3

Ensuring the Safety of Automated Driving

Technological Development and
Education for Enhanced Safety(Overview)
Osamu Hosaka, Hiroaki Kimura (Cabinet Office)1

Background and SignificanceEnsuring safety and reliability is the most important issue
in the practical implementation and deployment of
automated vehicles, and there is an urgent need to establish asafety assurance method for automated vehicles. In addition,communication between automated vehicles and othertraffic participants must be conducted flawlessly. Therefore,
SIP-adus (Cross-ministerial Strategic Innovation Promotion
Program (SIP) Automated Driving for Universal Services)
decided to address the following three issues for technologicaldevelopment for enhanced safety.(1)(i) Building of safety assurance environments in virtual environments(ii) S ustainable and effective measures against cyberattacks onconnected cars (cybersecurity)(iii) Establishment and dissemination of appropriate
methods for communicating the intentions of automatedvehicles to people, and effective education and awarenessmethods for people using automated vehicles andautomated driving services (safety education)
Currently, addressing the building of a safety assurance
method for automated driving is also the matter of greatest
interest globally, and various initiatives are in progress. SIPadus is also conducting safety assurance on public roads,
such as the FOTs (Field Operational Tests) in the Tokyo
waterfront area, and these driving tests on public roads andtesting of actual cars at testing sites are also important, but
evaluation via simulations that are reproduceable and that
can also generate critical conditions are essential as well.
Therefore, SIP-adus is particularly focusing on the evaluationof sensor performance, with the aim of building simulation
models which are highly consistent with real environments andcan be an alternative to test evaluations in real environments,and has commenced the development of a safety assurance
simulation platform in a virtual environment that can perform
evaluations under various conditions based on the relevant
models.
Regarding cybersecurity, in the first phase of SIP-adus,
the establishment of a method for evaluating the level ofcybersecurity protection against cyberattacks from outside
the vehicle was addressed, and an evaluation method for
during the development phase was developed. On the other
hand, because cyberattack technology continues to evolve, asystem to detect and monitor cyberattacks during vehicle
operation after a vehicle is on the market is also necessary.
Currently, Intrusion Detection Systems (IDS) for
cyberattacks against vehicles by third parties in bad-faith aregarnering attention as countermeasures against this issue.
The development of a method to evaluate the performance of
IDSs was addressed in the second phase of SIP-adus.
In addition, as a human-machine interface (HMI) issue,
research and development was conducted on the appropriate
indication and education methods to avoid communication
errors when automated vehicles encounter other vehicles,
pedestrians, etc. in mixed traffic.
The research and development of all three of these issues were
progressed in cooperation with the German Federal Ministry of
Education and Research (BMBF) under the Japanese-German
cooperation framework. [See Section 6 2) for more details]

2

Building of safety assurance
environments in virtual environments

Because the current evaluation method, which is centeredaround FOTs using actual vehicles on public roads, does not

104

SIP通し.indb 104

2023/03/01 16:47

3 Ensuring the Safety of Automated Driving
﻿Technological Development and Education for Enhanced Safety (Overview)

Experimental Evaluation
Public Road

Virtual Evaluation
SILS／MILS(Software in the Loop／Model in the Loop)

Proving Ground
Connect

Connect

HILS(Hardware in the Loop)
Camera

Radar

LiDAR

VILS
(Vehicle in the Loop)

Construction of sensor models that are highly
consistent with real phenomena

Fig.1: Consistency verification of DIVP’s sensor model

modeled based on the principle of reflectance properties,
and consistency is verified by comparing the results withthose of actual vehicle tests. In addition, there are "virtualenvironment models" which correspond to cameras, radar,and LiDAR respectively to make the simultaneous evaluationof these sensors possible. (Fig.2)

Virtual model

Actual physical model

Verification
SourceLight

TargetTrans

Ray tracing

Environmental model
3D modelReflection/Refr
actionSpatial rendering
SourcePropagation

Recieve

Sensor Model
Perception

Recognition
Physical value
outputVerification

allow for the intentional setting of the necessary driving
environment conditions, which makes it difficult todetermine whether or not an automated vehicle meets therequired safety and to evaluate all of the various events that
occur on public roads using actual vehicles, there was anurgent need to develop a method to assure the safety of
automated vehicles under specific driving environment
conditions. In addition, it is considered necessary to develop
simulation tools that focus on sensor performance evaluationin order to improve the efficiency of safety assurance using
actual vehicles, which takes an enormous amount of time inthe current development of automated vehicles.
Therefore, SIP-adus program to develop simulation
models which are highly consistent with real phenomena
and can be an alternative to test evaluations in realenvironments, in order to conduct safety assurance withhigh reproducibility under various traffic environments.
(Fig.1)A consortium consisting of three universities (Kanagawa
Institute of Technology, Ritsumeikan University, and Toyota
Technological Institute) and ten companies (Hitachi Astemo,Sony Semiconductor Solutions, Denso, Toyota TechnicalDevelopment, Pioneer, Biprogy, Mitsubishi Precision, Soken,
Solize, and Yushin), led by Professor Hideo Inoue of the
Kanagawa Institute of Technology, and the all-Japan
organization commenced the building of the Driving
Intelligence Validation Platform (DIVP®), a safety assurance
environment platform in a virtual environment. This "virtualenvironment model" is a model that enables the safetyassurance of an automated driving systems by modeling thegeometric information and reflectance and spatial
propagation properties of objects under traffic environment
scenarios, and detecting them with a refined sensor model(Fig.2); by changing the various environmental factors suchas time of day, the weather, vehicles, and infrastructure, it ispossible to evaluate a system under any conditions.
In this model, in order to accurately simulate the sensor
outputs, the detection principles of each sensor and physicalphenomena in the electromagnetic wave band used are

Allocation of improvement requests

Consistency verification andenhancement research based on
sensor output comparisonsFig.2: Virtual environment model and sensor model

From FY2021, we have cooperated with the AD-URBAN(FOT project of Automated Driving system Under Real city
environment Based on Academic researcher’s Neutral
knowledge) step being undertaken by the Kanazawa
University consortium to share scenarios where there wasrecognition failure in FOTs of LiDAR and cameras, evaluate
the reproducibility in a virtual environment, verify thedifferences in recognition between the real environment andvirtual environment, and improve the accuracy of the model.
[See Section 3 2) for more details]
In addition, in order to ascertain the future needs ofcustomers and to identify issues regarding the continuous
operation of the DIVP data platform, a scenario package wasprepared that stimulated sensor weakness scenarios in theTokyo Waterfront City area, invited participants fromautomobile manufacturers and sensor manufacturers, andconducted a monitoring evaluation in two separate steps.
The results of the monitoring evaluation were generally

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 105

1052023/03/01 16:47

3 Ensuring the Safety of Automated Driving
﻿Technological Development and Education for Enhanced Safety (Overview)

Organization of simulation types according to purpose

Customer
environment(existing Sim)

SILS/HILS

Scenario

Environment

Traﬃc flow
scenario

Map data

SpatialrenderingSensor
Perception
Recognition

Sensor model (true value model)<PointMILS/SILS

Want to utilize existing
assets and own scenarios

SDMG®
DIVP-Format

DIVP-I/F

Want a virtual environment output for
sensor development and recognition
learning

User control
model

User vehicle
model(MATLAB/Simulink, etc.)(CarMaker/CarSim/
ASM etc.)

Want to develop and evaluate systems
and sensors

DIVPⓇ "Environment, Space Rendering,Sensor" modelSensing Weakness
Scenario

Automated vehicles
Fusion/Control
Vehicle control

Reference
control model

Reference
vehicle model

DIVP-I/F

LegendConnection using standard IF

Fig.3: Connectivity between customer environment and DIVP

favorable, and the simulation experience conducted in STEP1 was rated higher than other simulators in terms of thereliability of the simulation and the adequacy of the assets. In
STEP-2, several companies from among the participants in
STEP-1 were selected and a detailed evaluation verification
was conducted on the connectivity of the system with various
existing simulation environments, the output of simulation
results when the scenario environment was arranged basedon the virtual environment that had been prepared, and theconnectivity of the system with the participants’ various
models and systems. (Fig.3)
Automated vehicles utilize a complex combination of
sensors such as cameras, radar, and LiDAR, and it is essential
to evaluate them via sensor fusion. In this case, evaluation of
combinations of various sensors is necessary, andstandardization of the interfaces is important to realize thisin a simulation. For this reason, the standards of the Germanbased standardization organization Association forStandardization of Automation and Measuring Systems(ASAM) were used to ensure compatibility with existing
simulations.
In addition, SIP-adus started a cooperative project called
the VIrtual Validation methodology for Intelligent Driving
systems (VIVID) with the German research project VIVALDI
in October 2020, and the standardization of safety assurance
systems and simulation interfaces for automated driving byorganizations such as the International Organization forStandardization (ISO) and ASAM while strengthening
international cooperation efforts was promoted throughthese activities. [See Section 6 5) for more details]
On the other hand, the establishment of safety assurance
methods for automated vehicles requires verification of thevalidity of scenario definitions and evaluation criteria. To
accelerate the practical implementation of safety assurance
technology, DIVP has cooperated with the Safety Assurance
KUdos for Reliable Autonomous Vehicles (SAKURA) Project
being conducted by the Ministry of Economy, Trade andIndustry and the Ministry of Land, Infrastructure andTransport, based on a proposal by the Safety Assurance
Subcommittee of the Japan Automobile Manufacturers
Association (JAMA), and established a working task force andsteering committee to jointly promote the project in FY2021.
As a result of the DIVP activities over the past four years,
the practical implementation of a platform that enables
streamlined scenario generation, recognition performance
evaluation, and vehicle control verification is in sight, andV-Drive Technologies, a new company (100% owned by
Biprogy) that provides DIVP’s research results as a tool chain,
was established in July and started general sales in September.

3

Cybersecurity

Automated vehicles need to acquire road traffic
environment data such as high precision 3D maps and traffic
signal information via communications. Cyberattacks are amajor threat to automated vehicles and other vehicles with
communication functions, and it is extremely important totake cybersecurity measures in anticipation of cyberattacks.
Several cases of cyberattacks on vehicles have already beenreported at international conferences, and Intrusion
Detection Systems (IDS) are attracting attention as acountermeasure against cyberattack methods, which
continue to evolve even after vehicles are put on the market.
On the other hand, since IDS is a countermeasure against
unknown cyberattacks, it is difficult to evaluate its
performance. Therefore, SIP-adus has worked on an
evaluation guideline for IDSs. In this project, in addition to
examining IDS in cooperation with an industry organization(JASPAR), IDS performance evaluation was conducted using
a testbed and actual vehicles, IDS evaluation methods were
established, and the formulation of IDS Evaluation
Methodology and Guidelines was completed. The results ofthis research were transferred to JASPAR in August 2022 andare planned to be used as industry guidelines in the future.
Although the number of vehicles with communication
functions is increasing each year, there have not been manyhacking incidents that concerned safety because they havebeen used mainly for entertainment purposes. However, thenumber of such incidents is anticipated to increase as
automated driving commences in the future and information

106

SIP通し.indb 106

2023/03/01 16:47

3 Ensuring the Safety of Automated Driving
﻿Technological Development and Education for Enhanced Safety (Overview)

obtained from communications is used for control.
Therefore, the active collection of threat informationfrom cyberattacks on connected cars was attempted, and thedevelopment of observation, collection, analysis, and storage
methods, as well as the definition of basic specifications for a
threat information sharing system to support initial response
activities was conducted.
The basic specifications of this threat information sharing
system will be transferred to an industry organization(J-Auto-ISAC) at the end of FY2022, and the future
improvement and utilization is also planned. (Fig.4)
Competitive Area

V-SOC*

Cooperative Area

Formulation of IDS evaluation methodology andguidelines (Activity a)The operational transfer of the
IDS evaluation guidelines to an industry association
was completed in August 2022.
Surveillance

Cooperation

PSIRT*

Initialresponse

information
sharing

Research on threat information collection
and initial response support for connectedcars (Activity b)The final goal isoperational transfer of the basic system
specifications, etc. for initial support forincident response to an industry
organization by the end of FY2022.

IDS

AttackAttackdetection

ThreatInformation
Sharing System

Attack* V-SOC: Vehicle-Security Operation Center. Monitors vehicles and responds in cooperation with the PSIRT when an incident is detected.
* PSIRT: Product Security Incident response Team. Receives incident reports from V-SOC and third parties and responds.

Fig.4: Scope and final goals of the activities (in red, within the dotted lines)

Regarding this project, based on the cooperation
framework between SIP-adus and BMBF, a cooperation
project called SAVE (Securing Automated Vehicles) wascommenced in October 2020 with the German research
project SecForCARs (Security For Connected, AutonomousCars), and international cooperation was promoted throughstudies, etc. of threat information sharing systems. [See Section
6 7) for more details]

guidelines. The initiatives focused on the following three
details as the cooperative area related to HMI.
Assuming movement and logistics services that use(1) 
automated vehicles equivalent to SAE Level 4TM, smoothcommunication methods that ensure the safety of
automated vehicles and surrounding traffic participants(pedestrians, bicycles, drivers of other vehicles, etc.) andthat allow them to clearly understand each other’s
intentions were derived. [See Section 3 5) for moredetails](2) An HMI was developed for the appropriate taking over ofdriving when the driving environment conditions deviate
or when the function of the automated driving systemdeteriorates, and education methods for drivers were
derived. [See Section 3 6) for more details](3) As a preliminary step before the introduction and spreadof vehicles with automated driving at SAE level 3TM or
higher, the knowledge that drivers, pedestrians, etc.
should acquire regarding driver assistance systems
equivalent to SAE level 2TM and effective education
methods for them were derived. [See Section 3 4) for
more details]
Regarding this project, cooperation with a German
research project was commenced in July 2019, based on thecooperation framework between SIP-adus and BMBF, andthe setting of research topics and verification of the validityof the research methods for the three issues mentioned above
was progressed. In addition, the results were widely
disseminated worldwide, and international standardization
was conducted while strengthening international cooperation
efforts. [See section 6 4) for more details][References]
(1)

Cross-ministerial Strategic Innovation Promotion Program (SIP)
Automated Driving for Universal Services R&D Plan, https://www8.

4

cao.go.jp/cstp/gaiyo/sip/keikaku2/4_jidosoko.pdf, (accessed 30th June

Safety education

2022)

Road safety is ensured by the three complementaryelements of people, vehicles, and the traffic environment, and
it is difficult to ensure safety through vehicles alone. Currently,
various driving safety support systems such as Autonomous
Emergency Braking (AEB) are widely used and are steadily
contributing to the reduction of accidents, but conversely,
accidents caused by overconfidence are sometimes seen. It istherefore necessary for all traffic participants to have a correctunderstanding of automated driving. Regarding this issue,
SIP-adus investigated the ideal HMI, including appropriate
indication and education methods, while taking international
trends into consideration, and conducted studies toward thedevelopment of the necessary technology and creation of[Contacts]
Secretariat of Science, Technology and Innovation Policy Cabinet Office,1-6-1 Nagata-cho, Chiyoda-ku, Tokyo 100-8914, 03-6257-1334, Contact:
Osamu Hosaka, Hiroaki Kimura

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 107

107

2023/03/01 16:47

3

Ensuring the Safety of Automated Driving

1) Development of Driving Intelligence
Validation Platform (DIVP®) for
Automated Driving Safety Assurance
Hideo Inoue (Kanagawa Institute of Technology)(Abstract) As automated driving systems become more complex, there is a need to ensure the safety of countlessdriving environments. However, safety validation on current automated vehicle relies on comprehensive results
evaluations in real driving environments and are quite costly in terms of people, material, money, and time. It is alsodifficult to test the physical perception and recognition performance limits of camera, radar, LiDAR, and other suchexternal sensors with respect to physical phenomena that occur in the real world. In other words, there is the issue of
how far to go in constructing systems to guarantee safety (How safe is safe enough?) Against this backdrop, thisresearch project will build an assessment platform for virtual simulations that can achieve "highly consistent modelingof driving environments, spatial propagation, and sensors" with real phenomena, as is needed to conduct automateddriving safety assurance. The goal is to enable detailed and efficient safety assurance for automated driving systems
under a large number of scenarios.
Keywords: automated driving, virtual environment, external sensor, virtual simulation, safety assurance

1 Project Background and Overview
The National Highway Traffic Safety Administration(NHTSA) of U.S. Department of Transportation has investigated
accidents involving self-driving vehicles and reported cases inwhich the sensors failed to detect objects and detected objects
were not properly recognized.(1)(2) Currently, a variety of different
approaches to safety assurance are being tried in countries
around the world. As a typical example, the PEGASUS project,founded by Germany's Bundesministerium für Wirtschaft undEnergie (BMWi) and its successor, the SET Level project,
propose validation and verification methodologies based on
driving scenarios.(3)(4)Through these activities, it is essential to utilize a simulation
evaluation environment to evaluate the safety of automatedvehicles under weather conditions that are difficult to reproduce
and accident/near-miss conditions that are dangerous toreproduce. In particular, whether sensors can correctly perceiveand recognize objects is important point and a simulation
infrastructure equipped with sensor models that are highly
consistent with real phenomena supporting these processes, isneeded. However, so far, there has been no full-scale researchand development of a simulation with high fidelity on sensors’
performance. As part of the second phase of SIP-adus, theDriving Intelligence Validation Platform ("DIVP®") project, a
research and development project undertaken by a consortiumof collaborating sensor manufacturers, software firms,

universities, and other organizations, was launched at the endof 2018. (Fig.1) The project aims to build a simulation platform
for automated driving safety evaluation in a virtual spacefocusing on sensor models that are highly consistent with actual
phenomena.
In this project, above-mentioned 13 organizations inindustry-academia collaboration (Fig.2), by connecting witheach other by utilizing their respective expertise, work on theconstruction of interface specifications for the safety evaluation
platform including automated driving control models in
addition to a series of virtual space models of "driving
environment-spatial propagation-sensors", with the aim ofcontributing to global standardization.

consistent sensor modeling
2 Highlywith real phenomena
External recognition sensors differ from standard vehicle
component models and play a dynamic role in connecting
driving environment models with automated driving control.
Conventional simulators put an emphasis on evaluating
whether or not system control functions properly, and sensor
models based on true value (normal function) are often used.
As mentioned above, automated vehicle safety assurance
requires identifying the strengths and weaknesses (limits) ofperipheral surveillance sensors and making improvements tosystem design, sensors, and sensory perception algorithms.

108

SIP通し.indb 108

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

図1 実現象と一致性の高いセンサモデルの必要性
Real vehicle test
Public Road

Virtual test
SILS／
／MILS

Proving Ground（Software in the Loop／／Model in the Loop））

Connect

Connect

HILS

VILS

（Hardware in the Loop））

Camera

LiDAR

Radar

（Vehicle in the Loop））

Highly Consistent Sensor Model

Source : Kanagawa Institute of technology, MITSUBISHI PRECISION CO.,LTD., DENSO Corporation, Pioneer Smart Sensing Innovations Corporation, Hitachi Automotive Systems, Ltd.

1

Fig.1: Necessity of sensor models highly consistent with real phenomenaRealenvironment

図2 DIVP® プロジェクトの構成

Real Physics
Based Virtualization

Open SCENARIO*2Test data Generator

Environmental
conditions

5

3

Temporalmodifications

Open DRIVE

Sensing weakness
scenariosearch algorithm(AI)

Virtual environment

Sensingweakness DB

Measurement & validation
Sensor model

Environment model
Environment2Road furnitureand rules

1

Road shape

Automated
control

SensorMeasurement & validation

Database
Sensing weakness
scenario database
construction

VehicleEnvironment

Space design

Visible light

Perception

Recognition

Camera modeling

Camera modelingSony Semiconductor
Solutions Corporation

Radar modeling*1

Test data
generating toolMillimeter-wave
Ray tracing*1

Recognition
Radar modeling*2

Perception

Fusion*2

Recognition

Perception

VehiclemaneuverRisk prediction model

Driving
Path planning

LiDAR modeling

Infrared light
Ray tracing

Performance Validation
 Intended performance Performance limits Sensing weakness
 Traffic disturbance Human errors*1

Perception

Path/Ray tracing

4 Moving object

Automated
driving model

VehicleMotion control

Recognition

Platform with standard I / F*1 Ritsumeikan finished Feb-2021, DENSO finished June-2021, Hitachi finished Sept-2021*2 TTDC, U-shin, Toyoda-univ joined Mar-2021

Fig.2: DIVP project structure

However, with true value-based sensor models, it is difficult toincorporate electromagnetic wave spatial propagation
validation results into models. This presents problems forincorporating into models the kinds of scenarios that sensors
struggle with. This project is taking the reflectance properties(retro, diffusion, specular reflection, etc.) and transmission
properties of millimeter wave radar waves, visible light used incameras, and near infrared light used in LiDAR, and is building
a physical model and designing a spatial propagation model for
ray tracing and other applications. Additionally, using
Peripheral structuressophisticated experimentation and measuring technologies, weare building a physical model for real phenomena that changes
based on peripheral environment effects such as rain, fog, andperipheral illumination such as sunlight. This project's focus on
incorporating spatial propagation properties as seen throughsensors into a model consisting of driving environments, spatial
propagation, and sensors based on electromagnetic principles
sets the project apart from other DIVP simulation platforms.
(Fig.3) Specific examples of each model are given below.

Space
Rain, fog, snow, etc.

Targets

Virtualenvironment
model seen
throughsensors

Camera

Radar

LiDAR

Light source
Ray tracing
Road surface
Precisely modelized sensor internals for more detailed perception output

Windshield

Image sensor

ISP*/recognition section

Photoelectric conversion
Photoelectric conversion
Raw signal processing
Raw Signal Processing

OutputInput

In-sensor
model

Lens

Fig.3: Sensor model for replicating spatial propagation (camera example)

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 109

1092023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

図6 精緻な反射を再現する3つのモデル（ミリ波レーダ）

2.1. Camera model
Rather than the RGB workflow which can show images tohumans, the camera model in the DIVP simulates spectral
properties that are input into CMOS and other semiconductors.
Moreover, sunlight has been formulated as a sky model,
allowing precise sunlight sources to be simulated by inputting
the time of day, latitude, and longitude. As shown below, objects
are given defined reflection properties and highly realistic
simulation images are achieved. (Fig.4)
図4 物標の反射特性の有無の影響
No Property

With Property

Flat, no color, no texture

Precise & real reflection duplication

Scattered model

Reflector model

RCS model

For small target (vehicle、person etc.)
Radar equation (distance 4th power)
Physical Optics approximation

For large target (Building, road surface etc.)
Friis equation (distance squared rule)
Geometrical Optics approximation

For analysis duration reduction
Radar equation (distance 4th power)

1st layer εr1 ，μr1

t1

Nth layer εrn μ，
rn

tn

2nd layer εr2，μr2
3rd layer εr3，μr3

Multi-layer model(Transfer Matrix Method)t2
t3

Input parameters

Dielectric constant,
Magnetic permeability, Thickness(from reflection characteristics)

Angle characteristics ofreflection characteristics

Bi-static RCS

Fig.6: Three models for replicating precise reflection (millimeter wave radar)

9

Fig.4: Impact from presence or absence of object reflection properties
Source : MITSUBISHI PRECISION CO.,LTD.

4This allows for replicating the darkness of a tunnel or thestrong sunlight backlighting seen when exiting a tunnel,
scenarios where obtaining visible light is problematic, and forvalidating that high dynamic range ("HDR") camera models
have sufficient visibility to discern objects. (Fig.5)
図5 HDRカメラの効果検証例

Normal (NML) Camera

High Dynamic Range (HDR) Camera

Fig.5: Sample image of Verification for HDR camera performance

Source ： Sony Semicondtor Solutions Corporion

2.2. Millimeter wave radar model
Millimeter wave radar is the sensor for which it is mostdifficult to create a model. We have defined three reflection
models according to the behavior of radio waves on illuminatedobjects, and use them for each object accordingly. The physical
optics approach is used as a setting model for small objects suchas vehicles and people, while the geometric optics approach isused as a reflection model for roads and other large objects. The
radar cross-section ("RCS") model uses predefined objects forpurposes such as shortening analysis time. Each model is used
for different scenario objects. (Fig.6)
The output of millimeter wave data is replicated well,including in the form of maps showing reflection intensity via
X-Y coordinates (heat maps), as well as X-Y reflection point
maps that show locations of strong reflection as target object
reflection points. (Fig.7)

Fig.7: Examples of millimeter wave radar model output

2.3. LiDAR model
Near infrared light, which is used in LiDAR, is a sensor thatis relatively easy to model due to its directivity properties. The
LiDAR model shown below can assess background light andother environmental disturbances by performing 360° scans. By
creating a model for scanning the near infrared light emitted by
LiDAR and precisely replicating things like object footprint
based on near-infrared light spread and the effects of sunlightand other background light, this project has made it possible toconduct simulations that are highly consistent with real
phenomena, even for LiDAR. (Fig.8)

Fig.8: LiDAR model

2.4. Sensor output consistency validation
For the inputs and outputs of each sensor model, this project
conducts quantitative consistency validations for simulation
models by comparing test results in real environment. (Fig.9)

110

SIP通し.indb 110

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance
図9 現象実験ベースの一致性検証の取組み(Fig.10)(1) Traffic participants and other objects(2) Peripheral structures(3) Road surfaces(4) Surrounding environment such as sunlight source, rain, fog(5) Interface between sensor and environment (windshield,
Radar snow and ice deposits, etc.)

SystemIdentification
Simulation
Modeling

Gap Analysis

figure10 DIVP®’s virtual world model overview(4)Surrounding environment(1)Participants and Objects

Correlation(5)Interface

Light sources

Traffic participants

Interface between sensors and space・Camera: windshield
・LiDAR and Radar: sensor surface

Verification test(2)Peripheral
Static objects

Rain, fog and snow(3)Road surfaces
Changes due to weather conditions (e.g. reflectance)

Fig.9: Efforts involving test-based consistency validation

Results show a high consistency with light source-related
scenarios in which sensor use is problematic, includingnighttime and back lighting such as from sunlight, as well as
conditions such as rainwater deposited on glass surfaces. For
falling snow scenarios and other phenomena difficult toreplicate, we are validating consistency with models based onelectromagnetic wave reflection and propagation analyses via
test measurement.

Source : DENSO Corporation, SOKEN, INC, MITSUBISHI PRECISION CO.,LTD.

9

Fig.10: Virtual environment from the perspective of sensors
10

3.2. Environment model development process
Fig.11 shows the main components of the DIVP simulator
structure. In addition to the OpenSCENARIO® and OpenDrive®
data reading for scenarios defined in Germany's ASAM
standard activities, there are seven characteristics of thestructure: (1) a file defining the physical properties of materials
validated based on reflection principles (commonly known as
DIVP materials), (2) SDMG® (Space Designed Model
Generator), (3) database of events with sensing weakness
scenario and tools for compiling those events and generating avirtual environment model, (4) environment and space designmodel, (5) sensor perception model and recognition algorithm,(6) standardization of interfaces connecting inputs and outputs
for each model, and (7) 3D model for objects and other assets.
Measurement and validation of reflection principles
involves compiling a physical properties file (DIVP materials)
as a simulation model consisting of reflex properties that arehighly consistent with real phenomena. This is achieved bydevising measurement systems for visible light, near-infrared
light, and millimeter waves, and accurately measuring physicalproperties. (Fig.12)

a virtual environment model
3 Building
to address sensor detection weaknesses
3.1. Defining a virtual environment model
The European PEGASUS project defines six levels of driving
environment scenarios and provides a high precision 3D map
used for automated driving. These have defined driving
environment data useful for automated driving control (selflocation, traffic flow, routes, etc.), but there are few virtualenvironment models for which physical properties such asreflex, transmission, etc. from the perspective of externalsensors have been defined. As virtual environment architecture
as it relates to things such as sensor reflection properties, thisproject defines the following five areas and aims to build avirtual environment model that incorporates properties such as
electromagnetic wave reflection, transmission, and attenuation.
Scenario
Scenario description
Dynamic content

Rendering(7)3D model(1) DIVP : Defined material property values verified based on the principle of reflection

DIVPⓇ（OpenDRIVE, CRG）

(1)Physical properties model(Reflection, attenuation, etc.)(2)SDMG(3)Sensing weaknesses
Scenario database

Perception

Recognition

AD control

Vehicle control(Map, object)(OpenSCENARIO)

Static content

Sensor fusion

Physical property

User Model

User Model

User Model

User Model

Interface : ROS,FMI/FMU, Simulink/S-funktion(4)Environmental and(5)Sensor
space designing models
perception(3) Original database based on JAMA and SIP-FOT data

Sensorrecognition

SensorfusionAD control

VehicleModel(6) I/F DIVPⓇ: I/F capable of outputting spatial, perceptual, and cognitive models

Fig.11: Structure of process from sensing weakness scenario description to virtual environment models generation

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 111

111

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

Fig.12: Millimeter wave measurement instruments
figure13 Space design model development process
Process1
Requirement
definition

Process0

Phenomenon analysis

 Analyze a phenomenon
 Investigation ontheoretical formula

Process2

 Measuring
 Definition ofmodeling target
phenomena Parameter Definition

𝐷𝐷𝐷𝐷𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚

Model development

 Investigation on modeling methods
 Reconstruction of theoreticalformula𝑁𝑁𝑁𝑁 𝑠𝑠𝑠𝑠 (𝐷𝐷𝐷𝐷𝑖𝑖𝑖𝑖 ) = � Model construction
 Implementation

𝑛𝑛𝑛𝑛𝑣𝑣𝑣𝑣

𝑁𝑁𝑁𝑁(𝐷𝐷𝐷𝐷𝑖𝑖𝑖𝑖 , 𝑣𝑣𝑣𝑣𝑗𝑗𝑗𝑗 )𝑗𝑗𝑗𝑗=1 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝑣𝑣𝑣𝑣𝑗𝑗𝑗𝑗 𝐷𝐷𝐷𝐷𝑖𝑖𝑖𝑖 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴

Theoretical formula of attenuation
0

Process4

Model design

Theoretical formula of particle size distribution

e.g. Rain attenuation and scattering

𝐾𝐾𝐾𝐾𝑟𝑟𝑟𝑟 = 10𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙10 𝑒𝑒𝑒𝑒 ∗ 103 �Process3

Reproductive
experiment𝛿𝛿𝛿𝛿𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝐷𝐷𝐷𝐷 𝑁𝑁𝑁𝑁 𝑠𝑠𝑠𝑠 𝐷𝐷𝐷𝐷 𝑑𝑑𝑑𝑑𝐷𝐷𝐷𝐷 [𝑑𝑑𝑑𝑑 𝐵𝐵𝐵𝐵 ⁄𝑘𝑘𝑘𝑘 𝑚𝑚𝑚𝑚�

Attenuation estimation

雨滴粒形分布からの推定減衰量
Estimated result from theoretical formula

実測減衰量

Real

𝛿𝛿𝛿𝛿𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝐷𝐷𝐷𝐷 =Attenuation

Theoretical formula of scattering
𝜆𝜆𝜆𝜆2
Re � 2𝑛𝑛𝑛𝑛 + 1 (𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛 + 𝑏𝑏𝑏𝑏𝑛𝑛𝑛𝑛 )
2𝜋𝜋𝜋𝜋
𝑛𝑛𝑛𝑛

Precipitation

Fig.13: Space model development process
Source : SOKEN, INC

133.3. Development process for space models for propagation, attenuation, etc.
Replicating sensor detection with precision requires
accurately replicating physical phenomena such as sunlightsources, rain, and fog. With cameras, for example, visible lightemitted from light sources it propagates space and reaches
object surfaces. After being reflected off of or passing throughsurfaces, propagates space and reaches the camera lens. Lightpassing through the lens undergoes photoelectric conversion,
becomes electrical signals, and gets subjected to different kindsof control. (Fig.3)
Light diffusion, attenuation, etc. Events which occur at thistime, such as light diffusion and attenuation, can have significant
effects on the light input into cameras and therefore requireprecise replication. This is the case with millimeter-waveband
radio waves in the case of millimeter wave radar, and with nearinfrared light in the case of LiDAR. This project is also definedby a development process for highly realistic propagation,
15
attenuation, and other space models, with a focus on five areas:(1) analyzing phenomena and studying theoretical formulas,(2) defining phenomena for modeling, (3) replicating andmeasuring phenomena, (4) designing detailed models based ontheoretical formulas, and (5) implementing models. (Fig.13)

3.4. Examples of nighttime road surface reflection
models (cameras）
Precisely replicating road surface reflection properties forheadlight light sources at night is important for camera and
LiDAR models. As light sources hit surfaces at a low incidence
angle when emitted from one's vehicle headlights, we optimized

reflectance as a retroreflective model based on test values. This
図15enabled
夜間路面の再現
camera simulation imaging highly consistent with realenvironments. (Fig.14)

Real

Simulation

Fig.14: Nighttime image replication via optimizationof headlight reflections on road surfaces

3.5. Examples of falling rain space models for LiDAR
This project is developing weather-related space models forsuch things as backlight, rain, and fog, which are difficult toreplicate and evaluate using real vehicles. For a space model for
falling rain, we are replicating and validating as a simulation
model the effects on LiDAR signals by fake points caused byraindrop reflection according to real phenomena based on
rainfall, as well as fake images caused by specular reflection on
wet road surfaces. (Fig.15)

3.6. Building on 3D asset models for virtual environments
As mentioned above, this project is partly focused onmodels for virtual environment from the perspective of sensors,
as well as sensor models to support those models. Through this112

SIP通し.indb 112

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

図16

雨がLiDARセンサに与える影響

LiDAR (Doubts due to rain)

LiDAR (Reflecting like a mirror) Rainfall
20mm/h Target vehicle inmirror on roadside Rainfall
180mm/h LiDAR error percept
vehicle in mirror

Fig.15: Effects of rainfall on LiDAR sensors (test)
Source : Kanagawa Istituteof technology, SOKEN, INC

Passenger vehicles (11 models)

Traffic signal

Motorcycles and special vehicles

Pedestrians and their belongings

歩行者

Traffic signs and construction equipment

NCAP dummies

Backlight

AnimalLarge vehicles (including towing)

Rainfall(Raindrops, Spatial Damping/Scattering, Doubt)

Night headlight

Fig.16: Replicable assets / space model expansion

has established to build up evaluation scenarios—(1) NCAP
and other such assessment, and (2) evaluations conducted via
modeling communities of the FOTs at Odaiba and C1
Metropolitan expressway (to be focused on sensing weakness
scenarios) —and work to create models consisting of driving
environments, space propagation, and sensor perception andrecognition. To create scenarios, it is important to establish
virtual environment units as package scenarios in line with
objectives, and conduct a series of reliable safety assurance
using these package scenario units. (Fig.17)

project, we will gradually measure the reflection properties ofobjects needed for various kinds of evaluation environments,
and will conduct comparative evaluations of real test values andsimulation values while expanding our scope to include DIVP
materials. For the 3D assets for the environmental models thatwill support these efforts, we are expanding these assets into alibrary that includes objects, roads, and peripheral structures.
Being expended upon are assessment scenarios from things likethe NCAP (New Car Assessment Programme), and highpriority asset models and space models for modeling of sensing
weakness scenarios with, i.e., the Odaiba region and C1: theinner circular route of the Metropolitan Expressway, where theautomated vehicle field operational tests (FOTs) are being
conducted. (Fig.16)

図18 DIVP®シナリオパッケージのロードマップ
Laboratory

PG : Proving GroundCG : Community Ground(Odaiba, Metropolitan highway C1)

Dynamic

Real traffic
environmental
Sensingweaknessconditionof DIVP simulator towards
4 Application
automated driving safety assurance
StaticNCAP

Just how many safety assurance scenarios can be planned
for remains a question to be answered, but assessment efficiency
can be greatly improved by establishing high-consistency
conditions in a virtual environment. For this reason, this projectBasicvalidation

Fig.17: DIVP package scenario roadmap
Source : DENSO Corporation, SOKEN, INC, MITSUBISHI PRECISION CO.,LTD.

18

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 113

113

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

Scenario creation, unit validation for each asset / sensor

CPFA-50
L1:Road Shape

CPNA-25

Closed-Loop validation

CPNA-75

CPNC-50

Scenario creation, unit validationfor each asset / sensor

CPLA-25

CPLA-50

No slopes, homogeneous / solid pavement, no irregularities (slopes, cracks, manholes),no features that could cause false sensor detection in the surrounding area―

L2: Objects / Rules
L3: Temporalmodifications

―

―

―

Obstacle vehicle――

Pedestrian (adult)
runs and crosses theroute from the driver‘sside (Far side).
At a vehicle width of
50% without braking.

Pedestrian (adult)
walks and crosses
the route from thedriver's side (Nearside).
At a vehicle width of
25% without braking.

Pedestrian (adult)
walks and crosses
the route from thedriver's side (Nearside).
At a vehicle width of
75% without braking.

Pedestrian (child) runs
and crosses the routefrom an obstacle (small
and large vehicle) fromthe opposite side of thedriver's seat (Near side).
At a vehicle width of
50% without braking.

Pedestrian (adult)
walking forward in thesame direction.
At a vehicle width of
25% without steering
operation to avoid acollision after braking or
FCW operation.

Pedestrian (adult)
walking forward in thesame direction.
At a vehicle width of
50% without braking.

L4: Moving objects

4m

6m

4m

AEB

FCW

2m

2m

75％

1.5m

25％

25％

1ｍ

2m
1ｍ

10m

10m

1m

1m

1ｍ

L5: Environmental
conditions

Day

Day/Night

Day/Night

Day

Day/Night

Day/Night

Fig.18: NCAP evaluation environment modeling
図20

4.1. Application toward NCAP assessment
As with advanced safety systems that have automateddriving control such as AEB (Automated Emergency Break),
ACC (Adaptive Cruise Control), and ALKS (Automated Lane
Keeping System), assessment protocols are defined in detailunder projects such as Euro-NCAP and J-NCAP. Although
there is a small amount of variation with regard to traffic
accident circumstances among countries, it is important tohave protocols that incorporate accident information, including
with respect to pedestrians, vehicles, and turning left or right atintersections, as well as scenarios such as cutting in and cuttingout with automated driving ALKS. It is safe to say that thissimulation can effectively replicate conditions such as whenpedestrians suddenly step out into the street at night, a scenario
where real vehicles are affected by peripheral light sources. This
project is gradually creating models from NCAP scenarios, and
30 scenarios for existing Euro-NCAP protocols have been madeinto models. (Fig.18)
Shown below is an example of a simulation that incorporates
a sensor model for a scenario where a pedestrian steps out fromthe shadow of a parked car. (Fig.19) Data can be output tocameras, Radar, and LiDAR simultaneously for one assessment
scenario, enabling dynamic assessment.

NCAP車両陰からの歩行者飛出しシナリオ時のシミュレーション
Scenario of Pedestrian Emerging
from behind Vehicle

Camera Simulation

Fig.19: Simulation of a scenario involving a pedestrian stepping out from theshadow of a NCAP vehicle

20

4.2. 
Application toward real-world environment(Odaiba and C1 the Metropolitan Expressway)assessments
For the following package scenarios, we worked on building
an environment model for Odaiba and the C1 Metropolitan
Expressway, where the FOTs are being conducted. (Fig.20)
As virtual environments that provide a comprehensive
combination of real world environmental factors (driving
environment, roads, geographical features, dynamic objects,
weather, etc.), these locations are useful in assessing scenariosthat sensors struggle with. In coordination with other FOT
projects that are part of SIP-adus, sensing weakness condition
data (e.g., places, images, recognition output) generated inthese real-world environments can be relayed to DIVP
simulators, and can be sublimated to virtual environments thatare assessed by a greater number of users as Virtual Community
Grounds. The following shows examples of assessments
conducted in virtual environments.

Fig.20: Odaiba driving environment model (camera simulation)

114

SIP通し.indb 114

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

4.2.1. Applying toward white line detection weakness evaluation on road surfaces coated withheat resistant paint (LiDAR)

4.2.2. Applying toward signal recognition limit evaluation (identifying rainfall conditions)

Using the DIVP simulator for Kanazawa University's
As part of automated driving field operational tests, ourautomated driving system ("AD-URBAN System," part of theresearch has identified a phenomenon whereby LiDAR
AD-URBAN Project), we evaluated recognition limits based on
struggles to detect white lines on road surfaces coated with heat
the rainfall levels of a signal recognition algorithm. We
resistant paint in front of the Odaiba Telecom Center (datareplicated severe rainfall conditions that are not easily replicatedmeasured as part of "the 2nd phase of SIP-adus Research on thein real-world vehicle evaluation by making modifications torecognition technology required for automated driving
these conditions as simulation parameters. We then derived
technology (levels 3 and 4)" Kanazawa University (5). We
evaluation conditions that arrive at signal recognition limits. In
determined that this is due to road surfaces coated with heatso doing, we were able to efficiently evaluate other algorithm
resistant paint tending to have stronger retroreflection
and improvement conditions with good reproducibility, andcomponents, and therefore approaching the reflection
were able to demonstrate the effectiveness of the DIVP
properties of white lines, and incorporated these findings into
simulator. (Fig.22)
the simulation model. Fig.21 shows simulation results. While
LiDAR can detect white lines on normal road surfaces (red■ Evaluation of signal recognition using real vehicles
lines in the Fig.), the simulation clearly shows that white lines→ Unclear recognition limits due to failure to encounterare not being detected on road surfaces coated with heat
heavy rainfall
resistantpaint.
図23 LiDARシミュレーション（台場駅前付近）

Normal road surface (white line → detectable(red line))■ Evaluation of signal recognition by DIVP simulator→ Evaluation of recognition limits by varying rainfall
according to parameters

Signal recognition failure
due to raindrops
confirmed

Thermal shielding paint (white line un-detectable)

Fig.22: Signal recognition limit validation via DIVP simulator

Fig.21: LiDAR simulation (area in front of Odaiba Station)
After contrast correction

23

Lv 1

Lv 1.4

Lv 1.6

Lv 1.8Lv1.9 ( Performance limit )

Lv 2

LiDAR
image

High
Correlation value

Matching
correlation
distribution

Low

The higher the correlation value (red), the easier it is to estimate self-location.

Fig.23: Validating LiDAR recognition limits at the road surface wetness level

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 115

115

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

4.2.3. Applying toward crosswalk recognition limit
evaluation (identifying road surface wetness
conditions)for how it can coordinate with other automated driving projects.

4.2.5. 
Evaluating safety when automated vehicles
turn right at intersections

As described in the previous section, we used a DIVP
simulator to evaluate the recognition limits of an algorithm formatching locations of sidewalks and other features through
LiDAR based on road surface wetness conditions in the ADURBAN system with points on an ortho map. We were able toreplicate wet road surfaces based on LiDAR output generated
via the DIVP and validate recognition limits according todifferent contrast levels. (Fig.23 Lv.1.9)

As another effort related to coordination with AD-URBAN,
we are working to replicate traffic conditions via DIVP
simulations from data obtained when real-world vehicles turnright at intersections, as well as replicate the ways in whichcameras, millimeter wave radar, and LiDAR perceive andprocess the world. This will lead to our developing performanceindicators for such things as recognition performance andsafety margin with respect to oncoming vehicles and otherobjects. As ours will be a synchronized simulation platform, it
should be able to handle safety assurance for both perception
capability and safety margin at intersections in a range of traffic
environments. (Fig.25) In the future, we will contribute tobuilding intersection evaluation scenarios and developing
algorithms for automated vehicles.

4.2.4. Applying toward evaluation of self-location
estimation algorithm robustness
The AD-URBAN system uses a self-location estimation
algorithm that conducts map matching using LiDAR. For thisrobustness verification, we created a virtual environment modelfor LiDAR output in the DIVP simulator that modeled a severe
condition wherein there were numerous parked cars lined up
along the road so as to obscure the road surface. As a result of
the assessment, we validated that AD-URBAN's self-location
estimation algorithm is always highly consistent with correct
values and has high robustness. (Fig.24) Consequently, as DIVP
can be configured with conditions that are difficult to replicate
in real-world spaces, it can contribute to system assessment.
It is therefore safe to say that SIP-adus is uniquely effective
DIVP® LiDAR & Camera Sim.

of the FOTs in the Tokyo
5 Results
waterfront area
5.1. Objectives and background
In preparation for commercializing DIVP and in order toevaluate the feasibility of using the technology in the business

LiDAR & Algorithm output

Effect on localizing accuracy

Parked vehicle
Hiding of the roadsurface occurs.
Estimated location(posterior probability density distribution)

Fig.24: Evaluation of self-location estimation algorithm robustness

Driving data

CameraMillimeter wave

Fig.25: Safety evaluation conducted via DIVP simulation for when vehicles turn right at intersections

116

SIP通し.indb 116

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assuranceof OEMs and suppliers while encouraging its use in customer
environments, we executed Step 1: Simulation trial in November
2021 then Step 2: Simulation evaluation in January 2022 as partof the FOTs in the Tokyo waterfront area.

5.2. STEP1: Simulation trial as FOTs in the Tokyo
waterfront area
As shown in Fig.26, we conducted a trial use of a simulation
model that replicated in a virtual space, the environment of theTokyo waterfront city area, a field in the Tokyo waterfront area
FOTs. The usability of model and the results of our simulations
were evaluated. 81 people from 56 companies, including OEM
companies, took part in this simulation trial.

Thermal coated roadWest of the Aqua City
intersection
Shade of the trees andwhite line
Near expressway
junction (westbound)

Aomi 1-chomeintersection
Green Yellow

Red

Arrow

5.3. STEP2: Simulation evaluation as FOTs in theTokyo waterfront area

Pedestrian crossing(in black outfit)
Manhole on roadsurface (eastbound)Fig.26: Sensing weakness scenario in Odaiba environments

To conduct the trial, we set up a special portal site where we
mainly provided technical data concerning DIVP characteristics,
particularly its consistency with real-world environments, andvideos showing simulation results. Many participants weremade aware of our model through our efforts to notify the highconsistency of DIVP using data from consistency evaluationthat measured to replicate physical phenomena.
The portal site also provided information through webinars.
Through a questionnaire conducted at the end of the webinar,
participants ask questions about the feasibility of using the
DIVP simulator in their business. The results are shown in
Fig.27. SDMG®, a tool for generating environment models andscenarios, shows promise for use in not just research but vehicle
development and design, with the possibility for expanding the
図26

アンケート結果サマリ

Which services SDMG canbe used for ?

Which services Simulation
PF can be used for ?
Vehicle Use

Vehicle Vehicle Use
Sales
7%
3%
Research

5%
VehicleDevelop.
/ Design
48%

34%
VehicleDevelop.
/ Design
50%

5%Operation/ Planning

26

Fig.27: Questionnaire results

use of simulations in product development operations. We also
confirmed the ability for SDMG to be used in real operations.
Simulation platforms also show similar promise for use inresearch and vehicle development and design. In addition, we
learned that, for research applications, simulation platforms
could be used in processes ranging from evaluation toadaptation in new system and algorithm research anddevelopment, as well as vehicle development and design.
For the write-in questions, some questionnaire respondents
said that the simulators had "been shown to be consistent andextremely useful," while others said things like "simulators weretool for quickly achieving what I want to do" and "I expect to geta lot of feedback from the users' point of view," these answers
suggesting the potential for further development of simulators.
The questionnaire results also highlighted the fact that
promoting the use of simulators for safety evaluation would need
to be an area of focus going forward. Feedback shows good result
the use of physical simulators to sensing weakness evaluation, so
we will continue to coordinate with efforts such as the research ofevaluation indicators while clearly communicating how thissimulation platform can benefit safety assurance.

In order to prepare DIVP for participating companies' work
environments, in Step 2 we deployed DIVP into theseenvironments and validated their connectivity. Using scenariosand environments that we modified based on prepared virtualenvironments, we attempted to link the outcomes of conducting
DIVP simulations with participating companies' various
models and systems. Furthermore, to promote further adoption
in participating companies, we are also working to ensure
connectivity among multiple existing simulation environments.
Eight companies participated in Step 2, and we conducted
DIVP evaluation in line with their individual needs. Specifically,
we validated the viability of data generation for machinelearning, conducted comparative validation with realenvironment test results using the companies' own recognition
models, and software coordination with Simulink®. (Fig.28)
Applicable cases Sensor DIVP virtual environment output
Data
generation forAI
training■ Massive image generation (realistic images)
MultiplecamerasIn-house
Camerasensormodel
Millimeter
evaluation wave radar

Scenarios...

Details of implementation■ Recognition SW (AI) development andvalidation
Recognition
Machine learning
SW improves
performanceand efficiency■ DIVP (camera and millimeter wave)
space rendering■ Performance evaluation of proprietary
camera recognition models and millimeterwave models (OEMs and suppliers)■ DIVP (camera and millimeter wave)
space rendering and perception

SW linkage is realized, and various
outputs of DIVP and various
simulations are coupledResearch
43%

SW linkageon
Simulink

CameraMillimeterwave radar

Other
SW

DIVP

Other
SW

5%
Operation/ Planning

Fig.28: Examples of user needs as identified through the FOTs
in the Tokyo waterfront area

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 117

117

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

Through these activities, we ascertained the needs of clientssuch as OEMs, suppliers, and universities for different types ofsystems with regard to the usage of DIVP virtual environments
and space rendering outputs. We have also conducted different
types of validations aimed at achieving automated driving
safety validations for DIVP virtual environments. Certain
validations are still being done based on requests fromparticipating companies.

cooperation and
6 International
standardization
Different countries are trying different approaches to safety
evaluation. It is a rivalry among powerful players: there is the
PEGASUS project completed in 2019 and SET Level projectthat followed on, both funded by BMWi, and there have also
been IEEE and ISO standardization and other developments by
the U.S.'s Mobileye and Germany's BMW based on SaFAD.
China is also planning its own safety assurance framework.
(Fig.29)
Meanwhile, due to a number of automated vehicle accidents
that have occurred in the U.S., the country's American

Automobile Association (AAA) needs to conduct proper,
objective safety evaluation that satisfy consumers, throughmeasures that include reducing accidents by deploying
automated driving technologies, and is disrupting the previously
system-based, system builder-oriented approach to safetyassurance.
While these efforts are happening overseas, DIVP project aspart of SIP-adus makes efforts for joint research on safety
evaluation through Japanese-Germany cooperation, while
taking part to ASAM (Association for Standardization ofAutomation and Measuring Systems) for standardization
activities based on the outcome of this cooperation.

6.1. Japanese-German cooperation VIVID project
DIVP and Germany's VIVALDI project are being jointly
conducted as an International cooperation project known as
VIVID, which aims to contribute to safety assurance for
automated driving based on sensor modeling. The project,launched in October 2020 with support from SIP-adus andGermany's Bundesministerium für Bildung und Forschung(BMBF), is currently developing a standardized automateddriving safety evaluation system and interface. (Fig.30)
At a joint meeting between Japan and Germany held in■
Study of the Safety
Envelope approach for AVs■Germany:▷ Germany, as chair, led thedevelopment of ISO 21448(SOTIF) and ISO 5083
(replacing ISO 4804 by
SaFAD) .
▷ Government-led projects:

PEGASUS, ongoing V&VM
and Set Level 4 to 5; Sim
evaluation of theeffectiveness of preventive
safety technologies of
VIVID:P.E.A.R.S., which is
being conducted incollaboration with Japan[Third-party evaluation organization]■: Develop related test
methods and standards for AVs.
■
: Develop standards for
scenario-based safety arguments■: Rating tests for LKA and
ACC■
Creation of "Framework for
Automated Driving SystemSafety■
Release of IEEE P2846(RSS-based standard)

See RSS, SFF for AV Safety Assurance

Based on RSS references■: Proceed with various
rule making based on RSS*2■: Develop standards forevaluation procedures for ICV*1 Level
3 and above in China▷SaFAD*3 => released in ISO 4804
▷Released in IEEE P2846=>Referenced to FRAV and VMAD
under GRVA in WP 29■:
Development of SFF*4■
China-ITS: Create
V1.0 of AV standard■: Develop China
SOTIF for intra-China traffic
environments=>Could be translated to ISO.
■ : Chair of WG9 to develop
ISO 34501-34504■: Launch of AVSC (Automated Vehicle Safety Consortium)
and IAMTS (International Alliance for Mobility Testing &Standardization)■: Release of UL 4600 (principles and processes forevaluating safety at the system level)

Fig.29: International trends study results

Figure 30*1 ICV : Intelligent and Connected Vehicle. *2 RSS : Responsibility-Sensitive Safety.
*3 SaFAD : Safety First for Automated Driving. *4 SFF : Safety Force Field
Source: Prepared by Deloitte Tohmatsu Consulting LLC based on publicly available informationJT1- Toolchain -JT2- Scenario -JT3- Sensor Model -JT4- Framework & metrics-

VIVALDI
DIVP

Outcome Build VILS under the DIVP
environmental model and transmit
data to the VIVALDI simulator via
OTA Proceed with preparations for a
joint study in view of standardizing
the data format and I/F

 Exchange sensing weakness
scenarios

Implement mutual data exchange
between the DIVP environmental
model and VIVALDI sensor model Agree on standardization of openmaterials

Proceed with preparations for a joint
study in view of standardizing thesensor model I/F Compare the process andevaluation method for mutual
understanding Proceed with the formulation anddefinition of AD safety assurance
evaluation standards as VIVID going
forwardFig.30: Organization for international cooperation and international standardization

118
1

SIP通し.indb 118

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

Berlin in June 2022, automated driving project stakeholders inboth countries were informed that the project had made majorachievements, including having had an impact on theimportance of sensors and environment models. (Fig.31)

scenario description, reflective properties for different
environment models, as well as I/F connecting different models
such as sensors, which are included as property in DIVP. As for
I/F related to camera input, our involvement to the organization
was ended up with standardizing the technology as OSI ver.3.0.
This represents an epoc-making achievement that largely
owes to the groundbreaking nature of DIVP and the winning of
supporters through the VIVID project. Further efforts will bemade toward research, development, and international
standardization with the goal of standardizing AD safety
evaluation. (Fig.32)

6.2. Participation in the international standardization organization ASAM (Germany)
In response to ASAM, through which a broad range ofstandardization efforts are being made with regard to automateddriving, DIVP project stakeholders are participating in major
working groups for technologies such as OpenDRIVE/OpenSCENARIO and OSI® (Open Simulation Interface). In
these working groups, we are issuing proposals forstandardization of such technologies linked to sensing weakness

図30

日独シンポジウム概要“German-Japanese Symposium on Safety Assurance forConnected and Automated Driving”

Representatives of funding institutions, key note speakers oflarge scale validation projects and event participants
Dr. Stefan Mengel
Head of division, electronics
and automated driving, BMBF
Mr. Reinhold Friedrich
Deputy Head, electronics andautomated driving, BMBF
Ernst Stöckl-Pukall,
Head of Unit Digitisation andIndustry 4.0, BMBK
Mr. Benjamin Engel
Global technology manager,
ASAM

Mr. Seigo Kuzumaki
SIP-Adus Program Director

Mr. Shigekazu Fukunaga
Director, ITS and Autonomous
Driving Promoting Office,
METI

Ministries
PJ
Research
Institutes
OEM
Mega
Suppliers

Vendor30

Figure 32

Fig.31: Organization for international cooperation and international standardization

Standardization status at the international cooperation
framework and ASAM

VIVID spirit of innovation
 Discussion on standardization of scenarios, materials, and I/F

 Overall image of standardization initiatives through collaboration with VIVID
Standardization process

Scenario
Sensing weakness

Geometric
movement(1) Scenario
Align with standard OpenScenario
 OpenDrive
Scenario format
standardization for
sensor weakness
 OpenScenario
 OpenLabel

Env. Model

Sensor Model

Current status
Scenarios

Fusion / AD Model
R&D

Property
PropertySpace
designPerception

Recognition

FusionSimulation technology and toolchain R&D

AD
Model

SDMG
SDMG

(2) Material
Standardization ofreactive property
format of object
surface for sensor
validation
 OpenMaterial

CameraRecognition I/F
RADAR
Materials
I/F

Commonality(3) Camera Perception I/FStandardized spectrum I/F
required for camera input
 OSI3.0(4) RADAR I/FStandardized sensor model
I/F retaining condentiality ofspatial design
 OSI4.0

VIVID

Complementarity
Safetyassurance
evaluation
strategyplanning

Definition ofstandardization
scopeStandardi
zationDiscussion
among experts

Agreement on
formatProposal
acceptance

✓ ✓

✓

....

Standardization
complete✓Fig.32: Japanese-Germany symposium overview

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 119

1192023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿1) Development of Driving Intelligence Validation Platform (DIVP®) for Automated Driving Safety Assurance

7 Conclusion
As a part of SIP-adus, this project has seen research anddevelopment efforts made through industry-academiagovernment collaboration. Our goal is to capitalize on the
newness of models consisting of driving environments, space
propagation, and sensor, ensure connectivity with othersimulators, and develop core technologies for the efficient andwidespread execution of increasingly complex automateddriving safety validation. All DIVP project members fervently
hope that virtual environment simulation technologies that cancreate detailed digital twins of real environments will makeconsumers more confident about automated driving safety, andpromote more automated driving in the society.

[References]
(1)	ODI RESUME: U.S. Department of Transportation, National Highway Traffic
Safety Administration, 01/19/2017,https://static.nhtsa.gov/odi/inv/2016/
INCLA-PE16007-7876.pdf, (see June 2021)(2)	Nikkei Business: The Difficulties of Communicating the Limits of Technologies(2016), https://business.nikkei.com/atcl/report/15/264450/071500039/, (see June
2021)(3)	SET Level Project Office (Deutsches Zentrum fuer Luft- und Raumfahrt e. V.
(DLR), German Aerospace Center (DLR), Institute of Transportation Systems):
SET Level Project, https://setlevel.de/en/project, (see June 2021)(4)	PEGASUS Project Office (Deutsches Zentrum für Luft- und Raumfahrt e. V.
(DLR), German Aerospace Center, Institute of Transportation Systems):
PEGASUS METHOD An Overview, https://www.pegasusprojekt.de/files/tmpl/
Pegasus-Abschlussveranstaltung/PEGASUS-Gesamtmethode.pdf, (see June
2021)(5)	Naoki Suganuma (Kanazawa University): "the second phase of SIP-adus-Research on the recognition technology required for automated driving
technology (levels 3 and 4)", (2020)[Contacts]
Advanced Vehicle Research Institute, Kanagawa Institute of Technology, 1030
Shimo-ogino, Atsugi, Kanagawa 243-0292, 046-291-3277, Contact: Hideo Inoue(Professor/Director)

120

SIP通し.indb 120

2023/03/01 16:47

3

Ensuring the Safety of Automated Driving

2 ) R esearch on the Recognition Technology Required
for Automated Driving Technology (Levels 3 and 4)
Naoki Suganuma, Keisuke Yoneda, Ryo Yanase, Akisue Kuramoto (Kanazawa University),
Takayoshi Yamashita, Hironobu Fujiyoshi (Chubu University), Junichi Meguro (Meijo University)(Abstract) Automated driving equivalent to Level 4 in urban areas requires advanced and autonomous recognition
functions using on-board AI, as well as road facilities, communication facilities, and other infrastructure to support
these functions. At the same time, the installation of road and communication facilities throughout Japan requires
massive budgets, so it is necessary to study the minimum necessary infrastructure and recognition technologies.
Therefore, in this project, for the purpose of advancing discussions of future cooperative areas, we conducted
research and investigations on the recognition technologies and infrastructure that will be essential for automateddriving systems through public road tests of automated vehicles, chiefly through universities and other organizations
that are capable of publishing a certain amount of acquired data and the technologies used to acquire that data.
Based on the knowledge gained from the public road tests, we investigated the factors and issues that cause
recognition failures in (non-cooperative) automated driving systems, and evaluated the effectiveness of road traffic
environment data provided by the infrastructure.
Keywords: traffic light recognition, object recognition, Global Navigation Satellite System, localization, emergency vehicle recognition, virtualenvironment, recognition limits

1

2

Preface

Automated driving equivalent to Level 4 in urban areas
requires advanced and autonomous recognition functions
using on-board AI, as well as road facilities, communication
facilities, and other infrastructure to support these functions.
However, since massive budgets are required to install road andcommunication facilities throughout Japan, it is necessary to
study the minimum necessary infrastructure and therecognition technologies required under that infrastructure. In
this project, we collaborated with universities that are alreadytesting automated driving on public roads and with universities
that are conducting cutting-edge research on related elemental
technologies to develop the recognition technologies required
for an automated driving system equivalent to Level 4 incomplex urban traffic environments with a mixture of ordinary
traffic participants and other automated vehicles. In addition,
we evaluated the marginal performance of various recognition
technologies as well as the effectiveness of infrastructurecooperative driving.

Overview and Results of the FOTs
(Field Operational Tests)

2.1. Summary of the FOTs
This project aims to develop and verify recognition
technologies for automated driving systems and technologies tolink with infrastructure information such as V2I/V2N. We built
a test vehicle equipped with multiple LiDAR units, millimeterwave radar, cameras, and other sensors for the recognition of
the surrounding environment, GNSS/INS and other localization
sensors, and V2X on-board terminals, and then conducted
FOTs. Fig.1 (a) shows an overview of the test vehicle.
In July 2019, we began FOTs in the central part of Kanazawa
City, Ishikawa Prefecture, and in September 2019 we also began
FOTs in the Tokyo waterfront area. In the Tokyo waterfrontarea, the vehicle ran for 182 days over a three-year period
through FY2021, covering approximately 3,212.8 km in
automated driving mode. Fig.1(b) shows the automated drivingin the Tokyo waterfront area.
Among the data collected in the FOTs, the driving data
under adverse conditions that tend to cause recognition failures
are considered to be useful data for general researchers who
study recognition algorithms, irrespective of the implementersof this project. For this reason, from the sensor data collected in

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 121

1212023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

LiDAR
Three cameras
Four microphones(side of vehicle body) (for traffic light recognition)

LiDAR (for all directions)
Eight cameras(for all directions)
LiDAR(lateral monitoring)
Nine millimeter wave radar units(for all directions)

GNSS/INS

information transmitted by wireless communication based on
V2I and V2N makes it possible to recognize the traffic lightstate from a greater distance. However, in the examination inthis project, it was assumed that the traffic light recognition
distance required to pass through an intersection isapproximately 120m, and it was found that both methods havesufficient recognition distance. On the other hand, in situations
where traffic lights are not physically visible, recognition using
an on-board camera becomes difficult, and V2I/V2N are
considered effective in such situations.

Wheel speed sensor[m](a) Installation of sensors in automated vehicles

Recognition distance

350

More distant recognition is possible

300
250
200
150

After passing(out of visual range)

V2N

100

V2I

50

Camera0
540

535

the various areas in this project, a dataset (AD-URBAN Open
Image Dataset v1) that includes sensor data under adverse
conditions such as backlighting, background assimilation,
concealment, nighttime, rainy weather, etc., which tend tocause recognition errors, was constructed. It was made available
beginning at the end of March 2022 to domestic institutions forthe limited purpose of research activities.(1)2.2. Evaluation of the Effectiveness of V2I/V2N Road
Traffic Environment Data
In the FOTs in the Tokyo waterfront area, it was possible toacquire road traffic environment data provided by wirelesscommunications from infrastructure such as V2I (Vehicle to
Infrastructure) and V2N (Vehicle to Network). Therefore, in
addition to evaluating automated driving functions, we also
conducted and evaluated infrastructure-cooperative drivingusing V2I/V2N.
V2I/V2N provide various types of road traffic environment
data. In this project, we specifically evaluated the traffic lightinformation provided by V2I/V2N and the emergency vehicle
location information provided by V2N.
Fig.2(a) shows the comparison of traffic recognition
distance between V2I, V2N and camera based recognition. The
results show that the distance at which the traffic light can berecognized (the recognition distance) is approximately 150m
when the autonomous recognition technology using the onboard camera is used. At the same time, the traffic light

[s]

555

Elapsed time

560(a) Comparison of traffic light recognition distance between
V2I/V2N and camera(b) Automated driving in the Tokyo waterfront area

Fig.1: Overview of the automated vehicle developed in this project

550

545

Early deceleration

12
10
8

Suppression of
deceleration

6
4
2
0

6

4

2

0

-2

-4

-6

-8

-10

(b) Effectiveness of deceleration suppression in the dilemmazone using V2I with remaining time information
Fig.2: Results of evaluation of V2I/V2N traffic light information

In this project, we also developed a technology to facilitate
intersection entry by utilizing information on the remaining
time at traffic lights transmitted from V2I, and verified theeffectiveness of this technology. Specifically, as shown in
Fig.2(b), in the so-called "dilemma zone" where the traffic lightcolor changes from green to yellow/red just before entering anintersection and sudden deceleration is unavoidable, we
confirmed that the system can start deceleration before theactual traffic light color changes, enabling smooth decelerationand stopping before the intersection. Since sudden deceleration
in a dilemma zone always occurs regardless of whether driving
in automated or manual modes when the decision to enter anintersection is based solely on the current traffic lightinformation, the number of remaining seconds via wirelesscommunication is considered to be a useful piece of information
for safe entry into an intersection.

122

SIP通し.indb 122

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

Next, we show the results of an evaluation of the emergency
vehicle location information provided by V2N. In thisevaluation, simulated emergency vehicle locations were
distributed by V2N via a server, and their usefulness wasevaluated based on the information received by the vehicle.
Fig.3 shows an example of receiving the location of a simulated
emergency vehicle in the Tokyo waterfront area. The results
show that the location of the simulated emergency vehicle canbe determined smoothly at the most cases. At the same time, we
observed that an error of about 20m occurs in someenvironments, making it difficult to determine the location of
the simulated emergency vehicle in detail. Therefore, if we
assume that an automated vehicle must autonomously takeevasive action when approached by a simulated emergency
vehicle, it may be necessary to combine autonomous recognition
technology with on-board sensors to determine the detailed
location of the emergency vehicle. On the other hand, whenusing on-board sensors, we can assume that it is difficult toconfirm the approach of an emergency vehicle at a distance oroutside the visible range, and therefore, V2N simulated
emergency vehicle location information may be particularly
useful in such situations.
Displacement ofapprox. 20m
Behaves likemakinga left turning
rivorydlttua jec
Ac traPosition
information
accuracy
degradation

Taken from Google maps

Fig.3: Example of V2N simulated emergency vehicle location
information acquisition data

Evaluation of Recognition

3 Technologies

In the development of automated driving systems and theevaluation of their safety, it is important to confirm whether
automated driving functions operate correctly under actual
traffic conditions by driving automated vehicles on actual roads,
as was done in this project. On the other hand, evaluations in
actual road environments have limitations in terms of time,cost, and efficiency. In addition, it is difficult to evaluate themarginal performance of automated driving functions in realenvironments. For this reason, this project also evaluatedrecognition technologies using virtual environments such as
DIVP® (Driving Intelligence Validation Platform), which was
implemented as another project. In this section, we present an
overview of the various recognition technologies developed inthis project, as well as the results of the marginal performance
evaluation using the virtual environments.

3.1. D evelopment of Traffic Light Recognition Technologies and Examination of Conditions That Make Recognition Difficult
Accurate recognition of traffic light state is important forsmooth intersection crossing in automated driving. In thisproject, we developed a state-of-the-art recognition algorithm
that maximizes the performance of traffic light recognition by
conducting technological development combining conventional
pattern recognition technology and the latest AI technology inutilizing the latest cameras developed for on-board applications.
In addition, we also evaluated marginal performance in failure
scenarios as found in the FOTs using the virtual environment
developed in the DIVP project to look at situations whererecognition proved difficult in the FOTs.
(1) Development of Traffic Light Recognition Technologies
The development of an algorithm for recognizing traffic
lights by image recognition using an on-board camera wasconducted from the viewpoint of the use of digital maps and thedevelopment of several types of image recognition algorithms.
When a digital map is used, it is possible to limit the recognition
range in the image, and thus false positives can be suppressedand high-speed processing can be expected. For the multiple
image recognition algorithm, we considered a method ofpattern recognition that focuses on features such as brightness
and the circular shape of lighted areas in the input image, and amethod of semantic segmentation that uses deep learning. The
AI technology is expected to enable recognition that takes into
account not only the local brightness information but also thesurrounding objects. We evaluated the performance of thealgorithm developed in this way on driving data from the Tokyo
waterfront area. Fig.4 shows an example of recognition resultsof traffic lights using the developed algorithm.

Fig.4: Example of traffic light recognition results(2) Issues Identified in the Tokyo Waterfront FOTs
Next, given the results of the FOTs in the Tokyo waterfrontarea, we analyzed the technical issues in the recognition of
traffic lights, which are mainly caused by environmental factors,
and evaluated the algorithm. As a result, as shown in Fig.5, we
confirmed that the recognition performance temporarily
degraded under the following environmental conditions:backlighting, forward lighting, concealment, backgroundassimilation, and nighttime. On the other hand, in many
situations, the influence on the decision of whether to enter anintersection is considered to be limited because the effects are

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 123

123

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

mainly temporary when approaching an intersection. At thesame time, due to the effects of backlighting, there are moments
when objects cannot be seen due to image saturation, which
may prevent appropriate decisions on whether to enter when an
event occurs at the start of an intersection. However, it was also
verified that the effects of such failures are limited in the image
when using an on-board grade camera, and therefore, it isdesirable to use information from other traffic lights when there
are multiple traffic lights at the intersection, or to use V2I/V2N
information when there is only a single traffic light at theintersection to assist in the decision. In the evaluation ofrecognition performance through FOTs, we demonstrated thatthe system can recognize green, red, and arrow traffic lights
within 120m, which is the target performance, more than 99%of the time by introducing a strategy of determining intersection
entry based on the status of multiple traffic lights.(2)(3) Evaluation of Marginal Performance in a Virtual
Environment
We worked to reproduce in a virtual environment the
problems that we have encountered in our FOTs. We reproduced
in a virtual environment situations such as the aforementioned
failure caused by backlighting and the rare situation shown in
Fig.6, in which sunlight reflecting off a building makes it
difficult to recognize a traffic light, and evaluated the consistencyof the reproduced images as well as the marginal performanceof the system. As a result, it was confirmed from the evaluation
results in the virtual environment that traffic light recognition
in rainy conditions, which did not cause recognition failures in
the actual FOTs due to only light rain, can fail under heavy
rainfall of several tens of mm/h (so-called downpour) (Fig.7).

3.2. Development of AI Technology Necessary forLong-Distance Object Detection
For safe and smooth automated driving, it is important toreliably detect vehicles, motorcycles, pedestrians, and otherobjects in the vicinity of the vehicle. For this reason, in thisproject we examined ways to improve the detection performanceof distant moving objects, which is an issue in moving object
detection.

(a) Concealment(b) Background assimilation(c) Nighttime(c) BacklightFig.5: Examples of scenarios in which traffic light recognition becomesdifficult, as confirmed by field operational tests in the Tokyo waterfront area(a) Actual environment(b) Virtual environment

Fig.6: Example of reproduction of reflected light from a building
in a virtual environment(a) Actual environment (light rain)(b) Virtual environment (heavy rain)

Fig.7: Evaluation of marginal performancein wet weather using a virtual environment

Development of Long-Distance Object Recognition(1) Technology
Based on the experience of the FOTs conducted to date, it isnecessary to recognize automobiles within 200m andpedestrians within 70m in urban traffic environments,
depending on the situation. Therefore, we developed arecognition algorithm to satisfy these requirements. First, since
the blurring of distant objects due to the on-board camera
being out of focus is a problem in image recognition using an
on-board camera, we developed an object recognition algorithm
that can deal with blurring, enabling more accurate detection of
distant objects with small pixel counts. In addition, since thesituations where long-distance recognition is required arelimited, such as at intersections to be passed, we also verified
that stable recognition is possible by using a digital map to limit
the recognition area in the image.
In LiDAR recognition, an object must be detected as a 3D
rectangular BOX from a group of observed points, but it is noteasy to stably recognize an object from a sparse group of distantpoints. Therefore, we verified that sensor fusion with therecognition results of camera images is effective in compensating
for the sparse groups of points in the distance. Using suchsensor fusion, we created a recognition algorithm that achieved
the target performance of 90% detection of automobiles within
200m and pedestrians within 70m in a normal environment.
(2) Issues identified in the Tokyo Waterfront Area FOTs

124

SIP通し.indb 124

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

Because object recognition by LiDAR and cameras
recognizes objects based on the overall image features of theobject based on the observed sensor information, recognition
can be difficult due to changes in environmental conditions
such as the arrangement of surrounding objects, degree ofconcealment, and weather conditions. Other issues include
background assimilation when an object is near a median strip
or a pole, and in the case of a camera, the effects of sunlight
conditions. In the case of causes of failures specific to certain
sensors, we can deal with them with the aforementioned sensor
fusion strategy, but in concealment scenarios, which are an
issue for both sensor types, recognition is inherently difficult
with on-board sensors. In the FOTs, we confirmed that the
failure scenarios in which concealment is a major factor
occurred during the judgment of approaching oncoming traffic
at right-turns at intersections. It is important to evaluate themarginal performance of the system in such situations because
the system must recognize vehicles farther away when passing
through intersections with many lanes.

under different conditions. Therefore, an evaluation of marginal
performance affected by blind spots was conducted using thevirtual environment being developed in the DIVP project. As
shown in Fig.8, the data generated is of a scenario in which thevehicle is turning right at an intersection and an oncoming
vehicle is approaching straight ahead in an environment inwhich there is also an oncoming truck waiting to turn right.
LiDAR and image data with different shielding rates weregenerated by changing the relative position of each vehicle.
The detection results of objects from the LiDAR and camera
sensor data for the intersection scenario generated in this way
were confirmed, and the marginal performance of each sensor
and sensor fusion was evaluated for different shielding rates asshown in Fig.9(a), (b), and (d). Fig.9(c) shows the distributionof the shielding rates of LiDAR and camera in the same frames.
From these results, we verified that the recognition performanceby sensor fusion will likely improve with different installation(3) Results of Blind Spot Evaluation in a Virtual Environment
In the FOTs, object recognition under the influence ofconcealment has been identified as an issue. However, becausesuch scenarios are encountered only momentarily whiledriving, it is difficult to create a large amount of driving data

Fig.8: Recognition of oncoming vehicles at intersections
in a blind spot environment

1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

Shielding rate: LiDAR

(2)

5
0-2

-75

-50

50

25

100

75-

Shielding rate: Camera

0.0

125

-

100

150

-

125

-

175

150

200

5

0-2

-

175

50

25-

(a) Detection rate by LiDAR

75

50-

100

75-

5

-12

100

0

-15

125

5

-17

150

0

-20

175

(b) Detection rate by camera

High shielding of both resultsin many false negatives
100
1.0
80

60

0.8

Lower camera
shielding than LiDAR

0.6

0.4

40

0.2
20

Shielding rate: LiDAR

0.0
0
0

20

40

60

80

100

(c) LiDAR/camera shielding rate distribution in the evaluation data

5

0-2

50

25-

75

50-

100

75-

5

-12

100

0

-15

125

5

-17

150

0

-20

175

(d) Detection rate by LiDAR/camera sensor fusion

Fig.9: Object detection results by sensor fusion in a blind spot environment

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 125

125

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

positions and observation methods for the sensors.

multi-GNSS, it was confirmed that a range of 30
cm@77%/1.5m@90% and 30 cm@57%/1.5m@92% could beachieved in Odaiba and Shinjuku, Tokyo, respectively. A
summary of the results is shown in Fig.10, which shows the
positioning method using vehicle trajectories as Conventional
1, and the method using Doppler to predict the initial RTK
search position as Conventional 2.
Next, we focused on determining locations that haveachieved a positioning accuracy of 0.3m or less, and confirmed
the possibility of achieving a positioning accuracy of 0.3m orless at 99% in the same evaluation locations. At the same time,
we examined the possibility of determining a positioning
accuracy of 0.3m, which is sufficient for automated driving, byutilizing the Centimeter Level Augmentation Service (CLAS),
which provides correction information broadcast from thequasi-zenith satellite MICHIBIKI. As a result, we were able toconfirm a positioning accuracy of 0.3m in 99% of the cases on acourse set in Odaiba, Tokyo.(3) A summary of the results isshown in Fig.11.

3.3. Development of Localization Technology Using
MICHIBIKI
In this project, while we used signals from GNSS (Global
Navigation Satellite System) satellites such as MICHIBIKI andINS (Inertial Navigation System), we also developed GNSS/INS
technology that can estimate positions applicable to automateddriving systems. In actual environments where automateddriving is used, there is a high possibility that the accuracy of
GNSS/INS will deteriorate because signals from GNSS cannot
be received. Therefore, we also examined the conditions underwhich automated driving systems using GNSS/INS can operate
stably.
(1) Study and Evaluation of GNSS/INS Applied to AutomatedDriving
In the development of GNSS/INS, we set two targets for theabsolute positioning accuracy required for automated driving: aposition estimation accuracy of 1.5m, which enables thedetermination of the travel lane, and a position estimation
accuracy of 0.3m, which makes possible the operation of an
automated driving system using only GNSS/INS. We also aimed
to achieve these goals by utilizing the quasi-zenith satellite
MICHIBIKI. First, to achieve a positioning accuracy of 1.5m, asa stable estimation method while still using a general-purpose
GNSS, we studied a method that maximizes the effectiveness of
GNSS Doppler. In addition, we developed a technique todetermine the achievement of 0.3m accuracy by utilizing
vehicle motion with a particular focus on the height direction.
In addition, to maintain highly accurate position estimationeven at locations where GNSS cannot be received, we conducteda study to improve the accuracy of dead reckoning (DR).
First, in the absolute positioning evaluation test using(a) Odaiba

1

North error [m]North［m］

0

-0.5

-1

−1500

−1000

−500

0

500

1000

East［m］(a) Results of reliability determination.

-1

-0.5

0

0.5

1

East error [m](b) Accuracy of positioning solution

Fig.11: Results of Reliability Determination in Odaiba

In addition, we improved the accuracy of DR, which isimportant for interpolating between points where highly
accurate position estimation is possible. First, we looked at theerrors in the DR sections and the factors causing them, andconfirmed that the initial azimuth angle has an effect in the caseof a 10-second DR. Next, to improve the performance of theinitial azimuth angle, we proposed a method to optimize theangle by using the FIX solutions for the past several tens ofseconds. In the evaluation test, we confirmed the effectivenessof the proposed method by achieving an error rate of 0.3m orless 87.7% of the time after 10 seconds of DR on an evaluation
course in an urban area (Fig.12).

90

90

80

+20％

87.7%

13% improvement
74.7%

70
60

60
50

● All conventional methods● High precision system in initial position● High precision system for initial orientation● High precision system for yaw rate
● High precision system for speed● Slip correction

40
30
20
10
0.2

0.3

0.4 0.5

Across Err［m］

1

Rate％

Rate［％］

0.5

100

100

0
0.1

0

−1000

(b) Shinjuku

70

500

−500

Fig.10: GNSS/INS absolute position evaluation results

80

FIX
Positive
Reference

1000

50
40
30
20

Conventional
methodProposed method

10
0
0.1

0.2

0.3

0.4 0.5

1

Across error［m］

Fig.12: DR error factor analysis and performance improvement

126

SIP通し.indb 126

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

Angle of incidenceθrRelationship between laser
Air

Direct reflection

Diffusereflection

Water

Reflectance(2) GNSS/INS Issues and Responses to Them
The results of this project showed that absolute positioning
performance required for automated driving can be obtained inenvironments where GNSS such as MICHIBIKI can be received,
and that a reliability determination of 30 cm can be made at 95%
or better in Tokyo waterfront areas such as Odaiba. However,even if CLAS could be received during the evaluation, thepositioning performance degraded at locations where thenumber of satellites decreases. For such locations, it is expectedthat complementary technologies such as map-matching usingroad infrastructure will be required to maintain positioning
accuracy.
In this project, we investigated a technology that can predict
positioning errors at arbitrary points using a virtual environment
by collecting high precision 3D map data and GNSS signal
errors as double differences. The developed technology is
expected to make it possible to predict where positioning
performance will deteriorate. Based on the above results, we
believe that safer operations of automated driving will bepossible by prioritizing the building of road infrastructurenecessary for location estimation in locations where positioning
performance is expected to deteriorate.

(2) Issues Identified in the Tokyo Waterfront FOTs
At the same time, the FOTs in the Tokyo waterfront area
revealed that the contrast between the white lines and theasphalt deteriorates on thermal shielding-coated road surfaces
and during rainfall, resulting in a loss of accuracy in matching.
As a countermeasure, we developed a method to correct thereflectance from the angle of incidence of the laser using theLambert model, which is a model of diffuse reflection of light(Fig.14). We have also confirmed that localization accuracy of
0.1m can be achieved by introducing a contrast correction
process that matches the entire LiDAR image to the contrast of
the map image.

Asphalt

Incident angle [deg]Correction

3.4. Development of Map-Matching Technology
LiDAR image before correction

Automated driving systems that use high precision maps
must be able to continuously estimate their own location withhigh accuracy. For this reason, we have developed a localization
technology based on map matching.
(1) Outline of the Developed Technology
As shown in Fig.13 , the localization technology developedin this project is realized by continuously estimating theposition by DR and map matching.(4) DR is capable of calculating
self-position at a high frequency by integrating velocity vectors
obtained from sensors, but it is not suitable for long-term
localization due to cumulative errors. To solve this problem, wedeveloped a method for robustly estimating accurate selfpositions by compensating for the cumulative errors in DR
through image matching (template matching) using roadsurface pattern images obtained in real time from LiDAR andreference map images created by acquiring road surface patterns
in advance.
Dead reckoning

Speed v

GPS position

Localization
Map matching

LiDAR image

Reference
Map imageFig.13: Overview of Localization technology

LiDAR image after correction

Fig.14: Correction of reflectance using Lambert reflection model(3) 
Evaluation of Marginal Performance in a Virtual
Environment
On the other hand, depending on the road surface material
and the wetness of the road surface, the contrast between thewhite lines and the asphalt may be significantly reduced,
resulting in incorrect localization. For this reason, we evaluatedthe marginal performance of self-localization by reproducing
scenarios that caused localization problems identified in the
FOTs using the virtual environment being developed in the
DIVP project.
In this evaluation, as shown in Fig.15, we changed thewetness of a normal asphalt road surface and a thermal
shielding-coated road surface, and compared the matching
results to obtain performance limits. In the DIVP virtualenvironment, the wetness of the road surfaces was defined infive levels: Level 0 is a dry road, Level 1 is a road where raindrops
have begun to seep in, Level 2 is a road where water has fully
seeped in and saturated the road surface, and Levels 3 and 4 area thin water film of 1-2 mm and a thick water film of several mm
or more, respectively. The results of the evaluation confirmedthat localization is possible even at the maximum wetness ofLevel 4 for a normal asphalt road surface. On the other hand, fora road surface with a thermal shielding coating, a wetness levelof 1.9 was confirmed to be the performance limit.

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 127

127

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

Lv. 1
LiDAR image

Lv. 2
Matching correlation
distribution

LiDAR image

Matching correlation
distribution(a) Ambulance(b) Patrol car

Fig.16: Example of emergency vehicle detection
Performance limits for
localization between thetwo conditions
Lv 1.9Lv 2

LiDAR image(After contrast
correction)

Matching correlation
distribution

Correlation value

High

Low
Correlation peaks incorrect position

Peak of correlation
cannot be obtained

Fig.15:  Map-matching results near the performance limit during rainfall
on a thermal shielding-coated road surface

3.5. Development of Emergency Vehicle Recognition Technology
As stipulated in Article 40 of the Road Traffic Act, when an
emergency vehicle approaches, it is necessary to take evasive
action so as not to obstruct its path. For this reason, wedeveloped a technology for recognizing emergency vehicles
using on-board sensors.
(1) Recognition of Emergency Vehicles by Camera
For the image-based recognition of emergency vehicles, werecognized the vehicles themselves, such as police cars andambulances, and detected the flashing of their emergency
lights. In this project, YOLO v4 was used as the recognition
algorithm for emergency vehicles. First, YOLO v4 was trainedusing a dataset for object detection collected in the Tokyo
waterfront area. The Tokyo waterfront area dataset contains
seven classes of objects for detection, including general vehicles,
pedestrians, bicycles, and different states of traffic lights. After
training on the Tokyo waterfront dataset, we added classes for
emergency vehicles, and conducted additional training nineclasses. In training the first seven classes, the system acquires
features suitable for object detection using video acquired froma camera mounted on the vehicle as input. Then, by training
additional classes, features suitable for emergency vehicles canbe additionally acquired. Fig.16 shows an example of emergency
vehicle detection results. At the same time, it may be difficult to
detect an approaching emergency vehicle using only image
recognition based methods when the vehicle is far away or in anenvironment where it is physically difficult to see the vehicle
due to blind spots or other reasons. For this reason, a method to(2) Recognition of Emergency Vehicles by Microphone
For emergency vehicle recognition by microphone, in
addition to determining the presence or absence of an
emergency vehicle, it is necessary to determine the location of
the vehicle. For this reason, in this project we developed a siren
sound recognition technology and a sound source localization
technology. For siren sound recognition, features were extracted
from the sound data of the on-board microphones, and thesiren sound is identified by machine learning. In this project,
SVM (Support Vector Machine) was used as the recognition
algorithm, and in addition to the test vehicle driving data, datafrom ESC-50, which is a publicly available environmental
sound dataset, and voice data scraped online were also used as
the training dataset. Table 1 shows the evaluation results of theambulance siren sound detection technology developed in thisproject. The Precision index indicates a value closer to 1 when
there are fewer false positives, and the Recall index indicates avalue closer to 1 when there are fewer false negatives. Table 1
shows that the sirens were identified with an accuracy of morethan 90%.
Table 1: Recognition rate of ambulance siren soundPrecision

Recall

0.964

0.965

Recognition rate

In addition, we studied the phenomenon of the recognition
rate decreasing due to attenuation of the siren sound, increasein noise, and the Doppler effect depending on the distance tothe emergency vehicle and the vehicle's speed. Fig.17 shows agraph of the recognition accuracy as a function of vehicle speed
and distance to the emergency vehicle. The figure shows that
1.2
0 km/h1

Recognition rate

Lv 1.8detect the siren sound of an emergency vehicle using
microphones mounted on the test vehicle shown in Fig.1 wasalso studied.

20 km/h

40 km/h

60 km/h

300

350

0.8
0.5
0.4
0.2
0

0

50

100

150

200

250

400

Distance [m]Fig.17: Recognition accuracy based on own vehicle speed and distance128

SIP通し.indb 128

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿2) Research on the Recognition Technology Required for Automated Driving Technology (Levels 3 and 4)

the recognition accuracy is 90% when the distance to theemergency vehicle is within 120m, but the recognition accuracy
decreases significantly when the distance to the emergency
vehicle is farther away. It can also be confirmed that therecognition accuracy generally decreases as the speed increases.
The test vehicle was equipped with four microphones forsound source localization (Fig.1). Using these on-board
microphones, we developed a technique for estimating thedirection and distance of a sound source by calculating the timedifference of sound arrival between the microphones.
To confirm the principle, the distance to the emergency
vehicle was fixed at 100m, the azimuth angle was varied whileboth vehicles were stationary, and the accuracy of the source
azimuth estimation by the two microphones was evaluated. As
shown in Table 2, it was confirmed that an azimuth estimation
with an error of about 10 degrees was possible under eachcondition.

[References]
(1)	AD-URBAN Open Image Dataset download page, https://github.com/
AdmoreKanazawa/open_data, (accessed 2022.08.31)(2)	K. Yoneda, A. Kuramoto, N. Suganuma, T. Asaka, M. Aldibaja, and R.
Yanase, Robust Traffic light and Arrow detection using Digital map with
Spatial prior information for Automated driving, Sensors, Vol.20, No.4,
2020.
(3)	Aoki Takanose, Yoshiki Atsumi, Kanamu Takikawa, Junichi Meguro,
Improvement of Reliability Determination Performance of Real Time
Kinematic Solutions Using Height Trajectory, Sensors, Vol.21, No.2,
2021.
(4)	N. Suganuma, D. Yamamoto and K. Yoneda, Localization forautonomous vehicle on urban roads, Journal of Advanced Control,
Automation and Robotics, Vol.1, No.1, pp. 47-53, 2015.

[Contacts]
Kanazawa University, Advanced Mobility Research Institute, Kakuma-cho,Kanazawa, Ishikawa 920-1192, 076-234-4714, Contact: Naoki Suganuma

Table 2: Accuracy of sound source azimuth estimation
with two microphones
True value [deg]Estimated position[deg]

Error [deg]

−36.2

−45.7

−9.5

9.4

1.9

−7.5

48.6

37.7

−10.9

4 Postscript
In this project, through universities as neutral research
institutes, we developed the recognition technologies required
for a Level 4-equivalent automated driving system in complex
urban traffic environments with a mixture of ordinary traffic
participants and other automated vehicles. We also evaluatedthe effectiveness of infrastructure-cooperative driving using
wireless infrastructure such as V2I/V2N through FOTs in theTokyo waterfront area. In addition, we reproduced in a virtualenvironment the problems of autonomous recognition
technologies based on on-board sensors that emerged in the
FOTs in the Tokyo waterfront area, and evaluated the marginal
performance of these recognition technologies.
In the future, for the introduction of automated driving
technology to society in various environments, including
prefectural and municipal roads, it will be important to evaluate
not only advancements in recognition and decision-making
technologies in automated driving systems, such as thosedeveloped in this project, but also their safety. At the same time,evaluations in actual road environments are limited in terms oftime, cost, and efficiency. For this reason, it will be important toconstruct frameworks for efficient and comprehensive safetyassurances using virtual environments, as was done in thisproject.

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 129

1292023/03/01 16:47

3

Ensuring the Safety of Automated Driving

3) Research of New Cyberattack Techniquesand Countermeasure Technologies
Ken Okuyama, Naohide Waguri, Shinichi Kan, Yuki Imagawa (PwC Consulting LLC)
(Abstract) As connected cars and automated driving systems develop and spread, a lot of information of advanced
maps and the vehicles, people, and infrastructure facilities mapped onto them is transmitted to vehicles via external
networks. This situation causes cybersecurity issues. Furthermore, since the introduction of UN-R155/R156 to thesafety standards for road vehicles—agreed-upon regulations for cybersecurity and software updates—it is also
necessary to take countermeasures against cyberattacks from a legal perspective. To solve these issues, this study
focused on intrusion detection systems (IDS) as a new countermeasure technology against cyberattacks after
shipment, and developed IDS evaluation guidelines to serve as a baseline for evaluation and testing when introducing
IDS. During the current fiscal year, we are studying methods of collecting and accumulating threat information onconnected cars and conducting collection experiments using methods such as honeypots, seeking to create amechanism to support an initial response when an incident actually occurs.
Keywords: IDS (Intrusion Detection System), Honeypot, Threat Information, STIX/TAXII, S-BOM

1

Purpose of this research study andoverview of activities

In accordance with the research and development plan,
objectives and goals of the second phase of SIP-adus (Crossministerial Strategic Innovation Promotion Program (SIP)
Automated Driving for Universal Services): Research of New
Cyberattack Techniques and Countermeasure Technologies,
the "Formulation of IDS Evaluation Methodology andGuidelines" and "Research study on Threat Information andInitial Response Support for Connected Cars" are being
conducted from August 2020 to February 2023.

Background related to post-shipment cybersecurity
Legal side

Practical side

WP29 UN-R155 cyberattacks must be detected andresponded to, and it is necessary to explain that thecompany's vehicles can detect and respond to suchattacks.

There are no existing laws, regulations, or guidelinesthat specify what types of attacks should be detected andto what extent, making it necessary for each company todefine its own requirements.

Purpose and Policy of Activity A
Focusing on on-board IDS, we will investigate evaluation methods for evaluating the ability of IDS to detect attacks
and subsequently recover the vehicle and organize them into IDS evaluation guidelines to contribute to post-shipment cybersecurity measures in the automotive industry as a whole.
Development

Manufacture

Operation

Disposal

Detection Response Recovery

Subject of activity A
Improved post-shipment cybersecurity

Fig.1: Purpose and Policy for Formulation of IDS Evaluation Methodology
and Guidelines

2.1. Derivation of basic test cases

Formulation of IDS Evaluation

2 Methodology and Guidelines

As shown in Fig.1, our objective with this topic is tocontribute to post-shipment cybersecurity measures. We hopeto achieve this by creating IDS evaluation guidelines and thenproviding them to industry organizations as a baseline forOEMs when selecting, verifying, and operating IDS. The main
target audience is OEMs that have just started considering the
introduction of on-board IDS with the intent of raising the levelof post-shipment cybersecurity of their vehicles.

In order to organize the points to be evaluated in theselection and testing phases of on-board IDS, we analyzed
previous published cases of attacks and derived the cybersecurity
events from them. Among the cybersecurity events, we selected
Extraction

Information gathering and analysis
Data Source

Examples of Attacks

Cybersecurity events

Basic test cases

Papers
Products
Public documents

Analysis

FilterFilter conditions
1. An attack that occurred on a vehicle which has been shared in the past (2019-2021) (*1) and which
every IDS should be capable of addressing (*2), and/or
2. affects the basic movements of the vehicle (running, turning, stopping).

Fig.2: Derivation of basic test cases

130

SIP通し.indb 130

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿3) Research of New Cyberattack Techniques and Countermeasure Technologies

those that an IDS should detect at a minimum, and summarized
them as basic test cases. (Fig.2)

2.2. Basic test case testing environment
Based on the basic test cases, three types of testing
environment were considered, as shown in Fig.3. In light of theincurred costs, this study examined a test procedure based on
the assumption that testing would be conducted in either asimulation environment (Fig.4) or a testbed environment.
(Fig.5)

Simulation environment

Testbed environment

Vehicle (bench) environment

Test environment without
using actual ECUs. Reproduction of on-board
networks using software.

An environment built withthe minimum hardware
required to meet the test
requirements.

An environment in which
input devices, ECUs, andactuators equivalent to those ofa real vehicle are connected.

CGW

GW

While simulating an on-board NW,
the (attack) messages are input to the
IDS according to the test requirements.

IDS
Simulate on-board
NW with software.
Virtual ECU (CGW)
VirtualECU

VirtualECU

Control unit
ECU

ECU

Control unit

ECU

ECU

Actuator

Actuator

IDS

VirtualECU

Fig.3: Types of IDS testing environments

IDS

2.3. Testing of test cases using actual IDS equipment
The purpose of actual IDS equipment testing is to verify thevalidity of the test method of the basic test case (whether thetest can be performed using the predefined test method).
For this purpose, one OEM (hereinafter referred to as "OEMA") provided the communication data and ECU for the onboard network necessary for testing a basic test case. In addition,
we asked ETAS Corporation and Arilou Information Security
Technologies Ltd. to provide IDS adapted from their respective
companies for the specific vehicle model of OEM A. We
conducted testing according to the test procedures of the basic
test cases, and confirmed whether the test results matched the
expected values.

2.4. Results of testing
The procedures shown in the test method for all test
perspectives tested were performed for the IDS of the twocompanies under the simulation and testbed environments. It
was confirmed that there were no errors in the procedures andthat the IDS performed as expected.

2.5. Activities for social implementation
A technical review meeting was held with JASPAR (Japan
Automotive Software Platform and Architecture), the future
recipient of the IDS evaluation guidelines, in preparation for
the transfer of the IDS evaluation guidelines. The IDS evaluation
guidelines were finalized by responding to feedback from therecipient right up until the transfer, which—including
administrative procedures—was then completed in August2022.

Fig.4: Simulation environment

3

Research Study on Threat
Information and Initial Response
Support for Connected Cars

Our goals with this topic have been to define basic
specifications for methods of collecting and accumulating
threat information on connected cars and for initial response
support using threat intelligence, and to transfer our outputs toindustry organizations in 2023. "Threat intelligence" isinformation collected, analyzed, and accumulated to support
responses to cyberattacks and other threat. In some industries,
there are threat intelligence sharing activities.(1)Sharing threat intelligence is expected to be effective inpreventing chain-reaction damage caused by similar
cyberattacks, but the currently shared threat intelligence comesmainly from the IT domain.
Fig.5: Testbed environment

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 1311312023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿3) Research of New Cyberattack Techniques and Countermeasure Technologies

3.1. Examination of basic specifications for information collection and storage

As a result of the current observation, we observed many
activities that match the characteristics of IoT malware (Miraietc.) as seen in IoT products, sending IDs and weak passwords
to telnet. This is considered an automated attack from a device
infected with malware and not an attack based on recognizing
the honeypot as an on-board device.
As part of the honeypot PoC, we have also been concurrently
studying efficient equipment search methodologies, and haveproposed two approaches: keyword search using an Internet
search engine to search websites for on-board unit products,
and a direct search using Censys*3 for keywords related to onboard unit.
The scope of the honeypot PoC up to FY2021 was honeypotsthat imitated aftermarket products among on-board unit, but in
FY2022, we are also studying the methodology for collecting
threat information using honeypots that imitate connected
systems as a whole.

In order for organizations in the automotive industry toobtain threat information more efficiently and proactively, we
conducted PoC (Proof of Concept) on collection methods otherthan public information collection, referring to examples of
implementation in the IT domain. During this activity, we
focused on honeypots*1 and playgrounds (one format ofCTF*2), and examined whether these methods can be applied tothe automotive domain.

3.1.1. Honeypot PoC
We created an automotive version of a honeypot used in the
IT field and conducted PoC. Specifically, we investigatedaftermarket products that can be detected via wide-areascanning, developed a honeypot prototype using thecorresponding products, and began cyberattack observation
experiments in late January 2021. Fig.6 shows an overview of
the automotive honeypot.

http
ssh

Cloud

telnet3.1.2. Playground experiments
Playground experiments were conducted to investigate
cyberattack methods targeting vehicles in connected systems,
as well as attacker's behavior and logs to determine if an attack
was targeted at a vehicle. These playgrounds were conducted ashacking contests in which participants report vulnerabilities
with the goal of exploiting the system, without the setting of any
specific flags.
For the playgrounds, we prepared an environment that
imitates a general connected system as shown in Fig.7. Although
there are various attack surfaces in a connected system, theattack surface in this playground is a user application or a web
portal site open to the public.

httpCollected
responses

ftp

ssh

Collected
responses

Devicedecovery
Systemtelnetftp

honeypot

MobileNW

honeypot

Fig.6: Overview of automotive honeypots

Playground environment(1) Attacks on web

applications via
the Internet

Attacker

Scope of honeypot
PoC(3) Attacks on AP

servers, etc. frominternal NWs, etc.

Attacker
AP Server
Web
Applications

Assumed attack
surface(5) Direct attack on vehicle
from internal NW
Connected
Server

Attacker

Drive
management
program etc.

(4) Attacks onconnected servers
from internal
NWs, etc.

DB(2) Attacks on DB fromDBinternal NW, etc. (Driving data,(User information
sensor information,systems)
operation records Attacker
etc.)
Attacker(6)
Direct attacks onvehicles via the Internet
Attacker

TCU/ECUAM on-board
deviceAutomated
driving
programs,
emergency callprograms, etc.

AM equipment
lot service, etc.

Fig.7: Playground environment*1 Equipment for observing attacks. A method to collect and observe access information and attack methods of attackers by disclosing a system on the Internet with
the intention of it being attacked.
*2 A method in which white hat hackers intentionally launch attacks against an environment that simulates a system to collect information on the availability of attacks
and their methods. A widely known method is to set a target (flag) for the attack and compete for points.
*3 C
 ensys is an IoT search engine. It can search for devices that are open to the public on the Internet.

132

SIP通し.indb 132

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿3) Research of New Cyberattack Techniques and Countermeasure Technologies

3.2. Methods of describing and sharing threat information
In order to collect and share threat information efficiently,
the threat information must have a certain level of structure anda method for sharing it must be formulated. As shown in Fig.8,
as vehicles become more connected and automated, there will
be more opportunities to adopt technologies used in IT
domain. We are conducting research focusing on STIX/TAXII,*4
which is the most popular protocol in the IT field and is capableof describing many types of information.
IT expert

IT domain

IT cybersecurity

Web Applications
Internet protocol

Connected car
cybersecurity

On-board unit
domainOn-board unit
systems expert

Vehicle functions

On-board systemcybersecurity

HardwareFig.8: Integration of automotive and IT domains

3.3. Utilization of threat information for initial response
In order to manage threat information and utilize it for an
initial response, it is necessary to determine whether the data
discovered and reported daily is relevant to the vehicles orsystems that will be managed. In particular, when open source
software (hereinafter referred to as "OSS") is used, a large
amount of vulnerability data is reported and frequent updates
are performed. In turn, this makes it necessary to constantly
update the relevant information and keep track of the version,increasing the difficulty in determining the relevance. Using
Software Bill of Material (S-BOM) to manage the components
included in software has been attracting attention as a solution
to these issues. Although the S-BOM can be created in-houseand used for vulnerability management, it can be usedthroughout the supply chain for a more comprehensive andefficient initial response. By creating an S-BOM and linking it
to other information sources, such as information sharing
systems, it is possible not only to determine whether or not theSupplier A

Complete vehicle manufacturer
SBOM

Information sharing
system information
sources

SBOM
On-board system A
Product nameSoft C1

Program D1
Program D2

Supplier B
SBOM

Supplier C
SBOMSoft C2

Program D2

Cooperation

Threat information

Advantages of S-BOM creation
Understanding of whether fleet
systems are vulnerable based on
the threat information collected
and shared
Smooth access to rele vantvulnerability information andinformation necessar y for
initial response in the event of
an incident

Fig.9: Use of S-BOM for initial responsecompany's vehicles and systems are vulnerable, but also tosmoothly access the relevant vulnerabilities and informationnecessary for an initial response in the event of an incident.
(Fig.9)3.4. Activities for social implementation
Our goals for this topic consist of two deliverables: basic
specifications for an information sharing system that
accumulates threat information in the automotive domain andcan be used to support initial response to cybersecurity
incidents, and an information collection guide that compiles
knowledge on attack methods and technologies for connected
systems as obtained from the honeypot and playground.
Technical review meetings with J-Automotive-ISAC, to which
the deliverables will be transferred, are being held on an ongoingbasis and approval has been received for the transfer of thedeliverables.
In order to cope with cyberattacks in the automotive domain,
it is important to take cybersecurity measures not only for
individual companies but also for the entire supply chain. In
order to make the deliverables easier to deploy in practice,meetings were held as necessary with J-Auto-ISAC, the final
recipient, in order to exchange opinions with the teams
conducting testing and research at the field level.
After the transfer, which is scheduled for December 2022, wewill continue discussions with J-Auto-ISAC for practical
deployment and additional activities.

4 Outcomes of the Project4.1. Development of IDS evaluation guidelines
For this topic, IDS evaluation guidelines were developed
mainly to contribute to accelerating the start-up of on-board
IDS development by OEMs/suppliers who are facing difficulties.
Regarding the basic test cases, test case procedures andexpected values were confirmed using actual equipment
through the collaboration of two companies, one OEM and one
IDS vendor, validating their suitability. The transfer of the test
cases to JASPAR has also been completed.
IDS for on-board units will be a typical technology andproduct that provides detection functions to cope with new
cyberattacks in the future, but we found through interviews
with some OEMs that this technology is not being studied at
the same level in the automotive industry as a whole. The targetof our evaluation was an off-the-shelf network monitoring IDS
using CAN communication as the target protocol. The maincontent of this project is testing and evaluation methods
focusing on detection functions, which is the main function of*4 STIX is a standardized language developed to describe threat information, while TAXII is a specified procedure for sharing threat information.

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 133

133

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿3) Research of New Cyberattack Techniques and Countermeasure Technologies

IDS. It is expected that OEMs/suppliers will introduce
appropriate IDS by referring to these IDS evaluation guidelines.

4.2. Research study on threat information and initial
response support for connected cars
This topic aims to prevent cybersecurity incidents andminimize damage in the event of an attack on connected cars
and automated driving systems, which are technology that haveseen remarkable development in recent years. The basic
specifications for an information sharing system and an
information gathering guide for proactive threat information
gathering are being prepared.
The deliverables for this topic will promote information
sharing activities in the automotive industry, where connectedcars are widely used and new cyberattack methods arecontinuously reported, and will contribute to enhancing first
response capabilities. We hope that the proposed specifications
will be used to develop and implement information sharing
systems, and that the guidance will be used as a reference forproactive information gathering in addition to publicly available
information. We also expect that these sharing systems andinformation collection efforts will continue to be improved andadapted to the automotive industry through continuous
operation and review.

technologies from overseas and other industries, and tocontinue research not only on information gathering, but also
on methods of analyzing collected information and improving
its accuracy and precision. In addition, cybersecurity on theside of connected services (such as servers and platforms),
which has been investigated and handled independently by
OEMs/suppliers, may become a new area of cooperative area,
and is therefore considered to be an area of future research andfocus.
Based on the above, data cybersecurity activities have an
important role in the development of automated driving
systems, and are expected to contribute to the development ofcybersecurity activities in the industry.

[References]
(1)	Cross-ministerial Strategic Innovation Promotion Program (SIP) Ensuring
Cybersecurity in Critical Infrastructure, etc. (b2) Information Sharing Platform
Technology "Information Sharing Design Guide Construction Edition", Hitachi,Ltd. https://www.nedo.go.jp/content/ 100904081.pdf, (Reference 2022.05.31)[Contacts]
Technology Division, PwC Consulting, LLC, 2-1, Otemachi 1-chome, Chiyoda-ku,
Tokyo 100-0004, 080-4426-1658, Contact: Ken Okuyama

4.3. Conclusion
Since ensuring automotive cybersecurity may have an
impact on automotive safety, it is appropriate to ensure thatminimum cybersecurity standards are met and that thecommon threats faced as a whole must be made a cooperative
area for the entire Japanese industry, or at least actively shared,in order to improve the development and operational efficiency
of connected services and help Japanese companies to maintaintheir international competitiveness. It is also important tostrategically approach standardization organizations so that thecybersecurity measures and information sharing mechanismsthat have been established can be used to the advantage ofJapanese companies, allowing Japan to make recommendations
for future international standards and the development of
automotive cybersecurity.
The final results of this project also lead into our further
research and areas of focus. For the IDS evaluation guidelines,
it will be possible to create guidelines for the evaluation of IDS
other than off-the-shelf products, other protocols such as
Ethernet, testing and evaluation methods for non-functional
requirements such as performance that cannot be generalized
at this time, and guidelines relating to IPSs (Intrusion
Prevention Systems).
With regard to the basic specifications for information
sharing systems and the information collection guidelines, it isnecessary to catch up with and utilize methodologies and new134

SIP通し.indb 134

2023/03/01 16:47

3

Ensuring the Safety of Automated Driving

4) Research of Education Methods for
Advanced Automated Driving Systems
Makoto Itoh, Huiping Zhou (University of Tsukuba), Yoshiko Goda, Masashi Toda (Kumamoto University), Maki Arame,
Junko Handa (Polytechnic University of Japan)(Abstract) This paper outlines the activities of "Research and Development of Knowledge that Drivers and Pedestrians
Should Acquire and Effective Educational Methods (Task C)" of the second phase of SIP-adus (Cross-ministerial
Strategic Innovation Promotion Program (SIP) Automated Driving for Universal Services). In Task C, based on themain research objectives, we have set three research themes for further developing educational methods: (1) proposalof an educational method based on individual characteristics, (2) proposal of a motivational method, and (3)development of complete educational materials that can be modularized with partial education in mind. In addition,using prototype teaching materials, we verified the effectiveness of providing general knowledge about automateddriving in advance using a driving simulator.
Keywords: education, training, motivation, knowledge, experience

1 Introduction
In order for the user to safely utilize an automateddriving system, the user must be able to make appropriate
judgments and take appropriate actions according to thesystem functions, the role of the user (driver), and thesituation in which he or she is placed. The human-machineinterface (HMI) installed in the vehicle should support theuser's judgment, and ideally, the HMI should support theuser in making appropriate judgments intuitively, regardlessof the user's knowledge. However, unfortunately, an ideal
HMI that can guarantee the user's appropriate judgment
and execution of actions has not yet been realized, and even
if it were technically feasible, it may not be possible to use
an ideal HMI due to cost constraints.
Therefore, it is necessary to consider how education andtraining should be conducted to provide users with thenecessar y knowledge and skills. This research anddevelopment is an effort to educate users on the safe use of
automated driving.

Proposal of educational methods

2 based on individual characteristics
In order to conduct education efficiently, it is necessary

to appropriately take into account the characteristics of therecipient of education, such as learning styles. In thisresearch and development, in addition to learning styles, we
focused on resilience characteristics, one of the individual
difference variables of personality characteristics, andexamined the effects on knowledge acquisition whentransitioning from Level 3 automated driving to manual
driving, taking into account the fact that automated driving
is a new learning experience.
Specifically, we analyzed the results of a survey of 3,240
participants conducted via the Internet during the firstphase of SIP-adus in relation to personality characteristics
and knowledge acquisition. Three types of educational
materials (leaflets, quizzes, and videos) were used, and theeffect of knowledge acquisition after learning with eachmaterial was compared.(1) A logistic regression analysis was
conducted using the binary variable of the ascending andnon-ascending groups as the dependent variable, and agroup of variables that were assumed to influence thedependent variable as the independent variable, using
propensity score matching. As a result, it was confirmed thatthe absorption effect of resilience characteristics differeddepending on the format of the teaching materials, and thatthe video teaching materials had the greatest potential toabsorb resilience characteristics.
In addition, a method that can absorb individual
characteristics and be effective in educating a large numberof trainees was investigated, as well as a method that can

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 135

135

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿4) Research of Education Methods for Advanced Automated Driving Systems

easily identify characteristics. As a result, we were able todevelop a shortened version of the learning style and career
resilience scale.(2) (3) Furthermore, we found that effective
teaching materials can differ depending on differences inlearning styles and career resilience. These results suggestthat adaptive learning is feasible if thre is enough time in a
education, where learning styles and career resilience can beeasily examined using a shortened version of the scale, andlearners are provided with more appropriate learningmaterials according to the results of the examination.

3 Proposal on motivation methods
The target audience for safety education is alltransportation users, and is diverse and wide-ranging interms of age groups, backgrounds, and learning styles. Manyof them are not interested in automated vehicles. Driver
education may include a course at the time of license
renewal, but it is necessary to devise ways to increase
learning effectiveness in a limited amount of time.
Therefore, in addition to teaching materials, a method toarouse interest in the learning content was studied. In thisstudy, we developed two types of motivational videos forsafety education ("Narrative" videos that focus on stories
that are expected to be emotionally moving, and "Factbased" videos that focus on objective facts), using the
motivational ARCS model. The results of formativeevaluation suggested that the two types of videos were able
to motivate at the same level.
In addition, in order to examine motivational methods
for automated driving education, we used two types ofmotivational videos we developed and examined theeffectiveness of motivational videos considering theinfluence of individual characteristics from a web-based
survey of 2,790 participants.(4) Propensity score matching
results verified the possibility that motivational videos
utilizing the narrative method can absorb individual
resilience factors compared to fact-based videos, and thatafter the fact scores increase with lower levels of resilience.
In addition, the use of narrative-based motivational videos
may be more effective when the individual's learning style istaken into account. It was also suggested that the use ofnarrative-based motivation may be able to absorb
differences in individual attributes such as gender, age,marital status, and whether or not they have children.
In addition, the methodology of group training toincrease motivation was also examined. We designed andimplemented an online training course, and found that thetraining improved knowledge about automated driving

systems and promoted attitude change.

4

Development of complete educational
materials that can be modularized withpartial education

One of the objectives of this project is to develop andaccumulate a variety of educational material modules thatcan be combined with each other so that users can learn
using educational materials that match their own
characteristics and learning styles. For the research, wedeveloped four modules including two types of motivational
videos as described in Section 3, and quiz materials andvideo materials for acquiring general knowledge about
automated driving. By developing a variety of teachingmaterials as short, micro-learning-enabled modules, it ispossible to learn each of them independently, or to combine
them according to learning goals or actual objectives. In
addition, a simple driving simulation system that enables
users to experience a driving takeover situation wasdeveloped as a web-based remote learning material. The
effectiveness of these teaching materials has been confirmed
through web-based surveys and verification experiments
using the simulator described in Section 5.
In addition, among the materials developed for the
research, the motivational videos and general knowledge
videos will be published as sample teaching materials for
actual use.

5

Experiments to verify educational
effectiveness for educational
opportunities

5.1. Educational opportunities
In this research project, we examined educational
opportunities for automated driving with a view to social
i mp l e m e nt at i o n . T h e r e a r e v a r i o u s e d u c at i o n a lopportunities, including mass media and websites, toprovide the knowledge necessary to use automated driving
or to interact with automated vehicles. Since the time andresources available for each of these opportunities arelimited, it is important to select the appropriate level of
specificity and granularity in line with the characteristics ofeach opportunity.
Opportunities to convey knowledge and information
about specific systems to actual users may include
opportunities for explanations at dealerships and car rental
offices and at the time of delivery, as well as the ability toconvey knowledge through in-vehicle systems. However,

136

SIP通し.indb 136

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿4) Research of Education Methods for Advanced Automated Driving Systems

according to the ongoing research conducted to date in thisproject, (5) it is believed that some people generally havealmost no knowledge of automated driving at all, andtherefore it is necessary for them to have general knowledge
to be able to understand the explanations of individual
systems. Therefore, in this project, we decided to verify theeffectiveness of communicating general knowledge about
automated driving in advance.
5.2. Simulator-based verification experiment on generalknowledge
First, we decided to verify the effectiveness of the generalknowledge provision as a trial before the prototype teachingmaterials in Section 2. (Experiment 1) Based on the
instructional materials used in the knowledge provision
study conducted in the first phase of SIP-adus, we organized
the general knowledge about automated driving, created an
explanatory video, and had the participants watch it.
Approximately one month later, an experiment on thehandover from automatic to manual driving was conductedusing a driving simulator. To test whether generalknowledge is useful for several specific systems, two types ofexperiments were conducted: one that provided Level
3-equivalent functions only at low-speed following, and theother that provided Level 3-equivalent functions with nospeed limit (between-subjects factor). Neither of the two
types provides a function for changing lanes.
Fig.1 shows an example of the experiment results.(6) Thisis a scene in which a broken-down car is stopped in front of
the driver's lane and the driver needs to change lanes toavoid it. Since the automated driving system in thisexperiment does not have a lane change function, the driver
must take over in order to continue driving. Fig.1 shows theratio of the number of people involved in a collision to thenumber of vehicles involved in a breakdown. As shown in
Fig.1, the results suggest that if general knowledge about
automated driving is provided in advance, the driver may be

Accident rates in the DS experiment

0.30

0.25

0.15

0.10

0.05

Prior training available

No prior training

Prior training available

No prior training

Lv3 automated driving

Lv3 automated driving forwith no speed limit

low-speed following only

Fig.1: Example of experiment results

able to perform an appropriate takeover when RtI ispresented by the system in a situation where doing so isnecessary.
Next, a verification experiment was conducted using thevideo materials developed in Section 4, which wereshortened to the time expected to be spent on actual
education. (Experiment 2) In this experiment, an
explanation of the system used in the experiment, alongwith general knowledge, was given approximately one
month before the simulator experiment was conducted. As
a result, many participants were observed to be unawarethat they were being asked to take over driving in the firsthalf of the experiment, even though the RtI was presented
to them. The difference is that in Experiment 1, theexplanation of the actual system was provided on the day ofthe experiment, whereas in Experiment 2, it was provided
one month before the experiment. Although the usefulnessof providing general knowledge was confirmed inExperiment 1, it is necessary to promote understanding of
the provided knowledge in order to enable truly safe use of
the system.
In order to promote understanding and retention, we
reorganized the teaching materials so that necessary
explanations are given in a timely manner, and we
confirmed that in many cases, the participants were able torespond appropriately to driving takeover. In addition, we
also examined the two points of including a specific systemin the general explanation, and of providing even a
simplified experience of the driving takeover, and confirmedthat better results were obtained in the verification
experiment when a specific system was included as a casestudy. The results also indicated that the effect of theexperience of switching drivers may work in a more positive
direction, although the effect was not statistically significant.
5.3. Verification experiment on how education andtraining should be conducted when actually using thesystemEven if the same automated driving system is used in thesense of Level 3, the functions, their limitations, andsituations in which driving takeover is required differ
depending on the type of vehicle and system, such aswhether there is a speed limit or not. Therefore, when
actually using a particular system, it is necessary to haveknowledge specific to that system and acquire the skills to
respond appropriately in situations where driving takeoveris required.
One of the important issues here is how effective theexperience is. From the first phase of SIP-adus results,(7)(8)
we discovered that various experiments gradually allow thedriver to respond more skillfully. On the other hand, when

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 137

137

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿4) Research of Education Methods for Advanced Automated Driving Systems

the driver suddenly encountered a serious situation such as
loss of sensor function, there was a delay in intervention. In
addition, it was confirmed that experience with functional
failure, which is considered to be one of the most severe
situations requiring driver handover, is effective when
encountering actual situations, and that such an experience
may be effective over a long period of time.(9) From theabove, it is considered important to have the trainees
experience specific driving takeover situations in advance.
However, it is not always possible to devote enough time tothe test-drive experience in the midst of complicatedprocedures such as contracting when buying or selling avehicle. Rather, it would be effective to provide users with
an opportunity to freely learn on their own in their sparetime. Therefore, we conducted an experiment to verify theeffectiveness of using the web-based system developed in
Section 4 to allow users to experience a possible driving
takeover situation using the same HMI as the vehicle they
will actually use at a later date. As a result, it was found thatthe participants were able to take quicker action during theactual driving takeover by experiencing such a situation inadvance.
In the web-based learning, there were some cases inwhich the learners skipped watching the video materials
that they should have viewed, and as a result, there weresome cases in which their understanding was consideredinsufficient.

[References]
(1)	Arame, M., et al.: Effects of Learning materials about AutomatedDriving Level 3 Focusing on Driving Frequency: Verification by
Propensity Score Matching, Proceedings of 2020 Fourth World
Conference on Smart Trends in Systems, Security and Sustainability(WorldS4), pp. 454-460, 2020.
(2)	Goda Y, et al.: Development of a Short-Form Learning Style Inventoryfor Automated Driving Safety Education. Proceedings of IEEE
International Conference on Teaching, Assessment, and Learning forEngineering (TALE) 2020, pp. 847-851, 2020.
(3)	Arame, M., et al.,: Development of a Short-Form Career Resilience
Inventory for Online learning -Verification based on analysis results of
online learning for automated driving-, IEEE TALE 2022 (under review)(4)	Arame, M., et al.: Using Narrative based Video on Gaining Safety
Driving: Focusing on Career Resilience and Learning Style in
Automated Driving Level 3, Proceedings of Fifth International Congresson Information and Communication Technology, ICICT 2021.
(5)	The Second Phase of SIP-adus Research and Development in FY2019:
Research and Study on HMI and Safety Education Methods for
Advanced Automated Driving, https://en.sip-adus.go.jp/rd/rddata/rd03/
e211. pdf, (accessed 6/29/2021)(6)	Zhou, H., et al.: Influence of Prior General Knowledge on Older Adults’
Takeover Performance and Attitude Toward Using Conditionally
Automated Driving Systems, Proc. HFES Annual Meeting, 2021.
(7)	The First Phase of SIP-adus Automated Driving System Outcome
Report: Establishment of Guidelines by Data Collection on Human
Factors such as HMI, https://en.sip-adus.go.jp/rd/rddata/rd01/121s.pdf,(accessed 6/292021)(8)	Zhou, H., et al.: How Do Levels of Explanation on System Limitations
Influence Driver Intervention to Conditional Driving Automation? inIEEE Transactions on Human-Machine Systems, Vol. 51, No. 3, pp.188197, 2021.
(9)	Zhou, H., et al.: Long-term Effect of Experiencing System Malfunction
on Driver Take-over Control in Conditional Driving Automation.
Proceedings of The 2019 IEEE international Conference on Systems,
Man, and Cybernetics (SMC 2019), pp.1950-1955, 2019.

[Contacts]

6 Social implementation

Institute of Systems and Information Enginerring, University of Tsukuba
1-1-1 Tennodai, Tsukuba , Ibaraki 305-8573, 029-853-5502, Contact: Makoto
Itoh (Professor, itoh.makoto.ge@u.tsukuba.ac.jp)

Instead of simply summarizing the results of this researchand development in this report, they should be utilized forsocial implementation. For those who plan education, etc.,
this report is a manual that summarizes the knowledge that
contributes to planning and development of teachingmaterials. In addition, as described in Section 4, we arepreparing to release motivational videos and generalknowledge video materials as samples. We hope you willfind the manual and other materials useful.
Since this manual was compiled, we have continued toanalyze data obtained as a result of our research anddevelopment activities, and as further findings are obtained,
we hope to take the opportunity to revise it as necessary.

138

SIP通し.indb 138

2023/03/01 16:47

3

Ensuring the Safety of Automated Driving5) R esearch on Communication between Low-Speed
Automated Transportation and Logistics Services
Vehicles and Surrounding Traffic Participants
Tatsuru Daimon, Masahiro Taima, Jieun Lee, Tomoyuki Furutani (Keio University)(Abstract) Aimed at addressing societal problems such as facilitating mobility for vulnerable road users, alleviating alack of drivers for transportation and logistics services, and reducing costs, technologies are currently being developed
for low-speed automated transportation and logistics service vehicles, and FOTs (Field Operational Tests) are being
done in rural regions. In contrast to traditional driver-operated vehicles, low-speed automated service vehicles needto see a number of issues addressed, including safety, security, and traffic efficiency, through better communication
with traffic participants, including pedestrians and other drivers. In order to achieve safe and reliable communication
between low-speed automated service vehicles and traffic participants, we analyzed communication characteristics
observed during the FOTs, and conducted experimental studies concerning communication methods (vehicle
behaviour, external HMI (eHMI), etc.) for relaying to traffic participants the intent and status of automated service
vehicles. Furthermore, through the field operational tests, we tested communication methods through eHMI identified
based on experimental studies, and confirmed the effectiveness of design recommendations for communication
methods.
Keywords: communication, expression of intent, eHMI (external human machine interface), vehicle behavior, traffic participant

1

Introduction

To ensure mobility for vulnerable road users in rural, or
depopulated areas, and to alleviate a shortage of drivers fortransportation and logistics services, research is being done
into the deployment and usage of transportation and logistics
services that make use of automated vehicles. At this early stage,
research is proceeding with the expectation that these services
will be used for low-speed travel in general road environments
and traffic situations where there are pedestrians, drivers, andother traffic participants. In some areas, including those where
FOTs are being done, human attendants who can manuallyintervene are needed for safety reasons in automatedtransportation or logistics service vehicles traveling at low
speed. However, technological advances should pave the way toa means of vehicle operation capable of handling situations andlimitations in areas of automated service vehicle deployment,
including means of operation where no human attendants, ortheir intervention, are needed. On the other hand, conventional
driver communication, such as gestures and eye contact, for the
purpose of ensuring safety and traffic facilitation for participantsin surrounding traffic, facilitates their understanding of thevehicle's intentions and status and their own judgment of theirown actions, and improves their sense of security. However,there is unlikely to be any active communication from humancrew member when automated service vehicles are being
operated without involvement by such crew member. These
research and development efforts(1) have focused on developing
and proposing design recommendations to support and achievesafe and seamless communication between automated service
vehicles and traffic participants. Through cooperation with
"automated service vehicle FOTs in areas primarily around
Michi-no-Eki (Roadside Stations) in rural regions,"(2) ("Michino-Eki FOTs"), we analyzed drive recorder video and identified
typical unsafe and inefficient means of communication between
automated service vehicles and traffic participants(3), shown in
Fig.1. For these typical cases, a focus was placed on methods
that included means of communication between traffic
participants and automated golf carts (6-passenger capacity)
traveling at up to 12 km/h along only electromagnetic induction
line, especially external human machine interfaces ("eHMI").
We conducted experiments in a virtual reality environment(VR experiments) using a head mounted display, experiments
using a driving simulator, and road experiments using anexperimental vehicle on campus and in other locations,researching and proposing means of communication for each

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 139

139

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿5) Research on Communication between Low-Speed Automated Transportation and Logistics Services Vehicles and Surrounding Traffic Participantscase. In cooperation with Michi-no-Eki FOTs, we also
conducted FOTs relating to means of communication involving
eHMI, and evaluated the effectiveness of these means.

avoidance behavior, etc. when the eHMI gave them visual orauditory information about the intent of the vehicles, orrequests to avoid the vehicles.

2.3. Results of the Experiment(a) Approach/avoid(b) Cross(c) Overtake

Fig.1: Categories for communication with traffic participants

Concerning avoidance behavior after a pedestrian notices anautomated service vehicle approaching from behind, we
observed numerous cases of inadequate avoidance bypedestrians, including them not getting off the roadway ormoving over a little bit to the road shoulder when no eHMI was
deployed. However, when the pedestrians were given
information from an eHMI, many practiced proper avoidance.

Communication methods for

2 approach and avoidance situations

Sufficiently move out of the vehicle's path

100％

1

For approach and avoidance situations, we researchedcommunication methods through VR experiments that focusedon eHMI, for example, that included visually or auditorily
conveying vehicles' intent to pedestrians in situations where
automated service vehicles approach pedestrians from behind.

2.1. eHMI specifications
For eHMI used for approach and avoidance situations, as
shown in Fig.2, we developed the phrases "now moving" toindicate vehicle intent and "please clear the way" as a request forpedestrians, adding auditory cues to the existing visual cues.

No eHMIeHMI (visual)"Now moving"eHMI (visual)"Please clear the way"
"Please clear
the way""Now moving"

Percentage

80%
60%

2

Others

2

2

4

7

40％

5

20％
0％

Not move over enough or at all

6

6
4

1
No eHMI

VisualAuditory"Please clear the way"

VisualAuditory"Now moving"Fig.3: Pedestrian avoidance behavior upon first notice

We also analyzed how much time it took for experimental
participants who conducted proper avoidance and moved
completely off the roadway after noticing an automated service
vehicle to complete the avoidance process. Those given any sortof auditory cues took roughly 5 seconds to complete theavoidance process, whereas those given visual cues tended totake longer, although there was considerable individual
variation. In a retrospective report released after the experiment
was completed, many experimental participants did not knowthat automated service vehicles could not drive themselvesautomatically separated from electromagnetic induction line.
When no eHMI was used and an automated service vehicle

eHMI (auditory)"Now moving"eHMI (auditory)"Please clear the way"

Fig.2: eHMI specifications

had stopped before the experimental participants turned
around and noticed them, many said that they didn't knowwhat they're being told about. Even when the eHMI was used,

2.2. Experiment method
We built road environments modeled after non-intersection
segments in areas surrounding Michi-no-Eki (Roadside
Stations) and had experimental participants—posing as
pedestrians—look back and confirm the state of an automatedservice vehicle approaching from or stopping behind them as
the pedestrians walked down the left side of a non-intersection
segment. We then studied those pedestrians' understanding,people told us, for example, that "I thought I just needed tomove over to the shoulder a little and the automated service
vehicle would correct a little and avoid me," "‘Now moving’ isnot enough to prompt me to avoid vehicles," "(‘Please clear theway’) makes me feel like I am being forced to take action, likeI'm being admonished in front of others" (Visual cues) arealways shown, but I'm not used to looking for them and so I
don't notice them right away."

140

SIP通し.indb 140

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿5) Research on Communication between Low-Speed Automated Transportation and Logistics Services Vehicles and Surrounding Traffic ParticipantsCommunication methods for street

3 crossing situations

For street crossing situations, regarding automated service
vehicle behavior and eHMI-based communication methods
identified through VR experiments focused on non-intersection
segments and parking areas, we studied communication
methods through on-premises road experiments that involved
both the presence and absence of human crew members.

3.1. Specifications for test vehicles, deceleration behavior, and eHMI
The experimental vehicle was a golf cart capable of automateddriving, as shown in Fig.4. As with the Michi-no-Eki FOTs, thisvehicle was capable of automated driving along electromagnetic
induction line buried under the street. For the eHMI, we
installed LED panels on the dashboard under the windshield.
We created three types of information communicated topedestrians by the experimental vehicle: "After you" (request topedestrians), "Vehicle will stop" (vehicle intent), and "Vehicle isin automated driving mode" (vehicle control state). We alsoestablished three types of deceleration behavior for street
crossing situations based on experimental vehicle deceleration
start point and stop point when traveling at 12 km/h (as shownin Fig.5): normal deceleration, early deceleration, and early
LED panel・Human crew member present
・Human crew member absent

・After you
・Vehicle will stop・Vehicle is in automated driving mode

Fig.4: Experimental vehicle and eHMI specifications

・Normal deceleration

・Early deceleration

・Early stop

12km/h

0km/h

0

5

10

15

20

25

Distance between automated service vehicle and participant (m)Fig.5: Deceleration behavior types

30

stop.

3.2. Experiment method
To simulate the residential roads with few sidewalks in areas
around Michi-no-Eki (Roadside Stations), we set up
comparatively narrow on-premises roads. We had experimental
participants, as they waited on the road shoulder to cross theroad, observe the behavior of automated service vehicles,
eHMI, and the presence or absence of human crew member, as
the vehicle approached at 12 km/h from up the right side of thepedestrians. We then examined whether the experimental
participants perceived that the automated vehicle had given
way to them or not, and also whether they could judge to crossthe street safely.

3.3. Results of the Experiment
The following are the results of tests involving the presenceof human crew member under the same driving condition as
was present in the Michi-no-Eki FOTs. Fig.6 shows the timingwith which pedestrians decided to start crossing as theexperimental vehicle approached. When no eHMI was present,pedestrians made the decision to start crossing several seconds
after the vehicle stopped. When an eHMI was used, pedestrians
made the decision to start crossing at about the same time oreven before the experimental vehicle stopped. Some participants
made the decision to cross even earlier when the vehicle
conducted early deceleration, and participants tended to startcrossing after the vehicle started decelerating but before it
stopped when given the "After you" cue.
When no eHMI was used, participants' anxiety when
deciding whether to start crossing, was high when vehicles
decelerated or stopped in earlier timing, and conversely was
low when the vehicle conducted normal deceleration. When an
eHMI was used, participants' anxiety was lessened when thevehicle performed an early stop.
To summarize the results of the above, we expect that, instreet crossing situations, the recommended communication
methods will be to conduct with deceleration behavior such as
Timing of decision to begin crossing (sec)

To summarize the results of the above, we expect that, inapproach and avoidance situations, the recommended
communication methods will convey avoidance requests topedestrians using eHMI that provide either both auditory andvisual cues or only auditory cues.

■ Early deceleration■ Early stop

■ Normal stop

7.5
Golf cart stop timing

5.0
2.5
0.0
-2.5
-5.0

No eHMI"After you"

"Vehicle will stop""In automateddriving mode"

When human attendants are present

Fig.6: Deciding timing to start crossing when human crew members arepresent

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 141

1412023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿5) Research on Communication between Low-Speed Automated Transportation and Logistics Services Vehicles and Surrounding Traffic Participants

4 overtaking situations

As a communication method to reduce unsafe behavior bydrivers approaching from behind an automated service vehicle
and overtaking it, we conducted DS experiment to study the
impact that had on such drivers' understanding and overtaking
behavior when they are provided with vehicle state, type, andcaution information when they overtake the automated vehicle,
rather than directly conveying vehicles' intent to stop or yield.

4.1. Road environment and eHMI specifications
As shown Table 1, road environments were constructedaccording to different types of centerlines. As shown in Fig.7,
we established three types of specifications for eHMI used forovertaking situations, indicating them on the rear of theautomated service vehicles. These are "Low-speed travel /Automated vehicle" (alternately displayed in one-second
intervals) to indicate vehicle state and type, "Be careful when
overtaking" to alert drivers when passing vehicles, and "Bewareof surroundings / Please proceed" (alternately displayed in onesecond intervals), which includes intent to yield as a comparison.
Table 1: Studying communication according to centerline types
Road
environment

Goal of communications fromgolf cart to following vehicle

Golf cart vehicle state/behavior

Yellow
centerline

・Restrict overtaking
・Efficiently corresponding tofollowing traffic

・Maintain travel/driving state・Stop after deceleratingWhite
centerline

・Alert when overtaking
・Efficiently corresponding tofollowing traffic

・Maintain travel/driving state

4.2. Experiment method
After building a road environment modeled after nonintersection segments in areas around Michi-no-Eki, we had
experimental participants approach and overtake automatedservice vehicles traveling along roads in the area. We studied
the impact that was had on drivers' understanding, overtaking
behavior, etc., by automated service vehicle behavior, lighting
system operation (direction or hazard indicator lamp, stoplamp) and the presence or absence of eHMI. We set up
contactless eye tracking systems on the DS, measured theparticipants' gazes as they drove, and analyzed where theylooked from the time they approached the automated service
vehicle to when they started and completed their overtaking.

Low-speed travel↓↑

Automated vehicle(a) Low-speed travel/ Automated vehicle

Beware of surroundings↓↑

Be careful when overtaking(b) Be careful when
overtaking

Please proceed(c) Beware of surroundings/ Please proceed

Fig.7: eHMI specifications

4.3. Results of the Experiment
The following is a summary of the results of experimentfocused on white centerlines. Fig.8 shows time spent looking at
surroundings, starting from when participants approached theautomated service vehicle to when they started to overtake it.
Participants looked at their surroundings for about five seconds
when no eHMI was used, for around eight seconds when issued
a "Be careful when overtaking" alert by the eHMI, and about
five seconds when issued a "Low-speed travel / Automatedvehicle" alert. When issued a "Beware of surroundings / Pleaseproceed" alert, participants spent relatively less time looking attheir surroundings.
Time spent looking at surroundings (sec)with early stop and normal stop, and supplementally to convey
Communication methods forthe vehicle's intention to stop or give way to them.

11
10
9
8
7
6
5
4
3
2

No eHMI"Low-speed travel""Automated vehicle""Be careful when "Beware of surroundings"
overtaking""Please proceed"

Fig.8: Time spent looking at surroundings,
from start of approach to start of overtaking

Regarding participants' recognition of receiving the rightof-way from automated service vehicles, recognition was low
when no eHMI was used. Recognition was highest when theeHMI issued the "Beware of surroundings / Please proceed"
alert. Although "Low-speed travel / Automated vehicle" and "Be
careful when overtaking" do not explicitly express vehicles'
intent to yield both alerts yielded greater such recognition whencompared to situations where no eHMI was used. Compared towhen not using an eHMI, participants receiving eHMI alerts
felt less irritation from the time they approached the automatedservice vehicle to when they started overtaking it.
To summarize the results of the above, we expect therecommended communication methods will use eHMI to give
drivers information on vehicle state, type, when they overtake
an automated service vehicle.

FOTs to verify communication

5 methods using eHMI

In conjunction with Michi-no-Eki FOTs and with a focus on
methods of communication between an automated service
vehicle and traffic participants that were identified through VR,
DS, and on premises road experiments, we used actual traffic
environments to verify communication methods for crossing
and overtaking situations.

142

SIP通し.indb 142

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿5) Research on Communication between Low-Speed Automated Transportation and Logistics Services Vehicles and Surrounding Traffic Participants

As shown in Fig.9, we installed an LED panel to serve asa rear eHMI on the rear cargo bed of an automated golf cart
experimental vehicle used in the on-premises roadexperiments. Activation and deactivation of front and rearlighting on the eHMI were manually controlled by humancrew members. We configured the front eHMI to display"After you" and "Vehicle will stop" for crossing situations,and configured the rear eHMI to display "Low-speed travel /Automated vehicle" (alternately displayed) and "Be carefulwhen overtaking" for overtaking situations. We configured
the front eHMI to display these messages roughly
concurrently with deceleration start, and for the rear eHMI
to display the messages constantly. We also mounted several
drive recorders and cameras on the experimental vehicle tocapture traffic participants' expressions and behaviors. The
eHMI and messages were used in each FOT location after
application was filed and approved for a relaxation ofregulations.

eHMI (front)

eHMI (rear)Fig.9: Experimental vehicle and eHMI used in the FOTs

5.2. Characteristics of the FOTs area and road traffic
conditions
With support from the Road Bureau of the Ministry ofLand, Infrastructure, Transport and Tourism, Highway
Industry Development Organization, local governmentorganizations, and local transportation service providers, we
conducted FOTs around Akagi-Kogen Michi-no-Eki (Iinan,
Iishi District, Shimane Prefecture, hereinafter "AkagiKogen") and the Yamakawa region of Miyama (Miyama
City, Fukuoka Prefecture, hereinafter "Miyama"). The AkagiKogen service route has many residential roads and little
vehicle traffic. The Miyama travel route has national
highways and considerable vehicle traffic, including large
vehicle traffic, and there is a considerable speed difference
between these vehicles and automated service vehicles. In
all of the FOTs areas, parking areas existed at the start/end
points and along the routes, and pedestrians could
encounter situations where they crossed the path of theautomated golf carts in the parking lots.

5.3. Experiment method
Under the guidance of local government organizations andservice managers, the experimental vehicle was run for 10 days
during daytime hours in each FOT area. To ensure traffic safety,
we followed the guidance we were given and had theexperimental vehicle slow down and pull over to the left when
the human crew members detected the approach of following
vehicles so that following drivers could overtake theexperimental vehicle safely. Using drive recorders and cameras
mounted on the experimental vehicles, we recorded video dataof traffic participants' expressions and behavior while thevehicle was in operation. We chose eHMI message types basedon the number of observations to be made and vehicle operation
state, including displaying no message at all.

5.4. Results of the Experiment
To ensure that applicable instances of communication were
missed, we recorded all traffic participants that were present
within a certain radius from the test vehicles. In Akagi-Kogen,this was 33 pedestrians and 114 vehicles (four-wheel vehicles
and large vehicles), and in Miyama this was 13 pedestrians and525 vehicles. We established the road crossing situations as
those where pedestrians noticed the test vehicle and crossed theroad. For vehicle passing situations, due to the large number ofincidents recorded, we selected and analyzed incidents at
random.
For crossing situations, this paper analyzes observations
made in Akagi-Kogen. We calculated the time it took forpedestrians to start crossing after noticing the experimental
vehicle or after the experimental vehicle stopped. These results
are shown in Fig.10. Although the number of observations is
insufficient, when pedestrians were provided with no cues froman eHMI, they took at least five seconds to start crossing. Whencues from an eHMI were provided, they took within two
seconds.
For overtaking situations, this paper conducts analyses ofobservations made in Miyama. The analyses, which look atsituations where following vehicles approach and overtake theexperimental vehicle from behind, examine the traveling stateof the following vehicles just before attempting to overtake theexperimental vehicle. These analyses also take into account
6.0

Crossing timing (sec)

5.1. Experimental vehicle and eHMI specifications

Golf cart stop timing

4.0

2.0

0.0

−2.0

2

No eHMI

3"After you"5

"Vehicle will stop"

Fig.10: Pedestrian crossing timing in Akagi-Kogen

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 143

143

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿5) Research on Communication between Low-Speed Automated Transportation and Logistics Services Vehicles and Surrounding Traffic Participants

Constant speed

Decelerate

Slow moving[References]

Stop

Oncoming vehicle
absent30

Low-speed travel /Automated vehicle

Oncoming vehicle
present

Speed upLow-speed travel /Automated vehicle3

Be careful when
overtaking

2

No eHMIBe careful when
overtaking

28

2 2

27

6

0%

4

1

4

No eHMI

2

6

20%

1

2

2

40%

60%

1

2
80%

100%

Fig.11: Travel state of following vehicles just before their start overtaking
in Miyama(1)	The Second Phase of SIP-adus Research and Development in FY2020: HMI andUser Education, https://en.sip-adus.go.jp/rd/rddata/rd04/e206s. pdf, (accessed
August 5, 2022)(2)	Commencement of Long-term Automated Driving FOTs in rural areas —Toward Formulating Standards for Road Spaces for Automated Driving, https://www.mlit.go.jp/common/001259382.pdf (accessed August 5, 2022)(3)	Tatsuru Daimon et al., Research on Communication between Low-Speed
Automated Transportation and Logistics Service Vehicles and Surrounding
Traffic Participants, SIP 2nd Phase: Automated Driving for Universal Services—
Mid-Term Results Report (2018-2020), pgs. 108-112, 2021

[Contacts]
Faculty of Science and Technology, Keio University, 3-14-1 Hiyoshi, Kohoku-ku,
Yokohama, Kanagawa 223-8522, 045-566-1441, Contact: Tatsuru Daimon

whether or not there were oncoming vehicles. These results areshown in Fig.11. Most of the service routes in Miyama arenational highways, with vehicles traveling at speeds muchhigher than the 12 km/h or slower of the experimental vehicle.
Although the number of observations was low when there were
oncoming vehicles, we observed following cars decelerating
before overtaking the experimental vehicle when provided withcues from the eHMI, suggesting these cues encourage slowing
down before overtaking.
We also observed a small number of cases where, even whenthe eHMI provides cues, oncoming vehicles and following
vehicles interfere with each other. This could be due to the roadenvironment or to experimental vehicle stop timing or
positioning when yielding to following vehicles. A more indepth investigation is needed with regard to communication
methods that take into account possible interference between
traffic participants.

6 Conclusion
Aimed at achieving safe and efficient communication
between an automated service vehicle traveling at low speed
and traffic participants, this paper reports on the results ofvarious experiments conducted to evaluate the effectiveness ofcommunication methods and eHMI design recommendations,among other things, for different traffic situations. We observed
instances of unsafe or inefficient communication that is difficult
to collect data on through VR, DS, and on-premises roadexperiments and that relies on actual road environments andtraffic conditions. Furthermore, we were unable to conduct anadequate number of observations due to FOT time limitations.
Further FOTs and other studies will need to be done in order toobserve communication situations and evaluate the effectivenessof communication methods.

144

SIP通し.indb 144

2023/03/01 16:47

3

Ensuring the Safety of Automated Driving

6) Research of HMI for Advanced AutomatedDriving Systems
Toshihisa Sato, Kunihiro Hasegawa, Yanbin Wu, Ken Kihara (National Institute of Advanced Industrial Science andTechnology (AIST)), Kimihiko Nakano, Yang Yo (The University of Tokyo)(Abstract) This paper outlines the efforts in "Development of evaluation methods of driver’s
OEDR (Object and Event Detection and Response) and HMI for enhancing driver’ takeover in a transition fromautomated to manual driving (Task B)" in the second phase of SIP-adus (Cross-ministerial Strategic Innovation
Promotion Program (SIP) Automated Driving for Universal Services). In Task B, we considered evaluation indices for thedriver's peripheral monitoring status before a takeover from automated to manual driving, and the effects of HMI on thedriver's understanding of the system. We conducted driving simulator experiments to study a method for thequantitative evaluation of the driver's OEDR in preparation for driving in a system-initiated driver takeover from Level 3
automated driving. From the experiment results, we identified an evaluation index for the driver's OEDR and determined
the time required for appropriate OEDR. In driver-initiated driver takeover situations from Level 2 automated driving, wefound that eye tracking are effective in evaluating the driver's attention state during Level 2 use. We also clarified the
HMI requirements so that the driver can understand the functional limits of the system appropriately and respond
appropriately before the functional limits are reached.
Keywords: human factors, driver takeover,evaluation methods, HMI ( human-machine interface) , driving simulator

1

Takeovers From Automated to
Manual Driving

When using automated driving, the driving mode may
switch from automated driving to manual driving due tocauses such as reaching a system functional limit or theboundary of the system’s operational design domain (ODD).
Here, we addressed the human factor issue by classifying
takeovers into two categories: system-initiated takeovers, inwhich the automated driving system announces the takeoverto the driver, and driver-initiated takeovers, in which thedriver takes over driving based on his/her own judgment
while the automated driving system is in use.
In system-initiated driver takeovers, when theautomated driving system is at Level 2, the driver needs tomonitor the system’s op erating status as well thesurrounding situation. Yet because it is not always the casethat the driver maintains a state of monitoring, it is essential
the system detects the state of the driver during automateddriving, and knows whether it is possible to switch driving
over to the driver. The first phase of SIP-adus addressed
different driver states, such as looking aside (not facing
forward),(1)(2) attention directed elsewhere (facing forward
but thinking about something other than driving and not
paying attention to the information necessary to drive),(1)(3)and dozing (decreased alertness). (1)(4)(5) The study was
conducted for evaluation indices for each state as well as theeffects of these states on driving performance during driver
takeovers. (6) Based on these findings, we also examined
methods for predicting performance after a driver takeoverbased on the driver's eyeball movements.(7) Furthermore, in
addition to estimating the driver's state during automateddriving and analyzing the effects on driving performance,
we also studied measures to prevent the driver's alertnesslevel from decreasing during automated driving.(8)(9)Efforts in the second phase related to system-initiateddriver takeovers focused on Level 3 use of an automateddriving system. We investigated a method for quantitatively
evaluating the state in which the driver is ready for thedriving task, i.e., the state in which the driver is aware of thesurrounding situation, in order to appropriately carry out
driver takeovers while the driver is performing non-driving
related activities (NDRA) during automated driving at Level 3.
For driver-initiated takeovers, we investigated a method
to evaluate the state in which the driver can appropriately
understand the system’s functional limits at Level 2
automated driving and respond appropriately. In addition,
we experimentally studied the requirements for HMI thatenables the driver to appropriately understand thefunctional limits of the system and respond appropriately.

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 145

145

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿6) Research of HMI for Advanced Automated Driving Systems- We measured the driver's eye movement and head
movements during automated driving. Driving behavior
after switching to manual driving was also measured, andthe lane change success rate was calculated.

2 System-Initiated Driver Takeovers
2.1. Overview of Issues Addressed

2.3. S tudy of Methods for Evaluating Driver’s OEDR
Before a Driver Takeover

In Level 3 automated driving, in order to properly
perform a driver takeover while the driver is doing
something other than driving during automated driving, itis necessary to appropriately shift the driver's attention fromthe non-driving related activity to the driving task. To thispurpose, an effective way to prepare for the driver takeoveris by having the driver recognize the surrounding situation,such as the speed of his/her own vehicle, the movement ofother vehicles around him/her, and the state of the roadenvironment, before performing the driving task manually.
The key issue here is how to evaluate whether the driver isappropriately aware of the surrounding situation. Based on
this awareness of issues, for this topic, we addressed thefollowing issues. (Fig.1)
・Methods for evaluating the driver's awareness of thesurroundings prior to a change of driving
・Time required for proper OEDR・Validity of evaluation methods for OEDR by driver(comparison with standard evaluation methods forperipheral awareness)
・HMI requirements to promote proper OEDR

(1) Experiment conditions
In this experiment, in which 30 drivers participated, thefollowing conditions were set: a condition in which thedriver is given a one-minute warning before the takeover, acondition in which the driver stops playing Tetris andbecomes aware of his/her surroundings one minute before
the takeover, and a condition in which the driver changes
without such a warning and without becoming aware of his/her surroundings in advance.
(2) Results of the experiments: Driving performance afterdriver takeover
The percentage of drivers who were able to change lanes
within a specific section without colliding with othervehicles after the takeover was calculated, and thepercentage was significantly higher in the condition inwhich the driver became aware of his/her surroundings
before the takeover than in the condition in which thedriver did not become aware of his/her surroundings. The
results suggest that, in Level 3 automated driving, the driver
can perform a takeover more appropriately when switchingfrom a state in which the driver is doing something otherthan driving to manual driving by creating a preparatory
state for the takeover by becoming aware of the surrounding
situation.
(3) Results of the experiments: Evaluation index of driver's
OEDR
An analysis of the forward fixation rate after the start ofperipheral awareness based on the driver's visual behavior
as detected by an eye camera showed that the forward
fixation rate tended to increase for 20 seconds after the startof peripheral awareness, and stabilized at a higher value
from 30 seconds after the start of peripheral recognition to

2.2. Experiment Scenarios
Each of the four issues above was examined in driving
simulator experiments. The road traffic scenarios were thesame for all experiments and were as follows- T he vehicle traveled at approximately 60 km/h in thecenter lane of a motorway with three lanes in eachdirection in automated driving mode.
- D uring the automated driving, participants in theexperiment played a game (Tetris) using a tablet PC.
- A few minutes after the start of automated driving, we
initiated a driver takeover, and a lane change was
performed by manual driving within a specific section.

Transition from automated driving Level 3 to manual driving-Planned driver takeover
SystemDriver takeover request (RtI: Request to Intervene)

Automated driving Level 3

Driver takeover mode automated driving OFF (Level 0)

Providing information on the "monitoring of peripheral conditions”

Driver NDRA: Non-Driving Related Activities

Attentiveness: state of readiness
How to evaluate?

Preparation for driver takeover

Manual driving(Non-Driving Related Activities)

Results obtained
from the project's
experiments

The process of increasing situational awareness by paying
more attention to either the front or the periphery (mirrors).
➡ If a takeover is performed at this point, thelane change success rate is lower than that aftera more stable percentage of eye fixation to thefront and periphery.

Stable percentage of eye fixation to front and periphery (mirror) = state of being able to recognize both
➡If the driving is taken over at this point, the lane change
success rate is the same as that of manual driving.
Effective HMI requirements

・The point of a takeover (start of manual driving) can be determined.
・Assuming the posture for driving operations

Evaluation index: driver's eye movement, head movement

Fig.1: System-initiated driver takeover efforts and results

146

SIP通し.indb 146

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿6) Research of HMI for Advanced Automated Driving Systems

the point at which the driver takeover was performed.
Furthermore, the forward fixation rate remained stable after
the driver takeover. The eye movement other than forward
fixation were directed at side mirrors, rearview mirror, andinstruments.
This result suggests that the forward fixation rate after
the start of OEDR can be an effective evaluation index of thedriver's peripheral monitoring, and that the driver can be
judged to be aware of the surrounding traffic situation when
the low forward fixation rate increases immediately after thestart of awareness and then remains stable.
2.4. Examination of the Time Required for Proper OEDR
(1) Experiment conditions
The experiment conditions consisted of the times taken
for the driver to become aware of his/her surroundings
before taking over driving. There were four experimentconditions set at 5, 10, 20, and 55 seconds, and a 0-second
condition (taking over driving from the state of playing agame), plus a manual driving condition rather than an
automated driving condition. There were 30 participants inthe experiment.
(2) Results of the Experiment
The rates of forward and mirror gazing were calculated
for the four conditions in which OEDR was studied, and thechanges from the start of peripheral awareness werecompared. The results showed that the changes from thestart of peripheral awareness were the same in allconditions, and that the rate of forward fixation increased
until 20 seconds from the start, after which it tended tostabilize. Put differently, no such stable shift was observed in
the 5-, 10-, and 20-second conditions. The results of thedriving performance after the takeover showed that the lanechange success rate in the 55-second condition was higher
than that in the 0-, 5-, 10- and 20-second conditions, andwas as high as that in the manual driving condition.
This result indicates that high performance was observedin the driver takeover after the stable shift in the forward
fixation rate, which supports the results in 2.3.
2.5. Study of Validity of Evaluation Methods for Driver
OEDR (Comparison With Standard Evaluation
Methods for OEDR)(1) Experiment conditions
The time for the driver to become aware of his/her
surroundings before the driving takeover was set to 55
seconds, and the verbal thinking method was used to obtainutterances of what the driver was thinking during that time.
In the DS experiment with 20 drivers, we obtained
utterances related to awareness of the surrounding situation,such as, "The flow of cars in the right lane is faster than mycar."(2) Results of the Experiment
From the obtained utterances, we tabulated the numberof utterances related to the situation in front and thenumber of utterances related to the situation in the
peripheral area. The number of forward/periphery-related
utterances from the start of the driver's peripheral awarenessand the change in the rate of gaze to the front/mirrors overthe same time period were compared. The number offorward/periphery-related utterances and the rate of eye
fixation to the front/mirrors showed nearly the samechange.
This result indicates that the change in the eye fixation
rate is large in the sections where the change in the numberof utterances is large. Conversely, the change in the numberof utterances is stable in the sections where the change inthe eye fixation rate is stable. Assuming that the number ofutterances indicates the degree of awareness of the situation,
the result suggests that the change in eye fixation rate may
correspond to the degree of awareness of the situation.
2.6. HMI Requirements to Promote Proper OEDR
(1) Experiment conditions
In order to examine HMI requirements for promoting
OEDR when asking drivers to become aware of theirsurroundings before driver takeovers, we set the following
conditions: a condition in which the driver is informed of
the takeover timing by being told "Takeover after x seconds,"
a condition in which a countdown is given in addition toclearly stating the takeover timing, a condition in which thedriver takes hold of the steering wheel and assumes thedriving posture while becoming aware of his/her periphery,
and a condition in which an alarm is sounded if the driver
does not become aware of his/her periphery. A total of 120
drivers (20 per condition) participated in the experiment.
(2) Results of the ExperimentComparisons of the times it took to complete lanechanges and of the collision rate showed that the results ofthe condition in which the driver is given a countdown in
addition to being informed of the takeover timing and thecondition in which the driver assumes the driving postureand takes the steering wheel during OEDR were better thanthose of the other conditions. At the same time, thecondition in which an alarm was sounded when the driver
failed to become aware of his/her periphery had no effect on
improving driving performance.
These results suggest that knowing the time of a driver
takeover and assuming a posture for driving operations are
effective in improving driving performance after the
takeover.

3 Driver-Initiated Driver Takeovers
Level 2 driver assistance requires the driver to be awareof the surrounding situation and of the system status during

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 147

147

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿6) Research of HMI for Advanced Automated Driving Systems

automated driving, and to take over driving when thesystem reaches functional limits. In this study, weinvestigated a method for evaluating whether the driverunderstands the system status appropriately. We also studied
HMIs that facilitate the driver's understanding of the system
to ensure that the driver can take over driving without fail.
3.1. Method for Evaluating the Driver's Understanding of
the System
We interpreted a driver state equivalent to manual
driving as the state in which the driver is able to coexist withthe system, i.e., he/she understands the system state. We
analyzed the differences in driver behavior during manual
driving and Level 2 driver assistance to investigate a method
for evaluating the state of system understanding.
A driving simulator experiment was conducted with 10
experiment participants,(10) driving in the second lane of aroad with three lanes in each direction by manual driving orLevel 2 driver assistance. We set up potential risk situations
in which the participants should pay attention to theirsurroundings while driving. The participants' eye
movements while driving were measured using a noncontact eye camera. Significant differences were observed in
the duration of eye fixation in several areas, with a tendency
for the participants to spend less time looking at the front
and instruments and more time looking at the peripheryand mirrors during Level 2 driver assistance than during
manual driving.
3.2. HMI to Facilitate System Understanding
Candidates for HMIs that would help drivers understandsystem detection failures and false detections were selected
and operated in a driving simulator environment. An HMI
that displays the recognition results of the traffic situation inreal time was used in a driving experiment with 18
participants.(11) (12) The results of a questionnaire showed
that, when the HMI display was present, drivers understoodthat the automated driving system may not be able torecognize objects other than vehicles, suggesting that the
HMI may promote driver understanding of the system.
3.3. HMI for Preventing Vehicle-to-Vehicle Accidents at
Signalized Intersections
The purpose of this experiment was to investigate therequirements for an HMI to prevent vehicle-to-vehicle
accidents near signalized intersections through appropriate
driver intervention during driving by Level 2 driverassistance. Two types of HMIs were proposed: one that
presents static environmental information based on map
information for the purpose of clearly indicating dangerous
locations (static HMI), and one that presents object
recognition information based on on-board sensor
information for the purpose of confirming the Level 2
driving support mechanism (sensor HMI). We evaluatedthe effectiveness of these HMIs through experiments usinga driving simulator. The HMIs proposed in this study
display the presented information on a heads-up display
fixed to the dashboard of the driving simulator.
The road environment used in the experiment simulated
a Japanese national highway, with two lanes in eachdire c t ion and an addit iona l r ig ht-tur n lane ne arintersections, resulting in three lanes. In the straight-ahead
scenario, the vehicle followed another vehicle ahead of it in
the second lane at 60 km/h using the Level 2 driverassistance system. The driver did not need to perform any
dynamic driving tasks such as acceleration/deceleration or
steering operations when the driver assistance system wasin operation. In the risk scenario, another vehicle appearedin the right-turn lane of the oncoming lane at a signalizedintersection, and when the driver's vehicle entered theintersection, the other vehicle suddenly started to turn right.
The situation was such that if the driver did not intervene indriving, a collision would have occurred. The Level 2 driverassistance system did not request any driver intervention,and the driver had to detect the hazardous event or anomaly
on his/her own and take the initiative to avoid the collision.
The four experiment conditions were a combination of thepresence and absence of static HMIs and sensor HMIs. An
ana lysis of t he exp er iment d at a showe d t hat t hecombination of a static HMI and sensor HMI resulted in thedriver intervention with the largest margin between vehicles
and significantly increased the driver's sense of driving
safety compared to driving without HMIs.(13) (14)In the right-turn scenario, the vehicle makes a right turnat a signalized intersection in Level 2 driving. In the risk
scenario, a motorcycle slips out from the oncoming lane
while the vehicle is turning right, and the driver must
intervene to avoid a collision with the motorcycle. The
driver intervention time and the collision margin time uponintervention were analyzed. The driver intervention timetended to be shorter and the collision margin time tended
to increase when the static HMI and the sensor HMI wereused together.
3.4. HMI for Accident Prevention During Traffic Signal
Changes at Signalized Intersections
During Level 2 driver assistance, in which control bysignal recognition is not performed, the driver is required toperform driver-initiated intervention to stop the vehicle
when the signal turns yellow on approach to a signalizedintersection when engaging follow control by adaptive
cruise control. The purpose of this experiment was toinvestigate HMI requirements to support safe travel at asignalized intersection when the signal light color changes
just before the vehicle enters the intersection during Level 2
driver assistance.
We proposed a static HMI that presents static
environmental information based on map information and148

SIP通し.indb 148

2023/03/01 16:47

3.Ensuring the Safety of Automated Driving
﻿6) Research of HMI for Advanced Automated Driving Systems

a dynamic HMI that presents dynamic environmental
information based on infrastructure information in addition
to static environmental information. The dynamic HMI
presented information such as the predicted color of thetraffic light ahead when the vehicle reaches a signalizedintersection. The information was displayed on a heads-up
display fixed to the dashboard of the driving simulator, andan audible notification was made when the information wasdisplayed. The effectiveness of the system was evaluatedusing a driving simulator experiment.
The road environment used in the experiment simulated
a Japanese national highway, with two lanes in eachdire c t ion and an addit iona l r ig ht-tur n lane ne arintersections, resulting in three lanes. In the straight-ahead
scenario, the vehicle followed another vehicle ahead of it in
the second lane at 60 km/h using a Level 2 driver assistancesystem. When the vehicle and the vehicle ahead of it
approach the signalized intersection where the risk scenario
occurs, the light changes to yellow. The vehicle ahead passes
through the intersection without stopping, but if the vehicle
following it continues to drive with the driver assistancesystem, the traffic light turns red near the stop line. We
analyzed the responses under three conditions: no HMI,
static HMI, and dynamic HMI. It was shown that when
dynamic HMI was used, the vehicle decelerated slowly andstopped with a margin of safety.
This is thought to be because traffic prediction support
information on the traffic signal enabled the vehicle toprepare for deceleration and to stop with a margin of safety. (15) (16)In the left-turn scenario, the vehicle turns left at asignalized intersection by driving at Level 2. In this risk
scenario, the traffic signal changes to yellow as the vehicle
and the vehicle ahead of it approach the intersection. The
preceding vehicle passes through the signal intersection
without stopping, but before this vehicle enters theintersection, the light turns red. If the driver does notintervene, he or she will run the red light. Analysis of the
stopping situation at the stop line and the driverintervention time after the signal change showed that theuse of dynamic HMI increased the number of times thedriver could stop in front of the stop line. In addition, thetime required for the driver to intervene tended to decrease.

Real Proving Ground, Proceedings of DSC 2019 Europe, pp.43-50, 2019.
(4)	Toshihisa Sato, et al.: Evaluation of Driver Drowsiness While Using
Automated Driving Systems on Driving Simulator, Test Course andPublic Roads, In: Krömker H. (eds) HCI in Mobility, Transport, andAutomotive Systems. Driving Behavior, Urban and Smart Mobility,
HCII 2020. Lecture Notes in Computer Science, Vol 12213. pp.72-85,
Springer, 2020.
(5)	Yanbin Wu, et al.: The Relationship Between Drowsiness Level andTakeover Performance in Automated Driving, In: Krömker, H. (eds)
HCI in Mobility, Transport, and Automotive Systems. Driving Behavior,
Urban and Smart Mobility. HCII 2020. Lecture Notes in Computer
Science, Vol 12213. pp. 125-142, Springer, 2020.
(6)	Damee Choi, et al.: Effects of cognitive and visual loads on driving
performance after take-over request (TOR) in automated driving,
Applied Ergonomics, Vol. 85, 103074, 2020.
(7)	Yanbin Wu, et al.: Eye movements predict driver reaction time totakeover request in automated driving: A real-vehicle study,
Transportation Research Part F: Traffic Psychology and Behaviour, Vol.
81, pp. 355-363, 2021.
(8)	Yanbin Wu, et al.: Effects of scheduled manual driving on drowsinessand response to take over request: A simulator study towards
understanding drivers in automated driving, Accident Analysis &Prevention, Vol. 124, pp.202-209, 2019.
(9)	Yanbin Wu, et al.: Age-related differences in effects of non-driving
related tasks on takeover performance in automated driving, Journal of
Safety Research, Vol. 72, pp.231-238, 2020.
(10)	Koichiro Inoue et al.: Transformation of driver mental models byadvanced driver assistance systems, 18th ITS Symposium, 2020.
(11)	Koichiro Inoue et al.: Effects of Presenting Object Recognition Results
to Drivers on Mental Models for Advanced Driver Assistance Systems,
29th Transportation and Logistics Sector Conference (TRANSLOG
2020), 2020.
(12)	B o Yang, et al.: Influences on Drivers’ Understandings of Systems byPresenting Image Recognition Results,IEEE Intelligent Vehicles
Symposium Workshops, 2021.
(13)	Saito, T. et al.: HMI requirements to support appropriate driver takeover
during level-2 driver assistance on prefectural and municipal roads at
signalized intersections, 19th ITS Symposium, 2021.
(14)	B o Yang, et al.: Influences of Different Traffic Information on Driver
Behaviors While Interacting with Oncoming Traffic in Level 2
Automated Driving, International Journal of Human–Computer
Interaction, pp.1-9, 2022.
(15)	Saito, T. et al.: HMI requirements to support proper driver takeover
during Level 2 driver assistance on prefectural and municipal roads,
30th Transportation and Logistics Sector Conference (TRANSLOG),
2021.
(16)	Yang, P. et al.: Investigation of HMI requirements for safe signalizedintersection passage during Level 2 driving, 58th Annual Conference of
the Japan Council of Traffic Science, 2022.

[Contacts]
National Research and Development Agency Human Mobility ResearchCenter, National Institute of Advanced Industrial Science and Technology(AIST), 1-1-1 Higashi (Central 6), Tsukuba, Ibaraki, Contact: Sato Toshihisa[References]
(1)	Yanbin Wu, et al.: Assessing the Mental States of Fallback-Ready Drivers
in Automated Driving by Electrooculography, Proceedings of 2019 IEEE
Intelligent Transportation Systems Conference (ITSC), pp. 4018-4023,
2019.
(2)	Toshihisa Sato, et al.: Evaluation of Driver Visual Distraction in
Automated Driving Systems in Driving Simulator, Test Course, andPublic Roads Experiments, Proceedings of 7th International Conference
on Driver Distraction and Inattention (Book of Abstracts, DDI 2021),
pp. 58-60, 2021.
(3)	Toshihisa Sato, et al.: Comparison of Driver Conditions in AutomatedDriving Systems and Transition Behaviors in Driving Simulator versus

SIP 2nd Phase: Automated Driving for Universal Services - Final Results Report (2018-2022)

SIP通し.indb 149

149

2023/03/01 16:47