Technical Document
The ICT Platform For Connected Vehicles
2021.11.29© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Ver.1.0

Introduction
The automobile industry is facing a once-in-a-century period of transformation at the
moment. With the introduction of many connected vehicles in the market in recent years,
various services have been realized for drivers, passengers, automakers, insurancecompanies, and repair factories. In the future, further widespread and market expansion
are expected.
The spread of connected vehicles enables utilization of big data detected by sensors.
The utilization of data analysis results using ICT platform technology is expected tocontribute to safe transportation (zero traffic accidents/congestion), solutions to social
issues including disaster damage mitigation, and mobility services that provide new value.
This document presents an overview of technical results on the following platform system.
・ High-speed/effective collection of big data generated from connected vehicles
・ Verification of use cases with high technical difficulties as ICT platform for connectedvehicles (for example, real world simulation in cyberspace)・ System supporting End to End verification tests with actual vehicles in the real field
We hope this technical document will be helpful broadly not only for those engaged inconnected vehicles, ICT, and mobility services, but also for other industries.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

1Index ICT Platform for Connected Vehicles
 Field Trials• Overview and Architecture• System, Hardware, and Software Configurations• Use Case 1: Static Map Generation• Use Case 2: Obstacle Detection• Use Case 3: Congestion Detection

 Future Initiatives© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

2ICT Platform for Connected Vehicles© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

3

Overview of ICT Platform forConnected Vehicles
The ICT platform for connected vehicles connects a vehicle with data communication modules,
edge computing, and the cloud via LTE and 5G mobile networks and IoT networks. It also
applies computing resources to collect, accumulate, and analyze vehicle data.

Data center

Edge computing

Data collection,
accumulation and analysis

IoT network

5G and LTE© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

4

Requirements for ICT Platform forConnected Vehicles
The major requirements for developing an ICT platform for connected vehicles are ensuring the capacity for the numberof vehicles connected to the system, real-time execution, and precision, as the value and use of data depend on their
being obtained in real time from a sufficiently large number of vehicles and processed with a high level of precision.

Requirement
1

Requirement
2

Requirement
3

Number ofconnected
vehiclesReal-time
execution

Precision© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Handling of ultra-large volumes and ultra-hightransactions to collect and apply vehicle and image
data from tens of millions of vehicles

Real-time collection and compilation of the position
information of vehicles driving at high speeds to send
real-time information to companies so they can provide
their services

Estimation of the position of landscape features,
including vehicles, signs, and traffic signals, at a higher
precision than that of GPS for managing theinformation on the data center side5

Field Trials© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

6

Objectives of Field Trials
The objective of the field trials was to assess technologies used in cloud and communications infrastructure and toclarify requirements of next-generation vehicles toward a full-scale rollout of connected vehicles and autonomous
driving cars.

Objectives of Field Trials

1

2

Determining cloud andcommunications
infrastructure technologies
for large-scale platforms
and identifying
technological requirements

Determining what cloud technologies will contribute to thedevelopment of connected vehicles and autonomous driving as wellas the technological requirements for realizing a next-generation
communications infrastructure. To this end, we fully applied theexpertise in design, verification, and evaluation cultivated throughour collaboration to date while setting 2023 as the target year.

Identifying requirements fornext-generation vehicle
development

Identifying technical restrictions on the cloud and communications
infrastructure and requirements for vehicle development at an earlystage, as a longer process is taken to address these needs
compared to those of ICT, toward the full-scale rollout of connectedvehicles targeted in 2023 and increased integration of autonomous
driving vehicles.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

7

Roadmap of Field Trials
The field trials were conducted by setting phased targets over a mid- to long-term period up to FY2020.

Mid- to Long-Term Roadmap of Field Trials
JUMP
STEP
HOP

2020

2019

2018
Performance
Vehicles connected: > 1,000,000 vehicles> 10,000,000 vehicles> 100,000,000 vehicles

Processing time:

About 10 minutes

About 10 seconds

A few seconds

Data processed:> 100 items> 1,000 items> 10,000 items© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

8Overview of Field Trials
The goals of the field trials include a scalability evaluation of a large-scale infrastructure technology, which is used forthe cyber-physical system (CPS), toward the full-scale rollout of the connected vehicles. The top priority was toreproduce the necessary conditions for automated driving and mobility services in the future. Vehicle data was
obtained from test cars driven in the Odaiba area.
Data Collection (from Vehicles)

Current Status and Prediction (Example)

In-vehicle status data:
Static map generation forroute planning

CAN & ECU data

Detection and display ofcongestion status by lane

Peripheral monitoring data:

Data processing
Data integration
Video

Lidar point cloud datae.g., White line detection

Detection of fallen objects and obstacles,
notification alerting following vehicles

Data Collection (from Surroundings)
Detailed road information• High-precision map• Accident black spots
Traffic status• Congestion information• Construction work,
accidents• Weather, road surface

Data processing
Data integration

information© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

9System Architecture for Field Trials
Verifications were based on the following architecture.
Image data
streamQueue

CAN datastore

Micro-batchprocessing
for CAN data

Image data
streamDynamic
deployment ofapplications

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Area selection

Vehicle position
data

Area
notification

Delivery and notification

CAN data
streamDynamic map

Batch processing
for image data

CAN data processing

Reception of
message

Image data processing

Edge

Reception, enrichment and annotation, queue

Data center

Queue

Image
datastore

Vehicle position
data

Network Edge

Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

10Verification Based on Use Cases© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

11

Overview of Verification Process
Verification was conducted on three use cases that focused on the number of connected vehicles, real-time execution,and precision as the key issues for developing an ICT platform for connected vehicles.
No.

1

2

3

Use Cases

Overview

Static map generation

Images are collected from the front-view camera of vehicles moving throughthe Odaiba area to generate static maps using a map generation applicationand image processing platform.

Obstacle detection

Vehicles moving through the Odaiba area detect obstacles in a pseudo mannerand send video data of the obstacles to the cloud. The cloud side identifies the
obstacles and notifies vehicles that are expected to drive on the road where
the obstacles are, in order to alert them.

Congestion detection

Congestion data for a specific road is created based on data obtained from avehicle moving through the Odaiba area, and other vehicles on the same roadare requested to send video data. The cloud side determines the level ofcongestion for each lane based on the video and notifies vehicles that are
expected to take that lane, in order to alert them about the traffic jam.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

12

Verification of Use Case for Static Map
Generation© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

13

Verification Overview
In this use case, we identified the technical issues involved in static map generation by running the system, from thecollection of video taken by vehicle onboard cameras to static map storage, under actual field conditions.

Overview

Verification Points

We verified the capability of collecting video data from specificvehicles in the area designated by an operator (Odaiba) for the
purpose of static map generation.
We also verified that the center requested the transferring video
data to the vehicles and handle the series of processing,including receiving, converting, and analyzing video data as wellas the production of a static map properly.

1.

Functionality
Confirm flow of processes involved in static map
generation.

2.

Real-time execution
Confirm processing time from data collection
instructed by the operator to static map generation.

Batch processing
for image data

Data
centerDelivery andnotification

Reception,enrichment andannotation, queue

Image data stream
processing

CAN data stream
processing

Vehicle© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

14

Verification 1: Description and Results
As for functionality, we conducted verification for the series of processes (1 to 8) from notification anddelivery to the generation of lane maps using test cars, and we confirmed that all processes operated
as required.
Verification Targets
Function

Delivery andnotification

1

Processing of delivery
and notification

2

Reception of message,enrichment, andannotation, queue

CAN data stream
processing

3

Processing of CAN
data stream

Image data stream
processing

4

Processing of image
data stream

Batch processingof image data

5

Objectrecognition/detection

6

Object position
estimation

7

Generation of lanemaps

8

Storage into static
map datastore© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Delivery and notification

1

Request for sending video of Odaiba area

Reception,enrichment, andannotation

Verification Diagram

Data center

No.

Batch processing for image data

5

Selectionof area

Objectrecognition/
detection

2

6

Objectposition
estimation

7 Generation oflanes-precision
static vector
maps

8
Static map

4

Reception, enrichment
and annotation, queue

Image data stream processing
Video frame
chunking

Annotation

3
Receptionof message

CAN
synchronization

Image
datastore

CAN data stream processing
Raw data
conversion to
SI unit values

CAN
datastore

Video data for map generation

VehicleCategory

15

Verification 2
To determine the time required for map generation, we measured and analyzed the time for each process, from therequest for video collection to static map generation.
Verification Description
Category

No.

Item

Description

Request for video
collection

1

Time taken to request video
collection

Time elapsed between the time when a message requesting video collection is sent by the platform side andthe start of the first video transmission

Static map generation
and registration

2

Time taken from video
transfer to preprocessing

The time elapsed between when all chunks of the video have been transferred and the completion of
preprocessing (storing into image datastore)

3

Time taken for object
recognition/detection

The time taken for recognition/detection of an object to the delivery of all video chunks

4Time taken for lane map
generation

The time taken for preprocessing (data shaping and other processes up to when the data is sent to thecongestion detection container), including waiting for the delivery of data chunks

Verification Diagram

3
1

2

4

Receivenotification

Notification
threadVehicleVideo transfer
threadPreprocessing
and notification

DatacenterSend ①

Collection
requestSend ②

Send ③

Send ④

Send ⑤

Send ⑥

Send ⑦

Video processing, CAN synchronization, and position hinting

Image
processing
container
Collaborative
image
processing© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Storage in
image
datastore

Databaseregistration
completed

Objectrecognition/detectionObjectposition
estimation

Generation of lanes-precision static vector maps

16Verification 2: Results
In the field trial, the actual time taken from the request for video collection to static map generation for the trial run
route*1 was approx. 35 minutes*2.

Processing time: approx. 35 minutes
Notification
threadReceivenotification

Video transfer and preprocessing: approx. 11 minutes
VehiclesVideo transfer
threadSend ①

Send ②

Send ③

Send ④

Send ⑤

Send ⑥

Send ⑦

Video transfer
Preprocess Request
for video
ing andnotification collection

Video processing, CAN synchronization, and position hinting

Static map generation: approx. 23 minutes
Storagein image
datastore

Databaseregistration
completed

Preprocessing

DatacenterImage
processing
container

Objectrecognition/
detection

1
minuteImage
processing
container

Object position estimation*2

12 minutes

Generation of lanes-precision static vector maps*2

23 minutes*1 A single vehicle driving approximately 2 km. For more information, see Appendix: Trial Routes for Verification.
*2 An additional 12 minutes or so are required when adopting serial processing of position estimation using Visual SLAM. Actual processing time is not affected in this verification, in which parallel
processing for lane map generation was used.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

17

Summary
Verification
PointsSummary

Future Issues

1. Functionality

We confirmed process flow from video collection to static
map generation using test cars. The processes fromvideo collection to static map generation included:

While we were able to verify the basic operationof the functions, the following future issues must
be resolved for commercialization.

・ Delivery and notification
・ Reception, enrichment, and annotation
・ CAN data stream processing
・ Image data stream processing
・ Object recognition/detection
・ Object position estimation
・ Lane map generation
・ Storage in the static map datastore• Precision measurement and enhancement formaps generated using only data obtained fromthe front-view camera (monocular camera) of
vehicles and GPS data
• Realization of processing for extracting andsending map data to vehicles based on theircurrent location• Realization of efficient data collection by roador by lane, as opposed to the current
assumption of mesh-based data collection

2. Real-time
execution

Approx. 35 minutes overall were required from therequest to send video to map generation, including time
for interval operations that needed to be executed
manually.

—• Video transfer and preprocessing: approx. 11 minutes• Static map generation: approx. 23 minutes© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

18

Verification of Use Case for Obstacle
Detection© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

19

Verification Overview
In this use case, video data of obstacles taken by the onboard camera were sent from the vehicles to the platform, andthe platform provided notification to the vehicles after identifying the obstacles to identify issues in real-time stream
processing.

Overview

Verification Points

The platform identifies an obstacle from the video taken by anonboard camera and sent by a vehicle and then notifies
following vehicles about the obstacle. Notification takes place in
two stages.
Stage 1: Priority on responsivenessStage 2: Priority on precise positioning
Edge computing node, data center

1.

Functionality
Confirm the operation of processes from obstacle detection
to notification.

2.

Real-time execution
Confirm that the following vehicle is notified within sevenseconds of an obstacle being detected by a preceding
vehicle to prioritize responsiveness.

Priority on precise
positioning

Priority on response andrough positioning

O
b
s
t
a
c
l
1
e*

！

Following vehicle

Preceding vehicle*1 Due to the difficulty of creating obstacles on a public road, traffic lights were used as hypothetical obstacles.
© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

20

Verification 1: Description and Results
As for functionality, we conducted verification tests for the series of processes from obstacle detection to notification using test cars.
We verified the following items and confirmed the correct operation of all processes.
(2) Processing of video data for object recognition

Verification Points
Verification Points
1

Receiving video data of obstacles sent from a vehicle (DCM emulation)

2

Identifying traffic lights based on object recognition

3

Identifying the position of traffic light based on object position estimation

4

Searching for vehicles to notify about the obstacle

5

Sending requests for video collection and output to the console

Edge computing
node, data center

Synchronous
Asynchronous

3
Objectposition
estimation(3) Traffic lights identified by object position estimation
are shown4

Priority on response and rough positioning

2
Objectrecognition

Existence
notification

1
VehicleDetails
notification

Obstacle data

Vehicleselection

Vehicleposition

5！
Obstacle

Obstacle© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Obstacle© OpenStreetMap contributors

21

Verification 2: Description
As for real-time execution, we measured and verified response time from the obstacle detection to
notification.
Verification Points
Verification Points

Priority on responsiveness and rough positioning (fromobstacle detection to existence notification)

2

Priority on precise positioning (from obstacle detection
to details notification)

Key Technical Elements
No.

Technical Elements

3

Task offloading to edge computing node
Minimum required functions are offloaded to the edge
computing node to ensure real-time responsiveness toaccelerate notification*1

4Video chunking transfer
Sending video data in shorter chunks to shorten thewaiting time for the platform side to begin processing,to accelerate overall processing*2

Notification prioritizing
precise positioning

Data center

1

Synchronous

2

Position
estimation

1
Edge
computingnode

Measure time taken for the following stages of video
data collection, image processing, and the notification
process

VehiclesNo.

3

Objectrecognition

4
Obstacle

Obstacledata

Asynchronous

Details
notification

Notification
prioritizing
responsiveness

Existence
notification

Vehicleposition

Vehicleselection！
Obstacle

Obstacle*1 For details, see Appendix 4: Vertical-distributed computing technology*2 For details, see Appendix 7: Video chunking transfer technology© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

22

Verification 2: Results
We conducted two patterns of chunked video transfers to measure processing times from obstacle detection to notification. The verification
results are shown below. For notifications prioritizing responsiveness, we achieved our target of notifying vehicles within seven seconds* ofdetecting the obstacle.
(1) 1 second × 2 chunks① Notification prioritizing responsiveness: 4.092 sec.
Driver’sfunction
VehiclesDetectVideo generation
thread② Notification prioritizing precise positioning: 9.212 sec.
Display

Display
Generate①

Generate ②

Video transfer
threadSend ①

Edge
computingnode

Send ②

Recognize ①

Recognize ②

RegisterDatacenterRegisterPosition estimation(2) 0.5 seconds* × 4 chunks① Notification prioritizing responsiveness: 3.901 sec.
Driver’sfunction
VehiclesVideo
generation
threadVideo transfer
threadEdge
computing
node

Detec
t

② Notification prioritizing precise positioning: 8.560 sec.
Display

Display

Generate Generate Generate Generate
①
②
③
④
Send
①

Send
②

Send
③

Send
④

Recognize ① Recognize ② Recognize ③ Recognize ④ Register

Data
centerRegisterPosition estimation*1 While this document lists the fastest value from the results of several tests conducted, all tests achieved the target of notification within seven seconds.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

23

Summary
Verification
PointsSummary
We confirmed process flow from obstacle detection to
notification of nearby vehicles using test cars. The resultsof the verification for the two-stage notification process are
as follows.

1. Functionality

1. Priority on responsiveness- Identification of obstacles by object recognition
processing- Search of nearby vehicles based on vehicle
position and notification
2. Priority on precise positioning- Identification of obstacle position by objectposition-estimation processing- Search of nearby vehicles based on obstacle
position and notification
We were able to achieve the target of notification withinseven seconds of obstacle detection by combining thefollowing technologies.

1. Task offloading to the edge computing node
2. Video chunking transfer
2. Real-time execution

Consideration and Future Issues
While we were able to verify the basic operation of the functions, thefollowing future issues must be resolved for commercialization.
•

Evaluation of precision of object recognition and position estimation- Deviation from the actual object (position, quantity)- Degree of deterioration in precision due to weather and time zones•

Need to limit scope of data collection under current assumptions for
obstacle monitoring
Collecting video data from all vehicles receiving notification will result inhigh transfer and processing costs, so there is a need to limit the scopeof data collection. For example, notification targets can be limited tovehicles approaching the obstacle by combining vehicle identification
within the polygon range of high-performance spatial-temporal data
management technology*1 with a selective data collection algorithm*2.

•

Functions can be offloaded to vehicles to accelerate processing andreduce data traffic•

Load distribution of edge functions and optimal configuration of
applications- Offloading of partial processes in response to load status- Data collaboration between edge computing nodes

Although not included in this document, we also
constructed an architecture that effectively uses edge
computing by applying the following technologies.
3. High-performance spatial-temporal data processing*1
4. Dynamic node selection*3*1 For details, see Appendix 1: High-performance spatial-temporal data management technology and high-performance spatial-temporal query technology*2 For details, see Appendix 2: Data acquisition technology with target vehicle selection algorithm*3 For details, see Appendix 4: Vertical-distributed computing technology© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

24

Verification of Use Case for Congestion
Detection© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

25

Verification Overview
In this use case, we identified the technical issues involved in real-time execution of stream processing by running
processes from the collection of video captured by the vehicles’ onboard cameras to notification of analysis results,
under actual field conditions.

Overview

Verification Points

Data center

Platform identifies congestion from data captured by onboard
cameras and sent by vehicles and then notifies following vehicles.
The verification was executed in two steps.
Step 1. Potential congestion detection up to the notification andrequest for video collection
Step 2. Request for sending video up to the analysis andnotification of congestion causes

Potential congestion
detection

1. Functionality
Confirm process flow from detection of congestion to
notification.
2. Response time
Confirm that the process of requesting videos to be sent toanalyze congestion causes and notify vehicles is executed
within five minutes.

Request for video
collection

Analysis and notificationof congestion causes

Video data

Congestion
information

VehicleRequest forvideo collection© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

26

Verification 1: Description and Results
Regarding functionality, we conducted verification for the series of processes (1 to 8) from detection of congestion to
notification, and we confirmed that all processes operated as required.

Verification Targets

Verification Diagram

Potentialcongestions
detection

1

Mesh aggregation2

Suddenness index
calculation*1

3Lane aggregation

4

Area selection

5

Request for video
collection

Request for sending
video

Analysis andnotification ofcongestion cause

Function

6

Detection of line ofvehicles in next lanes

7

Congestion cause
analysis8

Notification ofcongestion

Potential congestions detection

1

Meshaggregation

Data center

No.

2

3

Suddenness
index
calculation

Request for sending
4 videoLane
aggregationArea
selection

Requestfor sending video
5

Congestion causes analysis and notification

6, 7

Detection ofline of vehicle
in next lanes

VehiclepositionCongestion
information
notification

8
CAN data

Video data

Congestion information

VehiclesCategory

Immobile state

Mobile state*1 For more details, see Appendix 5: Suddenness Index Calculation Method© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

27

Verification 2
We measured and analyzed the time taken for each process involved in requesting video, analyzing congestion
causes, and notifying vehicles.
Verification Points
Category

No.

Item

Description

Requesting video

1

Time taken to request video

Time from the request for sending video issued by the platform side to the start of first video transfer

Analysis andnotification oncongestion cause

2

Waiting time for initial chunk transfer

Time from the start of the first video transfer to the start of processing the chunk on the platform

3

Time taken for video transfer

Time for sending all video chunks

4Time taken for preprocessing

Preprocessing time (data shaping and other processes up to when the data is sent to the congestion
detection container), including waiting time for the delivery of data chunks

5Time taken for congestion detection

Processing time to detect congestion

6

Time taken for cause estimation

Processing to estimate congestion cause

7Time taken for notification

Time from the start of notification processing by the platform side to the reception of notification by
the vehicles

8

Total time

Time required for the entire process

Verification
Procedure

8
1

2

4
3

5

6

7

Notification thread

Receive notification

Receive notification

VehiclesVideo transfer thread

Send ① Send ② Send ③ Send ④ Send ⑤ Send ⑥ Send ⑦

Preprocessing and notification
Request video collection

DatacenterVideo processing, CAN synchronization,and position hinting

Image processing container© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Input data

Send notification

Notify congestion

Estimate causes

28

Verification 2: Results
We measured and analyzed the processing time of each stage for the case*1 in which we achieved the fastest time
under actual field conditions. We found that congestion detection accounted for roughly 60% of total processing time,
while video transfer and preprocessing accounted for about 35% of the total.

Total time: 236.6 sec.
Receivenotification

Receivenotification

Notification
threadVideo transfer and preprocessing: 35%
VehiclesSend ① Send ② Send ③ Send ④ Send ⑤ Send ⑥ Send ⑦

VideotransferthreadVideo transfer: 80.5 sec.

Request forvideo

Preprocessing
and notification

Video processing, CAN synchronization,and position hinting

Send
notification

Input data

Preprocessing: 75.4 sec.
Congestion detection and cause analysis: 60%

DatacenterImage
processing
container

Detect congestion

Congestion detection: 137.8 sec.

Estimate
causes

7.1 sec.

*1 Overview of the trial run results under actual field conditions: successful processing: 6 trials; fastest time: 236.6 seconds; slowest time: 311.4 seconds; average time:
264.7 seconds© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

29

Summary
Verification Points

Summary and Consideration•

For the suddenness index calculation method, a desk study confirmedthat target lanes for aggregation can be reduced by up to 80%•

We used image data captured by real vehicles to verify operations forthis function and confirmed that it can recognize lines of vehicles
occurring in the real world

Future Issues••

1. Functionality•

We were able to verify the entire process, from collecting video datafrom appropriate vehicles and delivering congestion information toappropriate vehicles, in collaboration with the existing logic for vehicle
selection and notification

•
•
•
•
•

•

•
2. Real-time execution•

We constructed the system within the framework of Spark *1 andKubernetes*1 to enable parallel distributed computing in the next phaseof the process and confirmed that the system was operating properly
through collaboration between technologies•

We confirmed that it takes four to five minutes to execute congestion
detection and cause analysis for 70-second videos (video collected in
response to requests for 60-second videos)•

We analyzed the processing time and identified several areas of
improvement; in particular, the process can be shortened by about one
minute by improving the interface to reduce waiting time for processing
on the platform side

•

•
•

•

Use of machine learning to upgrade the suddenness
index table
Refinement of detecting congestion in a vehicle’s own
lane, adjacent lanes, and opposite lanes in area
selection, and selection of optimal lanes
Consideration of selection logic for an optimal vehicle invideo data collection
Response to unavailability and insufficient resolution of
CAN data
Assessment of significance between search results fora large amount of video data
Integration with edge computing
Integration with dynamic node selection technology

Enhancement of throughput and turnaround time (TAT)based on parallel computing for the suddenness index
calculation
Enhancement of throughput and TAT based on
improved interface for the suddenness index calculation
Outsourcing of the suddenness index table calculation
to raise scalability
Enhancement of throughput and TAT by eliminating file
entries from the preprocessing phase for recognizing
congestion in adjacent lanes
Enhancement of throughput and TAT by integrating the
preprocessing phase for recognizing congestion in
adjacent lanes with functions
Enhancement of throughput and TAT by avoiding
redundant video processing from the preprocessing
phase for recognizing congestion in adjacent lanes*1 For details on architecture using Spark/Kubernetes, see Appendix Architecture (Detailed Version)© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

30

Evaluation of Platform© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

31

Evaluation Overview
We conducted an evaluation focused on five components of the architecture for the ICT platform for connectedvehicles*1.
No.
Evaluated Components
Overview
Reception, enrichment and annotation,
queue

Conduct a load test on the collection platform for CAN data and image data sentfrom vehicles under the load of a simulated five million vehicles sending data at
intervals of ten seconds or less.

CAN data processing

Execute stream processing of CAN data sent from vehicles under the load of asimulated five million vehicles to evaluate possibility of real-time storage to the
dynamic database. Position data stored in the database are aggregated basedon micro-batch processing to ascertain performance trends.

3

Image data processing

Measure the maximum capacity of the image processing platform when
executing stream processing of video data sent from vehicles and batchprocessing of massive amounts of image data accumulated in the image
datastore.

4

Delivery and notification

Evaluate performance of search and selection processes for vehicles targeted in
the video request and data transfer phases.

5

Network edge computing

Evaluate the functionality and performance of the platform system featuring
several network functions at edge sites, (called 'Network Edge'), to determine
feasibility of optimizing data flow.

1

2*1 For details on system configuration, see Appendix: Platform Verification System© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

32

Evaluation of Annotation Platform© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

33

Evaluation Overview
Conduct a load test on the annotation platform under the load of a simulated five million vehicles sending data at
intervals of ten seconds or less.

Overview

Evaluation Points

Massive CAN data and image data sent from vehicles must beprocessed in real-time. We evaluated the performance of theannotation platform, which serves as the preprocessing phase for
utilizing data in stream processing.
Image data
streamReception, Enrichment andAnnotation, QueueCAN data
stream1.

Stream processing (preprocessing) of CAN data
Confirmed potential for simultaneously handling 600,000
driving vehicles (5 million × 12% of peak operational
capacity) and performance of each component under thevarious loads.

2.

Stream processing of image data (preprocessing)a. Conduct a load test and measure the processing time ofdynamic prioritization based on vehicle position andspeed information as well as simultaneous processing
within the area.
b. Conduct a load of the queue server and confirm
resource bottlenecks.

Batch
processing

Queue for
video

Queue for
CAN

Performance testing of the annotation platform by data type

Device shadow

Metadata query for annotation
Vehicle position
information
Message reception

Generator

CAN datatransferVideotransfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

34

Evaluation 1
We examined the potential for simultaneously handling a quantitative target of 600,000 driving vehicles (5 million ×
12% of peak operational capacity) and the performance trends for each component.

Description

Conditions

We tested performance trends for CAN data processing by dividing theprocess into two components.
•
Annotation server performance trends•
Queue server performance trends

Measure the maximum limit for the number ofvehicles for four separate cases of send intervals
ranging from one to ten seconds.
Reception, enrichment and annotation,
queue

Annotation Server Performance Trends

Queue Server Performance Trends(Hypothesis)
Shorter send intervals will result in increasing thenumber of messages, thereby reducing the
maximum number of vehicles.

(Hypothesis)
Shorter send intervals will not affect theoverall number of processed data blocks,
thereby maintaining the maximum numberof vehicles.

Queue for
CAN

Queue forvideo

Device shadow

Metadata query for annotation

Reception of message

Number ofprocessed data
blocks/second

Number ofmessages/second

Vehicle position
information• Number of vehicles: 100,000 to 1 million• Send interval: 1 sec., 2 sec., 5 sec., 10 sec.
Generator

Number ofvehiclesTarget number ofvehicles© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Target number ofvehiclesNumber ofvehiclesCAN datatransferVideotransfer

35

Evaluation 1: Results
We measured the maximum number of vehicles for each send interval and found that our performance target of
600,000 vehicles can be attained when messages were sent at an interval of at least five seconds.
Message sendinterval (seconds)

Maximum numberof vehicles

Results

Bottlenecks

1

306,000

Unattained

Annotation server2

540,000

Unattained

Annotation server

5648,000

Attained

Queue server

10648,000

Attained

Queue server

Queue Server Marginal Performance

Annotation Server Performance Trends

Number of processed data blocks/sec. for each sendinterval

Processed messages/sec. for each send interval
1

300 K
250 K
200 K
150 K
100 K

50 K
0
0

200 K

400 K

600 K

3

800 K

Number of vehicles
送信間隔1s1 sec. interval

送信間隔2s2 sec. interval

5 sec. interval
送信間隔5s© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

送信間隔10s
10
sec. interval

Number of processed data
blocks/sec.

Processed messages/sec.

350 K

1. Marginal performance: 300,000 messages/sec.
2. Marginal performance: 640,000 data blocks/sec.
3. Annotation server is not the bottleneck

800 K
700 K
600 K

2

500 K
400 K
300 K
200 K
100 K
0
0

100 K

200 K

300 K

400 K

500 K

600 K

700 K

800 K

Number of vehicles
1送信間隔1ssec. interval

2 sec. interval

送信間隔2s

5送信間隔5ssec. interval

10 sec. interval
送信間隔10s

Ideal value
理想値36

Evaluation 2-a
We measured the critical load and processing time of dynamic prioritization based on vehicle position and speed
information as well as simultaneous processing within the area.

Description

Conditions

In dynamic prioritization processing, vehicles are prioritized according tostatistical data aggregated from position information. Performance is
expected to be affected by the number of vehicles located within thescope of statistic data (inside the spatial-temporal mesh and the circulararea surrounding an event), so verification tests are conducted byaltering the number of vehicles under the following conditions.

The processing limit of vehicles with regard to load per second wasmeasured after designating vehicle position information.

(1) Increase in the number of vehicles in each mesh
Measure the performance involved in the process of retrieving the vehicles
list by increasing the number of vehicles in each mesh.
(2) Increase in the number of vehicles inside the circular area
Measure the performance involved in average speed calculation andcoverage detection for the retrieved vehicles list by increasing the numberof vehicles inside the circular area while fixing the number of vehicles
in each mesh.

Query fixed
metadata forannotationQueryvehiclepositions

Retrievevehicles listin meshDistillation ofhigh priority
vehicleAveragespeed
calculation

Coveragedetection

Reception, enrichment and annotation,
queue

Queue for
CAN

Queue forvideo

Device shadow
Metadata query for annotation

Message reception

Vehicleposition
information

Number of vehicles:
100 to 500

Applied load: 50 to 600 video/sec.
Generator

Scope ①

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Scope ②

CAN datatransferVideotransfer

37

Evaluation 2-a: Results (1/2)
By measuring the performance, we found that the maximum throughput declines as the number of vehicles in eachmesh rises. The bottleneck was the CPU of the annotation server.

CPU usage at marginal performance against changes
in the number of vehicles in each mesh

500

1000

100

450

900

90

400

800

80

350

700

300

600

250

500

200

400

150

300

100

200

30

50

100

20

0

10

0
0

100

200

300

400

500

Number of vehicles in each mesh70

CPU usage (%)

Processing time (mins.)

Maximum throughput

Maximum throughput linked to changes in thenumber of vehicles in each mesh

60
50
40

600
0
0

TPS limit

TAT© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

100

200
300
400
Number of vehicles in each mesh

500

600

38

Evaluation 2-a: Results (2/2)
As for the processing time, we observed the following two characteristics.
1. Processing for retrieving the vehicles list from the dynamic database accounts for over 95% of overall processing time.
2. The processing time of the average speed calculation and coverage detection for vehicles inside a circular area increases linearly withrespect to the number of vehicles inside the area.
Query fixed
metadata forannotation

Query vehicle
position

Retrievevehicles list inmeshDistillation ofhigh priority
vehiclesAveragespeed
calculation

Coveragedetection

Query fixed
metadata forannotation

Breakdown of Processing Time (Overall)

Query vehicle
position

Retrievevehicles list inmeshDistillation ofhigh priority
vehiclesAveragespeed
calculation

Coveragedetection

Breakdown of Processing Time (Post-Processing
for Retrieval of Vehicles List/Details)

700
18
16

Processing time (mins.)

Processing time (mins.)

600
500
400
300
200

14
12
10

2

8
6
4

2

100

0
0
(200, 0)(150, 50)

1(200, 0)(100, 100)

(50, 150)(0, 200)(150, 50)

(100, 100)

(50, 150)(0, 200)

Number of vehicles outside and inside the circular area

Number of vehicles outside and inside the circular area
Query fixed metadata for annotation

Query vehicle position

Retrieve vehicles list in mesh

Distillation of high priority vehicles

Average speed calculation

Coverage detection

1. Nearly all are retrieval processes for vehicles list (impact of the number of vehicles iswithin margin of error)© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Coverage detection (interior/exterior point determination)
Coverage detection (shape formation for determination)
Average speed calculation
Distillation of high priority vehicles

2. Linear increase39

Evaluation 2-b
Conduct a load test of the queue server and identify resource bottlenecks.

Description

Conditions

The marginal performance value of the queue server was
ascertained by sending video data on a regular basis. Dynamic
prioritization processing was not performed to position the queue
server as the bottleneck.
Profile of video data

The maximum number of vehicles processable for load per
second was measured.
Reception, enrichment and annotation,
queue

Queue for
CAN

Length of video: 2 seconds (approx. 2 MB)
Frame rate: 10 fps
Bit rate: standard (6,200 kbps)

Queue forvideo

Device shadow

Metadata query for annotation

Key area for ascertaining the critical load of the queue server

Time taken forstorage

Video data/sec.

Critical load value was determined by measuring processable load perunit time and time taken to store data in the queue.

Maximum numberof vehicles© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Message reception

Vehicleposition
information

Applied load: 1,000 to 4,000 video/sec.
Generator

Number ofvehiclesCAN datatransferVideotransfer

40

Evaluation 1-b: Results
We measured the marginal processing performance of the queue server with respect to load applied by video data. We found that theserver was capable of continuous processing up to loads of around 3,500 video/sec., but performance peaks out above this threshold.

Load (video/sec.)

Preprocessing
time (ms)

Waiting time for sending toqueue (ms)Time taken to complete
storage to queue (ms)

2000

0.918

2.121

30.336

3000

1.312

5.700

48.175

4000

2.653

54.743213.420

Disc I/O Usage by Queue Server at High Loads (%utl)

Processing Performance against Load from Video Data
0.35

3

100%

1

3,500

0.3

3,000

0.25

2,500
0.2
2,000

2

0.15

1,500
0.1

1,000

Time taken for storage (sec.)

Number of items processed (video/sec.)

4,000

1: Peak-out at 3,500 (video/sec.)
2: Rising load leads to longer processing time
3: Bottleneck: disc input/output

100%

0.05

500
0
0

1000

2000

3000

4000

0
5000

100%

Load (video/sec.)
Throughput
スループットAverage processing time
処理時間平均© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

41

Summary
Evaluation Points

Summary
Performance target of 600,000 vehicles was achieved when messages were sent at intervals of five and ten
seconds. The performance trend hypothesis confirmed that the bottleneck changes with send intervals.

•

Message send interval: one second, two seconds- Bottleneck: Annotation server- Reason: More messages are sent at shorter intervals•

Message send interval: five seconds, ten seconds- Bottleneck: Queue server- Reason: Total volume of data processed is unaffected by changes in send intervals

1. Stream processing of CAN data
(preprocessing)a. Dynamic prioritization processing
Process of retrieving the vehicles list from the dynamic database accounts for over 95% of the total processing
time. As expected, the processing time of average speed calculation and coverage detection for vehicles insidea circular area increases linearly with respect to the number of vehicles inside the area.

2. Stream processing of image
data (preprocessing)b. Storage processing to queue server
Server was confirmed to be capable of continuous processing up to loads of around 3,500 video/sec.
The bottleneck was confirmed to be queue server disc I/O.
Common considerations:
The current use case assumes that video data is only sent when an event occurs, so it can be assumed thatthe marginal performance of the overall system does not affect trial results since it applies to image processing
in a later phase. An effective method for data collection must be considered, however, for cases involving theregular collection of video data.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

42

CAN Data Processing© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

43

Evaluation Overview
The CAN data processing platform was verified by measuring the performance of each processing
method (stream processing, micro-batch processing).

Overview

Evaluation Points

Vehicles collect massive volumes of CAN data, and in some cases
the response takes priority over throughput. Therefore, theprocessing method must be determined according to performancerequirements. This verification involved measuring theperformance of stream processing and micro-batch processing.

1. CAN data stream processinga. The marginal performance for a guaranteed response wasconfirmed by using the process of raw data conversion to
SI unit values for regular CAN data as an example.

High throughput andSemi-real-time processing

Real-time processing

Queue forCAN

CAN data
streamprocessing

Micro-batchprocessing for
CAN data

Queue forpotentialcongestions

b. Possibility of achieving the target of 600,000 vehicles (5million × 12% of peak operational capacity) during thestorage processing of vehicle data was examined.
2. Micro-batch processing for CAN data
The effectiveness of detection during search processing was
confirmed by using potential congestions detection by meshand lane aggregation as an example.

Vehicleposition© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

44

Evaluation 1-a
Confirm the marginal performance for guaranteed response using the process of raw data conversion to
SI unit values for regular CAN data as an example.

Description

Conditions

We will realize a high-performance search by storing the latest vehicle position
data in the device shadow. On the other hand, this requires an additional
process of storing data in the device shadow during raw data conversion to SI
unit values. Therefore, we will also verify the level of performance
deterioration in the overall conversion process. Marginal performance isdefined as the maximum number of vehicles for which raw data conversion to
SI unit values is completed within the span of fixed processing intervals andcompared performance for the following two patterns.

The maximum number of vehicles that meet the following conditions
was measured.
• Average processing time for one second of data must be one
second or faster• Total processing time for inputted message load must be around180 seconds
Reception, enrichment andannotation, queue

Queue forCAN

① Without entries into the device shadow② With entries into the device shadow

Metadata
query forannotation

Queue

Queue forCAN

CAN data stream processing

Dataconversion ①

High-performance
spatial-temporalprocessing

Dataconversion ②

Reception of
messageData
conversion

Spatialtemporalprocessing

Execution cycle:
1 second

Vehicleposition
Deviceshadow

Vehicleposition

Deviceshadow

No. of vehicles: 40,000 to 260,000/sec.*
Loading time: 180 secs.
Generator

Send CAN
data© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATIONCAN data stream processing*Verification was conducted based on one-third of
the number of servers required for simultaneously
processing 600,000 driving vehicles, so the load
was also set at one-third of the volume.

45

Evaluation 1-a: Results
We increased the number of vehicles to 260,000 and measured the average processing time for one second of data
and the total processing time for the inputted message load. We confirmed that performance deteriorated due to theadditional process for entering data into the device shadow, reaching the marginal performance at around 240,000
vehicles/second.
Raw Data Conversion Time (With Entries into the Device
Shadow)

1400

1000

Leeway
800

120

600

400

60

200

Average processing time (ms)180

Total processing time (s)Average processing time (ms)

1200

1,400

Accumulated
unprocessed
tasks

1,200

180

1,000

Execution
timeout

800

120

600

400

60

Total processing time (s)Raw Data Conversion Time (Without Entries into theDevice Shadow)

200

0
0

50,000100,000

150,000

200,000

250,0000
300,000

Number of vehicles
Average processing time

Total processing time

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION0050,000100,000

150,000

200,000

250,0000
300,000

Number of vehicles

Average processing time

Total processing time46

Evaluation 1-b
We confirmed the possibility of processing vehicle data storage for the quantitative target of 600,000
simultaneously driving vehicles (5 million × 12% of peak operational capacity).

Description

Conditions

We verified storage processing for vehicle position data by comparing
performance based on the following technologies.
① Geohash② Spatial-temporal aggregation*1

The maximum number of items processed per second wasmeasured.

Marginal performance was defined as the maximum load at which
the processing can be maintained with replicating vehicle position
data.
Queue for
Kafka

Queue for
CAN

Data Conversion
Data
conversion① Geohashtechnology

Data
conversion② Spatial-temporal
aggregation
technology

Primary
data
Vehicleposition data

Vehicleposition data
Replicateddata

Asynchronous
replication*1 For details, see Appendix 1: High-performance spatial-temporal data management technology andhigh-performance spatial-temporal query technology© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Reception, enrichment andannotation, queue

Queue forCAN

CAN Data Stream Processing

Dataconversion

Spatialtemporalprocessing

Vehicleposition data

Metadata
query forannotation
Message
reception

400,000 to 800,000 vehicles
10 seconds of send interval
Generator

CAN datatransfer

47

Evaluation 1-b: Results
We measured the maximum number of vehicles and confirmed the possibility of processing 504,000 vehicles per second based on the
Geohash method and 684,000 vehicles per second based on spatial-temporal aggregation. Storage performance was enhanced by up to
1.36 times by adopting spatial-temporal aggregation.

(1) Geohash
Number ofvehicles(2) Spatial-Temporal Aggregation
Processing
within fixedintervals*1

Status ofreplication*2

Results

Number ofvehiclesProcessing
within fixedintervals*1

Status ofreplication*2

Results

414,000

Pass

Attained

414,000

Pass

Attained

504,000

Pass

Attained

504,000

Pass

Attained

540,000

Fail

Unattained

540,000

Pass

Attained

594,000

Fail

Unattained

594,000

Pass

Attained

Pass

Pass
648,000

Fail

Unattained

648,000

Pass

Attained

684,000

Fail

Unattained

684,000

Pass

Attained

774,000

Fail

Unattained

774,000

Fail

Unattained

864,000

Fail

Unattained

864,000

Fail

Unattained*1 Pass … Processing time ≦ Continuous processing possible at startup intervals, Fail … Processing time > Continuous processing difficult atstartup intervals*2 Pass … Continuous processing possible without stagnating replication, Fail … Replication stagnates, continuous processing difficult without
stagnating replication© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

48

Verification 2
We confirmed the effectiveness of detection during search processing through potential congestion
detection by mesh and lane aggregation as an example.

Description

Conditions

We adopted mesh aggregation and the Suddenness Index Calculation Method
for Aggregated Values for reducing the load on lane aggregation in the latterstage. In this verification, we measured aggregation times as shown below andcalculated total processing time to confirm the effect of using mesh aggregation
for the preceding stage.

Aggregation time was measured by changing position data
within the mesh and lanes.

Suddenness Index
Calculation(1) Lane aggregation only (without detection)

Search for position data

Suddenness index
calculation datastore

Lane aggregation

Time taken for laneaggregation

Calculatesuddenness index(2) Mesh aggregation + around 50% of total lane aggregation (detection for mesh aggregation only)
Search for
position dataMeshaggregation

Suddenness
Index
Calculation

Search for2. Time taken for lane aggregation*0.5

1. Time taken for meshaggregation(3) Mesh aggregation + 20% of total lane aggregation (detection for mesh aggregation +Suddenness Index Calculation)
Search for
position dataMeshaggregation

1. Time taken for meshaggregation

Micro-batch processing
for CAN data

Lane

Lane
レーン集計
位置情報検索position data
aggregation
aggregation

Suddenness
Index
Calculation

SearchLanefor
位置情報検索aggregation
position
data

レーン集計
aggregationLane

2. Time taken for lane aggregation*0.2© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Test
ClientMeshaggregationLane
aggregation

Vehicle position
data

Road map,
Polygon-based
mapQueue forpotentialcongestions

Number of meshes: 200,400,800
Lanes/mesh: 9,16,25,36,49
No. of vehicles /lane: 3,6,12,24,48

49

Evaluation 2: Results
We confirmed that the processing time can be reduced by using mesh aggregation in advance to detect target vehicles for laneaggregation. We consider the effect of detection is more effective with the increase in the number of lanes and vehicles due to the growing
time gap between mesh aggregation and lane aggregation.
Estimates of Processing Time According to Changes in the Number of Meshes (Seconds)
Meshes = 200

Meshes = 400

Meshes = 800

90

90

90

80

80

80

70

70

70

60

60

50

50

50

40

40

30

30

30

20

20

20

10

10

10

0

0

Down
40%

40

0

20
40
Lanes per mesh

① Only lane aggregation

60

Down
70%

0

20
40
Lanes per mesh

② Mesh aggregation + 50% of lane aggregation© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Down
70%

60

60

0
0

10

20
30
40
Lanes per mesh

50

60

③ Mesh aggregation + 20% of lane aggregation

50

Summary
Evaluation Points

1. CAN data stream
processing

Summary

Future Issues

We confirmed the possibility of expanding CAN data stream processing to 30 million
vehicles according to estimates based on the results of platform performance verification.

• Improve architecture to address
abnormal circumstances including
server failures• Processing for raw data conversion to SI unit values
We compared performance in cases with and without entries into the device shadow andconfirmed the marginal performance without entries to be 240,000 TPS.
We can raise the processing time by 1.3 to 1.5 times by adding processing for device
shadow storage to processing for raw data conversion to SI unit values.
• Storage of vehicle position data
We compared the performance of storage processing of vehicle position data based on
the Geohash method and spatial-temporal aggregation and confirmed the possibility ofhandling up to 504,000 vehicles under the Geohash method and up to 684,000 vehicles
under the spatial-temporal aggregation. Adoption of spatial-temporal aggregation
enhanced storage performance up to 1.36 times.

2. Micro-batchprocessing for CAN
data• Improve architecture to enable
response to spikes• Develop architecture for
economically rational data
management, such as shifting
infrequently used data to low-coststorage areas

To confirm the effect of detection, we calculated the following estimates based on the
results of platform performance verification. The results of platform performanceverification, however, have been excluded from this document.
(1) Lane aggregation only (without detection)(2) Mesh aggregation + around 50% of total lane aggregation (detection for meshaggregation only)(3) Mesh aggregation + 20% of total lane aggregation (detection for mesh aggregation +Suddenness Index Calculation)• Improve throughput and turnaroundtime (TAT) by adopting parallel
processing for raw data conversion
to SI unit values

We were able to reduce the number of target lanes for aggregation by including a
detection process in advance as in case (3), thereby achieving a maximum 70%
reduction in processing time compared to case (1), in which aggregation was conducted
on all lanes.
We consider detection to be more effective with the increase in the number of lanes andvehicles due to the growing time gap between mesh aggregation and lane aggregation.

• Improve scalability by storing thesuddenness index table to
distributed datastores© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION• Improve throughput and turnaroundtime (TAT) by enhancing theinterface• Use machine learning to upgrade
the suddenness index table

51

Image Data Processing© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

52

Evaluation Overview
We measured the performance of each processing method (stream processing, micro-batch processing, and batchprocessing) executed by the platform side to process video sent by vehicles.

Overview

Evaluation Points

The volumes of video data collected from vehicles is massive, so
the processing method must be determined according toperformance requirements. In this verification, we measured theperformance of stream, micro-batch, and batch processing.

1. Image data stream processinga. Executed stream processing for video data sent by vehicles
and measured the maximum load of real-time processing.
Since GPU processing represents the bottleneck for theentire system, processing tasks were off-loaded to off-site
GPU nodes depending on operational status.

Real-time and semi-real-time processing
Image data stream
processing (static
map generation)

Queue for
video

Image
datastore

Image stream
processing(obstacle detection)
Micro-batchprocessing for video(congestion detection)

Delivery andnotification© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Large-volume dataprocessing
Batch processing fordata (static map
generation)

Static map

b. Verified basic performance by changing data input to theapplication for detecting lines of vehicles and cause ofcongestion.
2. Batch processing for video
Executed batch processing for large volumes of video data
accumulated in the image datastore to verify the marginal
performance of GPU processing.

53

Evaluation 1-a
Confirmed the marginal performance for a guaranteed response by executing stream processing for video data sent from vehicles. Since
GPU processing represents the bottleneck for the entire system, processing tasks were off-loaded to off-site GPU nodes depending on
operational status. We also verified processing time, with or without task off-loading.

Description and Conditions
The load was applied in the form of image data by enabling dynamic
selection technology for processing nodes*1. Priorities (processing sites)
were changed by modifying the input criteria in the preceding process ofdynamic prioritization, and processing throughput for each site ① and dataprocessing time ② were measured.
Generator
Generator

Distributed via
amplifierAmplifier

Reception, enrichment, and annotation
High only
Receive

Annotation

Receive

Annotation

Mid
only

Low only

Load patterns:
High only, mid only, low only

Receive

Annotation(1) Measure throughput of image processing

Changes in throughput and processing time were verified byadjusting prioritization rules and load patterns to ensure theappropriate measurement of performance for each processing site.

Queue

Image processing (edgenode/data center)

High priority
queue forvideo

High priority
queue for video

Mid priority
queue forvideo

Mid priority
queue for video

Low priority
queue forvideo

Low priority
queue for video

Edge
processing

Maximum numberof terminals*1 For details, see Appendix 4: Vertical-distributed computing technology© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Request for
notification
Deployment patterns:
Edge node only. data
center only

(2) Analyze processing time for each task

Throughput

Number ofterminals

Delivery andnotification

Data center
processing

Retrievevideo

Retrieve video

Video image
conversion

Synchronize
with CAN
data

Video image
conversion

Inference

Synchronize
with CAN
data

Request for
notification

Inference

Request for
notification

54

Evaluation 1-a: Results (1/2)
The comparison between the performance with and without the use of dynamic selection technology for processing nodes found that byusing dynamic selection technology for processing nodes for high priority on-site processing we can maintain the response time throughkeeping the load at a certain level, thereby ensuring that processing is completed within a fixed amount of time regardless of load status.
We also confirmed that marginal performance values can be enhanced by off-loading image data processing off-site to increase thenumber of GPU nodes handling the tasks.

Total message processing time for image data processing
600

Processing time (s)

500

400

Without dynamic
selection:
Edge node: ⭕️Without dynamic
selection:
Edge node: ✖️With dynamic
selection:
Edge node: ⭕️Data center: -With dynamic
selection:
Edge node: ⭕️Data center: ⭕️With dynamic
selection:
Edge node: ⭕️Data center: ✖️Without dynamic
selection:
Marginal performance at
2 GPU: 3TPS

300

200

Without dynamic
selection:
Edge node: ✖️With dynamic selection(data center):Marginal performance at 4
GPU: 6 TPS

Load input time

With dynamic selection(edge node):
Maintain response time

100

0
0

1

2

3

4

5

6

7

8

9

Input throughput (Data/s)© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

55

Evaluation 1-a: Results (2/2)
As a result of analyzing time required for on-site and off-site image data processing, we found there were no effects
other than on the time taken to retrieve the video. Therefore, the time required for data transfer is the only additional
value when processing off-site.

Image processing time by processing site*
0.036

Off-site

0.525

On-site

0.446

0.050
0.000

0.685

0.415

0.691

0.004
0.200

Retrieve video

0.400

0.600

0.800

Video image conversion

1.000

1.200

CAN synchronization

1.400

1.600

1.800

Effect of off-site deployment on
processing time
- Increased:• Retrieve video• CAN synchronization(marginal increase in transfer
time due to small data size)- No change:• Video image conversion• Inference processing

Inference processing*Processing time is the average value resulting from multiple trials.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

56

Evaluation 1-b
We verified basic performance when the data input is changed to the application for detecting lines ofvehicles and cause of congestion.

Description and Conditions
We measured the processing time and throughput of the technology for
detecting vehicle lines and cause of congestion by varying the following data
input conditions.
i. Length of video
ii. Frame rateiii. Bit rate (resolution)
Since GPU use is differed in three sections during image processing, wedefined them as A, B, and C and separately analyzed processing times.

Cause inference

Congestion detection
Section A
Vehicledetection

Congestion
detection bylane

Section B

Section C

When the left lane is congested

Congestion
cause
inference

Variations in data input
Reception,enrichment andannotation, queue

Queue forvideo

Micro-batch processing for video

Detection ofline of vehicles
in adjacentlanes

Congestion
cause analysis

Delivery andnotification

Request for
notification

Confirmed basic performance for variations indata input
i. Length of video: 10 s, 30 s, 60 s
ii. Frame rate: 10 fps, 30 fps, 60 fps
iii. Bit rate:• 1,920 × 1,080 (1,080 p)• 3,840 × 2,160 (2,160 p)• 7,680 × 4,320 (4,320 p)

Detection oflanes insame/opposite
directions
When the right lane is congested© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

57

Evaluation 1-b: Results
Through the evaluation, we found the following relationships between processing time and variations in data input.
i. Length of video: Processing time increases linearly in proportion to the length of video in Sections A, B, and C
ii. Frame rate: Processing time increases linearly in proportion to the frame rate in Sections A and B, and remains within a fixed band for
Section Ciii. Bit rate: Processing time increases linearly in proportion to the bit rate in Sections A and B, and remains within a fixed band for Section C
II. Frame rate

III. Bit rate

Length of video and processing time(by process)

Frame rate and processing time (byprocess)

80

70

180

70

60

160

60
50
40
30
20

50
40
30
20
10

10
0
20

40

60

80

120
100
80
60
40

0
0

20

Length of video (s)

1

140

20

0
0

No.

Bit rate and processing time* (byprocess)

Processing time (s)Processing tie (s)Processing time (s)I. Length of video

40

60

80

0

10

Frame rate (fps)20

30

Section A

Section B

Section A

Section B

Section A

Section B

Section C

Other

Section C

Other

Section C

Other

Length ofvideo

Section A

Section B

Section C

10.1

19

13

10

Other

Total
13

No.

Frame rate
(fps)

Section A

Section B

Section C

1

10

19

13

10

13

55

1

2

30

36

31

11

13

91

2

3

60

61

57

10

16

144

3

55

2

30.2

39

26

12

16

93

3

60.6

67

50

15

23

155

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

40

Pixels (million)

Other

Total

No.

Pixels(million)

Section A

Section B

Section C

Other

1,080 p

2.07

19

13

10

13

2,160 p

8.29

41

18

10

19

4,320 p

33.17

156

43

11

43

Resolution*Time measured in seconds

58Evaluation 2
We verified the marginal performance of GPU processing by executing batch processing for large
volumes of video data accumulated in the image datastore.

Description and Conditions
Obtained marginal performance values for batch processing of large
volumes of video data accumulated in the image datastore. Variable
parameters were used as follows to measure throughput andprocessing time for object recognition/detection.
(1) Number of images processed in a single inference request (batchsize)(2) Image data size

Frame rate

1

10 fps2

10 fps200, 400, 600 vehicles
Single vehicle = 60-second video
Batch processing for image data

Image
datastore

Type of image data
No.

Variations in parameters

Resolution

Image size
(average)

6,200 kbps

High295 KB

6,200 kbps

Low

77 KB

Bit rate

330 fps6,200 kbps

High295 KB4

10 fps4,800 kbps

Low

291 KB5

10 fps8,000 kbps

High

298 KB© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Retrieve
image

Call for traffic
light/road sign
recognition

Load-balancing
for positioning

Variations in number of images(batch size) per request: 1, 16, 32
Traffic
light/road sign
recognition

Measurement of throughput andprocessing time for object
recognition/detection

59Evaluation 2: Results (1/2)
We found that the inference processing time can be reduced by increasing batch size. We also confirmed that batchprocessing time increases in proportion to the number of vehicles. Performance peaks at a batch size of around 32,
and an OOM error occurs when the batch size exceeds 40, due to lack of GPU memory (16 GB).

Breakdown of batch processing time(600 vehicles)

Processing time according to batch size
1:40:00

100

Processing time (m)1:30:00
1:20:00

Processing time

1:10:00
1:00:00

0:50:00

10

80

60
81

40

9

8

34

34

Batch size: 16

Batch size: 32

20
0Batch size: 1

0:40:00

Controller processing time for static map generation

0:30:00

Inference processing time

0:20:00
Batch size

0:10:00
0:00:00
200

400

600

Number of vehicles
Batch size: 1

Batch size: 16

Batch size: 32

Inference
processing time(average)

Inference processing
time (average) per image

1

53.9 ms

53.9 ms

16

363 ms22.69 ms

32

705 ms22.03 ms*

*Larger batch size requires more inference processing
time due to the greater number of images processed per
request.
© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

60

Evaluation 2: Results (2/2)
Performance characteristics for variations in image data were as follows.
• Resolution: Although the size of low-resolution image data was roughly 0.26 times that of high-resolution image data, processing time remained at around0.86 times that of high-resolution image data (since image resolution is an element that contributes to raising precision in subsequent object-positionestimation processing, high-resolution images are preferable if the effect on processing time is limited).
• Fame rate: In line with our hypothesis, an increase in the frame rate raises the volume of image data to be subjected to inference processing, thereby
extending batch processing time.
• Bit rate: The effect on batch processing time is marginal. We believe this is because image quality is fixed when extracting the frames, so the bit rate causes
no difference in data size.

Processing time according to differences in video data size
2:10:00

Magnification of processing time for baseline video
data and variable parameters

2:00:00

Resolution

Processing
time (m)Magnification

1:50:00
High resolution (baseline)43-

Low resolution (0.26 times)

37

0.86

1:40:00

Processing time

1:30:00
1:20:00
1:10:00

Frame rate

1:00:00

Processing
time (m)Magnification

0:50:00

10 fps (baseline)43-

0:40:00

30 fps (3 times)

122

3

0:30:00
Bit rate

0:20:00
0:10:00
0:00:00
200

400

600

Number of vehicles
Baseline

Image quality: Low resolution

Frame rate: 30© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Bit rate: Low

Processing
time (m)Magnification

6,200 kbps (baseline)

43-

4,800 kbps (0.77 times)

42

0.98

8,000 kbps (1.3 times)

44

1.02

Bit rate: High61

Summary
Evaluation
Points

1. Image data
streamprocessing

Summary

Future Issues

a. In the past, the limit of a single edge node was the marginal performance of the entire system.
However, we confirmed the following correlations by adopting vertical-distributed computing.

Improvement in throughput can be accelerated
further by combining the following.

• For high priority on-site processing, response time can be maintained by keeping the load at a certain
level, thereby ensuring that processing is completed within a fixed amount of time regardless of load
status.
• Marginal performance values are enhanced by off-loading image data processing off-site to increase
the number of GPU nodes handling the tasks (marginal performance = 1.5 × number of GPU nodes).

• Reducing traffic by off-loading inference
processing to vehicles• Boosting GPU resources through cloud integration• Reducing the cost of video transfer andprocessing by selective uploading by vehicles

b. The following is a summary of the basic performance verification for variations in data input to theapplication for detecting lines of vehicles and cause of congestion.

• Improving throughput and turnaround time (TAT)by eliminating the two-step disc entries during
preprocessingi. Length of video- Congestion detection: Processing time increases linearly in proportion to video length- Cause inference: Processing time is not dependent on video length and remains fixed**In this verification, we used the same video on repeat as input data, which resulted in magnifying theincrease in the number of congestions in proportion to video length, thereby extending processingtime.

• Improving throughput and turnaround time (TAT)by executing chunk processing via functional
integration with preprocessing• Supplementing deficiencies and lack of resolution
in the CAN data from the platform sideii. Frame rate
- Congestion detection: Processing time increases linearly in proportion to frame rate
- Cause inference: Processing time is not dependent on the frame rate and remains fixediii. Bit rate
- Congestion detection: Processing time increases linearly in proportion to the video length- Cause inference: Processing time is not dependent on the bit rate and remains fixed• Variations in batch size
We confirmed that throughput can be improved by increasing batch size to effectively use GPU. However,GPU becomes the bottleneck when batch size is raised too high, causing throughput to peak.
2. Batchprocessing for
video

—

• Variations in data size
We confirmed that by increasing frame rate, processing time increased at the same magnitude.
Also, since increasing the resolution of image quality to a given level did not affect processing time, highquality image processing for the improved precision of object-position-estimation processing is realistic.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

62

Delivery and Notification© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

63

Evaluation Overview
In the obstacle detection use case, the platform sends a request to collect video data prior to receiving video data fromvehicles. To reduce traffic volume for the data collection, we applied a method of selecting the target vehicles that use
information about the relative positions of the vehicles and obstacles.

Overview

Evaluation Points

We evaluated the performance of the data acquisition systemarchitecture, which incorporates a target vehicle selection
algorithm*1.

1. Processing time for vehicle selection
The relationship between the number of vehicles to be selected andprocessing time for vehicle selection was verified by reproducing asituation that involves searching a large volume of vehicle data.

Overview of Evaluation
Edge computing nodeand data center

Platform cost: low

VideoObstacle is outside
shooting angle

Optimalcondition
Obstacle

Low-spec
cameraToo close to
obstacle*1 The use of this algorithm is expected to reduce platform costs caused by the concentration of loads and processing times by collecting video data exclusively from optimal vehicles
based on their direction, angle of view, and other information, in addition to collecting vehicle position data. For details, see Appendix 2: Data acquisition technology with target vehicle
selection algorithm© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

64

Evaluation 1
We evaluated the relationship between the number of vehicles to be selected and the processing time for vehicle
selection by reproducing a situation that involves searching a large volume of vehicle data.

Description

Conditions(1) Search vehicle
position database(3) Direction detection
Bad
direction

Processing time was measured by varying the number ofvehicles in the search area.

Edge node

Vehicleposition data

Notification oftarget area

Radius of target search area: 100 m

Vehicle(4) Coverage detection

Good
distanceGood
distance

Bad
distance

Target area
selection

Number of vehicles: 100 to 2,000Good
direction(2) Distance detection

Delivery and notification

Data center

The video collection process using the target vehicle selection algorithm consists ofthese four steps. We measured processing time for each step to verify the
relationship between the number of vehicles to be selected and processing time forvehicle selection.

Good
coverage© OpenStreetMap contributors© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Rationale for setting the upper limit at 2,000 vehicles• Area of circle with 100 m radius: 100 × 100 x 3.14 = 31,400 m2• Grid for parking lot per vehicle: 3 × 5.5 = 16.5 m2• Number of vehicles that fits inside the circle: 31,400 ÷ 16.5 = 1,903

65

Evaluation 1: Results
We conducted the evaluation by increasing the number of target vehicles and observed an upward trend in processing time for databasesearch and coverage detection. In particular, coverage detection became the dominant factor when the number of target vehicles exceeded
1,000.
Arrangement of vehicles used in the evaluation

Relationship between the number of vehicles and processing time
300 ms
250 ms200 ms
150 msOther④ Coverage detection③ Direction detection② Distance detection① Database search

100 ms
50 ms
0 ms
100

200

500

1,000

2,000

© OpenStreetMap contributors• Target search area
• Center: “Aomi 1-chome” intersection• Radius: 100 m• Vehicle data
• 100 to 2,000 (positioned at a 90 m radius)• Distance good, direction good, coverage good© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION• Processing times for ② distance detection and ③ direction
detection were minimal• Processing times of ①database search and ④coverage detection
were increased by increase of the number of target vehicles• In particular, the processing time varied significantly depending on
the number of vehicles that require coverage detection• Around 280 ms for 2,000 vehicles• Coverage detection became the dominant factor when thenumber of vehicles exceeded 1,000• Followed the theoretical value for O(n)66Summary
Evaluation
Points

1. Processing timefor vehicle
selection

Summary

Considerations

We varied the number of vehicles to be selected forcollecting vehicle position data and measured theprocessing time for each step of the target vehicle
selection algorithm. As a result, we observed thefollowing results.

The following points were considered in terms of effectively
applying the target vehicle selection algorithm.

• Processing times for “Distance detection” and“Direction detection” are minimal• There was an upward trend in “Database search”
and “Coverage detection”• In particular, the processing time varies significantly
depending on the number of vehicles that require
coverage detection, measuring around 280 ms when
2,000 vehicles are selected© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION• “Distance detection” and “Direction detection” require
minimal processing time, so from the perspective of
eliminating excess notifications, the algorithm should beapplied in use cases where the priority is to provide a realtime response and in use cases that only require
notification to be completed.
• Given the nature of processing “Coverage detection,” thealgorithm should be applied in use cases where thepriority is on the precision of notification as well as in use
cases where the collection of large-volume data, such asvideo, is linked to the notification. When applying thealgorithm to use cases prioritizing a real-time response,
the trade-off between speed and the effect on processing
time must be considered.

67

Evaluation of Network-Edge Computing
Platform© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

68

Evaluation Overview (1/4)
To realize an optimal flow of CAN and video data within the widely dispersed sites of the connected vehicle system, the architecture
between the vehicles and application groups was designed to include a multiple-function platform system, and we conducted anexperiment to verify the platform’s effectiveness at addressing the technical issues of the connected vehicle system. This platform will becalled "Network Edge" to distinguish it from application groups that are also arranged on the edge node.
Monitoring sites
Network Edge

Center site 1

Center site 2

Application groups

Application groups

Scout platform

Annotation platform

Scout platform

Annotation platform

Status management

Aggregation ofmonitoring
metrics

Visualize

Network Edge

Network Edge

Vehicle notification function

Site-to-site offloading

Session management

Site-to-site offloading
Network Edge statussharing

Vehicle notification function

Network Edge statussharing

Delivery management

Site-to-site proxy

Site-to-site proxy

Delivery management

Session management

Queuing function

Queuing function

BrokerBroker

Proxy

Proxy

Routing management function

Routing management function

TLS termination

TLS termination

Load balancing

Load balancing

BGP Anycast

BGP Anycast

Mobile network (EPC)© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Mobile network (EPC)69

Evaluation Overview (2/4)
To define the data flow from vehicles to application groups dispersed at multiple sites, we defined basic use cases as
those that involve switching application server in response to vehicle movement and to wide-area routing (switching
due to the localized concentration of loads).

Vehicle movement
We switched application server in response to vehicle movement. Tasks that require processing based on networkedge computing were defined as applied use cases related to the basic case of vehicle movement.

Application
serverEnterprise Network
pGW

pGW
Mobile Network (EPC)

Considers data traffic management whenswitching networks in response to vehicle
movement

Movement

Basic Use Case
Vehicle movement

Applied Use Case
Supports change in access from one edge platform to another to enable tracking moving vehicles
Supports fast response and manual switching while tracking rapidly moving vehicles© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

70

Evaluation Overview (3/4)
Wide-area routing
The application server is switched from one to another in response to triggers such as site failures and load
distribution. Tasks that require processing based on network edge computing were defined as applied use cases
related to the basic use case of wide-area routing.
FailureRising load

Application
serverEnterprise Network

pGW

pGW
Mobile network (EPC)

Consider data traffic management whenswitching networks in response to network
equipment failure or load concentration

Basic Use Case

Applied Use Case
Reduce data volume and execute preprocessing by using the edge platform to reduce the load on network
infrastructure

Wide-area routing

Limit redundant/similar data sent from vehicles when there is a localized concentration of vehicles to prevent suchdata from affecting the platform

Select the edge platform based on the equipment’s load status, processing time, and access quality
Switch some of the vehicles to another edge server when network congestion is detected
Equipment units will expand resources under high load through the manual and automatic scaling of resources© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

71

Evaluation Overview (4/4)
We established an environment with multiple network edge computing sites to realize the basic use case and verified the optimal method
for routing management between vehicles and each Network Edge site as well as the synchronization method between edge nodes. We
also evaluated Transport Layer Security (TLS) performance, assuming that communication with vehicles will be encrypted based on TLS toensure security.
Evaluation Points

Overview

1. Optimal routing
management for sending
data from vehicles tocenter site andapplication servers

We considered two methods for routing management and verified the system’s behavior in response to vehicle
movement and wide-area routing (equipment failure) as well as the basic functions and performance of each method.
• DNS method: Requests are guided by BGP Anycast to the nearest DNS server, which selects the optimal
application groups (annotation server platform) to notify vehicles.
• Load Balancer(LB) method: Requests are guided by BGP Anycast to the nearest LB server, which forwards therequest from vehicles to the optimal annotation server.

2. Metrics-based routing
management
3. Performance of TLS
termination (CPUprocessing)

We verified the following aspects of network edge computing by adopting the LB method for routing management.
• Combine the applications of DCM emulation, edge nodes, and data centers and confirm completion of processing
along with our scenario• Obtain basic performance values• Confirm the possibility of scale-out based on multiple units• Confirm behavior in case of failures in network edge computing and application server• Route control and load balancing that utilize the collected metrics at the application level in addition to status
information such as site failures.

Basic Use
Case
Vehiclemovement
Wide-arearoutingWide-arearouting—

4. Message Queue (MQ)based data flow betweenedge sites

We considered an MQ-based method for data aggregation between sites in an environment with multiple distributed
Network Edge sites and confirmed its behavior and performance.
• Confirmed behavior of communication between vehicles and application groups via wide-area distributed MQ(including site-to-site data synchronization)• Verified performance of data transmission from vehicles• Verified performance of message notification to vehicles

5. TLS offload

We verified the behavior of TLS offload based on two methods and the performance of each method for reducing
CPU usage.
• TLS Accelerator: Processing mode that uses the OpenSSL EVP application programming interface• SmartNIC (ASIC): Processing mode that uses Kernel TLS—

6. 5G field trials

We uploaded and downloaded large volumes of data between vehicles mounted with LTE/5G modems and theplatforms and measured the performance value.

—© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Vehiclemovement
Wide-arearouting72

Evaluation 1
We confirmed the behavior of routing management based on the DNS and LB methods and measured basic performance as follows.

Description

Conditions

Based on the assumptions of the basic use cases(vehicle movement, wide-area routing), we
confirmed the behavior of routing management
using DNS and LB methods, and comparedswitching time for varying parameters. We alsomeasured the basic performance based on DNS
and LB methods.

Vehicle movement: APN was switched by the vehicle client to confirm the behavior of routing management
via BGP Anycast.

Vehicle movement:(1) Checking the behavior of routing management
when using DNS or LB method(2) Effect of different parameters on switching timeWide-area routing:(1) Management operation for varying failure
patterns under DNS and LB methods- Total failure of application server- Partial failure of application server- Failure of control unit
(2) Effect of different parameters on switching time

Parameters
HTTP

1.1, 2.0

TLS
Sessioncontinuity
Data volume(KB/s)
TTL

With/Without

APN
Client-side
processing

With/Without
LTE module

Mobilenetwork

16,2560,60

APN

LTE connection
management

DNS/LB server

Application server

Routingmanagement

Annotation stub

DNS/LB server

Application server

Routingmanagement

Pseudoannotation
processing

Wide-area routing: Pseudo-equipment failures were created to confirm aggregated routing managementwith the status management database.
Failure scenarios
Pattern

Assumed behavior

Total failure
Route management byofswitching to an
Application
alternative site
serverPartial
Route management by
failure ofswitching to an
Application alternative Application
serverserver at the same site
Routing
Switching to an
managemen alternative site by BGPt failure
Anycast© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Loading device(vehicle client)

Loading device(vehicle client)
GW

Client-side
processing

Mobilenetwork
GW

DNS/LB server

Application server

Routingmanagement

Annotation stubStatusmanagement
database

Statusmanagement

DNS/LB server

Application server

Routingmanagement

Pseudoannotation
processing

Statusmanagement
database

Statusmanagement73Evaluation 1: Results
The evaluation results for each use case and basic DNS and LB methods performance are shown below.

Vehicle Movement(1) Checking the behavior of routing management when using DNS or LB method
We confirmed operations in line with assumptions based on both methods. Switching time was around two seconds for either method, and we found that thedifference in the routing management method had no effect on switching time.
(2) Effect of different parameters on switching time
We found that the difference in HTTP versions, TLS, session continuity, and data volume had no effect on switching time.
Response time was affected by the difference in parameters, and we confirmed that session continuity had a particularly significant effect.
Session continuity

Response time (TAT)With

100 ms–150 ms

Initialprocessing
With

Without

50 ms–400 ms

Without

DNS

TCP

TLS

HTTP

Disconnection Subsequent

processing
With

DNS

TCP

TLS

HTTP Disconnection

Without

Wide-area Routing(1) Management operation for varying failure patterns under DNS and LB methods
We confirmed operations in line with assumptions based on both methods. We found that the failure
detection threshold (polling interval and timeout value) had a particularly significant effect on switchingtime.
(2) Effect of different parameters on switching time
We found that the difference in HTTP versions, TLS, session continuity, and data volume had no effecton switching time.

Processing time

DNS

LB

Routing processing

10 s–12 s

4–5 s

Timeout value

10 s

4s

Basic Performance
DNS method• Stable operational limit was around 10,000 queries per second• We believe that the load on a specific CPU during UDP reception was thebottleneck• While it is possible to process around 50,000 queries per second when large
loads are received without interruption, we assumed it was not applicable inpractice© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATIONLB method• Stable operational limit was around 10,000 queries per second• We believe that the CPU load of TLS processing was the bottleneck• Scale-up and tuning method are future issues

74Evaluations 2 and 3
We evaluated the routing management function based on metrics data and the basic performance of edge sites.

Description
We confirmed the collection of metrics data during
server failures and behavior of routing management
based on LB method.

Verification of basic performance:
Confirmed the number of connections that can beprocessed by placing a connection load on a single
edge site. We used a loading device for thisverification.
Verification of routing management using metrics data:
Confirmed routing management operations andprocessing performance during the following failures
under the connected load state measured in thebasic performance verification.
(1) Edge node failure*
Confirmed distribution of processing to analternative node within the same site and verify
processing performance during distribution.
*Edge node failure includes the following three patterns.
• LB server failure (power outage)• Failure in load balancing function of the LB server• Failure of BGP connection processes on the LB server which
is used for BGP connection with Anycast router(2) Site failure
Confirmed distributed processing at multiple sites
due to the following failures.
- Overload of application server- Total site failure© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Conditions
Load criteria
Concurrent
connections
Data size
Protocol

200,000 TPS*
8 KB
HTTPS (POST)*Concurrent connections are defined as the maximum processing
performance at a single site. Based on the results of the basic
performance verification, concurrent connections for verifying routing
management using metrics data were set at 150,000 TPS.
Edge node failure Processing is maintained by normal nodes within the same site

Assumed distributed operation
during failures

Loading device(vehicle client)

Site failure

Processing is distributed to an alternative site

OverloadPart of the data traffic is distributed to an alternative site

Total failure

All data traffic is distributed to an alternative site

Site 1

Network Edge

Application server

Application server 1

LB server 1
Routing
Management

Client-side
processing

Annotation stub

LB server 2
RT

RTApplication server 2

Routing
Management

Annotation stub

LB server 3
Routing
Management

Site 2

Statusmanagement
database

Network Edge

Application server

Application server 1

LB server 1
Routing
Management

Annotation stub

LB server 2
RT

RT

Status management

Application server 2

Routing
Management

Annotation stub

LB server 3
Routing
Management

Statusmanagement
database

Status management75

Evaluations 2 and 3: Results
The results of the basic performance evaluation, node failure tests, and verification of routing management and load distribution at multiple
sites are shown below.

Basic Performance Evaluation• The processing performance of a single site was concurrent connections of 150,000 TPS. Although our initial projection was a load of 200,000 TPS,
around eight HTTP requests are being processed per TLS connection, which we believe is behind the reduced load.
• Average CPU usage was 25% and did not cause a bottleneck. Using the TLS session resumption, we conducted the verification during session start-up
by setting new TLS connections at a maximum of 20,000 TPS. Therefore, CPU usage may cause a bottleneck depending on the proportion of newconnections.

Routing Management Based on Metrics Data
(1) Edge node failure• We confirmed that processing was maintained in each of the node failure cases by
distributing to normal nodes, but we observed a phenomenon where processing was notpossible for several seconds immediately following a power outage in Network Edge
• Compared to an average CPU usage of 25% under normal conditions, it rose to 35% during
a failure of two units• In terms of hardware specifications, 5% of CPU usage was for realizing network receptiontransmission (Rx/Tx) of 1 Gbps
• We confirmed that the client resent data in response to a processing failure in the LB server
We must consider the possibility that this may increase the load on vehicle-mounted
devices

CPU usage during node failure
LB server 1

LB server 2

LB server 3

25%

25%

25%

Duration of failure
35%

35%

(2) Site failure• During an application server overload, we confirmed that some of the data traffic was
Response time for distributed processing during application server overload
allocated to an alternative site in line with our assumptions• This added a delay of approximately 10 ms to the processing response time for
Site１
Site 2transmissions between center sites• We confirmed that all processing was distributed to an alternative site in response to a site
failureDetecting failure
Detecting failure• We observed an increase in processing response time immediately after the switch© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

76

Evaluation 4
We conducted a trial run using test vehicles to confirm whether the set of scenarios, including the behavior of application groups such as
annotation processing and vehicle notification, function as planned. We also verified the performance of wide-area distributed Message
Queue (MQ) by uploading pseudo-data and placing a load on the client-side network connection.

Description
Evaluated operation and basic performance of datatransmission from vehicles and message notification
to vehicles using network edge computing thatincorporates wide-area distributed MQ.
Operation:
Confirmed that we can establish communication
between vehicles and application groups via the widearea distributed MQ in line with the scenario by using
test vehicles and the verification platform.
Basic performance:
Used a load testing device to measure processing
performance for data transmission from vehicles andmessage notification to vehicles as well as data arrival
times during site-to-site offloading.
(1) Data transmission from vehicles
a. Measured response performance and request
processing performance from vehicle sideb. Measure data arrival time from LB server toapplication groups during site-to-site offloading(2) Message notification to vehicles
Measured message arrival time from applicationgroups to vehicles© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Conditions
The evaluation was conducted by implementing Network Edge servers and application groups incorporatingwide-area distributed MQ in Tokyo and Osaka.
*Kudanshita area
Scenario for the operation evaluation:
was selected as
1. Processing by equipment at each site in Tokyo and Osaka
the trial route for2. Offloading of the data transmission and notification based on
the operation
assumptions of site failure and other incidents
evaluation
3. Offloading only the notification based on the assumption of vehicle
movement
Tokyo Network Edge
Load criteria for basic performance evaluation:(1)-aTotal number ofrequests
Number of clients

Network Edge Server

200,000
RT
200

Message size

128 B–1 MBProtocolHTTP

LB server 1
Routingmanagement

Application server 1

ProxyAnnotation stub

Broker
Application server 2
Site-to-site proxy 1

RT

Load testing device

Application server

MQ servers 1–3

Annotation stubAggregation
among sites(1)-b
Number of
Instantaneous requests
Message size

128 B–1 MBProtocolHTTP

1–2,000
Osaka Network Edge
Network Edge Server(2)
Number of
Instantaneous requests
Message size

RT

Application server 1

Aggregation
among sites

Annotation stub

1–1000128 B

RTApplication serverSite-to-site proxy 1

LB server 1

MQ servers 1–3

Routingmanagement

Proxy

Application server 2
Annotation stub

Broker77

Evaluation 4: Results
The results of the operation evaluation for wide-area distributed MQ and basic performance evaluation are shown below.

Operation Evaluation of Wide-Area Distributed MQ
No.

We verified that processing was completed normally in all evaluation
scenarios from 1 to 3 within the trial system comprising vehicles, theTokyo and Osaka edge sites and center site.

Pattern

Test result

1

Processing by equipment at each site in Tokyo and Osaka

Good

2

Offloading of the data transmission and notification based on the assumptions ofsite failure and other incidents

Good

3

Offloading only the notification based on the assumption of vehicle movement

Good

Basic Performance Evaluation(Req/sec)(1) Data transmission from vehicles(1)-a Measurement of response performance and request processing performancefrom the vehicle side
• Confirmed the possibility of processing 100,000 requests per second per MQ
server when the message size is up to 1 KB
• Observed no difference in results between HTTP/1.1 and HTTP/2• The wire rate limit became dominant once the message size exceeded 100 KB(1)-b Measurement of data arrival time from the LB server to application groups
during site-to-site offloading• Offloading resulted in a minimum delay of 50 ms as overhead• With the same total data transmission and the message size exceeds 100 KB,
any increase in the number of instantaneous requests was accompanied by amarked increase in average arrival time(e.g., Average arrival time is lower when sending the same number (1,000) of 100 KB requests as 1 MB
requests

Processing performance for each message size

Capable of processing around
100,000 requests per MQ server

Request processing performanceand publishing performance tothe MQ are dominant(msec)

Message size

Wire rate limit between theload device and LB server

Average arrival time for each message size
Rise in serializing/de-serializing
cost of the MQ with increase in message size

Exceeds site-to-site bandwidth limit (8Gbps)

Confirmed fixed cost of delays (about
50 ms) unrelated with criteria(2) Message notification to vehicles• We confirmed that even scaling up the number of simultaneously connectedclients in the logic queue to around 1,000 had a limited effect on the arrival timeof the notification message.
Total number of
instantaneous
requests© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

78

Evaluation 5
As described in the basic performance verification of Evaluation 1, CPU processing may become a bottleneck that lowers performance
when there is an increase in simultaneous connections to an edge site. We verified the enhancement of the processing performance basedon hardware offloading.

Description

Conditions

We evaluated the degree of enhancement in processing
performance for routing management that can berealized by introducing hardware offloading. We also
confirmed CPU usage.

Expected improvement in performance
We hoped to raise processing performance while
lowering CPU usage by adopting hardware offloading.
Processing
performance(TPS)

Reduce CPU usage
CPU usage (%)

Load criteria
Maximum number of simultaneousconnections per server
Data size
Application protocol
TLS

22,000 vehicles*
8 KB
HTTPS (POST)
TLS: v1.2; Cipher: ECDHE-RSA-AES128-SHA256*Set the load scenario for the loading device simulating vehicle-mounted client packets for 22,000 vehicles

Environment composition
Hardware offloading mechanism

OpenSSL version

LB server 1

None

OpenSSL 1.1.1f

LB server 2

TLS Accelerator installed

OpenSSL 1.1.1f

LB server 3

SmartNIC (ASIC) installed

OpenSSL 3.0.0-alpha13

Loading device(vehicle client)
Raise processing
performance

Network Edge

Application server

LB server 1

Application server 1
Routingmanagement

Time (Sec.)CPU processing: processing performance
CPU processing: CPU usage
Hardware offloading: processing performance
Hardware offloading: CPU usage© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

DCM
emulation
application

Annotation stub

LB server 2
RT

RT

TLS Accelerator

Routingmanagement

Application server 2
LB server 3
SmartNIC

Routingmanagement

Annotation stub79

Evaluation 5: Results
We measured the performance and CPU usage of each edge server. While servers 2 and 3 incorporating hardware offloading
demonstrated stable processing, the maximum performance was far below that of server 1 without hardware offloading. We believe wewere unable to confirm the expected performance enhancement because verification was conducted based on processing packets withsmall volumes of data that would not benefit from hardware offloading.

(TPS)

Confirmed maximum performanceby LB server 1 without hardware
offloading

30,000Server 3 based on a hardware
offloading (SmartNIC) excelled inaverage performance

25,00020,00015,000
LB server 1
LB server 2
LB server 3

10,000
5,000
0
0

2

4

6

8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84(Sec.)

LB server 1

LB server 2

LB server 3

Avg. CPU (%)

75.87 / 11.15*

19.79

77.57

Avg. transaction rate (TPS)

5,977.57

9,152.61

18,989.47

Max. transaction rate (TPS)

26,993

11,59921,182

With LB server 3 (offloading mechanism:
SmartNIC), verification was conducted
only for Tx Offloading. Rx Offloading is amodel that leaves processing completely
to the CPU, and we assume we canreduce CPU usage by using this model.

*CPU usage for LB server 1 was calculated separately as an average value for the earlier part of the measurement when it peaked
and for the latter part of measurement when it plateaued. Sections from 0 s to 8 s and beyond 74 s, where TPS is 0, have not been
included in the calculation.
© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

80

Evaluation 6
We measured performance values for uploading and downloading large volumes of data between vehicles mounted with LTE/5G modems
and the platform.

Description

Conditions

We established connections for communicating large volumes of data
between vehicles and the platform and upload/download large files andmeasured the processing time and throughput until HTTP POST/GET wascompleted for each transmission.

Wireless environment• 5G testbed network
 Experimental equipment for handling different environments andsettings than commercial models
 28 Ghz band 400 Mhz width

Platform
HTTP server(for sending/receiving files)

Large volumedata(upload)Commercial LTE
equipment
and 5G testbed
network

Large volume
data(download)• LTE communications network
 Commercial equipment with different execution speeds
depending on the place and time of use and communication
environment
Limitations• Size of large volume files: 128 MB to 2 GB
• Assume that files were uploaded/downloaded under circumstanceswhere CAN data is being regularly sent from vehicles• Vehicles moved within the 5G trial site at Odaiba, where verificationof LTE was also conducted• We could not verify the communication handover due to equipment
limitations

VehicleFile upload
application

File
download
application© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

81

Evaluation 6: Results
We connected the vehicles with the 5G testbed network and obtained basic performance values at regular points on the field. As a result,
we confirmed that using 5G broadband significantly shortened both upload and download times compared to LTE.

Band used when uploading 2 GB file

Performance result for sending 2 GB data*1
Upload5G testbed
network

Reference:commercial
LTE network

250 Mbps

Download
5G testbed
network

Reference:commercial
LTE network

67 sec.
10 Mbps
27 min., 38 sec.

Networkthroughput

Average value:
243.61 Mbps

Average value:
9.88 Mbps

Average value:
861.87 Mbps

Average value:
27.91 Mbps

Maximum
value:
262.25 Mbps

Maximum
value: 20.69
Mbps

Maximum
value: 914.93
Mbps

Maximum
value: 44.42
Mbps

Time (seconds)

Band used when uploading 2 GB file

900 Mbps

Processingtime

Average value:
67 s

Average value:
27 m, 38 s

Average value:
19 s

Average value:
9 m, 47 s

19 sec.

30 Mbps
9 min., 47
sec.
*1 Results are values obtained in an experimental environment, which is different than acommercial 5G network.
© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATIONTime (seconds)

82

Summary
We implemented an architecture between the vehicles and application groups that consists of an arrangement of Network Edges withmultiple functions (routing management, protocols, queueing, site-to-site offloading, vehicle notification management, and statusmanagement) and conducted trials to confirm its effectiveness. The results of each verification and future issues that came to light aresummarized below.
Evaluation Points
1. Optimal routing
management for sending
data from vehicles tocenter site and application
servers

Result Summary and Considerations

Future Issues

We evaluated routing management behavior during vehicle movement and wide-area routing for
DNS and LB methods and also assessed the basic performance of the two methods.
• We verified that routing management can be executed using either method without issues• There was no major difference in switching time between DNS method and LB method• We verified the possibility of CPU becoming the bottleneck that lowers performance

It became apparent that we needed to consider methods for
scaling up, scaling out, and tuning, so we addressed the issue
while evaluating “2. Metrics-based routing management.”

We evaluated the wide-area routing behavior and processing performance of edge sites for routing
management based on LB method in case of overload and site failure.
• Verified that processing performance was 150,000 TPS for a single site.
• In this verification we did not observe any performance caused by CPU bottleneck.
• Processing was maintained in each of the node failure cases by distributing to normal nodes.
• We observed cases where messages were resent at the time of the switch.
• During a site failure (overload, malfunctions), we verified that allocating processing to analternative site raised processing response.

Advanced metrics monitoring• In this evaluation, routing was managed based on themonitoring status and Round-Trip Time (RTT) of the target
servers. We must conduct tests in real environments to verify
basic technologies that will enable the use of AI to analyze
accumulated data for predicting congestion and providing
navigation when congestion is detected.

4. Message Queue (MQ)based data flow betweenedge sites

We evaluated behavior and processing performance for transmitting messages and notifications,
including aggregation among sites based on wide-area distributed MQ that responds to vehicle
movement.
• We observed site-to-site aggregation and message notification based on wide-area distributed MQ
in line with our hypothesis.
• When sending messages, we were able to process approximately 100,000 requests per server.
• We observed a delay (50 ms) in the arrival of messages to the application caused by aggregation.
• The effect on arrival time for message notification was limited.

Aggregation with priority management of application servers• We mainly evaluated routing management based on the usecase of vehicle movement and equipment failure. In the future,
we must consider aggregation with priority management
based on data processed by application servers to assess
advanced management methods that enable selecting highpriority queues for higher priority data.

5. Verification of TLS
offloading

We evaluated reduced CPU usage based on hardware offloading and operational performance.
• We verified that hardware offloading can lower CPU usage and raise performance but could notconfirm any advantages for ensuring maximum performance.
• We verified that SmartNIC demonstrated superior performance for offloading.

Rx Offload by SmartNIC and expansion in data size
• In this evaluation, there was only the Tx Offload function when
testing SmartNIC. In the future, we must address Rx Offload
and conduct additional evaluations. We will also need to verify
reception of large volume data (from around 500 KB to 1 MB)
assuming image and video data sent from vehicles.

6. 5G field verification

We conducted upload and download tests for large volume data via LTE and 5G networks andconfirmed that 5G is capable of handling extremely large volumes of data at a high speed.

ー

2. Metrics-based routing
management

3. Performance evaluation of
TLS termination (CPUprocessing)© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

83

Future Initiatives© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

84

Future Initiatives
Since December 2018, we have been conducting verification tests to evaluate feasibility of typical services
in the connected vehicle field and ICT platform technology.
As a result, we achieved the following two objectives, as well as our performance targets concerning thenumber of connected vehicles and real-time response.
At the time of the spread of connected vehicles and autonomous vehicles,
・ Evaluation of cloud platform technology/communications infrastructure technology
・ Requirements arrangement for next-generation vehicles
In preparation for further expansion of connected vehicles, we will work on followings and plan to continue
developing technologies that can solve various social issues facing our society, such as traffic accidents andcongestion.
・ Improvement of ICT platform for connected vehicles
・ Effective utilization of big data from connected vehicles
In the future, we will advance technological development in collaboration with various companies,
organizations and services in order to provide communications and computational resources beyond thelimits of current infrastructure.
In addition, we aim to realize a sustainable smart mobility society that brings safety and security in the
coming autonomous driving era.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION85

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Trademarks
The owners of the registered trademarks or trademarks related to company names and product names mentioned inthis document are as follows.

• AMD, the AMD Arrow logo, ATI and a combination of these, as well as Radeon are trademarks of Advanced Micro Devices, Inc.
• Amazon, Amazon Web Services, AWS, Amazon EC2 and Amazon S3 are trademarks of Amazon.com, Inc. and its affiliated companies in the UnitedStates and other countries.
• Apache, Tomcat, Apache Hadoop, Apache Ignite, Apache Zookeeper, Apache Kafka and Apache Spark are registered trademarks or trademarks ofApache Software Foundation in the United States and other countries.
• Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and other countries.
• Elasticsearch is the registered trademark or trademark of Elastic NV in the United States and other countries.
• Hadoop and Spark are registered trademarks or trademarks of Apache Software Foundation in the United States and other countries.
• Java is he registered trademark of Oracle Corporation, its subsidiaries and affiliated companies in the United States and other countries.
• Kubernetes is the trademark or registered trademark of The Linux Foundation in the United States and other countries.
• Linux is the registered trademark or trademark od Linus Torvalds in Japan and other countries.
• Microsoft, Windows, Windows Aero, Internet Explorer, Office, OneNote, Outlook, Excel, PowerPoint, Windows Live, Windows Media and the Windows
logo are trademarks or registered trademarks of U.S. Microsoft Corporation in the United States and other countries. Screen images are being usedin accordance with guidelines published by Microsoft Corporation.
• Oracle is the registered trademark of Oracle Corporation and its affiliated companies.
• PostgreSQL is the trademark or registered trademark of PostgreSQL in the United States and other countries.
• Redis is a trademark of Salvatore Sanfilippo.
• TensorFlow is the registered trademark or trademark of Google Inc. in the United States and other countries.
• Ubuntu is the registered trademark of U.K. Canonical Ltd.
• Intel, Intel Inside, the Intel Inside logo, Intel SpeedStep, Intel Core, Intel vPro, vPro Inside, Intel Atom, Celeron and Pentium are the trademarks orregistered trademarks of Intel Corporation and its subsidiaries in the United States and other countries.
Other company names and product names are the trademarks or registered trademarks of the respective company.
© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

87

Appendix© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

88

About ICT Technologies© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

89

About ICT Technologies (1/4)
Appendix 1: High-performance spatial-temporal data management technology and high-performance spatial-temporal query technology(1)(2)(3)(4)These technologies enable massive volumes of vehicle data to be stored instantaneously, allowing a high-performance query to be carriedout within certain parameters (time and location). Specifically, this spatial-temporal database manages time and positioning information
generated by moving objects including vehicles and smartphones as one-dimensional data using a unique method and allows querying fora vehicle within a given timeframe and location in addition to high-speed queries for roads, parking, and other complex-shaped spaces. A
high-performance query for sensor information contained in a given rectangular range defined by certain time and spatial information ismade possible by storing sensor data generated by massive volumes of dynamic objects in a real-world physical space along withassociated time and spatial information. Furthermore, this high-performance spatial-temporal query technology can be built upon to
manage more advanced spatial-temporal data, including queries for ranges that are more complex than simple rectangular shapes such as
roads. For example, this enables the high-speed extraction of vehicles and smartphones (i.e., people) in locations expressed by a polygon
shape such as a road, park, or school district.

Appendix 2: Data acquisition technology with target vehicle selection algorithm

This technology prioritizes the order of data aggregation based on meta-information such as position data, time information, sensor type,resolution, observation range, and situations in which vision may be blocked by surrounding vehicles. Data aggregated from connectedvehicles vary from vehicle data, video data, and image data to sensor data, all in massive volumes. These data collected from a series ofvehicles adds up to a tremendous volume. Increase in data traffic raises concerns over the heavy load on network and data processing.
This technology facilitates the efficient aggregation of data by minimizing the overlap of observation range as well as adjustments in theamount of data to be aggregated depending on the load condition of the telecommunications channel and analysis platform.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

90

About ICT Technologies (2/4)
Appendix 3: Congestion detection by lane
This technology analyzes footage taken by vehicle-mounted cameras and driving data such as vehicle speed and position information to detectvehicle congestion per lane. By inferring the section of the lane where traffic may be congested, vehicles at the front and the back end of thequeue are identified, and queue length is calculated from their positions. This technology can be applied to footage from monocular cameras
such as inexpensive drive recorders. Current traffic congestion information is provided on a road-by-road basis by the Vehicle Information andCommunication System (VICS) or a map app. If the technology is put to practical use or becomes widespread, drivers will be able to obtain more
precise congestion information. Furthermore, the cause of road congestion can be inferred by combining the footage of the area around thevehicle at the front of the queue and its position data with the position data of landscape features such as surrounding facilities and traffic lights.

Appendix 4: Vertical-distributed computing technology
This technology realizes high-speed processing of applications by distributing the processing functions formerly carried out at data centers toedge servers. Striving to have all devices immediately respond to inputs may from time to time hinder the realization of a high-speed response
as a result of insufficient server resources. In practice, however, the necessity of a high-speed response differs depending on the vehicle’s
overall situation (speed, direction, time period, etc.). This technology is intended to efficiently deploy limited server resources by dynamicallychanging the server that performs the response process depending on vehicle circumstances while dynamically distributing the app to ensurehigh-speed responses to vehicles that need them. This approach will help us more efficiently use the limited resource of distributed computing
capable of accommodating many vehicles and performing aggregation and responding to massive volumes of data.

Appendix 5: Suddenness Index Calculation Method
This technology creates an index at a high speed and based on the divergence of periodicity and/or suddenness of the value aggregated in thechronological log from its stationary level. It learns the average and distribution of aggregates such as the number of vehicles on different
temporal axes (all period, day of the week, time of the day, day of the week × time of the day) and creates an index that represents the level ofsuddenness of the newest aggregate. The suddenness index by time axis will be integrated and unified using a weighted linear combination andtaking into account the number of times it was aggregated. A sudden increase in the number of aggregated vehicles will be detected as apotential place for sudden traffic congestion. To optimize traffic flow, this technology should be applied before (3) Congestion detection by lane,
as communication cost cuts and server load reductions can be expected, by prioritizing the area for retrieving footage or by controlling thefrequency of retrieval.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

91

About ICT Technologies (3/4)
Appendix 6: Object-position-estimation technology(5)This technology estimates the position coordinates of landscape features along the road by triangulation using footage from inexpensive,commercially available monocular drive recorders and GPS. It improves the precision of position estimation by selecting quality data from thelarge volume of aggregated footage and data and by statistically and automatically selecting estimation results. Accurate recognition of verticallandscape features such as road signs, signboards, and traffic lights, and high-precision estimates of their positions, helps the plotting of
information onto maps. If the technology is put to practical use or becomes widespread, we can expect the creation of high-precision maps thatreflect the most recent road conditions for automated driving or a cost reduction in relation to updating information about landscape features.

Appendix 7: Video chunking transfer technology
This technology enables the transfer of footage taken by vehicle-mounted cameras in units equaling to or smaller than one second (at theextreme end, it can be transferred in a single-frame unit). Since the minimum length of video data transferred using a conventional
technology is one second, waiting times of one second or more is inevitable at the platform before it can start processing. Applying thistechnology can accelerate the process by the shortened waiting time. Additionally, note that CAN data linked to video data can begenerated by a unit of one second each and can be sent as it is generated by transmitting it in parallel.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

92

About ICT Technologies (4/4)
Bibliography(1) A. Isomura, “Real-time spatiotemporal data utilization for future mobility services,” RedisConf19, June 2019(2) RedisConf19 presentation(https://www.slideshare.net/RedisLabs/realtime-spatiotemporal-data-utilization-for-future-mobility-services-atsushi-isomura)(3) RedisConf19 video (https://www.youtube.com/watch?v=jVnIkwpcL3U)(4) NTT GIJUTU Journal (https://www.ntt.co.jp/journal/1911/JN20191118_h.html)(5) Aki Hayashi, Yuki Yokohata, Takahiro Hata, Kouhei Mori, and Kazuaki Obana, The Road Environment Measurement for Automatic Road
Map Generation, 2021, Volume 52, Issue 2, pp. 419–424, published in 2021, released on February 19, 2021

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

93

Trial Routes for Verification© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

94

Trial Routes for Verification
Routes were set in the Odaiba area for use case verification. The route shown in red below was set as the basic route,and the trial was extended to the routes shown by broken gray lines as necessary.

START© OpenStreetMap contributors© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

95

Platform Verification System© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

96

Architecture
Platform verification was performed for each platform of ① to ⑤ shown below.
Image data
stream

Queue

Image
datastore

Batch processing
for image data

CAN datastore

Micro-batchprocessing
for CAN data

Static map③

Area selection②

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Vehicle position
data

Area
notification

Delivery and notification

Dynamic
deployment ofapplications

Image data
stream④
Dynamic map

Edge

Reception of
message

CAN data
stream

CAN data processing

Queue

Image data processing

Data center

Reception, enrichment and annotation, queue①

Vehicle position
data

Network edge

⑤ Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

97Architecture (Detailed Version)
Category

Dynamic selection forprocessing nodes

Image data processing
Batch processing for image data
Generation of
Objectlanes-precision
positionstatic vector
estimation
maps
Objectrecognition/
detection

Dynamic deployment of applications
Monitorresources

Deployapplications

Image
datastore

Reception, enrichment and annotation, queue

DatacenterQueue forimage

Video frame
chunking

CAN
synchronization

Objectrecognition/
detection

Detection ofline of vehicles

Congestion
cause
inference

CAN
datastorelane
aggregation

Edge

Vehicleposition data
Notification on target
area
High performancespacial-temporalprocessing

CAN data stream
Raw data
conversion to
SI unit values

Video
reception

Staticannotation

Select vehicle

Storage/query

Metadata query for annotation

CAN reception

Select lane

Vehicleposition data

Storage/query

CAN
datastore

Objectrecognition/
detection

Queue forCAN

Reception of message

Target area selection

High performancespacial-temporalprocessing

CAN data stream

Image data stream
CAN
synchronization

Static map

Select edge

Image data stream

Queue forvideo

Datastore

Notification and delivery

Suddenness
index
calculation

Raw data
conversion to
SI unit values

Video frame
chunking

Queue

Mesh
aggregation

Queue for
CAN

Queue for
image

Function

Dynamic map

Micro-batch processing
for CAN data

Image data stream

Objectpositionestimation

Queue forvideo

CAN data processing

Platform

Dynamic
prioritization

Deviceshadow

Network Edge platform
Load balancer function

NetworkEdge

Loadbalancing

BGP Anycast

Site-to-site offloading

Queuing function
TLS
termination

BrokerNetwork Edge
status sharing

Proxy

Site-to-site
proxy

Vehicle notification function
Notification
statusmanagement

Sessionmanagement

4G/5G

Vehicles(DCM)

Data transfer
Send CAN
Send video
data
data

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception
Receive
Receiverequestnotification98

List of Software
List of software used in the platform verification.
No.

Name ofsoftwareDescription

Reception,enrichment andannotation, queue

CAN data
processing

Image dataprocessing

Dynamic map

Notification/delivery

Network edge
computing

1 CentOS

OS

○

○

○

○

○

ー

2 OpenJDKJava operating environment

○

○

○

○

○

ー

3 Netty

Framework wrapping Java NIO

○

ー

○

ー

○

ー

4 Apache Kafka

Middleware for processing distributed messaging

○

○

○

ー

ー

ー

5 NATS

Middleware for processing distributed messaging

ー

ー

ー

ー

ー

○

6 Apache HDFS

Distributed file system

ー

○

ー

ー

ー

ー

7 Apache YARN

Distributed resource management mechanism

ー

○

ー

ー

ー

ー

8 Apache Spark

Parallel distribution processing engine

ー

○

ー

ー

ー

ー

9 RedisDistributed in-memory KVS

ー

○

ー

ー

○

ー

10 Apache Ignite

Distributed in-memory database

ー

ー

ー

○

ー

ー

11 ElasticsearchDistributed query/analysis engine

ー

ー

ー

ー

○

ー

12 UltraMonkey

Software load balancer

○

ー

○

ー

ー

ー

Platform providing container-type virtualization
environment
Middleware for integrated management of containers
running on multiple nodes

ー

ー

○

ー

○

ー

ー

ー

○

ー

○

ー

TLS decoding middleware

○

ー

○

ー

○

ー

Relational Database Management System(RDBMS）
Extension to handle geospatial information using a
PostgreSQL database

ー

ー

ー

○

ー

ー

ー

ー

ー

○

ー

ー

18 Nginx

Used as a load balancer for web server software

ー

ー

ー

ー

ー

○

19 OpenSSL

SSL/TLS communication library

ー

ー

ー

ー

ー

○

20 FRRouting

Routing protocol management software

ー

ー

ー

ー

ー

○

13 Docker
14 Kubernetes
15 Hitch
16 PostgreSQL
17 PostGIS© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

99

Architecture
Reception, enrichment and annotation, and queue platform verification components are highlighted in blue.
Image data
streamQueue

CAN datastore

Micro-batchprocessing
for CAN data

Image data
streamDynamic
deployment ofapplications

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Area selection

Vehicle position
data

Area
notification

Delivery and notification

CAN data
streamStatic map

Dynamic map

Batch processing
for image data

CAN data processing

Reception of
message

Image data processing

Edge

Reception, enrichment and annotation, queue

Data center

Queue

Image
datastore

Vehicle position
data

Network edge

Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

100List of Functions
Functions of the reception, enrichment and annotation, and queue component

No.

Category

1

Reception of
message

Name of function

Description
Receives message by HTTP protocol

2

3

Metadata query
for annotation

Receive CAN and video data

Query fixed metadata forannotation, fixed/dynamic
method selection(1) Refer to the annotation master and retrieve the initial value of annotation
information(2) Refer to the data type of the message and process area detection if it is avideo message of an event(3) Return the retrieved annotation information if prioritization will not be
performed

Area detection

Prioritizes event video messages in the following sequence.
(1) Retrieves a list of vehicles present in the surrounding mesh(2) Narrows down the data by calculating the average moving speed, etc.
(3) Detects coverage(4) Detects the area threshold4

Output data queue

5

Queue for CAN, queue for
video

Sends messages to queue for CAN and queue for video based on annotation
information

Temporarily stores an annotated message
Queue© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

101

List of Hardware
List of hardware used in the reception, enrichment and annotation, and queue component.

No.

1

2

3

4

5

Name of server

Generator server

Reception, enrichment
and annotation server

Device shadow

Vehicle position data

Queue server

Name of function

Send CAN and video data

Receive CAN and video data, query
fixed metadata for annotation,
fixed/dynamic method selection, area
detection, and output data queue——

Queue for CAN, queue for video© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Number ofservers25

6Server specs
CPU (Xeon Gold 5118): 2 P, 24 C, 48 T
Memory: 192 GB
Disk (HDD): 1,200 GB
CPU (Xeon Gold 5118): 2 P, 24 C, 48 T
Memory: 192 GB
Disk (HDD): 1,200 GB

1

CPU (Xeon Gold 5118): 2 P, 24 C, 48 T
Memory: 192 GB
Disk (HDD): 1,200 GB

3

CPU (Xeon Gold 5118): 2 P, 24 C, 48 T
Memory: 192 GB
Disk (HDD): 1,200 GB

9

CPU (Xeon Gold 5118): 2 P, 24 C, 48 T
Memory: 288 GB
Disk (SSD): 1,600 GB

102

Architecture
Components used in the CAN data processing platform verification are highlighted in blue.
Image data
streamQueue

CAN datastore

Micro-batchprocessing
for CAN data

Image data
streamDynamic
deployment ofapplications

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Area selection

Vehicle position
data

Area
notification

Delivery and notification

CAN data
streamStatic map

Dynamic map

Batch processing
for image data

CAN data processing

Reception of
message

Image data processing

Edge

Reception, enrichment and annotation, queue

Data center

Queue

Image
datastore

Vehicle position
data

Network edge

Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

103

List of Functions
Functions of the CAN data processing component
No.

Category

Name of function

Description(1) Retrieves message stored in data queue
Raw data conversion to SI unit (2) Filters(3) Interprets binary data contained in the message and converts it intovaluesdata with high readability

1
CAN data stream
2

Vehicle position registration

Output data to the vehicle position data and device shadow after
performing raw data conversion to SI unit values

3

Output CAN datastore

Stores in the event store the data sent from CAN at the time of the event
and after performing raw data conversion to SI unit values

4

Mesh aggregation

Retrieves vehicle information in the designated mesh and calculates thenumber of vehicles

Suddenness index calculation

Determines if the number of vehicles calculated by mesh aggregation has
suddenly increased compared to the usual state within the mesh

Lane aggregation

Calculates the average moving speed of vehicles in the lanes within themesh by using a suddenness index calculation and picks up those thatwere below the threshold as a potentially congested lane5

Micro-batchprocessing for CAN
data (aggregation
process)

6

7

Queue

Queue for CAN, queue forpotential congestions© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Temporarily stores annotated CAN message and the results of congestion
detection

104

List of Hardware
List of hardware used in the CAN data processing component.
No.

Name of server

1

Queue server

2

CAN data stream server

3

Device shadow

4

Vehicle position data

5

CAN datastore

6

Micro-batch processing
for CAN data server

7

Suddenness index
calculation server

Name of function
Queue for CAN, queue for potential
congestions

Number of servers
8Server specs
CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 288 GB
Disk (SSD): 1,600 GB

Raw data conversion to SI unit values,
a: 5
vehicle position registration, output
b: 15
CAN datastore

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk: 1,200 GB—

1

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk (HDD): 1,200 GB

3

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk (HDD): 1,200 GB

1

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk: 1,200 GB

3

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk: 1,200 GB

1

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 288 GB
Disk: 1,600 GB——

Mesh aggregation, lane aggregation

Suddenness index calculation© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

105

Architecture
Components used in the verification of the image data processing platform are highlighted in blue.
Image data
streamQueue

CAN datastore

Micro-batchprocessing
for CAN data

Image data
streamDynamic
deployment ofapplications

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Area selection

Vehicle position
data

Area
notification

Delivery and notification

CAN data
streamStatic map

Dynamic map

Batch processing
for image data

CAN data processing

Reception of
message

Image data processing

Edge

Reception, enrichment and annotation, queue

Data center

Queue

Image
datastore

Vehicle position
data

Network edge

Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

106

List of Functions
Functions of image data processing component:
No.

Category

Name of function

1

Obstacle detection andnotification2

Obstacle inference

Description
(1)
(2)
(3)
(4)

Processes video data into chunks of image data
Retrieves CAN data linked to image data from the CAN datastore
Sends image data to obstacle inference containers and identifies objects that appear repeatedly
Stores obstacle information in a queue for an obstacle and forwards obstacle information to the notification process

Detects obstacles appearing in the image data3

Call for an object position
estimation and notification of
obstacle details(1) Retrieves CAN data linked to image data from the CAN datastore(2) Sends image and CAN data to an object-position-estimation container and receives the estimated object position
results(3) Forwards obstacle information to the notification process

4

Estimated object position

Performs an object position estimation and outputs the object label and position coordinates (lat/long)5

Detection of line of vehicles
in adjacent lanes andcongestion cause analysis(1) Checks the image data showing the picture of the vehicle to analyze the object detection method and circumstancesin regard to the footage taken and detects the occurrence of congestion in the adjacent lanes(2) Infers the cause of congestion based on information of features around the place where the occurrence ofcongestion was detected

6

Aggregation of monitoring
metrics, aggregation, andthreshold assessment(1) Aggregates the monitoring metrics information of the GPU node deployed to each site(2) Calculates the average GPU usage rate per unit of time(3) Assesses whether the average calculated for the aggregation of monitoring metrics exceeds the threshold

7

Deployment planning andsite selection(1) Preplans a deployment site when a bottleneck is detected within a site(2) When a bottleneck is detected within a site, decides on the final site for deployment based on the situation of theplanned deployment site and then proceeds with deployment

8

Recognition of traffic
light/road sign and estimatedobject position

Image data stream

Dynamic
deployment ofapplications

9
10

Batch processing
for image data

(1) Detects traffic lights and road signs appearing in image data
(2) Estimates the object position and output object label and position coordinates (lat/long)

Generation of a road network Synchronizes video and CAN data to DC in other sites and generates a road network map including lanes
map including lanes
Static map generation

Combines a road network including lanes and the estimated object position results and records them on a static map© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

107

List of Hardware
List of hardware used in the image data processing component.
No.

Name of server

Location

1

Queue server

Edge node

Queue for video

1

Edge node

Obstacle detection and notification,
obstacle inference2

Detection of line of vehicles in adjacentlanes and congestion cause analysis

22

Image data stream server(with GPU)
3

Data center

Name of function

Number ofservers

Server specs
CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 288 GB
Disk (SSD): 1,600 GB
GPU: Nvidia V100-PCle
CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 288 GB
Disk (SSD): 1,600 GB

Recognition of a traffic light/road sign andestimated object position, generation of a4road network including lanes, andgeneration of a static map

4Image data stream server（without GPU）

Data center

Call for an estimated object position andnotification of obstacle details, and an
estimated object position9

CPU (Xeon Silver 4110): 2P, 16C, 32T
Memory: 192 GB
Disk (HDD): 1,200 GB

5Image data stream LB
serverData center

Call for an estimated object position1

CPU (Xeon E5-2630v4): 2P, 20C, 40T
Memory: 128 GB
Disk (HDD): 300 GB

6

Container management
serverData center

Aggregation of monitoring metrics and athreshold assessment, and deployment
planning and site selection1

CPU (Xeon E5-2630v4): 2P, 20C, 40T
Memory: 128 GB
Disk (HDD): 300 GB

7

CAN datastore

Edge node anddata center—1 eachCPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk (HDD): 1,200 GB

8Image datastore

Data center

Mesh aggregation, lane aggregation

4

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 288 GB
Disk (SSD): 1,600 GB© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

108

Architecture
Components used in the notification and delivery platform verification are highlighted in blue.
Image data
streamQueue

CAN datastore

Micro-batchprocessing
for CAN data

Image data
streamDynamic
deployment ofapplications

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Area selection

Vehicle position
data

Area
notification

Delivery and notification

CAN data
streamStatic map

Dynamic map

Batch processing
for image data

CAN data processing

Reception of
message

Image data processing

Edge

Reception, enrichment and annotation, queue

Data center

Queue

Image
datastore

Vehicle position
data

Network edge

Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

109

List of Functions
Functions of notification and delivery process component:
No.

Category

Name of function

Description
Identifies the edge node for transferring the notification request.

1

Target area
selection

Query for target edge
server

Query for target
vehicle2
Notification on
target area
3

Request for
notification

Narrows down target vehicles to transmit image data send request. The following two
patterns of logic are assumed to be used for narrowing down.
(1) Query for target vehicle using only vehicle position data
(2) More precise query taking into account the specs and angle of the vehicle-mounted
cameraSends a list of vehicles in mesh identified through a query for a target vehicle tonotification process© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

110

List of Hardware
List of hardware used in the notification and delivery processing component.
No.

1

2

3

Name of server

Target area selection
serverNotification on the target
area server

Name of function

Query for the target edge server

Query for the target vehicle,
request for notification

Vehicle position data

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION—

Number ofservers

Server specs

1

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk (HDD): 1,200 GB

1

CPU (Xeon E5-2630 v4): 2P, 40C, 80T
Memory: 128 GB
Disk (HDD): 1 TB

3

CPU (Xeon Gold 5118): 2P, 24C, 48T
Memory: 192 GB
Disk (HDD): 1,200 GB

111

Architecture
Sections shown inside the broken blue line have been verified in the network edge platform verification.
Image data
streamQueue

CAN datastore

Micro-batchprocessing
for CAN data

Image data
streamDynamic
deployment ofapplications

Image data
stream

CAN data
stream

CAN datastore

Metadata query
for annotation

Device shadow

High-performance
spatial-temporalprocessing

High-performance
spatial-temporalprocessing

Area selection

Vehicle position
data

Area
notification

Delivery and notification

CAN data
streamStatic map

Dynamic map

Batch processing
for image data

CAN data processing

Reception of
message

Image data processing

Edge

Reception, enrichment and annotation, queue

Data center

Queue

Image
datastore

Vehicle position
data

Network edge

Network edge computing
Statusmanagement

Routingmanagement

Protocolconversion

Queuing

Notification
statusmanagement

Site-to-site
offloading

Vehicles(DCM)

4G and 5G networks
Data transfer© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Data reception

112

System Configuration
System configuration used for the network edge platform verification.
Center site
Estimated object
position
Obstacle database
Details notification

Insert latency emulation
between the Center siteand edge site 1

Center Servers

Edge site 2

Edge site 1
10 G dedicated line

Terminal routerfor mobile
network

BGP

BGP (Anycast)Servers for the edge processing app

Terminal routerfor mobile
network

BGP (Anycast)Dynamic annotation
Inference processing
Obstacle notification
Selective collection
and notification ofvehicle data

Load test device/load node

Dynamic annotation
Inference processing
Obstacle notification
Selective collection
and notification ofvehicle data

Servers for the edge processing app

LB/DNS statusmanagement
MQ
Site-to-site proxy

LB/DNS statusmanagement
MQ
Site-to-site proxy
Anycast router

Network Edge servers

Network Edge servers
pGW

pGW

APN Osaka

APN Tokyo

Mobile network (EPC)

Mobile connection

Vehicle-mounted client© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

113

List of Functions
The below table shows the functions of the Network Edge platform. In the verification, these functions were equippedmainly to an IA server, and scaling out was enabled by increasing the number of servers.
No.

Name of function
BGP Anycast

1

Routingmanagement

Load balancing
TLS termination

Description of function
Guides data from vehicles to the nearest Network Edge by propagating the IP address
where the message should be sent through BGP Anycast
Sorts messages and distributes load based on the established policy
Authentication and decrypts TLS traffic

2

Protocolconversion—

Converts the communication protocol (HTTP, MQTT, WebSocket, etc.)

3

Queuing—

Holds the received message over a certain period

Network Edge statusmanagement and sharing

Manages the status of other Network Edges and selects a site for sharing messages

4Site-to-site
offloading

5

Notification statusmanagement

6

Statusmanagement

Site-to-site proxy

Sends/receives a message between edge sites

Session management

Manages the connection status between the vehicle and edge site

Notification statusmanagement

Manages the notification status of the notification messages to vehicles

Monitoring metrics

Monitors metrics information of the Network Edge

Visualize

Visualizes the aggregated monitoring metrics information© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

114

Hardware and Software Configurations
Hardware and software configurations of the Network Edge server for verifications 1 through 4
Verification 1 DNS, LB, annotation server

Verifications 2 and 3 of the Network Edge, annotation server

Hypervisor specs

Hardware specs

CPUIntel Xeon E5-2650 2.00 GHz 8 C/16 T

CPUIntel Xeon Gold 6140 2.3 GHz 18C/36T ×2

RAM

64 GB

RAM

384 GBSAS

1.67 TB (300 G × 8 RAID50)

SSD

480 GB (480 G ×2 RAID1)Hypervisor

ESXi-6.5.20180502001-standard

Software specs

Virtual machine specs

OS

Ubuntu 18.04.3.0 LTS

CPU

8vCPUs

KernelLinux Kernel 4.15.0-135-generic

RAM

32 GB

Nginx(LB)

1.14.0

HDD

64 GB

OS

Ubuntu 18.04.1.0 LTS amd64

Verification４ of the Network Edge, annotation server

KernelLinux Kernel 4.15.0-45-generic

Hardware specs

Nginx (LB)

1.15.6

CPUIntel Xeon Gold 6140 2.3 GHz 18 C/36 T x2

dnsdist (DNS)

1.3.3 (Lua 5.1.4)

RAM

384 GB

Knot DNS (DNS)

2.6.9

SSD

480 GB (480 G ×2 RAID1)

Software specs© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATIONOS

Ubuntu 20.04.1 LTS

KernelLinux Kernel 5.4.0-65-generic

Nginx(LB)

1.19.3115

Hardware and Software Configurations
Hardware and software configurations of the Network Edge server for verification 5.
Verification 5 of all Network Edges
Hardware specs
CPUIntel Xeon Gold 6140 (2.3 GHz, 18 C) ×2

RAM

384 GB DDR4

SSD

480 GB (RAID1)

Software specs
OS

Ubuntu 20.04.1 LTS

nginx (LB)

1.18.0TLS (Cipher)v1.2 (ECDHE-RSA-AES128-SHA256)

Verification 5 of Network Edge 2 (TLS Accelerator, QuickAssist
Adapter)
Hardware specs
CPU w/ HT

Yes

NIC

Intel X710

QAT board

QuickAssist Adapter 8950 ×3

Software specs
KernelLinux Kernel 5.4.0-65-generic

OpenSSL1.1.1f

Verification 5 of Network Edge 1 (CPU)Verification 5 of Network Edge 3 (ASIC based SmartNIC)

Hardware specs

Hardware specs

CPU w/ HTYes

CPU w/ HTNoNICIntel X710 ×2

NICIntel X710

SmartNIC

Mellanox ConnectX-6 Dx

Software specs
KernelLinux Kernel 5.4.0-42-generic

Software specs

OpenSSL1.1.1fKernelLinux Kernel 5.4.0-65-generic (ktls enabled)

OpenSSL

3.0.0-alpha13© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

116

Evaluation 1: Evaluation Architecture
To realize for connecting the closest NW edge to the vehicle and the wide-area routing due to site failure or increased load, therouting management of the vehicle data traffic must be administered properly. This evaluation tested the basic functions andperformance of the DNS method as well as the LB method, which we designed as a routing management method. Eacharchitecture of the two methods is illustrated below.*

DNS method

LB method

Load information of the application server, network, etc., is kept in a statusmanagement database. Routing management is done by referring to
information in the database through the process at DNS, selecting the
optimum application server, and the DNS response.

Load equipment(client vehicle)
Connect

Load information of the application server, network, etc., is kept in a statusmanagement database. Routing management is done by referring to the
database when processing at LB, selecting the optimum application server,
and linking with a subsequent process.

Application server

Application server

Application server
疑似受信アノテーション処理
Application server
疑似受信アノテーション処理

Application server
疑似受信アノテーション処理
Application server
疑似受信アノテーション処理

Annotation processing

Load equipment(client vehicle)
LB and the application server
are connected in a closed
connection, communications
will not be encrypted

Annotation processing

DNS server
Processby client

DNS query andresponse

DNS queryandresponse

Site selection

LB server
Processby client

HTTP
processing

Connect
Selection based on the network
response time and information
about the application server load

Statusmanagement
database

TLS
termination

Site selection

Statusmanagement
database* Use of BGP Anycast is assumed for the selection of connecting edge site (DNS server, LB server). BGP Anycast selects a connecting site based on
the advertised route information between gateways. It is introduced to realize reconnection when pGW changes within the connecting mobile
network along with the vehicle’s motion.

© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

117

Evaluations 2 and 3: Evaluation Architecture
Routing management based on monitoring metrics information is required to realize wide-area routing that is responsive to thefailure or load status at each edge site. We test the routing management based on monitoring metrics information under LB
method in Evaluations 2 and 3. A basic performance evaluation of an edge site was also carried out. The architecture isillustrated below.
BGP route exchange
between Anycast RT and
LB server

Load equipment(client vehicle)

Process
distribution by
ECMP

Select connecting site by using
BGP Anycast at aggregated RT
Connect

Process by client

AggregatedRT

Anycast RT

Site 1
Network Edge

Application server

LB server 1
Application server 1

Routingmanagement

Annotation processing

LB server 2
Application server 2

Routingmanagement

Annotation processing

LB server 3
LB distributes a portion of the process
to other sites when overload is
detected in the entire site based onmonitoring the metrics information.

Routingmanagement

Statusmanagement
database

Status management

Constantly aggregates monitoring metrics
information. Selects a connecting Application
server based on the aggregated monitoring
metrics information.

Site 2
Network Edge

Application server

LB server 1
Application server 1

Routingmanagement

Annotation processing

AggregatedRT

Anycast RT

LB server 2
Application server 2

Routingmanagement

Annotation processing

LB server 3
Routingmanagement© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Statusmanagement
database

Status management

118

Evaluation 4: Evaluation Architecture
In an environment where a Network Edge is dispersed to multiple sites, synchronizing data between sites becomes importantbecause communication with vehicles must continue even when availability is lost due to failure at the Network Edge site. We
will address this by introducing a Message Queue (MQ) to a Network Edge to improve data synchronization and availability
across the entire Network Edge. This verification tested data synchronization between edge sites using wide-area-distributed
MQ. The verification architecture is illustrated below.

Data transmission from vehicles

Messages notification to vehicles

Off-loading processes among edge sites is necessary as application
processing may become overloaded when connections from vehicles areconcentrated to a certain site.
When overload is detected by load status monitoring of application processing,
only the necessary data is off-loaded through site-to-site proxy, to beprocessed at other sites.

Connected edge node may shift from one to the next along with thenotification target vehicle's motion. Therefore, notification status management
becomes necessary by synchronizing a notification message between edge
nodes. MQ logic that covers multiple sites is created to synchronize data
between sites. Metadata of a notification message is shared at each site, andthe message is retrieved from MQ logic and vehicles receive a notification
upon triggering events such as a connection request from vehicles.

Proxy

Application server

Client vehicle

Annotation
processing

Process byclientBrokerProxyBrokerProxyBrokerMonitorSite-to-site
proxy

Select edge
node

Network Edge
Routingmanagement

Process byclientNetwork Edge
Routingmanagement

Client vehicle

Proxy

BrokerProxyBrokerProxyBrokerNotification
platform

Annotation
processing

Site-to-site
proxy

MQ logic

Network Edge
Site-to-site
proxy

Select edge
node

Select the necessary process and offProxyload to other sites when application
processing overload is detected.
Proxy

BrokerMonitor

Proxy

BrokerBroker© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

Application server

Annotation
processing

Site-to-site
proxy

Client vehicle

Process byclientroutingmanagement

Network Edge

Proxy

BrokerProxyBrokerProxyBrokerNotification message from theconnected edge platform can bereceived even when the vehicle is inmotion, by synchronizing a notification
message using MQ logic.

119© 2021 TOYOTA MOTOR CORPORATION, NTT CORPORATION

120