論文 / 著書情報
Article / Book Information
題目(和文)

協調認知を実現するためのSDNに基づくV2Xプラットフォームに関する研究

Title(English)

SDN-Based V2X Platform for Cooperative Perception

著者(和文)

李宗典

Author(English)

Zongdian Li

出典(和文)

学位:博士(学術),学位授与機関:東京工業大学,
報告番号:甲第12588号,
授与年月日:2023年9月22日,
学位の種別:課程博士,
審査員:阪口 啓,廣川 二郎,岡田 健一,TRAN GIA KHANH,西尾 理志,原井 洋明

Citation(English)

Degree:Doctor (Academic),
Conferring organization: Tokyo Institute of Technology,
Report number:甲第12588号,
Conferred date:2023/9/22,
Degree Type:Course doctor,
Examiner:,,,,,

学位種別(和文)

博士論文

Type(English)

Doctoral Thesis

Powered by T2R2 (Tokyo Institute Research Repository)

Doctoral Dissertation

SDN-Based V2X Platform for
Cooperative Perception

Supervisor

Professor Kei Sakaguchi

Department of Electrical and Electronic Engineering
Graduate School of Engineering
Tokyo Institute of Technology

Zongdian Li

Contents
Acknowledgments

vii

Abstract

ix

Chapter 1

Introduction

1

1.1

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

1.2

Research objectives and related works . . . . . . . . . . . . . . . . . . . . . . .

6

1.3

Summary of contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12Chapter 2

Cooperative Perception and SDVN

17

2.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

2.2

Trade-offs in cooperative perception . . . . . . . . . . . . . . . . . . . . . . . .

18

2.3

Transition from SDN to SDVN . . . . . . . . . . . . . . . . . . . . . . . . . .

25

Chapter 3

Software Defined Dynamic mmWave V2X Network for Coopera-

tive Perception

33

3.1

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34

3.2

Architecture design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

3.3

Prototype system and indoor verification . . . . . . . . . . . . . . . . . . . . .

40

3.4

Outdoor proof of concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

48

3.5

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

57

Chapter 4

Het-SDVN: Software Defined Heterogeneous V2X Network for

Cooperative Perception

61

4.1

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

62

4.2

Design of Het-SDVN architecture . . . . . . . . . . . . . . . . . . . . . . . . .

63

ii
4.3

Local SDVN framework for RSU and evaluation . . . . . . . . . . . . . . . . .

68

4.4

Proof of concept of Het-SDVN . . . . . . . . . . . . . . . . . . . . . . . . . . .

78

4.5

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

90

Chapter 5

Orchestration of Mobility-Aware HD Map Distribution in Het-

SDVN

95

5.1

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

96

5.2

Design of mobility-aware map partition . . . . . . . . . . . . . . . . . . . . . .

97

5.3

Design of mmWave map distribution system . . . . . . . . . . . . . . . . . . . 100

5.4

Proof of concept and evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.5

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Chapter 6

Final Remarks and Future Work

113

6.1

Summary of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

6.2

Suggested future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Appendix I List of Publications

119

I.1

Journal papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

I.2

Journal papers not related to this thesis . . . . . . . . . . . . . . . . . . . . . 119

I.3

International conferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

I.4

International conferences not related to this thesis . . . . . . . . . . . . . . . . 120

I.5

Domestic conferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Appendix IIAbbreviations

123

References

127

iii

List of Figures
1.1

Taxonomy of driving automation and CDA [1] [2]. . . . . . . . . . . . . . . . .

2

1.2

Standardization progress of IEEE and 3GPP on V2X. . . . . . . . . . . . . . .

5

1.3

Current vehicular networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

1.4

Thesis organization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

2.1

ETSI CPM format [3]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

2.2

Example of object/track-level cooperative perception [4]. . . . . . . . . . . . .

20

2.3

Dynamic LiDAR points as the features for cooperative perception. . . . . . . .

21

2.4

RS-LiDAR-32 packet structure [5]. . . . . . . . . . . . . . . . . . . . . . . . .

22

2.5

Two fusion paradigms for cooperative perception. . . . . . . . . . . . . . . . .

23

2.6

Illustration of perspective transformation (v: vehicle velocity). . . . . . . . . .

24

2.7

Three-layer architecture of SDN. . . . . . . . . . . . . . . . . . . . . . . . . . .

26

3.1

Software-defined dynamic mmWave V2X network for cooperative perception. .

37

3.2

Creation of SDVN topology (LLDP: link layer discovery protocol, CAM: cooperative awareness message, CPM: collective perception message). . . . . . . . .

39

3.3

Visualization of the SDVN topology. . . . . . . . . . . . . . . . . . . . . . . .

39

3.4

Prototype details of indoor proof-of-concept. . . . . . . . . . . . . . . . . . . .

43

3.5

Scenario of cooperative perception over mmWave V2V. . . . . . . . . . . . . .

44

3.6

Scenario and demonstration for software-defined mmWave V2V. . . . . . . . .

46

3.7

Scenario of cooperative perception over mmWave V2I. . . . . . . . . . . . . . .

47

3.8

Scenario and demonstration for software-defined mmWave V2I. . . . . . . . . .

49

3.9

Hardware configuration for the outdoor prototype system. . . . . . . . . . . .

51

3.10 Photos of field trial. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

52

3.11 Interactions between C-plane and D-plane. . . . . . . . . . . . . . . . . . . . .

53

3.12 Point clouds of OBU from dynamic cooperative perception. . . . . . . . . . . .

56

iv
3.13 Network performance with mmWave V2X. . . . . . . . . . . . . . . . . . . . .
4.1

59

Overview of the Het-SDVN architecture (hierarchical C-plane and heterogeneous V2X D-plane). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63

4.2

SDVN-based framework for cooperative perception. . . . . . . . . . . . . . . .

69

4.3

DSRC message-triggered dynamic sensor ON/OFF. . . . . . . . . . . . . . . .

71

4.4

Sequence of RSU-centric cooperative perception. . . . . . . . . . . . . . . . . .

72

4.5

Experiment overview of Tokyo Tech smart mobility test field (three RSUs deployed, one Robocar used as CAV, cameras and LiDARs installed for contextual
awareness, Wi-Fi and WiGig antennas providing V2X connectivity, and Intel
NUCs empowering computing capability at RSUs; the test CAV drives along
the 360 m course (blue arrows) at a velocity of 10 km/h) [6]. . . . . . . . . . .

73

4.6

Total delay and the breakdown. . . . . . . . . . . . . . . . . . . . . . . . . . .

75

4.7

Power consumption under different sensor OFF timing. . . . . . . . . . . . . .

76

4.8

Cooperative perception under higher-layer orchestration. . . . . . . . . . . . .

77

4.9

Experimental topology for Het-SDVN demonstration. . . . . . . . . . . . . . .

79

4.10 Proof-of-concept environment. . . . . . . . . . . . . . . . . . . . . . . . . . . .

80

4.11 Sequence diagrams of Het-SDVN for CP. (a). Detected object-based CP. (b).
Raw LiDAR data-based CP. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83

4.12 RSSI maps of the SDVN WiFi C/D-planes. . . . . . . . . . . . . . . . . . . . .

85

4.13 Throughputs of Wi-Fi as local SDVN D-plane. . . . . . . . . . . . . . . . . . .

86

4.14 Variation of link data rates under SDVN controller orchestrations. . . . . . . .

88

4.15 Visualization of received CP data on Vehicle1 PC. (a). RSU1 and RSU2 send
detected objects. (b) RSU1 and RSU2 send raw LiDAR data. (c) Vehicle1
performs detection with raw LiDAR data from CP. . . . . . . . . . . . . . . .

93

5.1

An example HD map containing pointcloud data and semantic information. . .

97

5.2

Illustration of overlap optimization. . . . . . . . . . . . . . . . . . . . . . . . .

99

5.3

System overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.4

System implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.5

Proof-of-concept scenario and testbed setup (overlaid on Google Earth Image(Mar. 13, 2019), Tokyo Tech, Japan). . . . . . . . . . . . . . . . . . . . . . . . 106

5.6

Functionality verification through real driving tests (vehicle velocity: 20 km/h,
map size of Midorigaoka area: 267 MB). . . . . . . . . . . . . . . . . . . . . . 108

v
5.7

Evaluation results of mmWave map distribution system. . . . . . . . . . . . . . 111

6.1

Het-SDVN for constructing mobility digital twins. . . . . . . . . . . . . . . . . 116

6.2

Het-SDVN-based HD map distribution system. . . . . . . . . . . . . . . . . . . 117

6.3

Cooperative perception-assisted path planning system. . . . . . . . . . . . . . 118

vi

List of Tables
1.1

QoS requirements for advanced V2X application groups [7]. . . . . . . . . . . .

4

1.2

Performances of existing V2X technologies. . . . . . . . . . . . . . . . . . . . .

7

1.3

Related works of mmWave V2X and SDVN. . . . . . . . . . . . . . . . . . . .

10

3.1

Indoor Prototype Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . .

41

3.2

Outdoor Prototype Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . .

50

4.1

List of heterogeneous V2X technologies and characteristics. . . . . . . . . . . .

65

4.2

Proof-of-concept hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

81

4.3

Average data rate (raw LiDAR data, detected objects). . . . . . . . . . . . . .

84

4.4

Throughput and PDR of mmWave (WiGig) D-plane. . . . . . . . . . . . . . .

87

4.5

Latency performance of CP and SDVN controllers. . . . . . . . . . . . . . . .

89

5.1

System hardware. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

5.2

Overlap optimization results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

6.1

Comparisons with existing SDVN implementation studies. . . . . . . . . . . . 115

6.2

Comparisons with existing cooperative perception studies. . . . . . . . . . . . 115

vii

Acknowledgments
Since I came to Japan in 2018, five years have passed in the blink of an eye. Countlessjoys, passions, tears, and sweat have become indistinct in my memory. At the moment I
stop writing, all that remains in my heart is gratitude and a profound sense of appreciation.
I never imagined that I would become a doctoral candidate in engineering and the first
doctorate in my family. My parents surely feel proud. To accomplish all of this, the person
I am most grateful to is my supervisor, Prof. Sakaguchi. Without his selflessness, patience,and professional guidance, I would not have been able to successfully complete my doctoral
research. His dedication to work and unwavering passion for academia inspired and influenced
every aspiring young researcher, including myself. Prof. Sakaguchi provided me with all
the necessary resources for my growth in research, including opportunities for exchange at
RWTH Aachen University and collaborative research with top-tier companies like Denso. I
have greatly benefited from these experiences.
I also extend my sincere gratitude to Prof. Hirogawa, Prof. Okada, Associate Prof. Tran,
Associate Prof. Nishio, and Dr. Harai as the committee members of my defense. Throughout
the entire defense, they provided numerous professional and invaluable insights, helping me
refine the thesis and prompting profound reflection on the shortcomings that still linger within
my works. I have gained clarity on the direction I need to dedicate myself in my future career.
I am deeply grateful for the time and efforts they have devoted to me.
Furthermore, I appreciate Specially Associated Prof. Tao Yu for his commitment to fastand clear feedback on research problems I encounter. He treats every student as a kind brotherand provides us with countless support in campus life. I am also grateful to Emeritus Prof.
Araki for his valuable comments during my seminar presentations. I thank Ms. Minami and
Ms. Funabashi for creating a comfortable lab environment for our easy study and research. I
still remember the warm support from Associate Prof. Maruta during his stay at Sakaguchi
Lab. His contribution to the SSS and smart mobility field paves the way for our good research

viii
and publications. I’d like to express my sincere gratitude to him.
Lastly, I appreciate my parents, who give continuous support and understanding during
my study abroad. They taught me the wisdom of living and fostered me to be a qualified
adult. I thank the lab family, too. My outstanding seniors, Mr. Fukatsu, Ms. Yin, Project
Assistant Prof. Nakazato, etc., taught me knowledge and research skills selflessly and showedme the way to become a good researcher. My talented juniors always assisted my experiments
and created a relaxing atmosphere in the lab. I wish them a bright and enjoyable future.

ix

Abstract
Automated driving represents one of the most attractive innovations in this century. Yet,
we haven’t seen the roll-out of fully safe and automated vehicles. Vehicles on markets claiming
to have automated driving capabilities are inevitably entangled in various road accidents dueto the limited or impaired visibility of local sensors. Cooperative perception has the potential
to address such safety concerns by allowing automated vehicles to share perspectives obtained
from local sensors through vehicle-to-everything (V2X) communication and gather perceived
data from mounted sensors on roadside units (RSU). However, traditional V2X networks havedifficulty meeting strict communication requirements posed by this safety-critical application.
In this dissertation, a novel V2X network architecture is proposed to support cooperative
perception. The emerging network technology, software defined networking (SDN), is introduced and integrated with millimeter wave (mmWave) to establish a centralized mmWave V2X
network with ultra-high data rate and ultra-low latency performances. Three key SDN functions are developed to arrange multi-hop V2X for cooperative perception. Proof-of-concept
experiments have demonstrated the effectiveness of the proposed V2X network.
Next, in order to enhance the scalability of the previous network architecture, Het-SDVN,
an evolved network that consists of hierarchical control planes, geographically distributed but
logically centralized sub-SDVNs, and heterogeneous V2X, is introduced. The design of local
SDVN controllers in RSUs, the division and collaboration between local and global SDVN
controllers, and the utilization of heterogeneous V2X, are elaborated. The proposed HetSDVN has been set up in Ookayama Capmus, Tokyo Institute of Technology. Based on thisplatform, a proof of concept that demonstrate the effectiveness of Het-SDVN in supporting
CP has been carried out. The evaluation results are comprehensively discussed.
Finally, this dissertation presents the design, implementation, and field test of mobilityaware high-definition (HD) map distribution, which can be regarded as an extension of cooperative perception, based on the proposed centralized V2X network architecture.

x

1

Chapter 1
Introduction
1.1

Background

1.1.1

Position of V2X in ITS and automated driving

Intelligent transportation system (ITS) represents the grand vision of human society towards aroad revolution. Cutting-edge technologies are exploited to modernize transportation systems
in order to make commuting safer, more efficient, and more enjoyable. In this process, vehicleto-everything (V2X) communication plays a fundamental role. According to the International
Organization for Standardization (ISO) Technical Committees (TC)-204 [8], current efforts
in developing ITS focus on enriching road functionalities and raising vehicle automation levels. V2X communication can contribute to road functionalities by delivering three-category
V2X applications: (1) road safety; (2) traffic efficiency; (3) telematics and infotainment. The
road safety applications include the electronic emergency brake light (EEBL), forward collision warning (FCW), blind spot warning/lane change warning (BSW/LCW), etc [9]. With
pre-warning messages from these applications, human drivers can fine-tune their driving decisions to prevent potential accidents. The green light optimal speed advisory (GLOSA) andemergency vehicle warning (EVW) are two examples of traffic efficiency applications [10].
With real-time information about traffic status (volume, signal phase, construction, etc.),
these applications can suggest human drivers with high-level route planning and fine-grained
maneuvering, lowing down the probability of encountering traffic congestion and reducing
harmful gas emission. Telematics and infotainment applications can bring considerable economic values. Their roll-out speed is therefore leading compared to other V2X applications.

2
For example, the vehicle near-field payment (VNFP) has been extensively used for electronic
toll collection (ETC) in super markets, parking lots, petrol stations, etc., and the in-vehicle
infotainment (IVI) system has become an integral part of commercialized vehicles to provide
localization and navigation services, radio/video broadcasting, online games, etc.

Cooperative Driving
Automation (CDA)SAE J3216
Level 1

Perspective

Agreement-seeking

Intent-sharing

Status-sharing

Level 3

Level 5Class A Class B Class C Class D

Figure 1.1: Taxonomy of driving automation and CDA [1] [2].
In recent years, automated driving technology has experienced a significant leap in development, driving the prosperity of the automotive industry. The original equipment manufacturers (OEMs), such as Tesla, Toyota, Honda, etc., roll out their new products with Level 3
automation in succession. The levels of automation are referred to the Society of Automotive
Engineers (SAE) J3016 definitions [1], as shown in Fig. 1.1. From Level 3, automated systems
dominate driving behaviors while requiring minimum intervention from human drivers in an
emergency. So far, this level of automation has represented the highest level achievable solely
through artificial intelligence (AI) equipped by a single vehicle. To realize fully automateddriving (Level 4 - 5 automation), vehicles must overcome the well-known constraints of local
sensors by their range, field of view (FOV), and vulnerability to occlusions, bad weather,and lighting conditions to meet high safety and reliability expectations. In this sense, V2X
communication will be a key enabler for fully automated driving as it allows vehicles toexchange sensor information and be aware of road status, over multiple air interfaces, including vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), vehicle-to-pedestrian (V2P), and3vehicle-to-network (V2N) communications, when vehicles suffer catastrophic sensor degradation. With V2X capabilities, SAE J3216 has extended its definitions of automation levels intoanother dimension, called cooperative driving automation (CDA) [2], in which four classes of
CDA are defined depending on cooperation degrees and types of exchanged information:• Class A (status-sharing): Perception information about the traffic environment andinformation about the sensor (i.e., "Here I am, and here is what I see.")• Class B (intent-sharing): Information about planned future actions of the sender (i.e.,"This is what I plan to do.")• Class C (agreement-seeking): A sequence of collaborative messages among specific vehicles intended to influence local planning of specific driving actions (i.e., "Let’s do this
together.")• Class D (prescriptive): The direction of specific action(s) to specific traffic participants,
provided by a prescribing entity (e.g., emergency vehicles) and adhered to by a receiving
entity (i.e., "I will do as directed.")Figure 1.1. draws the contribution scope of Class A - D to driving automation.

1.1.2

Advanced V2X applications

Together with earlier mentioned V2X applications, the European Telecommunications Standards Institute (ETSI) Technical Report (TR) 102 638 [11] and the Third Generation Partnership Project (3GPP) TR 22.885 [12] have introduced a richer set of V2X applications,
including cooperative adaptive cruise control, curve speed warning, control loss warning,
etc. These applications based on the first-generation V2X technologies are designed to assist
human drivers, targeting Level 1 - 2 vehicle automation. With the rapid development of
automated driving technologies and the commercialization of Level 3 automotive products,
a novel set of V2X applications will be generated to serve both human and AI drivers. AI
drivers feature stronger processing capabilities than human drivers, which in other words, require more detailed information for analysis or larger amounts of sensing data as input. This
fact raises an increasing quality of service (QoS) requirement for those advanced (novel) V2X
applications, which can overwhelm the capabilities of the first-generation V2X technologies.
3GPP TR 22.886 describes some advanced V2X applications and summarizes them into
four groups: vehicle platooning, advanced driving, extended sensors, and remote driving [7].

4- Vehicle platooning: This allows vehicles to dynamically form a group, traveling in close
proximity. All the vehicles in the platoon receive periodic data from the leading vehicle
to keep a meter-level distance between them. The platooning enhances traffic efficiency
while reducing wind resistance and energy consumption.
- Advanced driving: This enables semi-automated or fully-automated driving. Each vehicle and/or RSU shares data obtained from its local sensors with vehicles in proximity,thus allowing vehicles to coordinate their trajectories or maneuvers. This results in safe
traveling, collision avoidance, and enhanced traffic efficiency.
- Extended sensors: This allows the exchange of raw or processed data gathered throughlocal sensors or live video data between vehicles, RSUs, pedestrian devices, and V2X application servers. The vehicles can enhance the perception of their environment beyond
the local sensor range and gain a more comprehensive view of the local situation.
- Remote driving: This enables a remote driver or a V2X application to operate a remotevehicle, catering to those passengers who cannot drive themselves or a remote vehicle
located in dangerous environments.

Table 1.1: QoS requirements for advanced V2X application groups [7].
Latency

Reliability

Data rate

Communication

(ms)

(%)

(Mbps)

Range (m)Vehicle platooning

10 - 2590 - 99.99

50 - 65

80 - 350

Advanced driving

3 - 100

99.99 - 99.999

10 - 53360 - 700

Extended sensors

3 - 100

90 - 99.999

10 - 1000

50 - 1000

Remote driving

20

99.999Group nameUL: 25
DL: 1

Ubiquitous

Due to the feature enhancements, the utilization of V2X messages with a higher payload,such as the maneuver coordination message (MCM) [13] and collective perception message(CPM) [14], and even the need for raw data transmission in extended sensors, a demanding
V2X requirement is observed from Table 1.1, which includes a maximum data rate of 1000
Mbps, a minimum end-to-end latency of 3 ms, and a reliability of 99.999%. This requirement

5
far surpasses that of enabling basic V2X applications with the small cooperative awarenessmessage (CAM) [15] or decentralized environmental notification message (DENM) [16].
Note that the definition of extended sensors in 3GPP aligns with the collective perception
concept in ETSI. In this thesis, they will be collectively referred to as cooperative perception.

1.1.3

Overview of standardized V2X technologies

Figure 1.2: Standardization progress of IEEE and 3GPP on V2X.
The Institute of Electrical and Electronics Engineers (IEEE) and 3GPP are two main
forces of V2X standardization. They have different road maps. Figure 1.2 draws the timelineof the standardization activities of IEEE and 3GPP for V2X. IEEE 802.11p and long-term
evolution (LTE)-V2X are the first-generation V2X technologies, which were specified in 2010
and 2016, respectively. IEEE 802.11p enhances the PHY and MAC layers for Wireless Accessin Vehicular Environments (WAVE) based on IEEE 802.11 series standards. At the physicallayer, IEEE 802.11p operates in the 5.9 GHz band with a 10 MHz channel width. Orthogonalfrequency division multiplexing (OFDM) with convolutional coding is employed. At the MAC
layer, IEEE 802.11p simplifies the communication procedure to enable low latency by defining
a new rule called "outside of the context of BSS (OCB)". To support prioritization and QoS

6differentiation, enhanced distributed channel access (EDCA) is adopted. IEEE 802.11p is thebasis of dedicated short-range communications (DSRC) in US, ITS-G5 in Europe, and ITSConnect in Japan. Their commercial deployment started in 2017, 2019, and 2015, respectively.
IEEE is now standardizing the next-generation V2X (NGV) technology, i.e., IEEE 802.11bd,
as a successor of IEEE 802.11p. It will leverage the newest Wi-Fi technologies and a mmWave
band from 57 GHz to 71 GHz, thus providing higher QoS support than IEEE 802.11p.
3GPP announced the LTE-V2X in Release 14. Besides the legacy cellular interface Uu,
a sidelink interface called PC5 was specified to support direct communications. LTE PC5
works in the 5.9 GHz band and supports two resource configuration modes inside/outside theeNodeB coverage. Release 15 mainly discussed the PC5 enhancements of LTE-V2X (64QAM,
carrier aggregation, MIMO, etc.). Shortly after the commercialization of 5G new radio (NR),
3GPP standardized the first version of NR-V2X in Release 16 in Jun. 2020. Although the 5G
NR utilizes two frequency bands (FR1: 410 MHz - 7.125 GHz; FR2: 24.25 GHz - 52.6 GHz),the main focus of NR-V2X in Release 16 and 17 is FR1. NR-V2X made extensive modifications to the channel structures, introduced advanced coding technologies (Polar/LDPC),
supported adaptive operations on sidelinks (HARQ), etc. It also enabled the cross-radio access technology (RAT) scheduling mechanism (LTE sidelink controlled by NR Uu; NR sidelink
controlled LTE Uu). Release 17 mainly studied the support to vulnerable road user (VRU)
scenarios, the energy-saving mechanisms as well as the enhancements to resource allocation.
Release 17 has been frozen in Jun. 2022, but future releases will continue to investigate theutilization of mmWave bands up to 100 GHz on top of the dedicated 5.9 GHz ITS band, as
well as sidelink communication capability enabled in these frequency bands as well [17].
From the standardization overview, it can be noticed that most standardized V2X choose
the sub-6 GHz band, but there is a clear trend that higher frequency bands will be utilized
for V2X. Before the deployment, plenty of technical obstacles need to be overcome, such as
the adverse propagation characteristics, limited coverage, and unpredictable reliability.

1.2

Research objectives and related works

This thesis presents the author’s efforts to enable cooperative perception, one of the most challenging (advanced) V2X applications described in Sect. 1.1.2, by introducing novel RATs andnetwork designs to enhance the performance of existing vehicular networks. The motivation
for studying cooperative perception can be concluded into three points:

7
1. Cooperative perception is a safety-critical V2X application containing great economic
and social values. It assists both human and AI drivers and contributes to all levels ofdriving automation (Class A of CDA). If this application can be installed on all commercialized vehicles and effectively supported by the vehicular network, road fatalities
will become zero and fully automated driving will come true.
2. Many beyond 5G and 6G applications rely on the data from cooperative perception.
For example, digital twin is a promising application that gathers sensor data from its
physical systems to build a high-fidelity cyber world. To build a mobility digital twin,
cooperative perception is necessary. In [18], the see-through application for drivers withaugmented reality (AR) glasses utilizes RSU sensor data from cooperative perception.
3. As shown in Table 1.1, cooperative perception requires the most stringent QoS (1000Mbps, 3 ms, 99.999%). If the vehicular network has the capability to support cooperative perception, other V2X applications will definitely be enabled.
However, existing V2X technologies and vehicular networks have considerable limitations.
Cooperative perception cannot be naturally supported, thus requiring new designs.

1.2.1

Limitations of existing V2X technologies

Table 1.2: Performances of existing V2X technologies.
Features

Frequencyband
Bandwidth
Supported
data rate
Range

IEEE 802.11p

Cellular V2X

DSRC

ITS-G5ITS Connect

LTE-V2X

NR-V2X

in US

in Europein Japan(Sidelink)(Sidelink)

5.9 GHz

5.9 GHz760 MHz

5.9 GHz

10 MHz10 MHz9 MHz

3 - 27 Mbps

3 - 27 Mbps

≤ 18 Mbps

1000 m1000 m

200 m

2.5 GHz,
5.9 GHz

10, 2010, 20,MHz30, 40 MHz

R14: ≤ 31.7 Mbps
R15: ≤ 73.5 Mbps
R14: 320 m
R15: 500 m≤ 400 Mbps
1000 m8
As stated in Sect. 1.1.3, the standardized and commercialized V2X technologies today
mainly operate in the sub-6 GHz band. Table 1.2 summarizes their performances. It can beobserved that the first-generation V2X adopts a small bandwidth between 9 - 20 MHz. The
IEEE 802.11p-based ITS Connect working in the 760 MHz band can only support a data rateof less than 18 Mbps. In the 5.9 GHz ITS band, IEEE 802.11p-based DSRC and ITS-G5 with
10 MHz bandwidth have the same supported data rate but limited to 27 Mbps. With a larger
bandwidth of 20 MHz, the LTE-V2X achieves a higher data rate of 31.7 Mbps in Release 14
and 73.5 Mbps in Release 15. However, LTE-V2X can only enable advanced V2X applications
at low automated driving levels. Even the NR-V2X has introduced a 40 MHz bandwidth and ahigher-order modulation scheme (256QAM) for the sidelink, at the condition of a sing carrier,the peak data rate cannot exceed 400 Mbps. This performance is sufficient for most advanced
V2X applications at high automated driving levels except cooperative perception.
The reason that the mmWave band (FR2) of NR-V2X is not mentioned here, is because
this frequency band has not been specified for the sidelink. Currently, the FR2 is used by
NR Uu. With the abundant spectrum resource, NR Uu has the potential to meet the data
rate requirement of cooperative perception (1 Gbps), but is unlikely to satisfy the latency
requirement (3 ms), since the data has to go through the core network of NR. The feasibilityof exploiting mmWave for direct communications needs to be further evaluated.

1.2.2

Limitations of current vehicular networks

There are two types of vehicular networks. One is the vehicular ad-hoc network (VANET), theother is the cellular V2X network, as shown in Fig. 1.3 (a) and (b), respectively. The VANET
is a fully distributed and self-roganized network. When the RAT is IEEE 802.11p, vehicles
can easily and reliably form a network via broadcasting. In order to support cooperative
perception in the VANET, mmWave has to be introduced. Constrained by the severe pathloss and weak penetration capability of mmWave, multi-hop routing will become a dominant
way to form the network. This poses great challenges to the VANET as the routing protocols
such as GeoNetworking create topology information also via broadcasting [19]. Because ofusing directional antennas, mmWave is much more sensitive to the topology change than IEEE
802.11p. The time for topology generation in the VANET cannot be ignored by mmWave,otherwise an unexpected packet loss will occur.
In contrast, the C-V2X introduces centralized nodes (i.e., base stations) to manage the network. Topology information collection and resource allocation can be done efficiently through9(a) Vehicular ad-hoc network (VANET)(b) Cellular V2X (C-V2X)

Figure 1.3: Current vehicular networks.

10
the control channel. This provide a chance to utilize mmWave. Nevertheless, since mmWave
has not been specified for the sidelink, and in the current NR-V2X architecture, the network
function (NF) for leveraging RATs from heterogeneous network (e.g., IEEE 802.11ad/WiGig
working in the unlicensed 60 GHz band) has not been explored, mmWave V2X with ultra-low
latency cannot be enabled for cooperative perception.

1.2.3

Related works

Table 1.3: Related works of mmWave V2X and SDVN.
TargetReferenceContributions & Limitations✓ Derive required data rates of cooperative perception[20,21]✓ Showcase the necessity of mmWave (30 GHz, 60 GHz)

mmWave✗ Only consider one V2V link (without multi-hop)

V2X✓ Design beamforming and blockage recovery schemes[22]✓ Construct a mmWave V2X testbed✗ Only improve single-hop mmWave V2I performance

[23]
[24]

SDVN
Solutions

[25]
[26]
[27]

SDVN
Testbeds

[28]
[29]
[30]

✓ Build a remote driving system with 22.6 GHz NR mmWave✗ Use a LOS link and not for cooperative perception✓ Mitigate interference in multi-hop mmWave V2V✗ Under a distributed network architecture (VANETs)✓ Enumerate benefits and open challenges✗ Theoretical works only
✓ Evaluate the performances of 5G SDVN
✗ Theoretical and simulation works only
✓ Investigate multi-hop routing with SDVN
✗ Theoretical and simulation works only
✓ Implement SDVN in Mininet-WiFi and test RAT handover✗ Based on an emulator and no mmWave✓ Implement and test wireless access management in SDVN
✗ Data plane adopts Wi-Fi (no mmWave)✓ Implement and test mobility management in SDVN
✗ Data plane adopts Wi-Fi (no mmWave)

11
The motivation of utilizing mmWave for cooperative perception has been thoroughly confirmed. The researchers calculated the required data rate of cooperative perception for safe
driving in specific traffic scenarios (overtaking [20], intersection crossing [21]). It showed thatmmWave can support safe driving at a higher velocity than V2X technologies at 5 GHz. As theinterests in mmWave increase, some researchers attempt to resolve the obstacles hindering its
practical use. For example, the researchers in [22] optimize the codebook to accelerate beamforming and leverage non-light-of-sight (NLOS) paths to combat blockage. However, these
PHY layer optimizations can only ensure the mmWave performance in one hop. Similarly, theresearchers in [23] tested the remote driving with mmWave complied with NR standards. The
QoS requirement was met but only one light-of-sight (LOS) link was employed. In mmWave
scenarios, especially for cooperative perception, multi-hop links should be imagined. The
researchers noticed that the inter-vehicle interference of mmWave in multi-hop cooperative
perception degraded the achievable throughput, so they proposed ZigZag antenna configuration and dynamic bandwidth division to mitigate this impact [24]. However, their proposed
schemes were based on a distributed network. The resource configuration status was notified
to vehicles through broadcasting. Hence, their system will suffer from the same limitations of
the VANET as explained above. So far, there is little research conducting practical evaluations
on mmWave for cooperative perception under multi-hop scenarios.

In fact, centralized architectures for vehicular networks have been investigated in recent
years. Software defined networking (SDN), due to its flexibility and capability to perform
global management, draws growing attention. Due to the cost of building a test network,
most studies stayed in theoretical and simulation levels [25,26,27]. The researchers in [28]
implemented a software defined vehicular network (SDVN) with an emulator called MininetWiFi but only tested the handover management between Wi-Fi and LTE. A few group of
researchers established prototypes to demonstrate the network capability. For instance, in [29],the group implemented and tested wireless access management in SDVN, and another groupfocused on mobility management [30]. However, their scenarios are less challenging as the
management target is low-frequency Wi-Fi. There is still a lack of evaluations on the SDVN
performance in managing mobility-sensitive mmWave V2X for cooperative perception.

Table 1.3 summarizes the above-mentioned related works as well as their respective contributions and limitations.

12

1.3

Summary of contributions

To enable advanced V2X applications, this thesis develops an SDN-based vehicular networkarchitecture to satisfy their stringent QoS requirements in data rate, latency, and reliability. This thesis targets cooperative perception as it is the most challenging application toexisting vehicular networks, and at the same time, it has significant value in realizing roadsafety and fully automated driving. Due to the high data rate requirement for exchanging raw
sensor data, mmWave has to be introduced despite the well-known obstacles. Therefore, thefirst contribution of this thesis is the design and development of three key control functions(context manager, access manager, and routing manager) in the SDN/SDVN controller toorchestrate dynamic mmWave V2X networks for cooperative perception. Compared to the
distributed management in VANETs, the centralized management by the SDN/SDVN controller can collect topology information in real-time and perform forward-looking operations,for example, selecting the beam index of mmWave access points (AP) based on the vehicle
position and direction. In this way, the whole mmWave communication procedures, from theassociation to the routing, can be timely and adaptively adjusted under fast vehicle mobility
and environment changes. The latency and reliability requirements of cooperative perception
relying on multi-hop mmWave V2X are thus promised. Instead of discussing these benefits
at theoretical and simulation levels, an outdoor deployment is carried out and demonstrates
the network performance in real-life proofs of concept.
Next, to resolve the remaining challenges in the previously proposed architecture, including the network scalability, control plane availability in infrastructure-less areas, and theaccommodation of heterogeneous V2X technologies, this thesis designs the evolved networkarchitecture Het-SDVN, to achieve more comprehensive management of V2X resources forcooperative perception. Hierarchical control planes in Het-SDVN address the scalability andavailability issues of fully centralized architectures. To deploy the local control plane to the
RSUs, an SDVN framework for RSUs is developed. This framework also introduces sensor
management which saves power and reduces deployment costs. In addition, the developed
local SDVN controller can select the optimal RAT and transmission mode to support cooperative perception at different data-sharing levels. Through the real-life proof of concept
with a full-fledged network and smart mobility testbed, the functionalities, mechanisms, andperformances of the designed Het-SDVN are comprehensively evaluated.
Ultimately, with the demonstrated Het-SDVN architecture, this thesis explores its support

13
for other safety-critical or mission-critical V2X applications. Specifically, a mmWave-based
mobility-aware high-definition (HD) map distribution application is developed, which benefits
from the three-layer (MEC/Cloud - RSU - vehicle) cooperation enabled by Het-SDVN. Thisapplication is also demonstrated through a proof of concept.
The following parts of this thesis can be summarized as follows.
• Chapter 2: Cooperative Perception and SDVN
1. Surveys the state of the art of cooperative perception and highlights the importanceof raw sensor data sharing.
2. Introduces general concepts and principles about SDN and numerates its benefits
for the vehicular network as SDVN.
• Chapter 3: Software Defined Dynamic mmWave V2X Network for Cooperative Perception
1. Designs the SDN-based vehicular network architecture to dynamically manage themmWave V2X for cooperative perception. The core control functions and the RATs
for each network plane are introduced.
2. Conducts indoor and outdoor proofs of concept based on the established testbeds.
The indoor tests verify the control of a single link (V2V and V2I), while the outdoor
field trial demonstrates the muti-hop connectivity.
– These works are published in∗ Z. Li, T. Yu, R. Fukatsu, G. K. Tran and K. Sakaguchi, "Proof-of-Conceptof a SDN Based mmWave V2X Network for Safe Automated Driving," 2019
IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI,
USA, pp. 1-6, 2019.
∗ Z. Li, T. Yu, R. Fukatsu, G. K. Tran and K. Sakaguchi, "Towards Safe
Automated Driving: Design of Software-Defined Dynamic MmWave V2X
Networks and PoC Implementation," in IEEE Open Journal of VehicularTechnology, vol. 2, pp. 78-93, 2021.
• Chapter 4: Het-SDVN: Software Defined Heterogeneous V2X Network for Cooperative
Perception

14
1. Introduces the enhancements to the network architecture with hierarchical control
planes and heterogeneous V2X to support cooperative perception.
2. Designs an SDVN framework for RSUs to manage their local networks. This framework is practically implemented and tested in a field trial with system evaluations.
3. Builds Het-SDVN and analyzes its performance when a test vehicle requests cooperative perception, which involves the collaboration between the local and global
SDVN control planes for network scheduling.
– These works are published in∗ Z. Li, T. Yu, T. Suzuki and K. Sakaguchi, "Building an SDVN Framework for RSU-Centric Cooperative Perception with Heterogeneous V2X,"
2023 IEEE 20th Consumer Communications & Networking Conference(CCNC), Las Vegas, NV, USA, pp. 1-7, 2023.
∗ Z. Li, K. Wang, T. Yu and K. Sakaguchi, "Het-SDVN: SDN-Based Radio
Resource Management of Heterogeneous V2X for Cooperative Perception,"in IEEE Access, vol. 11, pp. 76255-76268, 2023.
• Chapter 5: Orchestration of Mobility-Aware HD Map Distribution in Het-SDVN
1. Proposes a map partition strategy that reduces the size of overlap.
2. Implements the map distribution system and conducts field trials.
– These works are published in∗ Z. Li, M. L. R. Lagahit, M. Matsuoka and K. Sakaguchi, "Design ofMobility-Aware Map Partition and Distribution System for Smooth Automated Driving," 2022 IEEE 33rd Annual International Symposium on
Personal, Indoor and Mobile Radio Communications (PIMRC), Kyoto,
Japan, pp. 628-634, 2022.
Figure 1.4. shows the organization of this thesis.

15

Figure 1.4: Thesis organization.

16

17

Chapter 2
Cooperative Perception and SDVN
2.1

Introduction

Enabling cooperative perception is the research target of this thesis. Chapter 1 hides somedetails about this advanced V2X application. One decade ago, commercialized vehicles havenot equipped with as many sensors as they do today. The most commonly used vehicle sensor
at that time was the global navigation satellite system (GNSS) in order to provide navigation
service for human drivers. In this period, cooperative perception was at the stage of "share
where I am." because the vehicle can only read its position from the GNSS. In recent years,
the vehicle’s level of automation rapidly grows with an increased number of sensors including
cameras, light detection and rangings (LiDARs), radars, etc. Cooperative perception evolves
to the new stage of "share what I see." Nevertheless, this evolution occurs so fast that the
research, development, and standardization of V2X technologies are left behind. The transmission of raw images and raw pointcloud data among multiple vehicles remains a challenge
in the current vehicular network. Some may question the necessity of sharing unprocessed
sensor data, emphasizing the high bandwidth requirements. This chapter present the reasons why the exchange of raw sensor data is crucial in certain circumstances. Additionally,
there is a viewpoint that believes introducing high-frequency V2X such as mmWave can resolve all limitations of the existing vehicular network in performances. However, this claim
is also subject to scrutiny. In conjunction with the insights presented in Chapter 1, thischapter introduces the concept and principles of software-defined networking (SDN), an innovative communication architecture designed for the next-generation Internet. By comparing
its advantages to the limitations of the current vehicular network, this chapter elucidates the

18
motivation behind adopting SDN as the primary platform to support cooperative perception.
At the end of this chapter, the challenges that lie ahead for achieving the anticipated SDVN
are uncovered. These challenges serve as the focal points of this thesis work.

2.2

Trade-offs in cooperative perception

2.2.1

Classification and state of the art

According to the definition of 3GPP, cooperative perception (referred to as extended sensors[7]) is an advanced V2X application for sharing sensor data to enhance the environmental
perception of the vehicles. Based on the processing level of shared sensor data, a researchteam gives their insight on the classification of cooperative perception [31]:• Raw/low-level cooperative perception: This means the raw data from local sensors areshared. Typically, the pointcloud data from the light detection and ranging (LiDAR)
and the images from the cameras can be regarded as raw data.
• Feature/middle-level cooperative perception: This means the data after a series of preprocessing steps are shared. Redundant information has been removed, while the valuable information that can contribution to the final detection is maintained, such as the
3D profile of objects, the reflection intensity, the object mobility characteristics, etc.
• Object/track-level cooperative perception: This means the data have been fully processed before being shared to the vehicles over V2X. These data are highly abstracted,
only preserving the semantic information about the classification results of objects orevents, and the numerical information about the accurate/estimated positions.
Considering the diversity of sensors and the variation of products among manufacturers,
the data format for the object/track-level cooperative perception is the easiest to be standardized because of the high-level abstraction. The cooperative awareness message (CAM) [15]is the first-generation V2X message, specified by ETSI, to share such highly abstracted sensor data. The CAM information includes the vehicle position, velocity, and heading derived
from the global navigation satellite system (GNSS) and inertial measurement unit (IMU). In
road safety applications, this message is broadcasted by vehicles periodically to notify their
existence when locating at the blind spots of others (vehicles, pedestrians, cyclists, etc.).
The decentralized environmental notification message (DENM) [16] is defined to alert road19

Figure 2.1: ETSI CPM format [3].

users of a detected event (accidents, constructions, temporary traffic controls, etc.), insteadof detected objects, to improve safety and traffic efficiency. Therefore the broadcasting of
DENM is event-triggered and targets the vehicles that plan to pass through the corresponding regions. Now, the ESTI is drafting a new specification for sharing detected objects inthe collective perception message (CPM) [3]. Figure 2.1 shows the structure of CPM. One
CPM message can contain information about up to 128 perceived objects and the sensors
that captured these objects. Intensive discussions are still ongoing regarding the generation
method of CPM and the type of objects which should be included into the list. Before it isfinalized, the researchers have already tried to reuse the CAM as a proxy format to include
the detected objects [32]. Although using the proxy-CAM is inefficient as it only contains
one perceived object at one-time broadcasting and does not include sensor attributes, demonstrating such messages and systems is a critical step for the standardization of object/tracklevel cooperative perception. Figure 2.2 provides an example of the visualization result fromobject/track-level cooperative perception.
The feature/middle-level cooperative perception recently gained considerable popularity
from research attention. Compared to sharing object information from detection, sharing socalled features loses less amount of information from raw data and avoids generating excessive
traffic volume in the V2X network, which is suitable for the vehicles to achieve cooperative
perception in bandwidth-limited environments. Nevertheless, the definition of features is toodiverse to be unified. It can vary in the methods and depth of data processing. For example,
in [33], the researchers proposed to use feature maps in cooperative perception, which are
generated from LiDAR pointcloud data by the convolutional layers in a neural network. By

20

Figure 2.2: Example of object/track-level cooperative perception [4].

fusing these feature maps, they achieved around 10% improvement for detection within 20 m
and 30% for further distances. Another group of researchers introduced the spatial confidence
map as a feature [34]. The spatial confidence map was similar to the cost map usually used
for navigation. It indicated the score/probability of object detection in each map grid. They
used fewer communication resources and achieved higher perception performance by focusingon perceptually critical areas. Figure 2.3 gives our example of features, where the dynamic
LiDAR points (blue in Fig. 2.3(b)) are extracted from one scan frame by the octree filter [x]as the feature. The permanently static points of trees, curbs, buildings, etc., do not contributeto the detection on roads so that they can be discarded.
It is also hard to form a unified data format for the raw/low-level cooperative perception,
because various sensors can be used for detection, such as cameras, LiDARs, and mmWave
radars, but the generated raw data are totally different (images, pointcloud frames, etc.).
Even for the same type of sensors, due to the diversity in their specifications (for example,
the number of lasers), the raw data may have distinct structures. Therefore, the drivers aredesigned individually to make the data readable. Figure 2.4 shows the LiDAR packet format
of a 32-beam LiDAR (RS-LiDAR-32) manufactured by the company RoboSense. The packet
payload is 1248 bytes, which is a common size in commercialized LiDAR products. With this21(a) Raw LiDAR scan frame.

(b) Extracted dynamic LiDAR points.

Figure 2.3: Dynamic LiDAR points as the features for cooperative perception.
structure, the size of one LiDAR scan frame can be calculated by
N = 1248 × 8 ×

R × 60ω
32 × 4(2.1)

where R represents the size of the LiDAR point rate (pts/s) and ω indicates the rotation speedof the motor (rpm). Since the R in single return mode is 6 × 105 pts/s, and the maximum ω
is 1200 rpm, one LiDAR frame at least occupies 2.34 Mbit.

2.2.2

Fusion techniques

For cooperative perception, fusion is the last step before the received data contribute to thedetection and navigation of automated vehicles. With respect to the three types of cooperative
perception in Sect. 2.2.1, the fusion can also be categorized into (1) raw/low-level data fusion;(2) feature/middle-level data fusion; and (3) object/track-level data fusion.
The fusion of the first and second types needs to take a trade-off between the detection
accuracy, bandwidth limitations, and computational capabilities. To that end, the researchersin [35] proposed two fusion paradigms, as known as early fusion and late fusion, respectively.
Figure 2.5 illustrates the difference between these two paradigms. In the case of raw LiDAR
data, i.e., pointcloud frames, early fusion involves merging these frames as a whole and thenfeeding them into the detection model. On the other hand, late fusion entails performing
detection on each frame individually and subsequently merging the final results (exactly thesame as object-track-level fusion). Each paradigm offers its pros and cons. Early fusion has
the potential to produce the optimal results as it preserves all information prior to detection.

22

Figure 2.4: RS-LiDAR-32 packet structure [5].

However, it necessitates a strong computational capability that only edge servers can supply,as implemented in [36]. Otherwise, the detection latency will significantly increase. In contrast, late fusion mitigates the bandwidth requirement but compromises the accuracy. Given
the uncertainty in data quality and sensor positions, late fusion may lead to detection failuresor duplicate detections, thereby impacting vehicle operation and driving safety.
For all types of fusion, perspective transformation is the fundamental step to integrate
cooperative perception data to the vehicle point of view [37]. The positional informationfrom CAM, CPM, or other localization sensors will be useful to derive perspective (coordinate)
relationships. As shown in Fig. 2.6, when an object is detected at the position Pr = [x, y, z, 1]T′′′

by RSU in the RSU perspective (r). To translate it into a position Pv = [x , y , z , 1]T in thevehicle perspective (v), perspective transformation should be done by
Pv = Trv (t) ∗ Pr

(2.2)

23

(b) Late fusion.

(a) Early fusion.

Figure 2.5: Two fusion paradigms for cooperative perception.
where Trv (t) is the transformation matrix from the RSU perspective to the vehicle perspective.
Trv (t) is expressed by

RotX.x RotY.x RotZ.x T ranslation.x RotX.y
Trv = 
 RotX.z


0





RotY.y RotZ.y T ranslation.y RotY.z RotZ.z T ranslation.z 


(2.3)



0

0

1

which dynamically changes with the vehicle movement.

2.2.3

Philosophy of data type selection

The object/track-level cooperative perception shares fully processed sensor data. The main
benefits consist in three aspects. Firstly, it does not require high communication bandwidth as
the sensing information has been highly abstracted and can be stored in a few bits. So legacy
V2X technologies like IEEE 802.11p and LTE-V2X are capable of this application. Secondly,
most standardization efforts up to now concentrate in the format for fully processed sensor
data, like the CAM, DENM, and CPM mentioned before. This is convenient for third-party
application suppliers because they do not need to consider what sensors are available andused for the detection in the environment to prepare multiple drivers. Thirdly, the emergenceand commercialization of advanced computing technologies like cloud and multi-access edge

24

Figure 2.6: Illustration of perspective transformation (v: vehicle velocity).

computing (MEC) provide strong computing resources to deploy big models and process thesensor data with ultra-low latency. These resources will become cheaper and easier to accessas the penetration rate increase. However, sharing fully processed data can bring some risks
as the cooperative perception receiver has no idea about the sender’s detection capability andaccuracy. Purely believing in these detection results may cause wrong behaviors.
The feature/middle-level cooperative perception only involves preliminary abstractions on
the raw sensor data, but by removing invaluable information, it has significantly reduced
the data amount and relaxed the bandwidth requirement. On the other hand, the features
preserve more environmental details than fully processed sensor data, which may produce a
better accuracy. However, one serious shortcoming of this approach is complexity. It requires
strict consistency between the cooperative perception sender and receiver (e.g., the design ofthe feature encoder/decoder). In addition, the definitions of features are too diverse to begeneralized for all cooperative perception use cases.
The raw/low-level cooperative perception is underexplored. The major concern still liesin the bandwidth limitation. The existing V2X technologies cannot satisfy the QoS requirement. This issue will be addressed after introducing the mmWave. The opportunities fromsharing raw sensor data are more than its challenges. Firstly, integrating raw sensor data canachieve the best detection accuracy due to the richness of information. Secondly, sharing raw

25
sensor data has the potential to resolve a highly controversial topic related to connected andautomated vehicles (CAV), i.e., the liability in accidents. Imagine the cooperative perception
scenario where a CAV receives fully processed sensor data from the RSU deployed by the government or telecom operators. If this CAV made detection errors and ran into an accident,
it bears the highest liability. However, if the CAV ran into the accident due to believing in
the detection results of RSUs, the government or telecom operator should share the liability.
The shared raw data will be used as evidence. To prevent such accidents, the CAV can also
double-check the received objects by performing detection on the raw data by itself.
It is worth noting that, all types of cooperative perception could be required according
to specific circumstances. The V2X network should be adaptive to their QoS demands, thus
motivating a powerful management architecture and functions.

2.3

Transition from SDN to SDVN

2.3.1

Concepts and principles of SDN

2.3.1.1

History overview

The software defined networking (SDN) technologies originate from the "Clean State" research
program at Stanford University, aiming to redesign today’s communication infrastructure andbreak the network’s ossification. This program was directed by famous Prof. Nick McKeown.
In 2008, their team introduced the concept of OpenFlow [38]. OpenFlow is a protocol thatallows the separation of network control and forwarding functions. It provides a standardized
interface between the control plane (C-plane: decides how to handle network traffic) and thedata plane (D-plane: forwards traffic based on the instructions from the C-plane). Seeing the
programmability brought by OpenFlow, Prof. Nick McKeown and his team further proposed
the concept of SDN in 2019, using OpenFlow as a critical component so that network managers
can to remotely control the behavior of the switch/router through a separate controller. Since
then, SDN has been placed at the forefront of academia and industrial attentions. In 2011,
the Open Networking Foundation (ONF) was established. This foundation aims to promote
the development and standardization of SDN. Its core members include Google, Facebook,
NTT, etc. In 2012, the ONF released the SDN whitebook [39], which marked a milestonethat the three-layer SDN architecture had earned global recognition.
Today, SDN, together with the network functions virtualization (NFV) are expected to26transform the networking landscape of beyond 5G and 6G networks. The open, flexible,
and efficient architecture of SDN enables rapid innovation and low cost for communication
infrastructure deployment. Moreover, SDN has expanded beyond its initial scope of the
backbone network and initial focus on separating control and data planes. The capabilities
of SDN are largely enriched, encompassing new functions such as network automation, radio
resource management (RRM), network slicing, etc. The author of this thesis believe that
there are still a plenty of fields where SDN can contribute and make innovations.
2.3.1.2

Three-layer architecture of SDN

Figure 2.7: Three-layer architecture of SDN.

27
The most striking feature of SDN is the three-layer network architecture, as shown in
Fig. 2.7. It consists of a data plane (D-plane), a control plane (C-plane), and an application
plane from bottom to top. Each plane serves a specific function and has its own set of
responsibilities.

• D-plane: The D-plane is also referred to as the user plane, forwarding plane or infrastructure plane. The D-plane devices, such as the physical switches, physical routers,
and their virtualized counterparts, simply execute the forwarding actions as instructedby the C-plane. These devices are controllable and programmable by the SDN controller of the C-plane via the OpenFlow protocol. This plane takes care of the actual
transmission and switching of user/application data packets over the network.
• C-plane: The C-plane is responsible for making high-level decisions about traffic forwarding and resource utilization. It relies on the SDN controller to manage and control
the behavior of the network. On the one hand, the SDN controller listens to the instructions from the application plane through the northbound interface (NBI). On the other,hand, the SDN controller translates the network policies and requirements into lowlevel actions and injects to the D-plane devices through the southbound interface (SBI).
The OpenFlow protocol is an implementation of the SBIs. Through the OpenFlow, theC-plane can obtain a global view of the network to make adaptive policies.
• Application plane: The application plane consists of software applications and services
that interact with the network through the C-plane. The NBIs of SDN provides programmability and flexibility for the application plane to implement various network
services. For instance, the multi-media application deployed by the third party canacquire the QoS support from the C-plane without dealing with the complicated Dplane network. The application layer works as an interface for network managers anddevelopers to customize their own network policies and services.

The three-layer architecture for communication infrastructures will construct a logically
centralized network, but it does not hinder the appearance of geographically distributed Cplanes. When there are multiple peer C-planes, eastbound/westbound interfaces (EBI/WBI)
between the SDN controllers and their collaboration mechanisms should be well-defined.

28
2.3.1.3

Benefits of introducing SDN

Attempting to restructure the traditional network that has existed vast communication infrastructures and users is not a task that can be accomplished solely with courage. It requires
wisdom and the persuasive power of the new architecture. SDN offers irreplaceable cost andperformance advantages, whether for telecom operators or application providers, although it
may degrade the vendor’s profit. These advantages are summarized as follows:
1. Improved network performance: SDN enables intelligent traffic management and optimization. The centralized C-plane collects global network information including topology, link status, resources, etc., for sophisticated algorithms (including today’s prevalent
AI models) to analyze and generate the globally optimized strategies for traffic routing,load balancing, or resource slicing. Compared to the distributed and competing-based
networking, SDN provides an efficient and reliable network environment.
2. Increased flexibility and agility: SDN enables dynamic network reconfiguration adaptive to the changing requirements of applications. The convenient NBIs allow network
managers or software applications to reshape the network environment according totheir current needs. In addition, new services, policies, and functions can quickly function in the D-plane with the help of the centralized C-plane through the SBIs, avoiding
unnecessary time consuming in on-site hardware configurations.
3. Cost efficiency: SDN offers cost efficiency from two aspects. Firstly, the D-plane consistsof virtualized or generic forwarding devices. These devices remove proprietary C-planes
and only needs to support SBI protocols so their prices will drop. The cost of scaling
the network will become cheaper for telecom operators. Secondly, SDN increases theautomation of network management with centralized intelligence. The tasks for network
managers become fewer and easier so it can save human resources.
4. Enhanced security: A self-organized network is hidden with numerous security vulnerabilities in distributed devices. With the centralized C-plane of SDN, network managers can enforce security policies consistently across the network. They can implement
firewalls, monitor traffic anomaly in real-time, and quickly eliminate security threats.
Moreover, the applications can choose their preferred security policies through the Cplane, which enhances the overall network security.

29
5. Prosperous and innovative ecosystem: SDN provides an open and programmable platform with plentiful well-defined NBIs and SBIs to application developers and researchers
to deploy and test cutting-edge services, functions, and protocols. This will promote
rapid innovation, foster collaboration among academia, telecom operators, content suppliers, etc., and encourage the development of novel networking solutions.

2.3.2

Challenges towards SDVN

The concept of SDN was initially proposed for the next-generation Internet. Nowadays, it isalso conceived as one of the key technologies for 5G and beyond networks. This means thetelecommunication industry has noticed the great potential of SDN and planned to adapt it to
the wireless/mobile network, for radio access network (RAN) management, backhaul network
management, multi-access edge computing (MEC) resource management, etc. The softwaredefined wireless networks (SDWN) [40] or software-defined mobile networks (SDMN) [41] referto the wireless/mobile networks which follow the SDN principles in their architectures. Before
5G commercialization, the researchers in [42] constructed a software-defined mmWave meshbackhaul network to realize dynamic MEC container migration and content delivery based on
the mobile user’s trajectory. This provided a great example for 5G network designers to see
how they can leverage SDN to serve mobile users. The vehicular network is a special type of
wireless/mobile network that features higher mobility. However, the software-defined vehicular networks (SDVN) have not drawn extensive discussions from academia and industries.
From the author’s point of view, it is due to some realistic challenges.

2.3.2.1

Resource abstraction

In the vehicular network, the roles of vehicles/onboard units (OBU), roadside units (RSUs),
and base stations (eNodeB, gNodeB) are not only communication infrastructures but also
computing and storage infrastructures. Managing these heterogeneous communication, computing, and storage resources requires a high-level abstraction in the SDVN C-plane. So far,there have been well-defined abstractions for the computing resources in MEC and Cloud
owing to the fast development of NFV and NFV platforms such as Docker, Kubernetes, etc.
Nevertheless, there is still a lack of methods and platforms for communication resource abstractions. This hinders the possibility of doing fine-grained radio resource allocation. As theheterogeneity of V2X technologies increases after introducing mmWave, THz, VLC, etc., and30
the two sets of V2X communication standards (IEEE, 3GPP) develop in parallel, the SDVN
must implement a standardized or consistent abstraction layer for them. Only then can SDN
unleash its advantages of global resource management in the context of vehicular networks,
while providing the flexibility and programmability required for V2X applications.

2.3.2.2

Mobility management

As mentioned above, the vehicular network features higher mobility, which indicates a highly
dynamic node topology and frequent connection handovers. This poses a challenge to the
traditional SDN and low-mobility SDWN/SDMN. They have stationary or semi-stationary
topology, without the need to implement a real-time and accurate node tracking functionin the C-plane. For the SDVN, the vehicle’s position, velocity, direction, destination, etc.,
must be collected to the C-plane. Based on the information, the SDVN controller predicts
the future topology and adapts the network behaviors, such as routing, association, service
migration, etc., accordingly by issuing forward-looking policies or modifying OpenFlow tables.
After introducing mmWave V2X, the requirements for mobility management further increase
due to the use of directional antennas and the vulnerabilities to environmental changes. How
to stably and accurately collect and synchronize vehicle mobility information becomes critical.

2.3.2.3

Interoperability

Interoperability is a non-negligible issue in the vehicular network due to the heterogeneityof V2X technologies. Resolving this issue needs efforts from vendors, standardization organizations, and network operators. Due to the support of a different set of V2X technologies(IEEE, 3GPP), some vehicles may not be able to join the network to obtain the service or be
designated as a relay vehicle for multi-hop routing by the SDVN controller. To overcome thischallenge, it needs to specify a new protocol of SBIs for exchanging information about the
supported V2X technologies of vehicles, RSUs, and base stations between the SDVN D-plane
and C-plane. With the V2X capability information from the D-plane, the SDVN controller
can instruct the vehicles that share the same V2X technologies to form a network. On theother hand, the application plane can select the most appropriate QoS guarantee mechanism
from the C-plane for vehicles considering the V2X interfaces they have.

31
2.3.2.4

Lack of proofs of concept

Similar to the initial situation of SDN practice, although Stanford University allowed the useof the campus network for SDN technology trials, for large-scale network tests, the operators
became very cautious as there were a huge number of users, and any technical failure mightlead to unpredictable economic loss. Hence, SDN deployment in real carrier networks waslaunched in recent years after demonstrating technical maturity and reliability. The SDVN
components and test requirements are much more complex than that of SDN. Firstly, the experimental equipment is costly. Besides the communication infrastructures, the testbed needs
to prepare vehicles. Secondly, due to vehicle mobility, a wide test field is required. Thirdly,
safety issues are the primary concern in real traffic. Because of these reasons, most studies
about SDVN were performed by simulations. Some researchers have established simplified
testbeds [29,30], but such works are still insufficient. Proofs of concept are essential for researchers to develop SDVN frameworks compatible with commercial off-the-shelf equipment
and to verify the network performance and C-plane algorithms in practical traffic scenarios.
It is worth noting the contributions of this thesis fully/partially cover the challenges described above, which shed light on SDN adoption to enhance traditional vehicular networks.

32

33

Chapter 3
Software Defined Dynamic mmWave
V2X Network for Cooperative
Perception
Chapter 2 has delivered a comprehensive introduction to cooperative perception in termsof the categories, fusion techniques, and required data rates. The significance of sharing raw
sensor data has been highlighted, considering the benefits for high-level automated driving
in reducing latency, enhancing detection accuracy, improving maneuvering reliability, andclarifying liability. To fulfill the demanding communication requirements (a data rate of over
1 Gbps, and a latency of less than 10 ms), millimeter wave (mmWave) technology operatingin the 30 - 300 GHz band has to be utilized. Despite of mmWave’s potential to enable ultrahigh data rate and ultra-low latency communications, its severe path loss, weak penetration
rate, and antenna directivity hinder the establishment of a reliable, scalable, and efficient
mmWave vehicular network. Traditional broadcasting-dominated vehicle-to-everything (V2X)
networks should be transformed into multi-hop V2X networks. In this chapter, the author
decides to integrate the software defined networking (SDN) and mmWave technologies andproposes a novel architecture called software-defined dynamic mmWave V2X network. Thisarchitecture takes advantage of the centralization, flexibility, and programmability of SDN forresource management and the outstanding communication performance of mmWave for V2X.
Hence, raw sensor data sharing can be effectively supported. This chapter also presents the
prototyping process, in which the necessary hardware and software equipment are enumerated.
Finally, this chapter gives a detailed introduction to the proofs of concept (PoCs).

34

3.1

Motivation

The primary public concern regarding automated driving is safety, especially after the firstroad death caused by a testing automated vehicle in 2018 [43]. V2X communication is now
considered as a promising solution to addressing the safety limitation of the commercialized
automated vehicles in non-line-of-sight (NLOS) perception, by allowing sensor data sharing
among vehicles (vehicle-to-vehicle, V2V), with nearby RSUs (vehicle-to-infrastructure, V2I)
or smart devices (phones, watches, etc.) taken by road users (vehicle-to-pedestrian, V2P) [44].
On the other hand, technology evolutions of V2X accommodate the growing demands in data
rates (larger than assumed 1320% for machine-to-machine/internet of things (M2M/IoT) from2016 to 2021) and latency reduction [45]. The IEEE 802.11p based ITS-G5 [46]/dedicatedshort-range communication (DSRC) [47], operating in the licensed ITS 5.9 GHz band, canhardly exceed 6 Mbps with high mobility, although it ensures low latency less than 10 ms.
As an alternative, 3GPP-conducted cellular V2X (C-V2X) has concluded the design of longterm evolution V2X (LTE-V2X) in Release 14. It supports higher throughputs using LTE
radio interface (Uu) and side-link interface (PC5) [48]. However, the peak data rate is still
lower than 28.8 Mbps and the end-to-end latency cannot go below 100 ms [49]. In order tofulfill the stringent quality of service (QoS) requirement of safety-critical V2X applications,especially the target cooperative perception in this thesis, which requires a data rate of over
1 Gbps, end-to-end latency of less than 10 ms, communication range of larger than 50 m andreliability of above 99.99%, evolutions for DSRC and C-V2X, known as 802.11bd and new
radio (NR) V2X respectively, are underway, taking millimeter wave (mmWave) into accountbecause of its abundant spectrum resources to enhance communication performances [50].
The massive multi-hop connections due to the introduction of mmWave and the hugetraffic volume generated by raw sensor data sharing will definitely challenge the capability of
traditional vehicular ad hoc networks (VANETs) in their capacity, support to high mobility,
and resource management efficiency. In addition, the increasing penetration of automated vehicles and ITS infrastructures demands rapid protocol updates and application deployment.
Software defined networking (SDN) emerges as a revolutionary network paradigm that breaks
the inherent coupling of data plane (D-plane) and control plane (C-plane). This paradigm
surpasses the conventional one in terms of flexibility, programmability, global awareness andcentralized management. In order to diffuse its advantages into the vehicular networks, preliminary attempts have been made by some researchers in architecture design and performance

35
analysis. Nevertheless, most of the architecture works focus on the technical combination
with other emerging technologies like fog computing [51] and multi-access edge computing(MEC) [52] to satisfy the general QoS requirements of V2X, rather than orienting a specificsafety service like cooperative perception. Besides, nearly all the performance evaluations are
based on network simulators (ns-2, ns-3) and emulation tools (Mininet, Mininet-WiFi) [53],thus lacking field trials with real testbeds.
The contributions of this chapter lie in two fronts. Firstly, in order to support raw sensor
data-based cooperative perception and tackle the inevitable challenges using mmWave, theauthor proposes a software-defined dynamic mmWave V2X network, which not only achieves
real-time context tracking in fast-changing network topologies, but also enables flexible radio
resource management (RRM) for mmWave fronthaul/backhaul associations and transmissions. Secondly, the author conducts indoor verification and outdoor proof of concept that
demonstrate the proposed network can serve the cooperative perception with a good performance. The testbeds are constructed by commercial off-the-shelf (COTS) equipment, and thefunctionalities are implemented by open-source and developer-friendly software frameworks.

3.2

Architecture design

3.2.1

Safety requirements

The common idea to improve the safety of automated vehicles is attaching multiple andmulti-type sensors, e.g., cameras and light detection and rangings (LiDARs) on board, which
however exerts much pressure on the processing capability of a single vehicle. The sensor
detection performance is also highly constrained by frequent blockage and extreme weather.
To ensure the reliability of information acquisition, delivering alarm messages via V2X thenbecomes a new safety demand. Either DSRC or LTE-V2X can support this application foruse cases like pre-crash warning [12]. Nonetheless, alarm messages discard many environmental details due to the restricted message size. Complete safety is hard to meet with thelimited context information. In this regard, raw sensor data retains full environmental details, but meanwhile its dissemination desires higher performances of vehicular networks. The
requirements on three different network layers are summarized as follows:
1) High-speed V2X access layer: The high-definition (HD) dynamic maps composed by
sensor data are crucial for the ADSs to safely maneuver vehicles, in particular as a means

36
to provide decimeter localization that consumer-grade GPS cannot achieve. For a LiDARgenerated HD map, the total data volume collected for the duration of one hour is about 1
TB, which corresponds to a 2.2 Gbps data output [54]. The required data rate for cooperative
perception is analyzed based on an overtaking scenario, which reveals that over 1 Gbps LiDAR
information per is needed on V2V links to ensure safety for vehicle velocities above 70 km/h[55]. Hence, an estimation of 1 Gbps can be reasonably considered as the typical data raterequirement for exchanging HD information via V2X links.
2) Low-latency and economical backhaul layer: Vehicular backhaul refers to the associations from the source OBU to the destination OBU or RSU. The backhaul carries hugeamounts of user traffic and should guarantee the QoS correspondingly, of which the latency
is the most significant metric for safety-related services. For example, an end-to-end latencyof less than 20 ms is tolerable for remote driving [7]. But HD map exchanging requires amore strict latency of less than 10 ms [54]. Besides, since a considerable number of RSU are
being deployed to facilitate the penetration of automated vehicles, cost-efficient alternatives
for optical fibers are much preferred to reduce the budget of vast backhaul construction.
3) Ubiquitous and reliable control layer: Inadequate challenges are posed to the vehicular
network when transmitting alarm and other context information embedded in the cooperative awareness message (CAM) [15] and the decentralized environmental notification message(DENM) [16]. However, the dissemination of raw sensor data or HD map can cause traffic
congestion, load imbalance, etc that degrade the network performances. Therefore, a control
layer with wide coverage should be supplied to manage global network resources and coordinate multi-hop flow transmission. Moreover, it is necessary to monitor the real-time vehicle
status reliably in high dynamic contexts for safe automated driving.

3.2.2

Architecture description

Figure 3.1 gives an overview of the software-defined dynamic mmWave V2X network where
cooperative perception is performing at urban intersections. In a mixed traffic environment,
legacy vehicles and vehicles with OBUs coexist. For driving safety, alarm messages are broadcasted by OBUs and RSUs via LTE-V2X and DSRC. The contexts of legacy vehicles arecaptured by nearby RSUs which then send proxy notifications. As is mentioned above, HD
dynamic maps measured by versatile sensors (e.g., LiDAR) are critical for automated vehicles
to ensure safety. They can be basically divided into two types: the local dynamic map andthe global dynamic map. The former indicates the single HD map captured by self-sensors

37

Figure 3.1: Software-defined dynamic mmWave V2X network for cooperative perception.

connected with an OBU or a RSU. This sort of maps can provide the context information ofsurroundings, but it only covers LOS regions and is subject to the visual range of equipped
sensors. In other words, the local dynamic map only helps with local environmental perception, which is inadequate for 100% driving safety. The latter indicates the merged multi-source
HD map. Besides its own HD map, a certain automated vehicle can acquire more than one
local dynamic maps from vehicles ahead or fixed RSUs that monitor particular areas at highviews. The received maps is then merged into a global dynamic map that provides extra
information of NLOS regions and enables proactive risk awareness and reaction.
The proposed architecture is intended to ensure global HD dynamic maps available foreach OBU in demand. Cooperative perception of HD dynamic maps is running on the application plane. It utilizes the functions of SDN controller in the C-plane via the northboundinterface (NBI) to distribute local dynamic maps according to the context information (velocity, location, destination, etc) collected from the participants, i.e., OBUs and RSUs via thesouthbound interface (SBI). The role of three major elements in this architecture is describedin detail:• SDVN RSU: Existing infrastructures like street lamps or traffic lights can be easily
retrofitted into RSUs by mounting sensing, computing and communication modules. A RSU

38
can directly share its raw/processed HD map with vehicular OBUs via V2I communications
or simply forward map data via the backhaul network as relays. In the SDVN, RSUs belong
to the D-plane, handling network traffic according to the decisions made by the C-plane andperforming resource/mobility management (e.g., power allocation, channel/RAT handover,etc) under the instruction of SDVN controller.
• SDVN OBU: The OBU is commonly used to represent the automated vehicle but originally it refers to the platform of automated driving system, which takes raw sensing data
coming from various on-board sensors (LiDAR, camera, GPS, IMU, etc) as input, develops
dynamic path planning as intermediate results and finally maps to the appropriate action of
the vehicular engine system as output. For the compatibility of cooperative perception, V2X
module is added to receive external information. Multi-source data fusion is implemented before the path planning. In the SDVN, OBUs also belong to the D-plane and are controllable
by the SDVN controller in terms of network selection, data relay and handoff.
• SDVN controller: As the core of the proposed architecture, SDVN controller locates at
the C-plane and is responsible for tracking the network status including the dynamic topology
and the resource distribution. It also accepts control edicts from the applications and applies
them to the environment using specific protocols (OpenFlow, NetConf, SNMP, etc). In the
scenario of cooperative perception, SDVN controller implements corresponding NBIs and
SBIs for three key function components, i.e., context manager, access manager, and routing
manager. The context manager periodically requests the D-plane elements to upload theircontext information to the C-plane. Based on the context information, the access manager
associates OBUs with the most appropriate RSUs. And then the routing manager pushes
flows to promote the data sharing.
In contrast to the geographically static network topology owned by traditional SDN controllers, with periodically collected context information from vehicles and a static HD map,
the SDVN controller in the proposed architecture establishes a geographically dynamic network topology, which not only reflects the node and link status but also presents the obstacle
information (e.g., buildings, trees, non-connected vehicles) that is critical to control mmWave
V2X communications. The access and routing managers can utilize the topology information
to reduce the blocking probability of mmWave caused by unconnected obstacles appearing
between a certain V2X link. Vehicles and RSUs with sensors capture these unconnected obstacles and upload them as context information using collective perception message (CPM).
Figure 3.2 illustrates the topology creation. Figure 3.3 shows the visualization of the topology.

39

Dynamic SDVN topology
HD map

Buildings, trees, etc.

CAM and CPM packets(detected objects)

SDVN controller

LLDP packets

CAM and CPM packets(f = 10 – 20 Hz)

LLDP packets

Connected Vehicle

Connected Vehicle

Figure 3.2: Creation of SDVN topology (LLDP: link layer discovery protocol, CAM:
cooperative awareness message, CPM: collective perception message).

Stationary
objects

Figure 3.3: Visualization of the SDVN topology.

Mobileobjects

40
In order to meet the safety requirements from the aspect of vehicular network performances, mmWave as well as legacy V2X technologies are exploited:
1) MmWave V2X access layer: The broadband mmWave is the only means to support
cooperative perception of HD maps, because the conventional V2X technologies, i.e., DSRC
and LTE-V2X can not provide multi-gigabit data rates due to the limited bandwidth. In
the proposed architecture, the access of mmWave V2X is under the coordination of SDN
controller. With a global network view, the SDN controller can monitor the V2X status (e.glink stability) and help vehicles create dynamic V2X accordingly. Therefore, the negative
impacts of severe path loss, blockage and fast fading on mmWave V2X can be timely noticed
and minimized.
2) MmWave backhaul layer: The mmWave meshed network is a cost-efficient architecturethat can be adopted to compose the vehicular backhaul. It can make full use of the network
capacity enhanced by mmWave V2X. With SDN, the backhaul can be dynamically constructedin adaptation to the context information of the OBUs (e.g., position). Moreover, the QoS
requirements such as optimum latency or bandwidth can be flexibly met by implementing
various routing algorithms on the C-plane. Fast failover is another feature of such backhaul
networks, since the SDN controller monitors the global link status in real time and can quickly
handle any network failure.
3) LTE-V2X/DSRC control layer: Though the legacy V2X technologies cannot support thesafety information (e.g., HD map) exchange which requires gigabit data rates and millisecond
latency, they have demonstrated their adaptability and reliability of supporting a numberof V2X services (e.g., pre-crash warning) in highly mobile environments. Hence, they are
qualified to provide stable control signals for the SDVN C-plane. Since the LTE base station
has a high penetration rate both in urban and rural areas, it can provide extensive user
coverage. Meanwhile, free control resources will be usable as more and more DSRC modules
are deployed in smart cities.

3.3

Prototype system and indoor verification

3.3.1

Prototype system

The deployment of the software-defined dynamic mmWave V2X network is initiated by the indoor implementation based on a prototype system. Table 3.1 lists the hardware components.

41

Table 3.1: Indoor Prototype Hardware
Hardware Name

Specifications
Model: Hokuyo UST-10LX

2D LiDAR

Scan angle: 270◦ Resolution: 0.25◦
Rotation rate: 40 Hz Accuracy: ± 40 mm
Model: Velodyne Lidar VLP-16

3D LiDAR

Scan angle: 30◦ (vertical), 360◦ (horizontal)
Resolution: 2◦ (vertical), 0.1◦ -0.4◦ (horizontal)
Rotation rate: 5-20 Hz Accuracy: ± 30 cm
Model: Cisco Aironet 1850 Series

Wi-Fi AP

Standards: IEEE 802.11n, 802.11ac MIMO: 4×4
Antenna: 2.4 GHz/3 dBi, 5 GHz/5 dBi, omni
Model: Intel Lens Antenna

Wigig Antenna

Scan angle: ±4.5◦ (vertical), ±17◦ (horizontal)(Type A)

3 dB bandwidth: 7◦ (vertical), 3.5◦ (horizontal)
Gain: 25.97 dBi
Model: Intel Lens Antenna

Wigig Antenna

Scan angle: ±7◦ (vertical), ±13.5◦ (horizontal)(Type B)

3 dB bandwidth: 6.4◦ (vertical), 2.9◦ (horizontal)
Gain: 25.4 dBi
Model: Kobuki Turtlebot II

Mobile Robot

Maximum translational velocity: 70 cm/s
Payload: 5 kg (hard floor), 4 kg (carpet)

PC/Laptop

OS: Ubuntu 16.04
Wireless LAN: IEEE 802.11a/b/g/n/ac42
The OBU consists of one 2D/3D LiDAR, a pair of WiGig antennas and one control PC. And
it is built on the Kobuki Turtlebot II, a low-cost mobile robot for education and research.
The equipped 2D/3D LiDARs are used to create dynamic HD maps for automated driving.
And the WiGig antennas enable ultra-high rate mmWave V2X communications for map sharing. As the brain of the OBU, the control PC processes the input data, plans the navigationroute and interacts with the engine of the robot based on the ROS, an open-source robotics
software project that commonly utilized by application developers of automated driving. The
RSU is composed of one 3D LiDAR, a pair of WiGig antennas and one management PC. Oneof the WiGig antenna is used for mmWave access, the other is used for mmWave backhaul.
The management PC deploys application agents for C-plane registration and map merging,or simply forwards raw sensor data to the next hop through the OVS, an open-source implementation of a distributed virtual multilayer switch that supports OpenFlow protocols. The
SDN controller is running on a PC with the ODL framework, which is the most pervasive
open-source SDN controller that provides abundant native features and well-defined APIs
for network monitoring and traffic management. In addition, ODL supports flexible deployments of customized applications or protocols, which contributes to the extension of control
functions for the software-defined vehicular network.
Figure 3.4(a) and Fig. 3.4(b) present the architecture of the prototype system from theperspective of hardware configuration and software platform respectively. Since Wi-Fi is
widely used indoors, it can replace DRSC and LTE-V2X in the PoC. As seen in Fig. 3.4(a),wireless C-plane is constructed by a cluster of Wi-Fi APs that are deployed on the corridor
roof. Basically, the out-band control channel carries two types of messages. One is thecontext messages collected from D-plane elements (OBUs, RSUs) periodically. The otheris the control messages (access control, routing control) sent from the SDN controller once
cooperative perception is activated. WiGig antennas establish mmWave V2X links for Dplane, where big LiDAR traffic is transmitted through the organized path to the target OBU.
According to the specifications, on the one hand, 2D LiDARs have shorter detection range
but higher accuracy comparing with 3D LiDARs, so they are preferred to be used for theindoor PoC. On the other hand, 3D LiDARs generate more data volumes, which is suitable tohighlight the performances of mmWave V2X transmission. Therefore in this PoC, 2D LiDAR
maps will only be utilized for self-navigation while 3D LiDAR maps are shared to ensure
better safety.
Open-source software also plays an important role in this prototype system. The three43(a) Hardware configuration.

(b) Software framework.

Figure 3.4: Prototype details of indoor proof-of-concept.

44
representatives are ODL, ROS and OVS, as shown in Fig. 3.4(b). SDN control relies on the
ODL platform, in which both native and customized components are implemented. Native
components such as the metrics collector and OpenFlow plugins provide basic network information and configuration tools. Upon them, customized components, i.e., context manager,
access manager and routing manager in this case, achieve extensive functions for cooperative
perception in the mmWave V2X networks. Automated driving of OBUs depends on the ROS,
which provides hardware abstraction for robotics motion control by applications or algorithms,
and device drivers for 2D/3D LiDARs. Based on these fundamentals, simultaneous localization and mapping (SLAM) is implemented to create static and dynamic maps for automatednavigation. In addition, OVS enables the map sharing among D-plane elements. Throughthe OpenFlow SBI, OVS updates flow tables that decide the data forwarding according to
the dynamic routing decisions made by SDN controller.

Figure 3.5: Scenario of cooperative perception over mmWave V2V.

3.3.2

Cooperative perception over mmWave V2V

Although on-board sensors are replacing human drivers’ eyes, the vision of automated vehicles
can still be blocked by obstacles while driving. For example, when inter-vehicle distancebecomes too short, the front vehicle may decrease the LOS area of the behind vehicle, as
shown in Fig. 3.5. This will lead to potential traffic hazards if the behind vehicle tries toovertake without the notification of oncoming vehicles or passes through a busy intersection
without the information of both sides. Therefore, in order to acquire a boarder view for safe

45
driving, cooperative perception is necessary between the front vehicle and the behind vehicle
over mmWave V2V. Under the proposed architecture, SDN controller periodically collects
the context information of the two vehicles. Context information such as vehicle position,
received signal strength indicator (RSSI), and etc can be the reference for mmWave accessconfiguration. As soon as the V2V link is established, routing configuration messages are
distributed to both vehicles to direct the data flows from OBUs.
To demonstrate the performance of our prototype system in above-mentioned scenario,
an indoor PoC 1 is carried out. The test field is a long corridor over 50 m with a single
fork. Two OBUs are deployed to represent the vehicle front and the vehicle behind. Since
the corridor width is narrow, about 2 m, it is hard for the OBU to detect the region behindthe fork before the OBU comes very close. Therefore, the OBU front will take responsibility
to transmit its 3D LiDAR map which contains the information of this NLOS area for the
OBU behind. As Fig. 3.6(a) indicates, the Wi-Fi APs on the roof provide wireless C-plane.
SDN controller activates cooperative perception via mmWave V2V according to the vehicle’s
relative distance (e.g., 5 m), which can be determined by different safety requirements based1

https://www.youtube.com/watch?v=a5fd5oXv4qI(a) SDN control through Wi-Fi signal.

46(b) Cooperative perception via mmWave V2V.

(c) Enhanced driving safety.

Figure 3.6: Scenario and demonstration for software-defined mmWave V2V.

47
on the speed or the degree of visual obstruction. Fig. 3.6(b) shows the map transmission
process. The received 3D LiDAR map is presented at the top right of the figure. Apparently,
the perception of the OBU behind is greatly enhanced that it can touch the field beyond LOS,
as circled in Fig. 3.6(c), which highlights the benefit of mmWave V2V for large LiDAR datatransmission. Besides, with software-defined mmWave V2V communications, the timing forcooperative perception can be flexibly modified by applications depending on specific safety
requirements.

Figure 3.7: Scenario of cooperative perception over mmWave V2I.

3.3.3

Cooperative perception over mmWave V2I

Besides V2V communications, another sensible method to improve the driving safety is tocommunicate with RSUs, of which wider deployments can be envisioned in the future ITS.
To save costs, RSUs are probably installed on the existing road infrastructures like street
lamps and traffic lights. Meanwhile, their higher altitude than OBUs’ brings them bird’s eye
views for broader monitoring. This contributes to the early warning in blind areas when avehicle tries to pass the intersection with heavy traffic, as shown in Fig. 3.7. By cooperative perception over mmWave V2I, dynamic HD maps from the RSU can be exploited tocompensate for the loss of vehicle perception in NLOS regions. Under the proposed archi-

48
tecture, this process is also managed by SDN controller. Once a RSU is built, its locationand coverage will be record to the global topology. When vehicles move towards the RSU,
their real-time positions are collected and used for control. If the relative distance is within
the coverage, SDN controller starts mmWave access and routing configurations to perform
cooperative perception.
To demonstrate our prototype system also adapts to this scenario, an indoor PoC is implemented. The test field keeps the same as previous mmWave V2V demonstration, where afork of the corridor results in a NLOS region for upcoming vehicles. In this case, one OBU
is deployed on the Kobuki robot and one RSU is installed on a multilayer bracket. The
bracket is placed near the fork, but at the opposite side for global LiDAR monitoring. The
communication coverage of RSU is an importance parameter that should be predefined andregistered to the SDN controller. Although transmission throughput suffers from severe reflections in indoors, it remains higher than 1 Gbps up to 35 m. Moreover, transmission latency
fluctuates slightly within the whole distance of the corridor (approximately 2.5 ms/packet).
Considering the NLOS distance from the fork, the coverage value is set to 5 m. Fig. 3.8(a)shows the scenario when the OBU is moving out of the coverage. In the top right of the
figure, the merged LiDAR map only contains the static map (grey points) and the OBU map(red points), which means the cooperative perception is still off. Since the indicated NLOS
region is uncovered with dynamic point clouds, automated driving is under risks. Instead,Fig. 3.8(b) shows the scenario when the OBU moves into the coverage. At this moment,cooperative perception is initiated by SDN controller. In the top right of the figure, the RSUmap (green points) appears in the merged LiDAR map and its dynamic point clouds alreadycover the NLOS region before the OBU arrives at the fork. Hence, with software-definedmmWave V2I communications, the driving safety is greatly improved.

3.4

Outdoor proof of concept

3.4.1

Prototype hardware

The indoor prototype system has achieved the network functions for software-defined mmWave
V2X, and the indoor PoC has demonstrated its effectiveness for safety enhancement under
two typical modes of cooperative perception, i.e., mmWave V2V and mmWave V2I communications. In order to test the robustness and reliability of our network in real environments,

49(a) Driving without cooperative perception.

(b) Cooperative perception via mmWave V2I.

Figure 3.8: Scenario and demonstration for software-defined mmWave V2I.

50
Table 3.2: Outdoor Prototype Hardware
Hardware Name

Specifications
Model: NETGEAR X10 R9000

WiGig AP

Standards: IEEE 802.11ac, 802.11ad
Bands: 800 Mbps @ 2.4 GHz, 1733 Mbps @ 5 GHz,
4600 Mbps @ 60 GHz
WAN interface: WiMAX 2+ @ 2.5 GHz

WiMAX Card

Speed: Uplink ⩽ 30 Mbps, Downlink ⩽ 440 Mbps
Size: 111×62×13.3 mm
Model: Acer TravelMate Series

PC/Laptop

OS: Ubuntu 16.04 USB ports: 4
WLAN interface: 802.11ac, 802.11ad, Bluetooth

VehicleModel1 : Toyota Estima (grey) w OBU
Model2 : Toyota Ractis (blue) w/o OBU

an outdoor PoC 2 will proceed by replacing or adding some hardware for the new outdoor
prototype system. Table 3.2 lists the additional hardware. In this case, the OBU is directly
deployed to a real vehicle rather than the mobile robot. A 3D LiDAR with the same type
as indoors is fixed on the top of the vehicle. The model of the control PC is substituted byAcer, which builds in 802.11ad chipsets. It comes with the 32-element phased antenna array.
Therefore, it’s not necessary for the OBU to equip another WiGig antenna as indoors so thevolume can be more compact and the power consumption can be less. As for the RSU, one of
its antennas that serves for mmWave access is replaced by NETGEAR X10 R9000, which uses
802.11ad module and can switch to the AP mode. Saha et al. completed a review of the connection performances between Acer and NETGEAR X10 R9000 using WiGig, showing thatthe maximum transmission range in outdoors is about 20 m when the average throughput isabove 1 Gbps, and the Gbps communication is possible for client angles in [−60◦ , 75◦ ] or AP
angles in [−45◦ , 30◦ ] [56]. Since the outdoor field is too wide to be covered by Wi-Fi signals,
WiMAX cards are utilized to provide wireless C-plane at 2.5 GHz, performing the same roles2

https://www.youtube.com/watch?v=di1GUaA5Q4I

51

Figure 3.9: Hardware configuration for the outdoor prototype system.
of DSRC and LTE-V2X as the system design. The supported speeds for uplink and downlink
transmission are adequate because the context messages and control messages won’t occupy
so much bandwidth.
Figure 3.9 gives an overview of the hardware configuration for the outdoor prototype
system, which includes one OBU, two RSUs and one SDN control server. Since the demonstrations of indoor prototype system doesn’t take mmWave backhaul capacity into account,the new prototype system will incorporate one mmWave backhaul link between the RSUs
for dynamic map routing. Therefore, the network performances of mmWave V2X access,mmWave backhaul and SDN management can be all confirmed in the outdoors. Note thateach D-plane and C-plane element is equipped with a WiMAX card for stable interconnections. And the only OBU appears twice in Fig. 3.9 just in order to highlight the association
handover with RSU1 and RSU2. Considering the limited perception range of 2D LiDARs,
they will not be used in the outdoor PoC.

3.4.2

Scenario description

The test field is selected at a L-shaped bend. In addition to the test vehicle with OBU,
non-test vehicles in the lane are also passing normally. The driving route of the OBU has
been marked on the photo of Fig. 3.10. To set up a dangerous scene artificially, a blue

52

Figure 3.10: Photos of field trial.

vehicle without OBU is hidden as an oncoming vehicle at the other end of the L-shaped bend,
which belongs to the NLOS region of the OBU. In Fig. 3.10, the locations of the two RSUs
are indicated by red points. They are deployed at each side of the bend, so their LiDAR
views can fully cover the dynamic environments of the whole area. Because the WiMAX
base station has very wide coverage, the SDN controller can be placed even miles away. In
order to test cooperative perception through the software-defined dynamic mmWave V2X
network, along the driving route, network functions will be implemented by four steps. ⃝1
Leave the start point: When the test vehicle begins to move from the start point, the OBU
is still out of the communication coverage of any RSU, which is set to 20 m in outdoors.
At the moment, SDN controller will not activate cooperative perception but continuously
collects and judges the context information (i.e., position) from the OBU via WiMAX Cplane. ⃝2 Associate with RSU1: When the test vehicle approaches RSU1. As soon as itenters the communication coverage, SDN controller will send configuration messages for V2X
access and backhaul routing, by which the local dynamic maps from RSU1 and RSU2 aresimultaneously forwarded to the OBU via mmWave D-plane. Then the OBU can create a
global dynamic map using map merging techniques. ⃝3 Handover to RSU2: After the test

53
vehicle passes RSU1, for the OBU, its dynamic map will not benefit the driving safety any
more. When the OBU enters the coverage of RSU2, SDN controller will send configuration
messages for mmWave V2X handover. In this case, only the dynamic map of RSU2 will be
forwarded to the OBU without utilizing the mmWave backhaul. ⃝4 Pass the bend: With
cooperative perception, the test vehicle is expected to drive through the bend safely. After it
leaves the coverage of RSU2. SDN controller will deactivate cooperative perception.
As a matter of fact, it is the interactions of C-plane and D-plane that achieves the SDN
control in above mentioned scenario. The details are described in Fig. 3.11. The process starts
from the registration of RSUs as soon as they are deployed. The registration information
includes ID numbers, locations, AP parameters and etc. This information is integrated on
the C-plane and used to generate a backhaul network topology for V2X. From the SLAM,
the OBU subscribes the context information (vehicle position) and uploads to SDN controller
almost in real time. When the trigger condition is reached, i.e., within the communication
coverage, SDN controller will activate cooperative perception by firstly distributing new flowtables to all D-plane elements for map data routing. Secondly, mmWave V2X access will beestablished or handover under the coordination of SDN controller. Note that the MEH whichindicates the mobile edge host plays a role of map merging and can be located both in RSU
and OBU.

Figure 3.11: Interactions between C-plane and D-plane.

54

3.4.3

Results of field trials

The graphical user interface (GUI) of OBU presents the received and merged LiDAR point
clouds from dynamic cooperative perception, which are important resources for SLAM-basednavigation. In Fig. 3.12, the three sub-figures correspond to the three critical steps of safe
automated driving mentioned in the last subsection. Fig. 3.12(a) shows the dynamic HD
map of OBU when it starts to move. In the map, the white point clouds form the static map
and will be updated in a long period. Besides, the green point clouds from the OBU LiDAR
compose the dynamic vehicle map. Except the static map and the vehicle map, no RSU maps
appear in the GUI since cooperative perception is not activated yet, i.e., the OBU has notreached the communication coverage of RSU1. In this case, the perception range of OBU is
too limited to detect the potential risk at the other side of the bend.
Figure 3.12(b) and Fig. 3.12(c) show the dynamic HD maps of OBU when mmWave
V2X association and handover are implemented by dynamic cooperative perception. In Fig.
3.12(b), RSU1 map (red point clouds) and RSU2 map (yellow point clouds) both appear.
This is in accordance with the mechanism, that all local dynamic maps of RSUs will be
forwarded to the OBU through the mmWave backhaul and V2X links under SDN control.
The merged global dynamic map greatly extends the perception range of OBU as it can detect
an oncoming vehicle in the NLOS region (56.86 m away, 5.12 s in advance when vehicle velocity
equals 40 km/h), and then take earlier measures (e.g., emergency brake) to avoid the collision.
If there is no terrain restriction, this perception performance can be further enhanced. The
deployed mmWave backhaul antennas support a communication range over 200 m, thus avehicle at 80 km/h will get more than 10 s to respond. Furthermore, Fig. 3.12(c) presents
the handover scene, in which the mmWave V2X switches to the RSU2 since the RSU1 map
no longer provides safety-related information. At the moment, the global map only consistsof the static map, the vehicle map and the RSU2 map. Although the red point clouds remainin the GUI, they already became static. On the contrary, the yellow point clouds fully cover
the moving vehicle, which indicates that the RSU2 map is dynamically transmitting to the
OBU.
Figure 3.13 presents critical network performance indicators measured while the vehicle isstaying within the RSU coverage (i.e., 20 m). As Fig. 3.13(a) shows, with mmWave V2X, bothof the transmission control protocol (TCP) and user datagram protocol (UDP) throughputs
exceed 1 Gbps. A peak value of 2.01 Gbps appears when using TCP and keeping the LOS
distance of 5 m between RSU and OBU. Even at the coverage edge, the throughputs of TCP

55(a) Leave the start point.

(b) Associate with RSU1.

56(c) Handover to RSU2.

Figure 3.12: Point clouds of OBU from dynamic cooperative perception.

and UDP still sustain at a relatively high level, i.e., 1.65 Gbps and 1.52 Gbps, respectively,
which far surpass the maximum theoretical values of legacy V2X (e.g., 27 Mbps of DSRC
and 28.8 Mbps of LTE PC5). Since the LiDAR utilized in the field trial generates UDP
packets with a size of 1248 bytes, the latency performance of mmWave V2X for one LiDAR
packet transmission is evaluated. As shown in Fig. 3.13(b), the average round-trip time(RTT) at 5 m and 20 m are 1.116 ms and 1.048 ms, respectively. It does not reveal any
appreciable performance degradation as the LOS distance increases. Besides, all measured
latency within the RSU coverage is less than 3 ms which is already the toughest latency
requirement for sensor information sharing claimed in the 3GPP Technical Specification (TS)22.186 [57]. In addition, the packet loss rate of UDP transmission via mmWave V2X isrecorded to characterize its reliability. As Fig. 3.13(c) shows, although the packet loss rate
slightly grows as the LOS distance increases, the highest value is 0.043% at 20 m. It keeps
below 0.05%. Therefore in this network, mmWave V2X can ensure the reliability of at least
99.95% for LiDAR map transmission within the RSU coverage.
Throughout the entire process of automated driving, SDN management does endow efficiency and flexibility to mmWave association and backhaul routing, and also provides re-57
markable safety enhancements to the OBU, which demonstrates the benefits of our proposed
network architecture.

3.4.4

Discussions on scalability

It is worth emphasizing that the proposed network system has sufficient scalability, although
the PoCs were conducted with a limited number of RSUs and vehicles. It can be illustrated
from three aspects: 1) C-plane capacity, 2) D-plane capacity, and 3) SDN controller performance variation under an increasing number of D-plane elements. Specifically, in the outdoor
testbed, the WiMAX router used for the C-plane can support an uplink data rate of 75 Mbps.
The C-plane periodically collects the context information of vehicles at 10 Hz, which occupies the largest portion of the C-plane bandwidth. The common V2X message that includes
such context information is the CAM, of which the packet size ranges from 200 to 800 bytes.
Therefore, the C-plane has enough capacity to collect CAMs from at least 1171 vehicles. On
the other hand, the D-plane deploys the 60 GHz WiGig. The WiGig modules in the testbed
have an achievable data rate of 2.5 Gbps (MCS 2-9). Since the 3D LiDAR (VLP-16) generates packets at a speed of 7.52 Mbps, if each vehicle or RSU equips only one 3D LiDAR, theD-plane can support cooperative perception among 332 vehicles or RSUs. Indeed, the SDN
controller’s processing capability may degrade as the number of associated D-plane elements
grows. A prior study has investigated the variation of flow installation latency of some commonly used SDN controller frameworks when the OpenFlow switch number increased from 2
to 128 [58]. However, no significant latency increase (0.78 - 4.68 ms) was observed for the ODL
controller, exactly the one adopted in the testbed. Hence, it can be concluded the proposedsystem has scalability for large-scale V2X networks and extensive cooperative perception.

3.5

Conclusion

The commercial roll-out of 5G will create a new era for ITS. As two key technologies of 5G,
SDN and mmWave communications have many use cases in vehicular networks to promote safe
automated driving. The benefits of mmWave V2X and SDVN have been revealed by plentyof state-of-the-art works, respectively. However, the research concerning their integration isstill vacant, especially lacking full-fledged prototype systems for PoCs. This chapter discussed
the performance requirements of vehicular networks to enable safe automated driving. Based58(a) Transmission throughput between RSU and OBU within the coverage.

(b) Transmission latency for one LiDAR packet of 1248 bytes.

59(c) Packet loss rate of UDP transmission.

Figure 3.13: Network performance with mmWave V2X.
on the discussion, a software-defined dynamic mmWave V2X network was proposed for cooperative perception. Beyond that, this chapter introduced the necessary hardware equipment
and software frameworks to build prototype systems for indoor and outdoor PoCs. In theindoor PoC, the two modes of software-defined cooperative perception were implemented overmmWave V2V/V2I communications, showing that the indoor prototype system can achieve
the network functions of the proposed architecture. Furthermore, with additional hardware,
the outdoor PoC was carried out in real road environments. Under the dynamic managementof SDN controller, the cooperative perception, which distributed raw LiDAR pointcloud datafrom RSUs to the vehicle over mmWave in a challenging U-bend driving scenario, was successfully performed, which verified mmWave performance, the feasibility of the proposed networkarchitecture, and the safety benefit of cooperative perception to vehicles.

60

61

Chapter 4
Het-SDVN: Software Defined
Heterogeneous V2X Network for
Cooperative Perception
In order to cater to customer and automotive industry desires for advanced V2X applications and deal with their soaring QoS demands, academia, standardization organizations,and telecom operators start to throw their attention to the underexploited higher frequency
bands, such as the mmWave band (30 - 300 GHz), terahertz band (100 GHz - 10 THz), andthe visible light communication (VLC) band (400 - 800 THz). It is practical to envision sucha future, where these high-frequency radio access technologies (RATs) and the traditional
V2X technologies in the low-frequency bands (5.9 GHz, 2.5 GHz, 760 MHz, etc.) coexist
and provide fundamental support to the smart mobility ecosystem. More specifically, they
will be dynamically assigned to help different applications or enable the same application fordifferent use cases or driving automation levels. As the V2X network becomes more and moreheterogeneous, two main bottlenecks have been exposed:

1. Complexity of network management: As pointed out, low-frequency and high-frequency
V2X will coexist in the network. They feature different characteristics in propagation(path loss, penetration capability, etc.). The network management must be adaptive tocooperative perception requirements and selected V2X. For example, broadcasting thatused to be applied in IEEE 802.11p is not effective for mmWave. Multi-hop transmission
is required by mmWave V2X, so the routing topology needs to be determined efficiently.

62
2. Interoperability between diverse networks: Currently, vehicular ad-hoc networks (VANETs)
led by the European Telecommunications Standards Institute (ETSI) and cellular V2X(C-V2X) led by the 3rd Generation Partnership Project (3GPP) are independent networks. They possess their own V2X technical stacks. IEEE 802.11p and its successor
IEEE 802.11bd are used for VANETs, while LTE and NR are adopted by C-V2X. Theinteraction and handover among these technologies have not been clearly specified.

4.1

Motivation

In the previous chapter 3, the author has designed an SDN-based mmWave V2X network andcarried out extensive proofs of concept, in which both the 3GPP network (LTE) and IEEE
network (IEEE 802.11ad) were involved. Their interoperability was achieved by serving different planes in SDVN (LTE as C-plane, IEEE 802.11ad as D-plane). To the best of theauthors’ knowledge, it was the first prototype of mmWave SDVN and the first demonstrationof SDVN supporting cooperative perception. Nevertheless, it left some defects for improvement. One was at the D-plane. Only mmWave was exploited, so the management focused on
multi-hops, which was inefficient for cooperative perception with processed sensor data that
could be broadcasted. The other was at C-plane. The whole network only relied on a single
C-plane. The scalability and stability were questionable. It may fail to meet the required
latency when the network gets huge and crowded.
Therefore, in this chapter, additional contributions are achieved in four aspects:• A hierarchical SDVN architecture is designed. By dividing the roles and scopes of local
and global C-planes, the global SDVN controller will only focus on cross-network operations and extreme scenarios. The local SDVN controllers will handle the networking
within sub-SDVN networks. In this way, control overhead can be diluted, and latency
can be promised.
• Heterogeneous V2X is considered in this new hierarchical SDVN (Het-SDVN). After
presenting a survey on the existing and emerging V2X technologies, the applicabilityof IEEE-based V2X, 3GPP-based V2X, and even satellite networks, to the Het-SDVN
C/D-planes is comprehensively discussed.
• A local SDVN framework suitable for RSUs is developed. This framework adds sensor management, which leverages the legacy V2X messages over DSRC to control the63
ON/OFF of RSU sensors for cooperative perception. The system shows good performance in application initiation delay and energy consumption after deployment.
• A full-fledged Het-SDVN testbed is set up. The testbed incorporates major ITS components, such as vehicles with OBUs, RSUs, and multi-access edge computing (MEC)
servers. Based on the testbed, a proof of concept is carried out to demonstrate HetSDVN’s capability to provide cooperative perception within and across sub-SDVN networks. The holistic system performances of C-planes and D-planes are evaluated.

4.2

Design of Het-SDVN architecture

Figure 4.1: Overview of the Het-SDVN architecture (hierarchical C-plane and heterogeneous
V2X D-plane).

4.2.1

Architecture description

The primary goal of the proposed Het-SDVN architecture is to effectively fulfill the stringent
and diverse QoS requirements of V2X applications, specifically those pertaining to cooperative

64
perception. To that end, it is crucial to have global optimization on network decisions andminimize control overhead. Extensive studies have exposed that a fully centralized control
architecture exhibits remarkable efficiency in adapting to highly dynamic vehicular networks[27]. However, these architectures encounter significant performance degradation when theservice range of the SDVN controller expands substantially. In order to retain the benefitsof centralization while simultaneously mitigating excessive overhead, a two-layer hierarchical
SDVN C-plane is introduced and illustrated in Fig. 4.1.
1) Local SDVN C-plane: This C-plane manages one sub-SDVN network, where the central
node is an RSU. The RSU hosts C-plane functions and orchestrates communication resources
for D-plane applications. The local SDVN controller processes the requests sent by the onboard units (OBU) of connected vehicles and dynamically controls V2X networks (typically,
V2V and V2I) to transfer the contents.
2) Global SDVN C-plane: Global C-plane manages geographically decentralized subSDVN networks. This C-plane takes advantage of MEC or Cloud resources and wide-areacommunication systems such as cellular networks and satellite networks. The global SDVN
controller holds an extended network view by synthesizing the context information (e.g.,
topology, resource distribution, and traffic information) from the underlying local SDVN controllers. If vehicles send requests that exceed the capabilities of local SDVN controllers, orvehicles move to infrastructure-less areas, it hands over to the global SDVN controller tocontrol the network.
3) RSU and OBU D-plane: The Het-SDVN D-plane consists of stationary RSUs and mobile
OBUs. There have been a variety of wireless interfaces that support data transfer in this Dplane. IEEE-based interface technologies (IEEE 802.11p, 802.11n/ac, 802.11ad/ay, etc.) andtheir 3GPP-based counterparts (LTE and NR sidelinks) are two significant representatives.
Application data, for example, raw or processed sensor data for cooperative perception, isrouted over a relay network of RSUs and OBUs, which is configured by SDVN controllers
using southbound interfaces like OpenFlow. The vehicle density, channel conditions, QoS
requirements, etc., may affect the selection of heterogeneous V2X interfaces at each hop.

4.2.2

Utilization of heterogeneous V2X

Table 4.1 summarizes the legacy and emerging radio access technologies (RATs) that are usedor potentially to be used for V2X.

65
Table 4.1: List of heterogeneous V2X technologies and characteristics.
Technology

Data Rate

Coverage

D-planeC-planeLow-frequency V2X
Wi-Fi (2.4/5/6 GHz)

★★✩

★★✩

Local

Local

IEEE 802.11p (5.9 GHz)

★✩✩

★★✩

Local

Local

LTE-V2X (5.9 GHz)

★✩✩

★★✩

Local

Local

NR-V2X (FR1)

★★✩

★★✩

Local

Local

High-frequency V2X
NR-V2X (FR2)

★★★

★✩✩

Local

N/A

IEEE 802.11ad (60 GHz)

★★★

★✩✩

Local

N/A

THz (0.1 - 10 THz)

★★★

★✩✩

Local

N/A

VLC (400 - 800 THz)

★★✩

★✩✩

Local

N/A

Other V2X

4.2.2.1

IEEE 802.11bd (5.9/60 GHz)

★★★

★★✩

Local

Local

Starlink (Ku, Ka)

★★✩

★★★

GlobalGlobal

Low-frequency V2X technologies

Low-frequency V2X mostly operates in the sub-6 GHz band. IEEE 802.11p is a representativeof the first-generation V2X technologies, which becomes the basis of DRSC in America, ITSG5 in Europe, and ITS-Connect in Japan. IEEE 802.11p performs at 5.9 GHz and provides adata rate of up to 28.8 Mbps to support basic V2X applications as defined in ESTI technicalreport (TR) 102 638 [11]. Then, the 3GPP launches formal discussions on enhancing cellular
networks for V2X since Release 14 (Rel-14). In 2016, LTE-V2X was finalized. Further
enhancements to LTE-V2X sidelink (PC5) for direct communications, and the legacy airinterface (Uu), were studied under Rel-15. In addition, NR-V2X was developed since Rel-16,
which was frozen in 2020, while the discussions on its enhancements now continue in Rel-17
and Rel-18. NR-V2X targets two frequency bands (FR1: 410 MHz - 7.125 GHz, FR2: 24.25
GHz - 52.6 GHz), but the primary focus for the first-generation NR-V2X design is in FR1 toenable low-latency reliable sidelink communications at 5.9 GHz. Cellular technology-based
V2X (C-V2X) outperforms IEEE 802.11p in terms of data rates, latency, and reliability.
However, IEEE 802.11p won’t be obsolete owing to its successful mass-market applications

66
over the last decades and economy. Their coexistence and interoperability should be one of
the major focuses of the newly designed V2X architecture.
Other than the IEEE 802.11p and C-V2X, Wi-Fi based on the IEEE 802.11 family ofstandards, which works in the unlicensed 2.4/5/6 GHz bands, can be potentially used forV2X due to its low cost and ease of deployment. Wi-Fi has become ubiquitous in urban areasand its performance is improving drastically as the generation evolves. In Wi-Fi 6 (802.11ax),
a data rate of over 1 Gbps is supported under the single-carrier and single-stream settings.
However, one of the well-known drawbacks of Wi-Fi is the catastrophic interference which
heavily degrades its practical performance.
Since the aforementioned technologies all operate in low frequency bands, they suffer fromtrivial path loss and penetration loss, thus earning a good coverage. In this Het-SDVN, they
can serve as C-planes in different layers. For example, Wi-Fi, DSRC, PC5-based LTE, andNR (FR1) provide local C-planes, while Uu-based LTE provides global C-planes as it canreach several kilometers. Apparently, low-frequency V2X can contribute to C-planes andD-planes simultaneously (in-band mode) or separately (out-band mode), because they havefully met the QoS requirements of basic V2X applications (including detected object-based
and feature-based cooperative perception).
4.2.2.2

High-frequency V2X technologies

V2X operating in higher-frequency bands now attracts great interest due to the vast unexploited spectrum resources. The mmWave, terahertz (THz), and visible light communication(VLC) are promising technologies to be used for V2X. NR-V2X has to fulfill the peak data
rate of 20 Gbps as required by IMT-2020, thus introducing the mmWave band (FR2). Ina field trial of remote driving [23], the NR-V2X system deployed at FR2 demonstrates its
ability to provide broadband, reliable and low-latency communications. At the same time,
the applicability of IEEE 802.11ad to V2X is being evaluated by many projects [17]. IEEE
802.11ad, also known as WiGig, is an earlier, commercialized mmWave technology working
in the unlicensed 60 GHz band. It enables the peak data rate of 6.75 Gbps with 2.16 GHz
bandwidth.
Although THz and VLC were not included in the design of NR-V2X, they have been
identified as key 6G-V2X technologies [59]. Of course, their utilization still faces serious challenges, such as the propagation measurement and channel modeling for THz communications,and the interference management for VLC. An indoor THz system measurement validates a

67
104 Gbps wireless link performance [60], which shows THz’s potential to enable immersive
V2X applications with AR/VR/XR. In [61], an automotive VLC system using light-emitting
diode (LED) transmitters and camera receivers achieves a system performance of 45 Mbps
without bit errors. These preliminary evaluations pave the way for their future adaptation tomore challenging V2X scenarios.
Owing to abundant bandwidth allocation, high-frequency V2X enables ultra-high data
rate and ultra-low latency communications, so that raw data-based cooperative perception is
supported. However, the coverage is sacrificed as the frequency increases. In addition to severe
path loss, antenna directivity and weakened penetration capability make broadcast-based
communication infeasible. Hence, in this Het-SDVN, high-frequency V2X is only suitable forD-plane transmission. The SDVN controllers should manage their multi-hop communications
and select appropriate D-plane nodes as relays.
4.2.2.3

Other V2X technologies

Other V2X technologies target both low- and high-frequency communications. In March
2018, a new task group, TGbd, was established by IEEE for next generation V2X (NGV).
The task is to develop IEEE 802.11bd as the successor to IEEE 802.11p. IEEE 802.11bd will
work in the 5.9 GHz band and optionally in the mmWave band from 57 GHz to 71 GHz.
In addition to having interoperability, coexistence, backward compatibility, and fairness withIEEE 802.11p, IEEE 802.11bd will inherit advanced PHY and MAC techniques implemented
in the latest Wi-Fi standards (IEEE 802.11/n/ac/ax) to enhance the 5.9 GHz performance forat least two times higher throughput and communication range. As for the mmWave band,
it is proposed to reuse portions of the PHY and MAC layers from IEEE 802.11ad/ay [50].
However, detailed specifications are ongoing. Considering the coverage, for this Het-SDVN,
5.9 GHz IEEE 802.11bd can play roles in the local C-plane and D-plane, while its mmWave
version should serve the D-plane.
In February 2021, SpaceX, the US company, launched its commercial low Earth orbit(LEO) satellite communication service, known as Starlink. This system comprises over 4000
small satellites in LEO and operates within the Ku (12 - 18 GHz) and Ka (27 - 40 GHz) frequency bands, as authorized by the Federal Communications Commission (FCC). According
to the disclosed specification [62], it offers impressive mobile user performance, with up to 250
Mbps downlink and 30 Mbps uplink speeds, comparable to LTE systems. Notably, Starlink
ensures ubiquitous coverage, enabling internet access even in rural and infrastructure-less ar-

68eas. This unique capability makes it an exceptional choice for the global C-plane and D-planeof Het-SDVN, except the LTE-V2X.

4.2.3

Profits to cooperative perception

The proposed Het-SDVN resolves the connectivity and scalability issues of existing SDVN
architectures. With reliable C-plane associations, local/global SDVN controllers master realtime context information in D-plane and respond quickly to application requests from vehicles.
Regarding cooperative perception, this Het-SDVN provides support from two aspects.
Firstly, the local SDVN provides cooperative perception for road/driving safety. This typeof cooperative perception requires the exchange of sensing data focused on the immediate
regions. Typical scenarios include safe overtaking and safe ramp merging, where an RSU withlocal SDVN controller manages the transmission of raw or processed sensing data over V2I
and V2V links, allowing for the notification or assistance in detecting the presence of vehicles
that pose a risk of collision. Secondly, the global SDVN provides cooperative perception for
traffic efficiency. Along the planned route that an automated vehicle will take, congestion
and accident information perceived by sensors in other sub-SDVNs are useful for route replanning. The global SDVN controller manages the distribution of the above information
over backhauls according to requests. There is a special case of cooperative perception where
the global SDVN guarantees safety. It is on roads with harsh driving conditions (lighting,surface quality, etc) and without infrastructure support (usually in rural or mountainous
areas). Vehicles can get the latest sensing information from passing vehicles for safety bysending requests to the global SDVN controller, which establishes V2N2V connections using
wide-area communication systems.

4.3

Local SDVN framework for RSU and evaluation

4.3.1

Framework composition

The proposed framework is established by integrating Ryu and Robot Operating System(ROS), two popular open-source projects in SDN and robotics, respectively. The Ryu is
specialized in network traffic management and is capable of centralized and dynamic routing
policy enforcement depending on (network) environmental changes. On the other hand, theROS community has developed a plethora of robotics applications (nodes) for sensing and69

Figure 4.2: SDVN-based framework for cooperative perception.

70
communications. The drivers for most commercial-off-the-shelf sensors are available in ROS.
Therefore, ROS can work as the major platform to provide perception capabilities and sensor
management.
Figure 4.2 shows the framework composition. The boxes in the figure represent the fundamental hardware and software components. As of now, vision cameras and LiDARs havebecome the standard equipment of RSUs for comprehensive monitoring of surrounding environments. In addition, to supply V2X connectivity in different situations, RSUs will leverage
two different V2X interface technologies. One is the classical DSRC interface at 5.9 GHz
for small message broadcasting. There are already commercialized DSRC products for RSUs(e.g., Cohda Wireless MK6 [63]). The other is the promising mmWave interface for directional raw sensing data transmission. The technical candidates include 5G-NR V2X at FR2(24.25-52.6 GHz), IEEE 802.11ad/ay and IEEE 802.11bd at 60 GHz. In some areas, streetlamps have been equipped with mmWave V2X for highly accurate positioning and enhanced
perception [17]. Hence, the coexistence of DSRC and mmWave V2X interfaces on RSUs willsoon be a trend.
The software components for service coordination are functioning in ROS. The DSRC
Message Encoder/Decoder is a ROS driver for DSRC interfaces which parses incoming CAM
messages and publishes corresponding vehicle information. The Sensor Pre-Configure isa ROS node for sensor management, in which a suite of sensor communication APIs is
defined to control sensor status (e.g., the ON/OFF switching of cameras or LiDARs, thespeed of laser motor rotation, etc.). The trigger of this node is the message from DSRC
Message Encoder/Decoder that indicates the emergence of connected automated vehicle(CAV) users. Once cooperative perception is triggered, the Multi-Sensor Calibration andObject Detection & Tracking will handle the recognition tasks and generate an object list
to format CPM messages. The AP Association Monitor waits for the access of mmWave
V2X and reports the status to the Routing Policy Generation which summarizes associated vehicle contexts and creates an optimal (delay or bandwidth-based) routing policy.
The Ryu controller can map the routing policy to specific forwarding tables compatible
with OpenFlow v1.3. The flow entries will be issued to the datapaths, i.e., Open Virtual
Switch (OVS) bridges, of RSU and authenticated vehicles. Then, the point cloud data flows
can be shared with specified targets.

71

4.3.2

Inherent mechanisms

4.3.2.1

Energy-efficient dynamic sensor ON/OFF strategy

Figure 4.3: DSRC message-triggered dynamic sensor ON/OFF.
The energy consumption of RSUs depends on the number and capability of equipped
sensors, which is not negligible. Take RoboSense LiDARs as an example. The average power
consumptions for the 16, 32, and 80-beam series are 12 W, 13.5 W, and 38 W, respectively. On
the other hand, turning RSU sensors ON all the time is neither energy-efficient nor judicious
since there are periods when no CAVs appear in the service range. Empirical information canbe leveraged to switch the ON/OFF of sensors. In [64], the authors predict vehicle traffic basedon prior traffic information and employ reinforcement learning (Deep Q-learning) to decide
whether to turn RSU ON/OFF. However, machine learning approaches incur more systemcomplexity and additional requirements on computational abilities. Instead, we propose an
intuitive strategy, i.e., to exploit existing and broadcasted DSRC messages as a trigger for
dynamic sensor ON/OFF, as depicted in Fig. 4.3.

72
The switching of sensor ON/OFF is based on the reception of DSRC messages, mainly
CAM messages, which indicate the incoming CAVs. When CAVs are out of DSRC coverage(hundreds of meters away), RSU sensors are turned OFF for energy-saving. When CAVs enter
the range of DSRC but not mmWave (150 - 200 m), RSU sensors will be turned ON, andcooperative perception will be activated firstly to share CPM messages via DSRC interfaces.
Further, as soon as mmWave V2X is associated, shared information will become raw sensor
data like LiDAR point clouds and be steered by the SDN controller to CAVs for more finegrained perception.
Likewise, when CAVs depart, RSU sensors will be switched OFF. Note that it is worthyof an in-depth discussion on the exact timing of sensor OFF after completing the service to
avoid unnecessary energy consumption.
4.3.2.2

Multi-RSUs service scheduling

Figure 4.4: Sequence of RSU-centric cooperative perception.
Along the traveling course of CAVs, multiple RSUs provide cooperative perception subse-

73
quently. The service consistency is one of our pursuing features in this RSU system. Figure
4.4 explains the entire sequence of cooperative perception, from the sensor initiation and sensing data provision to the subsequent RSU handover. The DSRC association and mmWave
association are of milestone significance in the described procedures. The former represents
the wake-up of RSU sensors and the startup of CPM message broadcasting, and the latter
triggers raw sensing data sharing and SDN fine-grained flow control. When the RSUs are
deployed geographically close to each other (in urban scenarios), owing to the wide coverageof DSRC, CAVs are able to activate the service preparation in the subsequent RSU while
enjoying cooperative perception in the current RSU. When the RSUs are deployed far apart(in rural or highway scenarios), CAVs can either trigger each RSUs’ service independentlyor request the coordination of primary SDVN controller which will proactively activate the
RSUs along the traveling course by driving intention sharing and route the point clouds ofmultiple RSUs through backhaul.

4.3.3

Implementation and Evaluation

Figure 4.5: Experiment overview of Tokyo Tech smart mobility test field (three RSUs
deployed, one Robocar used as CAV, cameras and LiDARs installed for contextual
awareness, Wi-Fi and WiGig antennas providing V2X connectivity, and Intel NUCs
empowering computing capability at RSUs; the test CAV drives along the 360 m course(blue arrows) at a velocity of 10 km/h) [6].

74
4.3.3.1

Implementation

Figure 4.5 gives an overview of the experimental environment. This smart mobility test field
is located on the Ookayama Campus of Tokyo Institute of Technology, Japan. There havebeen three well-equipped RSUs deployed. Each of the RSUs has camera and LiDAR sensors,
as well as V2X capabilities compliant with IEEE 802.11p/DSRC and IEEE 802.11ad/WiGig.
Due to Japanese outdoor wireless regulations, we abandon DSRC and substitute it with 5.6
GHz Wi-Fi. The Wi-Fi routers belong to the Buffalo WAPM-1266WDPR Series. They canmaintain connectivity up to 300 m outdoors [65]. In contrast, the WiGig antennas madeby Panasonic can only reach around 150 m for line-of-sight (LOS) communications. Inside
the waterproof box, Intel NUC PCs are embedded as RSU control units where we deploy
developed functionalities according to the framework design presented in Section II. Notethat to enable ETSI-defined V2X messages (CAM, CPM), we utilize OpenC2X [66]. It is an
open-sourced prototyping platform that completely implements ETSI ITS-G5 protocol stack.
The tested CAV model is ZMP Robocar MV2 [67]. Except for a more powerful control PC
for autonomous driving-related functions (provided by Autoware.ai [68]), the CAV’s hardware
configuration is almost the same as RSUs, including a 32-beam LiDAR on the rooftop, front
cameras, Wi-Fi and WiGig antennas for V2X communications. Besides the OpenC2X, weinstalled OVS in the CAV control PC. The WiGig interface is logically attached to an internal
OVS bridge, which will communicate with the Ryu controller in RSUs automatically after
entering the mmWave coverage and then download the routing policy for raw sensor data
sharing. In this experiment, the shared data is LiDAR point clouds.
When the CAV receives point clouds from RSUs, it will either forward or utilize the data
itself. After mutli-hop relays, the received data will be merged together with the CAV’s
own observations (so-called sensor fusion process). This process can be implemented by acoordination transformation. Specifically, if Vm = [x, y, z, 1]T is a LiDAR point from a certain
RSU in the static map (m) domain, we need to apply Tmc ∗ Vm to translate it into a point′′′

Vc = [x , y , z , 1]T in the CAV (c) domain, where

RotX.x RotY.x RotZ.x T ranslation.x RotX.y
Tmc = 
 RotX.z


0





RotY.y RotZ.y T ranslation.y RotY.z RotZ.z T ranslation.z 


(4.1)



0

0

1

Tmc is a dynamic transformation matrix from the static map coordinate frame to the moving

75
CAV coordinate frame. Using ROS TF/TF2 packages, this matrix can be easily derived.
4.3.3.2

Evaluation

Figure 4.6: Total delay and the breakdown.

Time consumption: In a previous study [4], the necessary time consumption for CPM
message delivery has been measured. Including the processing delay in Autoware, ≤ 100 ms
can be achieved in the worst case. In our evaluation, we pay more attention to the time
consumed throughout the handover from CPM-based cooperative perception to LiDAR point
cloud-based cooperative perception. The total time consumption can be expressed as
Ttotal = Tdsrc + Tros + Tryu + TmmW

(4.2)

where Tdsrc indicates the delay for trigger message (CAM) delivery, including the reading
delay from GPS to OpenC2X sender and the transmission delay from OpenC2X sender to
OpenC2X receiver via 5.6 GHz Wi-Fi; Tros and Tryu represent the processing delay in ROS

76
coordinator and Ryu SDN controller, respectively; TmmW is the one-hop V2I transmission
delay for one-scan LiDAR point clouds via 60 GHz WiGig.
Figure 4.6 reflects the breakdown of delay measurements. The total time consumption
Ttotal is around 30 ms. Some of the delay, such as Tros and Tryu , will certainly increase as
the serving CAV number increases. For one CAV, activating the LiDAR sensor, computing
the routing policy and issuing flow tables just consume less than 10 ms in total. However,Tdsrc costs more than 20 ms (83% of Ttotal ). We speculate that most time of Tdsrc is spentin the codec of DSRC messages. As for the mmWave transmission time TmmW , a superior
performance is found, as only 0.5 ms are needed for one-scan LiDAR point clouds transmission.
In the case of the RSU LiDAR (RS-LiDAR-32) with a rotation speed of 600 rpm, one-scandata size equals 94036.8 bytes, which implies that the mmWave data rate is over 1.5 Gbps
during the experiment.

Figure 4.7: Power consumption under different sensor OFF timing.
Power consumption: As we mentioned in Section II. C, the exact timing of sensor OFF
after completing cooperative perception is worth discussing because it can avoid unnecessary

77energy consumption. We examine the power consumption in three cases: OFF after losing
mmWave, OFF after losing Wi-Fi, OFF after gaining visibility of blind spots, and comparethem to the case when the LiDAR sensor is always turned ON. The average power consumption, Pavg can be expressed as
Pavg =

Psensor × Vcav × TON
Lcourse(4.3)

where Psensor indicates the sensor power in its specification (13.5 W for RS-LiDAR-32 we
deployed); Vcav is the CAV’s average velocity; TON is the duration of sensor ON along thedriving course; Lcourse represents the length of the course.
Figure 4.7 shows the results of average power consumption of RSU sensors when thetest CAV moves along the course (white). The power reduction percentages for the three
examined cases are 54%, 49.5% and 60%, respectively. This implies that, if an RSU can

Figure 4.8: Cooperative perception under higher-layer orchestration.

78
master accurate information about the exact timing of each CAV when their safety needs
are met by cooperative perception, it is possible to take the most effective sensor ON/OFF
strategy to save power. This information can be reported by CAVs actively or notified by theupper-layer orchestrator after prediction according to historical data.
Scalability: Our ultimate target is to promote this framework system to be deployedin every RSU of society. To that end, scalability is an indispensable requirement. The
software part can be developed in containers for convenient migration to RSU operating
systems. Moreover, to manage those distributed RSU systems jointly, we can rely on upperlayer orchestrator, such as the global SDVN controller introduced in Sect. 3.2.1. Figure
4.8 shows an example of how SDVN controller in MEC/Cloud can contribute to the RSU
cooperative perception. When the CAV velocity is extremely fast or the road scenario isextremely complex, the CAV may require more sensor data for safety. In this case, MEC
SDVN controller will notify the critical RSUs along the course of the CAV and issue routing
policies to them. Finally, the sensor data originated from multiple RSUs can be relayed via
backhaul networks and reach the target CAV via V2X. Obviously, a single RSU system cannot
perform such a large-scale coordination.

4.4

Proof of concept of Het-SDVN

4.4.1

Experimental topology

To simplify the experimental topology without losing hierarchical mechanisms, the scenariothat a vehicle benefits from cooperative perception (CP) in the current local sub-SDVN
network and requests CP from another sub-SDVN network is considered, as shown in Fig.
4.9. In this scenario, Vehicle1 runs CP for safety. The nearest CP server at RSU1 handles
the CP request containing an accurate vehicle position. Sensing data from RSU1 is thendelivered to Vehicle1 in its demanded format (detected objects, raw data) through a V2X
network organized by the local SDVN controller. For the purpose of efficiency, Vehicle1 also
needs perceptual information of RSU2, which exceeds the charge area of RSU1. Hence, thistype of CP request goes to the CP server at MEC or Cloud. The global SDVN controller is
responsible for sensing data routing across these two sub-SDVN networks.
In this topology, D-plane consists of two parts: the local SDVN D-plane within RSU1
coverage and the global D-plane connecting the two sub-SDVN networks. The former D-

79

Base Station

Global C-Plane

CP Server &Global SDVN Controller

Global D-Plane

MEC/CloudCP Server &Local SDVN Controller

V2N

Global C-Plane
Ethernet

Ethernet
Local C-PlaneV

2I

Global D-Plane(Backhaul)

RSU1(Nearest)

V2V

RSU2
(Remote)t1: Vehicle1 requests CP data from RSU2
Vehicle1 (Behind)

Vehicle2 (Front)

t0: RSU1 provides CP to Vehicle1

SDVN C-plane
SDVN D-plane
Wired Link
Wireless Link

Figure 4.9: Experimental topology for Het-SDVN demonstration.
plane includes three nodes, two vehicles and one RSU, as well as multiple V2V/V2I links
to forward sensing data. The latter D-plane are mainly wired networks, where sensing dataneeds to pass through large amounts of backhaul switches and Ethernet links in between.
As for C-plane, the described CP relies on the local SDVN C-plane of RSU1 and the global
SDVN C-plane to schedule network resources. These two C-planes require different coverage.
The former C-plane only needs to cover a target region, like an intersection with high accident
probability where RSU1 can locate for safety, while the latter C-plane must be ubiquitous tomitigate the risk of local SDVN controller failures and provide CP even in RSU-less areas.

4.4.2

Environment setup

The smart mobility field for the proof of concept is placed at Tokyo Institute of Technology,Ōokayama campus [6]. This field includes smart vehicles and RSUs for automated driving
and V2X-related tests. The proof-of-concept environment is shown in Fig. 4.10. It is seenthat hardware equipment for sensing, communication, and computation is installed on the80
exploited vehicles and RSUs. Table 4.2 reveals the details about the equipment. Here, LiDARs
are used to monitor dynamic 3D objects. Therefore, the dense pointclouds become major data
sources of CP. The 32-laser LiDARs and 80-laser LiDARs are deployed on vehicles and RSUs,
respectively.
For D-plane implementation, Wi-Fi (IEEE 802.11a/n/ac) at 5 GHz and WiGig (IEEE802.11ad) at 60 GHz are two options for the local SDVN D-plane. To support the WiGig,
directional antennas are installed on RSUs and the rooftop of vehicles so that mmWave V2I
and V2V are enabled. The global SDVN D-plane directly uses campus Ethernet, but a backup
could be 4G LTE. Pocket LTE devices are put on vehicles to provide access to the global SDVN
C-plane. As for the local SDVN C-plane, Wi-Fi (IEEE 802.11b/g/n) at 2.4 GHz is adopted.
These Wi-Fi APs are set up on RSUs.
In addition, computation devices with customized capabilities are supplied in order toperform CP and the functions of SDVN controllers on vehicles, RSUs, and the MEC server.
LiDAR
V2V antenna
Car1正面

SDVN C-plane
SDVN D-plane

OBU

Car1后备箱

Front
antenna
Car2正面Rear antenna
Car2后备箱

V2I antenna

RSU1写真Wi-Fi AP
Vehicle1
Vehicle2

RSU1
MEC Server

Local SDVN controller

Server写真

Global SDVN Controller
RSU LiDAR

Photo of the proof of concept in RSU1 local SDVN

Figure 4.10: Proof-of-concept environment.

RSU2

81

Table 4.2: Proof-of-concept hardware
Category

PurposeSpecifications
Number of lasers: 80

Sensor

RSU LiDAR

Frame rate: 10/20 Hz
Output: 44.982 Mbps (UDP packets)
Number of lasers: 32

Vehicle LiDAR

Frame rate: 5/10/20 Hz
Output: 15.6 Mbps (UDP packets)
Standard: IEEE 802.11ad

Local D-PlaneAP

Frequency: 60 GHz (2.16 GHz BW)
Data rate: 385 Mbps - 2.5 Gbps
Standard: IEEE 802.11a/n/ac
Frequency: 5 GHz (20 MHz BW)
Data rate: 6 Mbps - 173.3 Mbps
Standard: 4G LTE

Global C-Plane

Frequency: 2.5 GHz (20 MHz BW)
Coverage: Up to 12 km
Standard: IEEE 802.11b/g/n

Local C-Plane

Frequency: 2.4 GHz (20 MHz BW)
Coverage: Up to 400 m
Model: Mouse DAIV 7N (32 GB)

Vehicle PC
Computer

CPU: 8-core Intel Core i9-11900K
GPU: NVIDIA GeForce RTX 3080
Model: Jetson AGX Orin (32 GB)

RSU PCCPU: 12-core Arm Cortex-A78AE
GPU: 2048-core NVIDIA Ampere
Model: Precision Tower 7920 (64 GB)

MEC Server

CPU: 10-core Intel Xeon Silver 4210
GPU: NVIDIA Quadro RTX 5000

82

4.4.3

Network orchestration

The design of network orchestration in this proof of concept showcases the hierarchy andheterogeneity principles of the proposed Het-SDVN architecture. Figure 4.11 explicitly draws
the application, networking, and data sequences when Vehicle1 runs CP and claims for various
data formats.
In Fig. 4.11(a) and (b), the CP client on Vehicle1 broadcasts CP requests, indicating a
region of interest (ROI) and format preference, which are received by the nearest local CP
server on RSU1. As shown in Fig. 4.11(a), when Vehicle1 specifies the format as detected
objects and the ROI as the current sub-SDVN, the local CP server on RSU1 sends a request
to the local SDVN controller and performs LiDAR detection. The local SDVN controller
on RSU1 creates OpenFlow tables and installs them on the Open vSwitches of Vehicle1 and
RSU1, respectively. The OpenFlow table for RSU1 is used to broadcast the detected objects
of RSU1 via UDP over 5 GHz Wi-Fi. When the ROI changes to the RSU2 sub-SDVN, thelocal CP server on RSU1 notifies the global CP server on MEC and requests the local SDVN
controller to enable broadcast-based networking. The global CP server forwards the Vehicle1
request to RSU2 and simultaneously sends a request to the global SDVN controller for routing
the detected-object data over the backhaul networks.
The sequence is almost the same when raw LiDAR data becomes the preferred format, asshown in Fig. 4.11(b). The major difference lies in the local SDVN. The large data size of
LiDAR pointcloud makes it intractable for broadcasting. Therefore, the local SDVN controller
on RSU1 installs new flow tables on Vehicle1, Vehicle2, and RSU1. The OpenFlow table for
Vehicle2 is used to relay the raw data to Vehicle1 through a V2I and a V2V link of 60 GHz
WiGig.
In this proof of concept, both the CP client and servers are implemented as ROS2 nodes[69], leveraging the detection algorithm (CenterPoint) in autoware.universe [70], a ROS2-based
open-source framework for automated driving. Besides, a Python-written SDN controller
framework called Ryu is used to realize the local and global SDVN functions [71]. Ryu provides
OpenFlow components and well-defined API. For the development of various interfaces (i.e.,
the CP client-server interface, CP server-to-SDVN controller interface, and SDVN local-toglobal interface), Python sockets and Python Flask are utilized. Flask enables the creation of
HTTP-based RESTful APIs, which are widely used by web applications and as northboundinterfaces of SDN controllers.

83
Vehicle1
CP
ClientVehicle2

Open
vSwitch

RSU2

RSU1

Open
vSwitchOpen
vSwitchLocal CP
ServerLocal SDVN
Controller

Open
vSwitch

MEC Server

Local CP
ServerGlobalCP Server

Global SDVN
Controller

Local SDVN

CP request (ROI: current sub-SDVN; Format: detected objects)

V2X request
Routing table(OpenFlow)

Routing table(OpenFlow)
UDP Broadcasting
FusionDetected objects

UDP Broadcasting

CP request (ROI: RSU2 sub-SDVN; Format: detected objects)

V2X request

Vehicle1 CP request
Backhaul request
Routing table(OpenFlow)

Routing table(OpenFlow)
UDP Broadcasting
FusionGlobal SDVN

Vehicle1 CP request

Detected objects

UDP Broadcasting(a) Vehicle1 requests CP in a detected-object format

Vehicle1
CP
ClientVehicle2

Open
vSwitch

RSU2

RSU1

Open
vSwitchOpen
vSwitchLocal CP
ServerCP request (ROI: current sub-SDVN; Format: raw LiDAR data)

Local SDVN
Controller

Open
vSwitch

MEC Server

Local CP
ServerGlobalCP Server

Global SDVN
Controller

Local SDVN
V2X request

Routing table(OpenFlow)

Fusion

RSU1 LiDAR data

Routing table(OpenFlow)

Routing table(OpenFlow)

RSU1 LiDAR data

CP request (ROI: RSU2 sub-SDVN; Format: raw LiDAR data)

Routing table(OpenFlow)

Fusion

RSU1+RSU2
LiDAR data

Routing table(OpenFlow)
RSU1+RSU2
LiDAR data

V2X request

Vehicle1 CP request

Global SDVN
Vehicle1 CP request
Backhaul request
Routing table(OpenFlow)

Routing table(OpenFlow)
RSU2 LiDAR data(b) Vehicle1 requests CP in a raw-data format

Figure 4.11: Sequence diagrams of Het-SDVN for CP. (a). Detected object-based CP. (b).
Raw LiDAR data-based CP.

84

4.4.4

Evaluation and discussion

Proof-of-concept results are discussed from three aspects: the required data rate for two
different types of CP, the performance of the deployed local SDVN network and the overall
performance including the backhaul performance (global D-plane), controller reaction delay
and the visualization of CP.
4.4.4.1

Required data rate

Table 4.3: Average data rate (raw LiDAR data, detected objects).
Pointcloud Data Rate

Detection Data Rate

Detection Data Rate

RSU LiDAR

RSU1

RSU2

46.5 Mbps

2.84 Mbps

1.39 Mbps

Table 4.3 shows the practical output data rate of RSU1/RSU2 LiDAR and the data rates
of detected objects on RSU1 and RSU2, respectively. The former value depends on the
LiDAR specification. RoboSense 80-laser LiDAR Ruby-Lite is used by RSU1 and RSU2.
The measured average data rate is 46.5 Mbps, about 3 times that of the RoboSense 32-laser
LiDAR used by Vehicle1. It makes sense because RSU LiDARs are supposed to monitor
more precisely to ensure road safety. The latter values are affected by the surrounding object
number, the detected object number, the used detection algorithm, the message format, etc.
In this measurement, object detection is done by the ROS2 node lidar_centerpoint ofautoware.universe. It publishes detected objects in ROS2 message format DetectedObjects,
which are transferred via UDP. In 10 minutes, the average detection data rates of RSU1 and
RSU2 are 2.84 Mbps and 1.39 Mbps, respectively.
Comparing the measured data rates, it shows that sharing raw LiDAR data for CP poses
greater challenges to network performances, especially to the throughput (about 20 times
higher than the detected objects with only one LiDAR). It is crucial to consider different
radio access technologies and transmission modes of V2X in the local SDVN network.
4.4.4.2

Local SDVN performance

Vehicle1 receives CP data within the local SDVN of RSU1. Therefore, key performance
metrics like the C-plane coverage and D-plane throughputs should be evaluated.

85

Figure 4.12: RSSI maps of the SDVN WiFi C/D-planes.

86

100

Throughput (Mbps)

8060

LiDAR data rate
(RSU1 + RSU2)
LiDAR data rate
(RSU1)

40
Detection data rate
(RSU1 + RSU2)
Detection data rate
(RSU1)20

0

A (Farthest)

BC

D (Closest)

Measurement Position
Figure 4.13: Throughputs of Wi-Fi as local SDVN D-plane.
Figure 4.12 showcases the received signal strength indicator (RSSI) maps of RSU1 Wi-Fi as
the local C/D-planes. As shown in Fig. 4.12(a), RSU1 is located at a T-type intersection (redcircle). Since the RSSI values are measured under the condition that the Wi-Fi association isavailable, it demonstrates that using 2.4 GHz Wi-Fi as the local C-plane on RSU1 can cover
all three main roads to the intersection. So does the 5 GHz Wi-Fi D-plane, as shown in Fig.
4.12(b). Four red triangles are marked along the experimental course (from bottom to top:
A→B→C→D), which indicate the positions where Wi-Fi D-plane throughputs are recorded.
Figure 4.13 plots the average Wi-Fi D-plane throughputs. The error bars represent standard deviations and the dash lines indicate the required data rates under four processes
depicted in Fig. 4.11. It is seen that the practical throughput of 5 GHz Wi-Fi outdoors
degrades as the distance from RSU1 increases. When it comes to point B, this D-plane has
failed to support the broadcasting of RSU1 LiDAR data. Even at the closest point D, the87throughput improves, which, however, is still far from the requirement of transmitting both
RSU1 and RSU2 LiDAR data. It demonstrates that the 5 GHz Wi-Fi D-plane is only capable of detected object-based CP. Heterogeneous V2X, especially high-frequency V2X like
mmWave is necessary in order to enable raw data-based CP.
In contrast, the 60 GHz WiGig as another choice of local SDVN D-plane has extraordinary
throughput performance. The peak throughput observed on one WiGig V2I/V2V link reaches
2.20 Gbps. Due to the directivity of WiGig antennas, this performance is only sustained inline-of-sight communications. In addition, when multi-hop mmWave V2X occurs on a straightlane/road, like the topology of this proof of concept (RSU1→Vehicle2→Vehicle1), interference is non-negligible. This impact has been studied in [72], where the authors proposed toconfigure mmWave antennas as ZigZag to mitigate inter-vehicle interference. In this work,
channel management is implemented as a function of the local SDVN controller to relax interference. In Fig. 4.10, Ch2 (60.48 GHz) and Ch3 (62.64 GHz) are allocated to the mmWave
V2I and V2V links, respectively. Table 4.4 compares the throughputs and packet delivery
ratios (PDR) of mmWave D-plane with and without such channel control. It is shown thatby applying channel control, the average D-plane throughput increases from 602.1 Mbps to1.84 Gbps and the PDR also improves from 66.33% to 99.99%.
Table 4.4: Throughput and PDR of mmWave (WiGig) D-plane.
mmWave D-Plane

mmWave D-Plane

w/o Channel Control

w Channel Control

602.1 Mbps

1.84 Gbps

66.33 %

99.99 %

Throughput
PDR

4.4.4.3

Overall performanceThis proof of concept aims to effectively integrate the local and global SDVN networks. To
examine network operations, the data rates of mmWave and backhaul interfaces on Vehicle1,
Vehicle2 and RSU1 are recored using Wireshark after synchronization. Figure 4.14 shows thedata rate variation when SDVN controllers are requested to orchestrate the network for CP
following the sequence of Fig. 4.11(b). It is seen that the mmWave V2I and V2V links start tocarry raw LiDAR data (from RSU1) after the local SDVN controller responds. When Vehicle1
demands raw LiDAR data from RSU2, the global SDVN controller successfully arranges the

88backhaul network, which can be observed from the significant growth of backhaul data rates.
Meanwhile, the data rates on mmWave V2I and V2V links double, which implies that RSU2
LiDAR data get properly transmitted over the local SDVN network.

107

10

mmWave V2I
mmWave V2V
Backhual

9

Data rate (bps)8

Global SDVN controller
receives backhaul request

7

Local SDVN controller
receives V2X request

6
5
4
3
2
1
0

0

5

10

15

20

25

30

35

40

45

50

55

60

Time (seconds)
Figure 4.14: Variation of link data rates under SDVN controller orchestrations.
The local and global C-planes use heterogeneous networks (2.4 GHz Wi-Fi and Ethernet).
It is necessary to evaluate their HTTP latency and flow installation latency separately. Table
4.5 presents the test results. The HTTP latency over Wi-Fi C-plane is slightly larger andmore unstable than that over Ethernet, probably due to the outdoor wireless interference.
Nevertheless, if the global CP server is deployed on Cloud, instead of the MEC server inthis proof of concept. The HTTP latency can range from tens to hundreds of milliseconds.
Therefore, safety-critical requests need to be firstly processed by the local CP server. As
for flow installation latency, it is measured by CBench [58], one of the benchmarking tools
for SDN controllers. The CBench emulates one OpenFlow switch (connecting to the Ryu

89controllers) and calculates the number of flow modifications per second. The number of test
iterations is set to 20. Other parameters keep default and the same. It is seen that the local
and global SDVN controllers present similar flow installation efficiency regardless of network
types. Since the network flow can be configured within 3 ms, it is likely to support advanced
V2X applications which require an end-to-end latency of less than 10 ms.
Table 4.5: Latency performance of CP and SDVN controllers.
CP Client-Local CP Server

Local CP Server-Global CP Server

HTTP Latency

HTTP Latency

3.307 ms

2.539 ms

Local SDVN Controller

Global SDVN Controller

Flow Installation Latency

Flow Installation Latency

1.796 ms

2.693 ms

On Vehicle1 PC, the received CP data are visualized using the ROS2 tool Rviz, as shownin Fig. 4.15. Figure 4.15(a) reflects the detected objects from RSU1 and RSU2 as coloredbounding boxes with motion vectors, representing their categories, sizes, and mobility. These
attributes are extracted from raw LiDAR data using the detection method CenterPoint, whichsignificantly reduces the data size so that legacy V2X is still capable of broadcasting detected
objects. Although sharing raw data is resource-consuming, demanding high-frequency V2X
and additional bandwidth allocation, the LiDAR pointcloud data from RSU1 and RSU2, asshown in Fig. 4.15(b), are imperative for high-level automated driving in terms of reliability and liability. Regarding reliability, these raw data complement the pointcloud density of
Vehicle1 LiDAR and light up the vacancy spots along the driving course, which can increase
maneuvering agility and the tolerable time for safe maneuvering [20,21]. As for liability,
automated vehicles share higher portions of liability for incidents as the levels of autonomy
upgrade, because they gradually take over the driving responsibility from human drivers [73].
RSUs, however, share very limited liability when they don’t control vehicles directly. Therefore, Vehicle1 has the obligation to identify the trustworthiness of detection results from RSU1
and RSU2 if their authentication information (e.g., detection capability) is incomplete. With
raw data at hand, Vehicle1 can perform detection by itself, as shown in Fig. 4.15(c), so thesafety-critical decisions (brake, deceleration) can be double-checked. The visualization results

90clearly demonstrate the capability of the proposed Het-SDVN in managing heterogeneous V2X
resources to enable diverse CP effectively.
4.4.4.4

Discussions on scalability

The Het-SDVN introduces hierarchical C-planes and D-planes. In the proof of concept, theglobal C-plane is through the LTE at 2.5 GHz and Ethernet. The LTE capacity here is equal
to the WiMAX capacity in Chapter 3, which supports the collection of mobility informationfrom 1171 vehicles. For both the global C-plane and global D-plane, the Ethernet network is
extensible and has enough capacity to connect a large number of RSUs. As for the local Cplane, since the 2.4 GHz Wi-Fi (IEEE 802.11b/g/n) is used, its capacity is 11 - 72.2 Mbps incoverage, which allows 171 - 1128 vehicles to upload their mobility information at 10 Hz. The
local D-plane has two different RATs, the 5 GHz Wi-Fi (IEEE 802.11a/n/ac) and 60 GHz
WiGig, for sharing detected objects and raw LiDAR data, respectively. The former takes
around 2.84 Mbps bandwidth, while the latter costs about 46.5 Mbps bandwidth with the 80laser LiDAR (Ruby Lite). Accordingly, the Wi-Fi D-plane has a capacity of 6 - 173.3 Mbps,
and the WiGig D-plane has a capacity of up to 2.5 Gbps. Their supported number of D-plane
units (vehicles or RSUs) is 2 - 61 and 53, respectively. By improving the hardware, thisnumber can be further increased. Moreover, the local SDVN framework for RSUs adopts the
Ryu controller. In the benchmark evaluation [58], the flow installation of the Ryu controller
only delayed 2.32 ms when the number of OpenFlow switches increased from 2 to 128, whichreflected the good scalability of the deployed local SDVN controller in the proof of concept.

4.5

Conclusion

The emergence of advanced V2X applications facilitates the advancement of V2X communications and the innovation on traditional vehicular networks. In order to effectively support
various types of cooperative perception (CP) and ensure this safety-critical V2X application is deliverable to vehicles anywhere and anytime, this chapter proposed Het-SDVN, an
SDN-based V2X network architecture, consisting of hierarchical C-planes, geographically distributed but logically centralized sub-SDVNs, and heterogeneous V2X. The network roles wereclearly defined. For CP in local SDVNs, communications are orchestrated by RSUs with local
SDVN controllers; for CP in other SDVNs or infrastructure-rare regions, communications forcrossing sub-SDVNs are orchestrated by global SDVN controllers at MEC/Cloud servers. To

91empower the RSUs with SDVN functions, this chapter presented a framework that integrates
ROS, Ryu, and OVS and adds customized functional components for application, network,and sensor management. This framework was deployed and tested, showing that a maximum
power saving ratio of 60% was achieved with the CAM-triggered sensor ON/OFF strategy,
and the delay for CP preparation was less than 30 ms. Eventually, the holistic Het-SDVN
was set up in Tokyo Institute of Technology, and its support for CP has been demonstrated
via a proof of concept. Evaluation results validated the feasibility of Wi-Fi as local C-planes
and stressed the importance of mmWave and the necessity of interference management when
performing raw sensor data sharing. Moreover, no significant difference in flow installation
performance was observed between the local and global SDVN controllers over different networks. They were all under 3 ms, showing the potential to meet the latency requirement (lessthan 10 ms) of advanced V2X applications [7].

92(a) Detected object-based CP.

(b) Raw LiDAR data-based CP.

93(c) Detection performed with raw LiDAR data from CP.

Figure 4.15: Visualization of received CP data on Vehicle1 PC. (a). RSU1 and RSU2 send
detected objects. (b) RSU1 and RSU2 send raw LiDAR data. (c) Vehicle1 performs
detection with raw LiDAR data from CP.

94

95

Chapter 5
Orchestration of Mobility-Aware HD
Map Distribution in Het-SDVN
In Chapter 4, a hierarchical SDVN architecture with heterogeneous V2X (Het-SDVN) has
been proposed and demonstrated. Through dynamic collaborations between vehicles and twolayers of C-planes/SDVN controllers at RSUs and MEC/Cloud, respectively, cooperative perception (CP), the safety-critical V2X application which also adopts a three-layer architecture(CP client, local CP server, global CP server), are effectively enabled. High-definition (HD)map distribution can be regarded as an extension of CP. HD maps play an essential role inautomated driving systems. The 3D pointcloud data they include, contribute to precise localization at a centimeter level, environmental awareness beyond the field of view (FOV), andsafe maneuvering based on transient information like traffic signs and landmarks. The current
dilemma is that vehicles cannot store an entire end-to-end map due to the overwhelming data
size. These maps are expected to be partitioned to reduce their size and distributed to moving
vehicles over the air. Existing methods simply evenly cut map tiles and distribute them via
legacy Wi-Fi. From the author’s perspective, map partitioning and distribution need a more
intricate design that takes into account the vehicle’s speed and the network’s performance.
In this chapter, the author states the necessary conditions for map partition and proposes a
strategy to guarantee an overlap between partitions and optimize the overlap’s length to avoid
localization failure. To take advantage of Het-SDVN, the author develops a vehicle contextbased millimeter-wave (mmWave) map distribution system with roadside units (RSU) underedge/cloud orchestration. The proposed system has been demonstrated by road tests with
automated vehicles and Het-SDVN network infrastructures.

96

5.1

Motivation

The development of automated driving technologies is at a crucial moment. Consumers are
not merely satisfied with the Advanced Driver-Assistance Systems (ADAS). They pursue
higher-level, Level 3-5, automated driving systems as laid out by the Society of Automotive
Engineers (SAE) International [1]. To that end, an increasing number of sensors including
global navigation satellite system (GNSS) antennas, radars, cameras, light detection andranging (LiDAR), etc, are being mounted on vehicles to enhance different capabilities. The
creation of high-definition (HD) maps is one of the major functions enabled by this improved
vehicular sensing system.
HD maps are 3D maps that contain spatial and semantic information about the roadway
environment. These maps play a significant role in automated driving. It helps automatedvehicles (AV) achieve centimeter-level localization accuracy, perform fine-grained path planning, and mitigate the difficulty of contextual perception [74]. As shown in Fig. 5.1, thespatial information is presented in form of pointclouds measured by LiDARs, while the semantic information (e.g., road markings) is denoted by vector elements, a.k.a. vector maps.
In contrast to HD maps, navigational maps for human drivers are of meter-level accuracy
and mainly comprise semantic information [75], it is possible to download a city or state-wide
map at once over cellular networks (e.g., 221.4 MB for Tokyo [76]). However, the tremendous
amount of pointcloud data in HD maps can easily overwhelm the storage capacity of AVs.
It is reported that the total data volume of an HD map collected for one hour is about 1
TB [54].
Therefore, updating these maps, after partitioning, over air interfaces during automateddriving seems to be a means to unleash the issue of storage. This also paves the way for moreadvanced vehicle-to-everything (V2X) communications to support ultra-fast map distribution.
However, existing V2X technologies have well-known limitations in data rates. The dedicatedshort range communications (DSRC) at 5.9 GHz only supports a peak data rate of 27 Mbps,
which further degrades to 6 Mbps with high vehicle densities [47]. Although the long term
evolution (LTE) V2X makes certain improvements, its achievable data rate is still below 28.8
Mbps [17]. In 2020, new radio (NR) V2X was standardized in 3GPP Rel-16 to enhance thecommunication performance [77] but its commercialization schedule is hard to predict.
In this chapter, the author addresses the challenges above from two fronts: 1) Section 3.1
proposes an adaptive strategy in map partition to retain overlap to avoid localization failure

97

Figure 5.1: An example HD map containing pointcloud data and semantic information.
at boundaries due to the network delay and optimize map extents; 2) Section 3.2 designs anefficient map distribution system using millimeter wave (mmWave) under the orchestration ofedge/cloud servers. In addition, proof-of-concept trials in real driving scenarios are presented
in Section 3.3. The performances of the proposed system are comprehensively evaluated.

5.2

Design of mobility-aware map partition

In addition to existing map partition considerations, such as software file size limitations andpointcloud density standards, this section will be introducing considerations for the proposed
inclusion of a strategy to make sure that there is an overlap between partitioned HD maps
and optimize its roadway length. The guarantee of overlap is crucial. It works as a safety
feature to avoid localization failure, when utilizing scan matching, at map boundaries causedby network delay. Meanwhile, optimizing the overlap’s roadway length further extends thereach of the partitioned HD maps. Both of these measures ensure that the automated vehicle
will always have an HD map throughout its trip. To realize this, the proposed map partition
takes the vehicle’s speed and its relationship with two factors: the time it takes to connect
and download the HD map from the RSU into account.

98

5.2.1

Connection time

Beamforming (BF) is now widely applied in wireless communications to compensate for thesevere path loss between the transmitter and the receiver. Meanwhile, this technology introduces non-negligible overhead in the initial access to the networks. It needs to perform an
exhaustive search to locate a suitable communication path between the vehicles and RSUs.
The high mobility of vehicles would complicate this process and cause a connection delay after
entering the RSU coverage. This delay in connection with the RSU coupled with the vehicle’s speed becomes the initial distance consideration to determine if there remains sufficient
overlap and how long the boundaries of the partitioned map could be extended.

5.2.2

Download time

After connecting to the network, the vehicle would be able to download the map partition.
The speed at which it can download the map depends on the practical throughput/data rateof the RSU. The time it takes to fully download the HD map together with the vehicle’s speed
is an additional consideration in determining overlap length sufficiency and the partitioned
map’s roadway length.

5.2.3

Overlap optimization

Figure 5.2 illustrates the necessity of overlap optimization between contiguous map partitions.
The vehicle’s capacity to store or load HD maps for automated driving is still highly constrained, due to either hardware limitations or legacy issues of the software. Assuming thatthe vehicle can store an HD map with a length of l, when updating maps near the boundary,a part of the new map has to be reserved to create an overlapping area for stable localization.
Conventionally, the overlap starts at the edge of the RSU coverage, i.e., the initial accesspoint when the vehicle’s speed is equal to 0. Such a method of delineating overlap occupies
a considerable part of the new map and confines its reach because of the limits of vehicle
storage.
Hence, it is imperative to save the overlap area by taking the connection time (t1 ) anddownload time (t2 ) into account. If the vehicle’s speed v0 is known, the saved length fromoverlapping in the new map can be represented aslsaved = lc + ld + σ1 + σ2

(5.1)

99
where lc = d1 = v0 t1 indicates the passing distance for V2X connection after entering RSU
coverage, ld = d2 − d1 indicates the passing distance for map downloading which is inversely
proportional to V2X data rates. σ1 and σ2 denote the standard deviations of d1 and d2
respectively. A minimum overlapping length of σ1 + σ2 can be reserved to keep redundancy
and enhance the system robustness. In this way overlap creation becomes adaptive to thevehicle’s speed and V2X performance, saving overlapping areas and extending the reach of
the new map.

Conventional Overlap Reservation

Map(𝑇" ), Length=𝑙

Map(𝑇! ), Length=𝑙

RSU

VehicleV2X Connection Position(V=0 km/h)

Optimized Overlap
Map(𝑇! ), Length=𝑙Overlap Area
Map(𝑇" ), Length=𝑙

V=60 km/h
RSU

Vehicle𝑑# = 0𝑡!

𝑑! (V2X Connection)𝑡"𝑑" (Map download completion)

Figure 5.2: Illustration of overlap optimization.

100

5.3

Design of mmWave map distribution system

5.3.1

Prior arts

5.3.1.1

Map distribution system

The wireless distribution of HD maps, especially the part of point cloud data (PCD) maps, ischallenging considering the limitations of current mobile services and traditional V2X technologies. Shimada et al. categorized the elements of HD maps into four layers: 1) permanent
static data (map data); 2) transient static data (roadside infrastructure); 3) transient dynamic
data (congestion, signal phase); 4) highly dynamic data (vehicles, pedestrians) [78]. Of thefour layers, PCD maps belong to the first layer.
From top to bottom, the data volume and required data rate for transmission largely increase. As such, existing works mostly focus on the distribution of higher-layer map data in amanner called cooperative perception through V2X communications. These data are embedded in light V2X messages (e.g., CAM, DENM) and broadcasted by vehicles or roadside units(RSUs) at a low data rate using DSRC technologies. The typical systems include Tsukada
et. al’s Proxy CAM [79], AutoC2X [4] and Miucic et.al’s prototype system employing BSMbased cooperative perception [80]. In the works of Mizutani et al, they have evaluated the
distribution of PCD maps during automated driving [81] by showing that the vehicle successfully downloaded PCD maps over Wi-Fi at 60 Mbps in 1.16 s. However, the downloaded
maps were highly downsampled which can be ineffective for localization under extreme conditions. Besides, PCD maps need to assist other automated driving functions, such as semantic
segmentation, road boundary detection, etc, which require much higher resolutions. Hence,
new V2X technologies are being pursued to meet the demanding requirements for such map
distribution.
5.3.1.2mmWave V2X networks

The mmWave, as a promising technology for 5G and beyond, has been incorporated into
the frequency range of 5G NR (FR2: 24.5 GHz - 52.6 GHz). Owing to the rich unexploitedspectrum resource, mmWave can realize the new features of 5G systems including the ultrareliable and low latency communications (URLLC), enhanced mobile broadband (eMBB),
and massive machine type communication (mMTC). More so, commercialized mmWave technologies have shown their great potential for access and backhaul networks [82]. For example,

101IEEE 802.11ad (a.k.a WiGig) operating in the unlicensed 60 GHz band can allocate 2.16
GHz bandwidth for communications, which corresponds to a data rate of 6.75 Gbps [83]. Its
follow-up standard IEEE 802.11ay supports channel bonding and adds MIMO, thus the peakdata rate can go up to 176 Gbps [84].
As a result, mmWave has countless applications in this data-hungry era. One of themis V2X networks, which is an important application scenario considering the huge market
scale of the automotive industry. In V2X, mmWave is mainly applied to the safety applications [17]. Particularly for cooperative perception, mmWave empowers the exchanging of raw
sensory data among vehicles [85,86,87], between vehicles and road infrastructures [88,89,90],
or between vehicles and networks [91]. In super smart society, the deployment of RSUs withmmWave V2X capabilities is accelerating. Examples of relevant projects include the Lightpole
Site by Ericsson/Philipps [92], the LuxTurrim5G project in Finland [93] and the Super Smart
Society (SSS) project in Tokyo Tech [6]. Therefore, it is evident that mmWave can contributeto smooth automated driving by ultra-fast and low-latency distribution of HD maps over
V2X, which is not possible in previous efforts.

5.3.2

System design

The overview of the system is shown in Fig. 5.3. An intact HD map covering the end-toend traveling of automated vehicles is partitioned into regional maps. Excluding extreme
conditions (e.g., in mountains or jungles), we assume that most of these regions have infrastructures that provide wireless connectivity between vehicles and the network. During
automated driving, the vehicle proactively reports context information to remote map servers
over cellular networks. The context information includes the real-time position, velocity, destination, etc. Remote servers then analyze this information to determine the parameters of
map partition and select the most appropriate roadside unit (RSU) to cache the partitioned
maps. The vehicle will be notified with the map uniform resource locator (URL) and description of the selected RSU. Finally, the RSU opens the service port for access. Remote servers
keep monitoring the mmWave association between the vehicle and RSU to ensure successful
map distribution.
There are three major participants in this system. Each of them shall satisfy certain
functionalities to fulfill their tasks.
1) Automated vehicles: A complex, integrated and intelligent software system is the basis
for automated driving. Typically, there are five subsystems, i.e., sensing, localization, per-

102
context msg

Overlapping Area

Core/Internet

Context Data
Analysis

Dynamic Map
Partition

Association
MonitorRSU Selection& Caching

map url & rsu info
Edge/Cloud servers

Region C

Region B

Region A

Map

RSU

Cellular networks
RSU

LTE/5G NR Sub6

RSU
RSU

① Post context info (position,

velocity, destination, etc)

RSU mmWave (WiGig)② Get map URL and info ofservice-providing RSU③ Receive new map fromedge/cloud-specified RSU

Figure 5.3: System overview.

ception, planning, and control. Localization heavily relies on LiDAR scan matching withHD maps, especially in urban canyons where the performance of GNSS severely degrades
due to non-line of sight (NLOS). In this design, vehicle context information is uploaded toremote servers, for which a proxy application is needed to organize sensor data and set up
communication. To separate communications between the control plane and data plane, eachvehicle shall have two types of wireless interfaces. For the control plane which exchanges
context and response messages, cellular interfaces (LTE/5G NR Sub6) are applicable due totheir wide coverage. For the data plane that transmits HD maps with ultra-low latency, it isbelieved that the mmWave interface is a must. In addition, an efficient database/file system
is necessary for HD map updates.
2) Edge/Cloud servers: In this system, map servers manage the service centrally. There
are a variety of options to deploy these servers on the cloud, such as Amazon AWS [94],Microsoft Azure [95], Google Cloud [96], etc. Recently, activities on configuring mobile edge
computing (MEC) in the vicinity of base stations are growing [97,98,99,100]. It’s an alternative
to bringing map servers to the edge which is closer to vehicles. As such, not only is the103
uncertain delay that occurs in the backbone avoidable, the enormous service requests to thecloud can be offloaded. Nonetheless, regardless of locations, map servers need to accomplish
their duties, as depicted in Fig. 5.3. Received context messages shall be parsed in Context
Data Analysis, and sent to Dynamic Map Partition as key references. Map servers must beaware of the local RSU topology for RSU Selection & Caching. Association Monitor serves asecurity purpose, recording the status/statistics of map distribution through mmWave.
3) Roadside units: Existing standards have not clarified the physical image of an RSU,
although it is no longer a novel concept in the ITS. Therefore, any stationary infrastructure
entity (street lights, base stations, etc) can potentially become an RSU as long as they equipsensing, computing, and communication capabilities. To fulfill the requirements of this system, RSUs must enable mmWave for V2X communications. Corresponding software shallsupport large file transmission. To cache partitioned HD maps from edge/cloud servers, there
shall be enough storage and a high-speed backhaul, for which both optical fiber and mmWave
are applicable. It is also expected that RSUs can have a high penetration rate in futuresociety to improve the quality-of-service (QoS) of HD map delivery.

5.3.3

System implementation

Table 5.1: System hardware.
PurposeTest Vehicle

HardwareSpecifications

Robocar MV2

Vmax ≤ 60 km/h

Garmin GPS 18x USB

ACC < 3 m, 95%

RoboSense RS-LiDAR-32

Laser: 32 channels,
ACC ± 5 cm
Intel Core i9-10980XE,

Edge Server

Tsukumo Workstation

NVIDIA RTX 3090,

WA9J-H200/XT

MEM 128 GB, SSD 4 TB,
Ubuntu 18.04 LTS
Intel Core i5-1145G7,

RSUIntel NUC 11 Pro Mini PC

MEM 8 GB, SSD 500 GB,
Ubuntu 16.04 LTS

A full-fledged proof-of-concept system consisting of a test vehicle, an edge server, and104

Edge Server

Database
VehicleDatabase

Map Resource
Database

RSU
Database

Orchestration
Context
Overlap
RSU
Registry
Estimation
Selection
Partitioned map
Vehicle info
Restful API
Selected RSU
eth0

w0

Ethernet

LTE@2.5GHz

Map URL,
Context info
Selected RSU info
w0
HD Map Service
Manager
Information
Aggregator

USB

GNSS Driver
LiDAR Driver

Point Map
LoaderNDT Matching

New map

IEEE 802.11ad(WiGig)@60GHz

w1

HD Mapw0

HD MapCaching• Directory• Database
RSU Info
Proxy

HD Map Proxy

LiDAR

USB

HTTP ServerGPS

Automated Driving

Map Downloader

Position data

eth0

RSU

Test Vehicle

Figure 5.4: System implementation.

several RSUs has been implemented. Most of the hardware is commercial-off-the-shelf (COTS)
products. For instance, the test vehicle RoboCar MV2 [101] is a product of ZMP. The edge
server adopts Tsukomo’s workstation WA9J-H200/XT [102], built with state-of-the-art GPU(NVIDIA Geforce RTX 3090) and extensive storage (Memory 128 GB, SSD 4 TB). In addition,
Intel mini-PC NUC 11 Pro [103] is installed in RSUs as the processing unit. To localize thetest vehicle, a GPS receiver (GPS 18x USB [104]) and a 3D LiDAR (RS-LiDAR-32 [105]) are
mounted on its rooftop. The hardware mentioned above has been summarized in Table 5.1.
Figure 5.4 reveals the functional components for the system implementation. Some ofthem are based on open-source projects (Autoware [106], SQLite [107], Docker [108]) andcommonly used Python frameworks (Flask [109], HTTP [110]).
1) Context registry: This component receives context information from the test vehicle via
designed RESTful APIs and registers it into the vehicle database. Beforehand, vehicle position
is read from the GPS receiver and formatted by the GNSS driver provided in Autoware, which

105
is the world’s first open-source automated driving project. This position data is published ata frequency of 10 Hz. The information aggregator component adds other meta information
including the vehicle ID, velocity, and destination to the context and posts them to the Flask
server launched at the edge via the 2.5 GHz LTE.
2) Overlap estimation: This component calculates the size of roadway length that canbe saved to extend the boundaries of partitioned maps and makes sure that there remains
an overlap in between, as discussed in Section III. The vehicle velocity from the context
registry and the network status of RSUs are important inputs. Our ultimate target is todo the partitioning online, however, for simplicity, we have made a partition set M withregular parameter pairs (Vi , Rj ), where Vi is the velocities between 0 − 60 km/h with ∆v =10 km/h, while Rj is the achievable data rates referring to the modulation and coding scheme(MCS) table of the commercialized mmWave communication standard IEEE 802.11ad/WiGig
in the unlicensed 60 GHz band [111]. As such, from the real-time inputs (Vt , Rt ), the desiredpartition can be obtained by argmin(∥(Vt , Rt ) − (Vi , Rj )∥). The tool that has been used toM(i,j)

partition the HD maps for this work is CloudCompare [112], free and open-source softwarefor 3D point cloud processing.
3) RSU selection: This component selects the most suitable RSU to cache and delivers
new HD maps to the test vehicle. To master the physical and logical topology of local RSUs,
the edge server periodically sends inquiry messages to them over Ethernet networks. RSU info
proxy helps RSUs to respond to their status/statistics, which is also implemented by Flask.
The collected RSU information is stored as a graph with metadata in the RSU database. For
simplicity, this implementation selects the closest RSU near the region boundary along thedriving course for new map delivery. The new map is cached to the selected RSU using the
HTTP service.
4) Map distribution: The map distribution is managed by HTTP servers in RSUs. The
HTTP server maintains the cached HD maps and corresponding URLs. mmWave is usedas the transmission medium between the test vehicle and RSUs. For this implementation,
IEEE 802.11ad/WiGig is the best choice because there have been plenty of mature antenna
products compliant with this mmWave standard. The antennas we used are fabricated by
Panasonic, which supports beamforming and MCS up to 9, indicating a maximum data rateof 2.5 Gbps.
After the map downloader in the test vehicle completes its task, the point map loader ofAutoware will load the new map to replace the current map without disturbing the local-

106
ization, and continue automated driving smoothly. Furthermore, an outdoor experiment isconducted to demonstrate these functionalities.

5.4

Proof of concept and evaluation

5.4.1

mmWave ITS testbed

As shown in Fig. 5.5, a mmWave ITS testbed consisting of the described components in ourdesigned system is established in the Ookayama Area of Tokyo Institute of Technology. We
consider a scenario where the test vehicle equipped with LiDAR and mmWave antennas is
performing automated driving toward the target area (Midorigaoka Area). Along the course,
a RSU capable of mmWave V2X is deployed near the boundary between the two areas. Theedge server which manages the HD map distribution is located in one of the buildings in theuniversity, and is connected to the RSUs through the high-speed campus LAN and mmWave
backhaul networks.
mmWaveAntenna

LiDAR

Edge Server
Test Vehicle

mmWave Antenna

RSU
RSU

Target Area

Figure 5.5: Proof-of-concept scenario and testbed setup (overlaid on Google Earth Image(Mar. 13, 2019), Tokyo Tech, Japan).

107(a) Navigation in Ookayama Area.

(b) Download New (Partitioned) Map.

108(c) Navigation in Midorigaoka Area.

Figure 5.6: Functionality verification through real driving tests (vehicle velocity: 20 km/h,
map size of Midorigaoka area: 267 MB).

5.4.2

Verification

The system functionalities are verified by real driving tests heading to the Midorigaoka Area
from the Ookayama Area. Figure 5.6 shows the vehicle driving process visualized in RViz, aconvenient tool of Autoware for topic visualization in the 3D space. The grey point clouds
represent the loaded PCD map, and the red point clouds represent the point clouds thatare generated from real-time LiDAR scanning. These two point clouds are used by thenormal distributions transform (NDT) algorithm, an efficient localization method for smooth
automated driving. The vehicle’s velocity for this test is 20 km/h which complies with theallowable speed limit at the university. From (a) to (c), we are able to demonstrate thatthe test vehicle was able to move smoothly while successfully receiving the new partitioned
map over mmWave without localization failure and releasing space by deleting the old map
from storage. Note that the map used for the distribution test had a size of 267 MB. This
selection is brought on by Autoware’s limit of loading PCD maps with a size up to 1 GB. The
system becomes slower as the map size increases, an issue that can be resolved by improving
hardware performance.

109

5.4.3

Evaluation

In this subsection, the system performance is evaluated from the following three aspects basedon the results of field tests.
1) Optimized overlap length: As discussed in Section II, if the system is mobility-aware andhas good knowledge of the network status (from real-time or experience data), we can optimize
the length of overlaps while keeping the automated driving robust. Table 5.2 summarizes theresults obtained from multiple measurements. The three velocities we have tested, i.e., 10km/h, 20 km/h, and 30 km/h, follow the speed limit in the region. They are often regarded as
the common velocities in urban areas. It is found that the average connection time (t1 = 0.941
s, 1.052 s, 0.857 s) is almost the same regardless of the vehicle’s velocity. Hence, the connection
distance is simply proportional to their speed. The average download time (t2 ) depends on
the network performance, which needs to be jointly considered with the vehicle’s speed to
calculate the download distance. From the results, there is a clear trend that the optimized
overlap length is increasing as the vehicle’s speed rises, compared with which the incrementof the minimum overlapping length (σ1 + σ2 ) is insignificant.
Table 5.2: Overlap optimization results.
Velocity

Average Connection

Average Download

Optimized Length(km/h)

Distance (m)Distance (m)

(m)

10

2.61

5.47

8.08 ± 0.74

20

5.84

10.50

16.34 ± 2.57

30

7.14

16.41

23.55 ± 4.54

2) Map distribution data rate: The IEEE 802.11ad/WiGig has good robustness against
mobility impact [17]. Figure 5.7(a) presents the experimental data rates when the vehicle
moves at 10 km/h, 20 km/h, and 30 km/h respectively. The average data rates at thesevelocities are all approximately equal to 1.1 Gbps. The velocity of 0 km/h means that thevehicle is stopped at the edge of RSU coverage to download the HD map where the data rate
becomes relatively lower (565 Mbps on average). Theoretical data rates complying with IEEE
802.11ad/WiGig mmWave standard are also drawn in the figure. Since the MCS supportedby the installed mmWave antennas is [1, 9], the minimum data rate is supposed to be 350
Mbps, while the maximum data rate can reach 2.5 Gbps. Comparing theoretical values withthe experimental results shows that even if there is a gap in the peak data rate, performance

110
has still far surpassed the legacy WLAN and traditional V2X technologies.
3) Map distribution latency: The previous study uses Wi-Fi for HD map distribution [81].
To highlight the superiority of mmWave, we compare three types of radio access technologies:
2.4 GHz Wi-Fi, 5 GHz Wi-Fi, and 60 GHz mmWave, in terms of map download latency fromthe RSU. The map used has a file size of 267 MB. Figure 5.7(b) shows the results. The 2.4
GHz Wi-Fi takes the longest time to finish downloading (around 24 s) and an improvement isobserved in the performance of 5 GHz Wi-Fi, which shortens the time by about 5 s. But, incomparison to the two Wi-Fis, the 60 GHz mmWave has shown an overwhelming advantage.
As in most of our trials, it can distribute the map within less than 2 seconds. Therefore, it
is critical to apply mmWave. The more efficient the map distribution system becomes, thesmoother and safer the automated driving will be.

5.5

Conclusion

With Het-SDVN, we can brighten a future where automated vehicles can perform safe andsmooth end-to-end travel and enjoy HD map services with edge/cloud connectivity and resource orchestration. Before that, two questions are waiting to be answered: 1) how to makeappropriate map partitions under storage limitations and safety constraints, and 2) how tofacilitate map distribution with state-of-the-art communication technologies? For the former,
this chapter investigated two key factors in map partition and proposed an adaptive strategy
to retain an optimized overlap for successful localization at the boundary between maps. As
for the latter, this chapter introduced mmWave to the map distribution system and presented
our implementation process. This system successfully delivered the required HD map to thevehicle in field trials. The experimental results revealed that the effectiveness of our overlap
optimization becomes more significant as the vehicle’s velocity increases. It was also provedthat mmWave has overwhelming advantages in map distribution (data rate over 1 Gbps, andlatency of less than 2 s for a map of 267 MB) versus legacy Wi-Fi and traditional V2X
technologies. Future efforts on this topic will focus on improving the scalability of this map
distribution system, for which experiments will be carried out with diverse vehicle densities,
at higher vehicle velocities, and in various ITS environments (e.g., RSU layout).

111(a) Experimental data rate (mmWave) v.s. vehicle velocity(b) Comparison of map download latency v.s. legacy Wi-Fi

Figure 5.7: Evaluation results of mmWave map distribution system.

112

113

Chapter 6
Final Remarks and Future Work
6.1

Summary of the thesis• Chapter 1: It highlighted the significance of V2X in ITS and automated driving andshowcased advanced V2X applications that have not yet been commercialized. Afterrevealing the research target "cooperative perception", it underscored the limitations ofexisting V2X technologies and current vehicular networks. At the end, it summarized
the author’s efforts and contributions towards the research target.
• Chapter 2: It provided a comprehensive overview about cooperative perception including the classification and fusion techniques. It presented an in-depth discussion on data
type selection for cooperative perception, emphasizing the values of raw data sharing.
In the second part, it introduced the concept and principles of SDN, showing the great
benefits that SDN can potentially bring to the vehicular network. Finally, it pointed
out the challenges ahead of the successful transition from SDN to SDVN.
• Chapter 3: It discussed the required performances for the enhanced vehicular networkin order to achieve safe automated driving. Based on the discussion, a software-defined
dynamic mmWave V2X network was proposed for cooperative perception. Then, it established prototype systems with commercialized hardware equipment and open-sourced
software frameworks for indoor and outdoor proofs of concept (PoCs). In the indoor
PoC, the two modes of software-defined cooperative perception were implemented overmmWave V2V/V2I communications, showing that the indoor prototype system canachieve the network functions of the proposed architecture. By replacing some equip-

114
ment, the outdoor PoC was carried out in a real road environment. Under the dynamic
management of SDN controller, the cooperative perception, which distributed raw LiDAR pointcloud data from RSUs to the vehicle over mmWave in a U-bend turning
scenario, was realized. These PoCs verified mmWave performance, the feasibility of the
proposed network architecture, and the safety benefit of cooperative perception.
• Chapter 4: In order to effectively support various types of cooperative perception (CP)and ensure this safety-critical V2X application is deliverable to vehicles anywhere andanytime, it proposed Het-SDVN, an SDN-based V2X network architecture, consisting of
hierarchical C-planes, geographically distributed but logically centralized sub-SDVNs,
and heterogeneous V2X. The network roles were clearly defined. For CP in local SDVNs, communications are orchestrated by RSUs with local SDVN controllers; for CP
in other SDVNs or infrastructure-rare regions, communications for crossing sub-SDVNs
are orchestrated by global SDVN controllers at MEC/Cloud servers. To empower the
RSUs with SDVN functions, it presented a framework that integrates ROS, Ryu, andOVS and adds customized functional components for application, network, and sensor
management. This framework was deployed and tested, showing that a maximum powersaving ratio of 60% was achieved with the CAM-triggered sensor ON/OFF strategy, andthe delay for CP preparation was less than 30 ms. Eventually, the holistic Het-SDVN
was set up in Tokyo Institute of Technology, and its support for CP has been demonstrated via a proof of concept. Evaluation results validated the feasibility of Wi-Fi as
local C-planes and stressed the importance of mmWave and the necessity of interference
management when performing raw sensor data sharing. Moreover, no significant difference in flow installation performance was observed between the local and global SDVN
controllers over different networks. They were all under 3 ms, showing the potential tomeet the latency requirement (less than 10 ms) of advanced V2X applications [7].
• Chapter 5: It created a new service of HD map distribution for future fully automatedvehicles by taking advantage of the designed Het-SDVN architecture. With the global
C-plane, the map distribution was mobility-aware, and the HD map could be properly
partitioned to optimize the size of overlap. To facilitate the map distribution, according
to the vehicle position, the partitioned map was cached to the nearest RSU and thendownloaded by the vehicle through the mmWave V2X network. The field trial showedthat the vehicle, MEC/Cloud server, and RSUs formed an interactive and collaborative

115
system that successfully delivered the demanded map to the vehicle without disturbing
its localization. This implementation also demonstrated the capability of the proposed
Het-SDVN to foster novel V2X applications other than cooperative perception.
Moreover, the unique contributions and added values of Chapter 3 and Chapter 4 works,compared to the existing SDVN implementation and cooperative perception studies, can beshowcased through Table 6.1 and Table 6.2.
Table 6.1: Comparisons with existing SDVN implementation studies.
State of the art

Ref. [113]

Ref. [28]Ref. [29]Ref. [30]

Chapter 3

Chapter 4

Architecture

Fully centralized

Fully centralized

Fully centralized

Hierarchical

Fully centralized

HierarchicalD-plane RATIEEE 802.11p

LTE & Wi-Fi

Wi-Fi

Wi-Fi

WiGig

WiGig & Wi-Fi

Target V2X

Basic

Basic

Basic

Basic

Advanced

Advanced & Basic

service

[12]

[12]

[12]

[12]

[7]

[12,7]

Context, access,Context, access,Controller
function

Load balance

RAT & Mobility
mgmt.

Access mgmt.

Mobility mgmt.

routing mgmt.

routing, RAT mgmt.

Evaluation

Small-scale

Small-scale

Small-scale

Small-scale

Large-scale test

Large-scale test

methodemulationemulation

testbedtestbed

field

field

Tool

NS-3 SUMO

Mininet-WiFi

Raspberry Pi

Zodiac FX

Vehicles & RSUs

Vehicles & RSUs

Table 6.2: Comparisons with existing cooperative perception studies.
State of the art

Ref. [114]Ref. [90]Ref. [85]Ref. [86]Chapter 3

Chapter 4
Point clouds & Objects

Data typeObjects

Objects

Features

Point clouds

Point clouds

Data rate requirement

Low

Low

Middle

High

High

High

Implemented RATIEEE 802.11p

IEEE 802.11p

WiGig

N/A

WiGig

WiGig & Wi-Fi

Transmission mode

Broadcast

Broadcast

Broadcast

N/A

Relay

Relay & Broadcast

V2I

V2I

V2V

N/A

V2I & V2N

V2V, V2I, V2N

Network controller

None

None

None

None

SDVN controller

SDVN controller

Contribution

SystemFusionFusionDetection

System & Network

System & Network

solutionsolution

Communication
method

6.2

Suggested future works

In the future, large-scale simulations and evaluations are needed to fill in the remaining gaps
between the proposals and experiments. Moreover, the proposed systems and technologies
have the potential to make significant contributions to several promising fields. We plan tomake some explorations in the following fields.

116

6.2.1

Het-SDVN for construction of mobility digital twin

Digital twin (DT) is considered a key application of beyond 5G and 6G. Constructing amobility DT needs to collect a huge amount of sensing information from the massive sensors
distributed in the environment to the DT server located at MEC/Cloud. It is similar to therequirement of cooperative perception, which collects the sensing data to the vehicle. Hence,the proposed Het-SDVN provides flexibility and programmability for establishing various
mobility DT, including car sharing, smart intersections, etc. In addition to the radio resource
management we mainly focused on in Chapter 3 and 4, the capability of SDVN controllers
can be further extended to manage the computing and storage resources distributed in thevehicular network. For instance, to quickly construct a mobility DT and provide the DT
service, the global SDVN controller in Het-SDVN can select an edge server to host the DT,
and then work with local SDVN controllers to route the required sensing data to that server.

Figure 6.1: Het-SDVN for constructing mobility digital twins.

6.2.2

Het-SDVN for scalable and dynamic HD map distribution

In Chapter 5, we demonstrated the application of mobility-aware HD map distribution in
the Het-SDVN. Although the mmWave V2I achieved ultra-fast map delivery from the RSU
to the vehicle and ensured the vehicle’s stable localization, however, if the vehicle number
continuously increases, the allocated bandwidth for each vehicle will reduce, resulting in a117higher HD map download latency and affecting the localization performance. To overcome
this challenge, we will investigate the clustering strategy for vehicles that request the same
map. We will also need to select the optimal vehicle in the cluster as the head, to receive theHD map from the RSU first, then distribute the map within the cluster via mmWave V2V
communications, as shown in Fig. 6.2. The clustering function can be deployed at the HetSDVN local C-plane. We can also study other issues that relate to this HD map distribution
system, such as inter-vehicle interference control, RAT handover, etc.

Figure 6.2: Het-SDVN-based HD map distribution system.

6.2.3

Path planning assisted by cooperative perception

With the proposed Het-SDVN, cooperative perception can be extensively supported. As we
indicated in Sect. 1.2, many other applications will benefit from the cooperative perception
data. Figure 6.3 shows our imagined future path-planning system. The vehicle’s path planningconsists of two phases: global path planning and local path planning. Before the global path
planning, with the help of the global SDVN controller, the detected traffic of RSUs can besent to the vehicle in the form of CPM for analysis and generating an optimal route. Whenthe vehicle performs local planning in a sub-SDVN network, the local SDVN controller cancoordinate the transmission of raw sensor data for fine-grained maneuvering and trajectory

118
control. It will increase the safety and efficiency of traditional path planning without V2X
information. Proofs of concept should be conducted to demonstrate this system.

Figure 6.3: Cooperative perception-assisted path planning system.

119

Appendix I
List of Publications
I.1

Journal papers• Z. Li, T. Yu, R. Fukatsu, G. K. Tran and K. Sakaguchi, "Towards Safe AutomatedDriving: Design of Software-Defined Dynamic MmWave V2X Networks and PoC Implementation," in IEEE Open Journal of Vehicular Technology, vol. 2, pp. 78-93, 2021.
• Z. Li, K. Wang, T. Yu and K. Sakaguchi, "Het-SDVN: SDN-Based Radio Resource
Management of Heterogeneous V2X for Cooperative Perception," in IEEE Access, vol.
11, pp. 76255-76268, 2023.

I.2

Journal papers not related to this thesis• J. Nakazato, M. Nakamura, T. Yu, Z. Li, K. Maruta, G. K. Tran and K. Sakaguchi,"Market Analysis of MEC-Assisted Beyond 5G Ecosystem," in IEEE Access, vol. 9, pp.
53996-54008, 2021.
• Y. Yin, H. Chen, Z. Li, T. Yu and K. Sakaguchi. "ZigZag Antenna Configuration for
MmWave V2V with Relay in Typical Road Scenarios: Design, Analysis and Experiment." IEICE Transactions on Communications, vol. E104-B, no.10, pp.1307-1317,
Oct. 2021.
• J. Nakazato, Z. Li, K. Maruta, K. Kubota, T. Yu, G. K. Tran, K. Sakaguchi and S.
Masuko. “MEC/Cloud Orchestrator to Facilitate Private/Local Beyond 5G with MEC
and Proof-of-Concept Implementation,” MPDI Sensors, Vol. 22, No. 14, July 2022.

120

I.3

International conferences• Z. Li, T. Yu, R. Fukatsu, G. K. Tran and K. Sakaguchi, "Proof-of-Concept of a SDN
Based mmWave V2X Network for Safe Automated Driving," 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA, pp. 1-6, 2019.
• Z. Li, M. L. R. Lagahit, M. Matsuoka and K. Sakaguchi, "Design of Mobility-Aware
Map Partition and Distribution System for Smooth Automated Driving," 2022 IEEE
33rd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Kyoto, Japan, pp. 628-634, 2022.
• Z. Li, T. Yu, T. Suzuki and K. Sakaguchi, "Building an SDVN Framework for RSUCentric Cooperative Perception with Heterogeneous V2X," 2023 IEEE 20th Consumer
Communications & Networking Conference (CCNC), Las Vegas, NV, USA, pp. 1-7,
2023.

I.4

International conferences not related to this thesis• J. Nakazato, M. Nakamura, T. Yu, Z. Li, G. K. Tran and K. Sakaguchi, "Design of MEC
5G Cellular Networks: Viewpoints from Telecom Operators and Backhaul Owners,"
2020 IEEE International Conference on Communications Workshops (ICC Workshops),
Dublin, Ireland, pp. 1-6, 2020.
• Y. Yin, H. Chen, Z. Li, R. Fukatsu, T. Yu and K. Sakaguchi, "Design of Antenna
Configuration for Interference Control in MmWave V2V Communication Systems," 2020
IEEE 92nd Vehicular Technology Conference (VTC2020-Fall), Victoria, BC, Canada,
pp. 1-5, 2020.
• K. Maruta, M. Takizawa, R. Fukatsu, Y. Wang, Z. Li and K. Sakaguchi, "Blind-Spot
Visualization via AR Glasses using Millimeter-Wave V2X for Safe Driving," 2021 IEEE
94th Vehicular Technology Conference (VTC2021-Fall), Norman, OK, USA, pp. 1-5,
2021.
• K. Wang, Z. Li, T. Yu and K. Sakaguchi, "Smart Mobility Digital Twin for AutomatedDriving: Design and Proof-of-Concept," 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring), Florence, Italy, pp. 1-6, 2023.

121• M. L. R. Lagahit, Z. Li, K. Sakaguchi and and M. Matsuoka, "Exploring Ground
Segmentation From LiDAR Scanning-Derived Images Using Convolutional Neural Networks," Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XLVIII-1/W1-2023,
221–226, 2023.

I.5

Domestic conferences• J. Nakazato, M. Nakamura, T. Yu, Z. Li, G. K. Tran and K. Sakaguchi, “Design of
MEC Cellular Networks: Viewpoints from Telecom Operator and Backhaul Owner,”
IEICE RCS Technical Report, Dec. 2019.
• J. Nakazato, Z. Li, K. Maruta and K. Sakaguchi, “Viewpoint from Local Telecom
Operators and Cloud Owners,” IEICE Technical Report, RCS, Mar. 2021.
• Z. Li, T. Yu, T. Suzuki and K. Sakaguchi, "Building an SDVN Framework for RSUCentric Cooperative Perception with Heterogeneous V2X," 2023 RCS Mobile Communication Workshop, pp. 1-7, Mar. 2023.

122

123

Appendix II
Abbreviations
3GPP

3rd Generation Partnership ProjectIEEE

Institute of Electrical and Electronics Engineers

ETSI

European Telecommunications Standards InstituteSAE

Society of Automotive Engineers

ISO

International Organization for Standardization

ITSIntelligent Transportation System

ONF

Open Network Foundation

EEBL

Electronic Emergency Brake Light

FCW

Foward Collision Warning

BSW/LCW

Blind Spot Warning/Lane Change Warning

GLOSA

Green Light Optimal Speed Advisory

EVW

Emergency Vehicle Warning

VNFP

Vehicle Near-Field Payment

ETC

Electronic Toll collection

IVI

In-Vehicle Infotainment

V2X

Vehicle-to-Everything Communication

V2V

Vehicle-to-Vehicle Communication

V2I

Vehicle-to-Infrastructure Communication

V2P

Vehicle-to-Pedestrian Communication

V2N

Vehicle-to-Network Communication

CDA

Cooperative Driving Automation

QoS

Quality of Service

124
CP

Cooperative Perception/Collective Perception

CAM

Cooperative Awareness Message

CPM

Collective Perception Message

DENM

Decentralized Environmental Notification Message

MCM

Maneuver Coordination Message

DSRC

Dedicated short-range communications

LTE

Long-Term Evolution

NR

New Radio

mmWaveMillimeter Wave

THz

Terahertz

VLC

Visible Light Communication

WAVE

Wireless Access in Vehicular Environments

MIMO

Multiple-Input and Multiple-Output

OFDM

Orthogonal Frequency Division Multiplexing

OCB

Outside of the Context of BS

EDCA

Enhanced Distributed Channel Access

HARQ

Hybrid Automatic Repeat Request

LDPC

Low-Density Parity-Check

VRU

Vulnerable Road User

RSU

Roadside Unit

OBU

Onboard Unit

CAV

Connected and Automated Vehicle

VANET

Vehicular Ad-Hoc Network

C-V2X

Cellular Vehicle-to-Everything Communication

SDN

Software Defined Networking

SDWN/SWMN

Software Defined Wireless Network/Software Defined Mobile Network

SDVN

Software Defined Vehicular Network

C-planeControl Plane

D-plane

D-Plane

NBI

Northbound Interface

SBI

Southbound Interface

WBI/EBI

Westbound Interface/Eastbound Interface

MEC

Multi-Access Edge Computing

125
NFV

Network Functions Virtualization

RAT

Radio Access Technology

RAN

Radio Access Network

RRM

Radio Resource Management

GNSS

Global Navigation Satellite System

GPS

Global Positioning System

IMU

Inertial Measurement Unit

LiDAR

Light Detection and Ranging

FoV

Field of View

LOS/NLOS

Line-Of-Sight/Non-Line-Of-Sight

RSSI

Radio Signal Strength Indicator

RTT

Round-Trip Time

TCP

Transmission Control Protocol

UDP

User Datagram Protocol

HTTP

Hypertext Transfer Protocol

PoC

Proof of Concept

COTS

Commercial Off-The-Shelf

ROS

Robot Operating System

OVSOpen Virtual Switch

ODL

OpenDaylight

SLAM

Simultaneous Localization and Mapping

NDT

Normal Distributions Transform

HD

High Definition

WLAN

Wireless Local Area Network

DT

Digital Twin

126

127

References[1] SAE J3016. Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle
Automated Driving Systems. Jun. 2018.
[2] SAE J3216. Taxonomy and Definitions for Terms Related to Cooperative Driving Automation for On-Road Motor Vehicles. Jul. 2021.
[3] ETSI TR 103-562 (V2.1.1). Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Analysis of the Collective Perception Service (CPS);
Release 2, Dec. 2019.
[4] Manabu Tsukada, Takaharu Oi, Akihide Ito, Mai Hirata, and Hiroshi Esaki. AutoC2X:
Open-source software to realize V2X cooperative perception among autonomous vehicles. In 2020 IEEE 92nd Vehicular Technology Conference (VTC2020-Fall), pp. 1–6,
2020.
[5] RS-LiDAR-32 User Manual.

https://www.robosense.ai/en/rslidar/RS-LiDAR-32.
[6] SSS Promotion Consortium. https://www.sss.e.titech.ac.jp/.
[7] 3GPP TR 22.886 (V16.2.0). Study on enhancement of 3GPP support for 5G V2X
services, Dec. 2018.
[8] ISO/TC 204. ITS Standardization Activities of ISO/TC 204, Oct. 2019.
[9] SAE J2945/1. On-Board Minimum Performance Requirements for V2V Safety Systems,
Mar. 2016.

128[10] China SAE T/China SAE 53-2017. Cooperative Intelligent Transportation System; Vehicular Communication; Application Layer Specification and Data Exchange Standard,
Sept. 2017.
[11] ETSI TR 102 638 (V1.1.1). Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Definitions, Jun. 2009.
[12] 3GPP TR 22.885 (V14.0.0). Study on LTE support for Vehicle-to-Everything (V2X)
services, Dec. 2015.
[13] ETSI TR 103 578 (0.0.12 Draft). Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Definitions, Jun. 2023.
[14] ETSI TS 103 324 (V2.1.1). Intelligent Transport System (ITS); Vehicular Communications; Basic Set of Applications; Collective Perception Service; Release 2, Jun. 2023.
[15] ETSI TS 102 637-2 (V1.2.1). Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Part 2: Specification of Cooperative Awareness
Basic Service, Mar. 2011.
[16] ETSI TS 102 637-3 (V1.1.1). Intelligent Transport Systems (ITS); Vehicular Communications; Basic Set of Applications; Part 3: Specifications of Decentralized Environmental
Notification Basic Service, Sept. 2010.
[17] Kei Sakaguchi, Ryuichi Fukatsu, Tao Yu, Eisuke Fukuda, Kim Mahler, Robert Heath,
Takeo Fujii, Kazuaki Takahashi, Alexey Khoryaev, Satoshi Nagata, and Takayuki
Shimizu. Towards mmWave V2X in 5G and Beyond to Support Automated Driving.
IEICE Transactions on Communications, Vol. advpub, p. 2020EBI0001, 2020.
[18] Kazuki Maruta, Miyuu Takizawa, Ryuichi Fukatsu, Yue Wang, Zongdian Li, and Kei
Sakaguchi. Blind-Spot Visualization via AR Glasses using Millimeter-Wave V2X for
Safe Driving. In 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall),pp. 1–5, 2021.
[19] ETSI TS 102 636-4-2 (V1.4.1). Intelligent Transport Systems (ITS); Vehicular Communications; GeoNetworking; Part 4: Geographical addressing and forwarding for pointto-point and point-to-multipoint communications; Sub-part 2: Media-dependent functionalities for ITS-G5, Feb. 2021.

129[20] Ryuichi Fukatsu and Kei Sakaguchi. Automated Driving with Cooperative Perception
Using Millimeter-Wave V2V Communications for Safe Overtaking. Sensors, Vol. 21,
No. 8, 2021.
[21] Ryuichi Fukatsu and Kei Sakaguchi. Automated Driving with Cooperative Perception Based on CVFH and Millimeter-Wave V2I Communications for Safe and Efficient
Passing through Intersections. Sensors, Vol. 21, No. 17, 2021.
[22] Song Wang, Jingqi Huang, and Xinyu Zhang. Demystifying Millimeter-Wave V2X: Towards Robust and Efficient Directional Connectivity under High Mobility. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking,
MobiCom ’20, New York, NY, USA, 2020. Association for Computing Machinery.
[23] Junhyeong Kim, You-Jun Choi, Gosan Noh, and Heesang Chung. On the feasibility of
remote driving applications over mmwave 5g vehicular communications: Implementation and demonstration. IEEE Transactions on Vehicular Technology, Vol. 72, No. 2,
pp. 2009–2023, 2023.
[24] Yue Yin, Tao Yu, Kazuki Maruta, and Kei Sakaguchi. Distributed and Scalable Radio Resource Management for mmWave V2V Relays towards Safe Automated Driving.
Sensors, Vol. 22, No. 1, 2022.
[25] Nelson Cardona, Estefanía Coronado, Steven Latré, Roberto Riggio, and Johann M.
Marquez-Barja. Software-Defined Vehicular Networking: Opportunities and Challenges.
IEEE Access, Vol. 8, pp. 219971–219995, 2020.
[26] Xiaohu Ge, Zipeng Li, and Shikuan Li. 5G Software Defined Vehicular Networks. IEEE
Communications Magazine, Vol. 55, No. 7, pp. 87–93, 2017.
[27] Zongjian He, Jiannong Cao, and Xuefeng Liu. SDVN: enabling rapid network innovation
for heterogeneous vehicular communication. IEEE Network, Vol. 30, No. 4, pp. 10–15,
2016.
[28] Ramon Dos Reis Fontes, Claudia Campolo, Christian Esteve Rothenberg, and Antonella Molinaro. From Theory to Experimental Evaluation: Resource Management in
Software-Defined Vehicular Networks. IEEE Access, Vol. 5, pp. 3069–3076, 2017.

130[29] Gokhan Secinti, Berk Canberk, Trung Q. Duong, and Lei Shu. Software Defined Architecture for VANET: A Testbed Implementation with Wireless Access Management.
IEEE Communications Magazine, Vol. 55, No. 7, pp. 135–141, 2017.
[30] Ousmane Sadio, Ibrahima Ngom, and Claude Lishou. Design and Prototyping of a
Software Defined Vehicular Networking. IEEE Transactions on Vehicular Technology,
Vol. 69, No. 1, pp. 842–850, 2020.
[31] Michael Aeberhard and Nico Kaempchen. High-level sensor data fusion architecture
for vehicle surround environment perception. In Proceedings of the 8th International
Workshop on Intelligent Transportation, pp. 1–6, Mar. 2011.
[32] Tomoya Kitazato, Manabu Tsukada, Hideya Ochiai, and Hiroshi Esaki. Proxy cooperative awareness message: an infrastructure-assisted V2V messaging. In 2016 Ninth
International Conference on Mobile Computing and Ubiquitous Networking (ICMU),
pp. 1–6, 2016.
[33] Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. F-cooper: Feature
based cooperative perception for autonomous vehicle edge computing system using 3d
point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing,
SEC ’19, p. 88–100, New York, NY, USA, 2019. Association for Computing Machinery.
[34] Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm:Communication-Efficient Collaborative Perception via Spatial Confidence Maps. In
Neurips 2022, pp. 1–6, Nov. 2022.
[35] Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. Cooperative
perception for 3d object detection in driving scenarios using infrastructure sensors. IEEE
Transactions on Intelligent Transportation Systems, Vol. 23, No. 3, pp. 1852–1864, 2022.
[36] Xumiao Zhang, Anlan Zhang, Jiachen Sun, Xiao Zhu, Y. Ethan Guo, Feng Qian, and
Z. Morley Mao. Emp: Edge-assisted multi-vehicle perception. MobiCom ’21, p. 545–558,
New York, NY, USA, 2021. Association for Computing Machinery.
[37] Seong-Woo Kim, Baoxing Qin, Zhuang Jie Chong, Xiaotong Shen, Wei Liu, Marcelo H.
Ang, Emilio Frazzoli, and Daniela Rus. Multivehicle cooperative driving using cooperative perception: Design and experimental validation. IEEE Transactions on Intelligent
Transportation Systems, Vol. 16, No. 2, pp. 663–680, 2015.

131[38] Nick McKeown, Tom Anderson, Hari Balakrishnan, Guru Parulkar, Larry Peterson,
Jennifer Rexford, Scott Shenker, and Jonathan Turner. Openflow: Enabling innovationin campus networks. Vol. 38, No. 2, p. 69–74, mar 2008.
[39] Open Network Foundation (ONF). Software-Defined Networking: The New Norm for
Networks, Apr. 2012.
[40] Salvatore Costanzo, Laura Galluccio, Giacomo Morabito, and Sergio Palazzo. Software
defined wireless networks: Unbridling sdns. In 2012 European Workshop on Software
Defined Networking, pp. 1–6, 2012.
[41] Tao Chen, Marja Matinmikko, Xianfu Chen, Xuan Zhou, and Petri Ahokangas. Software
defined mobile networks: concept, survey, and research directions. IEEE Communications Magazine, Vol. 53, No. 11, pp. 126–133, 2015.
[42] Gia Khanh Tran, Ricardo Santos, Hiroaki Ogawa, Makoto Nakamura, Kei Sakaguchi,
and Andreas Kassler. Context-based dynamic meshed backhaul construction for 5g
heterogeneous networks. Journal of Sensor and Actuator Networks, Vol. 7, No. 4, 2018.
[43] D. Wakabayashi. Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots
Roam. The New York Times, Mar. 2018.
[44] Mate Boban, Apostolos Kousaridas, Konstantinos Manolakis, Josef Eichinger, and Wen
Xu. Connected Roads of the Future: Use Cases, Requirements, and Design Considerations for Vehicle-to-Everything Communications. IEEE Vehicular Technology Magazine,
Vol. 13, No. 3, pp. 110–123, 2018.
[45] Cisco. Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update,2016-2021. White Paper, Feb. 2017.
[46] ETSI EN 302 663 (V1.2.0). Intelligent Transport Systems (ITS); Access layer specification for Intelligent Transport Systems operating in the 5 GHz frequency band, Nov.
2012.
[47] John B. Kenney. Dedicated Short-Range Communications (DSRC) Standards in the
United States. Proceedings of the IEEE, Vol. 99, No. 7, pp. 1162–1182, 2011.
[48] 3GPP TR 21.914 (V14.0.0). Summary of Rel-14 Work Items: Vehicle-to-Everything(V2X) related item, May 2018.

132[49] Elisabeth Uhlemann. Connected-Vehicles Applications Are Emerging [Connected Vehicles]. IEEE Vehicular Technology Magazine, Vol. 11, No. 1, pp. 25–96, 2016.
[50] Gaurang Naik, Biplav Choudhury, and Jung-Min Park. IEEE 802.11bd & 5G NR V2X:
Evolution of Radio Access Technologies for V2X Communications. IEEE Access, Vol. 7,
pp. 70169–70184, 2019.
[51] Ammara Anjum Khan, Mehran Abolhasan, and Wei Ni. 5G next generation VANETs
using SDN and fog computing framework. In 2018 15th IEEE Annual Consumer Communications & Networking Conference (CCNC), pp. 1–6, 2018.
[52] Lionel Nkenyereye, Lewis Nkenyereye, S. M. Riazul Islam, Chaker Abdelaziz Kerrache,
M. Abdullah-Al-Wadud, and Atif Alamri. Software Defined Network-Based MultiAccess Edge Framework for Vehicular Networks. IEEE Access, Vol. 8, pp. 4220–4234,
2020.
[53] Ramon R. Fontes, Samira Afzal, Samuel H. B. Brito, Mateus A. S. Santos, and Christian Esteve Rothenberg. Mininet-WiFi: Emulating software-defined wireless networks.
In 2015 11th International Conference on Network and Service Management (CNSM),
pp. 384–389, 2015.
[54] Heiko G. Seif and Xiaolong Hu. Autonomous Driving in the iCity—HD Maps as a Key
Challenge of the Automotive Industry. ELSEVIER, Engineering, Vol. 2, No. 2, pp.
159–162, 2016.
[55] Ryuichi Fukatsu and Kei Sakaguchi. Millimeter-Wave V2V Communications with Cooperative Perception for Automated Driving. In 2019 IEEE 89th Vehicular Technology
Conference (VTC2019-Spring), pp. 1–5, 2019.
[56] Swetank Kumar Saha, Hany Assasa, Adrian Loch, Naveen Muralidhar Prakash, Roshan
Shyamsunder, Shivang Aggarwal, Daniel Steinmetzer, Dimitrios Koutsonikolas, Joerg
Widmer, and Matthias Hollick. Fast and Infuriating: Performance and Pitfalls of 60
GHz WLANs Based on Consumer-Grade Hardware. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 1–9,
2018.
[57] 3GPP TS 21.886 (V16.2.0). Enhancement of 3GPP support for V2X scenarios, Jun.
2019.

133
[58] Liehuang Zhu, Md M. Karim, Kashif Sharif, Chang Xu, Fan Li, Xiaojiang Du, andMohsen Guizani. Sdn controllers: A comprehensive analysis and performance evaluation
study. ACM Comput. Surv., Vol. 53, No. 6, dec 2020.
[59] Md. Noor-A-Rahim, Zilong Liu, Haeyoung Lee, Mohammad Omar Khyam, Jianhua
He, Dirk Pesch, Klaus Moessner, Walid Saad, and H. Vincent Poor. 6G for Vehicle-toEverything (V2X) Communications: Enabling Technologies, Challenges, and Opportunities. Proceedings of the IEEE, Vol. 110, No. 6, pp. 712–734, 2022.
[60] Daniele Pirrone, Antonio Ferraro, Dimitrios C. Zografopoulos, Walter Fuscaldo, Pascal
Szriftgiser, Guillaume Ducournau, and Romeo Beccherelli. Metasurface-Based Filters
for High Data Rate THz Wireless Communication: Experimental Validation of a 14
Gbps OOK and 104 Gbps QAM-16 Wireless Link in the 300 GHz Band. IEEE Transactions on Wireless Communications, Vol. 21, No. 10, pp. 8688–8697, 2022.
[61] Yuki Goto, Isamu Takai, Takaya Yamazato, Hiraku Okada, Toshiaki Fujii, Shoji
Kawahito, Shintaro Arai, Tomohiro Yendo, and Koji Kamakura. A New Automotive
VLC System Using Optical Communication Image Sensor. IEEE Photonics Journal,
Vol. 8, No. 3, pp. 1–17, 2016.
[62] Starlink Specifications - Mobile Service Plan.

https://www.starlink.com/legal/documents/DOC-1002-69942-69.
[63] Cohda MK6 RSU.

https://www.cohdawireless.com/solutions/hardware/mk6-r

su/.
[64] Moyukh Laha and Raja Datta. Intelligent On/Off Switching of mmRSUs in Urban
Vehicular Networks: A Deep Q-Learning Approach. In 2022 National Conference on
Communications (NCC), pp. 221–226, 2022.
[65] Buffalo WAPM-1266WDPR Specification.

https://www.buffalo.jp/product/det

ail/wapm-1266wdpr.html.
[66] Sven Laux, Gurjashan Singh Pannu, Stefan Schneider, Jan Tiemann, Florian Klingler,
Christoph Sommer, and Falko Dressler. Demo: OpenC2X — An open source experimental and prototyping platform supporting ETSI ITS-G5. In 2016 IEEE Vehicular
Networking Conference (VNC), pp. 1–2, 2016.

134[67] RoboCar MV2. https://www.zmp.co.jp/products/robocar/robocar-mv2.
[68] Autoware Fundation. https://www.autoware.org/.
[69] ROS2 Foxy. https://docs.ros.org/en/foxy/.
[70] Autoware Universe Document - lidar_centerpoint.

https://autowarefoundation.g

ithub.io/autoware.universe/main/perception/lidar_centerpoint/.
[71] Ryu SDN Framework. https://ryu.readthedocs.io/en/latest/.
[72] Yue Yin, Haoze Chen, Zongdian Li, Tao Yu, and Kei Sakaguchi. Zigzag antenna configuration for mmWave v2v with relay in typical road scenarios: Design, analysis andexperiment. IEICE Transactions on Communications, Vol. 104, pp. 1307–1317, 2021.
[73] James M. Anderson, Nidhi Kalra, Karlyn D. Stanley, Paul Sorensen, Constantine Samaras, and Tobi A. Oluwatola. Autonomous Vehicle Technology: A Guide for Policymakers. RAND Corporation, Santa Monica, CA, 2016.
[74] J. Tang and S. Liu. Edge High-Precise Map Service for Autonomous Driving. ZTE
Technology Journal, Vol. 25, No. 3, Jun. 2019.
[75] Google Maps. https://cloud.google.com/.
[76] OpenStreetMap: Japan. https://download.geofabrik.de/asia/japan.html.
[77] Mario H. Castañeda Garcia, Alejandro Molina-Galan, Mate Boban, Javier Gozalvez,
Baldomero Coll-Perales, Taylan Şahin, and Apostolos Kousaridas. A Tutorial on 5G
NR V2X Communications. IEEE Communications Surveys Tutorials, Vol. 23, No. 3,
pp. 1972–2026, 2021.
[78] Hideki Shimada, Akihiro Yamaguchi, Hiroaki Takada, and Kenya Sato. Implementation and Evaluation of Local Dynamic Map in Safety Driving Systems. Journal ofTransportation Technologies, Vol. 5, pp. 102–112, 2015.
[79] Tomoya Kitazato, Manabu Tsukada, Hideya Ochiai, and Hiroshi Esaki. Proxy cooperative awareness message: an infrastructure-assisted V2V messaging. In 2016 Ninth
International Conference on Mobile Computing and Ubiquitous Networking (ICMU),
pp. 1–6, 2016.

135[80] Radovan Miucic, Ashish Sheikh, Zeljko Medenica, and Raju Kunde. V2X Applications
Using Collaborative Perception. In 2018 IEEE 88th Vehicular Technology Conference(VTC-Fall), pp. 1–6, 2018.
[81] Masaya Mizutani, Manabu Tsukada, Yuki Iida, and Hiroshi Esaki. 3D maps distributionof self-driving vehicles using roadside edges. In 2020 Eighth International Symposiumon Computing and Networking Workshops (CANDARW), pp. 40–45, 2020.
[82] Makoto Nakamura, Hiroaki Nishiuchi, Jin Nakazato, Konstantin Koslowski, Julian
Daube, Ricardo Santos, Gia Khanh Tran, and Kei Sakaguchi. Experimental Verification of SDN/NFV in Integrated mmWave Access and Mesh Backhaul Networks. IEICE
Transactions on Communications, Vol. advpub, p. 2020NVP0002, 2020.
[83] IEEE Standard for Information technology–Telecommunications and information exchange between systems–Local and metropolitan area networks–Specific requirementsPart 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY)
Specifications Amendment 3: Enhancements for Very High Throughput in the 60 GHz
Band. IEEE Std 802.11ad-2012 (Amendment to IEEE Std 802.11-2012, as amended byIEEE Std 802.11ae-2012 and IEEE Std 802.11aa-2012), pp. 1–628, 2012.
[84] IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems Local and Metropolitan Area Networks–Specific Requirements
Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY)
Specifications Amendment 2: Enhanced Throughput for Operation in License-exempt
Bands above 45 GHz. IEEE Std 802.11ay-2021 (Amendment to IEEE Std 802.11-2020
as amendment by IEEE Std 802.11ax-2021), pp. 1–768, 2021.
[85] Hang Qiu, Fawad Ahmad, Fan Bai, Marco Gruteser, and Ramesh Govindan. AVR:
Augmented Vehicular Reality. MobiSys ’18, p. 81–95, New York, NY, USA, 2018.
Association for Computing Machinery.
[86] Q. Chen, S. Tang, Q. Yang, and S. Fu. Cooper: Cooperative Perception for Connected
Autonomous Vehicles Based on 3D Point Clouds. In 2019 IEEE 39th International
Conference on Distributed Computing Systems (ICDCS), pp. 514–524, Los Alamitos,
CA, USA, Jun. 2019. IEEE Computer Society.

136[87] Chang-Heng Wang, Takayuki Shimizu, Haritha Muralidharan, and Akihiko Yamamuro.
Demo: A Real-Time High-Definition Vehicular Sensor Data Sharing System using Millimeter Wave V2V Communications. In 2020 IEEE Vehicular Networking Conference(VNC), pp. 1–2, 2020.
[88] Zongdian Li, Tao Yu, Ryuichi Fukatsu, Gia Khanh Tran, and Kei Sakaguchi. Towards
Safe Automated Driving: Design of Software-Defined Dynamic MmWave V2X Networks
and PoC Implementation. IEEE Open Journal of Vehicular Technology, Vol. 2, pp. 78–
93, 2021.
[89] Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. Cooperative
Perception for 3D Object Detection in Driving Scenarios Using Infrastructure Sensors.
IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 3, pp. 1852–
1864, 2022.
[90] Mao Shan, Karan Narula, Yung Fei Wong, Stewart Worrall, Malik Khan, Paul Alexander, and Eduardo Nebot. Demonstrations of Cooperative Perception: Safety and Robustness in Connected and Automated Vehicle Operations. Sensors, Vol. 21, No. 1,
2021.
[91] Xumiao Zhang, Anlan Zhang, Jiachen Sun, Xiao Zhu, Y. Ethan Guo, Feng Qian, and
Z. Morley Mao. EMP: Edge-Assisted Multi-Vehicle Perception. In Proceedings of the27th Annual International Conference on Mobile Computing and Networking, MobiCom’21, p. 545–558, New York, NY, USA, 2021. Association for Computing Machinery.
[92] Ericsson Network Lightpole Product (2017):"Ericsson Lightpole Site".

https://www.ericsson.com/en/portfolio/networks?nav=fgb_101_0561%7Cfg
b_101_0516%7Cfgb_101_0526.
[93] LuxTurrim5G project. https://www.luxturrim5g.com/.
[94] Amazon Web Services. https://aws.amazon.com/.
[95] Microsoft Azure. https://azure.microsoft.com/.
[96] Google Cloud. https://cloud.google.com/.

137[97] Makoto Nakamura, Hiroaki Nishiuchi, Konstantin Koslowski, Julian Daube, Ricardo
Santos, Gia Khanh Tran, and Kei Sakaguchi. Performance Evaluation of Prefetching
Algorithm for Real-Time Edge Content Delivery in 5G System. In 2019 IEEE 90th
Vehicular Technology Conference (VTC2019-Fall), pp. 1–5, 2019.
[98] Jun Zhang and Khaled B. Letaief. Mobile Edge Intelligence and Computing for theInternet of Vehicles. Proceedings of the IEEE, Vol. 108, No. 2, pp. 246–261, 2020.
[99] Huisheng Ma, Shufang Li, Erqing Zhang, Zhengnan Lv, Jing Hu, and Xinlei Wei. Cooperative Autonomous Driving Oriented MEC-Aided 5G-V2X: Prototype System Design,
Field Tests and AI-Based Optimization Tools. IEEE Access, Vol. 8, pp. 54288–54302,
2020.
[100] Francesco Spinelli and Vincenzo Mancuso. Toward Enabled Industrial Verticals in 5G:
A Survey on MEC-Based Approaches to Provisioning and Flexibility. IEEE Communications Surveys Tutorials, Vol. 23, No. 1, pp. 596–630, 2021.
[101] RoboCar MV2. https://www.zmp.co.jp/products/robocar/robocar-mv2.
[102] Tsukumo Model WA9J-H200/XT. https://www.tsukumo.co.jp/bto/pc/workstati
on/2020/WA9J-H200XT.html.
[103] Intel NUC 11 Pro.

https://www.intel.com/content/www/us/en/products/sku/205600/intel-nuc-11-pro-mini-pc-nuc11tnkv5/specifications.html.
[104] Garmin GPS 18x USB. https://www.garmin.com/en-US/p/27594.
[105] RoboSense RS-LiDAR-32. https://www.robosense.ai/en/rslidar/RS-LiDAR-32.
[106] Autoware Fundation. https://www.autoware.org/.
[107] SQLite. https://www.sqlite.org/index.html.
[108] Docker. https://www.docker.com/.
[109] Python Flask. https://flask.palletsprojects.com/.
[110] HTTP servers. https://docs.python.org/3/library/http.server.html.

138[111] Rohde & Schwarz. 802.11ad - WLAN at 60 GHz A Technology Introduction. White
Paper, pp. 23–24, Nov. 2017.
[112] CloudCompare. https://www.danielgm.net/cc/.
[113] Guiyang Luo, Haibo Zhou, Nan Cheng, Quan Yuan, Jinglin Li, Fangchun Yang, andXuemin Shen. Software-defined cooperative data sharing in edge computing assisted
5g-vanet. IEEE Transactions on Mobile Computing, Vol. 20, No. 3, pp. 1212–1229,
2021.
[114] Manabu Tsukada, Takaharu Oi, Masahiro Kitazawa, and Hiroshi Esaki. Networked
roadside perception units for autonomous driving. Sensors, Vol. 20, No. 18, 2020.