IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

1

Collaborative Learning of Communication Routes in
Edge-enabled Multi-access Vehicular Environment
Celimuge Wu, Senior Member, IEEE, Zhi Liu, Senior Member, IEEE, Fuqiang Liu, Member, IEEE,
Tsutomu Yoshinaga, Member, IEEE, Yusheng Ji, Senior Member, IEEE, Jie Li, Senior Member, IEEE

Abstract—Some vehicular Internet-of-Things (IoT) applications have a strict requirement on the end-to-end delay whereedge computing can be used to provide a short delay for end-users
by conducing efﬁcient caching and computing at the edge nodes.
However, a fast and efﬁcient communication route creation in
multi-access vehicular environment is an underexplored research
problem. In this paper, we propose a collaborative learningbased routing scheme for multi-access vehicular edge computing
environment. The proposed scheme employs a reinforcement
learning algorithm based on end-edge-cloud collaboration toﬁnd routes in a proactive manner with a low communication
overhead. The routes are also preemptively changed based on
the learned information. By integrating the “proactive” and“preemptive” approach, the proposed scheme can achieve a better
forwarding of packets as compared with existing alternatives. We
conduct extensive and realistic computer simulations to show theperformance advantage of the proposed scheme over existingbaselines.
Index Terms—Vehicular networks, routing protocol, collaborative learning, multi-access vehicular environment, fuzzy logic,
reinforcement learning.

I. I NTRODUCTION
Future vehicular Internet-of-Things (IoT) systems involve a
huge number of devices in multi-access environments where
different types of wireless spectrums should be efﬁciently
utilized [1]–[3]. At the same time, novel services, such as cooperative autonomous driving and intelligent transport systems
that demand unprecedented accuracy, latency and bandwidth
are emerging. These services have an extreme variance in theirresource demand with respect to time, location, context, as
well as individual patterns. Current vehicular IoT systems,
Copyright (c) 2020 IEEE. Personal use of this material is permitted.
However, permission to use this material for any other purposes must be
obtained from the IEEE by sending a request to pubs-permissions@ieee.org.
C. Wu and T. Yoshinaga are with the Graduate School ofInformatics and Engineering, The University of Electro-Communications,
1-5-1, Chofugaoka, Chofu-shi, Tokyo, 182-8585 Japan (e-mail:{celimuge,yoshinaga}@uec.ac.jp).
Z. Liu is with the Department of Mathematical and Systems Engineering,
Shizuoka University, No5. 3-5-1, Johoku Hamamatsu, 432-8561, Japan (email: liu@ieee.org)
F. Liu is with the the School of Electronics and Information Engineering,
Tongji University, 4800 Cao’an, Jiading, Shanghai, China, 201804 (e-mail:liufuqiang@tongji.edu.cn)
Y. Ji is with the Information Systems Architecture Research Division,
National Institute of Informatics, 2-1-2, Hitotsubashi, Chiyoda-ku, Tokyo 1018430 Japan (e-mail: kei@nii.ac.jp)
J. Li is with the Department of Computer Science and Engineering, Shanghai Jiaotong University, 800 Dongchuan RD. Minhang District, Shanghai,
China, 200240 (e-mail: lijiecs@sjtu.edu.cn)).
Manuscript received 2020.
Corresponding author: Zhi Liu (e-mail: liu@ieee.org).

such as autonomous driving, only consider the intelligenceof a single vehicle agent, and therefore a limitation exists. In
order to realize a more intelligent vehicular IoT system, thecollaboration between different vehicle/roadside units should
be utilized efﬁciently. This requires an efﬁcient and intelligent
networking scheme that can handle the highly mobile andvarying features of the environment.
The future vehicular IoT applications can be classiﬁed into
two main categories in terms of the underlying networking
technologies, namely, vehicle-to-infrastructure (V2I) applications and vehicle-to-vehicle (V2V) applications. Most V2I
applications, such as vehicular sensor data collections, aretrafﬁc-intensive, which means that the applications require a
communication approach that could deliver a large amountof data in a short time. In contrast, most vehicle-to-vehicle
applications are used to delivery safety messages or control
messages between vehicles, which are delay-sensitive. The
trafﬁc-intensive applications and delay-sensitive applications
have different types of quality-of-service (QoS) or qualityof-experience (QoE) requirements [4], [5], and therefore weshould consider the difference in the design of communication
protocols.
Meanwhile, there could be multiple types of communicationinterfaces available simultaneously for each vehicle, resulting
in the selection of best communication approach in a multiaccess environment particularly important. The edge computing [6]–[9] has been widely discussed for use in data caching
and computation ofﬂoading. However, the communicationroute selection in a multi-access vehicular environment was
not discussed adequately. Since the communication request
and the corresponding QoS requirement are always difﬁcult
to predict, we have to reserve some communication resources
before a request is made. However, it is difﬁcult to ﬁnd thebest route for each possible communication pair with low overhead, especially when the communication environment itself
changes fast with the vehicle movement. It is important todesign an intelligent method to handle these spatial-temporalchanges of vehicular environment. Recently, artiﬁcial intelligence based approaches have been attracting great interestin achieving intelligence in computer systems. However, due
to the above mentioned characteristics of vehicular networks,
conducting an efﬁcient learning in vehicular environments is a
difﬁcult scientiﬁc problem. It is important to design a learning
scheme that could evaluate and improve actions with low
communication overhead.
In this paper, we propose a reinforcement learning based
scheme for route selection in multi-access vehicular edge

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

computing environment. We employ an efﬁcient end-edgecloud collaboration to fasten the convergence speed of thelearning algorithm. The main contributions of this paper are
as follows.
• We propose a Q-learning-based routing scheme for multiaccess vehicular edge computing environments. The proposed scheme uses a “proactive” approach to ﬁnd communication routes with a low overhead, reducing thedelay in ﬁnding a route when a communication request
is received. The proposed scheme also employs a “preemptive” approach to replace an existing route with a
new one by dynamically learning a better route using areinforcement learning approach.
• We propose a decentralized approach for vehicle edge selection by jointly considering the vehicle velocity, vehicle
distribution, and the connectivity between vehicles basedon fuzzy logic.
• We achieve end-edge-cloud collaboration approach basedon a Q-learning algorithm. Each vehicle agent is able tolearn the best route by receiving the feedback from thecloud/edge.
• We consider different QoS requirements posed by different types of applications and select the best next hop routeaccording to the speciﬁc requirement of each application.
The remainder of the paper is organized as follows. We
ﬁrst give a brief survey of related work in Section II. Then,the details about the proposed scheme are explained in section
III. Section IV shows the simulation results for the evaluationof the proposed scheme, and ﬁnally section V draws ourconclusions.
II. R ELATED WORK
A. Edge computing in vehicular IoT
Most of the existing studies discuss how to conduct efﬁcient caching or computation ofﬂoading, and do not seriously
discuss how to ﬁnd a communication peer in a multi-accessvehicular environment. Su et al. [10] have proposed a crossentropy-based caching scheme for vehicular content networks.
The content access pattern, vehicle speed, and vehicle density
were considered in the content caching at the edge nodes
in order to facilitate a timely content delivery. Ale et al.
[11] have employed a bidirectional deep recurrent neuralnetwork (BRNN) to conduct online proactive caching foredge computing. The BRNN model was used to predict timeseries content requests in order to solve the difﬁculty incontent popularity recognition. In [12], the joint optimizationof content placement and content delivery in the vehicularedge computing was studied, and a deep deterministic policy
gradient framework was used to solve the problem.
Feng et al. [13] have introduced the concept of autonomousvehicular edge computing that utilizes the computational capabilities of vehicles in a decentralized manner. An efﬁcientjob caching approach was proposed to improve the schedulingof jobs based on the information exchange between neighbors.
Wang et al. [14] have proposed a multi-user non-cooperativegame-based approach for computation ofﬂoading in vehicularedge computing. They designed a payoff function taking

2

into account the node distance, application requirements,
communication overhead, and the contention for computing
resources. In [15], Liu et al. discussed the ofﬂoading problem of multiple tasks with task dependency, and proposeda task scheduling algorithm that prioritizes multiple tasks toguarantee the completion time constraint of each task whileconsidering the dependency relationship between tasks. The
problem of vehicle edge server selection for the task migration
has been discussed in [16]. The problem was formulated as aﬁnite horizon Markov decision process, and a time-aware taskofﬂoading approach was proposed to solve the problem.
Tan and Hu et al. [17] have proposed a joint caching
and computing approach where the resource allocation was
conducted by considering the vehicle mobility and service
deadline constraint. In [18], a deep reinforcement learningbased joint optimization of the edge computing and content
caching was managed to improve the proﬁts of mobile network
operator while ensuring user QoS in 5G-envisioned internetof-vehicles.
B. QoS control in vehicular environment
Since diverse applications that require different levels of
QoS constraints are expected for vehicular networks, theconventional one-ﬁt-all approach fails to satisfy the needs ofdifferent users. Wu and Zheng [19] have conducted a theoretical analysis on the uplink local delay between a vehicle and anedge node in a MEC-based VANET using stochastic geometry.
The distributions of vehicles and edge nodes were modeled as
an independent one-dimensional homogeneous Poisson point
process. The analytical result was validated through computer
simulations and the dominant factors on the transmission delay
were investigated. Zhang et al. [20] have proposed a serviceoriented hierarchical soft slicing framework to support multidimensional QoS in vehicular networks. Different network
slices were constructed to support the context information
service and the infotainment service, respectively, in order todifferentiate different QoS requirements.
In [21], a grid routing protocol was proposed to guarantee
QoS in the perception of complex vehicular IoT environments.
The grid identiﬁcation number was used to calculate thedistance between nodes and ﬁnd the least delay path. Garg
et al. [22] have discussed the integration of software deﬁned
networking (SDN) and edge computing for QoS guarantee
in vehicular networks, and proposed a mobility and QoSaware SDN framework for autonomous vehicles. Kumar etal. [23] have studied the multimedia content delivery fromcloud video streaming servers to moving vehicles. In orderto support sufﬁcient QoS for different video streaming cases,
a QoS-aware hierarchical web caching strategy was proposed
based on two metrics, namely, the load utilization ratio andthe query to connectivity ratio. Peng et al. [24] proposeda dynamic spectrum management framework to guarantee
QoS by considering the spectrum slicing, spectrum allocation,and transmission power adjustment. Three important issues,
speciﬁcally, spectrum slicing among base stations, spectrum
allocation for vehicles, and transmit power adjustment at basestations, were jointly solved through an alternate concave

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020search algorithm. In the above mentioned studies, an efﬁcient
integration of different communication approaches in a multiaccess vehicular environment is not discussed.
C. Communication resource allocation in multi-access vehicular environment
Sun et al. [25] have introduced a data delivery protocol
for vehicular networks based on a prediction of vehicular
trafﬁc volume. Li et al. [26] have addressed the data routing
problem from a vehicle to a ﬁxed destination with multihop forwarding. The routing decision was made by using
a geographical approach that divides a road segment into
smaller grids. However, an efﬁcient use of multiple accesstechnologies was not studied. In [27], an efﬁcient integrationof licensed and unlicensed spectrums was discussed. Anedge computing approach at vehicles was used to improve
the spectrum efﬁciency in the case where multiple typesof communication approaches exist. Nkenyereye et al. [28]have proposed an SDN-based multi-access edge computing
approach for vehicular networks. An OpenFlow algorithm wasdeveloped to facilitate the packet forwarding process based on
the target area and current route condition.
Different types of communications, including broadcast andunicast, coexist in vehicular networks. In [29], an approachfor neighbor discovery was proposed. The neighbor discovery
approach conducts mobility prediction based on Kalman ﬁlter
theory. Kuhlmorgen et al. [30] have proposed a packet forwarding scheme that takes advantage of both contention-based
forwarding and decentralized congestion control considering
the existence of mixed data trafﬁc.
Some studies have discussed the use of physical layer
technologies in improving the resource allocation efﬁciency.
Yang et al. [31] have studied the downlink radio resource
management problem for ultra-reliable and low latency communications in V2I systems. The problem was discussed by
exploiting the beneﬁts of massive MIMO. A non-orthogonal
multiple access (NOMA) based resource allocation for vehicular networks has been discussed in [32]. However, the QoSaware resource allocation problems in a multi-access vehicularenvironment, especially the route selection and communication
approach selection, still need investigations.
III. P ROPOSED SCHEME
A. Problem deﬁnition and system overview
We assume that each node has three different types ofcommunication interfaces, speciﬁcally, cellular, IEEE 802.11p
and mmWave interface. These communication approaches do
not interfere with each other, and each node can switch
between them or utilize all the communication approaches
simultaneously. The use of different communication interfaces
can improve the wireless resource utilization efﬁciency while
requiring an approach to ﬁnd the best communication interface
for the transmission of each packet. Each node sends periodical
hello messages with IEEE 802.11p interface in order toexchange information among neighbors, and the hello interval

3

is 1 second by default. The vehicle identiﬁer, position information, vehicle velocity, and some other information (details
will be explained later) are attached in the hello messages.
The research problems we discuss here are: 1) how to
efﬁciently utilize different types of communication interfaces?;2) how to ﬁnd the best route for each vehicle with low
overhead in accordance with the requirement and ensure theupdate of the route when a change of environment occurs?;3) how to design a learning scheme that could adapt to the
dynamically changing vehicular environment?
We propose a reinforcement learning-based routing scheme
for multi-access vehicular edge computing environments. The
proposed scheme employs a Q-learning algorithm to ﬁnd thebest route for each agent, and uses an end-edge-cloud collaboration approach to achieve intelligence in the route selection.
The proposed scheme uses different learning approaches for
trafﬁc-intensive applications and delay-sensitive applications.
First, we introduce a route selection approach for trafﬁcintensive applications. Then, we introduce a vehicle-to-vehicle
route selection approach for delay-sensitive applications.
In our proposed scheme, IEEE 802.11p is used for transmission of both data and control messages while mmWave isonly used to transmit data among neighbors. Basic procedureof the proposed scheme consists of two stages. In the ﬁrst
stage, we choose the vehicle edge nodes 1 . In this stage,
we select all the vehicle edges, and ensure that they areconnected with each other through IEEE 802.11p link. In
the second stage, each agent (vehicle) learns the best routeto the corresponding destination by evaluating the reward(feedback) from the cloud (in the case of V2I communications)or from the communication partner (in the case of V2V
communications). As shown in Fig.1, an ordinary vehicle
could receive a reward from a base station (BS) or receivea discounted reward from an edge vehicle. In the former casethe vehicle connects with the BS using cellular interface, andin the latter case the vehicle connects to the BS through theedge vehicle using mmWave/IEEE 802.11p interface. Based on
the reward, the vehicle chooses the best action between two
possible next hops, speciﬁcally, the BS or the edge vehicle.
The route selection approach for trafﬁc-intensive applications is as follows. By selecting a vehicle edge node/basestation as the next hop (action), each agent could receive a
feedback from the next hop and evaluate the goodness of
the action. The reward is only allocated by the base station(BS), and is transmitted to each agent with a discounted value.
The learning for the delay-sensitive applications uses a similar
learning approach but with a difference in the allocation of the
reward. While the learning of the trafﬁc-intensive applications
allocates the reward based on the throughput each route canprovide, the learning of delay-sensitive routes allocates reward
based on the expected transmission delay.
B. Fuzzy logic-based vehicle edge selection
We use a fuzzy logic-based approach to evaluate eachvehicle is whether suitable for being an edge node or not.
1 In the following, “edge node” and “vehicle edge node” are used interchangeably; “non-edge node” denotes a vehicle that is not selected as anedge node.

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

4

weighted exponential moving average with a smooth factor of
0.7.
Connectivity Metric (CM ): Connectivity metric can becalculated in two ways depending on the available information.
First, we can use the ratio of “the number of hello messages
received from all one-hop neighbors” to “the number of hellomessages sent by all one-hop neighbors” as
CM (x) =The suitability value is calculated by considering the vehicle
velocity, vehicle distribution, and the connectivity between
vehicles. The suitability value is calculated as follows. First,
three different metrics, namely, stability metric, leadership
metric, and connectivity metric, are deﬁned. Then, we convert
these metrics to fuzzy values using fuzzy membership functions, and then apply some predeﬁned rules to calculate thefuzzy value for the suitability level. In the ﬁnal step, the fuzzy
value for the suitability is converted to a numerical value [33].
The suitability value is calculated for each vehicle within therange of 21 R where R is the one-hop communication distanceof IEEE 802.11p. If a node ﬁnds itself having the largest
suitability value, the node declares itself as an edge node. Here,“edge node” works a leader to manage a group of vehicles
and provide gateway supports for the ordinary vehicles in thegroup.
1) First step – deﬁnition of three factors: The stability
metric, leadership metric, and connectivity metric factor are
calculated based on the information in the hello messages
received from neighbors.
Stability Metric (SM ): Stability metric of node x is
calculated as follows.
||υ(x)| − avgy∈Nx |υ(y)||maxy∈Nx |υ(y)|c(x)
Number of neighbor vehicles

1 Bad

(1)

Medium

Good

1 Bad

0.8

where a higher value means a higher stability. Since hellomessages are exchanged between neighbors, each vehicle cancalculate its neighbors’ SM. Here, avgy∈Nx |υ(y)| is predictedfrom the information attached in the hello messages. SM is
updated periodically (one second interval) based on a weighted
exponential moving average with a smoothing factor of 0.7.
The value of smoothing factor is determined based on our
experience [33].
Leadership Metric (LM ): Leadership metric is calculated
as follows.
(
LM (x) = min 1,

where h(x) is the antenna height of node x.
2) Second step – fuzziﬁcation and fuzzy rules: Fuzzy logic
is used to evaluate whether a vehicle is suitable for working
as an edge node or not. This evaluation should be conductedas soon as possible in order to satisfy strict QoS requirements of vehicular IoT applications. Therefore, consideringcomputational complexity, we use triangular or trapezoidal
membership functions instead of non-liner membership functions, such as Gaussian membership functions that require
more computational resources in fuzzy reasoning. The fuzzy
membership functions are deﬁned as shown in Fig. 2. The
linguistic variables of the three metric are deﬁned as {Good,
Medium, Bad}.

)(2)

where c(x) denotes the number of vehicles traveling to thesame direction as the node x in its neighbors. The higher
the number, the higher chance the vehicle is elected as anedge node. Here, the number of vehicles in one-hop region is
acquired from the information attached in the hello messages.
LM is updated periodically (one second interval) based on a

Medium

1 Bad

Good

0.6
0.4

0.6
0.4

0.2

0.2

0

0
0 0.2 0.4 0.6 0.8 1
SM

Fig. 2.

Medium

Good

0.8

0.8
DegreeSM (x) = 1 −(3)Note that “the number of hello messages sent by all onehop neighbors” can be calculated by observing the sequencenumber of received hello messages since each hello message
is identiﬁed with a unique sequence number which is incremented by a predeﬁned value for each hello period. The otherway is to use the antenna height to show the connectivity
metric as a vehicle with a higher antenna always can provide
a better connectivity to the neighbor vehicles. In that case,
CM is calculated ash(x)CM (x) =.
(4)maxy∈Nx h(y)DegreeLearning based on end-edge-cloud collaboration.

DegreeFig. 1.

Num of hellos received from all NBs.
Num of hellos sent by all NBs

0.6
0.4
0.2

0 0.2 0.4 0.6 0.8 1
LM

0
0

0.2

0.4

0.6

0.8

1

CM

Fuzzy membership functions (left: SM , middle: LM , right: CM ).

The fuzzy rule is deﬁned in Table I.
Each node calculates the rank (a suitability value for being
an edge node) of each neighbor based on the IF/THEN rules
as deﬁned in Table I. The linguistic variables for the rank
are deﬁned as {Perfect, Good, Acceptable, Unpreferable, Bad,
VeryBad}. In Table I, Rule 1 is expressed as follows:
IF Velocity is Slow, Leadership is High, and Connectivity
is Good THEN Rank is Perfect.
Note that multiple rules could be applied for the same fuzzy
value. Here, we use the Min-Max method to combine theresults from multiple rules.

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

TABLE IF UZZY RULES

Stability
Good
Good
Good
Good
Good
Good
Good
Good
Good
MediumMediumMediumMediumMediumMediumMediumMediumMedium
Bad
Bad
Bad
Bad
Bad
Bad
Bad
Bad
Bad

Rule1
Rule2
Rule3
Rule4
Rule5
Rule6
Rule7
Rule8
Rule9
Rule10
Rule11
Rule12
Rule13
Rule14
Rule15
Rule16
Rule17
Rule18Rule19Rule20
Rule21
Rule22Rule23
Rule24Rule25
Rule26
Rule27

Leadership
Good
Good
Good
MediumMediumMedium
Bad
Bad
Bad
Good
Good
Good
MediumMediumMedium
Bad
Bad
Bad
Good
Good
Good
MediumMediumMediumBad
Bad
Bad

Connectivity
Good
MediumBad
Good
MediumBad
Good
MediumBad
Good
MediumBad
Good
MediumBad
Good
MediumBad
Good
MediumBad
Good
MediumBad
Good
MediumBad

Rank
Perfect
Good
Unpreferable
Good
Acceptable
Bad
Unpreferable
Bad
VeryBadGood
Acceptable
Bad
Acceptable
Unpreferable
Bad
Bad
Bad
VeryBadUnpreferable
Bad
VeryBadBad
Bad
VeryBadBad
VeryBadVeryBad3) Last step – defuzziﬁcation: Fig. 3 shows the output
membership function that is used to convert from a fuzzy
value to a numerical value. The process of conversion is calleddefuzziﬁcation. In this work, the center of gravity (COG)
method is used for the defuzziﬁcation.
1

VeryBadBad

Unpreferable

Acceptable Good

Perfect

0.8
0.6
0.4

node for information transmission. Note that the information
could be sensor data or the data required for task ofﬂoading,which means that ﬁnding a next hop node is mandatory forcommunications as well as computing.
1) Q-learning model: The Q-learning model is deﬁned asfollows. Vehicles are the agents, and the actions are the pairs ofa possible communication type and the next hop node for the
packet forwarding. The possible actions at each node would
be the set of its one-hop neighbors including base station. The
BS is responsible for sending back a reward for each action thevehicle executed. The reward will be further transmitted witha discount by other edge vehicles. Each vehicle adjusts own
behavior based on the feedback from the BS. The informationexchange between agents is done with hello messages. Eachnode maintains a Q-Table where each Q-value shows the value
for choosing m as the next hop to the RSU.
2) Update of Q-Table: The state is expressed by a pair of{destination, current node}, and the action is determined by{communication type, next hop}. In case of using the cellularcommunications, the next hop would be BS, and in the caseof IEEE 802.11p, the next hop node would be a neighbornode. Each node has to maintain a Q-value for each tripleof a destination, the communication type, and an one-hop
neighbor. Upon reception of each hello message, the Q-Table
is updated. The initial value for each Q-value is 0. Each vehicle
maintains a Q-value to the BS and each vehicle in two-hop
region. Considering the change of neighbors with the vehicle
movement, we release the corresponding Q-Table space ofold neighbors when necessary for the purpose of maintaining
information about new neighbors. The proposed scheme does
not maintain route for each possible destination considering
the size of Q-Table. For ﬁnding a route to other nodes, theproposed scheme uses a hierarchical routing approach where
different levels of gateway nodes exist. Note that for eachvehicle, there is at least one neighbor would be an edge nodethat is working as a gateway node and responsible for ﬁnding
a route to any other vehicle. The BS also performs a dutyof gateway. After reception of a hello message from node m,node c updates the corresponding Q-value as
Qc (d, t, m) ← α × LQ(c, m)
× {R + γ × maxy∈N Bm Qm (d, t, y)}+ (1 − α) × Qc (d, t, m).
(5)

0.2
0
0

Fig. 3.

5

0.2

0.4

0.6

0.8

1

Output membership function.

C. Q-learning based end-edge-cloud collaboration for trafﬁcintensive applications
Here, BS is connected to the cloud, and the vehicles are connected to the cloud through the BS. Therefore, the BS is treatedas the cloud in our learning model for simplicity. The selectededge vehicles are the edges. The ordinary vehicles are theend nodes. Our aim is to conduct an intelligent collaboration
within the end-edge-cloud architecture. The cloud and edge
nodes give suggestion to the end nodes to select the next hop

where d and t are the destination node and communication
type, respectively. LQ(c, m) is the link quality value between
node c and m, which is expressed by the hello reception ratio
between two nodes. N Bm denotes the one-hop neighbor setof node m. Here, the learning rate α and discount rate γ are
set to 0.8 and 0.9, respectively, based on our experience [34].
The reward Rm is calculated as{
R̄, if m is base station and c is an edge nodeR=
(6)0, otherwise
where R̄ ∈ [0, 1] is allocated by the BS according to thenumber of vehicles connected to the BS. The base station will
set the reward according the vehicle density. If the density ishigh, the cloud will give a high reward to the correspondingIEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

edge candidate. The reward is further discounted according to
the number of hops. The exploration is achieved by exchanging hello messages among neighbors. Therefore, for a routeselection, each node always can choose the pair of node andcommunication type showing the highest Q-value.
In case of V2V communication routes, the reward is updatedas{
1, if m is an edge nodeR=
(7)0, otherwise.
The reward is set to 1 here in order to utilize decentralized
communications as far as possible. The decentralized communication approaches include IEEE 802.11p and mmWave.
Since the mmWave communication is only possible in a lineof-sight communication link, we only use mmWave for thecommunications between an edge vehicle node and its nonedge neighbors. For this type of communication pairs, if bothmmWave and IEEE 802.11p are available, then the mmWave
is used.
Each edge vehicle attaches its own cost (the corresponding
Q-value) to its neighbors in the hello message. Upon receptionof a hello message from a neighbor vehicle, the vehicle
could update the corresponding cost in the case of using theneighbor as the next hop based on the Q-learning algorithm.
The exploration of Q-learning is achieved by the periodical
hello message exchange. Therefore, each agent is aware of
the best action based on the Q-values.
D. Q-learning based packet forwarding for delay-sensitive
unicast applications
The delay-sensitive applications could be used to send theinformation required for the collaboration between vehicles,
which could enable collaborative autonomous driving. The
data transfer can be either conducted by up to 2-hop V2V
communications or cellular communications. Here the reward
is calculated based on the delay. For V2V communications, thereward is sent by each vehicle to its neighbors. The selectededge vehicle could further transmit the discounted reward to
its neighbors, but the reward will not be disseminated to morethan two hops. This is achieved by only the edge vehicle nodes
attaching the Q-Table entries of one-hop neighbors to the hellomessages.
The reward from the BS through cellular interface is calculated as
DthR̂ = min(1, Spkt ∗Nue
)
(8)
S
∗Nue
+ pkt
+ Dbs
BWul
BWdl
where Dth , Spkt , Nue , BWul , BWdl , Dbs are the delay
requirement, packet size, number of user devices, uplinkbandwidth, down link bandwidth, and processing delay atthe base station. If the BS is able to satisfy the incoming
request, R̂ is set as 1, and otherwise set to a smaller value.
The processing delay includes all the times required for thescheduling, queueing, and computing except the propagation
delay.
For the path using IEEE 802.11p communications, thereward is received from a neighbor node. In this case, thereward from a vehicle is calculated as
Dth)R̂ = min(1,Spkt
BW11p ×HRR + D11p

6

(9)where BW11p is the bandwidth of IEEE 802.11p (27 Mbps)
and HRR is hello reception ration between two neighbors.
D11p is the processing delay at each vehicle, which includes
all the contention delay, the retransmission delay due to packet
collisions, and computing delay. Here, the discount rate is set
as 0.5 in order to avoid the use of 2-hop transmissions as
far as possible. Note that the parameters of Eq.(8) and Eq.(9)should be tuned based on the corresponding hardware andenvironment, which is beyond the scope of this paper.
E. Data dissemination for delay-sensitive broadcast applications
Data dissemination for delay-sensitive broadcast applications should be conducted through IEEE 802.11p V2V communications as it could be difﬁcult to use cellular communications to detect all the intended receivers. We propose amulti-hop broadcast approach based on the proposed edgearchitecture. As shown in Fig.4, the broadcast messages areforwarded by the edge nodes. The ordinary nodes do not
rebroadcast the messages, which can signiﬁcantly reduce theredundant forwarding. It is also possible to conﬁrm the packet
delivery status of a broadcast packet by sending the acknowledge message (ACK) from the edge nodes, which makes theretransmission of a broadcast message possible and therefore
ensure a high packet dissemination ratio. The forwarding
algorithm is also simple as follow. If the current node is anedge node, the node just rebroadcasts the packet, and otherwise
just receives the packet without further forwarding.

Fig. 4.

Edge-based multi-hop broadcast.

IV. S IMULATION R ESULTS
To evaluate the performance of the proposed scheme, we
conducted extensive computer simulations using ns-2.34 [35](see Table II). We used a freeway road having three lanes ineach direction, which was generated using the same approachas [34]. Each vehicle had three different types of wirelessinterfaces, namely, cellular, IEEE 802.11p, and mmWave. For
IEEE 802.11p, we used Nakagami model to include a realistic
fading environment, and the parameters were the same as[34]. The average transmission range of IEEE 802.11p was
set as 250 m. The proposed protocol was compared with
“Without edge” (without edge computing), “Random edge”(random edge selection), and “Edge without preemptive” (edgeIEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

80

A. Performance for trafﬁc-intensive applications
For trafﬁc-intensive applications, we evaluated the communication performance between vehicles and the cloud. The
number of cellular base station was 1, which means thatthe communication path to the cloud must go through thisbase station. For simplicity, we used a down link trafﬁc
where the data were sent from the cloud to all the vehicles
and data were cached at the base station. Note that, in thesimulation topology, the base station was “cloud”, and theselected vehicle edge nodes were “edges”. Therefore, “endedge-cloud collaboration” was simulated by the collaboration
among non-edge vehicles, edge vehicles, and the base station.
Fig.5 shows the TCP throughput for various numbers ofreceivers where the cellular bandwidth was 500 Mbps. The
maximum vehicle velocity was 100 Km/h. We can observethat “Without edge” approach fails to provide an acceptablethroughput since all the vehicles use cellular communications
and therefore the resource allocated to each vehicle is small.
“Random edge” achieves better performance by conducting
data caching at randomly selected edge nodes. However,
due to the inefﬁciency of the random edge selection, theperformance improvement is limited. By choosing the best
nodes for the edge nodes based on a fuzzy logic algorithm,
the proposed scheme shows the largest throughput. “Edgewithout preemptive” uses ﬁxed edge node and changes aroute only when the link becomes unavailable, resulting in
difﬁculty of adapting to the change of network environments.
The advantage of the proposed protocol over “Edge without
preemptive” explains that it is promising to conduct a routechange by preemptively ﬁnding a better route. The proposed
scheme is able to preemptively change to a better route byusing the Q-learning approach to explore all the possible paths
based on the exchange of periodical hello messages.

50
40
30
20

0
100

200

300

400

500

Number of nodes
Fig. 5.

TCP throughput for various numbers of receivers.

Fig.6 shows the TCP throughput for various maximum
vehicle velocities. The number of vehicle was 300. The performance of “Random edge” is affected by the vehicle velocity
signiﬁcantly, which shows the importance of using an efﬁcientedge selection algorithm. The proposed scheme can ﬁnd stableedge nodes by taking into account the velocity factor, vehicle
distribution, and the signal quality between vehicles for theedge selection. This ensures that the proposed scheme could
achieve the best performance for different vehicle velocities.
16
14

TCP throughput (Mbps)

Freeway, 2000m, 6 lanes
400
100 km/h
Ref. [34]IEEE 802.11p MAC
27 Mbps (unicast), 3 Mbps (broadcast)
Nakagami Model
1000 s60

10TABLE IIS IMULATION E NVIRONMENT

Topology
Number of nodes
Maximum velocity
Mobility generator
MAC
Data rate
Fading model
Simulation time

Without edge
Random edge
Edge without preemptive
Proposed70

TCP throughput (Mbps)

without preemptive route change). In “Without edge”, eachnode only uses the cellular interface for communications.
“Random edge” collects some nodes randomly to conduct
data caching. “Edge without preemptive” employs the same
approach as the proposed scheme for the edge selection but
does not conduct efﬁcient route change between different
wireless interfaces. Three different types of applications wereconsidered in the simulations, namely, trafﬁc-intensive applications, delay-sensitive unicast applications, and delay-sensitive
broadcast applications. In the following simulation result ﬁgures, each error bar shows the 95% conﬁdence interval of thecorresponding data.

7

12
10
8
6

Without edge
Random edge
Edge without preemptive
Proposed

4
2
0
70

80

90

100

110

120

Maximum allowable velocity (Km/h)Fig. 6.

TCP throughput for various vehicle velocities.

The effect of the available cellular bandwidth on the protocol performance is shown in Fig.7. The number of vehicle
was 300, and the maximum vehicle velocity was 100 Km/h.
In “Without edge”, all the receiver nodes directly get thedata from the cloud by using cellular communications, which
results in an inefﬁcient use of cellular resources. Therefore,
the throughput improvement with the increase of cellular
bandwidth is not notable. Other approaches could utilize
the cellular resources more efﬁciently by conducting edge
computing at edge nodes. However, the performance of “Random edge” is still unsatisfactory due to the blindness of therandom edge selection. “Edge without preemptive” performs

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

TCP throughput (Mbps)

30
Without edge
Random edge
Edge without preemptive
Proposed

25

20

90
80

End-to-end delay (ms)better than “Random edge” by selecting better and morestable edge nodes. The proposed scheme can further improve“Edge without preemptive” by ﬁnding the best route basedon Q-learning and switching between different communicationinterfaces.

8

70
60
50
40
30
20
Without edge
Random edge
Edge without preemptive
Proposed

10
15
0
100

200

300

400

500

Number of traffic flows

10

Fig. 8.

End-to-end delay for various vehicle densities.

5

100

200

300

400

500

Cellular bandwidth (Mbps)
Fig. 7.

TCP throughput for various cellular bandwidths.

B. Performance for delay-sensitive unicast applications
We generated UDP trafﬁcs to evaluate the performance ofthe proposed scheme for delay-sensitive unicast applications.
UDP packet size was 512 bytes, and the data rate for eachtrafﬁc ﬂow was set as 1 packet per second. The cellular
bandwidths for uplink and downlink were 250 Mbps and
500 Mbps, respectively. The processing delay at the basestation was set as 50 ms. Fig.8 shows the end-to-end delay
for various numbers of trafﬁc ﬂows. The delay of “Withoutedge” is the largest as the use of pure cellular transmissions always encounter the problem of insufﬁcient bandwidths
when then number of trafﬁc ﬂows is large. This is because
all the trafﬁc ﬂows go through the base station, which is
inefﬁcient for some communication pairs that are close toeach other geographically. “Random edge” shows a better
outcome by conducing some IEEE 802.11p communications
through randomly selected edge nodes without going throughthe base station. However, the random selection of edge nodeis unsatisfactory in terms of MAC layer contention efﬁciency.
Since the proposed scheme could select the best edge nodes
for selection, it can achieve the lowest delay, which is asigniﬁcant improvement especially when the number of trafﬁc
ﬂows is large. The difference between “edge w/o preemptive”
and the proposed scheme shows that it is difﬁcult to achievea good performance if the route is not changed until the routeis disconnected. It is important to switch efﬁciently between
the cellular communications and IEEE 802.11p according
to the usage ratio of cellular spectrum. This requires anadaptive algorithm that can evolve by efﬁciently perceiving
the environments.
We also conducted evaluations for different processing
latencies at the base station where the number of trafﬁc
ﬂows was 300. As shown in Fig.9, with the increase of theprocessing delay at the base station, the advantage of the
proposed scheme becomes more signiﬁcant. This is because
it becomes important to use other types of communication
approaches to support the cellular communications in orderto provide a low delay for all the communication pairs. The
proposed scheme always can achieve a low delay by using
IEEE 802.11p communications for some communication pairs.
By using the Q-learning algorithm, the proposed scheme canﬁnd the best communication interface and the corresponding
route for each communication pair, resulting in the lowest
delay.
120

100

End-to-end delay (ms)

0

80

60

40

20

Without edge
Random edge
Edge without preemptive
Proposed

0
20

40

60

80

Processing delay at BS (ms)Fig. 9. End-to-end delay for various processing latencies at the base station.

Fig.10 shows the end-to-end delay for various vehicle velocities. The number of trafﬁc ﬂows was 100. We can observethat the delay of “Without edge” and “Random edge” increases
as the vehicle velocity becomes faster. This explains theimportance of selecting stable edge nodes for the data caching.
By considering the vehicle velocity and vehicle distribution in
the edge node selection, the proposed scheme achieves a stablelatency for different vehicle velocities. The preemptive routechange approach also contributes to the short delay by ﬁnding
a better route before the current route is broken.

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

60

vehicle distribution, and the link quality between vehicles,
the proposed scheme is able to ﬁnd the best edges, andtherefore satisfy the low latency requirement of delay-sensitive
broadcast applications.

40
700
30
600
20

10

Without edge
Random edge
Edge without preemptive
Proposed

0
70

80

90

100

110

120

Maximum allowable velocity (Km/h)Fig. 10.

End-to-end delay for various vehicle velocities.

500

400

300

200
Without edge
Random edge
Edge without preemptive
Proposed100

C. Performance for delay-sensitive broadcast applications
We also conducted simulations for delay-sensitive broadcast
applications. The number of broadcast source nodes was 1, andall other vehicles were considered as the intended receivers.
Since the broadcast communication through cellular interface
is not realistic, here “Without edge” denotes the weighted ppersistence [33]. Fig.11 shows the packet dissemination ratio
for various vehicle densities. Since “Without edge” does notuse edge computing on packet dissemination, it results in a low
packet dissemination ratio due to the redundant rebroadcastof data. The edge selection of “Random edge” also cannot
achieve a satisfactory result. “Edge without preemptive” showsa higher packet dissemination ratio as compared with “Withoutedge” and “Random edge” by using a more efﬁcient edge
selection. Based on the efﬁcient edge node selection and theedge-based retransmission scheme, the proposed scheme canprovide the perfect packet dissemination ratio.
1

0
100

200

300

400

500

Number of nodes
Fig. 12.

End-to-end delay for various vehicle densities.

V. C ONCLUSIONSWe discussed the problem of route selection in multiaccess vehicular edge computing environment, and proposeda scheme based on a reinforcement learning approach. The
proposed scheme employs a “proactive” approach to ﬁnd
communication routes based on periodical hello message exchange with low overhead, and conducts “preemptive” changeof communication interfaces and routes to ensure a highperformance in varying network environments. The propose
scheme uses different learning criteria for trafﬁc-intensive
applications and delay-sensitive applications in order to satisfy
different QoS requirements. The simulation results show thatthe proposed approach can achieve a better performance thanexisting baselines.
ACKNOWLEDGMENT
This research was supported in part by JSPS KAKENHI
Grant Numbers, 18KK0279 and 19H04093, and the Telecommunications Advanced Foundation.

0.8

0.6

R EFERENCES

0.4

0.2
Without edge
Random edge
Edge without preemptive
Proposed
0
100

200

300

400

500

Number of nodes
Fig. 11.

End-to-end delay (ms)End-to-end delay (ms)50

Packet dissemination ratio

9

Packet dissemination ratio for various vehicle densities.

As shown in Fig.12, the proposed scheme shows the lowest
end-to-end delay. When the node density is high, the selection of edge node has more a remarkable impact on the
delay. Since the proposed scheme considers vehicle velocity,[1] C. Wu, Z. Liu, D. Zhang, T. Yoshinaga, and Y. Ji, “Spatial Intelligence
towards Trustworthy Vehicular IoT,” IEEE Commun. Mag., vol.56,
no.10, pp.22–27, Oct. 2018.
[2] P. Liu, Y. Ding, and Tingting Fu , “Optimal ThrowBoxes assignment for big data multicast in VDTNs,” Wireless Networks,
DOI:https://doi.org/10.1007/s11276-019-01974-z, March 2019.
[3] T. Fu, C. Wang, and N. Cheng, “Deep Learning based Joint Optimization of Renewable Energy Storage and Routing in Vehicular Energy
Network,” IEEE Internet Things J., DOI: 10.1109/JIOT.2020.2966660,
Early Access.
[4] X. He, K. Wang, and W. Xu, “QoE-Driven Content-Centric Caching
With Deep Reinforcement Learning in Edge-Enabled IoT,” IEEE Comput. Intell. Mag., vol.14, no.4, pp.12–20, Nov. 2019.
[5] X. He, K. Wang, H. Huang, T. Miyazaki, Y. Wang, and S. Guo,
“Green Resource Allocation based on Deep Reinforcement Learningin Content-Centric IoT,” IEEE Trans. Emerg. Topics Comput., DOI:
10.1109/TETC.2018.2805718, Early Access.

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

[6] N. Hassan, K. A. Yau, and C. Wu, “Edge Computing in 5G: A Review,”
IEEE Access, vol.7, pp.127276–127289, Aug. 2019.
[7] X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, and M. Bennis, “Optimized
Computation Ofﬂoading Performance in Virtual Edge Computing Systems via Deep Reinforcement Learning,” IEEE Internet Things J., vol.6,
no.3, pp.4005–4018, June 2019.
[8] Y. Wang, K. Wang, H. Huang, T. Miyazaki, and S. Guo, “Trafﬁc andcomputation co-ofﬂoading with reinforcement learning in fog computing
for industrial applications,” IEEE Trans. Industrial Informatics, vol.15,
no.2, pp.976–986, Feb. 2019.
[9] J. Feng, Z. Liu, C. Wu, and Y. Ji, “Mobile Edge Computing for Internetof Vehicles: Ofﬂoading Framework and Job Scheduling,” IEEE Veh.
Technol. Mag., vol.14, no.1, pp.28–36, March 2019.
[10] Z. Su, Y. Hui, Q. Xu, T. Yang, J. Liu, and Y. Jia, “An Edge Caching
Scheme to Distribute Content in Vehicular Networks,” IEEE Trans. Veh.
Technol., vol.67, no.6, pp.5346–5356, June 2018.
[11] L. Ale, N. Zhang, H. Wu, D. Chen, and T. Han, “Online Proactive
Caching in Mobile Edge Computing Using Bidirectional Deep Recurrent
Neural Network,” IEEE Internet Things J., vol.6, no.3, pp.5520–5530,
June 2019.
[12] G. Qiao, S. Leng, S. Maharjan, Y. Zhang, and N. Ansari, “DeepReinforcement Learning for Cooperative Content Caching in Vehicular
Edge Computing and Networks,” IEEE Internet Things J., vol.7, no.1,
pp.247–257, Jan. 2020.
[13] J. Feng, Z. Liu, C. Wu, and Y. Ji, “AVE: Autonomous Vehicular Edge
Computing Framework with ACO-Based Scheduling,” IEEE Trans. Veh.
Technol., vol.66, no.12, pp.10660–10675, Dec. 2017.
[14] Y. Wang, P. Lang, D. Tian, J. Zhou, X. Duan, Y. Cao, and D. Zhao,“A Game-based Computation Ofﬂoading Method in Vehicular Multiaccess Edge Computing Networks,” IEEE Internet Things J., DOI:
10.1109/JIOT.2020.2972061, Early Access.
[15] Y. Liu, S. Wang, Q. Zhao, S. Du, A. Zhou, X. Ma, and F. Yang,“Dependency-Aware Task Scheduling in Vehicular Edge Computing,”
IEEE Internet Things J., DOI: 10.1109/JIOT.2020.2972041, Early Access.
[16] X. Zhang, J. Zhang, Z. Liu, Q. Cui, X. Tao, and S. Wang, “MDPbased Task Ofﬂoading for Vehicular Edge Computing under Certain
and Uncertain Transition Probabilities,” IEEE Trans. Veh. Technol., DOI:
10.1109/TVT.2020.2965159, Early Access.
[17] L. T. Tan, R. Q. Hu, “Mobility-Aware Edge Caching and Computing in
Vehicle Networks: A Deep Reinforcement Learning,” IEEE Trans. Veh.
Technol., vol.67, no.11, pp.10190–10203, Nov. 2018.
[18] Z. Ning, K. Zhang, X. Wang, M. S. Obaidat, L. Gu, X. Hu, B. Hu, Y.
Guo, B. Sadoun, and R. Y. K. Kwok, “Joint Computing and Caching in
5G-Envisioned Internet of Vehicles: A Deep Reinforcement LearningBased Trafﬁc Control System,” IEEE Trans. Intell. Transp. Syst., DOI:
10.1109/TITS.2020.2970276, Early Access.
[19] Y. Wu, and J. Zheng, “Modeling and Analysis of the Uplink Local
Delay in MEC-based VANETs,” IEEE Trans. Veh. Technol., DOI:
10.1109/TVT.2020.2970551, Early Access.
[20] S. Zhang, H. Luo, J. Li, W. Shi, X. S. Shen, “HierarchicalSoft Slicing to Meet Multi-Dimensional QoS Demand in CacheEnabled Vehicular Networks,” IEEE Trans. Wireless Commun., DOI:
10.1109/TWC.2019.2962798, Early Access.
[21] D. Wang, Q. Zhang, J. Liu, and D. Yao, “A Novel QoS-Awared Grid
Routing Protocol in the Sensing Layer of Internet of Vehicles Basedon Reinforcement Learning,” IEEE Access, vol.7, pp.185730–185739,
Dec. 2019.
[22] S. Garg, K. Kaur, S. H. Ahmed, A. Bradai, G. Kaddoum, and M. Atiquzzaman, “MobQoS: Mobility-Aware and QoS-Driven SDN Framework
for Autonomous Vehicles,” IEEE Wireless Communications, vol.26,
no.4, pp.12–20, August 2019.
[23] N. Kumar, S. Zeadally, and J. J. P. C. Rodrigues, “QoS-Aware Hierarchical Web Caching Scheme for Online Video Streaming Applications inInternet-Based Vehicular Ad Hoc Networks,” IEEE Trans. Ind. Electron.,
vol.62, no.12, pp.7892–7900, Dec. 2015.
[24] H. Peng, Q. Ye, and X. Shen, “Spectrum Management for Multi-Access
Edge Computing in Autonomous Vehicular Networks,” IEEE Trans.
Intell. Transp. Syst., DOI: 10.1109/TITS.2019.2922656, Early Access.
[25] P. Sun, N. AlJeri, and A. Boukerche, “DACON: A Novel Trafﬁc
Prediction and Data-Highway-Assisted Content Delivery Protocol forIntelligent Vehicular Networks,” IEEE Trans. Sustain. Comput, DOI:
10.1109/TSUSC.2020.2971628, Early Access.
[26] F. Li, X. Song, H. Chen, X. Li, and Y. Wang, “Hierarchical Routing for
Vehicular Ad Hoc Networks via Reinforcement Learning,” IEEE Trans.
Veh. Technol., vol.68, no.2, pp.1852–1865, Feb. 2019.

10[27] C. Wu, X. Chen, T. Yoshinaga, Y. Ji, and Y. Zhang, “Integrating Licensed
and Unlicensed Spectrum in Internet-of-Vehicles with Mobile Edge
Computing,” IEEE Network, vol.33, no.4, pp.48-53, July/August 2019.
[28] L. Nkenyereye, L. Nkenyereye, S. M. R. Islam, C. A. Kerrache, M.
Abdullah-Al-Wadud, and Atif Alamri, “Software Deﬁned NetworkBased Multi-Access Edge Framework for Vehicular Networks,” IEEE
Access, vol.8, pp.4220–4234, Dec. 2019 .
[29] C. Liu, G. Zhang, W. Guo, and R. He, “Kalman Prediction-Based
Neighbor Discovery and Its Effect on Routing Protocol in Vehicular Ad
Hoc Networks,” IEEE Trans. Intell. Transp. Syst., vol.21, no.1, pp.159–
169, Jan. 2020.
[30] S. Kuhlmorgen, H. Lu, A. Festag, J. Kenney, S. Gemsheim, and G.
Fettweis, “Evaluation of Congestion-Enabled Forwarding With Mixed
Data Trafﬁc in Vehicular Communications,” IEEE Trans. Intell. Transp.
Syst., vol.21, no.1, pp.233–247, Jan. 2020.
[31] H. Yang, K. Zheng, L. Zhao, and L. Hanzo, “Twin-Timescale Radio
Resource Management for Ultra-Reliable and Low-Latency Vehicular
Networks,” IEEE Trans. Veh. Technol., vol.69, no.1, pp.1023–1036,
Jan. 2020.
[32] Y. Liu, H. Zhang, K. Long, A. Nallanathan, and V. C. M. Leung,“Energy-Efﬁcient Subchannel Matching and Power Allocation in NOMA
Autonomous Driving Vehicular Networks,” IEEE Wireless Communications, vol.26, no.4, pp.88–93, August 2019.
[33] C. Wu, S. Ohzahata, and T. Kato, “VANET Broadcast Protocol Basedon Fuzzy Logic and Lightweight Retransmission Mechanism,” IEICE
Trans. Commun., vol.95-B, no.2, pp.415–425, Feb. 2012.
[34] C. Wu, S. Ohzahata, and T. Kato, “Flexible, Portable, and Practicable
Solution for Routing in VANETs: A Fuzzy Constraint Q-Learning Approach,” IEEE Trans. Veh. Technol., vol.62, no.9, Nov. 2013, pp.4251–
4263.
[35] The Network Simulator - ns-2, http://www.isi.edu/nsnam/ns/, Accessedon Sep. 29, 2019.

Celimuge Wu (M’10–SM’18) received his ME degree from the Beijing Institute of Technology, China
in 2006, and his PhD degree from The University
of Electro-Communications, Japan in 2010. He iscurrently an associate professor with the Graduate School of Informatics and Engineering, The
University of Electro-Communications. His current
research interests include vehicular ad hoc networks,
sensor networks, intelligent transport systems, IoT,
and mobile cloud computing. He is/has been a TPC
Co-Chair of Wireless Days 2019, ICT-DM 2018, a
track Co-Chair of many international conferences including ICCCN 2019
and IEEE PIMRC 2016. He is/has been serving as an associate editor ofIEEE Access, IEICE Transactions on Communications, International Journalof Distributed Sensor Networks, MDPI Sensors, and a guest editor ofIEEE Transactions on Emerging Topics in Computational Intelligence, IEEE
Computational Intelligence Magazine, ACM/Springer MONET etc.

Zhi Liu (M’14–SM’19) received the B.E., fromthe University of Science and Technology of China,
China and Ph.D. degree in informatics in National
Institute of Informatics. He is currently an Assistant Professor at Shizuoka University. His researchinterest includes video network transmission, vehicular networks and mobile edge computing. He
has served as Guest Editor of journals including
Wireless Communications and Mobile Computing
and IEICE Transactions on communications. He has
been serving as the chair for number of international
conference and workshops. He is a senior member of IEEE.

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING, VOL. XX, NO. XX, XXX 2020

Fuqiang Liu (M’09) received the Bachelor’s degreefrom Tianjin University, Tianjin, China, in 1987 and
the Ph.D. degree from China University of Mining
and Technology, Xuzhou, China, in 1996. He is a
Professor with the School of Electronics and Information Engineering, Tongji University, Shanghai,
China, where he is currently the Director of the
Broadband Wireless Communication and Multimedia Laboratory. He is a Guest Professor with the
National Institute of Informatics, Tokyo, Japan. He
has published more than 300 scientiﬁc papers andnine books. His research mainly focuses on theories and technologies ofbroadband wireless communications (ﬁfth-generation mobile communicationsand vehicular communications/dedicated short-range communications) andtheir applications in automotive and intelligent transportation systems. Dr.
Liu has participated in numerous national research projects in China andhas received research funding from the United States, Finland, the European
Union, and Japan.

Tsutomu Yoshinaga (M) received the BE, ME, andDE degrees from Utsunomiya University in 1986,
1988, and 1997, respectively. From 1988 to July
2000, he was a research associate of the Faculty ofEngineering, Utsunomiya University. He was also
a visiting researcher at Electro-Technical Laboratory from 1997 to 1998. Since August 2000, he
has been with the Graduate School of Information
Systems, The University of ElectroCommunications,where he is now a professor. His research interests
include computer architecture, interconnection networks, and network computing. He is a fellow of IEICE, and a member of
ACM, IEEE, and IPSJ.

Yusheng Ji (M’94–SM’18) received B.E., M.E.,
and D.E. degrees in electrical engineering from the
University of Tokyo. She joined the National Center
for Science Information Systems, Japan (NACSIS)
in 1990. Currently, she is a Professor at the National Institute of Informatics (NII), and SOKENDAI(the Graduate University for Advanced Studies).
Her research interests include network architecture,resource management, and quality of service provisioning in wired and wireless communication networks. She is/has been an Editor of IEEE Transactions on Vehicular Technology, a Symposium Co-chair of IEEE GLOBECOM
2012, 2014, and a Track Co-chair of IEEE VTC2016-Fall, VTC2017-Fall etc.

Jie Li (M’94–SM’04) received the B.Eng. degree in computer science from Zhejiang University,
Hangzhou, China,the M.Eng. degree in electronicengineering and communication systems from the
China Academy of Posts and Telecommunications,
Beijing, China, and the Dr.Eng. degree from the University of Electro-Communications, Tokyo, Japan.
He is currently with Department of Computer Science and Engineering, Shanghai Jiaotong University,
Shanghai, China, where he is also a Chair Professor.
He was a full Professor with the Department ofComputer Science, University of Tsukuba, Japan. He was a Visiting Professor
with Yale University, USA, Inria Sophia Antipolis, and Inria Grenoble-RhoneAlpes, France. His current research interests include big data and AI, cloudand Edge computing, machine learning and networking, network security,
OS, and information system architectures. He has also served on the programcommittees for several international conferences. He is a Co-Chair of theIEEE Technical Community on Big Data, and the Founding Chair of theIEEE ComSoc Technical Committee on Big Data. He serves as an Associate
Editor for many IEEE journals and TRANSACTIONS.

11