# Data Transmission Evaluation and Allocation Mechanism of the Optimal Routing Path: An Asynchronous Advantage Actor-Critic (A3C) Approach

Authors: Yahui Ding; Jianli Guo; Xu Li; Xiujuan Shi; Peng Yu
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

The delay tolerant networks (DTN), which have special features, differ from the traditional networks and always encounter frequent disruptions in the process of transmission. In order to transmit data in DTN, lots of routing algorithms have been proposed, like “Minimum Expected Delay,” “Earliest Delivery,” and “Epidemic,” but all the above algorithms have not taken into account the buffer management and memory usage. With the development of intelligent algorithms, Deep Reinforcement Learning (DRL) algorithm can better adapt to the above network transmission. In this paper, we firstly build optimal models based on different scenarios so as to jointly consider the behaviors and the buffer of the communication nodes, aiming to ameliorate the process of the data transmission; then, we applied the Deep Q-learning Network (DQN) and Advantage Actor-Critic (A3C) approaches in different scenarios, intending to obtain end-to-end optimal paths of services and improve the transmission performance. In the end, we compared algorithms over different parameters and find that the models build in different scenarios can achieve 30% end-to-end delay decline and 80% throughput improvement, which show that our algorithms applied in are effective and the results are reliable.


---

## Body

## 1. Introduction

Delay tolerant network (DTN) which has high delay and lower delivery rate is a newly developing network framework, aiming to realize the interconnection and asynchronous data stable transmission in hybrid environment. DTN has a wide range of application, like the sensor networks and the mobile networks, which have obtained the attention and deep research of the academic and industries.Although DTN can be applied in many challenged scenarios, the reliability cannot be guaranteed for the characteristics of discontinuity and randomness. So many scholars have proposed plenty of routing algorithms based on “carry-store-forward” to improve the quality of transmission. The algorithms can be classified into two types of strategies; the first kind of algorithms gets better delivery rate through message copies, like the “Epidemic” algorithm forwards the data in the manner of flooding, but too many copies of the message occupy much memory and increase the overhead of network. The second kind of algorithms forwards the data through classifying, like “First Contact” chooses end-to-end paths randomly to forward data and takes no account of the priori data; “Minimum Expected Delay” uses Dijkstra algorithm to find the path if minimum delay, but which only considers the limited prior knowledge not necessarily global optimal. Although the above algorithms can provide great convenience for us, they also increase the risk if the security of an SDN (Software Defined Network (SDN)) network is compromised. So a new authentication scheme called the hidden pattern (THP) was proposed, which combines graphics password and digital challenge value to prevent multiple types of authentication attacks at the same time [1].DTN can complete the delivery of the message in the complex environments of frequent interruption just because the nodes can store messages. But the above routing algorithms have not considered the memory management, so it is important to determine the optimal end-to-end paths considering the effective management and usage of node capacity.So in our paper, we have did research on the DTN and forming three different scenarios when the communication links break down firstly, and then, we applied the DQN algorithm and A3C algorithm in our proposed optimal models; finally, we compared algorithms over different parameters and find that the models build in different scenarios can achieve 30% end-to-end delay decline and 80% throughput improvement.The main innovation of this paper lies in the following:(i)
We have researched on various scenes and different actions of the nodes, build optimal models in different scenarios(ii)
We adopt the DQN algorithm and A3C algorithm in the optimal models, with the aim of optimizing the throughput of the service data(iii)
We compare the DRL algorithm with other DTN routing algorithms over different parametersThe composition of this paper is as follows. Section1 outlines the characters of DTN and the related works. The optimal models which built over different scenarios can be found in Section 2.Section3 states the procedure and structure of the DQN algorithm and A3C algorithm. Section 4 gives the simulation topology and parameters. Section 5 shows the performances of the algorithms over different simulation parameters and gives the analysis results. At last, Section 6 states the conclusions and future improvements.


## 2. The Outline of DTN

For the existence of the Bundle Layer in DTN, it can implement store-and-forward message switching and the service of custody transfer. These two functions are described in details below.When forwarding messages from the source node to the destination node in TCP/IP network, the messages can query route to find the path in the relay nodes and cannot be stored permanently, for this network has continuous connection to complete the transmission. But in DTN networks, the nodes can store the messages for a period of time and move by carrying the messages when meet the appropriate node to forward the message in the manner of message bundle like the Figure1 shown.Figure 1
Carry-store-forward.After the nodes sending the messages to the next rely node in the form of bundles, if the node has not received the receipt confirmation information from the next node, then the node will choose an appropriate time to forward the bundle again. As Figure2 shown, the relay node will return a receipt to the previous node, and when the relay node forwarding messages to the next hop, the next relay node will also send a receipt to the relay node; this procedure is carried on until the destination node receive the bundle [2]. The purpose of the custody transfer is to increase the reliability of the data transmission; only the node receives the receipt from the next hop, or the message is overdue, or the memory is full; the node deletes the message.Figure 2
Custody transfer.Ensuring DTN completes the service data transmission is important, so the scholars have studied and improved the routing algorithm in specific scenarios and proposed many routing algorithms [3].In view of whether the infrastructure is required in the process of data forwarding, the DTN routing algorithms are composed of infrastructure-aided algorithms and non-infrastructure-aided algorithms as Figure3 shown.Figure 3
DTN routing algorithms.For the above problems, some recent studies [4–6] have proposed efficient cooperative caching schemes, in which data is cached at proper nodes or router nodes with limited sizes. But these papers need long time and large memory to broadcast the services. In [7], a joint optimization framework about caching, computation, and security for delay-tolerant data in M2M communication networks was proposed and adopted deep Q-network (DQN) in the model. In [8], a shortest weighted path-finding problem is formulated to identify the optimal route for secure data delivery between the source–destination pair, which can be solved by employing the Dijkstra’s or Bellman–Ford algorithm.In [9], this paper uses the reliability of travel time as the weight of path selection, and solving by Dijkstra algorithm can reflect the actual vehicle path selection more accurately. This method is a beneficial improvement to the problem of static path selection. A dynamic routing algorithm based on energy-efficient relay selection (RS), referred to as DRA-EERS, is proposed in [10] to adapt to the higher dynamics in time-varying software-defined wireless sensor networks. In [11], a solution to the data advertising problem that is based upon random linear network coding was provided; the simulation results show that the proposed approach is both highly scalable and can significantly decrease the time for advertisement message delivery. A routing architecture and algorithm based on deep neural networks was proposed in [12], which can help routers make packet forwarding decisions based on the current conditions of its surroundings. A limited copy algorithm MPWLC based on service probability was provided in [13]; not only is the number of copies limited but the storage resources of the satellite are also taken into account to ensure reliable data transmission. The simulation results show that the proposed algorithm can effectively improve the efficiency of the network and ensure the reliable data transmission. In [14], a mathematical framework for DTN is introduced and suggested and applies it to a space network that is simulated using an orbital analysis toolkit. In [15], the problem of autonomously avoiding memory overflows in a delay tolerant node was considered, and reinforcement learning was proposed to automate buffer management given that this paper can easily measure the relative rates of data coming in and out of the DTN node.


## 3. Scenarios and System Model

Definition 1 (connected directed graph).
We useG=V,E to denote the connected graph if
(i)
G is a directed graph(ii)
If the connections exist between nodevi∈V and node vj∈V, there will have ei,j∈E
The connected directed graph can be seen in Figure4, the communication nodes responsible for forwarding the messages, the schedule nodes responsible for scheduling of service data, the connections among the nodes are affected by the actual environments.
Assume in our graph that there existN nodes and M links and K services; the communication nodes and schedule nodes can assemble together in V=vi,i=1,2,⋯,N, and the broadband links and narrow-band links can assemble together in E=ej,j=1,2,⋯,M; the service data S=sk,k=1,2,⋯,n transmitting from the initial node vs to the end node vd and the end-to-end paths are expressed by P=pk,k=1,2,..,n=Vpk,Epk, and the time slots are expressed by T=t,t=1,2,..,n.
Due to the exceptional application environments of DQN, which always have long transmission delay and uncertain end-to-end paths, we have researched on the “carry-store-forward” and “custody transfer” mechanism and build optimal models in the following scenarios.
During service data communication, assume all service data fragments start with the shortest end-to-end pathpk. Each source node of service can send sη fragments altogether but can only delivery one fragment fkt in time slot t and ascertain the transmission situations of the fragments which has been delivered before simultaneously. Ifkt indicates the connection status from the node to the next-hop node in time slot t.
(1)Ifkt=1,theconectionisinterrupted,0,theconnectionisconnected.Definition 2 (only consider cache scenario).
We define a scenario only consider cache as Figure5 shown, if
(i)
All fragments choose the shortest paths when sending from the source node(ii)
If the fragments encounter interruption, the fragments only choose to store at the interrupted nodes and wait the nodes return to normal
In this scenario, we believe that every fragmentfkt just chooses to store at the interrupted nodes, but at the same time, the cache data will increase the node cache processing delay αft and the link interrupt waiting delay βft; otherwise, there are no communication links interrupted Ifkt=0; the fragments need to consider the delivery delay on the link γe. This process is iterated until all the fragments reached the destination and calculate the throughput. Suppose the total delay used to complete the service transmission is λ1:
(2)λ1=∑k=1K∑l=1ηml−1⋅γe+∑t=1TIfkt⋅αft+βft.
mk is assumed to be the interrupted node in the shortest path pk, and when all the services accomplish the transmission, the total length of fragments that reached the terminal nodes is ω1, and the throughput is ε1:
(3)ω1=∑k=1Kηfkt,
So the optimal model is:(4)maxε1=∑k=1Kω1v1,e1λ1v1,e1s.t.∀vi∈Vpk,Dvi=Ifkt∗αfkt+βfkt,1∀vi∈Vpk,Dej=γej,2∀vi∈Vps,∀ej∈Eps,DRη=∑l=1η∑i=1mDvi+∑j=1m−1Dej,3∀vi∈Vps,B≥BRk,4∀vi∈Vps,∑n=1k∑t=1TIfktfkt.5
Here,Dvi denotes the total delay when the fragments store in the nodes in (1), Dej denotes the transmission delay in the path in (2), (3) states that the sum of processing delay Dvi and transmission delay Dej cannot exceed the value bound sRk, (4) states that the bandwidth should be enough for the data transmission, and (5) denotes the maximum caching space of every node, so the total cache fragments cannot exceed the target value.Definition 3 (only consider detour scenario).
We define a scenario only considered choosing the detour path as Figure6 shown, if
(i)
All fragments choose the shortest paths when sending from the source node(ii)
If the fragments encounter interruption, the fragments only choose other available paths which have more identical nodes with the initial shortest path, so the fragments choose not to store and wait the nodes return to normal
In this scenario, we believe that every fragmentfkt just chooses other detour paths ps in interrupted nodes Ifkt=1; the shortest path is pk; otherwise, the links are connected Ifkt=0; the fragment just transmitted along the initial shortest path and only takes into account the delivery delay γej. The source nodes need to continuously pay attention to the transmission till all services accomplish data delivery. We assume that the fragments run into interruption in the uk-th node; the transmission delay of every service in the alternate path is DsfRkt. Assume the total delay of all services is λ2.
(5)λ=∑k=1K∑l=1η∑j=1mIfktjk−1γe+DRdfkt,when all the services accomplish the transmission, the total length of fragments that reached the terminal nodes is ω2 as formula (3), and the throughput is ε2, so the optimal model is
(6)maxε2=∑k=1Kω2vi,ejλ2vi,ejs.t.∀vi∈Vps,∀ej∈Eps,DRs=∑i=1u1−Ifktγei,1∀vi∈Vps,∀ej∈Eps,∀vi∈Vpd,∀ej∈Epd,DRs+DRd≤sRk,2∀vi∈Vps,B≥BRs,3∀vi∈Vpd,B≥BRd.4
Here, (1) states the transmission delayDRk in the initial shortest path, DRs denotes the transmission delay of every service in the alternate path, (2) states that the sum of transmission of the entire path cannot be exceed the target value sRk, and (3) and (4) state that the bandwidth of the entire path should be enough for the data transmission.Definition 4 (comprehensive scenario).
We define a scenario comprehensive the end-to-end path as Figure7 shown, if
(i)
All fragments choose the shortest paths when sending from the source node(ii)
If the fragments encounter interruption, the fragments jointly consider choosing other available paths which have more identical nodes with the initial shortest path or storing at the interrupted nodes, at last choose a path has the minimum end-to-end delay after comparing the above circumstances
In this scenario, we believe that every fragmentfkt takes both the storage and the detour paths into account; Uft indicates the choice of the fragment. If the fragments chooses to wait at the nodes Uft=1, which generates the waiting and transmitting delay λ1, which denotes by λ1=∑k=1K∑j=1ηmj−1⋅γe+∑j=1mkIftαft+βft; if the fragment chooses a detour path Ufkt=0 and tries to ensure that the available paths have more identical nodes with the initial shortest path and the total delivery delay λ2 is λ2=∑k=1K∑l=1η∑j=1mIftjk−1⋅γej+DRsft, the fragment will compare the above delays and choose the path that has minimum delay. The source nodes need to continuously pay attention to the transmission till all services accomplish data delivery. Assume the total delay of all services is λ3.
(7)λ3=∑n=1k∑l=1ηJviIfktT1+1−IfktT2+1−Jvimk−1γe,when all the services accomplish the transmission, the total length of fragments that reached the terminal nodes is as formula (3), so the optimal model is as follows:
(8)maxε3=∑k=1Kω3vi,ejλ3vi,ejs.t.∀vi∈Vp,Dvi=αfkt+βfkt,1∀ej∈Eps,Dei=∑u=1uγe,2∀vi∈Vps,∀ej∈Eps,DRη=∑l=1η∑i=1mDvi+∑j=1m−1Dej≤sRk,3∀vi∈Vps,∀ej∈Eps,∀vi∈Vpd,∀ej∈Epd,DRs+DRd≤sRk,4∀vi∈Vps,B≥BRs,5∀vi∈Vpd,B≥BRd,6∀vi∈Vps,∑n=1k∑l=1ηUfkt·fkt≤Cvi.7
Here,Dvi denotes the total delay when the fragments store in the nodes in (1), Dej denotes the transmission delay in the path in (2), Ds indicates the transmission delay of the alternate path in (3), the sum of transmission of the entire path cannot be exceed the target value sRk, (4) states that the sum of processing delay Dvi and transmission delay Dej cannot exceed the value bound sRk, (5) and (6) state that the bandwidth of the entire path should be enough for the data transmission and Bk=minBvi, ∀vi∈Vpk, and in (7), Cvi denotes the maximum caching space of every node, so the total cache fragments cannot exceed the target value.Figure 4
Connected directed graph.Figure 5
Only consider cache scenario.Figure 6
Only consider detour scenario.Figure 7
Comprehensive scenario.


## 4. DRL Algorithm Procedure and Structure

Deep Reinforcement Learning (DRL) is a special and environment-friendly machine learning method, which uses the environment feedback as input and is the learning from environment state to behavior mapping; RL can maximize the cumulative return of system behavior from the environment, mainly consisting of agents and the external environment. But traditional reinforcement learning has bottlenecks; it uses table to save every state and theQ value of every action based on state [16]. And deep Q-learning network (DQN) can adopt neural networks to solve the above problems and make the state and action as the input of neural network; it can obtain the Q value through the neural network not the table, reducing the memory consumption. In DQN, they use experience reply to avoid correlation of data samples, but every time the agent interacts with the environments needs huge memory and computing power, and experience reply can only generate data by the old policy. So, A3C uses multithread of CPU to realize the parallel actor learner for multiagent instance; each thread corresponds to different exploration strategies. This parallelization can be used to decorrelate data and replace experience replay to save storage cost.


### 4.1. DQN Algorithm

Deep Learning has been proved to be a powerful tool to solve nonconvex and high complexity problems and has been widely used in many ways. Reinforcement learning pays more attention to the maximal rewards over a long period time which obtained by interacting with environment and carry out the optimal action. The deep Q-learning adopts deep neural network to develop an action plan and behave well when deal with dynamic time-varying environments. So, DTN provides a promising technology for the data transmission in delay tolerant network.With regard to the reinforcement learning, it interacts with the environment through the agent, which can inspect the environment and obtain the states (t) and then take action at based on the state at the time slot t. Next, the external environment observes the action taken by agent and deliveries the latest state st+1 and the reward rt to the agent. The above process is aimed at finding the maximal reward value by the optimal policy π∗. DQN uses neural networks to approximate the value function Qs,a, and the reward value Q∗s,a can be obtained according to the following:
(9)Q∗s,a=E∗∑s∈Srs,a,s1+γmaxQ∗s1,a1.In which the reward is computed based on the states and the action a, γ represents the discount factor which means the future impact on the present calculation, and E∗∙ denotes the expectation function. Hence, DQN chooses the action which can get the Q value to the maximum.TheQ value updated at every step in DQN as follows:
(10)Qs,a=Qs,a+σrt+1+γmaxQs1,a−Qs,a.In whichσ denotes the leaning rate and should be in the range of [0,1], with the increase of the learning rate, the influence of the past on the present is getting smaller and smaller. The process of the DQN is shown in Figure 8.Figure 8
The work process of DQN algorithm.In DQN,<A, S, R, P > is a typical Quaternions [17], in which Action A indicates the action set taken by agent, State S indicates the state set observed from the environment, Reward R indicates the set of rewards value, and P denotes the deep learn mode in probability state space of agent learning. Based on the above typical Quaternions, the specific definition of DQN is shown as follows:
(1)
State Space. S=X1,X2,⋯,XK, in which it is a vector and denotes the location of the source nodes of the k-th service; the vector has k dimensions, and only one dimension is 1(2)
Action Space. A=A1,A2,⋯,AK, in which it is a vector and denotes the nodes of the k-th service nodes can be connected; the vector has k dimensions, and only one dimension is 1(3)
System Reward. After each time slot t, the system will get the immediate reward rt based on different taken action at. In our paper, we denote the reward as the cost of the current node to other nodes; the reward rt is smaller if the distance between the nodes is longer; otherwise, reward is bigger; rt is shown as follows:(11)rt=distancei,j+c,Distanceisbigger−100,NoConnectionedistancei,j‐c,Distanceissmaller.With above analysis, the DQN procedures for the optimal path and throughput are shown in Algorithm2.


### 4.2. A3C Algorithm

A3C uses the method of multithreading; at the same time, it interacts with the environment in multiple threads, and each thread summarizes the learning results to global net as shown. In addition, each thread regularly takes back the results of common learning from global net to guide the learning interaction between the thread and the environment. Through this method, A3C avoids the problem of strong correlation of experience playback and achieves asynchronous and concurrent learning model.The A3C algorithm is based on the actor-critic consists of the value functionVst,θv and policy function πa∣st;θ, which will not use the traditional Monte Carlo to update the parameters until the end of the scenario episode but uses temporal-difference learning to update parameters in each steps. About the actor-critic, it has two networks; one is the actor network, which is responsible for choosing the actions on account of the policy πa∣st;θ; the critic network is responsible for evaluating each action from the actor network. And after the actor network obtains the scores of the action, it will optimize the policy to get the maximal reward over the algorithm executions. The critic network use the following formula to calculate the score of the actions
(12)6θv=∂R−Vst;θv2∂θ,in which, theR denotes the reward of taking the action a. Through calculating the gradient of the formula (4), the actor network can update the parameter θ; the gradient can be seen from the following:
(13)6θv=R−Vst;θv6θlogπAi∣st;θ.In A3C, it has defined a new function which called advantage functionAs,a that can be shown as
(14)As,a=R−Vst;θ.It expresses that if the action chosen is better than the average, then the advantage function is positive; otherwise, it is negative.Figure9 shows the process of A3C; it has one global network which includes the functions of actor network and critical network. And it also has n workers; each worker has the same network structure as the global neural network, and each worker will interact with the environment independently to get experience data. These workers do not interfere with each other and run independently.Figure 9
The work process of A3C algorithm.Each worker and the environment interact with a certain amount of data, then calculate the gradient of the neural network loss function in its own thread; but these gradients do not update the neural network in its own thread but update the global neural network. In other words,n workers will independently use the accumulated gradient to update the common part of the neural network model parameters. Every once in a while, the thread will update the parameters of its own neural network to the parameters of the common neural network and then guide the subsequent environment interaction.The specific description of A3C algorithm is presented in Algorithm1.


## 4.1. DQN Algorithm

Deep Learning has been proved to be a powerful tool to solve nonconvex and high complexity problems and has been widely used in many ways. Reinforcement learning pays more attention to the maximal rewards over a long period time which obtained by interacting with environment and carry out the optimal action. The deep Q-learning adopts deep neural network to develop an action plan and behave well when deal with dynamic time-varying environments. So, DTN provides a promising technology for the data transmission in delay tolerant network.With regard to the reinforcement learning, it interacts with the environment through the agent, which can inspect the environment and obtain the states (t) and then take action at based on the state at the time slot t. Next, the external environment observes the action taken by agent and deliveries the latest state st+1 and the reward rt to the agent. The above process is aimed at finding the maximal reward value by the optimal policy π∗. DQN uses neural networks to approximate the value function Qs,a, and the reward value Q∗s,a can be obtained according to the following:
(9)Q∗s,a=E∗∑s∈Srs,a,s1+γmaxQ∗s1,a1.In which the reward is computed based on the states and the action a, γ represents the discount factor which means the future impact on the present calculation, and E∗∙ denotes the expectation function. Hence, DQN chooses the action which can get the Q value to the maximum.TheQ value updated at every step in DQN as follows:
(10)Qs,a=Qs,a+σrt+1+γmaxQs1,a−Qs,a.In whichσ denotes the leaning rate and should be in the range of [0,1], with the increase of the learning rate, the influence of the past on the present is getting smaller and smaller. The process of the DQN is shown in Figure 8.Figure 8
The work process of DQN algorithm.In DQN,<A, S, R, P > is a typical Quaternions [17], in which Action A indicates the action set taken by agent, State S indicates the state set observed from the environment, Reward R indicates the set of rewards value, and P denotes the deep learn mode in probability state space of agent learning. Based on the above typical Quaternions, the specific definition of DQN is shown as follows:
(1)
State Space. S=X1,X2,⋯,XK, in which it is a vector and denotes the location of the source nodes of the k-th service; the vector has k dimensions, and only one dimension is 1(2)
Action Space. A=A1,A2,⋯,AK, in which it is a vector and denotes the nodes of the k-th service nodes can be connected; the vector has k dimensions, and only one dimension is 1(3)
System Reward. After each time slot t, the system will get the immediate reward rt based on different taken action at. In our paper, we denote the reward as the cost of the current node to other nodes; the reward rt is smaller if the distance between the nodes is longer; otherwise, reward is bigger; rt is shown as follows:(11)rt=distancei,j+c,Distanceisbigger−100,NoConnectionedistancei,j‐c,Distanceissmaller.With above analysis, the DQN procedures for the optimal path and throughput are shown in Algorithm2.


## 4.2. A3C Algorithm

A3C uses the method of multithreading; at the same time, it interacts with the environment in multiple threads, and each thread summarizes the learning results to global net as shown. In addition, each thread regularly takes back the results of common learning from global net to guide the learning interaction between the thread and the environment. Through this method, A3C avoids the problem of strong correlation of experience playback and achieves asynchronous and concurrent learning model.The A3C algorithm is based on the actor-critic consists of the value functionVst,θv and policy function πa∣st;θ, which will not use the traditional Monte Carlo to update the parameters until the end of the scenario episode but uses temporal-difference learning to update parameters in each steps. About the actor-critic, it has two networks; one is the actor network, which is responsible for choosing the actions on account of the policy πa∣st;θ; the critic network is responsible for evaluating each action from the actor network. And after the actor network obtains the scores of the action, it will optimize the policy to get the maximal reward over the algorithm executions. The critic network use the following formula to calculate the score of the actions
(12)6θv=∂R−Vst;θv2∂θ,in which, theR denotes the reward of taking the action a. Through calculating the gradient of the formula (4), the actor network can update the parameter θ; the gradient can be seen from the following:
(13)6θv=R−Vst;θv6θlogπAi∣st;θ.In A3C, it has defined a new function which called advantage functionAs,a that can be shown as
(14)As,a=R−Vst;θ.It expresses that if the action chosen is better than the average, then the advantage function is positive; otherwise, it is negative.Figure9 shows the process of A3C; it has one global network which includes the functions of actor network and critical network. And it also has n workers; each worker has the same network structure as the global neural network, and each worker will interact with the environment independently to get experience data. These workers do not interfere with each other and run independently.Figure 9
The work process of A3C algorithm.Each worker and the environment interact with a certain amount of data, then calculate the gradient of the neural network loss function in its own thread; but these gradients do not update the neural network in its own thread but update the global neural network. In other words,n workers will independently use the accumulated gradient to update the common part of the neural network model parameters. Every once in a while, the thread will update the parameters of its own neural network to the parameters of the common neural network and then guide the subsequent environment interaction.The specific description of A3C algorithm is presented in Algorithm1.


## 5. Simulation and Analysis

### 5.1. Simulation Parameters

The simulation scenario is shown in Figure10, and we assume to send two services in total, and the services are send in the different source nodes, and every service will send hundreds of data segments; the topology not only has broadband links but also has narrowband links, so the two services have different shortest paths, so when the service segments encounter disruption in the process of service data transmission, they can choose to cache in the interrupted nodes or choose other detour paths to arrive the target nodes. The purple and the yellow links denote the service’s end-to-end path, respectively; the red links denote the interruption in the shortest end-to-end path of every service.Figure 10
Network topology.In the process of simulation, we assume the transmission rate of service data is same in every source node, and the number of data segment cache disunified in every node. And the transmission rate of service segments is the constant value3∗104bps.In the above topology, the source nodes may send many fragments, which can result in block at the interrupted node for too many fragments store at the node, making DQN algorithm consumes more time to analyze the queue problem in the node. If the simulation topology is more complicated, the DQN algorithm needs more time to train and ensure the end-to-end path of services. So in the simulation process of the DQN algorithm, we assume the DQN network consists of three layers of neural network, and the every layer has a certain number of neurons, and the learning rate of DQN is 0.001; the discount factor to calculate the reward is 0.09 [18], and the size of memory pool which used to execute experience replay is 2000. Other specific parameters can get in Table 1; based on the above settings, the DQN algorithm can find the optimal path of services.Table 1
The parameters of simulation.
ParametersSymbolValueTime slott0.1 sFragment sizef1400 bitsMaximum storage space of nodeC10 bitsVelocity of propagationvi3·104 bitsWaiting delayβeiatsCache rateαei700 bpsBandwidthB6000 bpsLearning rate for DQNσD0.001Discount factor for DQNγD0.09Memory pool capacity-DQNN2000Minibatch-DQNb320Execute steps-DQNT5000Global maximum iterations-A3CNtmt1000Maximum iterations in thread-A3CNtst10Discount factor for A3CγA0.9Learning rate for A3C-actorσa0.001Learning rate for A3C-criticσc0.001Entropy factorρ0.001For the execution process of the A3C algorithm, the simulation environment is the same in every worker which is the independent kernel in computer, and we set the learning rate of actor, and critic is 0.001; the discount factor to calculate the reward is 0.09; other specific parameters can get in Table1. Based on the parameters and the special structure of the algorithm, it has higher execution speed and performance than DQN algorithm.We will compare the performance of the algorithm from the following aspects, and the parameters of simulation are shown in Table1.The delivery rate can be expressed as the ratio of the number of fragments reached the destination node and the number of fragments send from the source nodes, as the following shows:(15)D=∑k=1KLdkLs,in whichLdk denotes the number of fragments that reach the destination node of service and Ls denotes the total number of fragments that send from the source nodes.The end-to-end delay can be represented as delay from the time of the source nodes start to send fragments to the time of the last fragment reach the destination, as the following shows:(16)De=td−ts,in which,td is the time of the last fragment reach the destination, and ts is the time of the service data start to transmit.The throughput of service can be expressed as the number of service data successfully transmit from the source node to the destination node per unit time, as the following shows:(17)Th=∑k=1KLdiDe,in which,∑k=1KLdi is the total service data reach the destination node and De is the end-to-end delay.Algorithm 1: Data transmission evaluation of the optimal routing path with A3C algorithm.
Input: The initial locations of the service nodesOutput: The optimal path and throughput of every service1 Initialize the topology of all the nodes and the start and end nodes of every service;2 Initialize thread step countert⟵1;3whileT<=Tmaxdo4 Reset gradients:dθ⟵0 and dθv⟵0;5 Synchronize thread-specific parametersθ1=θ and tstart=t;6 Get statest, that is the start node of everyservice;7whileterminalstnot is the end node of every service ort−tstart!=tmaxdo8  Performat that is the next-hop according to policy πat∣st;θ1;9  If all the constraints in models are satisfied, in consideration of the sequence of the fragments stay at the interrupted node and according to the choice of every fragment, like continue to store at the node or choose other detour paths, which can build different scenarios and have various the next-hopst+1 and reward rt;10t⟵t+1;11T⟵T+1;12end13 SetR=f0forterminalstVst,θv1fornon−terminalstorBootstrapfromlaststate14fori=t−1totstartdo15R⟵ri+γR;16  Accumulate gradients wrtθ1:dθ⟵dθ+R−Vst;θv6θ1logπAi∣st;θ;17  Accumulate gradients wrtθv1:dθv⟵dθv+∂R−Vst;θv2/∂θ;18end19 Perform asynchronous update ofθ using dθ and of θv using dθv;20endThe equilibrium of nodes is the number of average service carrying of every node, as the following shows:(18)N=∑i=1VNip,in which,Ni is the number of service carrying of the i-th node, and P is the total number of nodes in the topology.The equilibrium of links is the number of average service carrying of every link, as the following shows:(19)L=∑i=1ELiQ,in which,Li is the number of service carrying of the i-th link, and Q is the total number of links in the topology.Algorithm 2: Data transmission evaluation of the optimal routing path with DQN algorithm.
Input: The initial locations of the service nodesOutput: The optimal path and throughput of every service1 Initialize replay memoryD to capacity N;2 Initialize action-value functionQ with random weights θ;3 Initialize target action-value functionQ^ with weights θ−=θ;4forepisode=1toNdo5 Initialize the topology of all the nodes;6 Get the initial stateXk of all nodes and the distance between nodes;7 Set sequences1⟵e1,preprocess φ1⟵φs1;8fort=1toTdo9  Select a random actionat for every node i with probability € otherwise selectat=maxaQφst,a;Q;10  Execute actionat in emulator and observe reward rt;11  If all the constraints in models are satisfied, in consideration of the sequence of the fragments stay at the interrupted node and according to the choice of every fragment, like continue to store at the node or choose other detour paths, which can build different scenarios and have various the next-hop, Setst+1⟵st,at,et+1 and preprocess φt+1=φst+1, otherwise go back to step 7; 13 Store transition φt,at,rt,φt+1 in D;12  Sample random mini batch of transitionsφt,at,rt,φt+1 from D;13  Setyj=frjstopsatstepj+1rj+γmaxa1hatQφj+1,a1;θ−Otherwise14  Perform a gradient descent step onyj−Qφj,aj;θ2 with respect to the network parameters θ;15  EveryC step do reset Q^=Q;16end17end


## 5.1. Simulation Parameters

The simulation scenario is shown in Figure10, and we assume to send two services in total, and the services are send in the different source nodes, and every service will send hundreds of data segments; the topology not only has broadband links but also has narrowband links, so the two services have different shortest paths, so when the service segments encounter disruption in the process of service data transmission, they can choose to cache in the interrupted nodes or choose other detour paths to arrive the target nodes. The purple and the yellow links denote the service’s end-to-end path, respectively; the red links denote the interruption in the shortest end-to-end path of every service.Figure 10
Network topology.In the process of simulation, we assume the transmission rate of service data is same in every source node, and the number of data segment cache disunified in every node. And the transmission rate of service segments is the constant value3∗104bps.In the above topology, the source nodes may send many fragments, which can result in block at the interrupted node for too many fragments store at the node, making DQN algorithm consumes more time to analyze the queue problem in the node. If the simulation topology is more complicated, the DQN algorithm needs more time to train and ensure the end-to-end path of services. So in the simulation process of the DQN algorithm, we assume the DQN network consists of three layers of neural network, and the every layer has a certain number of neurons, and the learning rate of DQN is 0.001; the discount factor to calculate the reward is 0.09 [18], and the size of memory pool which used to execute experience replay is 2000. Other specific parameters can get in Table 1; based on the above settings, the DQN algorithm can find the optimal path of services.Table 1
The parameters of simulation.
ParametersSymbolValueTime slott0.1 sFragment sizef1400 bitsMaximum storage space of nodeC10 bitsVelocity of propagationvi3·104 bitsWaiting delayβeiatsCache rateαei700 bpsBandwidthB6000 bpsLearning rate for DQNσD0.001Discount factor for DQNγD0.09Memory pool capacity-DQNN2000Minibatch-DQNb320Execute steps-DQNT5000Global maximum iterations-A3CNtmt1000Maximum iterations in thread-A3CNtst10Discount factor for A3CγA0.9Learning rate for A3C-actorσa0.001Learning rate for A3C-criticσc0.001Entropy factorρ0.001For the execution process of the A3C algorithm, the simulation environment is the same in every worker which is the independent kernel in computer, and we set the learning rate of actor, and critic is 0.001; the discount factor to calculate the reward is 0.09; other specific parameters can get in Table1. Based on the parameters and the special structure of the algorithm, it has higher execution speed and performance than DQN algorithm.We will compare the performance of the algorithm from the following aspects, and the parameters of simulation are shown in Table1.The delivery rate can be expressed as the ratio of the number of fragments reached the destination node and the number of fragments send from the source nodes, as the following shows:(15)D=∑k=1KLdkLs,in whichLdk denotes the number of fragments that reach the destination node of service and Ls denotes the total number of fragments that send from the source nodes.The end-to-end delay can be represented as delay from the time of the source nodes start to send fragments to the time of the last fragment reach the destination, as the following shows:(16)De=td−ts,in which,td is the time of the last fragment reach the destination, and ts is the time of the service data start to transmit.The throughput of service can be expressed as the number of service data successfully transmit from the source node to the destination node per unit time, as the following shows:(17)Th=∑k=1KLdiDe,in which,∑k=1KLdi is the total service data reach the destination node and De is the end-to-end delay.Algorithm 1: Data transmission evaluation of the optimal routing path with A3C algorithm.
Input: The initial locations of the service nodesOutput: The optimal path and throughput of every service1 Initialize the topology of all the nodes and the start and end nodes of every service;2 Initialize thread step countert⟵1;3whileT<=Tmaxdo4 Reset gradients:dθ⟵0 and dθv⟵0;5 Synchronize thread-specific parametersθ1=θ and tstart=t;6 Get statest, that is the start node of everyservice;7whileterminalstnot is the end node of every service ort−tstart!=tmaxdo8  Performat that is the next-hop according to policy πat∣st;θ1;9  If all the constraints in models are satisfied, in consideration of the sequence of the fragments stay at the interrupted node and according to the choice of every fragment, like continue to store at the node or choose other detour paths, which can build different scenarios and have various the next-hopst+1 and reward rt;10t⟵t+1;11T⟵T+1;12end13 SetR=f0forterminalstVst,θv1fornon−terminalstorBootstrapfromlaststate14fori=t−1totstartdo15R⟵ri+γR;16  Accumulate gradients wrtθ1:dθ⟵dθ+R−Vst;θv6θ1logπAi∣st;θ;17  Accumulate gradients wrtθv1:dθv⟵dθv+∂R−Vst;θv2/∂θ;18end19 Perform asynchronous update ofθ using dθ and of θv using dθv;20endThe equilibrium of nodes is the number of average service carrying of every node, as the following shows:(18)N=∑i=1VNip,in which,Ni is the number of service carrying of the i-th node, and P is the total number of nodes in the topology.The equilibrium of links is the number of average service carrying of every link, as the following shows:(19)L=∑i=1ELiQ,in which,Li is the number of service carrying of the i-th link, and Q is the total number of links in the topology.Algorithm 2: Data transmission evaluation of the optimal routing path with DQN algorithm.
Input: The initial locations of the service nodesOutput: The optimal path and throughput of every service1 Initialize replay memoryD to capacity N;2 Initialize action-value functionQ with random weights θ;3 Initialize target action-value functionQ^ with weights θ−=θ;4forepisode=1toNdo5 Initialize the topology of all the nodes;6 Get the initial stateXk of all nodes and the distance between nodes;7 Set sequences1⟵e1,preprocess φ1⟵φs1;8fort=1toTdo9  Select a random actionat for every node i with probability € otherwise selectat=maxaQφst,a;Q;10  Execute actionat in emulator and observe reward rt;11  If all the constraints in models are satisfied, in consideration of the sequence of the fragments stay at the interrupted node and according to the choice of every fragment, like continue to store at the node or choose other detour paths, which can build different scenarios and have various the next-hop, Setst+1⟵st,at,et+1 and preprocess φt+1=φst+1, otherwise go back to step 7; 13 Store transition φt,at,rt,φt+1 in D;12  Sample random mini batch of transitionsφt,at,rt,φt+1 from D;13  Setyj=frjstopsatstepj+1rj+γmaxa1hatQφj+1,a1;θ−Otherwise14  Perform a gradient descent step onyj−Qφj,aj;θ2 with respect to the network parameters θ;15  EveryC step do reset Q^=Q;16end17end


## 6. Simulation Results

### 6.1. Simulation Algorithms

The “Epidemic” algorithm belongs to the spread routing; it forwards the data in the manner of flooding; that is to say, all nodes encountered will “infect” the message [19], so there will have lots of copies of the message in the network and occupy much memory and increase the overhead of network.But in the circumstance of sufficient network resources, the Epidemic algorithm will show faster delivery rate and be the alternative algorithm in this circumstance. If there are too many messages to transmit, some messages may be discarded, resulting in higher packet loss rate.The ED algorithm has not taken the queue problem into consideration, and routing path is determined when the source nodes send the service data, so the ED algorithm belongs to the source routing. But when there have more fragments and queuing, it will affect the computation of weight of the ED algorithm; thus, making the computation of the source route emerges errors and cannot produce the optimal path.The MED algorithm has taken the transmission delay, the propagation delay, and the average waiting delay into consideration; the goal of the algorithm is to find the path of minimum delay, and the path adopted is identical when the source nodes and destination nodes are same. After ensuring the path of source routing, even there has better choice, this algorithm will not change the routing choices [20], so it is just the optimal path over the limited prior knowledge and not necessarily global optimal, so the MED algorithm belongs to “time-invariant” algorithm.Deep Learning has been proved to be a powerful tool to solve nonconvex and high complexity problems and has been widely used in many ways. The deep Q-learning adopts deep neural network to develop an action plan and behaves well when deal with dynamic time-varying environments.A3C uses the method of multithreading; at the same time, it interacts with the environment in multiple threads, and each thread summarizes the learning results to global net. In addition, each thread regularly takes back the results of common learning from global net to guide the learning interaction between the thread and the environment.


### 6.2. Simulation Results

#### 6.2.1. Comparison of End-to-End Paths

The end-to-end paths of different algorithms can be seen in Figures11–13; owing to the different operating mechanism of algorithms, they choose different paths when encountering the interrupted nodes. We can see that only the DQN algorithm and A3C algorithm can change the route when facing different scenarios, and they often choose the paths that have the minimum end-to-end paths. In the comprehensive scenario, the DQN algorithm and A3C algorithm have compared the end-to-end delay of the above scenarios and have the optimal path in the three scenarios.Figure 11
Process of the above algorithms.Figure 12
Process of the DQN.Figure 13
Process of the A3C.


#### 6.2.2. Comparison of Delivery Rate

In this paper, we broadcast 2 services in this topology and record delivery rate based on different link break delay over different algorithms and assume the minimum delivery rate is 0.75. It is shown in Figure14 that as the link break delay increases, the delivery rate is decrease over majority of algorithms, but in the only consider detour scenario and the comprehensive scenario of DQN, the delivery rate is remain unchanged and is the maximum, because in these scenarios, the source nodes choose the detour path and not affected by the interrupted links, so increasing the delivery rate. Through the result of all algorithms shown in Table 2, we found that the DQN algorithm has higher delivery rate in the majority of circumstances, which shows the delivery rate improvement of our proposed models and the DQN algorithm. But the A3C algorithm has the highest delivery rate in every scenario, because the A3C has many subthreads which can find the optimal paths in a very short time. The A3C algorithm and DQN algorithm can satisfy the constrained delivery rate in most cases.Figure 14
The delivery rate.Table 2
The delivery rate.
Delivery rateAlgorithmsEpidemicEDMEDDQN-sce1DQN-sce2DQN-sce3A3CInterrupted delay300.75111111400.750.50.51111500.750.250.50.5111600.7500.250.25111700.7500.250111


#### 6.2.3. Comparison of End-to-End Delay

In DTN network, due to the particularity of data connection, there may be open circuit between data connections, which makes data have to be stored in the node waiting for the link to be connected again. However, the storage space at the node is limited. When some greedy algorithms are adopted, multiple data storage may cause the use of nodes pace, so that when the following data arrives, it will cause data loss.The waiting delay at the node is reflected in the total end-to-end delay of data transmission, including not only the transmission delay on the link but also the waiting delay at the node. When using epidemic, ED, and other algorithms, due to the particularity of the algorithm, it will copy multiple copies of data to be transmitted in the network, so compared with DQN and other reinforcement learning algorithms, it will increase the waiting delay at the node.Because the transmission of multiple copies of data will cause congestion at the node and when the transmission continues, it will increase the queuing delay. For the transmission of data, intelligent algorithms such as DQN will not transmit multiple copies of the same data but take the reward in the algorithm as the guidance, minimize the end-to-end delay in the network, and reduce the occurrence of congestion at the node, so the end-to-end delay of each algorithm is shown in the figure below.The total transmission delay of the service 1 and service 2 is shown in Figures15–18; it can be seen that in the three scenarios, the minimum transmission delay is 100 ms, and the A3C algorithm has the minimum transmission delay, and DQN algorithm has lower delay, but in the scenario 1, the total transmission delay of service 1 and service 2 is increased as the interrupted delay of link is increasing, and the specific data are shown in Table 3, because in this scenario when the fragments encounter the interruption, they choose to store at the node, so the delay keeps on rising. Because in our paper, we think the capacity of node is available, so over the Epidemic algorithm, the fragments can arrive the destination node smoothly, and the total transmission delay of the Epidemic is not too high. But over the ED algorithm, it has the highest transmission delay; for the ED algorithm is the source routing algorithm, it determines the transmission path when the source nodes send the fragments, so when the fragments encounter the interruption, they will not change the path and result the high delay.Figure 15
The end-to-end delay of service 1.Figure 16
The end-to-end delay of service 1.Figure 17
The end-to-end delay of service 1.Figure 18
The end-to-end delay of service 2.Table 3
The end-to-end delay of service 2.
End-to-end delayAlgorithmsEpidemicEDMEDDQN-sce1DQN-sce2DQN-sce3A3CInterrupted delay301101001008080808040110130110100808080501101401101208080806011015011012080808070110160110140808080


#### 6.2.4. Comparison of Throughput

The throughput of service is shown in Figures19–21; it can be seen that in the three scenarios, the A3C algorithm has the maximum throughput, which is better than the DQN algorithm; in the consider cache scenario, the throughput is decrease, for in this the scenario the transmission delay is a little higher and the nodes that reach the destination node are not a lot. The specific throughput data of all algorithms are shown in Table 4, and the throughput of ED algorithm and MED algorithm of service 1 and service 2 is also decrease, because the delay is increase as the interrupted delay of link increase. But in the only consider detour and comprehensive consider scenarios, there has the maximum throughput, which can be proved that our models and the adopted algorithm have improved the transmission.Figure 19
The throughput of service 1.Figure 20
The throughput of service 1.Figure 21
The throughput of service 1.Table 4
The throughput of service in the scenario of comprehensive.
ThroughputAlgorithmsEpidemicEDMEDDQNA3CInterrupted delay302.12.12.252.252.2540222.252.252.255020.90.922.256020022.257020022.258020022.25


#### 6.2.5. Comparison of Node Equilibrium and Link Equilibrium

The node equilibrium of services can be seen from Figure22; we can see the Epidemic algorithm has the maximum node equilibrium; for in the process of the algorithm, it forwards the fragments over the flooding manner, which means when the node comes into the communication scope of other nodes, if the node found that other nodes have not the fragment, it will send the fragment to the other nodes. So it can lead to many copies of fragments in the work, so every node may store every fragment of every service, so the equilibrium is the highest. The node equilibrium of the ED algorithm and the MED algorithm is a little lower, and average node equilibrium of DQN is a little higher, which has to be improved. And for the link equilibrium, the Epidemic algorithm has the maximum link equilibrium; the cause of the result is the same to the node equilibrium, because it forward too many duplicates in the network. The average link equilibrium of DQN is a little higher but not very large as the node equilibrium, so it also has to be improved, so the A3C algorithm has improved the node and link equilibrium, which has lower equilibrium compared to the DQN algorithm.Figure 22
The node and link equilibrium of services.


#### 6.2.6. Comparison of Total Reward and Loss

From Figure23, we can see that the A3C algorithm has higher reward and can converge very quickly. At first, the value of reward is random jitter, because the exact value cannot be obtained in a short time. And the reward of A3C can get close to its top value with 400 episodes, but DQN needs about 700 episodes. The results prove that our models and algorithms adopted can reach an optimal value and converge.Figure 23
The node and link equilibrium of services.


## 6.1. Simulation Algorithms

The “Epidemic” algorithm belongs to the spread routing; it forwards the data in the manner of flooding; that is to say, all nodes encountered will “infect” the message [19], so there will have lots of copies of the message in the network and occupy much memory and increase the overhead of network.But in the circumstance of sufficient network resources, the Epidemic algorithm will show faster delivery rate and be the alternative algorithm in this circumstance. If there are too many messages to transmit, some messages may be discarded, resulting in higher packet loss rate.The ED algorithm has not taken the queue problem into consideration, and routing path is determined when the source nodes send the service data, so the ED algorithm belongs to the source routing. But when there have more fragments and queuing, it will affect the computation of weight of the ED algorithm; thus, making the computation of the source route emerges errors and cannot produce the optimal path.The MED algorithm has taken the transmission delay, the propagation delay, and the average waiting delay into consideration; the goal of the algorithm is to find the path of minimum delay, and the path adopted is identical when the source nodes and destination nodes are same. After ensuring the path of source routing, even there has better choice, this algorithm will not change the routing choices [20], so it is just the optimal path over the limited prior knowledge and not necessarily global optimal, so the MED algorithm belongs to “time-invariant” algorithm.Deep Learning has been proved to be a powerful tool to solve nonconvex and high complexity problems and has been widely used in many ways. The deep Q-learning adopts deep neural network to develop an action plan and behaves well when deal with dynamic time-varying environments.A3C uses the method of multithreading; at the same time, it interacts with the environment in multiple threads, and each thread summarizes the learning results to global net. In addition, each thread regularly takes back the results of common learning from global net to guide the learning interaction between the thread and the environment.


## 6.2. Simulation Results

### 6.2.1. Comparison of End-to-End Paths

The end-to-end paths of different algorithms can be seen in Figures11–13; owing to the different operating mechanism of algorithms, they choose different paths when encountering the interrupted nodes. We can see that only the DQN algorithm and A3C algorithm can change the route when facing different scenarios, and they often choose the paths that have the minimum end-to-end paths. In the comprehensive scenario, the DQN algorithm and A3C algorithm have compared the end-to-end delay of the above scenarios and have the optimal path in the three scenarios.Figure 11
Process of the above algorithms.Figure 12
Process of the DQN.Figure 13
Process of the A3C.


### 6.2.2. Comparison of Delivery Rate

In this paper, we broadcast 2 services in this topology and record delivery rate based on different link break delay over different algorithms and assume the minimum delivery rate is 0.75. It is shown in Figure14 that as the link break delay increases, the delivery rate is decrease over majority of algorithms, but in the only consider detour scenario and the comprehensive scenario of DQN, the delivery rate is remain unchanged and is the maximum, because in these scenarios, the source nodes choose the detour path and not affected by the interrupted links, so increasing the delivery rate. Through the result of all algorithms shown in Table 2, we found that the DQN algorithm has higher delivery rate in the majority of circumstances, which shows the delivery rate improvement of our proposed models and the DQN algorithm. But the A3C algorithm has the highest delivery rate in every scenario, because the A3C has many subthreads which can find the optimal paths in a very short time. The A3C algorithm and DQN algorithm can satisfy the constrained delivery rate in most cases.Figure 14
The delivery rate.Table 2
The delivery rate.
Delivery rateAlgorithmsEpidemicEDMEDDQN-sce1DQN-sce2DQN-sce3A3CInterrupted delay300.75111111400.750.50.51111500.750.250.50.5111600.7500.250.25111700.7500.250111


### 6.2.3. Comparison of End-to-End Delay

In DTN network, due to the particularity of data connection, there may be open circuit between data connections, which makes data have to be stored in the node waiting for the link to be connected again. However, the storage space at the node is limited. When some greedy algorithms are adopted, multiple data storage may cause the use of nodes pace, so that when the following data arrives, it will cause data loss.The waiting delay at the node is reflected in the total end-to-end delay of data transmission, including not only the transmission delay on the link but also the waiting delay at the node. When using epidemic, ED, and other algorithms, due to the particularity of the algorithm, it will copy multiple copies of data to be transmitted in the network, so compared with DQN and other reinforcement learning algorithms, it will increase the waiting delay at the node.Because the transmission of multiple copies of data will cause congestion at the node and when the transmission continues, it will increase the queuing delay. For the transmission of data, intelligent algorithms such as DQN will not transmit multiple copies of the same data but take the reward in the algorithm as the guidance, minimize the end-to-end delay in the network, and reduce the occurrence of congestion at the node, so the end-to-end delay of each algorithm is shown in the figure below.The total transmission delay of the service 1 and service 2 is shown in Figures15–18; it can be seen that in the three scenarios, the minimum transmission delay is 100 ms, and the A3C algorithm has the minimum transmission delay, and DQN algorithm has lower delay, but in the scenario 1, the total transmission delay of service 1 and service 2 is increased as the interrupted delay of link is increasing, and the specific data are shown in Table 3, because in this scenario when the fragments encounter the interruption, they choose to store at the node, so the delay keeps on rising. Because in our paper, we think the capacity of node is available, so over the Epidemic algorithm, the fragments can arrive the destination node smoothly, and the total transmission delay of the Epidemic is not too high. But over the ED algorithm, it has the highest transmission delay; for the ED algorithm is the source routing algorithm, it determines the transmission path when the source nodes send the fragments, so when the fragments encounter the interruption, they will not change the path and result the high delay.Figure 15
The end-to-end delay of service 1.Figure 16
The end-to-end delay of service 1.Figure 17
The end-to-end delay of service 1.Figure 18
The end-to-end delay of service 2.Table 3
The end-to-end delay of service 2.
End-to-end delayAlgorithmsEpidemicEDMEDDQN-sce1DQN-sce2DQN-sce3A3CInterrupted delay301101001008080808040110130110100808080501101401101208080806011015011012080808070110160110140808080


### 6.2.4. Comparison of Throughput

The throughput of service is shown in Figures19–21; it can be seen that in the three scenarios, the A3C algorithm has the maximum throughput, which is better than the DQN algorithm; in the consider cache scenario, the throughput is decrease, for in this the scenario the transmission delay is a little higher and the nodes that reach the destination node are not a lot. The specific throughput data of all algorithms are shown in Table 4, and the throughput of ED algorithm and MED algorithm of service 1 and service 2 is also decrease, because the delay is increase as the interrupted delay of link increase. But in the only consider detour and comprehensive consider scenarios, there has the maximum throughput, which can be proved that our models and the adopted algorithm have improved the transmission.Figure 19
The throughput of service 1.Figure 20
The throughput of service 1.Figure 21
The throughput of service 1.Table 4
The throughput of service in the scenario of comprehensive.
ThroughputAlgorithmsEpidemicEDMEDDQNA3CInterrupted delay302.12.12.252.252.2540222.252.252.255020.90.922.256020022.257020022.258020022.25


### 6.2.5. Comparison of Node Equilibrium and Link Equilibrium

The node equilibrium of services can be seen from Figure22; we can see the Epidemic algorithm has the maximum node equilibrium; for in the process of the algorithm, it forwards the fragments over the flooding manner, which means when the node comes into the communication scope of other nodes, if the node found that other nodes have not the fragment, it will send the fragment to the other nodes. So it can lead to many copies of fragments in the work, so every node may store every fragment of every service, so the equilibrium is the highest. The node equilibrium of the ED algorithm and the MED algorithm is a little lower, and average node equilibrium of DQN is a little higher, which has to be improved. And for the link equilibrium, the Epidemic algorithm has the maximum link equilibrium; the cause of the result is the same to the node equilibrium, because it forward too many duplicates in the network. The average link equilibrium of DQN is a little higher but not very large as the node equilibrium, so it also has to be improved, so the A3C algorithm has improved the node and link equilibrium, which has lower equilibrium compared to the DQN algorithm.Figure 22
The node and link equilibrium of services.


### 6.2.6. Comparison of Total Reward and Loss

From Figure23, we can see that the A3C algorithm has higher reward and can converge very quickly. At first, the value of reward is random jitter, because the exact value cannot be obtained in a short time. And the reward of A3C can get close to its top value with 400 episodes, but DQN needs about 700 episodes. The results prove that our models and algorithms adopted can reach an optimal value and converge.Figure 23
The node and link equilibrium of services.


## 6.2.1. Comparison of End-to-End Paths

The end-to-end paths of different algorithms can be seen in Figures11–13; owing to the different operating mechanism of algorithms, they choose different paths when encountering the interrupted nodes. We can see that only the DQN algorithm and A3C algorithm can change the route when facing different scenarios, and they often choose the paths that have the minimum end-to-end paths. In the comprehensive scenario, the DQN algorithm and A3C algorithm have compared the end-to-end delay of the above scenarios and have the optimal path in the three scenarios.Figure 11
Process of the above algorithms.Figure 12
Process of the DQN.Figure 13
Process of the A3C.


## 6.2.2. Comparison of Delivery Rate

In this paper, we broadcast 2 services in this topology and record delivery rate based on different link break delay over different algorithms and assume the minimum delivery rate is 0.75. It is shown in Figure14 that as the link break delay increases, the delivery rate is decrease over majority of algorithms, but in the only consider detour scenario and the comprehensive scenario of DQN, the delivery rate is remain unchanged and is the maximum, because in these scenarios, the source nodes choose the detour path and not affected by the interrupted links, so increasing the delivery rate. Through the result of all algorithms shown in Table 2, we found that the DQN algorithm has higher delivery rate in the majority of circumstances, which shows the delivery rate improvement of our proposed models and the DQN algorithm. But the A3C algorithm has the highest delivery rate in every scenario, because the A3C has many subthreads which can find the optimal paths in a very short time. The A3C algorithm and DQN algorithm can satisfy the constrained delivery rate in most cases.Figure 14
The delivery rate.Table 2
The delivery rate.
Delivery rateAlgorithmsEpidemicEDMEDDQN-sce1DQN-sce2DQN-sce3A3CInterrupted delay300.75111111400.750.50.51111500.750.250.50.5111600.7500.250.25111700.7500.250111


## 6.2.3. Comparison of End-to-End Delay

In DTN network, due to the particularity of data connection, there may be open circuit between data connections, which makes data have to be stored in the node waiting for the link to be connected again. However, the storage space at the node is limited. When some greedy algorithms are adopted, multiple data storage may cause the use of nodes pace, so that when the following data arrives, it will cause data loss.The waiting delay at the node is reflected in the total end-to-end delay of data transmission, including not only the transmission delay on the link but also the waiting delay at the node. When using epidemic, ED, and other algorithms, due to the particularity of the algorithm, it will copy multiple copies of data to be transmitted in the network, so compared with DQN and other reinforcement learning algorithms, it will increase the waiting delay at the node.Because the transmission of multiple copies of data will cause congestion at the node and when the transmission continues, it will increase the queuing delay. For the transmission of data, intelligent algorithms such as DQN will not transmit multiple copies of the same data but take the reward in the algorithm as the guidance, minimize the end-to-end delay in the network, and reduce the occurrence of congestion at the node, so the end-to-end delay of each algorithm is shown in the figure below.The total transmission delay of the service 1 and service 2 is shown in Figures15–18; it can be seen that in the three scenarios, the minimum transmission delay is 100 ms, and the A3C algorithm has the minimum transmission delay, and DQN algorithm has lower delay, but in the scenario 1, the total transmission delay of service 1 and service 2 is increased as the interrupted delay of link is increasing, and the specific data are shown in Table 3, because in this scenario when the fragments encounter the interruption, they choose to store at the node, so the delay keeps on rising. Because in our paper, we think the capacity of node is available, so over the Epidemic algorithm, the fragments can arrive the destination node smoothly, and the total transmission delay of the Epidemic is not too high. But over the ED algorithm, it has the highest transmission delay; for the ED algorithm is the source routing algorithm, it determines the transmission path when the source nodes send the fragments, so when the fragments encounter the interruption, they will not change the path and result the high delay.Figure 15
The end-to-end delay of service 1.Figure 16
The end-to-end delay of service 1.Figure 17
The end-to-end delay of service 1.Figure 18
The end-to-end delay of service 2.Table 3
The end-to-end delay of service 2.
End-to-end delayAlgorithmsEpidemicEDMEDDQN-sce1DQN-sce2DQN-sce3A3CInterrupted delay301101001008080808040110130110100808080501101401101208080806011015011012080808070110160110140808080


## 6.2.4. Comparison of Throughput

The throughput of service is shown in Figures19–21; it can be seen that in the three scenarios, the A3C algorithm has the maximum throughput, which is better than the DQN algorithm; in the consider cache scenario, the throughput is decrease, for in this the scenario the transmission delay is a little higher and the nodes that reach the destination node are not a lot. The specific throughput data of all algorithms are shown in Table 4, and the throughput of ED algorithm and MED algorithm of service 1 and service 2 is also decrease, because the delay is increase as the interrupted delay of link increase. But in the only consider detour and comprehensive consider scenarios, there has the maximum throughput, which can be proved that our models and the adopted algorithm have improved the transmission.Figure 19
The throughput of service 1.Figure 20
The throughput of service 1.Figure 21
The throughput of service 1.Table 4
The throughput of service in the scenario of comprehensive.
ThroughputAlgorithmsEpidemicEDMEDDQNA3CInterrupted delay302.12.12.252.252.2540222.252.252.255020.90.922.256020022.257020022.258020022.25


## 6.2.5. Comparison of Node Equilibrium and Link Equilibrium

The node equilibrium of services can be seen from Figure22; we can see the Epidemic algorithm has the maximum node equilibrium; for in the process of the algorithm, it forwards the fragments over the flooding manner, which means when the node comes into the communication scope of other nodes, if the node found that other nodes have not the fragment, it will send the fragment to the other nodes. So it can lead to many copies of fragments in the work, so every node may store every fragment of every service, so the equilibrium is the highest. The node equilibrium of the ED algorithm and the MED algorithm is a little lower, and average node equilibrium of DQN is a little higher, which has to be improved. And for the link equilibrium, the Epidemic algorithm has the maximum link equilibrium; the cause of the result is the same to the node equilibrium, because it forward too many duplicates in the network. The average link equilibrium of DQN is a little higher but not very large as the node equilibrium, so it also has to be improved, so the A3C algorithm has improved the node and link equilibrium, which has lower equilibrium compared to the DQN algorithm.Figure 22
The node and link equilibrium of services.


## 6.2.6. Comparison of Total Reward and Loss

From Figure23, we can see that the A3C algorithm has higher reward and can converge very quickly. At first, the value of reward is random jitter, because the exact value cannot be obtained in a short time. And the reward of A3C can get close to its top value with 400 episodes, but DQN needs about 700 episodes. The results prove that our models and algorithms adopted can reach an optimal value and converge.Figure 23
The node and link equilibrium of services.


## 7. Conclusions

In this paper, we have proposed optimal models based on different scenarios consist of the only consider cache scenario, the only consider detour scenario, and comprehensive consider scenario; the models are intend to jointly consider the behavior and the buffer of the nodes to improve the performance of the data transmission. Owing to different choices of the nodes, there will form three scenarios, and we adopted the DQN algorithm to solve the complex nonlinear optimization problem and to get the optimal solutions, which consist of lower end-to-end delay, higher throughput, and better data delivery guarantees. The results of simulation show that compared to other algorithms like Epidemic, ED algorithm, and MED algorithm, the DQN algorithm we adopted has better performance improvement.As future work, we are going to improve the optimal models and decrease the overhead of nodes and links, expecting the application of DQN can be further studied in the delay tolerant networks.



---

*Source: 6685722-2021-09-03.xml*