# Intelligent Offloading Decision and Resource Allocations Schemes Based on RNN/DQN for Reliability Assurance in Software-Defined Massive Machine-Type Communications

Authors: Sa Math; Prohim Tam; Dae-Young Kim; Seokhoon Kim
Journal: Security and Communication Networks
 (2022)

---

## Abstract

The heterogeneous novelty applications present in the 5th generation (5G) era, including machine-type communication (mMTC), enhanced mobile broadband (eMBB) communication, and ultra-reliable low latency communication (URLLC), which required mobile edge computing (MEC) for local computation and services. The next-generation radio networking (NGRN) will rely on new radio (NR) with the millimeter-wavelength (mmWave) technologies that enable ultra-dense connectivities of the deployed heterogeneous mobile terminal gateways (MTG). However, the mission-critical mMTC applications will suffer from inadequate radio resource management and orchestration (MANO), which will diminish end-to-end (E2E) communication reliability in edge areas. This paper proposed optimal MTG selections and resource allocation (RA) based on the complementary between MTG loading prediction based on recurrent neural network-based long short-term memory (RNN-LSTM) and MTG loading adjustment based on the applied deep reinforcement learning (DRL) approaches, respectively. Furthermore, the RNN-LSTM enhances offloading and handover decisions with discrete-time predictions, while the DRL plays an essential role in adjusting the determined MTG during congestion situations. The proposed method contributed to remarkable outcomes in crucial performance metrics over reference approaches regarding computation and communication quality of service (QoS).


---

## Body

## 1. Introduction

The 5th generation (5G) systems are invented to overcome three mobile applications, including enhancing mobile broadband (eMBB), massive machine type (mMTC), and ultra-reliable and low latency communications (uRLLC), which obligated end-to-end (E2E) wide bandwidth and reliability assurance [1]. Software-defined mobile edge computing (SDMEC) based on the software-defined network (SDN) architecture takes significant roles for both horizontal and vertical management and orchestration (MANO) for virtual edge network infrastructures in terms of computation and communication services [2, 3]. Heterogenous demands are required in heterogeneous applications. The heterogeneity of service level agreement (SLA) is essential for independent servicing with the different application codes. The network function virtualization (NFV) candidate performs virtual network infrastructure (VNI) to support dedicated SLA computing [4, 5]. Furthermore, the SDN controller provides the novelty application programming interface (API) by northbound interface for integrating artificial intelligence (AI) approaches in enabling autonomous closed-loop MANO [6].To empower self-organizing networks (SON) in massive edge communications, the adoption of SDN and AI approaches has been addressed in the autonomous SON in perspective of intelligent communication flow forecasting, mobility predictions, congestion predictions, network failure classifications, decision making, channel allocations, offloading decisions, and autonomous configurations. Recurrent neural network (RNN) based on the specialist long short-term memory (LSTM) approach provides systematically reliable prediction problems for time-sequential issues which meet the network statuses fluctuations at each time step [7–9]. Furthermore, deep reinforcement learning (DRL) approaches are suitable for fluctuation resource adjustment in complicated network environments. The DRL approaches provide direct interactions between the control plane (CP) and data plane (DP) with the real-time state observation module [10–13]. SDN controller adjusts the network statuses based on the agent function.The next-generation mobility networks will contain multitier gateways (in-network gateways, edge gateway, and cloud backup gateway) and communication over high and low power networking. The in-network and edge gateways will be installed by utilizing different computation power gateways. In addition, the in-network gateway will handle private communications, including wireless body area network (WBAN), personal area networking (PAN), and local area network (LAN). The mission-critical applications mainly underline lightweight communications that require ultra-reliable alternative networking, especially during handover periods [14, 15]. However, the alternative processes contain high costs and are unsuitable for light computation and delay constraint applications. Likewise, the mobile edge computing (MEC) server will be attached to each mobile terminal gateway (MTG); thus, each route network loading metrics will be involved in MTG and MEC server communication and computation costs. To guarantee E2E communication reliability for critical application of mMTC, we adopt the resource allocation approaches based on DRL and optimal selection MTG selection based on network loading prediction, which utilizes LSTM. The adoption of DRL and LSTM is conducted based on the reference SDN-based communication architecture.The main contributions of the paper are summarized as follows:(1)
We conduct intelligent local communication MANO for next-generation mMTC real-time applications perspectives. The real-world mMTC aspects in terms of communication conditions, issues, and E2E communication reliability assurance were presented in this research.(2)
We deploy autonomous MTG loading reduction based on applied DRL for intelligent resource adjustment to meet the optimal MTG conditions. SDN controller provides the network configuration, and the CP invests the DRL model for resource allocation. Thus, the complementary SDN controller and DRL approaches adjust the resource constraint scenarios.(3)
We conduct the network loading prediction based on the LSTM approach to determine the handover decisions and perform intelligent MTG selections based on the minimum loading metric.(4)
We conduct the E2E QoS evaluations for the proposed approach. The significant comparisons between the proposed and powerful reference approaches are presented in this study.The rest of the manuscript is structured as follows. The related work is presented in Section2. Section 3 presents the mMTC communication system model, including the communication and computation overheads. Our solution approach is presented in Section 4. The experiment and numerical Table 1evaluation results with detailed interpretation are given in Section 5. Finally, Section 6 presents a conclusion and future work.Table 1
The summary of utilized notations and descriptions.
NotationDefinitionλi,kThe MT requests to MTG/MEC ati and k interfacesμi,kThe resource of MTG/MEC ati and k interfacesTi,kThe computation time interval of the input task of MTG/MEC ati and k interfacesCi,kTThe computation overhead and can be represented by the loading metric of MTG/MEC ati and k interfacesol,kOffloading decision of the SDN controllerTl,klocalLocal computation overhead (current MTG loading)Tl,klocal_totalTotal computation overhead on the networksWi,kτThe bandwidth of the wired betweeni-th and k interfacesQ¯The mean waiting time in the queue of MTG/MECN¯The mean number of requests waiting in each MTG/MEC at time slottPnThe probability ofn number of the task remaining in each MTG/MECTR1,k, TRi,kTR1,k and TRi,k are the transmission rate from first MTG/MEC to k MTG/MEC and the transmission rate current MTG, respectivelyπ∗a|sThe optimal policyπ∗ that performs maximum Q-value in paired state s and action aRtThe reward function at each time slottθ′The target network parameterθThe main network parameterstObservation state at each time slott, S∈Wi,kτ,λi,kτ,Ni,kτ,Qi,kτ,TRi,kτai,kτThe action tacking at thei and k MTG/MEC interfaces, at∈ai,k1,ai,k2,ai,k3,…,ai,ktQ∗s,aThe maximumQ-value in paired state s and action a


## 2. Related Work

In the mMTC systems, the offloading decision methods are challenging in terms of optimal level classifications of the mobile terminal gateways (MTG). Therefore, various MEC servers will integrate the 5G radio access networks (RAN). The MTG handles the communication interfaces between the mobile terminal (MT) and the edge server. The MEC server is integrated with MTG to empower radio serving capability for both computation and communication services. Furthermore, the adopted SDMEC in the fronthaul network enables MANO intelligent resources and services for vertical and horizontal network slicing (NS) [16–18]. The lightweight classification approaches based on machine learning (ML) models have been introduced for resource-constraint application, requiring minimal computation and quick computing [19, 20].On the other side, with the enhancement of RAN gateways, the intelligent edge cloud services rapidly grow with novelty caching methods. The software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS) are mitigated from mobile cloud computing (MCC) infrastructure [21, 22]. LSTM contributes the available channel assignment based on the joined traffic awareness prediction and effectively provides feasible MTG selection [23]. SDN controller handles the gateway configuration and makes the route policy based on the prediction metrics. The RNN-based LSTM structures improve the SDN-based routing reliability.The DRL effectively handles offloading decisions and resource allocation (RA) in massive NS [24–27]. Therefore, the resource constraint of edge components (e.g., base station, network device, etc.) will be obligated to the autonomous resource adjustment. Likewise, the complementary between optimal selection based on RNN-based LSTM and DRL approaches increases the optimal network conditions of each MTG and boosts the communication data rate in RAN areas.


## 3. Massive Machine-Type Communication System Model

This section elaborates the realization of mMTC communication systems in fronthaul edge networks and communication and computation overhead between MT and edge servers.


### 3.1. Massive Machine-Type Communication Architecture

As depicted in Figure1, the mMTC network will be conducted in the fronthaul areas since the remote cloud (e.g., MCC) will be required only when the request offloading task is invalid for local edge computing. Therefore, the reliability of MTG selections will increase the E2E communication quality of services (QoS). Furthermore, the handover processes between the registered MTG and feasible MTG of the MT, especially the vehicle network, will be considered on both reliable gateways and edge servers. Handover processes of MTG-to-MTG, MTG-to-MEC, and MEC-to-MEC will be frequently performed in mobile communications.Figure 1
The mMTC communications in 5G edge infrastructure.


### 3.2. Computation Overhead

The loading metrics between MT and edge server interfaces will increase the E2E communication latency and reduce QoS. For time slotτ∈1,2,3,…,T, the cost Ci,kT of each computation at time T of MT i and edge node k can be denoted as the three tuples of Ci,kTμi,k,λi,k,Ti,k, and μi,k denotes the allocation resource for serving the requested task λi,k with the specific allocation timer assignment Ti,k. Each timer Ti,k can be varied based on the computation task required, and the mission-critical application will be necessary to set the minimum timer and increase the serving resource μi,k metric. The handover processes and offloading requirement comprise two rulemakings, local (without handover) and edge offloading (required handover), represented by ol,k∈0,1. When ol,k=0, the SDN controller will maintain the communication connection of the MT. When ol,k=1, the SDN controller will determine the available communication connection of the requested MT. The loading time Tl,klocal at each network node can be modeled as the poison processes:(1)Ti,klocal=λi,kμi,k.Regarding (1), the total execution time Tl,klocal_total in a single direction from MT to edge sever, which consists of i∈1,2,3,…,n MT and k∈1,2,3,…,m edge servers, can be expressed as follows:(2)Tl,klocal_total=∑0≤i≤n∑0≤k≤mλi,kμi,k.Furthermore, the network overheads of each serving node at the particular queue interface can be determined and modeled asM/M1/K queueing system with the limitation of K capacity of the server. The M/M1/K can be represented by both MT and edge server statues, andρ is the serving ratio between the arriving task λ and serving resource μ. For μ>0, user traffic P0 and n users traffic Pn in the system can be measured as follows: (3a)ρ=λμ,μ>0,(3b)P0=1−ρ1−ρK+1,ρ≠1,1K+1,ρ=1,(3c)Pn=1−ρρn1−ρK+1,ρ≠1,1K+1,ρ=1.The mean waiting timeQ¯ length in the single individual edge server can be measured as(4)Q¯=ρ1−ρ−KKρK+11−ρK+1,ρ≠1,ρK−12K+1,ρ=1.Then, for the mean number of user trafficN¯ in the single individual edge, the system can be modeled as(5)N¯=Q¯+1−P0.


### 3.3. Communication Overhead

The communication overhead will be considered on the computation and communication delays. There arecommunication parametersbetween SDR and MEC (itok interface) in wired networks, including bandwidth W, transmission power P, the noise power between the i to k interface ∂i,k2, and wired gain Hi,k. Thus, the transmission rate from aggregation server to another global server can be expressed as(6)TR1,k=Wlog21+P1H1,k∂k2,k∈2,…,K+1 and i∈2,…,I+1, and i≠k; then, the transmission rate between iandk nodes can be expressed as(7)TRi,k=Wlog21+PiHi,k∂i,k2.The communication overheads will rely on physical data plane (DP) resources (e.g., gateway power, channel bandwidth, link bandwidth, data rate, latency, etc.). At the same time, the minimum communication cost can be performed whenever the SDN controller selects the optimal MTG with high computation power and the lowest delay.


## 3.1. Massive Machine-Type Communication Architecture

As depicted in Figure1, the mMTC network will be conducted in the fronthaul areas since the remote cloud (e.g., MCC) will be required only when the request offloading task is invalid for local edge computing. Therefore, the reliability of MTG selections will increase the E2E communication quality of services (QoS). Furthermore, the handover processes between the registered MTG and feasible MTG of the MT, especially the vehicle network, will be considered on both reliable gateways and edge servers. Handover processes of MTG-to-MTG, MTG-to-MEC, and MEC-to-MEC will be frequently performed in mobile communications.Figure 1
The mMTC communications in 5G edge infrastructure.


## 3.2. Computation Overhead

The loading metrics between MT and edge server interfaces will increase the E2E communication latency and reduce QoS. For time slotτ∈1,2,3,…,T, the cost Ci,kT of each computation at time T of MT i and edge node k can be denoted as the three tuples of Ci,kTμi,k,λi,k,Ti,k, and μi,k denotes the allocation resource for serving the requested task λi,k with the specific allocation timer assignment Ti,k. Each timer Ti,k can be varied based on the computation task required, and the mission-critical application will be necessary to set the minimum timer and increase the serving resource μi,k metric. The handover processes and offloading requirement comprise two rulemakings, local (without handover) and edge offloading (required handover), represented by ol,k∈0,1. When ol,k=0, the SDN controller will maintain the communication connection of the MT. When ol,k=1, the SDN controller will determine the available communication connection of the requested MT. The loading time Tl,klocal at each network node can be modeled as the poison processes:(1)Ti,klocal=λi,kμi,k.Regarding (1), the total execution time Tl,klocal_total in a single direction from MT to edge sever, which consists of i∈1,2,3,…,n MT and k∈1,2,3,…,m edge servers, can be expressed as follows:(2)Tl,klocal_total=∑0≤i≤n∑0≤k≤mλi,kμi,k.Furthermore, the network overheads of each serving node at the particular queue interface can be determined and modeled asM/M1/K queueing system with the limitation of K capacity of the server. The M/M1/K can be represented by both MT and edge server statues, andρ is the serving ratio between the arriving task λ and serving resource μ. For μ>0, user traffic P0 and n users traffic Pn in the system can be measured as follows: (3a)ρ=λμ,μ>0,(3b)P0=1−ρ1−ρK+1,ρ≠1,1K+1,ρ=1,(3c)Pn=1−ρρn1−ρK+1,ρ≠1,1K+1,ρ=1.The mean waiting timeQ¯ length in the single individual edge server can be measured as(4)Q¯=ρ1−ρ−KKρK+11−ρK+1,ρ≠1,ρK−12K+1,ρ=1.Then, for the mean number of user trafficN¯ in the single individual edge, the system can be modeled as(5)N¯=Q¯+1−P0.


## 3.3. Communication Overhead

The communication overhead will be considered on the computation and communication delays. There arecommunication parametersbetween SDR and MEC (itok interface) in wired networks, including bandwidth W, transmission power P, the noise power between the i to k interface ∂i,k2, and wired gain Hi,k. Thus, the transmission rate from aggregation server to another global server can be expressed as(6)TR1,k=Wlog21+P1H1,k∂k2,k∈2,…,K+1 and i∈2,…,I+1, and i≠k; then, the transmission rate between iandk nodes can be expressed as(7)TRi,k=Wlog21+PiHi,k∂i,k2.The communication overheads will rely on physical data plane (DP) resources (e.g., gateway power, channel bandwidth, link bandwidth, data rate, latency, etc.). At the same time, the minimum communication cost can be performed whenever the SDN controller selects the optimal MTG with high computation power and the lowest delay.


## 4. Our Solutions

In our solution, we adopt an SDN controller with DQN for autonomous resource allocation and SDN controller with LSTM-based prediction of real-time loading of MTG for offloading decision. Thus, we decouple the works of real-time MTG predictions and resource allocations for scalability and reducing the computing time of the SDN controller, while the prediction and allocation models can be computed independently.


### 4.1. MTG Loading Metric Prediction

In the time series session of each communication interval of MT in the mMTC, the loading observations of communication stateS with N times series can be modeled as S×N sparse metric. The primary contribution of the LSTM model is to solve the long-term dependency problems, which is powerful for handling the time-discrete loading metrics of the MTG. The LSTM gates significantly control the memory space for long-term related information. In addition, LSTM consists of many gates of controlling the information, which restricts passing through from the cell state. Cell state can be stored in the current subject information or forgotten. The new information will be stored and the old subject forgotten since the input gate (σ sigmoid layer) will be considered on which values will have to be updated. The sigmoid layer will be run to define the part of the state that will be chosen to output. The sigmoid layer plays an essential role in maintaining the participating value for feasible prediction. The cell state will be put through the tanh by multiplying with the sigmoid output gates. This paper defined RNN to work through the common variant LSTM as depicted in Figure 2.Figure 2
Illustration of the peepholes diagram of LSTM.(8)zt=σWzct−1ht−1,xt+bz,rt=σWrct−1ht−1,xt+br,st=tanhWrt∗ct−1,1−ztxt+bs.ct=1−zt∗ct−1+zt∗st,ht=ct−1ht−11+tanh,where z,r,ands are the input values (weight values) to the input gates, including input gate, decision gates (keep or forget), and output gate, respectively. ct is the new state and ht is the hidden state which are the output of the LSTM. The root mean square error (RMSE) of m observation state points was utilized as a prediction loss function of our model at the time t and is expressed as follows:(9)Loss=1m∑i=1myit−y^it,where yit and y^it present the target and predicted values at the time t at the i-th MTG node, respectively. The deviation between the target and predicted values will influence the SDN controller MTG selection and offloading decision.The SDN controller executes a prediction system for determining the appropriate serving entity to allocate the resource. However, this paper considers the MTG selection as the entity that appropriates resource allocation. Thus, the SDN controller considers that the prediction system will be executed for optimal MTG selection for handover or offloading selection after executing the MTG loading adjustment phase. Thus, the network loading prediction will be a part of the routing decisions between MT and MEC servers. Furthermore, in the NG mMTC environments, each MT will be involved in many MTG and MEC servers. Therefore, after executing the loading adjustment phase in heterogeneous MTG, the loading metric of various MTG cannot be insured to be adjusted within the optimal condition. Moreover, the loading adjustment phase is based on the applied software-defined DRL, and the MTG conditions will be varied, reflecting the requests of the massive mobile terminal (MMT). Therefore, the prediction phase is required to be executed to forecast the possibility of loading metrics at each MTG interface.The network loading dataset is the time series data that contains the fluctuation based on the time domain. Based on the experiences of network loading from the previous communications, the feasible loading can be predicted. In this paper, we consider the loading metric of MTG and MEC server as a single interface that integrated thei MTG and k MEC server into i,k interface. In real-world communication, the loading metrics can occur during heavy traffic requests from heterogeneous MT devices. The mTMC communications share analogous features, and the congestion will arise in the bottleneck area while insufficient handling resources, especially during the handover processes and insufficient channel of the MTG. The network dataset was generated using computer software, namely, network simulation version 3 (NS3), a time-discrete C, and C++ programming language. Figure 3 shows the fluctuation of network loading data points for training the LSTM models.Figure 3
The features of the network loading dataset.On the other hand, the proposed scheme handles the offloading decisions based on the time series prediction problem. As depicted in Figure4, the SDN controller decides on the predicted metric. After completing the predictions, the SDN controller will consider the offloading decision based on the size of request tasks. If the task can be executed locally, the route installation will be dismissed. However, if offloading is required, the SDN controller will update the route and configure it to the flow table. In the feasible communications, the RA phase will adjust the network environment. In our communication system, we consider the handover requirement of the mobility and resource limitation into one perspective. Based on Algorithm 1 from line 24, whenever Ci,kT metric in the condition of μi,k>λi,k, then the SDN will maintain the selected connection for communication between MT and edge server. However, whenever Ci,kT metric in the condition of μi,k≤λi,k, we consider the required offloading decision. Thus, the SDN controller establishes the communication by handover to MTG with minimum cost Ci,kT.Figure 4
The autonomous MTG offloading selections based on SDN and LSTM.


### 4.2. MTG Loading Adjustment

We adopt an SDN controller to perform network configuration based on the recommended actionat∈ai,k1,ai,k2,ai,k3,…,ai,kt to handle the predicted loading MTG, as depicted in Figure 4. The observation state S∈Wi,kτ,λi,kτ,Ni,kτ,Qi,kτ,TRi,kτ represents the relative of mMTC state S fluctuations based on the waiting time Wi,kτ, request task λi,kτ, number of queuing tasks Ni,kτ, length of the utilized MTG resources Qi,kτ, and the communication ratio between MT and MTG TRi,kτ. As depicted in Figure 5, at each time step t, the SDN controller matches the action with state Q∗s,a, the controller will receive the immediate reward rt, and the optimal reward will be returned when the optimal policy π∗a|s selects the optimal action in state s. In our approach, the SDN controller obtains the Q-value based on the Bellman optimal equation, as follows:(10)Qπs,a=E∑I=0∞γi|π,st=s,at=a,(11)π∗a|s=argmaxaQθπs,a,(12)Q∗s,a=r+γmaxa′Qθ∗s′,a′.Figure 5
The network condition adjustment of MTG in mMTC based on DRL.As illustrated in (10), (11), and (12), the optimal Q-value represents the expectation metrics with discount factor γ∈0,1, and our targeted network can be defined as yi: (13)yi=ri,γ=0,r+γmaxa′Qθ∗s′,a′,0<γ<1.The prediction error between our main parametersθ and target parameters θ′ of the k sampling from replay buffer D is defined as(14)Lθ=1k∑i=1kyi−Qθsi,ai2.Algorithm 1: SDN-based DQN in MTG resource adjustment and MTG selections.
[DQN-based MTG Resource Adjustment](1)
Initial the main, target parameters, and replay buffer,θ,θ′ and D respectively.DefineN number of episodes(2)
for each step in the  episodes, then(3)
Statest observation(4)
DQNagent selects action based on the optimal policyπ∗a|s(5)
Actionat selection and explore next state s′ and obtain the reward RtAt each time slott, SDN controller executes the action at∈ai,k1,ai,k2,ai,k3,…,ai,kt(6)
if the size of st,at,Rtst′ the size of replay buffer Dcachest,at,Rt,st′ into the replay buffer D(7)
elseReplace queue tail element with the currentst,at,Rt,st′ as the FIFO process.(8)
End ifTransition to next network statest←st′(9)
Random mini-batch ofk samples from replay buffer D(10)
Compute the target network value:(11)
yt=rt+γmaxa′Q∗st′,a′(12)
Compute and minimize the loss:(13)
Lθ=1/k∑i=1kyi−Qθsi,ai2(14)
Update the target networkθ′, based on the updated θ:(15)
Q∗s,a=r+γmaxa′Qθ∗s′,a′(16)
End for[Offloading Decision and MTG Selection]Offloading Making of MTG,referred to Figure 4Input the adjusted MTG statuses observation in each time stepst∈SMonitoring the MTG costCi,kT in each time slot T(17)
while communication is not terminated, then(18)
if the current MTG cost Ci,kT under threshold μi,k>λi,k in allocated Ti,kthenSDN controller maintains the connection without handover processelsetheμi,k≤λi,k in allocated Ti,k, then(19)
SDN controller compares the different predicted loading metrics of each MTG and selects the MTG with minimumCi,kT A new connection is established.(20)
End if(21)
End whileBased on Algorithm 1, the proposed approach adjusts the MTG resource based on the applied DRL approach. The SDN controller configures the RA based on the optimal action recommended by the deep learning agent. The MTG conditions will autonomously decrease the costCi,kT, while the SDN controller will augment the μi,k at each Ti,k for increasing the serving rate in each edge node to reduce loading metric and computation cost. Moreover, the MTG states will be enhanced by reducing the waiting time Wi,kτ of each request at MTG i and MEC k, and the number of requests waiting in the queue-buffer, memory length, and set timer can be adjusted as Ni,kτ,Qi,kτ, and TRi,kτ, respectively. SDN controller can be independently executed between the offloading decision and RA. During the communication after and before offloading processes, the RA can be executed to adjust the loading metric of the edge node. Furthermore, the RA phase maintains the loading network separately from the RA module. In our proposed method, the RA is targeted to apply only to the active MTG device.


## 4.1. MTG Loading Metric Prediction

In the time series session of each communication interval of MT in the mMTC, the loading observations of communication stateS with N times series can be modeled as S×N sparse metric. The primary contribution of the LSTM model is to solve the long-term dependency problems, which is powerful for handling the time-discrete loading metrics of the MTG. The LSTM gates significantly control the memory space for long-term related information. In addition, LSTM consists of many gates of controlling the information, which restricts passing through from the cell state. Cell state can be stored in the current subject information or forgotten. The new information will be stored and the old subject forgotten since the input gate (σ sigmoid layer) will be considered on which values will have to be updated. The sigmoid layer will be run to define the part of the state that will be chosen to output. The sigmoid layer plays an essential role in maintaining the participating value for feasible prediction. The cell state will be put through the tanh by multiplying with the sigmoid output gates. This paper defined RNN to work through the common variant LSTM as depicted in Figure 2.Figure 2
Illustration of the peepholes diagram of LSTM.(8)zt=σWzct−1ht−1,xt+bz,rt=σWrct−1ht−1,xt+br,st=tanhWrt∗ct−1,1−ztxt+bs.ct=1−zt∗ct−1+zt∗st,ht=ct−1ht−11+tanh,where z,r,ands are the input values (weight values) to the input gates, including input gate, decision gates (keep or forget), and output gate, respectively. ct is the new state and ht is the hidden state which are the output of the LSTM. The root mean square error (RMSE) of m observation state points was utilized as a prediction loss function of our model at the time t and is expressed as follows:(9)Loss=1m∑i=1myit−y^it,where yit and y^it present the target and predicted values at the time t at the i-th MTG node, respectively. The deviation between the target and predicted values will influence the SDN controller MTG selection and offloading decision.The SDN controller executes a prediction system for determining the appropriate serving entity to allocate the resource. However, this paper considers the MTG selection as the entity that appropriates resource allocation. Thus, the SDN controller considers that the prediction system will be executed for optimal MTG selection for handover or offloading selection after executing the MTG loading adjustment phase. Thus, the network loading prediction will be a part of the routing decisions between MT and MEC servers. Furthermore, in the NG mMTC environments, each MT will be involved in many MTG and MEC servers. Therefore, after executing the loading adjustment phase in heterogeneous MTG, the loading metric of various MTG cannot be insured to be adjusted within the optimal condition. Moreover, the loading adjustment phase is based on the applied software-defined DRL, and the MTG conditions will be varied, reflecting the requests of the massive mobile terminal (MMT). Therefore, the prediction phase is required to be executed to forecast the possibility of loading metrics at each MTG interface.The network loading dataset is the time series data that contains the fluctuation based on the time domain. Based on the experiences of network loading from the previous communications, the feasible loading can be predicted. In this paper, we consider the loading metric of MTG and MEC server as a single interface that integrated thei MTG and k MEC server into i,k interface. In real-world communication, the loading metrics can occur during heavy traffic requests from heterogeneous MT devices. The mTMC communications share analogous features, and the congestion will arise in the bottleneck area while insufficient handling resources, especially during the handover processes and insufficient channel of the MTG. The network dataset was generated using computer software, namely, network simulation version 3 (NS3), a time-discrete C, and C++ programming language. Figure 3 shows the fluctuation of network loading data points for training the LSTM models.Figure 3
The features of the network loading dataset.On the other hand, the proposed scheme handles the offloading decisions based on the time series prediction problem. As depicted in Figure4, the SDN controller decides on the predicted metric. After completing the predictions, the SDN controller will consider the offloading decision based on the size of request tasks. If the task can be executed locally, the route installation will be dismissed. However, if offloading is required, the SDN controller will update the route and configure it to the flow table. In the feasible communications, the RA phase will adjust the network environment. In our communication system, we consider the handover requirement of the mobility and resource limitation into one perspective. Based on Algorithm 1 from line 24, whenever Ci,kT metric in the condition of μi,k>λi,k, then the SDN will maintain the selected connection for communication between MT and edge server. However, whenever Ci,kT metric in the condition of μi,k≤λi,k, we consider the required offloading decision. Thus, the SDN controller establishes the communication by handover to MTG with minimum cost Ci,kT.Figure 4
The autonomous MTG offloading selections based on SDN and LSTM.


## 4.2. MTG Loading Adjustment

We adopt an SDN controller to perform network configuration based on the recommended actionat∈ai,k1,ai,k2,ai,k3,…,ai,kt to handle the predicted loading MTG, as depicted in Figure 4. The observation state S∈Wi,kτ,λi,kτ,Ni,kτ,Qi,kτ,TRi,kτ represents the relative of mMTC state S fluctuations based on the waiting time Wi,kτ, request task λi,kτ, number of queuing tasks Ni,kτ, length of the utilized MTG resources Qi,kτ, and the communication ratio between MT and MTG TRi,kτ. As depicted in Figure 5, at each time step t, the SDN controller matches the action with state Q∗s,a, the controller will receive the immediate reward rt, and the optimal reward will be returned when the optimal policy π∗a|s selects the optimal action in state s. In our approach, the SDN controller obtains the Q-value based on the Bellman optimal equation, as follows:(10)Qπs,a=E∑I=0∞γi|π,st=s,at=a,(11)π∗a|s=argmaxaQθπs,a,(12)Q∗s,a=r+γmaxa′Qθ∗s′,a′.Figure 5
The network condition adjustment of MTG in mMTC based on DRL.As illustrated in (10), (11), and (12), the optimal Q-value represents the expectation metrics with discount factor γ∈0,1, and our targeted network can be defined as yi: (13)yi=ri,γ=0,r+γmaxa′Qθ∗s′,a′,0<γ<1.The prediction error between our main parametersθ and target parameters θ′ of the k sampling from replay buffer D is defined as(14)Lθ=1k∑i=1kyi−Qθsi,ai2.Algorithm 1: SDN-based DQN in MTG resource adjustment and MTG selections.
[DQN-based MTG Resource Adjustment](1)
Initial the main, target parameters, and replay buffer,θ,θ′ and D respectively.DefineN number of episodes(2)
for each step in the  episodes, then(3)
Statest observation(4)
DQNagent selects action based on the optimal policyπ∗a|s(5)
Actionat selection and explore next state s′ and obtain the reward RtAt each time slott, SDN controller executes the action at∈ai,k1,ai,k2,ai,k3,…,ai,kt(6)
if the size of st,at,Rtst′ the size of replay buffer Dcachest,at,Rt,st′ into the replay buffer D(7)
elseReplace queue tail element with the currentst,at,Rt,st′ as the FIFO process.(8)
End ifTransition to next network statest←st′(9)
Random mini-batch ofk samples from replay buffer D(10)
Compute the target network value:(11)
yt=rt+γmaxa′Q∗st′,a′(12)
Compute and minimize the loss:(13)
Lθ=1/k∑i=1kyi−Qθsi,ai2(14)
Update the target networkθ′, based on the updated θ:(15)
Q∗s,a=r+γmaxa′Qθ∗s′,a′(16)
End for[Offloading Decision and MTG Selection]Offloading Making of MTG,referred to Figure 4Input the adjusted MTG statuses observation in each time stepst∈SMonitoring the MTG costCi,kT in each time slot T(17)
while communication is not terminated, then(18)
if the current MTG cost Ci,kT under threshold μi,k>λi,k in allocated Ti,kthenSDN controller maintains the connection without handover processelsetheμi,k≤λi,k in allocated Ti,k, then(19)
SDN controller compares the different predicted loading metrics of each MTG and selects the MTG with minimumCi,kT A new connection is established.(20)
End if(21)
End whileBased on Algorithm 1, the proposed approach adjusts the MTG resource based on the applied DRL approach. The SDN controller configures the RA based on the optimal action recommended by the deep learning agent. The MTG conditions will autonomously decrease the costCi,kT, while the SDN controller will augment the μi,k at each Ti,k for increasing the serving rate in each edge node to reduce loading metric and computation cost. Moreover, the MTG states will be enhanced by reducing the waiting time Wi,kτ of each request at MTG i and MEC k, and the number of requests waiting in the queue-buffer, memory length, and set timer can be adjusted as Ni,kτ,Qi,kτ, and TRi,kτ, respectively. SDN controller can be independently executed between the offloading decision and RA. During the communication after and before offloading processes, the RA can be executed to adjust the loading metric of the edge node. Furthermore, the RA phase maintains the loading network separately from the RA module. In our proposed method, the RA is targeted to apply only to the active MTG device.


## 5. Experimentation and Numerical Evaluations

We evaluate the proposed system QoS metrics into communication overhead (QoS of DP communications) and computation overhead (conditions of the MTG). The loading metrics of MTG were assessed based on the software-based experiment, which utilized an OpenAI-Gym environment to conduct MTG computation evaluations. In addition, the summary parameters which are utilized to perform the experimentation are listed in Tables2 and 3 for the primary experiment hyperparameters [28–30]. Network simulation version 3 (NS3) was also utilized to perform the E2E mMTC experiment. Plus, the communication conditions of MTG were enumerated to reflect with adjusted loading metrics of DRL outcomes. The MTG was set to 4, mMTC loading between 0 to 250 at each MTG, 20 to 72 Mbps for user data rate, and 9 Gbps for edge link bandwidth. In the experimentation, the MTG and MEC are supposed to be integrated as a single interface between MTG and MEC link. So, the communication overhead between MTG and MEC was dismissed in consideration metrics.Table 2
The experiment components.
ToolsDetailed descriptionOSUbuntu 20.04.2 LTS release 20.04NS330.0Tensorflow-CPU2.3.0Matplotlib3.02RAM16 GBProgramC++, Python (version 3.7.1)Keras2.2.4Gym0.18.0MTG4MTG loading0 to 250MEC server4User data rate20 to 72 MbpsPacket size1024 bytesP2P link9 GbpsP2P link delay2 millisecondsSimulation time430 secondsTable 3
The summary of hyperparameter description.
HyperparametersDetailed valuePredicted lengthLast 81 pointsRNN specialistLSTMNumber of neurons of input and output layers50Mini-batch size32OptimizerAdam/SGDLoss functionRMSEActivation function of 1st hidden layerTanhActivation function of nth hidden layersReLuActivation function of the output layerLinearReplay bufferD5000lr0.02Discount factorγ0≤γ<1Optimal state space (threshold)0≤st≤1PolicyπBoltzmann policyFigure6 depicts the prediction error compared to the actual loading metrics. As illustrated in the given graph, the gap between the predicted and actual loading is under 0.0001, which is the minimum value. Therefore, the minimum loss value shows the maximum accuracy of the SDN controller in terms of the effectiveness of offloading decision. As a result, the CP entities will orchestrate the feasible MTG assigned to serving the MT connections after processing the handover. Therefore, the significantly determinable loading occurring of MTG will reduce the handover loading time and lessen communication failures in ultra-dense radio gateways. Figure 7 shows the symmetry points between the predicted and actual loading metrics that occur during communications. The graphs illustrated that the predicted and actual values are under a single shape. Therefore, our LSTM-based prediction contributes the satisfying outcomes. Each loading metric in discrete-time communication represented the predicted MTG loading during the congested networks. Thus, the SDN controller will decide to continue the current offloading or handover requirement based on each predicted metric.Figure 6
The error values between the predicted and actual loading metrics.Figure 7
The symmetry points between the predicted and actual loading metrics.Figure8 depicts the accumulative rewards between different discount factors values γ=0.5,γ=0.6,γ=0.7,γ=0.8, and γ=0.9. The increasing γ values will also increase the reward number in our proposed environment. The maximum accumulative of reward values represented the SDN controller effectiveness in configuring optimal resources to improve MTG network conditions, while the optimal resource allocation will return the optimal reward value. The optimal rewards also illustrate the ability to handle the congested network in DP. Whenever the loading metric of DP is being adjusted, the communication overhead will be dramatically reduced. However, the SDN controller will suffer from the computation cost, including time and resource constraints, whenever the SDN controller increases the discount factor value close to 1. Furthermore, the optimal discount factor value will be required to balance the agent experiences to explore the optimal action at each communication time-space with the adequate exploration time.Figure 8
The maximum reward accumulation comparisons betweenγ = 0.5, γ = 0.6, γ = 0.7, γ = 0.8, and γ = 0.9.The enhanced reward metrics represented the practical action. The SDN controller achieves optimal MTG condition at the maximum discount factor value, as depicted in Figure9. The maximum of bad state counts was identified at the discount factor γ=0.5. There are two different QoS perspectives between DP and CP. As the SDN controller extends the exploration of expectation allocation action, the high possibility of reducing the loading in DP can be achieved. However, the computation overhead of the CP will be arising corresponding to the exploration periods of the SDN controller. Figures 9 and 10 illustrate optimal condition counts between various discount factor γ values. The graphs presented that γ=0.9 attains the maximum optimal MTG condition during the communications. The SDN controller will be required to adjust the exploration value of γ to meet the computation resources of resource-constraint and real-time communications. However, minimum computation overheads of MTG will dramatically increase the data rates and reliability assurance between MT and edge servers.Figure 9
The good MTG state observations comparisons betweenγ = 0.5, γ = 0.6, γ = 0.7, γ = 0.8, and γ = 0.9.Figure 10
The bad MTG state observations comparisons betweenγ = 0.5, γ = 0.6, γ = 0.7, γ = 0.8, and γ = 0.9.Figure11 illustrates the E2E communication delay of the proposed approach, which applied both prediction and resource allocation models with only prediction-based offloading (RNN approach), the random selection of the MTG (random approach), and experienced-based handling (experience). The graphs showed that the proposed approach achieved remarkable outperformance over the various schemes. The maximum delay that occurred during communication of the proposed scheme, RNN, random, and experience-based approaches is 9.283714 milliseconds, 37.21746 milliseconds, 107.2179 milliseconds, and 207.2595 milliseconds, respectively. Based on the presented metrics, the commutation delays are reduced according to the loading metric of the MTG.Figure 11
The comparisons of E2E communication delays between the proposed and existing approaches.Figure12 presents the E2E communication drop ratio comparisons. The proposed approach reaches the minimum drop ratio of 0.0024582%, while the RNN, random, and experience-based approaches have a minimum drop ratio of 0.00471154%, 0.0147492%, and 0.0500206%, respectively. Corresponding to the E2E communication drop ratio, Figure 13 illustrates the communication reliability comparisons between the proposed and conventional schemes. The proposed scheme reaches the maximum reliability value of 99.9975418%, while the maximum reliability values of the RNN, random, and experience approaches are 99.99528846%, 99.9852508%, and 99.9499794%, respectively. Thus, based on the significant performance metrics in terms of computation and communication QoS, the proposed scheme meets the perspective of URLLC of resource constraint application for next-generation communications.Figure 12
The communication drop ratio comparisons between the proposed scheme, RNN, random, and experience-based approaches.Figure 13
The communication reliability ratio comparisons between the proposed scheme, RNN, random, and experience-based approaches.


## 6. Conclusion and Future Work

This paper presents decoupled MTG loading prediction based on the LSTM approach and MTG loading adjustment based on autonomous resource allocation with the applied DQN in software-defined RAN and SDMEC architectures. The proposed scheme was targeted to cope with the resource limitation of 5G edge network systems for mMTC communications. The decoupling of the prediction model from the resource allocation model meets the large-scale and independency computation problems, which are schedulable for different purposes handling. Furthermore, the proposed approach enhances the offloading decision based on the loading awareness of MTG conditions and reduces the handover process times during alternative gateway requirements. Additionally, the proposed method reaches remarkable performance in both communication and computation metrics. To improve the outcomes of the findings, we will be extending E2E experimentation with a comprehensive analysis of various network environments and large-scale MT devices to reflect the realization of mMTC in 5G systems.



---

*Source: 4289216-2022-04-21.xml*