# Offloading in Mobile Edge Computing Based on Federated Reinforcement Learning

Authors: Yu Dai; Qing Xue; Zhen Gao; Qiuhong Zhang; Lei Yang
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Mobile edge computing (MEC) has become a more and more popular technology, which plays a very important role in various fields. In view of the task of offloading of multiple users, most of the existing studies do not take into account data sharing and cooperation among users, which can easily lead to less generalization of the model trained by a single user, and some data sharing may also cause privacy leakage. Then, this paper uses the method of federated reinforcement learning to solve this problem in order to insure privacy. Besides, considering the poor quality of local models, which leads to the poor versatility of the overall parameters, this paper proposes a federated reinforcement learning method based on Attention mechanism to aggregate the parameter weights, which makes the new model more generalized. The experimental results show that the federated reinforcement learning task offloading model with Attention mechanism can reduce the processing delay of the task.


---

## Body

## 1. Introduction

In recent decades, the development of the Internet and wireless communication technology provides a very convenient channel for exchanging information in people’s daily life. Internet of things has been used in smart homes, intelligent transportation, human health detection, disaster management, and many other fields. In order to solve the problem of big data computing, some researchers have introduced mobile cloud computing (MCC) [1] service based on the centralized cloud data center to cope with the rapid increase in a number of mobile applications and the long-term limitations of battery technology. However, the offloading of MCC only reduces the energy consumption burden of the mobile device, but at the same time, because the data transmission is very complex, the delay for the mobile device to obtain the processing result data is very large. In 2014, the European Telecommunication Standards Association (ETSI) explained the basic concepts and various aspects of MEC (Mobile Edge Computing) [2], placing the MEC server near the place where the data was generated, enabling the server to process user requests at the edge of the network, reducing transmission and location-aware delays, reducing network load, and improving the user experience.Offloading computing tasks to edge servers is usually affected by many factors, such as software and hardware environment, network bandwidth, wireless channel interference, connection quality of communication links, overall performance of mobile devices, and so on. Therefore, offloading computing tasks in MEC is a key research issue, that is, whether computing tasks are executed locally or offloaded from edge nodes. In addition, many mobile applications in real scenarios (e.g., face recognition, gesture recognition, and augmented reality [3]) are composed of dependent tasks; that is, there are storage dependencies and computational dependencies between tasks. Therefore, it is unrealistic to offload some computing tasks to the edge server and ignore the relationship between them. In addition, the existing models based on heuristic or reinforcement learning offloading algorithms lack generalization due to the lack of data sharing among devices, and each device only trains the model locally. At the same time, some methods require the device to send the user’s personal data to the central server, which may lead to serious user privacy and data security problems.In the case of dealing with the offloading problem of dependent tasks, the computational offloading method proposed in this paper uses federated reinforcement learning (FRL) to solve the problem of weak generalization and privacy leakage caused by the lack of data transmission between users. At the same time, this paper considers that the trained models are different due to the different computing power and data quality of each mobile device, and it is unreasonable to simply aggregate each device model through average, so this paper considers the weighted method (i.e., weighted federation aggregation) and proposes a federation reinforcement learning task offloading model which integrates Attention [4] mechanism.The rest of this paper is organized as follows. Section2 introduces the related work of task computing offloading. Section 3 gives the task model and computing model of computing task offloading. Section 4 details the computational task offloading network model combined with reinforcement learning and proposes a federated reinforcement learning computing task offloading algorithm with Attention mechanism. Section 5 verifies the effectiveness of the proposed method through simulation analysis. Section 6 summarizes the full text.


## 2. Related Work

Recently, researchers have done a lot of research on solving the problem of task offloading, which can be roughly divided into the following two types of algorithms: one is the use of traditional algorithms such as heuristic or approximation. In the previous research, some research works considered whether the task of the mobile device was offloaded to the edge node, while others considered which edge computing node to offload the computing task to. Also, some works apply Reinforcement Learning (RL) method to make offloading decision. RL is an effective method for computing offloading decision. Considering the reward feedback from the environment in the future state, the RL agents can adjust their strategies to achieve the best long-term goals, which solves the problem of low efficiency of traditional methods [5]. Therefore, RL is a very important solution in the dynamic MEC system.


### 2.1. Traditional Algorithms Such as Heuristics

The mobile cloud computing system with multiple users and multiple tasks is considered in [6]. On the premise of minimizing the total cost, an effective approximate solution MUMTO uses a separable semistereotyped relaxation method to solve the binary offloading decision. In [7], the total network revenue is considered as the optimization objective, and the offloading problem is transformed into a convex problem, and an optimization algorithm based on multiplier alternating direction method is proposed to solve the problem of task offloading and content caching. From the point of view of prolonging the life of mobile devices, a dynamic offloading algorithm based on Lyapunov optimization is proposed in [8], which achieves the goal of energy saving on the premise of meeting the execution time requirements of a given application. The above calculation offloading method does not take into account that the offloaded tasks may include computational dependencies, which may result in the extra communication cost between the tasks, while the algorithm proposed in this paper takes into account the offloading of dependent tasks and modeling.A new algorithm is proposed in [9]. Starting from a minimum-delay scheduling scheme, tasks are migrated between the local core and the cloud, and dynamic voltage and frequency scaling techniques are applied to achieve energy saving. Aiming at the problem of task migration, a linear time rescheduling algorithm is proposed. In [10], the authors model the application as a directed acyclic graph (DAG). The UE side of the user equipment decides which subtask should be offloaded to which compute node (AP). In their work, the problem is described as a scheduling problem. In order to solve the NP-hard problem, the author proposes a heuristic algorithm based on the list scheduling algorithm. The algorithm takes into account of the transmission time between AP and the offloading time from UE to AP. Reference [11] proposes a minimum cost offload partition (MCOP) algorithm, which aims to find the optimal partition scheme under different cost models and mobile environments (i.e., to determine which parts of the application must run on mobile devices and which parts run on cloud or edge servers). In [12], the authors propose a heuristic mobility-aware offloading algorithm (HMAOA) to obtain the approximate optimal offloading scheme. The original global optimization problem is converted into multiple local optimization problems. Each local optimization problem is then decomposed into two subproblems.References [9–12] are all based on heuristic or approximate algorithms to solve the problem of task offloading in DAG diagrams. When the external environment of the MEC system changes, the model may not be suitable for the changing environment, so it is necessary to update the mathematical model accordingly. Therefore, these algorithms are difficult to fully adapt to the dynamic changing MEC environment. The algorithm proposed in this paper effectively solves the problem that it is difficult to adapt to the changing environment based on heuristic or approximate algorithms by using reinforcement learning.


### 2.2. Offloading Algorithm Based on Reinforcement Learning

In recent years, deep reinforcement learning has been widely concerned by academia. In Reinforcement Learning [13], the agent does not need to know part of the information of the network in advance and achieves the optimization goal of maximizing the long-term benefits of energy consumption and delay by adjusting strategies. Researchers have recently begun to use deep reinforcement learning algorithms to solve this problem [14].In [15], an optimization framework based on deep Q network (DQN) is proposed to solve the problem of joint task offload and resource allocation in wireless MEC when minimizing energy consumption and delay. Reference [16] considers that in a super-dense network, each task can select and offload multiple base stations, and the DQN algorithm is used to obtain the optimal offloading strategy. Considering the load of edge nodes in [17], and the arrival deadlines of some tasks may not be processed and discarded, the authors propose a model-free distributed reinforcement learning algorithm. The model trained by the offloading algorithm based on reinforcement learning lack generalization due to the lack of data exchange between devices, and each device trains the model locally.In addition, some methods require users to send data from this device to a central server, which may cause serious user privacy disclosure and sensitive data security problems, even the transmission of anonymous data will still put user privacy at risk, because attackers can recover by comparing anonymous information with other data [18]. Federated Learning is a concept proposed by Google in 2016 to solve privacy issues on mobile devices. Federated Reinforcement Learning (FRL) [19] is based on the sharing of experiences among agents so that more samples can be collected in a short period of time. On the other hand, the global model of federated reinforcement learning training is to aggregate the user model with weighted average, which leads to the problem of rigid average contribution of user model to global training parameters. In this paper, an FRL task offloading model with Attention mechanism is proposed from the point of view of protecting users’ privacy.


## 2.1. Traditional Algorithms Such as Heuristics

The mobile cloud computing system with multiple users and multiple tasks is considered in [6]. On the premise of minimizing the total cost, an effective approximate solution MUMTO uses a separable semistereotyped relaxation method to solve the binary offloading decision. In [7], the total network revenue is considered as the optimization objective, and the offloading problem is transformed into a convex problem, and an optimization algorithm based on multiplier alternating direction method is proposed to solve the problem of task offloading and content caching. From the point of view of prolonging the life of mobile devices, a dynamic offloading algorithm based on Lyapunov optimization is proposed in [8], which achieves the goal of energy saving on the premise of meeting the execution time requirements of a given application. The above calculation offloading method does not take into account that the offloaded tasks may include computational dependencies, which may result in the extra communication cost between the tasks, while the algorithm proposed in this paper takes into account the offloading of dependent tasks and modeling.A new algorithm is proposed in [9]. Starting from a minimum-delay scheduling scheme, tasks are migrated between the local core and the cloud, and dynamic voltage and frequency scaling techniques are applied to achieve energy saving. Aiming at the problem of task migration, a linear time rescheduling algorithm is proposed. In [10], the authors model the application as a directed acyclic graph (DAG). The UE side of the user equipment decides which subtask should be offloaded to which compute node (AP). In their work, the problem is described as a scheduling problem. In order to solve the NP-hard problem, the author proposes a heuristic algorithm based on the list scheduling algorithm. The algorithm takes into account of the transmission time between AP and the offloading time from UE to AP. Reference [11] proposes a minimum cost offload partition (MCOP) algorithm, which aims to find the optimal partition scheme under different cost models and mobile environments (i.e., to determine which parts of the application must run on mobile devices and which parts run on cloud or edge servers). In [12], the authors propose a heuristic mobility-aware offloading algorithm (HMAOA) to obtain the approximate optimal offloading scheme. The original global optimization problem is converted into multiple local optimization problems. Each local optimization problem is then decomposed into two subproblems.References [9–12] are all based on heuristic or approximate algorithms to solve the problem of task offloading in DAG diagrams. When the external environment of the MEC system changes, the model may not be suitable for the changing environment, so it is necessary to update the mathematical model accordingly. Therefore, these algorithms are difficult to fully adapt to the dynamic changing MEC environment. The algorithm proposed in this paper effectively solves the problem that it is difficult to adapt to the changing environment based on heuristic or approximate algorithms by using reinforcement learning.


## 2.2. Offloading Algorithm Based on Reinforcement Learning

In recent years, deep reinforcement learning has been widely concerned by academia. In Reinforcement Learning [13], the agent does not need to know part of the information of the network in advance and achieves the optimization goal of maximizing the long-term benefits of energy consumption and delay by adjusting strategies. Researchers have recently begun to use deep reinforcement learning algorithms to solve this problem [14].In [15], an optimization framework based on deep Q network (DQN) is proposed to solve the problem of joint task offload and resource allocation in wireless MEC when minimizing energy consumption and delay. Reference [16] considers that in a super-dense network, each task can select and offload multiple base stations, and the DQN algorithm is used to obtain the optimal offloading strategy. Considering the load of edge nodes in [17], and the arrival deadlines of some tasks may not be processed and discarded, the authors propose a model-free distributed reinforcement learning algorithm. The model trained by the offloading algorithm based on reinforcement learning lack generalization due to the lack of data exchange between devices, and each device trains the model locally.In addition, some methods require users to send data from this device to a central server, which may cause serious user privacy disclosure and sensitive data security problems, even the transmission of anonymous data will still put user privacy at risk, because attackers can recover by comparing anonymous information with other data [18]. Federated Learning is a concept proposed by Google in 2016 to solve privacy issues on mobile devices. Federated Reinforcement Learning (FRL) [19] is based on the sharing of experiences among agents so that more samples can be collected in a short period of time. On the other hand, the global model of federated reinforcement learning training is to aggregate the user model with weighted average, which leads to the problem of rigid average contribution of user model to global training parameters. In this paper, an FRL task offloading model with Attention mechanism is proposed from the point of view of protecting users’ privacy.


## 3. System Model

In this paper, we consider an environment consisting of a set of local mobile devicesC=1,2,…M, a base station (BS), and an MEC server connected to the BS, as shown in Figure 1. At the same time, considering that an application can be decomposed into multiple interdependent tasks in real life, this paper considers multidependent tasks where the optimization goal is to achieve the least time to complete all tasks in a task-dependent graph. For each mobile device, there is a task graph G containing dependent tasks that needs to be processed. When the task arrives randomly, the task graph is calculated and processed in the fashion of First Input First Output (FIFO). In this paper, a task graph is represented as GT,E, where T represents the set of task vertices, E represents a set of edges, and e=ti,tj represents a directed edge from ti to tj, indicating that there is data transferring from ti to tj, where ti is the parent node of tj, which can only be executed after the completion of ti execution, emphasizing that the order in which the tasks are executed cannot be changed. The optimization goal of computing the offloading problem is to make an offloading decision for each task in the DAG diagram, that is, whether the task is executed locally or whether the task as a whole is offloaded to the MEC through channel transmission. For a DAG diagram of any user, all task offloading decisions are represented as A1:n=a1,a2,…,an, where ai represents the offloading decision of the user’s ti task, which is executed locally on the device when ai=0 and offloaded to MEC when ai=1. Tasks are processed sequentially on order of FIFO and can be processed only when the task arrives.Figure 1
System model.The local execution timeTiL and the MEC execution time Tic can be calculated by the following formulae individually:(1)TiL=dataifL,(2)Tic=dataif,where datai is the amount of data in task ti, fL is the local CPU cycles of the device, and f is the CPU cycles of the MEC. If a task needs to be offloaded to the edge node, it will result in uplink transmission delay Tiul as formula (3). If a task needs to transmit results locally from the edge node, it will generate a delay Tidl in the downlink as shown in formula (4):(3)Tiul=dataisRul,(4)Tidl=datairRdl,where Rul is the uplink transfer rate between the mobile device and the MEC and Rdl is the downlink transfer rate between the MEC and the mobile device. datais is the amount of data that a task sends from the device to the MEC, and datair is the amount of data of the task results downloaded from the edge server for the device.In the process of local computing, computing at edge nodes, and uplink and downlink transmission, there may be some cases in which resources are occupied, so the start execution time of a task is not only affected by the task dependency graph, but also affected by the longer waiting time caused by the occupation of resources. The local start execution time of the taskti is RTiL, and the start execution time of the task ti at MEC is RTic, as shown in formulae (5) and (6). The computation of start times for uploading and downloading task data RTiul and RTidl is shown in formulae (7) and (8):(5)RTiL=maxFiL,maxk∈predtiFTkL,FTkdl,(6)RTic=maxFic,maxFTiul,maxk∈predtiFTkc,(7)RTiul=maxFiul,maxk∈predtiFTkL,FTkdl,(8)RTidl=maxFidl,FTic,where FiL=maxFi−1L,FTi−1L is the start time of locally available resources, Fic=maxFi−1c,FTi−1c is the start time of the resources available on the MEC, Fiul=maxFi−1ul,FTi−1ul is the start time of the available resources in the uplink of the channel, and Fidl=maxFi−1dl,FTi−1dl is the start time of the available resources in the downlink of the channel. The local execution end time FTiL, the end time of MEC FTic, the end time of uplink transmission FTiul, and the end time of downlink transmission FTidl are expressed as shown in the following formulae:(9)FTiL=RTiL+TiL,(10)FTic=RTic+Tic,(11)FTiul=RTiul+Tiul,(12)FTidl=RTidl+Tidl.In this paper, we define the optimization goal as minimizing the execution delayTA1:n. Because it calculates the delay of the tasks in the DAG graph, it needs to be executed by all nodes, that is, to minimize the execution time of the end node, as shown in the following formula:(13)TA1:n=maxk∈KFTkL,FTkdl,where K is the node with a degree of 0 in the task graph (i.e., the exited task), which stipulates that the start and end of the task graph need to be executed locally.


## 4. Network Model

### 4.1. Reinforcement Learning Offloading Model

Because the deep reinforcement learning algorithm is learned from the interaction with the environment and does not need a priori knowledge of the system, it has been widely used in communication and network [20].


#### 4.1.1. Task Modeling MDP Process

In this paper, each task is modeled as a Markov decision process (MDP). The MDP process is formally expressed asP=S,A,R, S represents the state of the agent during the interaction between the agent and the environment, A represents the action selected by the agent in a certain state of the environment, R represents the reward obtained by the agent after performing the action, and the state, action, and reward work together to form the environment of agent interaction. The definitions of state, action, and reward are as follows:(1)
The offloading of each task is related to the amount of data of each task in the DAG task graph where the task node is located, the required CPU cycles, and the shape of the DAG graph. The offloading of taskti is related to the offloading of task t1 to task ti in DAG because the offloading of task t1 to task ti may affect the start execution time of task ti. In this paper, the state is defined as follows:(14)state:DATA,TCPU,GT,E,A1:i,whereDATA is the amount of data of each task, Tcpu is the number of CPU cycles required for each task to process, and GT,E is the embedding of the graph (for each task, including the index of the current task, the index of its direct parent task, and the index of direct child tasks), and A1:i is the offloading decision from task t1 to task ti.(2)
There are two kinds of action choices for each task: one is to execute locally; the other is to offload it to the MEC to execute. In this paper, the action is defined as follows:(15)action:0,1.(3)
The purpose of this paper is to minimize the total task execution delayTA1:n. Define the reward as shown in formula (16). Because the optimization goal of this paper is to minimize the total task execution delay, the total execution delay is composed of the processing delay of each task, and the agent is given a reward after each task interacts with the environment, so reward is defined as the processing time of each task. The smaller the delay is, the larger the reward value is, and vice versa.(16)reward:TA1:i−1−TA1:i.


#### 4.1.2. seq2seq Structure

Because of the interdependent relationship between tasks, this paper uses the seq2seq [21] structure as the component model of reinforcement learning. The seq2seq model includes the Encoder and the Decoder, where both the Encoder and Decoder are composed of the LSTM model, as shown in Figure 2.Figure 2
Task offloading architecture.The input of Encoder isDATA,TCPU,GT,E in state (i.e., the embedded vector t1,t2,…,tn of a task graph, where n represents the number of tasks contained in each graph). Decoder outputs the offloading decision of a task graph, and the action is expressed as a1,a2,…,an. In order to convert the DAG task graph into embedded vectors and input into the model, the tasks are sorted and indexed according to the ascending order of the critical paths, and the formula is as follows:(17)rankti=Tio,ifti∈K,Tio+maxtj∈childtiranktj,ifti∉K,where Tio=minTiL,Tic+Tiul+Tidl indicates the delay of completing the task, and K is the end node of the task.The functions of Encoder and Decoder are set togenc and gdec, and the output ei of Encoder is represented as shown in formula (18). The aggregate intermediate vector c of the Encoder encoded output ei and the dj in Decoder can be expressed as shown in formulae (19) and (20):(18)ei=gencti,ei−1,(19)c=1n∑i=0nei,(20)dj=gdecc,aj−1,dj−1,where dj is connected to two fully connected layers that approximately represent the vsj and πaj|sj of task tj, where vsj and πaj|sj represent value functions and policy functions, respectively. During training, the action aj is generated by strategy πaj|sj sampling, where aj=maxπaj|sj represents the execution action of task tj.


### 4.2. An Offloading Model of Federated Reinforcement Learning Based on Attention Mechanism

In federated learning, multiple device nodes participate in the training and aggregate the model weights shared by the service BS nodes. In order to make better use of the parameter information provided by each equipment node, the service BS node needs to allocate different weight coefficients according to their contribution. Considering the difference between equipment models, the Attention mechanism can be applied to achieve the demand. In this paper, an FRL task offloading model based on Attention mechanism is proposed, and the training process is as follows:(i)
Associate the mobile device with the serving BS and report the local mobile device status to the BS. After receiving the information, BS sends the previous round of global model parameters to the local device to perform the steps of model release.(ii)
When the local mobile device learns that it can participate in the training, it receives the global model parameters obtained from the BS.(iii)
Each mobile device begins to train its local model parameters based on global model parameters and local data. After the local training, each mobile device collects its training evaluation indicators (e.g., average reward, average loss, etc.) during the training phase and sends them and local model parameters to BS. Then, BS calculates the aggregation weight of each equipment according to the training evaluation index. In this paper, attention mechanism is used to provide aggregation weights for different mobile devices. The update process for weighted federation aggregation is shown in Figure3.Figure 3
Federated Reinforcement Learning model of Attention mechanism.In this model, the indexes ofm mobile devices are input into the Encoder of the model, and Query is the optimal goal in Decoder. By comparing the similarity between the indexes of each mobile device model and the optimal goal, the contribution weight of each device model to the optimal goal is calculated. In this paper, the average loss, average reward and the computing power of mobile devices are used to calculate the attention weight of the contribution of the global model.(i)
Average loss: the quality of the model is related to the size of the loss, the greater the loss means that the model cannot well deal with the problems to be solved. On the contrary, the smaller the loss, the better the performance of the model, so we can consider using it. So, the loss can be evaluated on a model. The average lossLavg is the average calculation of the loss function of a certain period of time by a device.(ii)
Average reward: contrary to the average loss, the average reward can also evaluate the quality of the model. When the reward is larger, the effect of the model is better, and when the reward is smaller, the model may have some deviation from the problem solved. Therefore, the average reward can also evaluate the model. Average rewardRavg is the average reward for the mobile device over a certain period of time.(iii)
Computing power of mobile devices: the computing power of mobile devices can affect whether tasks need to be offloaded. When the computing power of mobile devices is large, the delay caused by local processing tasks is usually far less than the delay caused by transmitting task data, so the device will get a better user experience effect when the tasks are processed locally. On the contrary, if the processing power of the mobile device is too small, the processing delay will be very long, so it is better to offload the task to the edge server. Therefore, the computing power of the mobile devicecomu also has a certain impact on the model parameters.In this paper, we use the above indexK=Lavg,Ravg,comu to evaluate the attention mechanism. The goal of the training model is to get more rewards and achieve the minimum loss. Therefore, the optimization goal set in this paper is shown in formula (21). Q consists of minimizing the loss function, maximizing the reward, and maximizing the computing power.(21)Q=minLavg,maxRavg,maxcomu.Then, the weight of the contribution of the device is calculated by using the Attention mechanism. In this paper, we use the method of dot product to calculate the similarity betweenQ and K to get the corresponding weight, and then use the softmax function to normalize the weight, and introduce the weight factor wu to calculate the contribution of the local model to the global model. The specific calculation is shown as follows:(22)wu=AttentionQ,K=softmaxQKTdk,where dk represents the dimension of K, from which the contribution of each mobile device model is calculated by using the Attention mechanism, and the weight of the model is weighted to calculate the parameters of the global model. After aggregation, this parameter is sent to each local model. The optimized global parameter aggregation formula is shown in the following formula. The specific algorithm is shown in Algorithm 1.(23)Fθt=∑u∈UwuFθtu.Algorithm 1: A Federated Reinforcement Learning task offloading model integrated with Attention mechanism.
(1)
for each equipment in parallel do(2)
for local stepj=1,…,n do(3)
for eachstep do(4)
Sample actiona∼πθs, reward and s′(5)
Inputs and s′ to get Qws′,a′ and Qws,a(6)
δ=r+γQws′,a′−Qws,a(7)
θ=θ+α∇θlogπθs,aQws,a\(8)
w←w+βδ(9)
a←a′, s←s′(10)
end for(11)
K=Lavg,Ravg,comu(12)
end for(13)
Q=minLavg,maxRavg,maxcomu(14)
wu=AttentionQ,K(15)
Fθt=∑u∈UwuFθtu(16)
end for


## 4.1. Reinforcement Learning Offloading Model

Because the deep reinforcement learning algorithm is learned from the interaction with the environment and does not need a priori knowledge of the system, it has been widely used in communication and network [20].


### 4.1.1. Task Modeling MDP Process

In this paper, each task is modeled as a Markov decision process (MDP). The MDP process is formally expressed asP=S,A,R, S represents the state of the agent during the interaction between the agent and the environment, A represents the action selected by the agent in a certain state of the environment, R represents the reward obtained by the agent after performing the action, and the state, action, and reward work together to form the environment of agent interaction. The definitions of state, action, and reward are as follows:(1)
The offloading of each task is related to the amount of data of each task in the DAG task graph where the task node is located, the required CPU cycles, and the shape of the DAG graph. The offloading of taskti is related to the offloading of task t1 to task ti in DAG because the offloading of task t1 to task ti may affect the start execution time of task ti. In this paper, the state is defined as follows:(14)state:DATA,TCPU,GT,E,A1:i,whereDATA is the amount of data of each task, Tcpu is the number of CPU cycles required for each task to process, and GT,E is the embedding of the graph (for each task, including the index of the current task, the index of its direct parent task, and the index of direct child tasks), and A1:i is the offloading decision from task t1 to task ti.(2)
There are two kinds of action choices for each task: one is to execute locally; the other is to offload it to the MEC to execute. In this paper, the action is defined as follows:(15)action:0,1.(3)
The purpose of this paper is to minimize the total task execution delayTA1:n. Define the reward as shown in formula (16). Because the optimization goal of this paper is to minimize the total task execution delay, the total execution delay is composed of the processing delay of each task, and the agent is given a reward after each task interacts with the environment, so reward is defined as the processing time of each task. The smaller the delay is, the larger the reward value is, and vice versa.(16)reward:TA1:i−1−TA1:i.


### 4.1.2. seq2seq Structure

Because of the interdependent relationship between tasks, this paper uses the seq2seq [21] structure as the component model of reinforcement learning. The seq2seq model includes the Encoder and the Decoder, where both the Encoder and Decoder are composed of the LSTM model, as shown in Figure 2.Figure 2
Task offloading architecture.The input of Encoder isDATA,TCPU,GT,E in state (i.e., the embedded vector t1,t2,…,tn of a task graph, where n represents the number of tasks contained in each graph). Decoder outputs the offloading decision of a task graph, and the action is expressed as a1,a2,…,an. In order to convert the DAG task graph into embedded vectors and input into the model, the tasks are sorted and indexed according to the ascending order of the critical paths, and the formula is as follows:(17)rankti=Tio,ifti∈K,Tio+maxtj∈childtiranktj,ifti∉K,where Tio=minTiL,Tic+Tiul+Tidl indicates the delay of completing the task, and K is the end node of the task.The functions of Encoder and Decoder are set togenc and gdec, and the output ei of Encoder is represented as shown in formula (18). The aggregate intermediate vector c of the Encoder encoded output ei and the dj in Decoder can be expressed as shown in formulae (19) and (20):(18)ei=gencti,ei−1,(19)c=1n∑i=0nei,(20)dj=gdecc,aj−1,dj−1,where dj is connected to two fully connected layers that approximately represent the vsj and πaj|sj of task tj, where vsj and πaj|sj represent value functions and policy functions, respectively. During training, the action aj is generated by strategy πaj|sj sampling, where aj=maxπaj|sj represents the execution action of task tj.


## 4.1.1. Task Modeling MDP Process

In this paper, each task is modeled as a Markov decision process (MDP). The MDP process is formally expressed asP=S,A,R, S represents the state of the agent during the interaction between the agent and the environment, A represents the action selected by the agent in a certain state of the environment, R represents the reward obtained by the agent after performing the action, and the state, action, and reward work together to form the environment of agent interaction. The definitions of state, action, and reward are as follows:(1)
The offloading of each task is related to the amount of data of each task in the DAG task graph where the task node is located, the required CPU cycles, and the shape of the DAG graph. The offloading of taskti is related to the offloading of task t1 to task ti in DAG because the offloading of task t1 to task ti may affect the start execution time of task ti. In this paper, the state is defined as follows:(14)state:DATA,TCPU,GT,E,A1:i,whereDATA is the amount of data of each task, Tcpu is the number of CPU cycles required for each task to process, and GT,E is the embedding of the graph (for each task, including the index of the current task, the index of its direct parent task, and the index of direct child tasks), and A1:i is the offloading decision from task t1 to task ti.(2)
There are two kinds of action choices for each task: one is to execute locally; the other is to offload it to the MEC to execute. In this paper, the action is defined as follows:(15)action:0,1.(3)
The purpose of this paper is to minimize the total task execution delayTA1:n. Define the reward as shown in formula (16). Because the optimization goal of this paper is to minimize the total task execution delay, the total execution delay is composed of the processing delay of each task, and the agent is given a reward after each task interacts with the environment, so reward is defined as the processing time of each task. The smaller the delay is, the larger the reward value is, and vice versa.(16)reward:TA1:i−1−TA1:i.


## 4.1.2. seq2seq Structure

Because of the interdependent relationship between tasks, this paper uses the seq2seq [21] structure as the component model of reinforcement learning. The seq2seq model includes the Encoder and the Decoder, where both the Encoder and Decoder are composed of the LSTM model, as shown in Figure 2.Figure 2
Task offloading architecture.The input of Encoder isDATA,TCPU,GT,E in state (i.e., the embedded vector t1,t2,…,tn of a task graph, where n represents the number of tasks contained in each graph). Decoder outputs the offloading decision of a task graph, and the action is expressed as a1,a2,…,an. In order to convert the DAG task graph into embedded vectors and input into the model, the tasks are sorted and indexed according to the ascending order of the critical paths, and the formula is as follows:(17)rankti=Tio,ifti∈K,Tio+maxtj∈childtiranktj,ifti∉K,where Tio=minTiL,Tic+Tiul+Tidl indicates the delay of completing the task, and K is the end node of the task.The functions of Encoder and Decoder are set togenc and gdec, and the output ei of Encoder is represented as shown in formula (18). The aggregate intermediate vector c of the Encoder encoded output ei and the dj in Decoder can be expressed as shown in formulae (19) and (20):(18)ei=gencti,ei−1,(19)c=1n∑i=0nei,(20)dj=gdecc,aj−1,dj−1,where dj is connected to two fully connected layers that approximately represent the vsj and πaj|sj of task tj, where vsj and πaj|sj represent value functions and policy functions, respectively. During training, the action aj is generated by strategy πaj|sj sampling, where aj=maxπaj|sj represents the execution action of task tj.


## 4.2. An Offloading Model of Federated Reinforcement Learning Based on Attention Mechanism

In federated learning, multiple device nodes participate in the training and aggregate the model weights shared by the service BS nodes. In order to make better use of the parameter information provided by each equipment node, the service BS node needs to allocate different weight coefficients according to their contribution. Considering the difference between equipment models, the Attention mechanism can be applied to achieve the demand. In this paper, an FRL task offloading model based on Attention mechanism is proposed, and the training process is as follows:(i)
Associate the mobile device with the serving BS and report the local mobile device status to the BS. After receiving the information, BS sends the previous round of global model parameters to the local device to perform the steps of model release.(ii)
When the local mobile device learns that it can participate in the training, it receives the global model parameters obtained from the BS.(iii)
Each mobile device begins to train its local model parameters based on global model parameters and local data. After the local training, each mobile device collects its training evaluation indicators (e.g., average reward, average loss, etc.) during the training phase and sends them and local model parameters to BS. Then, BS calculates the aggregation weight of each equipment according to the training evaluation index. In this paper, attention mechanism is used to provide aggregation weights for different mobile devices. The update process for weighted federation aggregation is shown in Figure3.Figure 3
Federated Reinforcement Learning model of Attention mechanism.In this model, the indexes ofm mobile devices are input into the Encoder of the model, and Query is the optimal goal in Decoder. By comparing the similarity between the indexes of each mobile device model and the optimal goal, the contribution weight of each device model to the optimal goal is calculated. In this paper, the average loss, average reward and the computing power of mobile devices are used to calculate the attention weight of the contribution of the global model.(i)
Average loss: the quality of the model is related to the size of the loss, the greater the loss means that the model cannot well deal with the problems to be solved. On the contrary, the smaller the loss, the better the performance of the model, so we can consider using it. So, the loss can be evaluated on a model. The average lossLavg is the average calculation of the loss function of a certain period of time by a device.(ii)
Average reward: contrary to the average loss, the average reward can also evaluate the quality of the model. When the reward is larger, the effect of the model is better, and when the reward is smaller, the model may have some deviation from the problem solved. Therefore, the average reward can also evaluate the model. Average rewardRavg is the average reward for the mobile device over a certain period of time.(iii)
Computing power of mobile devices: the computing power of mobile devices can affect whether tasks need to be offloaded. When the computing power of mobile devices is large, the delay caused by local processing tasks is usually far less than the delay caused by transmitting task data, so the device will get a better user experience effect when the tasks are processed locally. On the contrary, if the processing power of the mobile device is too small, the processing delay will be very long, so it is better to offload the task to the edge server. Therefore, the computing power of the mobile devicecomu also has a certain impact on the model parameters.In this paper, we use the above indexK=Lavg,Ravg,comu to evaluate the attention mechanism. The goal of the training model is to get more rewards and achieve the minimum loss. Therefore, the optimization goal set in this paper is shown in formula (21). Q consists of minimizing the loss function, maximizing the reward, and maximizing the computing power.(21)Q=minLavg,maxRavg,maxcomu.Then, the weight of the contribution of the device is calculated by using the Attention mechanism. In this paper, we use the method of dot product to calculate the similarity betweenQ and K to get the corresponding weight, and then use the softmax function to normalize the weight, and introduce the weight factor wu to calculate the contribution of the local model to the global model. The specific calculation is shown as follows:(22)wu=AttentionQ,K=softmaxQKTdk,where dk represents the dimension of K, from which the contribution of each mobile device model is calculated by using the Attention mechanism, and the weight of the model is weighted to calculate the parameters of the global model. After aggregation, this parameter is sent to each local model. The optimized global parameter aggregation formula is shown in the following formula. The specific algorithm is shown in Algorithm 1.(23)Fθt=∑u∈UwuFθtu.Algorithm 1: A Federated Reinforcement Learning task offloading model integrated with Attention mechanism.
(1)
for each equipment in parallel do(2)
for local stepj=1,…,n do(3)
for eachstep do(4)
Sample actiona∼πθs, reward and s′(5)
Inputs and s′ to get Qws′,a′ and Qws,a(6)
δ=r+γQws′,a′−Qws,a(7)
θ=θ+α∇θlogπθs,aQws,a\(8)
w←w+βδ(9)
a←a′, s←s′(10)
end for(11)
K=Lavg,Ravg,comu(12)
end for(13)
Q=minLavg,maxRavg,maxcomu(14)
wu=AttentionQ,K(15)
Fθt=∑u∈UwuFθtu(16)
end for


## 5. Experiment

### 5.1. Experimental Setup

#### 5.1.1. Setting Model Parameters

System Model Parameters. Considering 20 users and one edge server node, the task arrival rate is 0.3, the user CPU clock rate is 2.5 GHz, the clock rate of the server is 41.8 GHz, and the uplink and downlink transmission rate is 14 Mbps. In this paper, a synthetic DAG generator is implemented with [22].Network Model Parameters. The Encoder and Decoder of seq2seq neural network are set to two layers of LSTM, with 256 hidden units in each layer. The learning rate is set to 0.001 and the optimizer adopts RMSprop algorithm.


#### 5.1.2. Hardware Parameters

The hardware setting in the experiment can be seen in Table1.Table 1
Hardware parameter.
ConfigurationParameterCPUIntel® Xeon® CPU E5645 2.40 GHzMemory8 GBOperating systemWindows 10Development toolsPyCharm


### 5.2. Results and Analysis

According to [22], a synthetic DAG generator is implemented for simulation. 22 task graphs are used as training sets for training, and the other three task graphs are tested, and each task graph is composed of 20 tasks. The training is carried out on 22 training sets, and the experimental results are shown in Figure 4, which shows the change of the average reward with the increase of the number of iterations. It can be seen that after 1000 iterations, the average reward as a whole tends to be stabilized and shows an upward trend during the training process.Figure 4
Average reward of training set.Learning rate plays a very important role in deep learning and training. Figure5 shows that on the premise that the number of subtasks in each task graph is 10; the learning rate is set to 0.1, 0.01, and 0.0001, respectively. The experimental comparison of the proposed algorithms shows that the model with a learning rate of 0.001 has the highest delay. The performance of the model with learning rate of 0.01 and 0.0001 is similar, but the effect is the best at 0.001. Therefore, this paper uses 0.001 as the learning rate of model training.Figure 5
The average delay of the algorithm under different learning rates.Considering 20 users, the task graph of each user is composed of 10 subtasks. The Federated Reinforcement Learning Offloading Computation algorithm with Attention (FRLOC-Attention) is compared with DQN algorithm, Greedy algorithm, and Actor-Critic based offloading algorithm (AC). The experimental results are shown in Figure6. The results show that after 500 episodes, FRLOC-Attention shows better performance and minimum delay, while AC and DQN offloading algorithms are not as effective as FRLOC-Attention, while greedy algorithm (Greedy) produces a larger delay, about 590 ms, and has a larger delay than DQN algorithm and AC algorithm. Generally speaking, FRLOC-Attention has the lowest latency and the best performance.Figure 6
The number of tasks is 10, the average delay on different algorithms.In order to show that the proposed algorithm has a certain generality, this paper chooses to experiment on the number of different devices. Figure7 shows that the number of devices is 10, 20, and 30, and the DAG task graph of each device includes 20 subtasks. From the experimental results, we can see that no matter the number of devices, the task processing delay of FRLOC-Attention is obviously lower than that of DQN algorithm and Greedy algorithm. The performance of AC algorithm is similar to that of FRLOC-Attention; this is because AC algorithm is included in the process of FRLOC-Attention training model, but because this algorithm proposed uses federated reinforcement learning combined with Attention to make the model more generalized, when there are multiple devices, the performance of task offload is better.Figure 7
The average delay of each algorithm in the case of different number of devices.


## 5.1. Experimental Setup

### 5.1.1. Setting Model Parameters

System Model Parameters. Considering 20 users and one edge server node, the task arrival rate is 0.3, the user CPU clock rate is 2.5 GHz, the clock rate of the server is 41.8 GHz, and the uplink and downlink transmission rate is 14 Mbps. In this paper, a synthetic DAG generator is implemented with [22].Network Model Parameters. The Encoder and Decoder of seq2seq neural network are set to two layers of LSTM, with 256 hidden units in each layer. The learning rate is set to 0.001 and the optimizer adopts RMSprop algorithm.


### 5.1.2. Hardware Parameters

The hardware setting in the experiment can be seen in Table1.Table 1
Hardware parameter.
ConfigurationParameterCPUIntel® Xeon® CPU E5645 2.40 GHzMemory8 GBOperating systemWindows 10Development toolsPyCharm


## 5.1.1. Setting Model Parameters

System Model Parameters. Considering 20 users and one edge server node, the task arrival rate is 0.3, the user CPU clock rate is 2.5 GHz, the clock rate of the server is 41.8 GHz, and the uplink and downlink transmission rate is 14 Mbps. In this paper, a synthetic DAG generator is implemented with [22].Network Model Parameters. The Encoder and Decoder of seq2seq neural network are set to two layers of LSTM, with 256 hidden units in each layer. The learning rate is set to 0.001 and the optimizer adopts RMSprop algorithm.


## 5.1.2. Hardware Parameters

The hardware setting in the experiment can be seen in Table1.Table 1
Hardware parameter.
ConfigurationParameterCPUIntel® Xeon® CPU E5645 2.40 GHzMemory8 GBOperating systemWindows 10Development toolsPyCharm


## 5.2. Results and Analysis

According to [22], a synthetic DAG generator is implemented for simulation. 22 task graphs are used as training sets for training, and the other three task graphs are tested, and each task graph is composed of 20 tasks. The training is carried out on 22 training sets, and the experimental results are shown in Figure 4, which shows the change of the average reward with the increase of the number of iterations. It can be seen that after 1000 iterations, the average reward as a whole tends to be stabilized and shows an upward trend during the training process.Figure 4
Average reward of training set.Learning rate plays a very important role in deep learning and training. Figure5 shows that on the premise that the number of subtasks in each task graph is 10; the learning rate is set to 0.1, 0.01, and 0.0001, respectively. The experimental comparison of the proposed algorithms shows that the model with a learning rate of 0.001 has the highest delay. The performance of the model with learning rate of 0.01 and 0.0001 is similar, but the effect is the best at 0.001. Therefore, this paper uses 0.001 as the learning rate of model training.Figure 5
The average delay of the algorithm under different learning rates.Considering 20 users, the task graph of each user is composed of 10 subtasks. The Federated Reinforcement Learning Offloading Computation algorithm with Attention (FRLOC-Attention) is compared with DQN algorithm, Greedy algorithm, and Actor-Critic based offloading algorithm (AC). The experimental results are shown in Figure6. The results show that after 500 episodes, FRLOC-Attention shows better performance and minimum delay, while AC and DQN offloading algorithms are not as effective as FRLOC-Attention, while greedy algorithm (Greedy) produces a larger delay, about 590 ms, and has a larger delay than DQN algorithm and AC algorithm. Generally speaking, FRLOC-Attention has the lowest latency and the best performance.Figure 6
The number of tasks is 10, the average delay on different algorithms.In order to show that the proposed algorithm has a certain generality, this paper chooses to experiment on the number of different devices. Figure7 shows that the number of devices is 10, 20, and 30, and the DAG task graph of each device includes 20 subtasks. From the experimental results, we can see that no matter the number of devices, the task processing delay of FRLOC-Attention is obviously lower than that of DQN algorithm and Greedy algorithm. The performance of AC algorithm is similar to that of FRLOC-Attention; this is because AC algorithm is included in the process of FRLOC-Attention training model, but because this algorithm proposed uses federated reinforcement learning combined with Attention to make the model more generalized, when there are multiple devices, the performance of task offload is better.Figure 7
The average delay of each algorithm in the case of different number of devices.


## 6. Conclusion

Considering the lack of model generalization caused by the lack of data sharing among multiple users and the privacy data leakage caused by data sharing, this paper proposes a federated reinforcement learning task offloading algorithm based on Attention mechanism to solve the relevance problem of multiple mobile device training models. Experiments show that the proposed FRLOC-Attention algorithm saves more time than DQN algorithm, AC algorithm, and Greedy algorithm. On the basis of this algorithm, we can further consider the optimization of energy consumption caused by data transmission during task offloading in the future.



---

*Source: 6752527-2022-02-08.xml*