# Joint Optimization of Service Migration and Resource Allocation in Mobile Edge-Cloud Computing
## 4 Proposed A2C-Based Algorithm
### Dynamic Migration and Resource Allocation Algorithm Based on A2c  
Reinforcement learning is a robust approach for addressing dynamic programming challenges. Recent advancements in this field involve the integration of deep neural networks to represent both policy and value functions effectively. In this paper, we utilize the A2C method to optimize our objective. As illustrated in Figure 3, A2C not only refines the policy to select the optimal action for each state but also develops a value function that supports policy optimization. In DRL, for the given state \(s_{t}\) the expected reward obtained by selecting action \(a_{t}\) according to policy \(\pi\) is defined as the action value \(Q_{\pi}(s_{t},a_{t})\), which is defined as follows:  
\[Q_{\pi}(s_{t},a_{t})=E_{\{S,\pi\}}[R_{t}|s_{t},a_{t}], \tag{21}\]  
where \(R_{t}\) represents the expected sum of reward under the strategy \(\pi\). \(R_{t}\) can be expressed as  
\[R_{t}=\sum\limits_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}), \tag{22}\]  
where \(\gamma^{t}\) represents the discount factor used to discount future rewards. According to [PERSON]'s equation, the action value \(Q_{\pi}(s_{t},a_{t})\) can be re-expressed as  
\[Q_{\pi}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma E_{s\sim S}[V_{\pi}(s_{t+1})], \tag{23}\]  
Figure 3: Training of A2C-based dynamic migration and resource allocation algorithm. and \(V_{\pi}(s_{t+1})\) is the expected reward obtained by following policy \(\pi\), called the state-value function, i.e.,  
\[V_{\pi}(s_{t+1})=E_{a_{t+1}\sim\pi}[Q_{\pi}(s_{t+1},a_{t+1})]. \tag{24}\]  
In the A2C framework, there are two primary components: the actor and the critic. The actor is responsible for maintaining a policy \(\pi\), guiding action selection, and interacting with the environment. The critic, on the other hand, learns the state-value function based on rewards derived from the interactions between the actor and the environment, and aids the actor in policy updates. The actor employs a neural network to model the policy function, generating action probabilities from the observed state. Its objective is to identify the optimal policy that maximizes the expected reward in the environment. We can define the objective function as  
\[J(\pi)=E_{s_{0}}[V_{\pi}(s_{0})]. \tag{25}\]  
We use \(\theta_{a}\) to denote the parameters of the actor's network and \(\theta_{c}\) to denote the parameters of the critic's network. We can rewrite the function \(J(\pi)\) as  
\[J(\pi_{\theta_{a}})=E[A(s_{t},a_{t})\cdot\log\pi_{\theta_{a}}(a_{t}|s_{t})], \tag{26}\]  
where \(A(s_{t},a_{t})\) is advantage function. It indicates whether the reward obtained by choosing action \(a_{t}\) is higher than the average reward in \(s_{t}\). We define the advantage function as  
\[A(s_{t},a_{t})=Q_{\theta_{c}}(s_{t},a_{t})-V_{\theta_{c}}(s_{t})\approx r(s_{ t},a_{t})+\gamma V_{\theta_{c}}(s_{t+1})-V_{\theta_{c}}(s_{t}). \tag{27}\]  
Then, the loss function of the actor's network is  
\[L_{Actor}=-\Big{[}\log\pi_{\theta_{a}}(a_{t}|s_{t})A(s_{t},a_{t})\Big{]}. \tag{28}\]  
Hence, we have  
\[\theta_{a}\leftarrow\theta_{a}+l_{a}(A(s_{t},a_{t})\frac{\partial\log\pi_{ \theta_{a}}(a_{t}|s_{t})}{\partial\theta_{a}}), \tag{29}\]  
where \(l_{a}\) is the learning rate of the actor network. As with the actor network, we use the critic network to estimate the state-value function \(V_{\theta_{c}}(s_{t})\), and its loss function can be expressed as  
\[R_{t}=\sum\limits_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}), \tag{22}\]0  
Similarly, we have  
\[R_{t}=\sum\limits_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}), \tag{22}\]1  
where \(l_{c}\) is the learning rate of the critic network. Algorithm 1 outlines the A2C-based approach for migrating services and allocating resources described in this paper. Initially, we initialize the algorithm parameters, as well as the actor and critic networks, and set up the MECC system based on input values (lines 1-4). At the beginning of each episode, the system's state is reset and transmitted to the agent (lines 6-8).