# Virtual Machine Migration Strategy Based on Multi-Agent Deep Reinforcement Learning
## 4 VM Migration Strategy Based on MADRL
### VM Migration Strategy Based on MADRL  
Each MEC server \(H_{n}\) has its own agent, and each agent is composed of two parts, namely actor and critic network. The actor network includes two parts: the current actor network \(\pi_{n}\) and the target actor network \(\pi^{\prime}{}_{n}\). The two network parameters are \(\theta_{n}\) and \(\theta^{\prime}{}_{n}\), respectively. Similarly, the critic network also includes two parts: the current critic network \(Q_{n}\) and the target critic network \(Q^{\prime}_{n}\). The two network parameters are \(\omega_{n}\) and \(\omega^{\prime}_{n}\), respectively.  
The VM migration strategy architecture based on MADRL is shown in Figure 2. In the process of model training, the current actor network \(\pi_{n}\) and the current critic network \(Q_{n}\) are trained online, and the target critic network \(Q^{\prime}_{n}\) is trained offline. The current actor network \(\pi_{n}\) of each agent receives the current observation state \(o_{n}\), and the MEC server takes the action \(a_{n}\) and obtains the reward \(r_{n}\), and then the MEC server moves on to the next state \(\phi^{\prime}_{n}\). Then, the quaternion \(\{s,a,r,s^{\prime}\}\) is saved into the experience replay set \(D\).  
Neural network training includes three steps. First update the current critic network, then update the current actor network, and finally update the target actor network and the target critic network.  
(1) Update the Current Critic Network  
At the beginning of each time period, each agent will randomly select \(s\) samples \(\{s,a,r,s^{\prime}\}\) from the experience replay set \(D\), and use the mean square error loss function shown in Formula (8) to update the parameter \(\omega_{n}\) of the current critic network through the gradient back propagation of the neural network,  
\[L(\omega_{n})=\frac{1}{s}\sum_{j=1}^{s}{(y_{j}-Q_{n}(s,a))}^{2}, \tag{8}\]  
where \(y_{j}\) is as shown in Equation (9),  
\[y_{j}=r+\gamma Q^{\prime}_{n}\big{(}s^{\prime},\pi^{\prime}_{n}\big{(}s^{ \prime}\big{)}\big{)}. \tag{9}\]  
(2) Update the Current Actor Network  
The current actor network updates parameters through policy gradient descent, as shown in Equation (10):  
\[\
abla_{\theta_{n}}l\approx\frac{1}{s}\sum_{j^{\prime}}\
abla_{\theta_{n}}log (\pi_{n}(o_{n}))\
abla_{a_{n}}Q_{n}(s,a). \tag{10}\]  
(3) Update the Target Actor Network and Target Critic Network  
Figure 2: VM Migration Strategy Architecture based on MADRL.  
The parameters of the target actor network and target critic network are updated as shown below:  
\[\theta^{\prime}{}_{n}=\tau\theta_{n}+(1-\tau)\theta^{\prime}{}_{n}\, \tag{11}\]  
\[\omega^{\prime}{}_{n}=\tau\omega_{n}+(1-\tau)\omega^{\prime}{}_{n}\, \tag{12}\]  
where \(\tau\in[0,1)\) represents the learning rate of the parameter.  
The VM migration strategy based on MADRL is shown in Algorithm 1.  
```
1.Initialize current actor network \(\pi\) and critic network \(Q\) with parameters \(\theta\) and \(\omega\)
2.Initialize target actor network \(\pi^{\prime}\) and critic network \(Q^{\prime}\) with parameters \(\theta^{\prime}\) and \(\omega^{\prime}\)
3.Initialize episode \(K\), discount factor \(\gamma\), learning rate \(\tau\), the number of samples \(s\), update frequency \(C\), experience replay set \(D\)
4.for episode \(k=1,2,\ldots,K\)do
5.for\(S_{l}\)\(i=1,2,\ldots,T\)do
6. choose action \(a_{n}=\pi_{n}(o_{n})\) for each MEC Server
7. execute action \(a=(a_{1},\cdots,a_{n})\), obtain reward \(r\) and next state \(s^{\prime}\)
8. store transitions \(\{s,a,r,s^{\prime}\}\) into replay set \(D\)
9.\(s\gets s\)
10.endfor
11.if\(k\%C==0\)then
12.for\(agent\)\(n=1,2,\ldots,n\)do
13. sample \(s\) transitions \(\{s,a,r,s^{\prime}\}\) from \(D\)
14. update current critic network according to (8), (9)
15. update current actor network according to (10)
16.endfor
17. update target actor network and target critic network according to (11), (12)
18.endif
19.endfor
```  
**Algorithm 1** VM migration strategy based on MADRL