# Virtual Machine Migration Strategy Based on Multi-Agent Deep Reinforcement Learning
## 4 VM Migration Strategy Based on MADRL
### Multi-Agent Deep Reinforcement Learning Model  
1. Agent Observation State The observed state \(o_{n}(i)\in\mathcal{S}\) represents the state that MEC server \(H_{n}\) can perceive during the time period \(SI_{i}\), which consists of two parts; namely, the current running status of the server and the parameters of the VM to be migrated, so it can be expressed as \(o_{n}(i)=\left\{H_{n},T_{n}^{1},T_{n}^{2},\ldots,T_{n}^{u}\right\}\).
2. Action Space According to the observed state of time period \(SI_{i}\) and the migration strategy \(\pi_{i}\), we can receive the action \(a_{n}(i)\), which is a vector composed of \(u\) elements, each element represents the destination server of VM \(T_{n}^{u}\) migration, and \(a_{n}(i)=\left(a_{n}^{1},\cdots,a_{n}^{u}\right)\). For VM \(T_{n}^{u}\), because the output \(p_{n,m}\) of neural network is the probability of migration from VM \(T_{n}^{u}\) to server \(H_{n}\), \(a_{n}^{u}\) chooses the server with the highest probability as the migration destination server through greedy algorithm, and migrates VM \(T_{n}^{u}\) to this server, so \(a_{n}^{u}=\max(p_{n,1},\cdots,p_{n,m})\). The action set of all agents is represented by \(\mathcal{A}\), which is an \(n\) dimensional vector.
3. Reward Function According to the observed state of \(o_{n}(i)\), the MEC server takes the action \(a_{n}(i)\) and obtains a timely reward of \(r_{n}(i)\). The optimization goal is to minimize the energy consumption of VM migration, so the reward function \(r_{n}(i)\) is as follows: \[r_{n}(i)=\left\{\begin{array}{c}\frac{1}{\sum_{n=1}^{\lceil{\gamma}\rceil} \sum_{j=1}^{\lceil{\delta}\rceil}a_{ij}\epsilon_{ij}^{u}},success\\ 0,fail\end{array}\right..\] (6)  
In the multi-agent environment, we use _sar_ to denote system status, joint action and joint reward, respectively, which are marked as \(s=\left\{o_{n}(i)\right\}\), \(a=\left\{a_{n}(i)\right\}\), \(r=\left\{r_{n}(i)\right\}\), respectively. In order to simplify the representation, the state, action and reward of MEC server \(H_{n}\) in the current time period are expressed by using the \(o_{n}\), \(a_{n}\) and \(r_{n}\). \(o^{\prime}{}_{n}\), \(a^{\prime}{}_{n}\), \(r^{\prime}{}_{n}\) are used to represent the status, action and reward of MEC server \(H_{n}\) in the next time period. Each agent expects to maximize its reward in the process of interacting with the environment. In this paper, the action value function of the action \(a_{n}^{u}\) taken by the MEC server \(H_{n}\) in the case of the observed state \(o_{n}\) in the time period \(SI_{i}\) is expressed by using \(Q_{n}(o_{n},a_{n})\). Based on the Behrman equation, the recursive relation of the action value function \(Q_{n}(o_{n},a_{n})\) is shown in the Formula (7),  
\[Q_{n}(o_{n},a_{n})=\mathbb{E}\big{[}r_{n}+\gamma\mathbb{E}\big{[}Q_{n}\big{(} o^{\prime}{}_{n},a^{\prime}{}_{n}\big{)}\big{]}\big{]}, \tag{7}\]  
where the discount factor \(\gamma\in[0,1)\) controls the ability of the agent to \"look forward to the future\". When \(\gamma=1\), the agent obtains a reward for \(Q_{n}\) without any discount in the future.