# Virtual Machine Migration Strategy Based on Multi-Agent Deep Reinforcement Learning
## 5 Experimental Results and Analysis  
Experiments compare the performance of the VM migration strategy based on MADRL strategy proposed in this paper with the following three VM migration strategies. Random migration strategy (Random): Randomly migrate VMs to a server. Nearest neighbor migration strategy (NN): Directly select the nearest MEC server to migrate the VM. Centralized migration strategy based on Actor-Critic algorithm (CON_AC): Based on the centralized network architecture of SDN, the migration strategy is generated by the SDN controller. Distributed migration strategy based on Actor-Critic algorithm (DIS_AC): Each agent interacts with the environment individually to learn the VM migration strategy, and the agents do not communicate with each other. In the experiment, the neural networks used by CON_AC, DIS_AC and MADRL are all actor and critic networks based on the parameter configuration described in the previous section, and they use the same reward function. The experiment has 2000 iterations, and the maximum number of iterative steps in each round is 50. Experiments compare the convergence of the above three migration strategies based on reinforcement learning when the number of servers is 10 and the number of VMs is 20, as shown in Figure 3.  
As can be seen from Figure 3, DIS_AC does not converge in this scenario. The DIS_AC method is based on the Actor-Critic method. During the training, each agent only makes the migration strategy according to the local information. When an agent makes a decision, the environment state has changed. It is precisely because the agents compete for resources and lack of awareness of the overall environment, which can easily lead to the fluctuation of the reward and the decrease of the reward value. The convergence of CON_AC is the best, and the MADRL strategy proposed in this paper is slightly inferior, but very close. This is because CON_AC can obtain the global server and VM status through the SDN controller, and only one agent exists in the SDN controller in this environment. On the other hand, MADRL has multiple agents, so it is necessary to construct the global state according to the observed state of each agent. Therefore, due to the interaction between \begin{table}
\begin{tabular}{c c} \hline
**Parameter** & **Value** \\ \hline number of VM & 5–20 \\ size of VM & 1000\(\sim\)10,000 bytes \\ tolerable delay & 0.01–0.3 s \\ number of CPU cycles required for tasks & 500–1500 cycles \\ speed & 0–90 m/s \\ computational capacity of a server & \(1\times 10^{6}\)–\(6\times 10^{6}\) cycles/s \\ bandwidth of a server & 1–3 MHz \\ size of memory & 1 TB \\ distance between servers & 200–1000 m \\ \hline \end{tabular}
\end{table} 
Table 1: Simulation experiment parameter. agents, the convergence speed of the model is a little slower than that of CON_AC, and it begins to converge in about the 800 th round. Figure 4 shows the time it takes for MADRL and CON_AC to obtain strategies as the number of servers increases with the number of VMs 10 and 20, respectively. The time it takes to obtain a strategy is expressed as the time it takes from the start of the triggered migration to the time before the migration strategy is executed. The experiments are all carried out on the same server, and the parameters are fixed. As we can see from Figure 4, MADRL has the lowest time consumption. The increase in the number of servers has little impact on MADRL, and only when the number of VMs increases, it will increase the time it takes to obtain strategies. However, CON_AC is very affected by the number of servers, and when the number of VMs is 10, the time spent increases 2.8 times as the number of servers increases from 5 to 20. The reason is that MADRL only needs local observation information to obtain the migration action, and does not need to communicate with other servers, which speeds up the perception of the current environment and generates the migration strategy more quickly. On the other hand, CON_AC needs to communicate with all servers to obtain the current global information.