# A Dynamic Service Placement Based on Deep Reinforcement Learning in Mobile Edge Computing
## 4 Dynamic Service Placement Framework Based on Deep Reinforcement Learning
### Migration Conflicting Resolution Mechanism
#### 4.2.2 Migration Conflict Resolution Mechanism  
We propose a new migration conflict resolution mechanism to avoid the invalid state and approximate the policy in the decision module. There are two main stages included in our resolution mechanism: stage one is to find the edge servers with conflicting services; stage two is to make migration decisions for the conflicting edge nodes. The details are shown in Algorithm 1. The input is action \(a_{t}\) at time slot \(t\), and the output is the updated action of service placement decisions while enabling conflict resolution for edge servers. According to the action \(a_{t}\), which is produced during the learning process in time slot \(t\), we do the pre-migration for each service \(v_{h}\in\mathbf{V}\). After that, we check the status of each edge sever \(m_{j}\in\mathbf{M}\) by calculating the total number of services \(N_{m_{j}}\) pre-migrated to \(m_{j}\). We compare the number of total requests \(N_{m_{j}}\) with the storage \(R(m_{i})\) of destination server \(m_{j}\). If \(N_{m_{j}}>R(m_{i})\), \(m_{j}\) is a conflicting edge server. Otherwise, the migration to \(m_{j}\) is successful. Based on that, we start to make migration decisions. For the conflicting edge server \(m_{j}\), we first build conflict set \(\mathbb{C}_{m_{j}}=\{v_{h}^{m_{j}}\}\), which is composed of all the services requesting to migrate on server \(m_{j}\) at the same time. Then, we choose service \(v_{h}\) in set \(\mathbb{C}_{m_{j}}\) with maximum \(\zeta(v_{h})\). Here, we introduce a novel definition: migration feasibility factor. **Definition 4** (**migration feasibility factor**). : _Let \(\zeta(v_{h})(t)\) indicate the migration feasibility factor of service \(v_{h}\in\mathbb{C}_{m_{j}}\) at time slot \(t\), where \(\zeta(v_{h})(t)=D^{u}_{u_{h}}(t)+\frac{\omega}{D^{u}_{u_{h}}(t)}\) and \(\omega>0\)._  
We use \(D^{u}_{u_{i}}(t)\) to denote the migration delay of the service \(v_{h}\) that is serving user \(u_{h}\) at time slot \(t\), and we use \(D^{l}_{u_{h}}(t)\) to denote the communication delay produced when service \(v_{h}\) is not placed on the edge server within a user's area. These two parametersare negatively correlated, which means that when the service migrates or is close to the users' area, the value of communication delay will be \(D_{u_{h}}^{l}(t)=0\) or less. Here, we use a constant \(\varpi\) to adjust the relationship, where \(\varpi>0\). Therefore, the migration feasibility factor considers the impact of these two parameters on users' delays. In line 9, we update the conflict set with \(\mathbb{C}_{m_{j}}=\mathbb{C}_{m_{j}}/v_{h}\). For the rest of the services in set \(\mathbb{C}_{m_{j}}\), we migrate \(v_{h^{\prime}}\) to the nearest edge server that meets the storage resource requirements, as denoted in lines 10 to 12. In line 13, we record the current state of services placement in \(a_{t}^{\prime}\) and update the action \(a_{t}=a_{t}^{\prime}\). If the storage resources are adequate, there will be no conflict, which means that the services migrate to \(m_{j}\) successfully. Finally, we will keep the original service placement decisions of action \(a_{t}\) in line 15.