# A Dynamic Service Placement Based on Deep Reinforcement Learning in Mobile Edge Computing
## 4 Dynamic Service Placement Framework Based on Deep Reinforcement Learning
### Deep Reinforcement Learning Formulation  
Since the decision making during dynamic service placement is a stochastic optimization, our framework is studied based on the deep deterministic policy gradient (DDPG) algorithm [25]. In this paper, the objective of the agent is to realize dynamic service placement for multiple mobile users while minimizing the total delay. We first summarize the state and action spaces, reward function, and the state transition policy that are used in our reinforcement learning framework. In order to describe the environment of edge servers and mobile users for the agent concisely and correctly, the state space includes the knowledge of services placed on the edge servers and the status of users that are supplied by these services. To that end, the state is designed as follows.  
**Definition 1** (State).: _The state \(s_{t}\) describes the environment of the edge network, which is a vector consisting of \(s_{t}=[\mathbf{r}_{t},\hat{\mathbf{u}}_{t}]\), \(\mathbf{r}_{t}=(r_{1}(t),r_{2}(t),...,r_{t}(t),...,r_{m}(t))\) is the vector of rest storages, where \(r_{j}(t)\) denotes the rest storage on edge server \(\mathbf{m}_{j}\), \(\hat{\mathbf{u}}_{t}=(\hat{u}_{1}(t),\hat{u}_{2}(t),...,\hat{u}_{i}(t),...,\hat{u }_{n}(t))\) is the vector of positions on each users' trajectories, where \(\hat{u}_{i}(t)\) denotes the position of \(u_{i}\) at time slot \(t\)._  
We consider that the services on each edge server make decentralized decisions according to the trajectories of mobile users by training the agent. The action \(a_{t}\) is designed as follows:  
**Definition 2** (Action).: _The action space \(a_{t}=[\hat{\mathbf{m}}_{1},\hat{\mathbf{m}}_{2},...,\hat{\mathbf{m}}_{h},...,\hat{\mathbf{m} }_{n}]_{t}\) is the migration action, where \(\hat{\mathbf{m}}_{h}(t)=[\hat{m}_{h}(t)^{-},\hat{m}_{h}(t)^{+}]\) denotes the alternative range of edge servers during the migration process of service \(\mathbf{v}_{h}\) at time slot \(t\)._  
For each service, the alternative range of edge servers is represented by a continuous set of numbers \([\hat{m}_{h}(t)^{-},\hat{m}_{h}(t)^{+}]\), where \(\hat{m}_{h}(t)^{-}\) denotes the lower bound (minimum number of server) that can be selected during the service \(\mathbf{v}_{h}\)'s migration, and \(\hat{m}_{h}(t)^{+}\) denotes the upper bound (maximum number of servers).  
Since our problem is an online learning process, the value of the reward cannot determine the final total delay for multiple mobile users in each time slot directly; however, it will drive their behaviors to obtain a better performance. In order to realize a decentralized dynamic service placement strategy, we minimize the total delay while completing the  
Figure 2: The overview of the DSP-DRL framework.  
processed tasks for the mobile users within a limited migration cost. Thus, here is the specific definition of the reward function.  
**Definition 3** (**Reward**).: _The reward \(r\) is measured by the average delay feedback of multiple mobile users comparing with \(r=\sum_{u\in U}\frac{\overline{z_{u_{i}}(t)-z_{u_{i}}(t)}}{|U|}\)._  
Here, \(|U|\) denotes the total number of users and \(z_{u_{i}}(t)\) denotes the total delay of multiple users according to the decisions by the deep neural network at time slot \(t\). We use \(\overline{z_{u_{i}}}(t)\) to denote the total delay that the service stays on the original edge server without migration.