# A Dynamic Service Placement Based on Deep Reinforcement Learning in Mobile Edge Computing
## 4 Dynamic Service Placement Framework Based on Deep Reinforcement Learning
### Dynamic Service Placement Based on Deep Reinforcement Learning  
In this subsection, we propose a dynamic service placement strategy based on deep reinforcement learning. According to the characteristic of the decision-making process, our scheme studies are based on the deep deterministic policy gradient (DDFG) algorithm. The main idea is to use a deep reinforcement learning agent to perform the dynamic service placement of multiple mobile users to minimize the total delay. The specific steps are shown in Algorithm 2. We use the sets of edge nodes \(\mathbf{M}\), services \(\mathbf{V}\), and users \(\mathbf{U}\) as the input. The output is the dynamic service placement scheme \(\mathbf{X}\). In lines 1 to 3, we initialize the preliminary parameters of the reinforcement learning agent, which includes the main network, the target network, and the replay buffer. In line 4, we start to train the agent by running a number of \(\kappa\) episodes with our environment. Each edge server can learn to determine the placement strategy (migration or keeping the original position) of services gradually and independently after training for \(\kappa\) episodes. We start to initialize environmental parameters for edge servers and users, and we generate an initial state \(s_{1}\) in line 5. The training process in one time period \(T\) starts from lines 6 to 15. For each time slot, we select an action \(a_{t}=\mu(s_{t}|\theta^{\mu})+\delta_{t}\) to determine the destination of migration by running the current policy network \(\theta^{\mu}\) and exploration noise \(\delta_{t}\). Since the movements of users are erratic and autonomous, we detect any migration conflicts and resolve them based on Algorithm 1 in line 8. For each user agent, we execute action \(a_{t}\) and observe reward \(r_{t}\) and new state \(s_{t+1}\) from the environment. Then, we store the transition tuple \((s_{t},a_{t},r_{t},s_{t+1})\) into the replay buffer \(B\) in line 10. In lines 12 to 14, the actor and critic network of the user agent will be updated according to the mini-batch of \(\mathbf{I}\) transitions from \(B\). In line 11, we update the critic network, which takes the state \(s_{t}\) and action \(a_{t}\) as input, and it outputs the action value [19]. Specifically, the critic approximates the action-value function \(Q(s,a|\theta^{Q})\) by minimizing the following loss function:  
\[L=\frac{1}{\mathbf{I}}\sum_{w=1}^{\mathbf{I}}(\tau_{w}+\gamma Q^{\prime}(s^{{}^ {\prime}}_{w},a^{\prime}|\theta^{Q^{\prime}})-Q(s_{w},a|\theta^{Q})) \tag{16}\]  
In line 12, we update the actor network, which represents the policy parameterized by \(\theta\). It maximizes \(\bigtriangledown_{\theta^{\prime}}\) using stochastic gradient ascent, which is given by:  
\[\bigtriangledown_{\theta^{\prime}}J\approx\frac{1}{\mathbf{I}}\sum_{w=1}^{ \mathbf{I}}\bigtriangledown_{a}Q^{\prime}(s_{w},a|\theta^{Q})|_{a=a_{w}} \bigtriangledown_{\theta^{\prime}}\mu(s_{w}|\theta^{\mu}) \tag{17}\]  
Finally, the target network is updated by \(\theta^{\prime\prime}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\prime\prime}\) and \(\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\).