# Joint Pricing, Server Orchestration and Network Slice Deployment in Mobile Edge Computing Networks
## 4 The Three-Stage Stackelberg Game
### The Double-Layer BDQ Model for the NPL and RAL
#### 5.2.4 The Implementation of the Double-Layer MBDQ  
We present the training procedure for the double-layer MBDQ (DMBDQ) in Algorithm 3. \(\mathbf{\varphi}\) and \(\mathbf{\xi}\) are set as the network parameters for the NPL and RAL. In each training round, the NPL first selects an action based on the greedy policy (line 5). Then, it needs to judge whether there is an illegal action or not and performs the illegal action modification if it exists (line 6). The RAL receives the NPL's action, obtains the current state and selects an action using greedy policy (lines 7, 13). After that, the RAL performs illegal action judgment and correction (line 14). The resource allocation scheme will be obtained using a math solver (line 15). In this paper, we use the MOSEK solver [38]. If P4 is insoluble, we will give a penalty on \(R_{t}^{NP}\) and \(R_{t}^{RA}\). Once \(\hat{a}_{t}^{NP}\) and \(\hat{a}_{t}^{RA}\) are determined, the rewards are computed and the actions are fed back to the IPL (lines 16-17). We then store transition \((s_{t}^{NP},\hat{a}_{t}^{NP},R_{t}^{NP},s_{t+1}^{NP})\) into the replay memory buffer of the NPL (line 18). After that, we use PER to select a mini-batch from the buffer and update the network parameters of the NPL by gradient descent (lines 19-20). Note that the RAL's state of the next round is determined by the NPL's action in the next round, so the RAL needs to delay storing experience and updating the network to the next round (lines 8-12). Finally, we update the target networks every fixed number of rounds (line 21).  
```
1:Initialize NPL with random weights \(\mathbf{\varphi}\) and target network with \(\mathbf{\hat{\varphi}}=\mathbf{\varphi}\); replay memory buffer \(\mathcal{D}^{NP}\).
2:Initialize RAL with random weights \(\mathbf{\xi}\) and target network with \(\mathbf{\hat{\xi}}=\mathbf{\xi}\); replay memory buffer \(\mathcal{D}^{RA}\).
3:NPL receives pricing information from IPL and gets initial state \(s_{1}^{NP}\).
4:for\(t=1\) to \(T\)do
5: Select a random action \(\hat{a}_{t}^{NP}\) with probability \(\epsilon\), otherwise get \(\hat{a}_{t}^{NP}\) by (23).
6: Modify \(\hat{a}_{t}^{NP}\) by Algorithm 2.
7: RAL receives information and gets the state \(s_{t}^{RA}\).
8:if\(t>1\)then
9: Store transition \((s_{t-1}^{RA},s_{t-1}^{RA},R_{t-1}^{RA},s_{t}^{RA})\) in \(\mathcal{D}^{RA}\).
10: Sample a mini-batch from \(D^{RA}\) by PER.
11: Perform gradient descent with respect to \(\mathbf{\xi}\).
12:endif
13: Select a random action \(\hat{a}_{t}^{RA}\) with probability \(\epsilon\), otherwise get \(\hat{a}_{t}^{RA}\) by (23).
14: Modify \(\hat{a}_{t}^{RA}\) by Algorithm 2 and obtain \(\mathbf{y}\) by (24).
15: Use a math solver to solve \(P4\) and obtain \(\mathbf{w}_{t},\mathbf{e}_{t},\mathbf{s}_{t}\).
16: Calculate rewards \(R_{t}^{NP}\) and \(R_{t}^{RA}\).
17: Feedback \(\hat{a}_{t}^{NP}\) and \(\hat{a}_{t}^{RA}\) to the IPL and get \(s_{t+1}^{NP}\).
18: Store transition \((s_{t}^{NP},\hat{a}_{t}^{NP},R_{t}^{NP},s_{t+1}^{NP})\) in \(\mathcal{D}^{NP}\).
19: Sample a mini-batch from \(D^{NP}\) by PER.
20: Perform gradient descent with respect to \(\mathbf{\varphi}\).
21: Set \(\mathbf{\hat{\varphi}}=\mathbf{\varphi}\) and \(\mathbf{\hat{\xi}}=\mathbf{\xi}\) every \(\omega\) rounds.
22:endfor
```  
**Algorithm 3** DMBDQ algorithm for the NPL and the RAL