# Joint Pricing, Server Orchestration and Network Slice Deployment in Mobile Edge Computing Networks
## 4 The Three-Stage Stackelberg Game
### The Double-Layer BDQ Model for the NPL and RAL
#### 5.2.1 The BDQ Model for the NPL  
The elements of the DRL problem in the NPL are summarized as follows:  
* State: \(s_{t}^{NP}=\{[u_{n,q}^{t}]_{N\times Q},\ \mathcal{L}_{t}\}\).
* Action: \(\hat{a}_{t}^{NP}=\{[h_{m}^{t}]_{M\times 1},[x_{m,n}^{t}]_{M\times N}\}\). \(h_{m}\) indicates which server the base station \(m\) is associated with. One thing to specify is that once the action \(\hat{a}_{t}^{NP}\) is given, the variables \(\mathbf{a}\), \(\mathbf{p}\) and \(\mathbf{k}\) can be easily derived.
* Reward: \(R_{t}^{NP}=U_{t}^{NP}\).  
The action space is \((M+MN)\)-dimensional and discrete. The first \(M\) dimensions take values in \(\{0,...,V-1\}\), and the last \(MN\) dimensions take values in \(\{0,1\}\). Thus, the action space is \(V^{M}2^{MN}\). The complexity of this neural network grows exponentially with \(M+MN\), making it difficult to converge. As such, we introduce the BDQ architecture [26] to solve this DRL problem.  
BDQ is a modification on D3 QN, and its evaluation network is shown in Figure 4. The input layer, partially hidden layers and state value estimator of BDQ are the same as those of D3 QN, i.e., the blue, yellow, and red regions in Figure 4. The difference is that BDQ changes the action advantage estimator into multiple independent branching networks, i.e., the purple region. Each branching network is responsible for evaluating the action advantage value for each action dimension. When the input layer provides environment state information, the hidden layer in the yellow region computes the latent representation, which is used to compute the state values (red region) as well as the factorized action advantages on the subsequent independent branches (purple region). The output layer (green region) is responsible for aggregating the state value and the factorized action advantages to output the state-action value for each action dimension. Then, each action dimension selects the action with the maximum state-action value, which is combined to form the final action. Specifically, suppose that action \(\hat{a}\) has \(F\) sub-actions, i.e., the action dimension is \(F\). The number of optional actions for each sub-action is \(|\mathcal{A}_{f}|\). \(A_{f}(s,\hat{a}^{f};\mathbf{\varphi})\) denotes the action advantage of sub-action \(f\) and \(V(s;\mathbf{\varphi})\) denotes the state value; then, the state-action value can be expressed as  
\[\hat{Q}_{f}(s_{t},\hat{a}_{t}^{f};\mathbf{\varphi})=V(s_{t};\mathbf{\varphi})+A_{f}(s _{t},\hat{a}_{t}^{f};\mathbf{\varphi})-\frac{1}{|\mathcal{A}_{f}|}\sum_{a^{f}\in \mathcal{A}_{f}}A_{f}(s_{t},\hat{a}^{f};\mathbf{\varphi}). \tag{20}\]  
BDQ averages the state-action values of all dimensions to compute the TD-target, i.e.,  
\[\hat{y}_{t}=R_{t}+\gamma\frac{1}{F}\sum_{f=1}^{F}\hat{Q}_{f}(s_{t+1},\arg\max _{\hat{a}_{t+1}^{f}}\hat{Q}_{f}(s_{t+1},\hat{a}_{t+1}^{f};\mathbf{\varphi});\hat{ \mathbf{\varphi}}), \tag{21}\]  
where \(\hat{\mathbf{\varphi}}\) is the network parameter of the target network. The loss is defined as the expected value of the TD-error across all the branches:  
\[L(\mathbf{\varphi})=\mathbb{E}[\frac{1}{F}\sum_{f=1}^{F}(\hat{y}_{t}-\hat{Q}_{f}( s_{t},\hat{a}_{t}^{f};\mathbf{\varphi}))^{2}]. \tag{22}\]  
When performing the PER, distributed TD-errors need to be aggregated into a uniform form. To preserve the magnitude of the error, the uniform prioritization error is defined as the sum of the absolute value of the TD-error across all the branches. In addition, the BDQ selects an action for each sub-action according to:\[\begin{split} a_{t}^{f,*}=\arg\max_{\hat{u}_{t}^{f}}\hat{Q}_{f}(s_{t},a_ {t}^{f};\mathbf{\varphi}).\end{split} \tag{23}\]  
For the BDQ problem of the NPL, there are \(M+MN\) branches, i.e., \(F=M+MN\). The first \(M\) branches represent \(\left[h_{m}\right]_{M\times 1}\), each with \(V\) optional actions. The last \(MN\) branches represent \(\left[x_{m,n}\right]_{M\times N}\), each with two optional actions. Thus, the number of network's outputs is reduced to \(VM+2 MN\).