# Joint Pricing, Server Orchestration and Network Slice Deployment in Mobile Edge Computing Networks
## 4 The Three-Stage Stackelberg Game
### Analysis of Stackelberg Equilibrium
When the integer decision variable \(\mathbf{y}\) is determined, the problem can be transformed into:  
\[\begin{split} P4:&\underset{\mathbf{w},\mathbf{e},\mathbf{s}}{ \text{maximize}}\;U^{RA}=-\sum_{n=1}^{N}(w_{d}d_{n}^{nerr}+w_{c}c_{n}^{nerr}) \\ \text{s.t. constraints \eqref{eq:P1},\eqref{eq:P2},\eqref{eq:P3}, \eqref{eq:P4},\eqref{eq:P5},\eqref{eq:P6},\eqref  
* State: \(s_{t}^{InP}=\left[u_{t,q}^{t}\right]_{N\times Q}\).
* Action: \(\hat{a}_{t}^{InP}=\mathcal{L}_{t}\).
* Reward: \(R_{t}^{InP}=U_{t}^{InP}\).  
The action space is 4-dimensional and discrete with each dimension taking values in \(\{0,1,\ldots,L-1\}\). The discrete action algorithm D3 QN [24] is suitable for solving this problem. D3 QN integrates the double deep Q-network (DDQN), prioritized experience replay, and the dueling architecture [40]. D3 QN achieves higher data utilization and can obtain good strategies. In the following, we will introduce D3 QN in detail.  
As shown in Figure 3, D3 QN contains two neural networks, which have the same structure but different parameters. The evaluation network with parameter \(\theta\) is responsible for action selection, and the target network with parameter \(\hat{\theta}\) is used to evaluate the action value [41]. The target network is a copy of the evaluation network, where \(\hat{\theta}\) is copied from \(\theta\) every \(\omega\) steps. Each network contains the input layer, the hidden layers and the output layer. The input layer is responsible for receiving environment state information. The hidden layer performs feature extraction and nonlinear transformation, and multiple hidden layers are fully connected to each other. D3 QN introduces a dueling architecture with two independent estimators. One is used to estimate the state value \(V(s;\theta)\), and the other is used to give the action advantage \(A(s,\hat{a};\theta)\). At the output layer, the outputs of the two estimators are combined to generate the state-action value \(\hat{Q}(s_{t},\hat{a}_{t};\theta)\), i.e.,  
\[\hat{Q}(s_{t},\hat{a}_{t};\theta)=V(s_{t};\theta)+A(s_{t},\hat{a}_{t};\theta) -\frac{1}{|\mathcal{A}|}\underset{\delta\in\mathcal{A}}{\sum}A(s_{t},\hat{a}; \theta), \tag{16}\]  
where \(|\mathcal{A}|\) is the number of all possible actions. The adoption of the dueling architecture ensures that the relative ordering of the advantage function for each action remains unchanged, which improves the algorithmic stability. During training, we store the transition \((s_{t},\hat{a}_{t},R_{t},s_{t+1})\) in the replay memory buffer and select a mini-batch from the buffer for training. The objective of the training is to minimize the loss function, i.e., \(L(\theta)=\mathbb{E}[(\hat{g}_{t}-\hat{Q}(s_{t},\hat{a}_{t};\theta))^{2}]\), where \(\hat{Q}(s_{t},\hat{a}_{t};\theta)\) is the state-action value output by the evaluation network. \(\hat{g}_{t}\) is the temporal difference (TD)-target, which is denoted as  
\[\hat{g}_{t}=R_{t}+\gamma\hat{Q}(s_{t+1},\arg\underset{\hat{a}_{t+1}}{\max}\hat {Q}(s_{t+1},\hat{a}_{t+1};\theta);\hat{\theta}), \tag{17}\]  
Figure 3: The architecture of D3 QN. where \(\gamma\) is the discount factor. It has been proven that experiences with high TD-error are more important because they provide higher learning efficiency. The authors in [42] proposed a framework for prioritized experience replay (PER) to increase the replay probability of transitions with higher TD-error, thus improving the sample efficiency. In D3 QN, PER is employed to select samples for training. In PER, the probability that transition \(i\) is replayed is denoted as \(P(i)=\frac{p_{i}^{n}}{\sum_{i}p_{i}^{n}}\), where \(\alpha\) determines the level of prioritization. \(p_{i}=|\delta_{i}|+\sigma\) and \(\delta_{i}\) is the TD-error, which is expressed as  
\[\delta_{i}(s_{i},\hat{a}_{i},R_{i},s_{i+1})=\hat{y}_{i}-\hat{Q}(s_{i},\hat{a}_ {i};\mathbf{\theta}). \tag{18}\]  
\(\sigma\) is a very small positive value used to guarantee the replay probability is not 0 when \(\delta_{i}=0\).