# Computation Offloading and Resource Allocation Based on P-DQN in LEO Satellite Edge Networks
## 4 P-DQN-Based Approach
### P-DQN Training  
The original Bellman equation in Q-learning and DQN is expressed as follows:  
\[Q(\boldsymbol{s},\boldsymbol{a})=\underset{\boldsymbol{n},\boldsymbol{s}_{ t+1}}{\mathbb{E}}\bigg{[}r_{t}+\gamma\max_{\boldsymbol{a}^{\prime}\in\mathcal{A}}Q( \boldsymbol{s}_{t+1},\boldsymbol{a}^{\prime})\Big{|}\boldsymbol{s}_{t}= \boldsymbol{s},\boldsymbol{a}_{t}=\boldsymbol{a}\bigg{]}. \tag{13}\]In the hybrid action space \(\mathcal{A}_{r}\) for each \(\mathbf{a}\in\mathcal{A}\), the action value function is defined as \(Q(\mathbf{s},\mathbf{a})=Q(\mathbf{s},v,\mathbf{n}_{v})\). If \(\mathbf{v}_{t}\) is the discrete action selected at slot \(t\) and \(\mathbf{n}_{v_{t}}\) is its associated continuous parameter, then the Bellman equation can be written as  
\[Q(\mathbf{s}_{t},v_{t},\mathbf{n}_{v_{t}})=\mathop{\mathbb{E}}_{\mathbf{r}_{t},\mathbf{s}_{t+1}} \Bigg{[}r_{t}+\gamma\max_{v\in V}\sup_{\mathbf{n}_{v}\in\mathcal{N}_{v}}Q(\mathbf{s}_{ t+1},v,\mathbf{n}_{v})\Big{|}\mathbf{s}_{t}=\mathbf{s},\mathbf{a}_{t}=(v_{t},\mathbf{n}_{v_{t}}) \Bigg{]}. \tag{14}\]  
First, solve \(\mathbf{n}_{v}^{*}=\arg\sup_{\mathbf{n}_{v}\in\mathcal{N}_{v}}Q(\mathbf{s}_{t+1},v,\mathbf{n}_ {v})\) for all \(v\in\mathcal{V}\) and then take the maximum \(Q(\mathbf{s}_{t+1},v,\mathbf{n}_{v}^{*})\). When \(Q\) is fixed, for any \(\mathbf{s}\) and \(v\), \(\arg\sup_{\mathbf{n}_{v}\in\mathcal{N}_{v}}Q(\mathbf{s},v,\mathbf{n}_{v})\) can be regarded as the mapping \(\mathbf{n}_{v}^{Q}:\mathcal{S}\rightarrow\mathcal{N}_{v}\), and then (14) becomes  
\[Q(\mathbf{s}_{t},v_{t},\mathbf{n}_{v_{t}})=\mathop{\mathbb{E}}_{\mathbf{r}_{t},\mathbf{s}_{t+1 }}\Bigg{[}r_{t}+\gamma\max_{v\in\mathcal{V}}Q\Big{(}\mathbf{s}_{t+1},v,\mathbf{n}_{v}^ {Q}(\mathbf{s}_{t+1})\Big{)}\ \bigg{|}\ \mathbf{s}_{t}=\mathbf{s}\Bigg{]}. \tag{15}\]  
Similar to DQN, the neural network \(Q(\mathbf{s},v,\mathbf{n}_{v};\mathbf{w})\) is used to approximate \(Q(\mathbf{s},v,\mathbf{n}_{v})\)[50], with \(\mathbf{w}\) as the network weight. For such \(Q(\mathbf{s},v,\mathbf{n}_{v};\mathbf{w})\), the deterministic policy network \(\mathbf{n}_{v}(\cdot;\theta):\mathcal{S}\rightarrow\mathcal{N}_{v}\) can approximate \(\mathbf{n}_{v}^{Q}(\mathbf{s})\), with \(\theta\) as the policy network weight. That is, given \(\mathbf{w}\), for each \(v\in\mathcal{V}\), \(\theta\) is obtained as  
\[Q(\mathbf{s},v,\mathbf{n}_{v}(\mathbf{s};\theta);\mathbf{w})\approx\sup_{\mathbf{n}_{v}\in \mathcal{N}_{v}}Q(\mathbf{s},v,\mathbf{n}_{v};\mathbf{w}), \tag{16}\]  
where \(\mathbf{w}\) is estimated by minimizing the mean square Bellman error by the gradient descent. In the \(t\)-th step (slot) and with the multi-step algorithm with \(j\), the \(j\)-th step's target becomes  
\[y_{t}=\sum_{i=0}^{j-1}\gamma^{i}r_{t+i}+\gamma^{j}\max_{v\in\mathcal{V}}Q(\mathbf{ s}_{t+j},v,\mathbf{n}_{v}(\mathbf{s}_{t+j},\mathbf{\theta}_{t});\mathbf{w}_{t}). \tag{17}\]  
Then, the least squared loss function is used to train \(\mathbf{w}\) as follows:  
\[L_{t}(\mathbf{w})=\frac{1}{2}[Q(\mathbf{s}_{t},v_{t},\mathbf{n}_{v};\mathbf{w})-y_{t}]^{2}. \tag{18}\]  
In particular, the objective is to find \(\theta\) maximizing \(Q(\mathbf{s},v,\mathbf{n}_{v}(\mathbf{s};\theta);\mathbf{w})\) when \(\mathbf{w}\) is fixed, whereas conventional loss functions typically require minimization.