## 4 The Proposed DRLEA Approach
### Conventional Methods
#### 4.2.2 DQN  
DQN is another classical RL [39]. It calculates the \(\mathcal{Q}\) value of each action based on the reward returned from the environment states and uses the DNN instead of \(\mathcal{Q}\) table in \(\mathcal{Q}\)-learning method to predict the optimal \(\mathcal{Q}\) value. Therefore, we can obtain the \(\mathcal{Q}\) value with the state \(s\) and the action \(A\) based on Equation (10).  
\[\mathcal{Q}(s,A)\leftarrow\mathcal{R}(s,A)+\alpha\max_{j}\mathcal{Q}(s^{ \prime},A_{j}). \tag{10}\]  
Different from \(\mathcal{Q}\)-learning searching \(\mathcal{Q}\) table to obtain the optimal solution, the DQN method builds a deep neural network to learn the \(\mathcal{Q}\) value of each possible action corresponding to the input environment state. The details of the proposed DQN-based antenna control method are shown in Algorithm 2.  
```
0:\(s_{t},\alpha,\epsilon,\mathcal{A}\).
0:\(\mathcal{Q}(s_{t},A_{1})\), \(\mathcal{Q}(s_{t},A_{2}),\ldots,\mathcal{Q}(s_{t},A_{I})\).
1: Initialize main network \(\mathcal{F}(\Theta)\) and target network \(\mathcal{F}(\Theta^{{}^{\prime}})\).
2: Initialize experience reply memory \(\mathcal{D}\).
3:for epoch = 0 to \(E\)do
4: Initialize the antenna parameters.
5:for step \(t=0\) to \(T\)do
6: Obtain the antenna parameters as state \(s_{t}\).
7: Randomly generates \(r\) in the range of (0,1).
8:if\(r>\epsilon\)then
9: Randomly select an action \(A\in\mathcal{A}\)
10:else
11:\(A=\operatorname*{arg\,max}_{j}\mathcal{Q}(s_{t},A_{j};\Theta)\)
12:endif
13: Perform the action \(A\).
14: Get reward \(\mathcal{R}(s_{t},A)\) by Equation (5).
15: Update \(s_{t}\) to \(s_{t+1}\).
16: Store \((s_{t},A,\mathcal{R}(s_{t},A),s_{t+1})\) in the \(\mathcal{D}\).
17: Get the output of the main network \(\mathcal{F}(\Theta)\): \(\mathcal{Q}(s_{t},A)=\mathcal{F}(s_{t},A;\Theta)\).
18: Generate target \(\mathcal{Q}\) value:
19:
20:\(\mathcal{Q}^{{}^{\prime}}(s_{t+1},A^{\prime})=\max\mathcal{F}(s_{t+1};\Theta^{ {}^{\prime}})\).
21: Update the main network \(\mathcal{F}(\Theta)\) to minimize the loss function: \[\begin{split}\mathcal{L}(\Theta)=& MSE(\mathcal{Q}(s_{t},A), \mathcal{R}(s_{t},A)+\\ &\alpha\mathcal{Q}^{{}^{\prime}}(s_{t+1},A^{\prime}))\end{split}\] (11)
22:endfor
23: Update the target network: \(\mathcal{F}(\Theta^{{}^{\prime}})\leftarrow\mathcal{F}(\Theta)\).
24:endfor
```  
**Algorithm 2** DQN-based method for HAPS antenna control.  
In Algorithm 2, \(E\) denotes the maximum number of epochs, \(T\) denotes the maximum number of steps, and \(\Theta\) denotes the weights of the deep neural network. We build two DQNs, one is the main network used for evaluating the \(\mathcal{Q}\) value of the action obtained by the \(\epsilon\)-greedy policy. This main network is trained during each step to estimate the approximate optimal action in the current state. The target network is updated with a copy of the latest learned parameters of the main network after each epoch. In other words, using a separate target network helps keep runaway bias from dominating the system numerically causing the estimated \(\mathcal{Q}\) values to diverge. Thus, using two DQNs instead of only one DQN can avoid the DQN algorithm to overestimate the true rewards [40]. We calculate the reward \(\mathcal{R}(s_{t},A)\) with the state \(s_{t}\) and the action \(A\) by Equation (5). In each step, we select an action based on the \(\epsilon\)-greedy method, and store the current state \(s_{t}\), the selected action \(A\), the reward \(\mathcal{R}(s_{t},A)\), and the next state \(s_{t+1}\) into the experience replay memory \(\mathcal{D}\) for training the main network \(\mathcal{F}(\Theta)\). Next, we train the main network \(\mathcal{F}(\Theta)\) to minimize the loss function \(\mathcal{L}(\Theta)\), as shown in Equation (11). After enough training, we input the current state \(s_{t}\) into the main network, then we can observed the predicted \(\mathcal{Q}\) value of all actions under the state \(s_{t}\). Thus, the optimal action \(A=\operatorname*{arg\,max}_{j}\mathcal{Q}(s_{t},A_{j};\Theta)\) can be obtained.