## 4 The Proposed DRLEA Approach
### Deep Reinforcement Learning Evolution Algorithm  
In the DQN method, the \(\mathcal{Q}\) value of each action is calculated based on the reward returned from the environment states. During the repeating of the experimental process, DQN can learn how to adjust antenna parameters to reduce the number of low-throughput users. Same with DQN approach, we obtain the \(\mathcal{Q}\) value by using the Equation (10). However, using the DQN method for the HAPS system is difficult to search for the optimal solution. We use Figure 6 to explain the reason.  
Figure 6: An example of how to find the optimal solution.  
In Figure 6, the 'pbest' is one of the sub-optimal solutions, the 'gbest' is the optimal solution. The DQN agent will adjust the antenna parameters step-by-step to find the optimal solution by using gradient descent. Nonetheless, if the initial state is located in the green point shown in Figure 6, the DQN agent cannot obtain the optimal solution even using the Epsilon-greedy method for action selection. To address this problem, we design a novel DRLEA algorithm as shown in Algorithm 4. The workflow of the proposed method is shown in Figure 7.  
Different from the DQN method with the same initial state at the beginning of each training epoch, the DRLEA will randomly generate a different initial state for each training epoch, such as the yellow points shown in Figure 6. For each epoch, the DRLEA performs many steps to search for the 'optimal' solution (actually is a sub-optimal solution). After a training epoch, the DRLEA compares the 'optimal' solution with the  
```
1:\(s_{t},\gamma,\epsilon\), actions \([a_{1},\dots,a_{l}]\).
2:\(\mathcal{Q}(s_{t},a_{1})\), \(\mathcal{Q}(s_{t},a_{2}),\dots,\mathcal{Q}(s_{t},a_{l})\).
3:Initialize main network \(\mathcal{F}(\Theta)\) and target network \(\mathcal{F}(\Theta^{{}^{\prime}})\).
4:Initialize experience reply memory \(\mathcal{D}\).
5:Initialize \(\mathcal{X}\) with random matrices.
6:\(\mathcal{X}=(x_{0},\dots,x_{i},\dots,x_{E})\) denotes the \(E\) different randomly generated antenna parameters.
7:\(x_{i}=[\phi_{3 dB},\phi_{tilt},\theta_{3 dB},\theta_{tilt}]\).
8:Initialize \(pbest\) and \(gbest\) with zero matrices.
9:for\(i=0\) to \(E\)do
10:Initialize the antenna parameters.
11:Initialize \(s_{t}=x_{e}\), \(pbest\).
12:for step \(t=0\) to \(T\)do
13:\(A=\arg\max_{j}\mathcal{Q}(s_{t},A_{j};\Theta)\)
14: Perform the action \(A\).
15: Get reward \(\mathcal{R}(s_{t},A)\) by Equation (5).
16:if\(\mathcal{R}(s_{t},A)\leq\max(0,\mathcal{R}(s_{t-1},A^{t-1}))\)then
17: Randomly select an action \(A\in\mathcal{A}\)
18: Perform the action \(A\).
19:endif
20:if\(\mathcal{R}(s_{t},A)\geq\mathcal{R}(s_{t-1},A^{t-1})\)then
21: Update \(pbest\): \(pbest=pbest+A\)
22:endif
23: Update \(s_{t+1}\): \(s_{t+1}=s_{t}+A\).
24: Store \((s_{t},A,\mathcal{R}(s_{t},A),s_{t+1})\) in the \(\mathcal{D}\).
25: Get the output of the main network \(\mathcal{F}(\Theta)\):
26:\(\mathcal{Q}(s_{t},A)=\mathcal{F}(s_{t},A;\Theta)\).
27: Generate target \(\mathcal{Q}\) value:
28:\(\mathcal{Q}\left(s_{t+1},A^{t+1}\right)=\max\mathcal{F}(s_{t+1};\Theta^{{}^{ \prime}})\).
29: Update the main network \(\mathcal{F}(\Theta)\) to minimize the loss function: \[\begin{split}&\mathcal{L}(\Theta)=\text{MSE}\bigg{(}\mathcal{Q}(s_ {t},A),\\ &\mathcal{R}(s_{t},A)+\frac{\alpha}{i+1}\mathcal{Q}^{{}^{\prime}} \Big{(}s_{t+1},A^{t+1}\Big{)}+\Big{(}\alpha-\frac{\alpha}{i+1}\Big{)}*\Big{(} 1-\frac{|s_{t}-gbest|}{|gbest|}\Big{)}\bigg{)}\end{split}\] (14)
30:endfor
31:if\(\mathcal{R}(s_{t},A)\geq\mathcal{R}_{\text{best}}\)then
32:\(\mathcal{R}_{\text{best}}=\mathcal{R}(s_{t},A)\)
33:\(gbest\gets pbest\)
34:endif
35: Update the target network: \(\mathcal{F}(\Theta^{{}^{\prime}})\leftarrow\mathcal{F}(\Theta)\).
36:endfor
```  
**Algorithm 4** Deep Reinforcement Learning Evolution Algorithm