## 4 The Proposed DRLEA Approach
### Markov Decision Process  
We define the state at time \(t\) as \(s_{t}\), and the selected action under the state \(s_{t}\) is \(A_{j}\). Moreover, we consider using the four antenna parameters [\(\phi_{tilt}\), \(\theta_{tilt}\), \(\phi_{3 dB}\), \(\theta_{3 dB}\)] at time \(t\) as the state \(s_{t}\). The action \(A_{j}\in\mathcal{A}\) is defined as the change of one set of antenna parameters, where \(\mathcal{A}\) denotes the action set of the HAPS. To reduce the computational complexity, we use discrete antenna parameters to reduce the number of actions. Moreover, to perform the all actions at once, we design the action mapping list as shown in Table 2. We assume that the number of antenna parameters of each antenna array is \(P=4\), and the number of values of each antenna parameter is \(V=3\). Thus, the number of actions of each antenna array is \(L=V^{P}\), and the number of actions of a HAPS is \(J=L^{N}\). \(A_{1}(a_{(1,1)},\ldots,a_{(N,1)})\) in Table 2 indicates that antenna array \(1\) to antenna array \(N\) performs action index \(0\). Figure 5: Throughput. The reward \(\mathcal{R}(s_{t},A)\) with the state \(s_{t}\) and the action \(A\) is represented by the Equation (5). \[\mathcal{R}(s_{t},A)=\frac{\sum_{i,j}^{\frac{K}{2}}\mathcal{T}_{t}^{\prime}- \mathcal{T}_{j}}{\frac{K}{2}}, \tag{5}\]  
where \(K\) denotes the number of users, \(\sum_{i}^{\frac{K}{2}}\mathcal{T}_{t}^{\prime}\) denotes the sum of the throughput of the 50 percent users with the least throughput after performing the selected action \(A\), and \(\sum_{j}^{\frac{K}{2}}\mathcal{T}_{j}\) denotes the sum of the throughput of the 50 percent users with the least throughput under the initial state. Thus, the antenna parameters control problem can be represented by a Markov Decision Process (MDP): \(\mathcal{M}\triangleq(\mathcal{S},\mathcal{A},\mathcal{R},p,\alpha)\), where \(\mathcal{S}\) denotes an infinite state space, \(p\) denotes the transition probability that characterizes the stochastic evolution of states in time, with the collection of probability distributions over the state space \(\mathcal{S}\), and \(\alpha\in[0,1)\) is the reward discount factor. The goal is to find a deterministic optimal policy \(\pi^{*}:\mathcal{S}\rightarrow\mathcal{A}\), such that:  
\[\pi^{*}:=\arg\max_{\pi\in\Phi}\mathbb{E}\left[\sum_{t=0}^{\infty}a^{t}r(s_{t}, \pi(s_{t}))\right], \tag{6}\]  
where \(\Phi\) is the set of all admissible deterministic policies. At time step \(t\), HAPS selects an action simultaneously based on the policies \(\pi\). The \(\mathcal{Q}\) function is shown as follows:  
\[\mathcal{Q}^{\pi}(s,A):=\mathbb{E}\left[\sum_{t=0}^{\infty}a^{t}\mathcal{R}(s_ {t},\pi(s_{t}))\mid s_{0}=s,A_{0}=A\right]. \tag{7}\]  
Thus, the optimal policy \(\pi^{*}\) can be obtained by:  
\[\pi^{*}(s) =\arg\max_{A}Q^{*}(s,A) \tag{8}\] \[=\mathbb{E}_{s^{\prime}\sim p(\cdot\mid s,A)}\left[\mathcal{R}(s, A)+\max_{A^{\prime}}\alpha\mathcal{Q}^{*}(s^{\prime},A^{\prime})\right]\]  
where \(s^{\prime}\) denotes the next state, \(\max_{A^{\prime}}Q^{*}(s^{\prime},A^{\prime})\) denotes performing the action \(A^{\prime}\) that can obtain the maximum \(\mathcal{Q}\) value under the state \(s^{\prime}\). \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Actions of the HAPS** & \begin{tabular}{l} **Actions of** \\ **the Antenna** \\ **Array**\(i\) \\ \end{tabular}  & \begin{tabular}{l} **Action** \\ **Index** \\ \end{tabular}  & \begin{tabular}{l} **\(\phi_{3 dB}\)** \\ **\end{tabular}  & \begin{tabular}{l} \(\theta_{3 dB}\)** \\ **\end{tabular}  & \ \begin{tabular}{l} \(\phi_{tilt}\)** \\ **\end{tabular}  & \begin{tabular}{l} \(\theta_{tilt}\)** \\ **\end{tabular}