## 2 Related Work  
However, due to wind pressure, it is difficult for HAPS to remain stationary. Thus, the degradation of the users' throughput and handovers to UEs' end happened [13; 14] after the HAPS coverage shifting. This kind of quasi-stationary state seriously impacts the performance of the communication system [15]. [PERSON] et al. [16; 17] researched the problem of maximization of coverage through optimization of the parameters of the HAPS antenna arrays, and proposed an optimized way to minimize both the coverage gaps between cells and the excessive cell overlap. [PERSON] et al. [18] studied the influence of handover performance when the HAPS is moved or rotated by winds. [PERSON] et al. [19] examined the swing state modeling of the cellular coverage geometry model and the influence of swing on handover. Many studies [11; 20; 21; 22; 23; 24; 25; 26; 27; 28] on antenna control of HAPS proposed employing antenna control methods to prevent interference between surrounding cells and HAPSs to alleviate the decrease in received signal power caused by HAPS shifting or rotation. [PERSON] et al. [20] proposed a beamforming method to reduce the impact of the degradation of system capacity caused by handover between two cells. [PERSON] et al. [21] analyzed the concentric circular antenna array (CCAA) and proposed a Genetic Algorithm (GA) to minimize the maximum side-lobe level (SLL). In [22], [PERSON] et al. further developed the discrete cuckoo search algorithm (IDCSA) used to reduce the maximum SLL under the constraint of a particular half-power bandwidth. To increase the system capacity, [PERSON] et al. [23] also researched SLL reduction. In contrast to existing approaches, they proposed a Symbiotic Organism Search (SOS) algorithm. The SOS algorithm requires no tuning of parameters, which makes it an attractive optimization method. In [24], the particle swarm optimization (PSO) GA is used for reducing the SLL to improve the carrier-to-interference ratio (CIR). However, in high-dimensional space, PSO is easy to enter a local optimum, such as other GAs, and the iterative process' convergence rate is low. These limitations motivate us to develop a new method with high convergence and better throughput performance. With the rapid development of deep learning techniques, reinforcement learning (RL) is widely used in various fields including in 5G and B5G [29; 30; 31]. [PERSON] et al. in [32] used Deep \(\mathcal{Q}\)-Network (DQN) for online learning on how to maximize the users' signal-to-interference plus noise ratio (SINR) and sum-rate capacity. The authors design a binary encoding for performing multiple relevant actions at once in the DQN structure. [PERSON] et al. in [33] used the RL method to solve the virtual network embedding problem (VNEP) in 5G and B5G. The authors considered that the conventional Deep Reinforcement Learning (DRL) usually obtains the sub-optimal solutions of VNEP, which leads to inefficient utilization of the resources and increases the cost of the allocation process. Thus, they proposed a relational graph convolutional neural network (GCNN) combined with DRL to automatically learn how to improve the quality of VNEP heuristics. In [34], the authors proposed a deep learning integrated RL, which combined deep learning (DL)and RL. The DL is used for preparing the optimized beamforming codebook and the RL is used for selecting the best beam out of the optimized beamforming codebook based on the user movements. According to those early works, we can find that the DRL technique can train the neural network with feedback from the environment. Thus, such as solving the beamforming problem in [32] that using the DRL solution does not require the CSI to find the SINR-optimal beamforming vector. Moreover, different from model-free RL methods that need a huge optimal solution searching overhead for solving a complex problem, the DRL approach uses the DNN to predict the optimal solution without searching overhead. The DRL solution can be trained by the SINR feedback from users.