# A Survey on Machine-Learning Techniques for UAV-Based Communications
## 5 Position Related Aspects
### Placement and Trajectory Design
The superior performance of this algorithm was demonstrated in the simulation results, where the spectral efficiency of the system achieved 91.3% maximum spectral efficiency, with lower complexity than conventional genetic algorithms, such as hill climbing and simulated annealing algorithms. In Reference [124], the self-optimization of multiple UAVs' trajectory in real-time sensing applications was tackled using a Q-learning learning method in a decentralized manner. A single-cell UAV orthogonal frequency division multiple access (OFDMA) network was considered, where the UAVs transmit the sensory data to a terrestrial BS over orthogonal subchannels to avoid mutual interference and the location of the terrestrial base station and the UAV was specified by 3-D cartesian coordinates. It was also considered that the UAVs perform the sensing tasks in a synchronized iterative manner and send the measured data to the base station, whereas the sensing quality of the UAVs was evaluated using the probabilistic sensing model [125]. In addition, a sense-and-send protocol was proposed that facilitates the coordination of the UAVs handling different tasks, and then, the probability for successful valid data transmission using nested Markov chains was investigated. A frame-level simulation of this protocol was built in MATLAB(r), and the results underlined the rapid convergence of the proposed algorithm compared to traditional single- and multi-agent Q-leaning algorithms. A distributed DRL algorithm for the navigation of a group of UAVs acting as BSs flying in a target region was developed in Reference [126]. The optimization problem was formulated as a partially observable MDP (POMDP), where the UAVs were capable of only observing the areas in their vicinity. Since each UAV has connectivity constraints and limited battery lifetime, the optimization problem aimed at improving the temporal average radio coverage and geographical fairness while minimizing the total energy consumption of the UAVs. In this algorithm, an agent, i.e., the UAV, and the environment interact at each of a sequence of discrete timeslots. The training of the algorithm was obtained using the observations of each UAV regarding the propagation environment. According to the simulation results, the proposed optimization method outperforms the state-of-the-art DRL-EC3 approach based on deep deterministic policy gradient (DDPG) [127] in terms of energy efficiency. A double Q-learning algorithm that handles the problem of trajectory design of UAVs was presented in Reference [128]. Although standard Q-learning algorithms use the same Q-table for selection and evaluation processes and usually tend to overestimation and suboptimal results, this algorithm uses two Q-tables to decouple the selection from the evaluation. The effectiveness of the proposed algorithm was demonstrated via extensive simulations, and the gain was up to 19.4% and 6.7% regarding the number of satisfied users compared to the random algorithm and Q-learning algorithm, respectively. Further advancements on UAV navigation were given in Reference [129]. In greater detail, a UAV navigation scheme based on DRL that can select the best UAV-to-ground links in real-time was presented for a massive MIMO system which included one ground station with a large number of antennas and multiple UAVs with single antennas. First, a DQN was constructed to extract the environment information and to obtain useful features of the massive MIMO channel, and then, the DQN was trained in order to facilitate the decision-making procedure based on the received signal strengths. Using the DQN, a Q-learning policy was also exploited for successfully optimizing UAV navigation while achieving increased coverage and rapid convergence. The high performance of the proposed DQN navigation scheme was confirmed through extensive simulation results, where the channel was Rician with a Rician factor of 6 dB and a ground station with 128 transmit antennas and 32 single-antenna UAVs flying at different velocities were considered.