# On the Application of Machine Learning to the Design of UAV-Based 5G Radio Access Networks
## 4 ML-Based Designs  
Design of wireless communications systems is dominated by model-based approaches. However, it is difficult to develop accurate models in highly dynamic environments and, moreover, most existing algorithms cannot perform full processing and utilization of the available data. As a result, much valuable information and patterns in the data are unexploited, so-called hidden information. ML techniques can be a powerful tool to design U-RANs using example data or past experience [15; 17; 18; 37]. ML algorithms can be classified as supervised, unsupervised, and _reinforcement learning_. In supervised learning (SL), the aim is to automatically acquire a mapping from the input to the output, whose correct values (referred to as labels) are provided by a supervisor during a training phase. In very complex scenarios with multiple UAVs, there are plenty of parameters that can be considered as input data training for SL. In the case of SL, a ML box can be used having as input the features and variables related to the defined trade-offs of the network and a loss function can be defined based on the given scenario. In unsupervised learning, there is no such supervisor, and the algorithm must autonomously find similarities, patterns, and differences in the available data without any prior training. However, a sufficient amount of training data may not be available in many cases. In some applications, the output of a system is a sequence of actions. In such a case, an ML algorithm should be able to find the best sequence of actions, i.e., a policy, to reach a given goal, by observing past good action sequences. Such learning methods are called reinforcement learning (RL) algorithms, and are based on sequential decisions, where the next step input depends on the decision of the system. In contrast to SL, where the algorithm learns through analyzing a labeled training dataset, RL algorithms discover the environment in order to learn which actions are the best to be taken at specific states. Due to their capability of online learning, RL algorithms can be particularly  
Figure 4: An example of satellite backhauling. suitable for U-RANs. Examples of basic RL elements, namely actions, states, and reward functions, are reported in Table 1. As an example, let us consider a non-flat area including many buildings, trees, and open area markets, and suppose the goal is to find the best location of an UAV that has to broadcast some data to the users. In this case, an RL approach will consider the number of covered users as a reward function. The state can be defined in terms of UAV movement directions, the agent is the UAV itself [15; 19], and the actions are UAV speed values and trajectories. This approach can help to find the best location of an UAV in online manner when no training dataset is available, while finding the best location of UAV through conventional optimization methods is too complicated [20]. Moreover, SL and RL can be combined together [1; 37; 63]. In general, a reward based function can be defined based on the scenario, e.g., increasing coverage or the number of the users as it is shown in Table 1. In a well-defined area with uniform distribution of the users, increasing the number of supported users can be considered as an improvement in coverage. Moreover, coverage can be increased by increasing UAVs altitude, but interference will be increased as well. Furthermore, hovering time decreases when UAV altitude is increased. UAVs make different actions in a defined period of time and the number of satisfied users are counted in each step. Based on the defined reward function, the configuration with the highest number of supported users is chosen. This can be done in real time if the user traffic is not delay-sensitive. On the contrary, when delay-sensitive applications must be supported by U-RANs encompassing many UAVs, RL may not be a convenient choice, as it might take a long time to find the best location based on the reward function [64].