# Deep Reinforcement Learning for UAV-Based SDWSN Data Collection
## 5 Simulation Outcomes  
Our simulation is carried out via MATLAB presuming an initial path within the UAV fuzzy range. The initial path is defined based on Figure 4 and after multiple episodes, the UAV path will converge into an optimal path considering the proposed cost function (3). The intention here is that the agent (UAV) visits the optimal data points as a part of its designed path to optimize the communication throughput and energy consumption by introducing the optimal location and velocity of the UAV within each state. The agent begins its journey from coordinate (0,0). The rewards are considered +10 for a constructive action while, for a destructive action, the reward is \(-1\). The learning policy using Q-learning is shown in Figure 3.  
To begin with, as outlined in Figure 4, a UAV fuzzy range is designed for data collection over 100 ground sensor nodes via gateways. The connectivity window is also mapped over the UAV's adjusted path within the fuzzy route shown in the pink circle. The designed fuzzy route is represented by the hatched area. The initial UAV path is applied via the defined RL model in the UAV fuzzy range shown in the pick route. Once the Q-table is sufficiently trained, the optimal path can be extracted by choosing the action with the highest Q-value for the current state. The initial and destination positions of UAVs are the same. The UAV is set to fly at an altitude of 100 m, which aligns with real-world conditions. At this height, the UAV can navigate smoothly above the canopies for data-gathering purposes, avoiding any obstructions caused by their height. The UAV velocity range is assumed to be between 5-40 m/s and the size of the grid considered is 1000 \(\times\) 1000 m\({}^{2}\). The simulation parameters are outlined in Table 1.  
The proposed RL algorithm is highly dependent on the learning parameters choices. Learning rate \(\alpha\) determines the extent to which the model adjusts its knowledge with each new data point. We have presumed the learning rate \(\alpha\) = 0.1 based on our simulations. The other factor is the discount factor \(\gamma\) which determines the significance of future rewards. We have presumed discount factor \(\gamma=0.9\) based on our simulations which ensure that the agent considers long-term rewards while making decisions. The exploration rate is another factor in defining the RL algorithm. The optimal choice for this parameter varies during the experiment, started with a high exploration rate (\(\epsilon\) = 1) and gradually decreased over time. This strategy, known as epsilon-greedy, facilitates the agent to explore the environment initially and then exploit the learned knowledge as it gains more experience. Figure 4: The proposed fuzzy vs. initial path for UAV path design. Following the training of the initial path via the proposed Q-Learning RL algorithm, the emerged path has the optimal energy consumption and communication bit rate on UAV ground connectivity. Following 300 episodes of experimentation, the resulting path achieves acceptable performance compared to the initial path design by reaching the episodes to the end. This episode's number choice can provide enough time for the agent to explore the environment and learn from a variety of states and actions. Additionally, it helps the learning algorithm converge to a stable policy relatively quickly, while extending the training episodes might only result in minor improvements at a much higher computational cost. Figure 5 illustrates the performance of our proposed optimized model, which enhances our initial path performance over the proposed Q-Learning RL. In this figure, the energy consumption and communication bit rate for UAV-Gateway connectivity are computed using Equations (1) and (2) for each episode, allowing us to assess the system's performance across multiple episodes.