## 4 Simulations
\[\ddot{x}=\frac{1}{m+Mass_{dist}}\big{(}u_{1}u_{x}-C_{dx}\dot{x}\big{)} \tag{54}\]  
\[\ddot{y}=\frac{1}{m+Mass_{dist}}\Big{(}u_{1}u_{y}-C_{dy}\dot{y}\Big{)} \tag{55}\]  
\[\bar{z}=\frac{1}{m+Mass_{dist}}\big{(}u_{1}(\cos\theta\cos\varphi)-C_{dz}\dot{ z}-gm\big{)} \tag{56}\]  
\[Mass_{dist}=Rand\cdot step(t-5) \tag{57}\]In Equation (57), \"_Rand_\" represents a random value between 0% and 70% of the mass of the robot. Every 5 seconds, the mass of the robot is perturbed by a weight disturbance with the value of \(Mass_{dist}\). An RL agent block from the reinforcement learning toolbox was used in the Simulink environment to implement the deep reinforcement learning policy. During the experiments, the presented RL algorithm interacted with the environment in the simulator, thereby learning the appropriate strategy for estimating and updating the parameters of the PID controllers and the gravity compensator. The achieved reward throughout the training process is shown in Figure 7.  
To train the neural networks, different values for hyperparameters were applied, aimed at finding the best possible values. Table 4 shows the optimal values obtained for the hyperparameters. During the aerial robot operation, the trained policy actively estimated the parameters of the controllers. The first experiment was done in two stages (the first time with the proposed controller and the second time with conventional PID controllers) to compare the performance of the proposed controller with that of the conventional PID controller. During the experiments, some weight disturbances (random values between 0% and 70% of the mass of the robot) perturbed the system. The first defined maneuver for the aerial robot was to get to the coordinates (x, y, z = 3.5, 3.5, 4) and stabilize its position in space. Figures 8-10 show the performance of the PID controller and the proposed controller. \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Critic Learning Rate & 0.0001 \\ Actor Learning Rate & 0.00001 \\ Critic Gradient Threshold & 1 \\ Actor Gradient Threshold & 4 \\ Variance & 0.3 \\ Variance Decay Rate & 0.00001 \\ Experience Buffer & 1,000,000 \\ Mini-Batch Size & 64 \\ Target Smooth Factor & 0.001 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The optimal values for hyperparameters. Figure 7: Episode reward and average reward recorded during training. Variations in the values of the manipulated variables \(u_{1},u_{2},u_{3}\) and \(u_{4}\) are plotted in Figures 11 and 12. Figures 13 and 14 exhibit changes in control gain values that occurred during the operation of the robot. Figure 8: Performance of the quadcopter in tracking the desired X and Y positions. Figure 10: Performance of the quadcopter in tracking the desired Roll and Pitch angles. Figure 9: Performance of the quadcopter in attaining desired height and yaw angle. A comparative study using a range of values was conducted to explore the influence of RL hyperparameters on the steady-state error of the altitude control. According to the results of the experiments, zero noise generated the highest steady-state error. Setting the noise to a very high value, however, prevented the actor from learning the best policy, resulting in more errors. It was observed that increasing the variance enhances the exploration of action space. For both actor and critic, the gradient threshold varied between 1, 4, and infinity. Table 5 summarizes the results of the trials. Figure 11: Variations in manipulated variables \(u_{1}\) and \(u_{2}\). Figure 12: Variations in manipulated variables \(u_{3}\) and \(u_{4}\). Figure 13: Variations in adaptive gain values of the X position PID controller. In the second experiment, a waypoint tracking maneuver was conducted to evaluate the ability of the robot, with the proposed control architecture, in tracking consecutive waypoints. Figure 15 illustrates the performance of the aerial robot in tracking the waypoints.