# Autonomous Driving Control Using the DDPG and RDPG Algorithms
## 4 Experimental Results  
In this section, we illustrate our simulation in detail. The Coastline map provided by AirSim was divided into the training and testing part. Then, the training path and two driving scenarios are shown in Figure 11, i.e., the starting point and the destination. Notice  
Figure 10: Critic architecture of the RDPG. Figure 9: Actor architecture of the RDPG. that the training and testing path are mutually exclusive to provide a fair assessment of the experiment; this is important and significant. The simulation settings are shown as follows:  
* Inputs: driving view image and driving speed;
* Due to the memory limitation, the driving view image was resized to \(224\times 224\) for both the training and testing procedures;
* The inputs at the previous timestamp were treated as the inputs as well for the memory property of the RDPG;
* Outputs: accelerator, brake, and steering;
* The range of the accelerator was set to \([0,1]\);
* The range of the brake was set to \([0,1]\);
* The range of the steering was set to \([-1,1]\) from left to right;
* Hyperparameters are detailed here: learning rate: 0.0001 (actor) and 0.001 (critic); learning rate decay: 0.9; replay buffer: 10,000; replay buffer threshold: 500; batch size: 64; \(\epsilon\) from the start: 1; \(\epsilon\) decay: 0.99; minimum \(\epsilon\): 0.01;
* The simulation was executed 1000 times for each DRL method;
* Arriving at the destination or violating a traffic regulation will reset the environment to the initial state;
* The specification of the experimental computer: CPU: Intel Core i7-9700 3.00 GHz, RAM: 16 GB DDR4 3200 MHz, and GPU: Nvidia GeForce RTX 2080 8 GB*2 with the Scalable Link Interface (SLI). Figure 12 shows the reward of the training procedure, which demonstrates the average reward values with respect to the corresponding algorithms of every 100 driving rounds. Note that the reason for considering every 100 driving rounds as a whole was that doing so allows presenting the simulation results more clearly. We can observe that the driving strategy with the RDPG approach had better performance: the reward values accumulated via the RDPG approach were apparently higher in the early phase. Namely, the RDPG had better convergence performance than the DDPG. In the training procedure also, the first time the driving strategy with RDPG reached the destination was in the 223 rd round, and that with the DDPG was in 419 th round. Similarly, the RDPG also had a higher arrival rate than the DDPG during the training procedure (Figure 13), especially in the early phase. Figure 11: (**a**) The starting point on the training path on the Coastline map; (**b**) the destination on the training path on Coastline map; (**c**) the training path on Coastline map. Next, for the testing procedure, the trajectories of the testing path of the DDPG and RDPG are presented in Figures 14 and 15 accordingly. Obviously, the autonomous driving mission was complete while using the RDPG, but not complete while utilizing the DDPG. In particular, the autonomous control strategy using the DDPG could not pass the sole U-turn, which did not appear in the training path. The intuition is that since RDPG has the memory property, which can adopt the inputs at sequential timestamps, it has better adaptability for a scenario that the model has never seen before, i.e., a U-turn. We performed the testing experiments 20 times. The strategy with the RDPG approach apparently possessed a higher average reward than that with the DDPG approach, i.e., 13,183.8819 vs. 7398.3034. Figure 12: The reward comparison between the DDPG and RDPG in the training procedure. Figure 13: The arrival rate comparison between the DDPG and RDPG in the training procedure. Last but not least, we used the route before the U-turn in the testing path to further discuss the performance of the DDPG and RDPG control strategies, i.e., vehicle steering and speed information, presented in Figures 16-19. Note that the reason for choosing this part was that both the DDPG and RDPG control strategies can complete it.