## 5 Simulation Analysis
### Simulation and Discussion  
The simulations are built upon the environment described in the last subsection. We introduce the RAINBOW algorithm to be the baseline and make comparisons with PPO. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Description** & **Value** \\ \hline \(v_{max}\) & Road speed limit & 13.89 m/s \\ \(a_{max}\) & Maximal acceleration of vehicles & 3.0 m/s\({}^{2}\) \\ \(d_{max}\) & Maximal deceleration of vehicles & \(-4.5\) m/s\({}^{2}\) \\ \(\Delta v_{d}\) & Default value for speed difference & \(-14\) m/s \\ \(\Delta a_{d}\) & Default value for acceleration & \(-7.2\) m/s\({}^{2}\) \\  & difference & \\ \(\epsilon\) & The value corresponds to the penalty & 0.01 \\  & item in Equation (18) & 0.01 \\ \(\chi\) & The GAE parameter & 1.0 \\ \(-\) & Learning rate of the Adam optimizer & 0.0001 \\ \(\epsilon_{1}\) & Clip parameter for policy & 0.3 \\ \(\epsilon_{2}\) & Clip parameter for value function & 10.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: General parameter settings for simulations. Figure 6: The simulation scenario. The action space of the RAINBOW agent is set to a 16-dimension vector because the value-based algorithm can only be applied to problems with continuous action space. We divide the acceleration of the ego vehicle between \(-4.5\) m/s\({}^{2}\) and \(3.0\) m/s\({}^{2}\) in steps of \(0.5\) m/s\({}^{2}\); the agent selects an action from the 16-dimension action space, and we will see that the optimal control cannot be implemented in this discrete-action-space way. The scatters in Figure 7 show the data point of different metrics during the training processes of PPO and RAINBOW. In addition, the line plots displayed above the scatters are the linear regression result of the corresponding data. It can be found that the energy consumption of the RAINBOW agent cannot be reduced significantly, despite the reward increases. Compared with RAINBOW, the energy consumption of the PPO agent converges to a nominal value, with extra sacrifice in terms of the travel time. The training profiles illuminate that the proposed PPO-based framework is capable of searching for a solution that enables the vehicle to run in a more energy-saving way. After the training procedure, the well-trained agents are utilized to evaluate their performance. The results are collected from 20 independent simulations, and the random seed is set to the same value for the same round simulation of different control methods. Figure 8 illustrates the evaluation results of PPO, RAINBOW, and IDM, whereas the IDM can reflect the general performance of human drivers. According to Figure 8a, we can see that the RL-based approaches will reduce the mobility of the ego vehicle. However, PPO can save electric energy to a great extent compared with IDM, which can be seen in Figure 8b. On the other hand, although the RAINBOW sacrifices traffic mobility, the energy consumption still remains at a high level. The result proves that the DQN-based algorithm cannot deal with the eco-driving problem in an ATSC environment. Moreover, the mean jerk is improved slightly under the control of PPO compared with the situation of IDM, and this can be caused by some braking behaviors with the intent to recover the energy. Thus, the overall comfort of driving under RL-based control may decrease slightly. On  
Figure 7: The change of reward, energy consumption, and travel time during the training processes of PPO and RAINBOW. average, PPO can save 41.06% energy consumption compared with manual driving (i.e., IDM) in an ATSC environment. Figure 9 presents the results in terms of electricity consumption for 20 groups of independent simulation-based evaluations in detail. The figure demonstrates that the proposed method lowers the consumed energy steadily compared with IDM, which is deemed as representative of normal human driving behavior.