## 4 Experiment Results and Discussion
### Training Process  
First, the human driver's manual driving data are collected with our test platform in the vehicle-following scenario. Two of our colleagues are selected to drive the host vehicle. They both have proficient driving skills but different driving styles. The first driver (Driver 1) is aggressive and tends to use large acceleration and deceleration, whereas the second driver (Driver 2) is non-aggressive, with smoother driving. The data collection for each driver lasts approximately 10 min, with a sampling time of 50 ms. The drivers commanded the vehicle with their driving habits respectively. Then, the supervisor networks with three layers and ten hidden neurons are trained by the collected samples with the MATLAB Neural Network Toolbox. The maximum number of iteration steps, performance goal, and learning rate are set as 1000, 0.001, and 0.01, respectively. Two supervisor networks with different driving styles are obtained from Driver 1 and Driver 2 for the next step.  
In the SRL stage, Algorithm 1 is performed for the CACC control. It should be mentioned that the convergence analysis of the actor-critic reinforcement learning algorithm has been given in [45]. It is shown that the estimation error and the weights in the action and critic networks remain uniformly ultimately boundedness (UUB) under the given conditions. Here, the training parameters are selected on basis of UUB conditions provided by [45]. Both the actor and critic networks have three layers and ten hidden neurons. The initial learning rate of the critic and actor networks is set as 0.3 to accelerate the convergence and decreases by 0.05 at each time step until it reaches 0.003. The discount factor is 0.9. The variance \(\sigma\) for action exploration is 0.05. For the gain scheduler, we expect that at the beginning of the training process, the actor network is updated dominantly by the supervisor. The weight of the reinforcement update then increases gradually, such that a near-optimal policy can finally be obtained. Thus, the initial interpolation parameter \(k_{s}\) is set as 0.2 and increases with 0.004 at each time step until it reaches 0.8. A typical driving profile is constructed as the training cycle [34; 36].  
Figure 4: Acceleration response of the host vehicle to the commanded acceleration and simulation output of the model: (**a**) acceleration and (**b**) braking.  
In our case, the preceding vehicle starts driving at a constant velocity of 50 km/h for 50 s. Then, two step acceleration signals, 0.42 m/s\({}^{2}\) and 0.83 m/s\({}^{2}\), and two step deceleration signals, \(-\)0.42 m/s\({}^{2}\) and \(-\)0.83 m/s\({}^{2}\), are successively given; each signal lasts 20 s. At 140 s, two periods of sine wave acceleration signals are given with a 40-s period and a 1 m/s\({}^{2}\) amplitude; then, the preceding vehicle drives at a constant velocity until the terminal time of 200 s. The initial velocity of the host vehicle is 60 km/h, and the initial inter-vehicle distance is 20 m. The host vehicle learns to follow the preceding vehicle while keeping a specific desired following range of 1 s time headway. The training step size is set as 1 s [34].  
In this study, an experiment consists of a maximum of 1000 consecutive trials of the training cycle. The experiment is considered successful if the host vehicle can follow the preceding vehicle with the desired range in a steady state at the end of a trial (trial number less than 1000). For comparison, the actor-critic RL algorithm without the supervisor is also adopted for the same training process as SRL, and 100 experiments are performed for each. The training results summarized in Table 1 show that with the proposed SRL algorithm, the training process is always successful, and fewer trials are needed than with RL.