## 3 The SRL-Based Control Strategy for CACC
### The SRL Control Algorithm
#### 3.2.5 SRL Learning  
During the learning process, the weights of the actor and critic are updated with error back-propagation at each time step. After taking the composite action \(u(t)\), the system transits to the next state \(x(t+1)\) and gives a reward \(r(t)\). For the critic network, the prediction error has the same expression with the temporal difference (TD) error as follows:  
\[e_{c}(t)=\alpha J(t)-(J(t-1)-r(t)) \tag{14}\]  
The squared error objective function is calculated as  
\[E_{c}(t)=\frac{1}{2}e_{c}^{2}(t) \tag{15}\]  
The objective function is minimized by the gradient descent approach as  
\[\omega_{c}(t+1)=\omega_{c}(t)+\Delta\omega_{c}(t) \tag{16}\]  
\[\Delta\omega_{c}(t)=l_{c}(t)\biggl{(}-\frac{\partial E_{c}(t)}{\partial\omega _{c}(t)}\biggr{)}=-l_{c}(t)\frac{\partial E_{c}(t)}{\partial J(t)}\frac{ \partial J(t)}{\partial\omega_{c}(t)} \tag{17}\]  
in which \(l_{c}(t)\) is the learning rate of the critic network at time \(t\), \(0<l_{c}(t)<1\). The adaptation of the actor network is regulated by the gain scheduler. The weights of the actor are updated according to the following rule as  
\[\omega_{a}(t+1)=\omega_{a}(t)+k_{s}\Delta\omega_{a}^{\text{RL}}(t)+(1-k_{s}) \Delta\omega_{a}^{\text{SL}}(t) \tag{18}\]  
in which \(\Delta\omega_{a}^{\text{RL}}(t)\) and \(\Delta\omega_{a}^{\text{SL}}(t)\) are the updates based on RL and the supervised learning, respectively. The error and objective function of RL are defined as  
\[e_{a}^{\text{RL}}(t)=J(t)-U_{c}(t) \tag{19}\]  
\[E_{a}^{\text{RL}}(t)=\frac{1}{2}\Bigl{(}e_{a}^{\text{RL}}(t)\Bigr{)}^{2} \tag{20}\]  
in which \(U_{c}(t)\) is the desired control objective. Here, \(U_{c}(t)\) is set to 0, because the reward tends to zero if optimal actions are being taken. The supervisory error and its objective function are calculated as  
\[e_{a}^{\text{SL}}(t)=u_{s}(t)-u_{a}(t) \tag{21}\]  
\[E_{a}^{\text{SL}}(t)=\frac{1}{2}\Big{(}e_{a}^{\text{SL}}(t)\Big{)}^{2} \tag{22}\]  
The RL and supervisory objective functions are minimized by the gradient descent approach, and thus the reinforcement-based update and the supervisory update are calculated as  
\[\Delta\omega_{a}^{\text{RL}}(t)=l_{a}(t)\bigg{(}-\frac{\partial E_{a}^{\text{RL }}(t)}{\partial\omega_{a}(t)}\bigg{)}=-l_{a}(t)\frac{\partial E_{a}^{\text{RL }}(t)}{\partial J(t)}\frac{\partial J(t)}{\partial u(t)}\frac{\partial u(t)}{ \partial\omega_{a}(t)} \tag{23}\]  
\[E_{c}(t)=\frac{1}{2}e_{c}^{2}(t) \tag{15}\]0  
in which \(l_{a}(t)\) is the learning rate of the actor network at time \(t\), \(0<l_{a}(t)<1\). The complete SRL algorithm is described in detail with the pseudocode as Algorithm 1. ```
1:Collect driving samples \(\mathbf{D}=\{e_{d}(t),v_{r}(t),a_{r}(t),a_{ds}(t)\}\) from a human driver in an infinite time horizon
2:Train the supervisor network with \(\mathbf{D}\) using the Levenberg-Marquardt method
3:Input the actor and critic learning rate \(l_{a},l_{c}\); the discount factor \(\alpha\); the exploration size \(\sigma\); and the interpolation parameter \(k_{s}\)
4:Initialize the weights of the actor and critic \(\omega_{a},\omega_{c}\) randomly
5:Repeat for each trial
6:\(x(t)\leftarrow\) initial state of the trial
7:repeat for each step of the trial
8:\(u_{s}(t)\leftarrow\) action given by the supervisor
9:\(u_{a}(t)\leftarrow\) action given by the actor
10:\(u_{E}(t)\gets u_{a}(t)+N(0,\sigma)\)
11:\(k_{s}\leftarrow\) interpolation parameter from the gain scheduler
12:\(u(t)\gets k_{s}u_{E}(t)+(1-k_{s})u_{s}(t)\)
13:execute action \(u(t)\) and observe the reward \(r(t)\) and new state \(x(t+1)\)
14:\(e_{c}(t)\leftarrow\alpha J(t)-(J(t-1)-r(t))\)
15: update the weights of the critic by (16) and (17)
16:\(e_{a}^{\text{RL}}(t)\gets I(t)-U_{c}(t)\)
17:\(e_{a}^{\text{SL}}(t)\gets u_{s}(t)-u_{a}(t)\)
18: update the weights of the actor by (18), (23) and (24)
19:\(x(t)\gets x(t+1)\)
20:until\(x(t)\) is terminal
```  
**Algorithm 1** Supervised Reinforcement Learning algorithm.