## 4 RL Control System  
### Reinforcement Learning for Control  
Reinforcement learning is a branch of machine learning with ample application to control systems. It differs both from supervised learning, since it does not require labelled data given from an external supervisor to learn from, and from unsupervised learning, \begin{table}
\begin{tabular}{c c c c c c c} \hline \multicolumn{2}{c}{**Driving Cycle**} & \multicolumn{2}{c}{**Jerk [m/s\({}^{3}\)] (Limit 0.9 m/s\({}^{3}\))**} & \multicolumn{2}{c}{\(a_{x}\) [m/s\({}^{2}\)] (Limit 2 m/s\({}^{2}\))**} \\ \hline \multicolumn{2}{c}{**Vehicle**} & \multicolumn{2}{c}{\(t_{h}=1.5\) s} & \(t_{h}=3\) s & \(t_{h}=4.5\) s & \(t_{h}=1.5\) s & \(t_{h}=3\) s & \(t_{h}=4.5\) s \\ \hline \multirow{3}{*}{\
\begin{tabular}{c} Follow 1 \\ \end{tabular} } & \(R_{0}=10^{-7}\) & 0.156 & 0.113 & 0.088 & 0.49 & 0.423 & 0.373 \\ \cline{2-7}  & \(R_{0}=10^{-5}\) & 0.154 & 0.111 & 0.086 & 0.495 & 0.428 & 0.374 \\ \cline{2-7}  & \(R_{0}=10^{-3}\) & 0.148 & 0.106 & 0.082 & 0.515 & 0.431 & 0.374 \\ \hline \multirow{3}{*}{
\begin{tabular}{c} Follow 2 \\ \end{tabular} } & \(R_{0}=10^{-7}\) & 0.122 & 0.073 & 0.019 & 0.452 & 0.36 & 0.291 \\ \cline{2-7}  & \(R_{0}=10^{-5}\) & 0.117 & 0.070 & 0.047 & 0.453 & 0.359 & 0.293 \\ \cline{2-7}  & \(R_{0}=10^{-3}\) & 0.108 & 0.063 & 0.042 & 0.456 & 0.354 & 0.287 \\ \hline \end{tabular}
\end{table} 
Table 2: Summary of comfort and safety analysis: RMS of jerk and longitudinal acceleration for the driving cycle under investigation. For a quick evaluation of the results, a colormap from green (lowest values) to red (highest value) is adopted to colour the table cells. Figure 8: Emergency braking manoeuvre results: follower 1 and follower 2 velocity (**top**) and inter-vehicle distance (**bottom**) \(t_{h}=[1.5\,\text{s},3\,\text{s},4.5\,\text{s}]\) with \(R_{0}=10^{-5}\). since it attempts to maximise a reward signal rather than trying to find hidden structure in unlabelled data. In essence, reinforcement learning attempts to learn how to map observations to actions that maximise the cumulative reward in the long run [27]. Compared to a more classical approach to control systems, reinforcement learning-based control attempts to compound all functions of a control system (such as state estimation and multiple high- and low-level control loops) into a single agent. The agent interacts with an environment by taking actions \(a\) and receiving observations of the states \(s\). When deployed, the agent uses a control policy to decide which actions should be taken as a function of the current observations, and the reinforcement learning algorithm is used in the training phase to train this policy (i.e., tune its parameters) to attain the desired behaviour. More specifically, while training, the agent attempts to maximise the cumulative reward in the long run; hence, the definition of the instantaneous reward is crucial in developing a reinforcement learning agent. The learning procedure does not require an external supervisor to label what the correct action will be. The agent itself must, in fact, identify the latter through a trial-and-error procedure and by interacting with the environment. Formally speaking, the policy is often described as a mapping \(\pi:\ S\to A\), where the state space \(S\) and the action space \(A\) define the (discrete or continuous) sets of possible states and actions. At each time step \(t\), the agent receives the observation \(s_{t}\) and takes an action \(a_{t}\). Consequently, it receives from the environment the instantaneous reward \(r_{t}\) and the new state \(s_{t+1}\). The cumulative reward \(R=\sum_{t}\gamma^{t}r_{t}\), which the RL algorithm attempts to maximize, is a discounted sum of the instantaneous rewards \((\gamma\in[0,\ 1]\) being the discount factor). The discounted sum is adopted to transform what would otherwise be an infinite-horizon optimisation problem into a finite-horizon one (there are also other reasons for discounting, such as not using predicted rewards that are too far in the future to be reliable).