## 4 RL Control System
### The DDPG Algorithm  
DDPG is a model-free, off-policy algorithm, meaning that the training phase takes place entirely before deployment and testing. A DDPG agent is also an actor-critic agent, as opposed to a value-based or a policy-based agent. In essence, this means that two different types of function approximators exist. * Actor, which receives observations and returns actions (thus playing the role of the policy). * The critic, which receives the observations and the actions taken by the actor, evaluates them and returns a prediction of the discounted long-term reward. During training, the actor's parameters are updated using the information given by the critic, and the critic itself is updated using the actual rewards received from the environment. DDPG uses four function approximators: an actor, a critic, a target actor, and a target critic, all four of which are modelled using feed-forward neural networks. The presence of a target actor and a target critic is a feature that was introduced to improve the stability of the learning process [28]. The target actor and critic networks(which are used when the agent is deployed) are periodically updated using the parameters of the actor and critic using a smoothing technique. More in detail, we can define a target actor \(\mu_{t}\), a target critic \(Q_{t}\), an actor \(\mu\) and a critic \(Q\). The weights of the target networks \(\theta^{\mu_{t}}\) and \(\theta^{Q_{t}}\) are at first initialized to the weights of the actor and critic networks \(\theta^{\mu}\) and \(\theta^{Q_{t}}\). Then, they are updated at every time step as follows:  
\[\theta^{\mu_{t}}\ = SF\cdot\theta^{\mu}+(1-SF)\cdot\theta^{\mu_{t}} \tag{22}\]  
\[\theta^{Q_{t}}\ = SF\cdot\theta^{Q}+(1-SF)\cdot\theta^{Q_{t}} \tag{23}\]  
where \(SF\) is a smoothing factor. In our work, this was set to \(SF=10^{-3}\). Furthermore, for all networks, we used three hidden layers with 56 neurons each and used the rectified linear unit (ReLu) as the activation function. The hidden layers and neurons are selected from previous work [29] and adapted to the platoon control problem by increasing the number of layers. Another notable feature of DDPG is that, since it is an actor-critic network using neural networks as function approximators, it can deal with continuous action and observation spaces that can be either continuous or discrete. This contrasts with other RL methods such as Q-learning, which can only deal with problems in which the action-value function can be represented as a table and where state and action spaces need to be small enough. Another significant feature of the DDPG algorithm is the experience replay memory, where the data needed for the update process of the parameters of the neural networks during training is stored. At each time step, the memory of the chosen capacity \(N\) receives and stores an array containing the current state \(s\), the current action \(a\), the obtained reward \(r\), and the state of the environment after the action. For every training iteration, a batch of \(n\) random arrays is sampled from the replay memory and is used to train the critic and actor networks through the respective loss functions \(L_{c}\) and \(L_{a}\):  
\[L_{c}=\frac{1}{n}\sum_{i=1}^{n}\left(y-Q\left(s,a\left|\theta^{Q}\right.\right) \right)^{2} \tag{24}\]  
\[L_{a}=\frac{1}{n}\sum_{i=1}^{n}Q(s,\mu(s)) \tag{25}\]  
In machine learning applications, the amplitude of the step moving towards the minimum of the loss function is called the learning rate. It is one of the main hyperparameter, and in this work it is set to \(5\cdot 10^{-5}\) for the actor network and to \(10^{-4}\) for the critic one. The capacity of the experience replay memory is set to \(N=10^{6}\). Furthermore, the sample time is set to 0.1 s, since it is small enough to correctly track the lead and depict the energetic aspects. It is typical to use, for this application, a sample time between 0.05 s and 0.2 s [30].