## 3 Path-Following Control Strategy with Deep Deterministic Policy Gradient
### Deep Deterministic Policy Gradient for Path-Following
#### 3.2.4 Implementation Details  
The actor and critic neural networks both consist of two hidden layers, as illustrated in Figure 6. Each layer includes rectified linear units (ReLU) activation, with 400 neurons in the first hidden layer and 300 neurons in the second. Notably, in the critic's network, the state vector connects to the first hidden layer, whereas the action is concatenated before the second hidden layer, following the structure of the original algorithm. This design allows the action to bypass the first layer, which improves the stability and performance of the networks [15]. The final layer of the actor is a tanh layer used to bound the action. Figure 5: Randomly generated paths, including both straight paths and complex curved paths. Figure 6: The two networks share the same structure and input, except that in the critic network, the action is concatenated before the second hidden layer. We initialize the weights following the method described in [34], with the exception that we used uniform distributions \([-3\times 10^{-3},3\times 10^{-3}]\) and \([-3\times 10^{-4},3\times 10^{-4}]\) to initialize the final layers of the actor and critic networks, respectively. This is done to prevent output saturation during the early stages of training. To optimize neural networks, we employ Adam [35] with a minibatch size of 64. Algorithm 2 outlines the training process based on DDPG, following the path-following strategy described above. The initial posture of the vehicle is sampled from a uniform distribution, with a position range of \([-1,1]\) meters and a heading angle range of \([-0.2618,0.2618]\) radians. We also introduce a warm-up technique to collect completely random experiences. During the initial training period, the agent uniformly samples random actions from the action space. After warming up, the actor network first generates an output, i.e., action, based on the observed state and adds sampled noise, transitions to the next state, computes the corresponding reward and stores it in the experience replay buffer in the form of a tuple until the number of experiences reaches the size of the set minibatch, at which point the networks are optimized and updated. The loss required for updating the parameters of the critic and actor networks can be calculated using Equations (17) and (19). After optimizing the networks, the parameters of the target networks are updated using a soft update strategy, as denoted by Equation (23). Specifically, a fraction of the updated network parameters are blended with the target network parameters, which helps to stabilize the learning process and avoid oscillations. \[\theta^{target}=\tau\theta^{updated}+(1-\tau)\theta^{target} \tag{23}\]  
where parameter \(\tau\) indicates how fast the update is carried on and the update is performed at each step after training the online networks. During training, a random path is generated for each episode and the agent's action is subject to exploration noise. The agent selected is determined based on its performance in the evaluation. The difference between training and evaluation is that during evaluation, the actions taken by the agent are solely based on the current learned policy without the addition of exploration noise. For each evaluation, the agent's performance is assessed across 10 randomly generated paths. Since evaluation occurs at regular intervals, we select the agent that achieves the most high rewards among the evaluations. Table 7 presents the relevant parameters and their values used in the training. We also conduct the training 10 times using 10 different seeds to ensure reputability. For instance, the uniform sampling of the initial position of the vehicle and the random generation of paths rely on a random number generator that is controlled by a seed.