## 5 Simulation Results  
Four simulation examples are shown to illustrate the effectiveness of the designed controller. The first and second examples show simplified scenarios, in which three vehicles are at the intersection. The goal of the simulations in scenario 3 is to shown that the proposed control system is able to adapt to various traffic environments, depending on the initial velocities and positions of the vehicles. The fourth example has increased complexity, in which seven vehicles are incorporated. In these simulations the closest three vehicles in the constraints of the optimization tasks are incorporated. The RL-based agent was trained through 500 simulation scenarios, in which the number of vehicles, their initial velocities and positions were selected randomly; i.e., the vehicle number was varied between 1 and 7, the initial positions were selected between \(-20\) m \(\ldots\)\(-40\) m and the initial velocities were varied between \(10\) km/h \(\ldots 50\) km/h. In the learning process of the neural networks, the following structures have been trained. The actor network had eight neurons in the input layer; three fully connected layers with 48 neurons and Rectified Linear Unit (ReLU) functions in each layer; and one neuron with hyperbolic tangent functions in the output layer. The critical network has the same structure. In the structure and parameter selection of the neural networks, the k-fold cross-validation technique [51] and a hidden-layer number optimization process [52] can be used. The sampling time in each episode is selected at \(T=0.05\) s. The terms in the reward function are considered with the same design parameters, such as \(Q_{1}=-0.1\) and  
Figure 4: Control structure in the learning process. \(Q_{2}=10\). The training process has been performed through Matlab 2020a Reinforcement Learning Toolbox [53] on an Intel i7 CPU. For the simplified examples, the first and second scenarios are illustrated in Figure 5. In these cases, vehicle 1 is the ego vehicle; vehicle 2 and vehicle 3 are the surrounding vehicles on the perpendicular road section. Both surrounding vehicles cross the route of vehicle 1, which results in a conflict situation. In the first scenario the following initial conditions were set: \(s_{1}=-40\) m, \(s_{2}=-36\) m, \(s_{3}=-41\) m and all of the vehicles have 50 km/h initial velocity. In the second scenario, vehicle 3 had different settings, while the other vehicles had the same initial conditions, i.e., \(s_{2}=-48\) m and \(v_{2}=30\) km/h. The safe distance was set to \(s_{safe}=8\) m.  
The results of the simplified scenarios are found in Figure 6. Figure 6a,b shows that the autonomous ego vehicle is able to guarantee keeping \(s_{safe}\). Nevertheless, in the two scenarios, the ordering of the vehicles in the intersection is different. Since in scenario 1 the surrounding vehicles have increased velocity, the autonomous vehicle decides to reduce its velocity to 18 km/h and allows the surrounding vehicles with right of way to enter the intersection. In scenario 2 the autonomous vehicle has enough time to go through the intersection before vehicle 3 enters it. Thus, in this case \(v_{1}\) can be increased after vehicle 2 leaves the intersection, see Figure 6d. The differences in the motion of vehicle 1 also influence the trajectories in Figure 6a,b. Moreover, the control input signals for each scenario are found in Figure 6e,f. In case of scenario 2 the acceleration of the ego vehicle is increased to 3 m/s\({}^{2}\) in 2.6 s to avoid a collision with vehicle 2, while in scenario 1 the acceleration is increased later than 3 s by a lower value. Figure 5: Illustration of the simplified intersection scenarios. In the next simulation examples, it is proposed that the proposed algorithm provides safe motion for the autonomous vehicle in various scenarios. The intersection scenario is illustrated in Figure 7, where vehicle 1 represents the autonomous vehicle and the other vehicles are the surrounding vehicles.