## 4 Experiments and Results
### Implementation in Carla Simulator  
For an iterative strategy algorithm, the research needs to easily measure the performance of the algorithm. Training cars in real environments can be damaging and costly. It is necessary to ensure the algorithm can operate safely and correctly in common scenarios before deploying it in real-world environments in the future. Additionally, in practical applications, it is not feasible to frequently update algorithms in real environments, so this process needs to be first deployed and tested in simulators. The debugging and performance evaluation of autonomous driving algorithms should be conducted in highly realistic simulation environments before real environments. It is important to have an autonomous driving simulator that can reliably reflect real scenarios. In this study, Carla is chosen as the simulation environment for autonomous driving development and testing [36]. Carla has a 3D city scene that supports perception, planning, and control. Figure 3 shows the top view of the city map in the Carla simulator used in this experiment. It runs on Unreal Engine 4 with a server-client architecture. In addition, it is open source and can be installed on Linux and Windows systems. It allows the creation of environments, agents, and required sensors via Python API, which meets the requirements of this study. In terms of experimental configuration, this study uses a Tesla V100 as the GPU and runs in a Linux environment. The system code is mainly based on the Python language. Carla can simulate common scenarios in the environment such as lane conditions, road conditions, obstacle distribution, and weather. Three-dimensional LiDAR can also be simulated in Carla, as shown in Figure 4. To maintain consistency with sensor configurations employed in subsequent real-world applications, in this study, the 3D-LiDAR parameters were set in the simulator as shown in Table 1.  
The 3D-LiDAR is configured with 16 channels, which is consistent with the model in the real environment of this research. The set rotation frequency is 10 revolutions per second, which can be adjusted to a higher frequency in the future when capturing more rapidly changing environments, such as high-speed scenes. The number of points per second can be calculated by multiplying the number of channels, points per revolution, and rotation frequency. It determines the resolution of the point cloud image and the density of the point cloud. To reduce the computational burden, the detection distance is reduced to 50 m, even though the default maximum detection distance is 100 m. The upper fov and lower fov together define the vertical field of view range. The difference between these must match the number of channels to ensure accurate simulation. The horizontal fov and vertical fov range define the scanning range. The point cloud data are generated every  
Figure 3: Top view of the experiment map. 0.1 s. The simulated environment works in parallel with the RL environment, and [PERSON] independently updates and transmits data into the RL network at this fixed interval. As a result, the agent obtains the latest environment data in real time and outputs actions, while the simulator executes the latest actions output by the agent. The experiment establishes a fundamental task, where the vehicle drives on urban roads while avoiding collisions. The objective is to maximize cumulative rewards by maintaining street driving for as long as possible. The experimental implementation follows these steps:  
1. Network Initialization: The first step is to define the structure of the Actor and Critic networks, including the input, hidden, and output layers. The learning rate of the actor is 0.001. The learning rate of the critic is 0.0001. The discount factor \(\gamma\) is 0.99. The range of weights is [\(-\)0.003, 0.003]. The weights and biases of the network are initialized in this process. The weight matrix is initialized with random values to control the effect of the input data on each neuron, with a core role as a linear transformation.