## 4 Experiments and Results
### Implementation in Carla Simulator
Bias vectors are initialized to zero, adjusting neuron outputs to enhance model expressiveness. During subsequent training, the optimization algorithm minimizes the loss function by calculating the gradient and constantly adjusting both. Target networks for training stability are initialized with weights identical to the main network. The replay buffer is initialized empty with a 10,000-capacity storage. Each experience corresponds to a state transition tuple, containing the current state, action, reward, next state, and terminal state Boolean. 2. Agent Generation: To prevent policy stagnation and enhance generalization, vehicles are spawned at random map locations for each episode, simulating real-world ran \begin{table}
\begin{tabular}{c c} \hline Channels & 16 \\ \hline rotation\_frequency (r/s) & 10.0 \\ \hline Range (m) & 50.0 \\ \hline upper\_fov (\({}^{\circ}\)) & 15.0 \\ \hline lower\_fov (\({}^{\circ}\)) & \(-\)15.0 \\ \hline horizontal\_fov (\({}^{\circ}\)) & 360.0 \\ \hline sensor\_tick (s) & 0.1 \\ \hline \end{tabular}
\end{table} 
Table 1: The parameter settings of 3D-LiDAR. Figure 4: Three-dimensional LiDAR view in Carla. domness and improving exploration efficiency. To prevent the model's strategy from falling into a fixed pattern and to enhance the generalization ability of the agent, the vehicle is spawned at random map locations for each episode. It also simulates the randomness of the real world and enhances the agent's exploration efficiency. * Environmental Information by sensor: The 3D-LiDAR inputs point cloud data containing three-dimensional coordinates. * Point Cloud Processing: Global features are extracted using the study's point cloud processing algorithm. * Actor-network Input: The 1024-dimensional features pass through two fully connected layers with 64 neurons each, activated by ReLU, outputting action vectors \(\text{a}_{1}\) and \(\text{a}_{2}\). \(\text{a}_{1}\) represents acceleration, ranging between \(-1\) and \(+1\), while \(\text{a}_{2}\) represents steering angle, ranging between \(-15\) degrees and \(+15\) degrees. Due to the nature of the DDPG, the output of actions allows for smooth steering angles and throttle control instead of the simple left or right direction in the DQN method. The state selects the action output after adding Gaussian noise according to the current strategy. Following the task objective, the reward function is designed as follows: the agent obtains a reward of 0.01 if it chooses to go forward; when it collides with an obstacle, the reward is \(-1\) and the episode is terminated. Since pulling out of the lane also results in a collision with the obstacle, there is no need to set it separately. After executing an action, it receives signals from the environment about the next state, the reward, and whether to terminate or not. Next, the transfer data are stored in the replay buffer. * Sample Data from the Replay Buffer: When the amount of data in the buffer exceeds the minimum batch size, a small random batch is sampled from the buffer. * Critic-network Implementation: This inputs state and action to output the scalar expected returns under the current policy. States are represented by point cloud global feature vectors. Actions come from the actor-network output. Both pass through fully connected layers for feature extraction, and then the state features and action features are fused into a vector that outputs the \(Q\)-value through the last layer of the fully connected network. The critic network uses the mean square error loss to update the weights. The output \(Q\)-value is used to guide the actor network to improve the driving strategy of the vehicle. * Actor-network Implementation: Following the critic network's performance evaluation, this step calculates actor network gradients and updates the parameters. * Target Network Updates: Soft updates maintain stability after the main network updates with \(\tau=0.001\). The training process repeats the above steps, and the results after 100 episodes are as follows.