## 3 System Model
### DRL Module  
The key parameters that constitute the DRL module are defined as follows: the ego vehicle serves as the agent that performs actions, while the environment represents the external world with which the agent interacts. The environment is modeled as a state \(s\in\mathcal{S}\), which can be represented at each time step \(t\) as \(s_{t}\). The state space \(\mathcal{S}\) is identical to the vehicle state information \(\mathcal{S}\) provided by the CARLA simulator. The action is defined as \(a\in\mathcal{A}\), where the action space \(\mathcal{A}\) is composed of all elements of the control information \(C\) except for \(c_{sp}\): \(A=\{c_{st},c_{th},c_{br}\}\). The reward is defined as \(r\in\mathcal{R}\). Policies are categorized into two types depending on whether their probability distributions are deterministic or stochastic. In the case of a deterministic policy, the selection of action \(a\) given state \(s\) is defined as \(\mu(s)=a\). In contrast, for a stochastic policy, the probability distribution of actions \(a\) given state \(s\) is denoted as \(\pi(a|s)\). The transition probability from a state \(s\) to a subsequent state \(s^{\prime}\) under action \(a\) is represented as \(T(s^{\prime}|s,a)\). The discount factor for future returns is denoted by \(\gamma\in[0,1]\). At each time step \(t\), the return is represented by \(G_{t}\), the corresponding state value function under policy \(\pi\) is denoted by \(\gamma_{t}\). Figure 1: Brief system operation of CARLA simulator and DRL module. Figure 3: A conceptual illustration of the approximate timing and granularity of the simulation operation. Figure 2: VAE architecture: the image features are extracted through four \(4\times 4\) convolutional layers to generate the latent space \(Z\), which is subsequently decoded into a segmentation image using four transposed convolutional layers. as \(V^{\pi}(s)\), and the action value function is given by \(Q^{\pi}(s,a)\), which are formally defined as follows:  
\[G_{t} = \sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1},\] \[V^{\pi}(s) = \mathbb{E}_{\pi}[G_{t}\mid s_{t}=s]=\sum_{a}\pi(a|s)\sum_{s^{ \prime}}T(s^{\prime}|s,a)\big{[}r(s,a,s^{\prime})+\gamma V^{\pi}(s^{\prime}) \big{]},\] \[Q^{\pi}(s,a) = \mathbb{E}_{\pi}[G_{t}\mid s_{t}=s,a_{t}=a]=\sum_{s^{\prime}}T(s^ {\prime}|s,a)\Bigg{[}r(s,a,s^{\prime})+\gamma\sum_{a^{\prime}}\pi(a^{\prime}| s^{\prime})Q^{\pi}(s^{\prime},a^{\prime})\Bigg{]}.\]  
Table 2 summarizes the key characteristics of the DRL modules utilized in the experiments, while Figure 4 provides a schematic comparison of the architectural structures of each algorithm. DDPG follows a standard actor-critic framework, employing target networks for both the actor and critic to stabilize \(Q\)-value estimation and policy learning. Let us now provide a formal mathematical formulation. The current reward \(r_{t}\) is combined with the output of the target critic \(Q^{\prime}\) evaluated in the next state \(s_{t+1}\) and the action selected by the target actor (policy) \(\mu^{\prime}\), resulting in the target \(Q\)-value \(y_{t}\) being the learning objective. The target network is introduced to mitigate the instability caused by abrupt parameter updates.