## 3 System Model
### Reward Function
#### 3.4.1 Reward  
The underlying philosophy of the reward's design is centered on promoting safe and compliant driving behavior. To this end, the reward function contains multiple contributing factors that encourage the agent to (1) maintain an appropriate speed, (2) stay within lane boundaries, (3) align its heading with the road direction, and (4) obey various traffic regulations. These objectives are jointly enforced through a composite reward function. Throughout each episode, driving decisions are made at every decision step, and a reward value ranging from 0 to 1 is assigned at each step. The cumulative reward accumulated over the course of an episode is employed as a quantitative measure of overall driving performance. The individual reward components and their corresponding formulations, computed at each step, are presented as described below. The Speed Reward \(R_{speed}\) is computed based on the current traffic signal state. Specifically, it is categorized as follows: \(R_{speed}^{G}\) for a green light, indicating normal driving con \begin{table}
\begin{tabular}{l l l} \hline \hline
**Reward and Penalty** & **Value** & **Terminology** \\ \hline Reward & (22) & Speed Reward (Green Light or No Traffic Light) \\  & (23) & Speed Reward (Red Light) \\  & (26) & Speed Reward (Yellow Light) \\  & (27) & Centerline Distance Reward \\  & (28) & Centerline Standard Deviation Reward \\  & (29) & Vehicle Heading Reward \\ \hline Penalty & \(-10\) & Vehicle stopped \\  & & Off-Track \\  & & Too Fast \\  & & Red Light Violation \\  & & Collision \\ \hline Total & (31) & Total Reward Function \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Reward and penalty parameters comprising the total reward function. ditions; \(R_{speed}^{R}\) for a red light, requiring the vehicle to stop; and \(R_{speed}^{Y}\) for a yellow light, suggesting deceleration. When a green light is active or the vehicle is navigating a section with no available traffic light information, the speed reward assigned is \(R_{speed}^{G}\), which can be calculated according to Equation (22):  
\[R_{speed}^{G}= \begin{cases}\frac{V_{ego}}{V_{min}},&\text{if}\;\;V_{ego}<V_{min},\\ 1.0-\frac{V_{ego}-V_{target}}{V_{max}-V_{target}},&\text{if}\;\;V_{target}<V_{ego }<V_{max},\\ 1.0,&\text{if}\;\;V_{min}<V_{ego}\leq V_{target}.\end{cases}  \tag{22}\]  
where \(V_{ego}\) denotes the current speed of the ego vehicle. The reward value \(R_{speed}^{G}\) is designed to yield higher scores when \(V_{ego}\) lies within the desirable range, between the minimum speed threshold \(V_{min}\) and the target speed \(V_{target}\). If \(V_{ego}\) falls below \(V_{min}\) or exceeds \(V_{target}\), the reward is gradually reduced. Furthermore, if the vehicle exceeds the maximum allowable speed \(V_{max}\), a negative reward is applied to penalize unsafe driving. In the experimental setup of this study, the parameters used were as follows: \(V_{target}\) = 25 km/h, \(V_{min}\) = 20 km/h and \(V_{max}\) = 35 km/h. In the case of a red traffic light, the Speed Reward value \(R_{speed}^{R}\) is formulated as in Equation (23), such that the vehicle is encouraged to approach the stop line gradually from its current position. \[R_{speed}^{R}=\left(w_{d}\times\hat{d}_{stop}\right)+\left(w_{v}\times\hat{o}_ {ego}\right), \tag{23}\]  
where \(w_{d}\) and \(w_{v}\) denote the weighting coefficients for the distance to the stop line and the ego vehicle's speed, respectively. In the experiment, they were set to 0.4 and 0.6. The terms \(\hat{d}_{stop}\) and \(\hat{o}_{ego}\) represent the normalized values of the distance from the vehicle's current position to the stop line \(D_{stop}\) and the current speed of the ego vehicle \(V_{ego}\), respectively. The corresponding normalization formulas are defined in Equations (24) and (25).