## 3 Problem Formulation and Modeling
### Modeling
#### 3.2.3 Reward  
Reward and punishment functions play a critical role in reinforcement-learning algorithms by assessing whether an agent's behavior aligns with expectations and providing feedback. In the real-time signal-priority control of self-driving-bus lines, the reward and punishment function primarily consists of three components: delay penalty, queue-length penalty, and equalization reward. 1. Delay penalty item. In a transportation network, a delay penalty term pertains to the penalty that a vehicle incurs when it takes longer than its expected arrival time to reach its destination, and the penalty is proportional to the duration of the vehicle's delay. The purpose of this reward function is to incentivize the vehicle to reach the target location expeditiously, reduce the vehicle's travel time, and enhance traffic efficiency. When the vehicle waits in front of the signal light, a specific delay penalty is incurred: \[R_{delay}=-\sum_{i=1}^{N}\sum_{j=1}^{M}w_{i,j}(t)\cdot d_{i,j}(t)\] (6) where \(n\) is the number of intersections; \(M\) is the number of lanes at each intersection; \(w_{i,j}\) is the weight between intersection \(i\) and state \(q_{j}\); \(d_{i,j}(t)\) is the average delay time at intersection \(i\) and state \(q_{j}\) at time \(t\). 2. Queue-length penalty item. In the traffic network, the queue-length penalty refers to a penalty imposed on a vehicle when it waits for the red signal at a traffic light for an extended duration, and the penalty is proportionate to the waiting time. This reward function aims to decrease vehicle waiting times at traffic lights, minimize queuing \begin{table}
\begin{tabular}{l} \hline
**Pseudocode of Action Space** \\ \hline action\_space = np.zeros (\((M,2)\)). \\ for \(k\) in range (\(M\)): \\ action\_space [\(k\)] = [\(\Delta a_{i}^{\min}(t)\), \(\Delta a_{i}^{\max}(t)\)] \\ \hline \end{tabular}
\end{table} 
Table 1: Pseudocode of action space. periods, and subsequently alleviate traffic congestion. When a vehicle waits in front of a signal light, it leads to the formation of a queue of a certain length:  
\[R_{queue}=-\sum_{i=1}^{N}\sum_{j=1}^{M}w_{i,j}(t)\cdot l_{i,j}(t) \tag{7}\]  
where \(l_{i,j}(t)\) is the average queue length in state \(q_{j}\) at intersection \(i\) at time \(t\). 3. Balance reward items. In a traffic network, the equalization-reward term involves assigning additional rewards to specific roads based on their vehicle distribution, thereby promoting traffic balance and alleviating traffic bottlenecks to enhance overall traffic efficiency. To prevent signal lights from favoring certain lanes or vehicle types, an equalization-reward term is introduced: \[R_{balance}=\sum_{i=1}^{N}\sum_{j=1}^{M}w_{i,j}(t)\cdot b_{i,j}(t)\] (8) where \(b_{i,j}(t)\) is the equalization factor at intersection \(i\) and state \(q_{j}\) at time \(t\), and can be determined by considering various factors such as vehicle type and speed. The balance coefficient \(b_{i,j}(t)\) is computed based on the traffic flow \(v_{ij}(t)\) in each lane and the maximum traffic flow \(v_{max}\) observed in the lane where the vehicle is currently located. The balance coefficient for each lane should be updated in real time, independently for each lane. Combining the above three factors, the reward and punishment function \(R\) is:  
\[R=R_{delay}+R_{queue}+R_{balance} \tag{9}\]  
Table 2 provides a framework and pseudo-code for the reward-punishment function implemented in the simulation. Among these factors, the larger the value of the queue-length penalty factor (queue_len_factor), the stricter the queuing penalty, and the model tends to reduce the length of vehicle queues. However, if the factor is set too large, it may make the system overly focus on the current queue length and ignore the delay for subsequent arriving vehicles. The larger the value of the delay penalty factor (delay_factor), the stricter the delay penalty, and the model tends to reduce vehicle delay. Nevertheless, if the factor is set too large, it may lead to increased traffic congestion.