## 2 Problem Statement and Modeling
### Description of MDP
#### 2.2.3 Reward  
When tackling the control challenges faced by CAVs at intersections, it is imperative to consider a range of factors, notably energy consumption and traffic efficiency among others. To comprehensively optimize the driving process of CAVs and attain optimal performance, this study designs a multi-objective reward function that considers diverse aspects to efficiently train agents. Specifically, the reward function aims to minimize energy consumption, mitigate traffic delays, and enhance driving comfort, all while prioritizing safety.  
For the component of travel efficiency, the reward function utilizes the vehicle's travel distance at each step as it crosses intersections, as detailed in Equation (5).  
\[r_{1}=|x_{t}-x_{t-1}| \tag{5}\]  
To reduce the electric energy consumption of the CAV at intersections, the reward function (\(r_{2}\)) for the energy consumption component is presented in (6).  
\[r_{2}=(1-\eta)[E_{vch}(v_{t},a_{t},x_{t})-E_{loss}(v_{t},a_{t},x_{t})], \tag{6}\]  
where \(\eta\) denotes the energy recovery factor, \(E_{vch}\) is a function capable of yielding the instantaneous electric energy of the CAV, and \(E_{loss}\) represents the energy loss caused by the driving resistance. \(E_{vch}\) and \(E_{loss}\) incorporate principles from vehicle dynamics and energy conversion, respectively, considering the energy brake-recovery mechanism. Specific details can be referenced in [35].  
Safety is always the top priority in vehicle operation. Here, the reward function (\(r_{3}\)) is formulated around time to collision (TTC), imposing greater penalties for actions that breach the safety margin, as demonstrated in (7).  
\[r_{3}=\left\{\begin{array}{l}e^{\kappa\cdot\text{TTC}^{*}\cdot(\frac{ \Delta r_{2}}{\Delta x_{1}})},\quad\text{TTC}_{t}<TTC^{*}\\ 0,\qquad\text{otherwise}\end{array}\right., \tag{7}\]where \(\alpha\) is penalty coefficient for dangerous driving, \(TTC_{t}\) denotes the time to the collision between the preceding and following vehicles at time \(t\), and \(TTC^{*}\) is the pre-defined safety threshold of \(TTC_{t}\), selected as 2 s according to previous literature [36].  
Enhancing driving comfort entails maintaining the smooth operation of the vehicle, reducing abrupt accelerations and decelerations to achieve a relatively smooth driving trajectory. Jerk, the derivative of acceleration, is used to characterize the stability of vehicle driving. The reward function (\(r_{4}\)) for driving comfort is calculated by (8) and (9).  
\[r_{4}=\left\{\begin{array}{l}\beta(|I_{t}|-J^{*}),|I_{t}|>J^{*}\\ \\ 0,\qquad otherwise\end{array}\right. \tag{8}\]  
\[J_{t}=\frac{da_{t}}{dt}, \tag{9}\]  
where \(\beta\) is the penalty coefficient for aggressive driving, \(J_{t}\) denotes the jerk of the vehicle at the time \(t\), and \(J^{*}\) represents the maximum permissible rate of change in acceleration that enables the vehicle to drive smoothly. According to previous research experience [37], the value of \(J^{*}\) can be taken as 4.  
Ultimately, the overall reward function (\(R_{t}\)) is derived by integrating the reward functions across all four indices, as shown in (10).  
\[R_{t}=r_{1}-\sum_{i=2}^{4}r_{i} \tag{10}\]