## 3 Flexible Merging Decision Methods
### Motion Decision Based on DRL  
Since the game at each moment may not have a pure-strategy Nash equilibrium or multiple equilibria, causing disagreements in the former, this raises the question of what action the vehicle should take at the current moment. The static game can be viewed as a finite-stage Markov decision process and, thus, can be combined with DRL algorithms to model the game. DRL optimizes reactive decision-making through continuous trial-and-error exploration and backtracking on experience. DRL's elements include observing the state \(O_{t}\), action space \(A\), reward \(r\), and decision networks \(\pi\), which can present as a tuple \((O,A,\pi,r)\). \(o_{t}\) is the input of \(\pi\) and \(a_{t}\) is the output of \(\pi\). The decision network \(Q_{\pi}(o_{t};\theta)\) is used to fit the functional relationship from the observed state to the output decision, and the reward \(r_{t}\) is used to correct the decision network parameter \(\theta\). The designed observation state should have completeness, which can completely reflect the traffic state at the current moment. Considering that the front vehicle in the same lane must be avoided, the observation state of the CAV needs to contain the longitudinal \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Allow** & **Not Allow** \\ \hline Merge & \([R_{CAV}(s_{\text{C1}},s_{\text{M1}}),R_{MDV}(s_{\text{M1}},s_{\text{C1}})]\) & \([R_{CAV}(s_{\text{C1}},s_{\text{M2}}),R_{MDV}(s_{\text{M2}},s_{\text{C1}})]\) \\ Not Merge & \([R_{CAV}(s_{\text{C2}},s_{\text{M1}}),R_{MDV}(s_{\text{M1}},s_{\text{C2}})]\) & \([R_{CAV}(s_{\text{C2}},s_{\text{M2}}),R_{MDV}(s_{\text{M2}},s_{\text{C2}})]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Matrix of game payoffs. position, speed, and acceleration information of the target vehicle, its own vehicle, and the respective front vehicle. Therefore, the observation state \(o_{t}\) is equal to  
\[o_{t}=\left[v,a,L_{CAV},v_{MDV},a_{MDV},L_{MDV},v_{Mf},a_{Mf},L_{Mf},v_{Cf},a_{Cf},L_{Cf}\right] \tag{11}\]  
where \(L_{*},v_{*},\ a_{*}\) are distances to the end of the acceleration lane of each vehicle, speed, and acceleration, respectively, and their subscripts CAV, MDV, _Hf_, and _Cf_ represent on-ramp CAVs,aniline MDVs, front vehicle MDVs, and front vehicle CAVs. The change in driving state is mainly reflected in the change in speed. But, the speed is continuous. Therefore, considering the accuracy and performance of the control as well as the sampling interval, the action space needs to be discretized. Also taking into account the discomfort caused by the speed change, this study also sets constraints on the amount of speed change possible for each step length. The action space is in the range of \(-6\ \text{m}/\text{s}^{2}\ \sim 6\ \text{m}/\text{s}^{2}\) with an interval of \(1\ \text{m}/\text{s}^{2}\):  
\[Action\ space=\{-6,-5,\cdots,5,6\} \tag{12}\]  
The reward function plays an important role in DRL for evaluating the effectiveness and safety of the decision and is the only signal to modify the parameters of the decision network. For the problem to be solved in this study, the reward function needs to consider the safety, comfort, timeliness, length of the remaining acceleration lanes, and number of games. The reward \(r\) takes the form of round rewards:  
\[r=r_{time}\Big{(}10-r_{safe}-r_{comf}-r_{lane}\Big{)}\beta^{n} \tag{13}\]  
\[r_{lane}=\frac{lane_{start}\ -lane_{end}}{lane_{start}} \tag{14}\]  
where \(r_{time}\) is the effectiveness factor, advance car's \(r_{time}=1\), give-way car's \(r_{time}=0.9\), \(\beta\) is the game attenuation coefficient, and \(n\) is the number of times the game is played. When the minimum distance \(d_{min}\) during convergence is greater than the safe distance, \(r_{safe}=0\); otherwise, \(r_{safe}=-1\). \(r_{comf}\) is equal to the variance of the speed during merging. \(r_{lane}\) is the remaining acceleration lane length bonus component, and \(lane_{start}\) and \(lane_{end}\) are the remaining acceleration lane lengths at the start and end of the game, respectively.