## 4 Cooperative Decision-Making in a Multi-Vehicle Cooperative Task  
### Problem Formulation  
In this paper, the unsignalized intersection environment, where the CAVs and HDVs coexist, was modeled as a model-free multi-agent system to solve the cooperative decision-making and control problems. The system was described as \(G=(V,E)\), where \(V=\{v_{1},v_{2},\ldots,v_{N}\}\) represents a non-empty finite with \(N\) CAVs and the edge set \(E\subseteq V\times V\) represents the connections among the CAVs. The CAV\({}_{i}\) makes decisions based on its observations of the local sensors, such as cameras and lidars, and its communication with its neighbors, denoted as \(N_{i}\doteq\{j|\varepsilon_{i,j}\in E,i\
eq j\}\), to enhance the range of perception. Considering that the decisions of \(N\) CAVs are interactive, the problem was considered a fully cooperative multi-agent task, which was modeled as a partially observed Markov decision process (POMDP). For the POMDP, each agent received a partial observation \(o_{i}\in O_{i}\) from the global state \(s\in S\). Based on the observation, the agents took joint action, \(a_{1},a_{2},\cdots,a_{N}\in A\) from the action set \(A\doteq\chi_{i\in V}A_{i}\) to interact with the environment and receive a reward from all the agents \(R=\{r_{1},r_{2},\ldots,r_{N}\}:S\times A_{1}\times\ldots A_{N}\rightarrow\mathbb{ R}^{N}\). The goal was that the agents would attempt to learn an optimal joint policy \(\pi_{i}:O_{i}\to A_{i}\) to maximize the expected return \(G=\mathbb{E}\Big{[}\sum_{t=0}^{T}\gamma^{t}r_{t}^{t}\Big{]}\) in the interaction with the environment, where \(\gamma\in(0,1]\) is the accumulated discount factor to quantify the importance of thefuture reward and T denotes the total steps of an episode. It can be defined by a tuple \((S,A,P,R,O,N,\gamma)\), where \(P\) denotes a state transition function. Action: In this paper, the control system of the vehicle was split into two levels, the high-level decision-making and the low-level controller, as depicted in Figure 2. According to the local observation, the high-level decision-making selected an action in the action space \(A_{i}\) defined as \(A_{i}=\{\)_hard acceleration_, _acceleration_, _idle_, _deceleration and hard deceleration_\(\}\). Then, based on the decision taken, the lower level controller, i.e., a PID controller, generated the corresponding throttle signals to maneuver the CAV. Observation: The observation \(o_{i}\) of the vehicle \(v_{i}\) included the information required to make decisions effectively. It was assumed that each vehicle could sense its state and exchange information with its neighbors, which is denoted as \(N_{i}\). The neighbors of a vehicle were defined as the vehicles that were within \(L_{c}\) meters and with the potential of a collision, i.e., a different priority on their respective routes. The observation \(o_{i}\) was defined as a matrix of \(b\times W\), where \(b\) denotes the upper bound of the number of its neighbors and \(W\) is the number of features \(c_{i}\) representing the state of a vehicle. Specifically, the observation feature is defined as \(c_{i}=[\)_i_prestent_\(x,y,v_{x},v_{y},h,ps]^{T}\), where _i_prestent is a binary flag denoting whether a vehicle can be observable. \(x,y,v_{x},v_{y}\) represent the absolute longitudinal position, the lateral position, the longitudinal speed, and the lateral speed for the ego vehicle, while the relative to the ego vehicle for observed vehicles; \(h\) denotes the heading; and \(ps\) represents the priority of crossing the intersection. The entire state of the system is the Cartesian product of the individual observation, that is, \(S=O_{1}\times O_{2}\times\cdots\times O_{N}\). Reward: The basic goals of a vehicle at the intersection include driving safely, crossing the intersection, and reaching its target lane under the right of way rules and in a timely manner. Several rewards are specified as follows. 1. The occurrence of a collision is detected by the body circle model.