## 4 Cooperative Decision-Making in a Multi-Vehicle Cooperative Task
### Cooperative Learning Algorithm for Multi-Agent Task  
The MARL method was leveraged to deal with the unsignalized intersection management due to the fact that it could produce an optimal policy through a continuous interaction with the traffic environment. A cooperative PPO-based decision-making method (Attn-MAPPO) is proposed in this section, as shown in Figure 3. The algorithm was based on a centralized training decentralized execution (CTDE) framework to reduce the environmental instability; that is, all the information of the agents was utilized during training and the agents only made decisions according to their own local observation after the training. Specifically, after choosing the actions from the policy to interact with the environment, all the vehicles executed them and reached a new state. Based on the new observation, an attention module was used to aggregate the information from the neighbors of a vehicle. Then, the output of the module was used to update the critic network. The critic network evaluated the taken actions and the agent network was updated. In this paper, each vehicle could observe its neighbors and collect their state information to make decisions. Although the relevant vehicle was selected as the focus, the aggregated information was not the most valuable information. For example, the state information of the neighbors which were more likely to collide required more attention. The attention mechanism was introduced in this paper to selectively pay attention to the neighbors' observations. More specifically, the structure of the attention representation is defined as Figure 4. The relevant message from other vehicles, \(m_{t}^{i}\), is summed by assigning the attention weights to the embedding of each agent, which is mathematically expressed as the following. \[m_{t}^{i}=\sum_{j\
eq i}\alpha_{t}^{i,j}W_{v}e_{t}^{j}, \tag{9}\]  
where \(e_{t}^{j}=f_{j}(o_{t}^{i})\) is the embedding where a multi-layer perceptron (MLP) \(f_{j}(\cdot)\) is used as the embedding function and \(W_{v}\) is a matrix to linearly transform \(e_{t}^{j}\) into a \"value\". The  
Figure 3: The framework of the proposed Attn-MAPPO algorithm. attention weight \(\alpha_{t}^{i,j}\) between the vehicles \(i\) and \(j\) was calculated using a _softmax_ function, as shown in the following. \[\alpha_{t}^{i,j}=\frac{\exp(\beta_{t}^{i,j})}{\sum_{j\
eq i}\exp(\beta_{t}^{i,j} )}, \tag{10}\]  
\[\beta_{t}^{i,j}=W_{q}e_{t}^{i}\cdot{(W_{k}e_{t}^{j})}^{T}, \tag{11}\]  
where \(\beta_{t}^{i,j}\) computes the correlation between the vehicles \(i\) and \(j\), the matrix \(W_{q}\) linearly transforms \(e_{t}^{i}\) into a \"query\", and \(W_{k}\) transforms \(e_{t}^{i}\) into a \"key\". In the cooperative multi-agent setting, each agent shared an actor network (i.e., policy) and a critic network (i.e., policy evaluation), which were trained using the individual trajectory \((o_{t}^{i},a_{t}^{i},r_{t}^{i})\). The actor network was trained to maximize the expected return. Each agent generated its action \(a_{t}^{i}\) from the policy \(\pi_{\theta}\) based on its local observations \(o_{t}^{i}\) (i.e., \(\pi(a_{t}^{i}|o_{t}^{i};\theta)\)  at the time slot \(t\). The most valuable information which an attention representation aggregated from the neighbors of an agent was access to the input of the critic network. An episodic setting was considered with each vehicle until a crash occurred or until it proceeded \(T\) seconds. Our objective was to obtain the optimal joint policy that maximized the accumulated reward. The PPO algorithm based on the actor-critic framework was an improvement of the policy gradient (PG) algorithm. The objective function of the PG algorithm is expressed as the following.