## 4 Cooperative Decision-Making in a Multi-Vehicle Cooperative Task
### Cooperative Learning Algorithm for Multi-Agent Task
\[\
abla J(\theta)=\mathbb{E}_{(s_{t},a_{t})\sim\pi_{\theta}}[\
abla_{\theta} \log\pi_{\theta}(a_{t}\Big{|}s_{t})\,A_{\pi_{\theta}}(s_{t},a_{t})], \tag{12}\]  
where  
\[A_{\pi_{\theta}}(s_{t},a_{t})=Q_{\pi_{\theta}}(s_{t},a_{t})-V_{\phi}(s_{t}) \tag{13}\]  
is an estimation of the advantage function at the step \(t\) and \(\theta,\phi\) are the parameters for identifying the policy and state value functions, respectively. By modifying the objective, the PPO constructs a clipped surrogate objective function to confine the policy updates to a small scope of around 1. The objective and update functions are illustrated as the following. \[J^{\text{PPO}}(\theta)=\mathbb{E}_{(o_{t}^{i},a_{t}^{i})\sim\pi_{\theta_{ \theta}}}[\psi(\rho_{\theta}^{i},A_{t}^{i})], \tag{14}\]  
Figure 4: Flowcharts of the attention representation. Figure 4.0  
where \(l\) is the learning rate and \(\rho_{\theta}^{i}=\frac{\pi_{\theta}(\sigma_{t}^{i}|\rho_{t}^{i})}{\pi_{\theta_{ old}}|\sigma_{t}^{i}|\sigma_{t}^{i}|}\) specifies the probability ratio of the old and new policies to select the action and  
Figure 4.1  
where \(\varepsilon\) is the penalty factor to prevent the policy from changing extremely. When \(A\geq 0\), the return for taking the action \(a_{t}^{i}\) is greater than the expected observation \(o_{t}^{i}\). Therefore, the updated policy should increase the probability of the action, but the increase amplitude should be restrained to \(\left(1+\varepsilon\right)\pi_{\theta_{old}}(a_{t}^{i}|o_{t}^{i})\). The opposite is true when \(A<0\). To decrease the variance, the advantage function \(A_{\pi_{\theta}}\) in Equation (13) is replaced by the generalized advantage estimation (GAE), which is expressed as the following. Figure 4.2  
where \(\delta_{t}^{i}(\gamma)=r_{t}^{i}+\gamma V_{\phi}(o_{t+1}^{i})-V_{\phi}(o_{t}^ {i})\) is the TD error and \(\lambda\) is a factor that balances the variance and bias of the estimation. When \(\lambda=0\), \(\hat{A}_{t}^{i}=r_{t}^{i}+\gamma V_{\phi}(o_{t+1}^{i})-V_{\phi}(o_{t}^{i})\), which is unbiased with a high variance. The GAE achieves both a low bias and a low variance by linearly integrating the \(n\)-step bootstrapping. The critic network \(V_{\phi}(\cdot)\) is updated using the loss function as follows. Figure 4.3  
Figure 4.4  
The whole pseudo-code for the proposed method is presented in Algorithm 1. ```
1:Initialize the actor network and target the actor network using the parameters \(\theta\) and \(\theta^{-}\);
2:Initialize the critic network and target the critic network using the parameters \(\phi\) and \(\phi^{-}\);
3:Initialize the memory buffer \(D_{i}\) and hyper-parameters \(lr\), \(\tau\), \(\epsilon\). 4:for episode = \(1\),..., \(M\)do
5:for\(t\leq T\) and not terminal do
6:for\(i\in V\)do
7: Observe \(o_{i}\) and select an action \(a_{i}\sim\pi_{\theta^{-}}\) using the \(\varepsilon\)-greedy strategy. 8: All agents execute the actions and receive their own reward \(r_{i}\). 9: Store trajectories \((o_{i},a_{i},r_{i})\) in \(D_{i}\). 10:endfor
11:for\(i\in V\)do
12: Obtain the attention representation of each agent using Equation (9). 13: Update the critic network \(\theta\) and the actor network \(\theta\) using a randomly sampled mini batch from \(D_{i}\) in Equations (15) and (19), respectively. 14: Update the target networks:\(\phi^{-}=\tau\phi+(1-\tau)\phi^{-},\theta^{-}=\tau\theta+(1-\tau)\theta^{-}\). 15:endfor
16:endfor
17: Initialize \(D_{i}\leftarrow\varnothing\), and reset the environment. 18:endfor
```  
**Algorithm 1**Attn-MAPPO