## 3 Method
### Simulation Reasoning  
In reinforcement learning, the agent updates its policy according to the reward signal feedback from the environment. In MARL, in order to make agents consider the impact of actions on the collective, we give each agent additional intrinsic rewards. Moreover, agents'observations are not comprehensive in the partially observable Markov decision process (POMDP) [36] so we cannot just use the historical observation to make judgments.  
To make that, we replicate the environment \(E\) to \(E^{c}\). The agent selects action \(a\) according to policy \(\pi\) in \(E\). We replace \(a\) to default action \(a^{c}\) and submit the joint action \(A=a_{1}...a_{1}^{c}...a_{N}\) to \(E^{c}\). Then continue to run the environment \(E^{c}\) for K steps while maintaining the policies of all agents unchanged. The operation of the simulated environment implies changes in the latent state of the environment. Performing a counterfactual action, in this case, can get the influence of that action on the latent state.  
Counting the data of these K steps, we get the phased collective reward \(R_{t}^{c}=\sum_{i=0}^{N}\sum_{l=0}^{K}r_{t,l+l}^{c}\). At the same time, the phased collective reward in the real environment is \(R_{t}^{c}\). Then \(r_{t}^{c}=R_{t}^{c}-R_{t}^{c}\) represents the difference between the two results when agent \(i\) execute action \(a\) and does not execute action \(a\). The main procedure of SR is shown in Algorithm 1.  
```
1:Initialize actor network \(\theta_{i}\) and critic network \(\phi_{i}\) for each agent \(i\in(1,N)\) with random parameters \(\theta,\phi\). Initialize hyperparameter predict step K, learning rate \(\alpha\), intrinsic factor \(\beta\).
2:for each episodes do
3:for t=1,T do
4: Copy the environment get \(E^{c}\)
5: Sample actions \(A=(a_{1},a_{2},\ldots,a_{N})\) from policies \(\pi_{1...N}\)
6: Execute actions in \(E\) and get reward \(r_{t}^{c}\)
7:for each agents do
8: Replace action \(a_{i}\), execute \(A^{\prime}\) in simulation environment \(E^{c}\), get reward \(r_{i,0}^{c}\)
9:for j=1,K do
10: Repeat 5,6 in \(E^{c}\) and get reward \(r_{i,j}^{c}\)
11:endfor
12:endfor
13: Compute phased simulation collective reward \(R_{i,t}^{c}=\sum_{i=1}^{N}\sum_{j=0}^{K}r_{i,j}^{c}\)
14: Compute phased collective reward \(R_{t}^{c}=\sum_{i=0}^{N}\sum_{j=0}^{K}r_{t+j}^{c}\)
15: Compute intrinsic reward \(r_{t}^{c}=R_{t}^{c}-R_{t}^{c}\) for each agent
16: Rewrite reward \(r_{t}=(1-\beta)r_{t}^{c}+\beta r_{t}^{c}\)
17: Compute gradient \(\
abla_{\theta},\
abla_{\phi}\).
18: Update policy and network \(\theta_{t+1}\leftarrow\theta_{t}-\alpha\
abla_{\theta}\), \(\phi_{t+1}\leftarrow\phi_{t}-\alpha\
abla_{\phi}\)
19:endfor
20:endfor
```  
**Algorithm 1** Simulation reasoning  
We know that part of the current training difficulties in reinforcement learning comes from the collection of samples. In a complex environment, the agent needs to continuously interact with the environment, including action decision-making and environmental state transition. This takes a lot of time and resources to collect and store the sample. Actually, our proposed simulation reasoning method is more time-intensive. Each simulation reasoning step takes extra time to replicate the environment and run additional \(K\) steps to collect simulation data. Nevertheless, the final collective reward is still a sufficient improvement compared to the baseline in Section 4. It is acceptable to trade efficiency for performance under the circumstance of sufficient hardware resources today.