# Environmental-Impact-Based Multi-Agent Reinforcement Learning
## 5 Experiments
### Experimental Results  
The total reward received by all agents is considered as a measure that explains how well the agents learned to cooperate [15]. According to Table 1, when trained based on the EMuReL method, the agents could earn 53.6% and 38.6% more total rewards than when they were trained based on the IA model, 19.7% and 44.2% more than when they were trained based on SI method, and 36.9% and 10.2% more than when they were trained by using only the extrinsic rewards in the Cleanup and Harvest environments, respectively (_percentage increase_ = \(\frac{new\ value-original\ value}{original\ value}\times 100\)). According to subplot (a) in Figure 2, the EMuReL method persistently outperformed the other methods starting from step \(2.15\times 10^{8}\) and had the minimum performance variance. Moreover, the EMuReL method improved the agents' cooperation while maintaining the equality between the obtained rewards of the agents better than other methods (subplot (b) in Figure 2). The mean equality of all methods is calculated by using the Gini coefficient as \(Equality=1-\frac{\sum_{i=1}^{N}\sum_{j=1}^{N}|R^{i}-R^{j}|}{2N\sum_{i=1}^{N}R^{i}}\)[15].  
All four methods performed almost identically in the Harvest environment using the PPO algorithm (see Figures A4 and A5 in Appendix A.1). The same results were reported by [PERSON] [8], comparing the baseline, SI, and SCM methods when the PPO algorithm is applied. Since the results reported by [PERSON] et al. [12], [PERSON] et al. [15] for training the agents based on A3C algorithm achieved higher collective rewards compared to our PPO results, we also used the A3C instead of the PPO algorithm to examine the performance of the EMuReLU method. We compare the results with four agents in Figure 3. This plot shows that the EMuReLU method with advantageous-IA agents (called advantageous EMuReLU method) outperformed other methods.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c}{**Environment (Algorithm)**} \\ \cline{2-3}  & **Cleanup (PPO)** & **Harvest (A3C)** \\ \hline Baseline & 484.4 & 622.2 \\ IA & 431.7 & 494.7 \\ SI & 554.2 & 475.5 \\ EMuReLU & **663.2** & **685.5** \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Collected rewards in Cleanup and Harvest environments.** The collective reward obtained by agents in the last 192,000 and 1,600,000 steps is averaged over 13 and 5 experiments in the Cleanup and Harvest environments, respectively. The results of the IA and EMuReLU methods are reported for the advantageous-IA agents. The largest number in each column is in boldface.  
Figure 2: **The results for the Cleanup environment.** The mean reward obtained by 5 agents over individual experiments of the baseline, IA, SI, and EMuReLU methods using the PPO algorithm. Each point of these curves shows the average collective reward over at least 96,000 environment steps (96 episodes of 1000 steps). The opaque curve is the mean of the results of 15 experiments. The shadows are the bands of the confidence interval obtained by estimating the unbiased variance of the collective rewards. (**a**) The comparison between the mean collective rewards of all methods computed by removing the best and worst results of each method. The IA and baseline methods have a sharp initial raise but then slow down, whereas the EMuReLU and SI have a relatively constant growth rate (EMuReLU as the smoothest) and that EMuReLU grows with a greater slop compared to the SI. (**b**) The comparison between the mean equality of all methods.