# Environmental-Impact-Based Multi-Agent Reinforcement Learning
## 6 Discussion  
Researchers have developed several MARL methods to train cooperative agents in SSD problems [32; 33]. They introduced intrinsic rewards as a stimulus that is internally computed by agents to accelerate and improve their learning process [34]. IA and SI are two state-of-the-art examples of reward reshaping methods that are based on two social concepts: inequity aversion and social influence [12; 15]. We propose the EMuReLU method based on the social responsibility concept. To this end, we define the environmental impact as a criterion to measure the role of each agent in reaching the current environment state and, in turn, making the agents continuously measure the cooperativeness of their fellows. We incorporate these impacts into the reward function of the IA method. So, in the advantageous case, the more the agents play an impactful role in reaching the current environment state, the less each agent feels socially responsible for them and, as a result, the less it penalizes itself with negative internal rewards induced by the IA method.  
To compute the agents' impacts, inspired by the SCM method [8], we propose the EICM structure, which extends the single-agent curiosity setup of the ICM method [14] to the MARL setting. The EICM structure utilizes the representation function \(\phi\) to isolate the impact of each agent's action on the current environment state and to make the agents \"curious\" about the behavior of others by assessing their environmental impacts. The EMuReLU method achieved better results compared to the IA and SI methods in the Cleanup and Harvest environments.  
In the Cleanup environment, as [15] explains, the (advantageous) IA method encourages the agents to contribute and clean up the river. Cleaning up the waste causes producing more apples in the field and makes other agents more successful in collecting more apples. So, it reduces the negative rewards fed by the IA method. Here, incorporating the environmental impacts in the IA method improves the agents' policies. Depending on what action each agent chooses, its impact on the environment is remarkably different in this environment. If the agent chooses one of the movement actions without collecting apples, it creates the minimum change in the environment state, which is simply the change of its location. Such agents are subject to social responsibility by the other agents. If, however, the movement action is with collecting an apple, the field conditions additionally change. The most impactful agents are those who clean the river. Such an agent, in addition to changing the condition of the river on a large scale, changes the condition of the field by producing several apples depending on the growth rate of the apples.  
After completing the learning phase of the EMuReLU method in the Cleanup environment, the behavior of the agents were investigated in the rendered videos for the polices learned in one of the last episodes. It was observed that one of the agents learned a circular movement pattern in the environment so that it traveled the entire length of the river and then entered the apple field. On the way back to the river, it collected the apples. This agent also followed a specific pattern to clean up the river. It cleaned the maximum possible width of the river whenever it applied the cleaning beam and cleaned the whole width of the river every time it crossed the river. When returning to the river area, it chose the shortest path according to the position of the apples in the field. The other agents maneuvered in the apple field and collected apples. None of the agents in their behavior pattern used the firing beam; neither did they prevent each other from moving in the direction of the apples.