# Environmental-Impact-Based Multi-Agent Reinforcement Learning
## 1 Introduction
To this end, researchers have enriched extrinsic rewards by the so-called _intrinsic rewards_[10] that capture aspects of the agents' behavior that are not necessarily encoded by the extrinsic reward. For each agent \(k\) at time step \(t\), this results in its _reshaped reward_\(r_{t}^{k}\), defined as a linear combination of its extrinsic reward \(e_{t}^{k}\) and intrinsic reward \(\hat{t}_{t}^{k}\):  
\[r_{t}^{k}=\alpha e_{t}^{k}+\beta\hat{t}_{t}^{k}.\]  
for constant scalers \(\alpha\) and \(\beta\). Insights into the nature of human social behavior have led to the introduction of two main categories of intrinsic rewards. The first focuses on self-preferencing attributes, including empowerment [11], social influence [12], and curiosity [13; 14]. The second is motivated by social welfare emotions, including envy and guilt [15], and empathy [16]. For example, in the inequity aversion (IA) model [15], which is based on envy and guilt, each agent compares its own and fellows' extrinsic rewards to detect inequities and balances its selfish desire for earning rewards by keeping the differences as small as possible. A _disadvantageous inequity_ occurs when agent \(k\) experiences \"envy\" by earning a lower reward than another agent \(j\), that is, if \(e_{t}^{j}-e_{t}^{k}>0\). Conversely, an _advantageous inequity_ happens when the agent experiences \"guilt\" by earning a higher reward than another agent \(j\), that is, if \(e_{t}^{k}-e_{t}^{j}>0\). To incorporate its feelings of envy and guilt, agent \(k\) averages these comparisons over all other agents, resulting in the following intrinsic reward:  
\[\hat{t}_{t}^{k}=-\frac{\alpha_{k}}{N-1}\sum_{j\
eq k}\max(e_{t}^{j}-e_{t}^{k},0)-\frac{\beta_{k}}{N-1}\sum_{j\
eq k}\max(e_{t}^{k}-e_{t}^{j},0), \tag{1}\]  
where \(N\) is the total number of agents and parameters \(\alpha_{k},\beta_{k}\in\mathbb{R}\) control the agent's aversion to disadvantageous and advantageous inequities, respectively. Regardless of whether it earns more or less than others, the agent decreases its obtained reward. Here, the feeling of guilt can be interpreted as a feeling of social responsibility; that is, the agent wants to revise its policy to reduce the reward inequity of itself compared to changing the others who are \"worse-performing\". However, the IA model measures the agents' performance only by their obtained rewards and ignores their roles in establishing the current environment state that resulted in those rewards. This may lead to equal treatments of defectors and cooperators that earn the same reward despite their different contributions to the obtained rewards. Namely, rather than only those who earned less than itself, each agent should also \"feel responsible\" for those who earned more but did not contribute to reaching the current rewarding environment state. Even among others who are equally less-earned, a higher social responsibility should be dedicated to those who also contributed less to reaching the current state. Similarly, compared to other higher-earning agents, the agent feels that it should work \"harder\" (by finding a better policy) to earn as much as they do, and this feeling should be stronger toward those who played a more effective role in changing the state. To take into effect the agents' roles in reaching the environment states, we propose an _environmental impact_ of each agent defined by the difference between the current local state and the hypothetical one where that agent would have been absent in the previous state. In the autonomous vehicle example, a vehicle's impact can be the difference in the current congestion in the presence and absence of that vehicle. When performing the comparisons in reshaping the reward function, each agent \(k\) computes the impact \(d_{t}^{k,j}\) of every other agent \(j\) and scales that agent's reward \(e_{t}^{j}\) by its impact \(d_{t}^{k,j}\). We demonstrate the effectiveness of the proposed impact criterion through experiments in the Cleanup and Harvest environments [15; 17; 18].