# Environmental-Impact-Based Multi-Agent Reinforcement Learning
## 3 The EMuReL Approach  
To define intrinsic rewards, we use the same equation as in the IA model (1) with two main differences. First, as in [15], to take into account the temporally distributed, rather than the single-instant reward, we replace the extrinsic reward \(e_{t}^{j}\) with the _temporal smoothed extrinsic rewards_\(w_{t}^{j}\) defined by  
\[w_{t}^{j}=\gamma\lambda w_{t-1}^{j}+e_{t}^{j}\qquad\forall t\geq 1,\ w_{0}^{j}=0, \tag{2}\]  
where \(\lambda\in[0,1]\) is a hyperparameter. Second, when performing the IA comparisons, agent \(k\) scales the reward of every agent \(j\) by \(d_{t}^{kj}\in[0,1]\), that is, agent \(j\)'s _environmental impact_ in establishing agent \(k\)'s local state at time \(t\). Hence, agent \(k\)'s intrinsic reward is  
\[i_{t}^{k}=-\frac{\alpha_{k}}{N-1}\sum_{j\
eq k}\max(d_{t}^{kj}w_{t}^{j}-w_{t} ^{k},0)-\frac{\beta_{k}}{N-1}\sum_{j\
eq k}\max(w_{t}^{k}-d_{t}^{kj}w_{t}^{j}, 0), \tag{3}\]  
where \(\alpha_{k},\beta_{k}\in\mathbb{R}\). The definition of the impact \(d_{t}^{kj}\) lies in the answer to the following question:  
Agent \(k\): \"How impactful was the presence of agent \(j\) in reaching my current local environment state \(s_{t}^{k}\)?\"  
So, agent \(k\) needs to compare its current local state \(s_{t}^{k}\) in the presence and absence of agent \(j\). This requires estimating the typically large size and detailed state \(s_{t}^{k}\) in the absence of agent \(j\) with a reasonable accuracy, which may not be practical. Moreover, \(s_{t}^{k}\) is often an image with many features that may not have been caused by the agents' actions. Therefore, rather than \(s_{t}^{k}\), we use a reduced-dimension _feature encoding function_\(\phi(s_{t}^{k};\theta_{\phi})\), briefly denoted as \(\phi(s_{t}^{k})\), that encodes the raw state \(s_{t}^{k}\) into a feature vector by using a deep neural network (Figure 10). This function encodes only those features of the environment state that are influenced by the agents' actions [14]. Now, inspired by the dropouts technique used in neural network regularization [22] to be able to extract the features of the local state \(s_{t}^{k}\) in the absence of agent \(j\), we need to estimate \(\phi(s_{t}^{k})\) when agent \(j\) is omitted. Thus, we first define the _estimated feature encoding function_\(\hat{\phi}\) that, instead of \(s_{t}^{k}\), takes the previous joint action \(\mathbf{a}_{t-1}\) and features \(\phi(s_{t-1}^{k})\) as its input and estimates \(\phi(s_{t}^{k})\) as its output, i.e.,  
\[\hat{\phi}(\phi(s_{t-1}^{k}),\mathbf{a}_{t-1})\approx\phi(s_{t}^{k}).\]  
Then, by omitting \(a_{t-1}^{j}\), resulting in the reduced joint action denoted by \(\mathbf{a}_{t-1}^{-j}\), we obtain the estimated features at state \(s_{t}^{k}\) in the absence of agent \(j\) (Figure 1). Taking the norm of the difference of the two estimated feature results in the impact of agent \(j\) in view of agent \(k\):  
\[d_{t}^{kj}=\frac{1}{2}\left\|\hat{\phi}(\phi(s_{t-1}^{k}),\mathbf{a}_{t-1})-\hat{ \phi}^{-j}(\phi(s_{t-1}^{k}),\mathbf{a}_{t-1}^{-j})\right\|_{2}^{2},\]  
where \(\|\cdot\|_{2}\) is the Euclidean norm and \(\hat{\phi}^{-j}\) is the same as \(\hat{\phi}\) but when agent \(j\) is eliminated. These disparities are scaled using unity-based normalization to bring all values into the range \([0,1]\). We emphasize that the elimination of an agent is different from when the agent is present but performs no operation (NOOP). To learn the feature encoding function \(\phi\) and its estimation \(\hat{\phi}\), inspired by the neural network structure of the Social Curiosity Module (SCM) [8] for multi-agent systems, we extend the Intrinsic Curiosity Module (ICM) [14] in the single-agent setup to our multi-agent RL system. We propose the Extended ICM (EICM) by embedding two associated predictors inside the architecture of each agent \(k\) (Figure 10).