# Environmental-Impact-Based Multi-Agent Reinforcement Learning
## 3 The EMuReL Approach
The first is the _forward model_ representing \(\hat{\phi}\) by a neural network function \(f\) with parameter \(\theta_{F}\) that predicts the encoded state \(\phi(s_{t}^{k})\) based on the previous joint action \(\mathbf{a}_{t-1}\) and encoded state \(\phi(s_{t-1}^{k})\). However, the state \(s_{t}^{k}\) is the result of applying the agents' actions \(\mathbf{a}_{t-1}\) on the previous global state \(s_{t-1}\), not local state \(s_{t-1}^{k}\). Hence, the forward model needsthe global state \(s_{t-1}\) or an estimation of it. To this end, we exploit the Model of Other Agents (MOA) [12], which is a neural network embedded in each agent's architecture, to predict the other agents' actions based on their previous actions and the previous state, i.e., \(P(\mathbf{a}_{t}\mid s_{t-1}^{k},\mathbf{a}_{t-1})\). Via its internal LSTM state \(u_{t-1}^{k}\), MOA implicitly models the state transition function \(\mathcal{T}(s_{t}|s_{t-1},\mathbf{a}_{t-1})\) to estimate the global state \(s_{t}\) (the green nodes in the gray rectangle in Figure 11) or \(s_{t-1}\) when the time is shifted negatively by one unit, i.e., \(\mathcal{T}(s_{t-1}|s_{t-2},\mathbf{a}_{t-2})\). By passing this internal LSTM state to the forward model, we implicitly provide an estimate of the global state \(s_{t-1}\). Therefore, the estimation of the encoded state \(\phi(s_{t-1}^{k})\) can be written as follows in terms of the neural network function \(f\):  
\[\hat{\phi}(\phi(s_{t-1}^{k}),\mathbf{a}_{t-1})=f(\phi(s_{t-1}^{k}),u_{t-1}^{k},\bm {a}_{t-1};\theta_{F}).\]  
During the training of the forward model, parameters \(\theta_{F}\) and \(\theta_{\phi}\) are learned to minimize \(L_{F}\) defined as the discrepancy between the predicted and actual encoded local states:  
\[L_{F}(\phi(s_{t}^{k}),\hat{\phi}\Big{(}\phi(s_{t-1}^{k}),\mathbf{a}_{t-1}\Big{)}= \frac{1}{2}\Big{\|}\hat{\phi}(\phi(s_{t-1}^{k}),\mathbf{a}_{t-1})-\phi(s_{t}^{k}) \Big{\|}_{2}^{2}. \tag{4}\]  
Second, to lead the forward model to extract those features that are more relevant to the agents' actions, the actions are estimated using the so called _inverse model_ and parameters \(\theta_{\phi}\) are tuned by minimizing the loss of the actions estimation. The inverse model is a neural network function \(g\) with parameters \(\theta_{I}\) that predicts the applied joint action \(\mathbf{a}_{t-1}\) given \(\phi(s_{t-1}^{k})\) and \(\phi(s_{t}^{k})\) as follows:  
\[\hat{\mathbf{a}}_{t-1}=g(\phi(s_{t-1}^{k}),\phi(s_{t}^{k}),u_{t-1}^{k};\theta_{I}).\]  
Then, parameters \(\theta_{I},\theta_{\phi}\) are learned to minimize \(L_{I}(\hat{\mathbf{a}}_{t-1},\mathbf{a}_{t-1})\), where \(L_{I}\) is a cross entropy over the predicted and actual actions of the agents:  
\[L_{I}(\hat{\mathbf{a}}_{t-1},\mathbf{a}_{t-1})=-\sum_{j=1}^{N}a_{t-1}^{j}\log(\hat{a }_{t-1}^{j}). \tag{5}\]  
The inverse model assists in learning a feature space that encodes only the relevant information to predict the actions of the agents, and the forward model makes the representation of these learned features more predictable. Thus, the feature space has no incentive to encode environmental features that are not affected by the agents' actions. Figure 1: **The elimination process of computing impacts by agent \(k\). ** In the iterative procedure of this process, each time, an agent is removed from the neural network computations of the forward model by setting the corresponding network weights of that agent in the input to zero. The disparity between the outputs of the forward model in the presence and absence of agent \(j\) is employed as a measure of agent \(j\)’s impact in predicting the features of the current local state from agent \(k\)’s viewpoint.