# Multi-Agent Reinforcement Learning: A Review of Challenges and Applications
## 3 The Limits of Multi-Agent Reinforcement Learning
### Scalability  
As the number of agents increases, there is a growth in the joint action space. For this reason, centralized approaches, in which an observer selects the actions after receiving the action-state information of every agent, require large amounts of computational resources and memory to work with more than a couple of agents. A possible solution to the curse of dimensionality in MARL is to use independent learners, but as we have seen, this approach is unable to obtain consistent results in a non-stationary environment. A third model of agent connection is the decentralized setting with networked agents. In this setting, every agent is able to interact with the environment and to exchange information with few other agents (typically, those in its vicinity), creating a time-varying communication network between all the agents. Algorithms developed for this setting are scalable to a massive number of agents and more real-world-oriented applications, as the absence of a central controller and uncertainty in communication links are typical requirements in a large number of applications. In [28], a distributed version of Q-learning, namely, QD-learning, is proposed under the assumption that each agent is only aware of its local action and reward and the inter-agent communication network is weakly connected. The optimal policy is achieved by agents sending their Q-values to their neighbors. The update of the Q-value is then computed locally for each agent using the following equation:  
\[\begin{split} Q^{n}_{i,u}(t+1)&=Q^{n}_{i,u}(t)- \beta_{i,u}(t)\sum_{l\in\Omega_{n}(t)}\left(Q^{n}_{i,u}(t)-Q^{n}_{l,u}(t) \right)+\\ &+\alpha_{i,u}(t)\bigg{(}r_{n}(\mathbf{x}_{t},\mathbf{u}_{t})+\gamma\min _{v\in U}Q^{n}_{x_{t+1,v}}(t)-Q^{n}_{i,u}(t)\bigg{)}\end{split} \tag{16}\]  
In can be seen from Equation (16) that the update is defined by two processes, consensus and innovation, where the former is the sum of the differences of the Q-value of the agent and the Q-values of its neighbors, controlled by the weight sequence \(\beta_{i,u}(t)\), and the latter is the traditional Q-learning update rule weighted by the sequence \(\alpha_{i,u}(t)\). In [29], the same setting of decentralized reinforcement learning with networked agents is addressed using an actor-critic algorithm. The policy is parametrized by a set \(\theta\in\Theta\), and the gradient of the globally averaged return \(J(\theta)\) with respect to \(\theta\) is given by  
\[\begin{split}\
abla_{\theta^{i}}J(\theta)&=\mathbb{ E}_{s\sim d_{\theta},\sim\tau_{\theta}}[\
abla_{\theta^{i}}\log\pi^{i}_{\theta^{i}}(s,a^{i})\cdot A_{\theta}(s,a)]\\ &=\mathbb{E}_{s\sim d_{\theta},\sim\tau_{\theta}}[\
abla_{\theta ^{i}}\log\pi^{i}_{\theta^{i}}(s,a^{i})\cdot A^{i}_{\theta}(s,a)]\end{split} \tag{17}\]  
The local advantage function \(A^{i}_{\theta}(s,a):S\times A\rightarrow\mathbb{R}\) is defined as  
\[A^{i}_{\theta}(s,a)=Q_{\theta}(s,a)-\hat{V}^{i}_{\theta}(s,a^{-i}) \tag{18}\]  
where \(\hat{V}^{i}_{\theta}(s,a^{-i})=\sum_{a^{i}\in A^{i}}\pi^{i}_{\theta^{i}}(s,a^ {i})\cdot Q_{\theta}(s,a^{i},a^{-i})\) and \(a^{-i}\) is the action chosen by every agent except for agent \(i\). The advantage function is not available considering only local information, so \(A_{\theta}\) is estimated with the state-value TD-error, which is an unbiased estimator of the advantage function.