# Multi-Agent Reinforcement Learning: A Review of Challenges and Applications
## 3 The Limits of Multi-Agent Reinforcement Learning
### Partial Observability
#### 3.3.1 Centralized Learning of Decentralized Policies  
Scalability and partial observability are connected problems, especially when considering the applications of algorithms to real-world problems. In a setting with an extremely high number of agents, as in the case of the number of autonomous driving cars in a city, it is impossible for every car to have full information about the state, as cars cannot exchange information with each other without a prohibitive communication overhead. The full information about the state could therefore not be useful for executing an optimal policy, but it could be useful during the learning phase to correctly estimate the value-functions or the policy gradients. To exploit this dichotomy, the paradigm of centralized learning of decentralized policies was introduced. The idea behind this is that, during learning, it is often possible to have access to a simulator of the environment or a central controller, or have additional information about the state. We can imagine a setting in which the agents are, for example, a swarm of drones trained in a closed building, such as a hangar, with limited visibility, and the extra information about the state of the environment is given by a fixed camera looking down on all of the drones. In [36], a comparison between various training schemes--centralized, independent learning, and centralized training--for decentralized policies using parameter sharing is presented. A parameter sharing variant of the single agent algorithm trust region policy optimization (TRPO) [37], namely, PS-TRPO, is presented. The decentralized paradigm obtained the best performance of all the training schemes in the evaluation phase. In addition, the scalability of the PS-TRPO algorithm was addressed using curriculum learning [38], in which the agents had to solve a number of sub-tasks that increased in difficulty in order to coordinate with an increasing number of agents. To exploit the utility of having a centralized learning phase in [39], a novel actor-critic architecture, called the configactual multi-agent policy gradient (COMA), was presented. In the training, a single centralized critic is used, which estimates the Q-function using the joint action and the full information on the state, or in the absence of a well-defined state, the concatenation of all the local observations made by the agents, and several decentralized actors that deploy a policy which maps from local observations to an action. Additionally, COMA tackles the challenge of multi-agent credit assignment in a cooperative setting where there is a unique reward shared to all the agents. Training is more efficient if each agent is able to determine how an action reflects on the success of the task. This is done through the use of a counterfactual baseline; each agent receives a shaped reward, which is the difference between the reward obtained at the time-step \(t\) and the reward that would be obtained if the agent actions were to change to the default \(D^{a}=r(s,\mathbf{u})-r(s,(\mathbf{u}^{-a},c^{a}))\). It can be shown that any action by the agent \(a\) that improves \(D^{a}\) also improves the true global reward. The main downsides of this approach are the necessity for a simulator of the environment to calculate \(r(s,(\mathbf{u}^{-a},c^{a}))\) and the choice of the default action \(c^{a}\), which is not always trivial. Both of these problems were addressed by the architecture of the critic which computed for each agent an advantage function as follows:  
\[A^{a}(s,u^{a})=Q(s,\mathbf{u})-\sum_{u^{a}}\pi^{a}(u^{a}|o^{a})Q(s,(\mathbf{u }^{-a},u^{a})) \tag{21}\]  
In particular, the actions \(\mathbf{u}^{-a}\) are given as an input to a network that determines the state-action value for each action of the agent \(a\) in a single forward pass. This algorithm (full pseudocode can be found in Algorithm 2) was tested in Starcraft, a combat-based video-game environment. Several homogeneous and heterogeneous unit combinations, with each one represented by an agent, were considered.