# Multi-Agent Reinforcement Learning: A Review of Challenges and Applications
## 3 The Limits of Multi-Agent Reinforcement Learning
### Partial Observability
#### 3.3.1 Centralized Learning of Decentralized Policies
It was shown that, in that setting, COMA reached competitive performance in regard to fully centralized methods with better training speed. While the paradigm of the centralized learning of decentralized policies is easily implementable for actor-critic and policy gradient methods, it is not as straightforward when considering value-based methods. A possible approach was presented in [40] that consists of decomposing the team value function into agent-wise value functions. The assumption on which this approach is based is that the joint action-value function can be factorized according to each agent's Q-function based only on each agent's local observation:  
\[Q((o^{1},o^{2},\ldots,o^{N}),(a^{1},a^{2},\ldots,a^{N}))\approx\sum_{i=1}^{N}Q_ {i}(o^{i},a^{i}) \tag{22}\]  
```
Initialise \(\varrho_{1}^{c}\), \(\delta_{1}^{c}\), \(\theta^{\pi}\) for each training episode \(e\)do  Empty buffer for\(e_{c}=1\)to\(\frac{BatchSize}{n}\)do \(s_{1}=\)initial state, \(t=0\), \(h_{0}^{a}=\mathbf{0}\) for each agent \(a\) while\(s_{t}\
eq\) terminal and \(t<T\)do \(t=t+1\) for each agent \(a\)do \(h_{t}^{a}=\) Actor \((\varrho_{t}^{a},h_{t-1}^{a},u_{t-1}^{a},a,u;\theta_{i})\)  Sample \(u_{t}^{a}\) from \(\pi(h_{t}^{a},e(e))\) endfor  Get reward \(r_{t}\) and next state \(s_{t+1}\) endwhile  Add episode to buffer endfor  Collate episodes in buffer into single batch for\(t=1\)to\(T\)do //from now processing all agents in parallel via single batch  Batch unroll RNN using states, actions and rewards  Calculate TD(\(\lambda\)) targets \(y_{t}^{a}\) using \(\hat{\varrho}_{t}^{c}\) endfor for\(t=T\)down to\(1\)do \(\Delta Q_{i}^{a}=y_{t}^{a}-Q(s_{i}^{a},u)\) \(\
abla\theta^{c}=\
abla\theta^{c}+\frac{\partial}{\partial\theta^{c}}(\Delta Q _{t}^{a})^{2}\) //calculate critic gradient \(\theta_{t+1}^{c}=\theta_{t}^{c}+a\
abla\theta^{c}\) // update critic weights  Every \(C\) steps reset \(\hat{\varrho}_{t}^{c}=\theta_{t}^{c}\) endfor for\(t=T\)down to\(1\)do \(A^{a}(s_{t}^{a},u)=Q(s_{t}^{a},u)-\sum_{u}Q(s_{t}^{a},u,u^{-a})\pi(u|h_{t}^{a})\) //calculate COMA \(\
abla\theta^{\pi}=\
abla\theta^{\pi}+\frac{\partial}{\partial\theta^{\pi}} \log\pi(u|h_{t}^{a})A^{a}(s_{t}^{a},u)\) // accumulate actor gradients endfor\(\theta_{t+1}^{\pi}=\theta_{t}^{\pi}+a\
abla\theta^{\pi}\) //update actor weights
```  
**Algorithm 2** COMA pseudocode [39]. Each agent taking a greedy action to maximize their returns is equivalent to a central controller maximizing the joint action-value functions. The value decomposition network (VDN) used DQN or DRQN, with the possibility of communication between agents at a low level (sharing observations) or high level (sharing network weights). It was tested in a two-dimensional grid environment, obtaining better performances than the centralizedand independent learners methods. The limit of this approach is that not every multi-agent problem can be approximated and solved as a summation of Q-functions. In [41], the VDN method was extended with the QMIX algorithm. The authors argued that a full factorization of the value function is not required to extract effective policies. It is sufficient that the result of an \(argmax\) on the joint action-value functions produces the same result to apply an \(argmax\) to all the individual action-value functions. This is possible if a monotonicity constraint is enforced between the total Q-value \(Q_{tot}\) and local agent Q-value \(Q_{a}\). \[\frac{\partial Q_{tot}}{\partial Q_{a}}>0,\forall a \tag{23}\]  
Each agent value function \(Q_{a}(\sigma_{i}^{a},u_{t-1}^{a})\) is represented by a DRQN that takes, as its input, the local observation and the last action at each time step. Then, the singular value functions are combined using a mixing network, which is a feed-forward neural network. The weights of the mixing network are bounded to be non-negative to enforce the condition presented in Equation (23). These weights are determined by a separate hyper-network which takes the augmented state as its input.