# Multi-Agent Reinforcement Learning: A Review of Challenges and Applications
## 3 The Limits of Multi-Agent Reinforcement Learning
### Partial Observability
#### 3.3.2 Communications between Agents  
In the case of the partial visibility of the environment, the use of communication between agents is often a necessity, as collaborating agents can then better infer the underlying state of the environment. Communication protocols are often hand-designed and optimized for the execution of particular tasks. In contrast with this approach, in [42], a simple neural communication model called CommNet was proposed, which learned a task-specific communication protocol that aided the performance of the agents. Considering a setting with \(J\) fully cooperative agents, a model was proposed with \(\mathbf{a}=\Phi(\mathbf{o})\), where \(\mathbf{a}\) is the concatenation of discrete actions for each agent \(j\) and \(\mathbf{o}\) is the concatenation of all agents observations. \(\Phi\) is built from modules \(f^{i}\) consisting of a multi-layer neural network with \(i\in 0,\dots,K\), where \(K\) is the number of communication steps of the network. Each agent \(j\) sends two input vectors to the module--the agent's hidden state \(h_{j}^{i}\) and the communication \(c_{j}^{i}\), and \(f^{i}\) outputs the next hidden state \(h_{j}^{i+1}\). The evolution of the hidden space and the communication message is regulated by the following equations:  
\[h_{j}^{i+1}=f^{i}(h_{j}^{i},c_{j}^{i}) \tag{25}\]  
\[c_{j}^{i+1}=\frac{1}{J-1}\sum_{j\
eq j}h_{j}^{i+1} \tag{26}\]  
The first hidden state is obtained through an encoder function \(h_{j}^{0}=r(s_{j})\), which takes the state observation of agent and outputs its hidden state. \(h_{j}^{0}\) is made of a single layer neural network. After the last round of communication, a decoder function, made of a single layer neural network followed by a softmax, is used to convert the hidden state \(h_{j}^{K}\) to a probability distribution over the space of action \(q(h_{j}^{K})\). The action is chosen by sampling this distribution. A variance of the broadcast transmission represented by Equation (26) is a local connectivity scheme:  
\[c_{j}^{i+1}=\frac{1}{|N(j)|}\sum_{j\in N(j)}h_{j}^{i+1} \tag{27}\]  
where \(N(j)\) is the neighborhood of agent \(j\). This model was tested with various difficult tasks, including a traffic simulation with four connected crossroads, with the objective being to avoid collisions while trying to maximize the vehicle flow. Promising results were obtained even with a very limited--if not absent--observability of the state. Another similar work addressing the communication problem in a multi-agent partially-observable setting was presented in [43] using a deep distributed recurrent Q-network (DDRQN) architecture. This approach takes inspiration from the single-agent DRQN algorithm [34] and generalizes it to a MA setting by making three fundamental modifications: Last-action inputs--giving each agent access to its previous action as an input for the next time-step; inter-agent weight sharing--a single network is learned causing a quicker learning process. Weight sharing allows diverse behavior between agents, as the agents receive different observations and thus evolve in different hidden states; disabling experience replay--this modification is made as the non-stationarity of the environment renders old experiences obsolete or misleading. As a proof of concept, this architecture was used for the solution of two multi-agent reinforcement learning problems based on the well known riddles of the hat riddle and the switch riddle. In these problems, the optimal solution can only be achieved if the agents reach a consensus on a communication protocol to use to solve the task. In Table 1, a summary of all the presented algorithm can be found with a brief explanation of its principal features and their scientific spreading. The latter was evaluated according to the number of search results from the Google Scholar indexing service. In Figure 4, a summary of all the algorithms with regard to the challenges they address is presented. Figure 4: Challenges encountered by each algorithm.