Learning to Ground Multi-Agent Communication with Autoencoders
Year: 2021

Abstract:
Communication requires having a common language, a lingua franca, between agents. This language could emerge via a consensus process, but it may require many generations of trial and error. Alternatively, the lingua franca can be given by the environment, where agents ground their language in representations of the observed world. We demonstrate a simple way to ground language in learned representations, which facilitates decentralized multi-agent communication and coordination. We find that a standard representation learning algorithm – autoencoding – is sufficient for arriving at a grounded common language. When agents broadcast these representations, they learn to understand and respond to each other’s utterances and achieve surprisingly strong task performance across a variety of multi-agent communication environments.

heading: 1 Introduction
text: An essential aspect of communication is that each pair of speaker and listener must share a common understanding of the symbols being used [8]. For artificial agents interacting in an environment, with a communication channel but without an agreed-upon communication protocol, this raises the question: how can meaningful communication emerge before a common language has been established?
To address this challenge, prior works have used supervised learning [18], centralized learning [14, 17, 28], or differentiable communication [6, 17, 28, 32, 41]. Yet, none of these mechanisms is representative of how communication emerges in nature, where animals and humans have evolved communication protocols without supervision and without a centralized coordinator [35]. The communication model that most closely resembles language learning in nature is a fully decentralized model, where agents’ policies are independently optimized. However, decentralized models perform poorly even in simple communication tasks [17] or with additional inductive biases [10].
We tackle this challenge by first making the following observations on why emergent communication is difficult in a decentralized multi-agent reinforcement learning setting. A key problem that prevents agents from learning meaningful communication is the lack of a common grounding in communication symbols [3, 10, 19]. In nature, the emergence of a common language is thought to be aided by physical biases and embodiment [29, 42] – we can only produce certain vocalizations, these sounds only can be heard a certain distance away, these sounds bear similarity to natural sounds in the environment, etc. – yet artificial communication protocols are not a priori grounded in aspects of the environment
Project page, code, and videos can be found at https://toruowo.github.io/marl-ae-comm/.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
dynamics. This poses a severe exploration problem as the chances of a consistent protocol being found and rewarded is extremely small [17]. Moreover, before a communication protocol is found, the random utterances transmitted between agents add to the already high variance of multi-agent reinforcement learning, making the learning problem even more challenging [10, 28].
To overcome the grounding problem, an important question to ask is: do agents really need to learn language grounding from scratch through random exploration in an environment where success is determined by chance? Perhaps nature has a different answer; previous studies in cognitive science and evolutionary linguistics [20, 36, 39, 40] have provided evidence for the hypothesis that communication first started from sounds whose meaning are grounded in the physical environment, then creatures adapted to make sense of those sounds and make use of them. Inspired by language learning in natural species, we propose a novel framework for grounding multi-agent communication: first ground speaking through learned representations of the world, then learn listening to interpret these grounded utterances. Surprisingly, even with the simple representation learning task of autoencoding, our approach eases the learning of communication in fully decentralized multi-agent settings and greatly improves agents’ performance in multi-agent coordination tasks that are nearly unsolvable without communication.
The contribution of our work can be summarized as follows:
• We formulate communication grounding as a representation learning problem and propose to use observation autoencoding to learn a common grounding across all agents.
• We experimentally validate that this is an effective approach for learning decentralized communication in MARL settings: a communication model trained with a simple autoencoder can consistently outperform baselines across various MARL environments.
• In turn, our work highlights the need to rethink the problem of emergent communication, where we demonstrate the essential need for visual grounding.

heading: 2 Related Work
text: In multi-agent reinforcement learning (MARL), achieving successful emergent communication with decentralized training and non-differentiable communication channel is an important yet challenging task that has not been satisfactorily addressed by existing works. Due to the non-stationary and nonMarkovian transition dynamics in multi-agent settings, straightforward implementation of standard reinforcement learning methods such as Actor-Critic [22] and DQN [31] perform poorly [17, 28].
Centralized learning is often used to alleviate the problem of high variance in MARL, for example learning a centralized value function that has access to the joint observation of all agents [9, 14, 28]. However, it turns out that MARL models are unable to solve tasks that rely on emergent communication, even with centralized learning and shared policy parameters across the agents [17]. Eccles et al. [10] provides an analysis that illustrates how MARL with communication poses a more difficult exploration problem than standard MARL, which is confirmed by empirical results in [17, 28]: communication exacerbates the sparse and high variance reward signal in MARL.
Many works therefore resort to differentiable communication [6, 17, 28, 32, 41], where agents are allowed to directly optimize each other’s communication policies through gradients. Among them, Choi el al. [6] explore a high-level idea that is similar to ours: generating messages that the model itself can interpret. However, these approaches impose a strong constraint on the nature of communication, which limits their applicability to many real-world multi-agent coordination tasks.
Jaques et al. [21] proposes a method that allows independently trained RL agents to communicate and coordinate. However, the proposed method requires that an agent either has access to policies of other agents or stays in close proximity to other agents. These constraints make it difficult for the same method to be applied to a wider range of tasks, such as those in which agents are not embodied or do not observe others directly. Eccles et al. [10] attempts to solve the same issue by introducing inductive biases for positive signaling and positive listening, but implementation requires numerous task-specific hyperparameter tuning, and the effectiveness is limited.
It is also worth noting that, while a large number of existing works on multi-agent communication take structured state information as input [4, 15, 16, 17, 32, 34], we train agents to learn a communication protocol directly from raw pixel observations. This presents additional challenges due to
the unstructured and ungrounded nature of pixel data, as shown in [3, 6, 24]. To our knowledge, this work is the first to effectively use representation learning to aid communication learning from pixel inputs in a wide range of MARL task settings.

heading: 3 Preliminaries
text: We model multi-agent reinforcement learning (MARL) with communication as a partiallyobservable general-sum Markov game [26, 38], where each agent can broadcast information to a shared communication channel. Each agent receives a partial observation of the underlying world state at every time step, including all information communicated in the shared channel. This observation is used to learn an appropriate policy that maximizes the agent’s environment reward. In this work, we parameterize the policy function using a deep neural network.
Formally, a decentralized MARL can be expressed as a partially observable Markov decision process as M = ⟨S,A, C,O, T , R, γ⟩, where N is the number of agents, S is a set of states spaces, A = {A1, ...,AN}, C = {C1, ..., CN}, and O = {O1, ...,ON} are a set of action, of communication, and of observation spaces respectively.
At time step t, an agent k observes a partial view o(k)t of the underlying true state st , and a set of communicated messages from the previous time step ct−1 = {c(1)t−1, ..., c (N) t−1}. The agent then chooses an action a(k)t ∈ Ak and a subsequent message to broadcast c(k)t ∈ Ck. Given the joint actions of all N agents at = {a(1)t , ..., a(N)t } ∈ (A1, ...,AN ), the transition function T : S×A1× ...×AN → ∆(S) maps the current state st and set of agent actions at to a distribution over the next state st+1. Since the transition function T is non-determinstic, we denote the probability distributions over S as ∆(S). Finally, each agent receives an individual reward r(k)t ∈ R(st, at) where R : S×A1× ...×AN → R. In our work, we consider a fully cooperative setting in which the objective of each agent is to maximize the total expected return of all agents:
maximize π:S→A×C
E [ ∑
t∈T
∑ k∈N γtR(st, at) ∣∣∣ (at, ct) ∼ π(k), st ∼ T (st−1) ]
(1)
for some finite time horizon T and discount factor γ.
In MARL, the aforementioned objective function is optimized using policy gradient. Specifically, we use asynchronous advantage actor-critic (A3C) [30] with Generalized Advantage Estimation [37] to optimize our policy network. The policy network in A3C outputs a distribution over the actions and the discounted future returns. Any other policy optimization algorithms could have been used in lieu of A3C. We do not use centralized training or self-play, and only consider decentralized training where each agent is parameterized by an independent policy.

heading: 4 Grounding Representation for Communication with Autoencoders
text: The main challenge of learning to communicate in fully decentralized MARL settings is that there is no grounded information to which agents can associate their symbolic utterances. This lack of grounding creates a dissonance across agents and poses a difficult exploration problem. Ultimately, the gradient signals received by agents are therefore largely inconsistent. As the time horizon, communication space, and the number of agents grow, this grounding problem becomes even more pronounced. This difficulty is highlighted in numerous prior works, with empirical results showing that agents often fail to use the communication channel at all during decentralized learning [3, 10, 19, 28].
We propose a simple and surprisingly effective approach to mitigate this issue: using a self-supervised representation learning task to learn a common grounding [43] across all agents. Having such a grounding enables speakers to communicate messages that are understandable to the listeners and convey meaningful information about entities in the environment, though agents need not use the same symbols to mean the same things. Specifically, we train each agent to independently learn to auto-encode its own observation and use the learned representation for communication. This approach offers the benefits of allowing fully decentralized training without needing additional architectural
image encoder
message encoder
(st)
Policy Gradient Communication
Grounding
agent 1
a(2)tc (2) t c (n) t a (n) ta (1) tc (1) t
agent 2 agent n
c(n)t 1c (1) t 1 c (2) t 1
Communicate
s(1)t s (2) t s (n) t
s(n)t c (1) t 1 . . . c (n) t 1
(ct 1)
a(n)tc (n) t
(n) (2) (1)
Figure 1: Overview: The overall schematic of our multi-agent system. All agents share the same individual model architecture, but each agent is independently trained to learn to auto-encode its own observation and use the learned representation for communication. At each time step, each agent observes an image representation of the environment as well as messages broadcasted by other agents during the last time step. The image pixels are processed through an Image Encoder; the broadcasted messages are processed through a Message Encoder; the image features and the message features are concatenated and passed through a Policy Network to predict the next action. The image features are also used to generate the next communication messages using the Communication Autoencoder.
bias or supervision signal. In Section 5, we show the effectiveness of our approach on a variety of MARL communication tasks.

heading: An overview of our method is shown in Figure 1, which illustrates the communication flow among agents at some arbitrary time step t. All agents share the same individual model architecture, and
text: each agent consists of two modules: a speaker module and a listener module. We describe the architecture details of a single agent k below.
4.1 Speaker Module
At each time step t, the speaker module takes in the agent’s observation o(k)t and outputs the agent’s next communication message c(k)t .
Image Encoder Given the raw pixel observation, the module first uses a image encoder to embed the pixels into a low-dimensional feature o(k)t ! im(o(k)t ) 2 R128. The image encoder is a convolutional neural network with 4 convolutional layers, and the output of this network is spatially pooled. We use the same image encoder in the listener module.
Communication Autoencoder The goal of the communication autoencoder is to take the current state observation and generate the next subsequent message. We use an autoencoder to learn a mapping from the feature space of image encoder to communication symbols, i.e. im(o (k) t ) ! c(k)t . The autoencoder consists of an encoder and a decoder, both parameterized by a 3-layer MLP. The decoder tries to reconstruct the input state from the communication message c(k)t ! ̂im(o(k)t ). The communication messages are quantized before being passed through the decoder. We use a straightthrough estimator to differentiate through the quantization [1]. The auxiliary objective function of our model is to minimize the reconstruction loss k im(o(k)t ) ̂im(o(k)t )k22. This loss is optimized jointly with the policy gradient loss from the listener module.
4.2 Listener Module
While the goal of the speaker module is to output grounded communication based on the agent’s private observation o(k)t , the goal of the listener module is to learn an optimal action policy based on both the observation o(k)t and communicated messages ct 1. At each time step t, the listener module outputs the agent’s next action a(k)t .
im(o (k) t )
<latexit sha1_base64="9Fgn9kpxREwS/dVOKmp4IDKcEoI=">AAACiHicfZFNb9QwEIa94auEry0ce4lYIW0RWiUIqe2tKhy4IIrEbiutQ+R4J1lr/RHZE+jKyoFfwxV+Dv8GZ7tI0CJGsvT4nXdsz7hspHCYpj8H0Y2bt27f2bkb37v/4OGj4e7jmTOt5TDlRhp7XjIHUmiYokAJ540FpkoJZ+XqdZ8/+wzWCaM/4rqBXLFai0pwhkEqhnu0WYrCU4QL9EJ13dgU+MmPV/vdfjEcpZN0E8l1yLYwIts4LXYHU7owvFWgkUvm3DxLG8w9syi4hC6mrYOG8RWrYR5QMwUu95suuuRZUBZJZWxYGpON+meFZ8q5tSqDUzFcuqu5XvxXbt5idZh7oZsWQfPLi6pWJmiSfiTJQljgKNcBGLcivDXhS2YZxzC4OKZvIDRj4V04+H0DlqGxzz1ltlbsogvN1fRFT/8zCv3bGCimGr5woxTTC0+1saqbZ7mnEiqkcgYWRxm1ol4itf2uCx+RXR37dZi9nGTpJPvwanR8sv2SHbJHnpIxycgBOSZvySmZEk6+km/kO/kRxVEaHURHl9ZosK15Qv6K6OQX3AnIXA==</latexit><latexit sha1_base64="ja7fR1eH9RhdwkzGaNhaf4QiBes=">AAACrXicjVFdaxQxFM2OX3X86FYf+zK4CFuRdUYE+1jUB19EBWdb2EyHTPbubNh8DMmd2iXMg7/GV3+N4L8xs11BWxEvBE7OPTfJOakaKRym6Y9BdO36jZu3dm7Hd+7eu7873Hswdaa1HHJupLEnFXMghYYcBUo4aSwwVUk4rlav+/7xGVgnjP6E6wYKxWotFoIzDFQ53KfNUpSeIpyjF6rrxqbEUz9eHXQH5XCUTtJNJVdBtgUjsq3/k5d7g5zODW8VaOSSOTfL0gYLzywKLqGLaeugYXzFapgFqJkCV/iN1S55HJh5sjA2LI3Jhv19wjPl3FpVQakYLt3lXk/+rTdrcXFYeKGbFkHzi4sWrUzQJH1uyVxY4CjXATBuRXhrwpfMMo4h3TimbyCYsfAuHPy+AcvQ2CeeMlsrdt4FczV92qN/CYX+JQwopho+c6MU03NPtbGqm2WFpxIWSOUULI4yakW9RGr7XRfizy6HfRVMn0+ydJJ9fDE6erX9tx2yTx6RMcnIS3JE3pIPJCecfCFfyTfyPXoW5RGNTi+k0WA785D8UVH9EyeZz9U=</latexit><latexit sha1_base64="ja7fR1eH9RhdwkzGaNhaf4QiBes=">AAACrXicjVFdaxQxFM2OX3X86FYf+zK4CFuRdUYE+1jUB19EBWdb2EyHTPbubNh8DMmd2iXMg7/GV3+N4L8xs11BWxEvBE7OPTfJOakaKRym6Y9BdO36jZu3dm7Hd+7eu7873Hswdaa1HHJupLEnFXMghYYcBUo4aSwwVUk4rlav+/7xGVgnjP6E6wYKxWotFoIzDFQ53KfNUpSeIpyjF6rrxqbEUz9eHXQH5XCUTtJNJVdBtgUjsq3/k5d7g5zODW8VaOSSOTfL0gYLzywKLqGLaeugYXzFapgFqJkCV/iN1S55HJh5sjA2LI3Jhv19wjPl3FpVQakYLt3lXk/+rTdrcXFYeKGbFkHzi4sWrUzQJH1uyVxY4CjXATBuRXhrwpfMMo4h3TimbyCYsfAuHPy+AcvQ2CeeMlsrdt4FczV92qN/CYX+JQwopho+c6MU03NPtbGqm2WFpxIWSOUULI4yakW9RGr7XRfizy6HfRVMn0+ydJJ9fDE6erX9tx2yTx6RMcnIS3JE3pIPJCecfCFfyTfyPXoW5RGNTi+k0WA785D8UVH9EyeZz9U=</latexit><latexit sha1_base64="ja7fR1eH9RhdwkzGaNhaf4QiBes=">AAACrXicjVFdaxQxFM2OX3X86FYf+zK4CFuRdUYE+1jUB19EBWdb2EyHTPbubNh8DMmd2iXMg7/GV3+N4L8xs11BWxEvBE7OPTfJOakaKRym6Y9BdO36jZu3dm7Hd+7eu7873Hswdaa1HHJupLEnFXMghYYcBUo4aSwwVUk4rlav+/7xGVgnjP6E6wYKxWotFoIzDFQ53KfNUpSeIpyjF6rrxqbEUz9eHXQH5XCUTtJNJVdBtgUjsq3/k5d7g5zODW8VaOSSOTfL0gYLzywKLqGLaeugYXzFapgFqJkCV/iN1S55HJh5sjA2LI3Jhv19wjPl3FpVQakYLt3lXk/+rTdrcXFYeKGbFkHzi4sWrUzQJH1uyVxY4CjXATBuRXhrwpfMMo4h3TimbyCYsfAuHPy+AcvQ2CeeMlsrdt4FczV92qN/CYX+JQwopho+c6MU03NPtbGqm2WFpxIWSOUULI4yakW9RGr7XRfizy6HfRVMn0+ydJJ9fDE6erX9tx2yTx6RMcnIS3JE3pIPJCecfCFfyTfyPXoW5RGNTi+k0WA785D8UVH9EyeZz9U=</latexit>
̂im(o (k) t ) <latexit sha1_base64="YU7rtaRhZBJ8J9MdzDeoPBYZGoc=">AAACjnicfZFNj9MwEIbd8LWEj+3CkUugQuoiVCUIARfECjjsBbFItLtSHSLXnSRW/RHZE9jKyplfwxV+C/8Gp1sk2EWMZOnxO+/YnvGikcJhmv4cRJcuX7l6bed6fOPmrdu7w707M2day2HKjTT2ZMEcSKFhigIlnDQWmFpIOF6s3vT5489gnTD6I64byBWrtCgFZxikYnif1gw9bWrRFZ4inKIXquvGpsBPfrza7/aL4SidpJtILkK2hRHZxlGxN5jSpeGtAo1cMufmWdpg7plFwSV0MW0dNIyvWAXzgJopcLnf9NIlD4OyTEpjw9KYbNQ/KzxTzq3VIjgVw9qdz/Xiv3LzFssXuRe6aRE0P7uobGWCJukHkyyFBY5yHYBxK8JbE14zyziG8cUxfQuhGQvvwsHvG7AMjX3kKbOVYqddaK6ij3v6n1Ho38ZAMdXwhRulmF56qo1V3TzLPZVQIpUzsDjKqBVVjdT2uy58RHZ+7Bdh9mSSpZPsw9PRwevtl+yQe+QBGZOMPCcH5JAckSnh5Cv5Rr6TH9Eweha9jF6dWaPBtuYu+Suiw19AYMsp</latexit><latexit sha1_base64="oCAjeJcB3MMOAajffD25/gExfpo=">AAACs3icjVFdi9QwFM3Ur7V+zeqjL9VBmBUZWlH0cVEffBEVnNmFyVjS9LYNk4+S3OoOoc/+Gl/9L4I/xnR2BN0V8ULg5Nxzk5yTopXCYZp+H0UXLl66fGXvanzt+o2bt8b7txfOdJbDnBtp7HHBHEihYY4CJRy3FpgqJBwV65dD/+gTWCeM/oCbFlaK1VpUgjMMVD6+RxuGnraN6HNPEU7QC9X3U5PjRz9dH/QH+XiSztJtJedBtgMTsqv/k+f7ozktDe8UaOSSObfM0hZXnlkUXEIf085By/ia1bAMUDMFbuW3hvvkQWDKpDI2LI3Jlv19wjPl3EYVQakYNu5sbyD/1lt2WD1feaHbDkHz04uqTiZokiG9pBQWOMpNAIxbEd6a8IZZxjFkHMf0FQQzFt6Eg9+2YBka+9BTZmvFTvpgrqaPBvQvodC/hAHFVMNnbpRiuvRUG6v6ZbbyVEKFVC7A4iSjVtQNUjvs+hB/djbs82DxeJals+z9k8nhi92/7ZG75D6Zkow8I4fkNXlH5oSTL+Qr+UZ+RE+jZVRE5ak0Gu1m7pA/KlI/Aeu00qI=</latexit><latexit sha1_base64="oCAjeJcB3MMOAajffD25/gExfpo=">AAACs3icjVFdi9QwFM3Ur7V+zeqjL9VBmBUZWlH0cVEffBEVnNmFyVjS9LYNk4+S3OoOoc/+Gl/9L4I/xnR2BN0V8ULg5Nxzk5yTopXCYZp+H0UXLl66fGXvanzt+o2bt8b7txfOdJbDnBtp7HHBHEihYY4CJRy3FpgqJBwV65dD/+gTWCeM/oCbFlaK1VpUgjMMVD6+RxuGnraN6HNPEU7QC9X3U5PjRz9dH/QH+XiSztJtJedBtgMTsqv/k+f7ozktDe8UaOSSObfM0hZXnlkUXEIf085By/ia1bAMUDMFbuW3hvvkQWDKpDI2LI3Jlv19wjPl3EYVQakYNu5sbyD/1lt2WD1feaHbDkHz04uqTiZokiG9pBQWOMpNAIxbEd6a8IZZxjFkHMf0FQQzFt6Eg9+2YBka+9BTZmvFTvpgrqaPBvQvodC/hAHFVMNnbpRiuvRUG6v6ZbbyVEKFVC7A4iSjVtQNUjvs+hB/djbs82DxeJals+z9k8nhi92/7ZG75D6Zkow8I4fkNXlH5oSTL+Qr+UZ+RE+jZVRE5ak0Gu1m7pA/KlI/Aeu00qI=</latexit><latexit sha1_base64="oCAjeJcB3MMOAajffD25/gExfpo=">AAACs3icjVFdi9QwFM3Ur7V+zeqjL9VBmBUZWlH0cVEffBEVnNmFyVjS9LYNk4+S3OoOoc/+Gl/9L4I/xnR2BN0V8ULg5Nxzk5yTopXCYZp+H0UXLl66fGXvanzt+o2bt8b7txfOdJbDnBtp7HHBHEihYY4CJRy3FpgqJBwV65dD/+gTWCeM/oCbFlaK1VpUgjMMVD6+RxuGnraN6HNPEU7QC9X3U5PjRz9dH/QH+XiSztJtJedBtgMTsqv/k+f7ozktDe8UaOSSObfM0hZXnlkUXEIf085By/ia1bAMUDMFbuW3hvvkQWDKpDI2LI3Jlv19wjPl3EYVQakYNu5sbyD/1lt2WD1feaHbDkHz04uqTiZokiG9pBQWOMpNAIxbEd6a8IZZxjFkHMf0FQQzFt6Eg9+2YBka+9BTZmvFTvpgrqaPBvQvodC/hAHFVMNnbpRiuvRUG6v6ZbbyVEKFVC7A4iSjVtQNUjvs+hB/djbs82DxeJals+z9k8nhi92/7ZG75D6Zkow8I4fkNXlH5oSTL+Qr+UZ+RE+jZVRE5ak0Gu1m7pA/KlI/Aeu00qI=</latexit>
c (n) t <latexit sha1_base64="f2ZMuhTMtn5OumvZJgwDxGEweks=">AAACdHicfVFba9RAFJ6NtxovbfVRH4KhUEWWRAT7WNQHX8QK7rayE5eT2ZPs0LmEmZPqEvIrfNUf5h/x2cl2BW3FDwa++c53Zs6lbJT0lGU/RtGVq9eu39i6Gd+6fefu9s7uvam3rRM4EVZZd1KCRyUNTkiSwpPGIehS4XF5+mqIH5+h89KaD7RqsNBQG1lJARSkj2JOn7p987if76TZOFsjuUzyDUnZBkfz3dGEL6xoNRoSCryf5VlDRQeOpFDYx7z12IA4hRpngRrQ6ItuXXGf7AVlkVTWhWMoWat/ZnSgvV/pMjg10NJfjA3iv2KzlqqDopOmaQmNOP+oalVCNhnaTxbSoSC1CgSEk6HWRCzBgaAwpDjmrzE04/BtePhdgw7IuicdB1dr+NKH5mr+dGD/M0rz2xhYzA1+FlZrMIuOG+t0P8uLjiusiKspOkpz7mS9JO6G27CI/OLYL5Pps3GejfP3z9PDl5uVbLEH7BHbZzl7wQ7ZG3bEJkwwzb6yb+z76Gf0MEqjvXNrNNrk3Gd/IRr/Ap5ZwTk=</latexit><latexit sha1_base64="qkn0q/PWLShigmcCsY7WLZoS9XU=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi5jR13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00+wDcez</latexit><latexit sha1_base64="qkn0q/PWLShigmcCsY7WLZoS9XU=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi5jR13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00+wDcez</latexit><latexit sha1_base64="qkn0q/PWLShigmcCsY7WLZoS9XU=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi5jR13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00+wDcez</latexit>
c (1) t 1 . . . c (n) t 1<latexit sha1_base64="Q/tTE9J/tIm2cTdaSKpfQ+8KN9M=">AAACj3icfZFNj9MwEIbd8LWEj23hyMXaCmkXQZUgpOWEKuAAB8Qi0e5KdagcZ5Ja64/IngBVlDu/hiv8Ff4NTrcI2EWMZOnxO+/YnnFeK+kxSX4MokuXr1y9tnM9vnHz1u3d4ejO3NvGCZgJq6w7ybkHJQ3MUKKCk9oB17mC4/z0RZ8//gjOS2ve47qGTPPKyFIKjkFaDvfEssVHafeh3U8POspUYdHT36I56JbDcTJJNkEvQrqFMdnG0XI0mLHCikaDQaG494s0qTFruUMpFHQxazzUXJzyChYBDdfgs3bTTEfvB6WgpXVhGaQb9c+Klmvv1zoPTs1x5c/nevFfuUWD5dOslaZuEIw4u6hsFEVL+8nQQjoQqNYBuHAyvJWKFXdcYJhfHLOXEJpx8CYc/LYGx9G6By3jrtL8cxeaq9jDnv5nlOaXMVDMDHwSVmtuipYZ63S3SLOWKSiRqTk4HKfMyWqFzPW7/iPS82O/CPPHkzSZpO+ejKfPt1+yQ+6RPbJPUnJIpuQVOSIzIsgX8pV8I9+jUXQYPYumZ9ZosK25S/6K6PVPng/J4Q==</latexit><latexit sha1_base64="pkZePaAaz45n815LYapDmQmMGN8=">AAACtHicjZFNbxMxEIad5aNl+WgKRy5WI6QWQbRGqHCsgAMXBEgkrRQvkdc72Vj1x8qeBaLV3vk1XPktHPgveNMgoEWIkSw9fucd2zMuaq0CZtm3QXLp8pWrW9vX0us3bt7aGe7engbXeAkT6bTzJ4UIoJWFCSrUcFJ7EKbQcFycPu/zxx/AB+XsO1zVkBtRWbVQUmCU5sM9OW/xIevet/vsoKNclw4D/SXag24+HGXjbB30IrANjMgm/s8+3x1MeOlkY8Ci1CKEGctqzFvhUUkNXcqbALWQp6KCWUQrDIS8XXfc0XtRKenC+bgs0rX6e0UrTAgrU0SnEbgM53O9+LfcrMHF07xVtm4QrDy7aNFoio7246Ol8iBRryII6VV8K5VL4YXEOOQ05S8gNuPhVTz4dQ1eoPP3Wy58ZcSnLjZX8Qc9/cuo7E9jpJRb+CidMcKWLbfOm27G8pZrWCDXU/A4Ytyraonc97v+t9j5YV+E6aMxy8bs7ePR0bPNv22Tu2SP7BNGnpAj8pK8IRMiyWfyhXwl35PDhCcygTNrMtjU3CF/RGJ/ABi90Vo=</latexit><latexit sha1_base64="pkZePaAaz45n815LYapDmQmMGN8=">AAACtHicjZFNbxMxEIad5aNl+WgKRy5WI6QWQbRGqHCsgAMXBEgkrRQvkdc72Vj1x8qeBaLV3vk1XPktHPgveNMgoEWIkSw9fucd2zMuaq0CZtm3QXLp8pWrW9vX0us3bt7aGe7engbXeAkT6bTzJ4UIoJWFCSrUcFJ7EKbQcFycPu/zxx/AB+XsO1zVkBtRWbVQUmCU5sM9OW/xIevet/vsoKNclw4D/SXag24+HGXjbB30IrANjMgm/s8+3x1MeOlkY8Ci1CKEGctqzFvhUUkNXcqbALWQp6KCWUQrDIS8XXfc0XtRKenC+bgs0rX6e0UrTAgrU0SnEbgM53O9+LfcrMHF07xVtm4QrDy7aNFoio7246Ol8iBRryII6VV8K5VL4YXEOOQ05S8gNuPhVTz4dQ1eoPP3Wy58ZcSnLjZX8Qc9/cuo7E9jpJRb+CidMcKWLbfOm27G8pZrWCDXU/A4Ytyraonc97v+t9j5YV+E6aMxy8bs7ePR0bPNv22Tu2SP7BNGnpAj8pK8IRMiyWfyhXwl35PDhCcygTNrMtjU3CF/RGJ/ABi90Vo=</latexit><latexit sha1_base64="pkZePaAaz45n815LYapDmQmMGN8=">AAACtHicjZFNbxMxEIad5aNl+WgKRy5WI6QWQbRGqHCsgAMXBEgkrRQvkdc72Vj1x8qeBaLV3vk1XPktHPgveNMgoEWIkSw9fucd2zMuaq0CZtm3QXLp8pWrW9vX0us3bt7aGe7engbXeAkT6bTzJ4UIoJWFCSrUcFJ7EKbQcFycPu/zxx/AB+XsO1zVkBtRWbVQUmCU5sM9OW/xIevet/vsoKNclw4D/SXag24+HGXjbB30IrANjMgm/s8+3x1MeOlkY8Ci1CKEGctqzFvhUUkNXcqbALWQp6KCWUQrDIS8XXfc0XtRKenC+bgs0rX6e0UrTAgrU0SnEbgM53O9+LfcrMHF07xVtm4QrDy7aNFoio7246Ol8iBRryII6VV8K5VL4YXEOOQ05S8gNuPhVTz4dQ1eoPP3Wy58ZcSnLjZX8Qc9/cuo7E9jpJRb+CidMcKWLbfOm27G8pZrWCDXU/A4Ytyraonc97v+t9j5YV+E6aMxy8bs7ePR0bPNv22Tu2SP7BNGnpAj8pK8IRMiyWfyhXwl35PDhCcygTNrMtjU3CF/RGJ/ABi90Vo=</latexit>
a (n) t <latexit sha1_base64="PkqRI09LOBn4opjQsmChvRaZY6E=">AAACdHicfVFba9RAFJ6NtxovbfVRH4KhUEWWRAT7WNQHX8QK7rayE5eT2ZPs0LmEmZPqEvIrfNUf5h/x2cl2BW3FDwa++c53Zs6lbJT0lGU/RtGVq9eu39i6Gd+6fefu9s7uvam3rRM4EVZZd1KCRyUNTkiSwpPGIehS4XF5+mqIH5+h89KaD7RqsNBQG1lJARSkjzCnT92+edzPd9JsnK2RXCb5hqRsg6P57mjCF1a0Gg0JBd7P8qyhogNHUijsY956bECcQo2zQA1o9EW3rrhP9oKySCrrwjGUrNU/MzrQ3q90GZwaaOkvxgbxX7FZS9VB0UnTtIRGnH9UtSohmwztJwvpUJBaBQLCyVBrIpbgQFAYUhzz1xiacfg2PPyuQQdk3ZOOg6s1fOlDczV/OrD/GaX5bQws5gY/C6s1mEXHjXW6n+VFxxVWxNUUHaU5d7JeEnfDbVhEfnHsl8n02TjPxvn75+nhy81KttgD9ojts5y9YIfsDTtiEyaYZl/ZN/Z99DN6GKXR3rk1Gm1y7rO/EI1/AZo5wTc=</latexit><latexit sha1_base64="C7NwhuG/vbnOL4fWRBpVnGZEWno=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpC8zoa7tjHnezzWE6SleRXIZsDUO2jv+zz7YGYz63otFoSCjwfpqlNeUtOJJCYRfzxmMN4ggqnAY0oNHn7aqtLnkUlHlSWheWoWSl/l7RgvZ+qYvg1EALfzHXi3/LTRsqX+StNHVDaMT5RWWjErJJP6NkLh0KUssAIJwMb03EAhwICpOMY/4GQzMO34WD39fogKx70nJwlYaTLjRX8ac9/csozS9joJgbPBZWazDzlhvrdDfN8pYrLImrCToaZtzJakHc9bv+t7KLw74Mk91Rlo6yj8+Ge6/W/7bBttlDtsMy9pztsX32gY2ZYJp9Z6fsLNqOXkb70dtzazRY1zxgf0T06Sero8ex</latexit><latexit sha1_base64="C7NwhuG/vbnOL4fWRBpVnGZEWno=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpC8zoa7tjHnezzWE6SleRXIZsDUO2jv+zz7YGYz63otFoSCjwfpqlNeUtOJJCYRfzxmMN4ggqnAY0oNHn7aqtLnkUlHlSWheWoWSl/l7RgvZ+qYvg1EALfzHXi3/LTRsqX+StNHVDaMT5RWWjErJJP6NkLh0KUssAIJwMb03EAhwICpOMY/4GQzMO34WD39fogKx70nJwlYaTLjRX8ac9/csozS9joJgbPBZWazDzlhvrdDfN8pYrLImrCToaZtzJakHc9bv+t7KLw74Mk91Rlo6yj8+Ge6/W/7bBttlDtsMy9pztsX32gY2ZYJp9Z6fsLNqOXkb70dtzazRY1zxgf0T06Sero8ex</latexit><latexit sha1_base64="C7NwhuG/vbnOL4fWRBpVnGZEWno=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpC8zoa7tjHnezzWE6SleRXIZsDUO2jv+zz7YGYz63otFoSCjwfpqlNeUtOJJCYRfzxmMN4ggqnAY0oNHn7aqtLnkUlHlSWheWoWSl/l7RgvZ+qYvg1EALfzHXi3/LTRsqX+StNHVDaMT5RWWjErJJP6NkLh0KUssAIJwMb03EAhwICpOMY/4GQzMO34WD39fogKx70nJwlYaTLjRX8ac9/csozS9joJgbPBZWazDzlhvrdDfN8pYrLImrCToaZtzJakHc9bv+t7KLw74Mk91Rlo6yj8+Ge6/W/7bBttlDtsMy9pztsX32gY2ZYJp9Z6fsLNqOXkb70dtzazRY1zxgf0T06Sero8ex</latexit>
comm(ct 1) <latexit sha1_base64="l6oZ+6b35mQmZK+c0MpS99YgvNY=">AAACiHicfVFNb9QwEPWGj5bw0W059hKxQioIVglCanurCgcuVYvEbiuto8jxTrJW/RHZE+jKyoFfwxV+Dv8GZ7tI0CJGsvT85s3Y86ZspHCYpj8H0Z279+5vbD6IHz56/GRruL0zdaa1HCbcSGMvSuZACg0TFCjhorHAVCnhvLx81+fPP4N1wuhPuGwgV6zWohKcYaCK4S5tFqLwFOEKPTdKdd0eLzy+zroXxXCUjtNVJLdBtgYjso6zYnswoXPDWwUauWTOzbK0wdwzi4JL6GLaOmgYv2Q1zALUTIHL/WqKLnkemHlSGRuOxmTF/lnhmXJuqcqgVAwX7mauJ/+Vm7VYHeRe6KZF0Pz6oaqVCZqktySZCwsc5TIAxq0If034glnGMRgXx/Q9hGEsnITGpw1Yhsa+9JTZWrGrLgxX01c9+p9Q6N/CgGKq4UvvNdNzT7WxqptluacSKqRyChZHGbWiXiC1/a0Li8hu2n4bTN+Ms3ScfXw7Ojper2ST7JJnZI9kZJ8ckQ/kjEwIJ1/JN/Kd/IjiKI32o8NraTRY1zwlf0V0/Av4/Mhq</latexit><latexit sha1_base64="gvKkV4MWCSccXOJw/5amjm5XJRU=">AAACrXicjVFNb9QwEPWGj5bwtYVjLxErpIJgSSokOFaUAxcESGRbaR0ixzvJWvVHZE+gKyuH/hqu/Bok/g3OdpGgRYiRLD2/eTP2vKlaKRym6Y9RdOXqtetb2zfim7du37k73rk3c6azHHJupLHHFXMghYYcBUo4bi0wVUk4qk4Oh/zRZ7BOGP0RVy0UijVa1IIzDFQ53qXtUpSeIpyi50apvt/jpcenWf+oHE/SabqO5DLINmBCNvF/8nJnlNOF4Z0CjVwy5+ZZ2mLhmUXBJfQx7Ry0jJ+wBuYBaqbAFX49ap88DMwiqY0NR2OyZn+v8Ew5t1JVUCqGS3cxN5B/y807rF8WXui2Q9D8/KG6kwmaZPAtWQgLHOUqAMatCH9N+JJZxjG4G8f0NYRhLLwNjd+1YBka+9hTZhvFTvswXEOfDOhfQqF/CQOKqYYvw0KYXniqjVX9PCs8lVAjlTOwOMmoFc0SqR1ufbA/u2j2ZTDbn2bpNPvwfHLwarO3bbJLHpA9kpEX5IC8Ie9JTjg5I1/JN/I9ehblEY0+nUuj0abmPvkjouYnRpLP4w==</latexit><latexit sha1_base64="gvKkV4MWCSccXOJw/5amjm5XJRU=">AAACrXicjVFNb9QwEPWGj5bwtYVjLxErpIJgSSokOFaUAxcESGRbaR0ixzvJWvVHZE+gKyuH/hqu/Bok/g3OdpGgRYiRLD2/eTP2vKlaKRym6Y9RdOXqtetb2zfim7du37k73rk3c6azHHJupLHHFXMghYYcBUo4bi0wVUk4qk4Oh/zRZ7BOGP0RVy0UijVa1IIzDFQ53qXtUpSeIpyi50apvt/jpcenWf+oHE/SabqO5DLINmBCNvF/8nJnlNOF4Z0CjVwy5+ZZ2mLhmUXBJfQx7Ry0jJ+wBuYBaqbAFX49ap88DMwiqY0NR2OyZn+v8Ew5t1JVUCqGS3cxN5B/y807rF8WXui2Q9D8/KG6kwmaZPAtWQgLHOUqAMatCH9N+JJZxjG4G8f0NYRhLLwNjd+1YBka+9hTZhvFTvswXEOfDOhfQqF/CQOKqYYvw0KYXniqjVX9PCs8lVAjlTOwOMmoFc0SqR1ufbA/u2j2ZTDbn2bpNPvwfHLwarO3bbJLHpA9kpEX5IC8Ie9JTjg5I1/JN/I9ehblEY0+nUuj0abmPvkjouYnRpLP4w==</latexit><latexit sha1_base64="gvKkV4MWCSccXOJw/5amjm5XJRU=">AAACrXicjVFNb9QwEPWGj5bwtYVjLxErpIJgSSokOFaUAxcESGRbaR0ixzvJWvVHZE+gKyuH/hqu/Bok/g3OdpGgRYiRLD2/eTP2vKlaKRym6Y9RdOXqtetb2zfim7du37k73rk3c6azHHJupLHHFXMghYYcBUo4bi0wVUk4qk4Oh/zRZ7BOGP0RVy0UijVa1IIzDFQ53qXtUpSeIpyi50apvt/jpcenWf+oHE/SabqO5DLINmBCNvF/8nJnlNOF4Z0CjVwy5+ZZ2mLhmUXBJfQx7Ry0jJ+wBuYBaqbAFX49ap88DMwiqY0NR2OyZn+v8Ew5t1JVUCqGS3cxN5B/y807rF8WXui2Q9D8/KG6kwmaZPAtWQgLHOUqAMatCH9N+JJZxjG4G8f0NYRhLLwNjd+1YBka+9hTZhvFTvswXEOfDOhfQqF/CQOKqYYvw0KYXniqjVX9PCs8lVAjlTOwOMmoFc0SqR1ufbA/u2j2ZTDbn2bpNPvwfHLwarO3bbJLHpA9kpEX5IC8Ie9JTjg5I1/JN/I9ehblEY0+nUuj0abmPvkjouYnRpLP4w==</latexit>
Policy Gradient
C o
m m
u n ic
a tio
n
A u to
e n c o
d e r
o (n) t <latexit sha1_base64="AijpuD5XdKwqeARXIV+woEF8hQM=">AAACdHicfVFba9RAFJ6NtxovbfVRH4KhUEWWRAT7WNQHX8QK7rayE5eT2ZPs0LmEmZPqEvIrfNUf5h/x2cl2BW3FDwa++c53Zs6lbJT0lGU/RtGVq9eu39i6Gd+6fefu9s7uvam3rRM4EVZZd1KCRyUNTkiSwpPGIehS4XF5+mqIH5+h89KaD7RqsNBQG1lJARSkj3ZOn7p987if76TZOFsjuUzyDUnZBkfz3dGEL6xoNRoSCryf5VlDRQeOpFDYx7z12IA4hRpngRrQ6ItuXXGf7AVlkVTWhWMoWat/ZnSgvV/pMjg10NJfjA3iv2KzlqqDopOmaQmNOP+oalVCNhnaTxbSoSC1CgSEk6HWRCzBgaAwpDjmrzE04/BtePhdgw7IuicdB1dr+NKH5mr+dGD/M0rz2xhYzA1+FlZrMIuOG+t0P8uLjiusiKspOkpz7mS9JO6G27CI/OLYL5Pps3GejfP3z9PDl5uVbLEH7BHbZzl7wQ7ZG3bEJkwwzb6yb+z76Gf0MEqjvXNrNNrk3Gd/IRr/ArcZwUU=</latexit><latexit sha1_base64="7We+Wtw1IUCjthukT8N7Q+XrA+c=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00/Kice/</latexit><latexit sha1_base64="7We+Wtw1IUCjthukT8N7Q+XrA+c=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00/Kice/</latexit><latexit sha1_base64="7We+Wtw1IUCjthukT8N7Q+XrA+c=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00/Kice/</latexit>
image encoder
message encoder
(
<latexit sha1_base64="SzgOjECc3/G/NjwHBGjVHycPgtA=">AAACcnicfZFba9RAFMdn46U13lr7pi/RRRApSyIFfSzVB1/ECu5uYSeUk9mT7LBzCTMn6hLyIXzVT9bv4QdwskbQVjww8Jv/OWfmXIpaSU9pejGKrl2/cXNn91Z8+87de/f39h/MvG2cwKmwyrqzAjwqaXBKkhSe1Q5BFwrnxfp1759/QuelNR9pU2OuoTKylAIoSHN+IquKt+d743SSbi25CtkAYzbY6fn+aMqXVjQaDQkF3i+ytKa8BUdSKOxi3nisQayhwkVAAxp93m7r7ZKnQVkmpXXhGEq26p8ZLWjvN7oIkRpo5S/7evFfvkVD5au8laZuCI349VHZqIRs0jefLKVDQWoTAISTodZErMCBoDCiOOZvMDTj8F14+H2NDsi65y0HV2n40oXmKn7Y0/8CpfkdGCjmBj8LqzWYZcuNdbpbZHnLFZbE1QwdjTPuZLUi7vpbFxaRXR77VZi9mGTpJPtwND4+GVayyx6xJ+wZy9hLdszeslM2ZYKt2Vf2jX0f/YgeRo+jYX/RaMg5YH9ZdPgTCjXAhg==</latexit><latexit sha1_base64="bPDTkEMP9FXbnEuQC8fXG0O8QIc=">AAACl3icjVFdaxQxFM2OH63jV6tP4svgIojIMiOF9s1SRfoiWnB3C5uh3MnemQ2bjyG5Y7sM8yN89df4N/w3ZrYraCvihcDJuecmOSdFraSnNP0xiG7cvHV7a/tOfPfe/QcPd3YfTbxtnMCxsMq60wI8KmlwTJIUntYOQRcKp8Xybd+ffkHnpTWfaVVjrqEyspQCKFBTfiSrirdnO8N0lK4ruQ6yDRiyTf2f/Gx3MOZzKxqNhoQC72dZWlPegiMpFHYxbzzWIJZQ4SxAAxp93q5NdcnzwMyT0rqwDCVr9veJFrT3K10EpQZa+Ku9nvxbb9ZQeZC30tQNoRGXF5WNSsgmfULJXDoUpFYBgHAyvDURC3AgKOQYx/wdBjMOP4SDP9bogKx72XJwlYaLLpir+Kse/UsozS9hQDE3eC6s1mDmLTfW6W6W5S1XWBJXE3Q0zLiT1YK463ddiD+7GvZ1MHk9ytJRdrI3PDza/Ns2e8qesRcsY/vskB2zT2zMBFuyr+wb+x49id5E76PjS2k02Mw8Zn9UdPITBKTHAA==</latexit><latexit sha1_base64="bPDTkEMP9FXbnEuQC8fXG0O8QIc=">AAACl3icjVFdaxQxFM2OH63jV6tP4svgIojIMiOF9s1SRfoiWnB3C5uh3MnemQ2bjyG5Y7sM8yN89df4N/w3ZrYraCvihcDJuecmOSdFraSnNP0xiG7cvHV7a/tOfPfe/QcPd3YfTbxtnMCxsMq60wI8KmlwTJIUntYOQRcKp8Xybd+ffkHnpTWfaVVjrqEyspQCKFBTfiSrirdnO8N0lK4ruQ6yDRiyTf2f/Gx3MOZzKxqNhoQC72dZWlPegiMpFHYxbzzWIJZQ4SxAAxp93q5NdcnzwMyT0rqwDCVr9veJFrT3K10EpQZa+Ku9nvxbb9ZQeZC30tQNoRGXF5WNSsgmfULJXDoUpFYBgHAyvDURC3AgKOQYx/wdBjMOP4SDP9bogKx72XJwlYaLLpir+Kse/UsozS9hQDE3eC6s1mDmLTfW6W6W5S1XWBJXE3Q0zLiT1YK463ddiD+7GvZ1MHk9ytJRdrI3PDza/Ns2e8qesRcsY/vskB2zT2zMBFuyr+wb+x49id5E76PjS2k02Mw8Zn9UdPITBKTHAA==</latexit><latexit sha1_base64="bPDTkEMP9FXbnEuQC8fXG0O8QIc=">AAACl3icjVFdaxQxFM2OH63jV6tP4svgIojIMiOF9s1SRfoiWnB3C5uh3MnemQ2bjyG5Y7sM8yN89df4N/w3ZrYraCvihcDJuecmOSdFraSnNP0xiG7cvHV7a/tOfPfe/QcPd3YfTbxtnMCxsMq60wI8KmlwTJIUntYOQRcKp8Xybd+ffkHnpTWfaVVjrqEyspQCKFBTfiSrirdnO8N0lK4ruQ6yDRiyTf2f/Gx3MOZzKxqNhoQC72dZWlPegiMpFHYxbzzWIJZQ4SxAAxp93q5NdcnzwMyT0rqwDCVr9veJFrT3K10EpQZa+Ku9nvxbb9ZQeZC30tQNoRGXF5WNSsgmfULJXDoUpFYBgHAyvDURC3AgKOQYx/wdBjMOP4SDP9bogKx72XJwlYaLLpir+Kse/UsozS9hQDE3eC6s1mDmLTfW6W6W5S1XWBJXE3Q0zLiT1YK463ddiD+7GvZ1MHk9ytJRdrI3PDza/Ns2e8qesRcsY/vskB2zT2zMBFuyr+wb+x49id5E76PjS2k02Mw8Zn9UdPITBKTHAA==</latexit>
o( )t <latexit sha1_base64="N82kK+S1vS9PY1MkZKJNB4AkyPQ=">AAACdHicfVFba9RAFJ6NtxovbfVRH4KhUEWWRAT7WNQHX8QK7rayE5eT2ZPs0LmEmZPqEvIrfNUf5h/x2cl2BW3FDwa++c53Zs6lbJT0lGU/RtGVq9eu39i6Gd+6fefu9s7uvam3rRM4EVZZd1KCRyUNTkiSwpPGIehS4XF5+mqIH5+h89KaD7RqsNBQG1lJARSkj3ZOn7r9/HE/30mzcbZGcpnkG5KyDY7mu6MJX1jRajQkFHg/y7OGig4cSaGwj3nrsQFxCjXOAjWg0RfduuI+2QvKIqmsC8dQslb/zOhAe7/SZXBqoKW/GBvEf8VmLVUHRSdN0xIacf5R1aqEbDK0nyykQ0FqFQgIJ0OtiViCA0FhSHHMX2NoxuHb8PC7Bh2QdU86Dq7W8KUPzdX86cD+Z5TmtzGwmBv8LKzWYBYdN9bpfpYXHVdYEVdTdJTm3Ml6SdwNt2ER+cWxXybTZ+M8G+fvn6eHLzcr2WIP2CO2z3L2gh2yN+yITZhgmn1l39j30c/oYZRGe+fWaLTJuc/+QjT+BTq3wQg=</latexit><latexit sha1_base64="iud3vMFNlLYbGl0OrfK/AAN+70w=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13Yne9zNNofpKF1FchmyNQzZOv7PPtsajPncikajIaHA+2mW1pS34EgKhV3MG481iCOocBrQgEaft6u2uuRRUOZJaV1YhpKV+ntFC9r7pS6CUwMt/MVcL/4tN22ofJG30tQNoRHnF5WNSsgm/YySuXQoSC0DgHAyvDURC3AgKEwyjvkbDM04fBcOfl+jA7LuScvBVRpOutBcxZ/29C+jNL+MgWJu8FhYrcHMW26s0900y1uusCSuJuhomHEnqwVx1+/638ouDvsyTHZHWTrKPj4b7r1a/9sG22YP2Q7L2HO2x/bZBzZmgmn2nZ2ys2g7ehntR2/PrdFgXfOA/RHRp59FVseC</latexit><latexit sha1_base64="iud3vMFNlLYbGl0OrfK/AAN+70w=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13Yne9zNNofpKF1FchmyNQzZOv7PPtsajPncikajIaHA+2mW1pS34EgKhV3MG481iCOocBrQgEaft6u2uuRRUOZJaV1YhpKV+ntFC9r7pS6CUwMt/MVcL/4tN22ofJG30tQNoRHnF5WNSsgm/YySuXQoSC0DgHAyvDURC3AgKEwyjvkbDM04fBcOfl+jA7LuScvBVRpOutBcxZ/29C+jNL+MgWJu8FhYrcHMW26s0900y1uusCSuJuhomHEnqwVx1+/638ouDvsyTHZHWTrKPj4b7r1a/9sG22YP2Q7L2HO2x/bZBzZmgmn2nZ2ys2g7ehntR2/PrdFgXfOA/RHRp59FVseC</latexit><latexit sha1_base64="iud3vMFNlLYbGl0OrfK/AAN+70w=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13Yne9zNNofpKF1FchmyNQzZOv7PPtsajPncikajIaHA+2mW1pS34EgKhV3MG481iCOocBrQgEaft6u2uuRRUOZJaV1YhpKV+ntFC9r7pS6CUwMt/MVcL/4tN22ofJG30tQNoRHnF5WNSsgm/YySuXQoSC0DgHAyvDURC3AgKEwyjvkbDM04fBcOfl+jA7LuScvBVRpOutBcxZ/29C+jNL+MgWJu8FhYrcHMW26s0900y1uusCSuJuhomHEnqwVx1+/638ouDvsyTHZHWTrKPj4b7r1a/9sG22YP2Q7L2HO2x/bZBzZmgmn2nZ2ys2g7ehntR2/PrdFgXfOA/RHRp59FVseC</latexit>
o <latexit sha1_base64="pbaN7Yle/L5V1xOdf+zP3ruyQj0=">AAACdHicfVFba9RAFJ6NtxpvrT7qQzAUqsiSFME+FvXBF7GCu63sxOVk9iQ7dC5h5qS6hPwKX/WH+Ud8drJdQVvxg4FvvvOdmXMpGyU9ZdmPUXTl6rXrN7Zuxrdu37l7b3vn/tTb1gmcCKusOynBo5IGJyRJ4UnjEHSp8Lg8fTXEj8/QeWnNB1o1WGiojaykAArSRzunT93e/pN+vp1m42yN5DLJNyRlGxzNd0YTvrCi1WhIKPB+lmcNFR04kkJhH/PWYwPiFGqcBWpAoy+6dcV9shuURVJZF46hZK3+mdGB9n6ly+DUQEt/MTaI/4rNWqoOik6apiU04vyjqlUJ2WRoP1lIh4LUKhAQToZaE7EEB4LCkOKYv8bQjMO34eF3DTog6552HFyt4Usfmqv5s4H9zyjNb2NgMTf4WVitwSw6bqzT/SwvOq6wIq6m6CjNuZP1krgbbsMi8otjv0ym++M8G+fvn6eHLzcr2WIP2WO2x3L2gh2yN+yITZhgmn1l39j30c/oUZRGu+fWaLTJecD+QjT+BTzBwQk=</latexit><latexit sha1_base64="DNUAenKGd/gB9ac2XxeyIn3grM8=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13Zn93E32xymo3QVyWXI1jBk6/g/+2xrMOZzKxqNhoQC76dZWlPegiMpFHYxbzzWII6gwmlAAxp93q7a6pJHQZknpXVhGUpW6u8VLWjvl7oITg208Bdzvfi33LSh8kXeSlM3hEacX1Q2KiGb9DNK5tKhILUMAMLJ8NZELMCBoDDJOOZvMDTj8F04+H2NDsi6Jy0HV2k46UJzFX/a07+M0vwyBoq5wWNhtQYzb7mxTnfTLG+5wpK4mqCjYcadrBbEXb/rfyu7OOzLMNkdZeko+/hsuPdq/W8bbJs9ZDssY8/ZHttnH9iYCabZd3bKzqLt6GW0H709t0aDdc0D9kdEn34CR4XHgw==</latexit><latexit sha1_base64="DNUAenKGd/gB9ac2XxeyIn3grM8=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13Zn93E32xymo3QVyWXI1jBk6/g/+2xrMOZzKxqNhoQC76dZWlPegiMpFHYxbzzWII6gwmlAAxp93q7a6pJHQZknpXVhGUpW6u8VLWjvl7oITg208Bdzvfi33LSh8kXeSlM3hEacX1Q2KiGb9DNK5tKhILUMAMLJ8NZELMCBoDDJOOZvMDTj8F04+H2NDsi6Jy0HV2k46UJzFX/a07+M0vwyBoq5wWNhtQYzb7mxTnfTLG+5wpK4mqCjYcadrBbEXb/rfyu7OOzLMNkdZeko+/hsuPdq/W8bbJs9ZDssY8/ZHttnH9iYCabZd3bKzqLt6GW0H709t0aDdc0D9kdEn34CR4XHgw==</latexit><latexit sha1_base64="DNUAenKGd/gB9ac2XxeyIn3grM8=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13Zn93E32xymo3QVyWXI1jBk6/g/+2xrMOZzKxqNhoQC76dZWlPegiMpFHYxbzzWII6gwmlAAxp93q7a6pJHQZknpXVhGUpW6u8VLWjvl7oITg208Bdzvfi33LSh8kXeSlM3hEacX1Q2KiGb9DNK5tKhILUMAMLJ8NZELMCBoDDJOOZvMDTj8F04+H2NDsi6Jy0HV2k46UJzFX/a07+M0vwyBoq5wWNhtQYzb7mxTnfTLG+5wpK4mqCjYcadrBbEXb/rfyu7OOzLMNkdZeko+/hsuPdq/W8bbJs9ZDssY8/ZHttnH9iYCabZd3bKzqLt6GW0H709t0aDdc0D9kdEn34CR4XHgw==</latexit>
o ( ) t <latexit sha1_base64="AijpuD5XdKwqeARXIV+woEF8hQM=">AAACdHicfVFba9RAFJ6NtxovbfVRH4KhUEWWRAT7WNQHX8QK7rayE5eT2ZPs0LmEmZPqEvIrfNUf5h/x2cl2BW3FDwa++c53Zs6lbJT0lGU/RtGVq9eu39i6Gd+6fefu9s7uvam3rRM4EVZZd1KCRyUNTkiSwpPGIehS4XF5+mqIH5+h89KaD7RqsNBQG1lJARSkj3ZOn7p987if76TZOFsjuUzyDUnZBkfz3dGEL6xoNRoSCryf5VlDRQeOpFDYx7z12IA4hRpngRrQ6ItuXXGf7AVlkVTWhWMoWat/ZnSgvV/pMjg10NJfjA3iv2KzlqqDopOmaQmNOP+oalVCNhnaTxbSoSC1CgSEk6HWRCzBgaAwpDjmrzE04/BtePhdgw7IuicdB1dr+NKH5mr+dGD/M0rz2xhYzA1+FlZrMIuOG+t0P8uLjiusiKspOkpz7mS9JO6G27CI/OLYL5Pps3GejfP3z9PDl5uVbLEH7BHbZzl7wQ7ZG3bEJkwwzb6yb+z76Gf0MEqjvXNrNNrk3Gd/IRr/ArcZwUU=</latexit><latexit sha1_base64="7We+Wtw1IUCjthukT8N7Q+XrA+c=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00/Kice/</latexit><latexit sha1_base64="7We+Wtw1IUCjthukT8N7Q+XrA+c=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00/Kice/</latexit><latexit sha1_base64="7We+Wtw1IUCjthukT8N7Q+XrA+c=">AAACmXicjZHbahRBEIZ7x1McT4le5mZwEaLIMhMEvYyHiyCIiu4msj0uNb01s036MHTXmCzDPIW3eZq8hW9jz2YFTUQsaPj6r7+6u6qLWklPafpjEF25eu36jY2b8a3bd+7e29y6P/G2cQLHwirrDgvwqKTBMUlSeFg7BF0oPCiOXvf5g2/ovLTmMy1rzDVURpZSAAXpi53R13bHPO5mm8N0lK4iuQzZGoZsHf9nn20NxnxuRaPRkFDg/TRLa8pbcCSFwi7mjccaxBFUOA1oQKPP21VbXfIoKPOktC4sQ8lK/b2iBe39UhfBqYEW/mKuF/+WmzZUvshbaeqG0Ijzi8pGJWSTfkbJXDoUpJYBQDgZ3pqIBTgQFCYZx/wNhmYcvgsHv6/RAVn3pOXgKg0nXWiu4k97+pdRml/GQDE3eCys1mDmLTfW6W6a5S1XWBJXE3Q0zLiT1YK463f9b2UXh30ZJrujLB1lH58N916t/22DbbOHbIdl7DnbY/vsAxszwTT7zk7ZWbQdvYz2o7fn1miwrnnA/ojo00/Kice/</latexit>
Figure 1: ver i : ll ti lti t t . ll t r t e sa e individual odel ar it t r , t t i i tl t i t l t t - e its o n observation a s t l r r r t ti f r i ti . t ti t , c agent observes an i a e r r s t ti f t ir t s ll s ss s r st t er agents during the last ti e st . i i ls r r ss t r I r; t r a casted essages are pr cesse t r a essa e c er; t e i a e feat res a t e essa e feat res are concatenated a asse t r a lic et r t re ict t e e t acti . e i a e feat res are also used to generate the next co unication essages using the o unication utoencoder.
bias or supervision signal. In Section 5, e sho the effectiveness of our approach on a variety of L co unication tasks.
n overvie of our ethod is sho n in Figure 1, hich illustrates the co unication flo a ong agents at so e arbitrary ti e step t. All agents share the sa e individual odel architecture, and each agent consists of two odules: a speaker odule and a listener odule. e describe the architecture details of a single agent k below.

heading: 4.1 Speaker odule
text: At each time step t, the speaker module takes in the agent’s observation o(k)t and outputs the agent’s next communication message c(k)t .
Image Encoder Given the raw pixel observation, the module first uses a image encoder to embed the pixels into a low-dimensional feature o(k)t → ϕim(o(k)t ) ∈ R128. The image encoder is a convolutional neural network with 4 convolutional layers, and the output of this network is spatially pooled. We use the same image encoder in the listener module.
Communication Autoencoder The goal of the communication autoencoder is to take the current state observation and generate the next subsequent message. We use an autoencoder to learn a mapping from the feature space of image encoder to communication symbols, i.e. ϕim(o (k) t ) → c(k)t . The autoencoder consists of an encoder and a decoder, both parameterized by a 3-layer MLP. The decoder tries to reconstruct the input state from the communication message c(k)t → ϕ̂im(o(k)t ). The communication messages are quantized before being passed through the decoder. We use a straightthrough estimator to differentiate through the quantization [1]. The auxiliary objective function of our model is to minimize the reconstruction loss ∥ϕim(o(k)t )− ϕ̂im(o(k)t )∥2. This loss is optimized jointly with the policy gradient loss from the listener module.

heading: 4.2 Listener Module
text: While the goal of the speaker module is to output grounded communication based on the agent’s private observation o(k)t , the goal of the listener module is to learn an optimal action policy based on both the observation o(k)t and communicated messages ct−1. At each time step t, the listener module outputs the agent’s next action a(k)t .
4
Message Encoder The message encoder linearly projects all messages communicated from the previous time step ct−1 using a shared embedding layer. The information across all agent message embeddings is combined through concatenation and passed through 3-layer MLP. The resulting message feature has a fixed dimension of 128, i.e. ϕcomm(ct) ∈ R128. Policy Network Each agent uses an independent policy head, which is a standard GRU [7] policy with a linear layer. The GRU policy concatenates the encoded image features and the message features ϕ (k) t = ϕim(o (k) t ) ◦ ϕcomm(ct), where ◦ is the concatenation operator across the feature dimension. The GRU policy predicts a distribution over the actions a ∼ π(ϕ(k)t ) and the corresponding expected returns. The predicted action distribution and expected returns are used for computing the policy gradient loss. This loss is jointly optimized with the autoencoder reconstruction loss from the speaker module.
The same setup is used for all experiments. The exact details of the network architecture and the corresponding training details are in the Appendix.

heading: 5 Experiments
text: In this section, we demonstrate that autoencoding is a simple and effective representation learning algorithm to ground communication in MARL. We evaluate our method on various multi-agent environments and qualitatively show that our method outperforms baseline methods. We then provide further ablations and analyses on the learned communication.

heading: 5.1 Environments
text: We introduce three multi-agent communication environments: CIFAR Game, FindGoal, and RedBlueDoors. Our work focuses on fully cooperative scenarios, but can also be extended to competitive or mixed scenarios.
Our environments cover a wide range of communication task settings, including (1) referential or non-referential, (2) ordinal or non-ordinal, and (3) two-agent versus generalized multi-agent. A referential game, often credited to Lewis signaling game [25], refers to a setup in which agents communicate through a series of message exchanges to solve a task. In contrast to non-referential games, constructing a communication protocol is critical to solving the task – where one can only arrive at a solution through communication. Referential games are referred to as a grounded learning environment, and therefore, communication in MARL has been studied mainly through the lens of referential games [12, 24]. Lastly, ordinal tasks refer to a family of problems where the ordering of the actions is critical for solving the task. The difference between ordinal and non-ordinal settings is illustrated in Figure 2, where the blue agent must wait for the red agent to open the door to successfully receive a reward. In contrast, non-ordinal tasks could benefit from shared information, but it is not necessary to complete the task. We now describe the environments used in our work in more detail:
CIFAR Game We design CIFAR Game following the setup of Multi-Step MNIST Game in [17], but with CIFAR-10 dataset [23] instead. This is a non-ordinal, two-agent, referential game. In CIFAR game, each agent independently observes a randomly drawn image from the CIFAR-10 dataset, and the goal is to communicate the observed image to the other agent within 5 environment time steps. At each time step, each agent broadcasts a set of communication symbols of length l. At the final time step, each agent must choose a class label from the 10 possible choices. At the end of the episode, an agent receives a reward of 0.5 for each correctly guessed class label, and both agents receive a reward of 1 only when both images are classified correctly.
MarlGrid Environments The second and third environments we consider are: FindGoal (Figure 2 left) and RedBlueDoors (Figure 2 right). Both environments are adapted from the GridWorld environment [5, 33] and environment states are randomized at every episode.
FindGoal is a non-ordinal, multi-agent, non-referential game. We use N = 3 agents, and at each time step, each agent observes a partial view of the environment centered at its current position. The task of agents is to reach the green goal location as fast as possible. Each agent receives an individual reward of 1 for completing the task and an additional reward of 1 when all agents have reached the goal. Hence, the optimal strategy of an agent is to communicate the goal location once it observes the goal. If all agents learn a sufficiently optimized search algorithm, they can maximize their reward without communication.
RedBlueDoors is an ordinal, two-agent, non-referential game. The environment consists of a red door and a blue door, both initially closed. The task of agents is to open both doors, but unlike in the previous two games, the ordering of actions executed by agents matters. A reward of 1 is given to both agents if and only if the red door is opened first and then the blue door. This means that any time the blue door is opened first, both agents receive a reward of 0, and the episode ends immediately. Hence, the optimal strategy for agents is to convey the information that the red door was opened. Since it is possible to solve the task through visual observation or by a single agent that opens both doors, communication is not necessary.
Compared to CIFAR Game, the MarlGrid environments have a higher-dimensional observation space and a more complex action space. The fact that these environments are non-referential exacerbates the visual-language grounding problem since communication can only exist in the form of cheap talk (i.e., costless, nonbinding, nonverifiable communication [13] that has no direct effect on the game state and agent payoffs). We hope to show from this set of environments that autoencoders can be used as a surprisingly simple and adaptable representation learning task to ground communication. It requires little effort to implement and almost no change across environments. Most importantly, as we will see in Section 5.4, autoencoded representation shows an impressive improvement over communication trained with reinforcement learning.

heading: 5.2 Baselines
text: To evaluate the effectiveness of grounded communication, we compare our method (ae-comm) against the following baselines: (1) a no-comm baseline, where agents are trained without a communication channel; (2) a rl-comm baseline*, where we do not make a distinction between the listener module and the speaker module, and the communication policy is learned in a way similar to the environment policy; (3) a rl-comm-with-biases baseline, where inductive biases for positive signaling and positive listening are added to rl-comm training as proposed in [10]; (4) a ae-rl-comm baseline, where the communication policy is learned by an additional policy network trained on top of the autoencoded representation in speaker module.

heading: 5.3 The Effectiveness of Grounded Communication
text: In Table 1 and Figure 3, we compare task performance of ae-comm agents with performance of baseline agents. We report all results with 95% confidence intervals, evaluated on 100 episodes per seed over 10 random seeds.
In CIFAR Game environment, all three baselines (no-comm, rl-comm, rl-comm-with-biases) could only obtain an average reward close to that of random guesses throughout the training process.
*The rl-comm baseline can be seen as an A3C version of RIAL [17] without parameter sharing.
In comparison, ae-comm agents achieve a much higher reward on average. Since this environment is a referential game, our results directly indicate that ae-comm agents learn to communicate more effectively than the baseline agents.
RedBlueDoors environment poses a challenging multi-agent coordination problem since the reward is extremely sparse. As shown in Figure 3(b), neither of the no-comm and rl-comm agents was able to learn a successful coordination strategy. Although rl-comm-with-biases agents outperform the other two baseline agents, they do not learn an optimal strategy that guarantees an average reward close to 1. In contrast, ae-comm agents converge to an optimal strategy after 150k of training.
In FindGoal environment, agents are able to solve the task without communication, but their performance can be improved with communication. Therefore, we use episode length instead of reward as the performance metric for this environment. To resolve the ambiguity in Figure 3(c), we also include numerical results in a table below. While all agents are able to obtain full rewards, Figure 3 shows that ae-comm agents are able to complete the episode much faster than other agents. We further verify that this improvement is indeed a result of the successful communication by providing further analysis in Section 5.5.
Our results indicate that a communication model trained with autoencoding tasks consistently outperforms the baselines across all environments. The observation that communication does not work well with reinforcement learning is consistent with observations made in prior works [10, 17, 28]. Furthermore, our results with autoencoders – a task that is often considered trivial – highlight that we as a community may have overlooked a critical representation learning component in MARL communication. In Section 5.4, we provide a more detailed discussion on the difficulty of training emergent communication via policy optimization.

heading: 5.4 The Role of Autoencoding
text: Given the success of agents trained with autoencoders, it is natural to ask whether a better communication protocol can emerge from the speaker module by jointly training it with a reinforcement learning policy. To this end, we train a GRU policy on top of the autoencoded representation (ae-rl-comm) and compare it against our previous model that was trained just with an autoencoder (ae-comm). The communication policy head is independent of the environment action policy head.
Surprisingly, we observed in Figure 4 that the model trained jointly with reinforcement learning consistently performed worse (except for in FindGoal, where performance of ae-rl-comm and ae-comm stayed about the same). We hypothesize that the lack of correlation between visual observation and the communication rewards hurts the agents’ performance. This lack of visualreward grounding could introduce conflicting gradient updates to the action policy, and thereby exacerbate the high-variance problem that already exists in reinforcement learning. In other words, optimization is harder whenever the joint-exploration problem for learning speaker and listener policies is introduced. Our observation suggests that specialized reward design and training at the level of [2] might be required for decentralized MARL communication. This prompts us to rethink how to address the lack of visual grounding in communication policy, where this work serves as a first step in this direction.

heading: 5.5 Analyzing the Effects of Communication Signals on Agent Behavior
text: Communication Embedding To analyze whether the agents have learned a meaningful visual grounding for communication, we first visualize the communication embedding. In Figure 5, we visualize the communication symbols transmitted by the agents trained on RedBlueDoors. The communication symbols are discrete with a length of l = 10 (1024 possible embedding choices), and we use approximately 4096 communication samples across 10 episodic runs. We embed the communication symbols using t-SNE [44] and further cluster them using DBSCAN [11]. In the figure, we visualize clusters by observing the correspondence between image states and communication symbols produced by agents in those states. For example, we observed that a specific communication cluster corresponded to an environment state when the red door was opened; this suggests a communication action where one agent signals the other agent to open the blue door and complete the task. Entropy of Action Distribution To measure whether communicated information directly influences other agents’ actions, we visualize the entropy of action distribution during episode rollouts. Suppose one agent shares information that is vital to solving the task. In that case, a decrease in entropy should be observed in the action distributions of other agents, as they act more deterministically towards solving the task.
As shown in Figure 6, we visualize the entropy of action distribution across 256 random episodic runs using policy parameters from a fully trained ae-comm model. The entropies are aligned using
environment milestone events: for FindGoal, this is when the first agent reaches the goal; for RedBlueDoors, this is when the red door is opened. Since the identity of the agents that solve the task first does not matter, entropy plots are computed with respect to the listener agents (i.e., agents that receive vital information from others). In FindGoal, this corresponds to the last agent to reach the goal; in RedBlueDoors, this corresponds to the agent opening the blue door. For both environments, we see a sharp fall-off in entropy as soon as the first agents finish the task. In contrast, agents trained without autoencoding act randomly regardless of whether other agents have completed the task. This reaffirms that the agents trained with an autoencoder can effectively transmit information to other agents.
Positive Signaling and Positive Listening We additionally investigate the two metrics suggested by [27] for measuring effectiveness of communication, positive signaling and positive listening. Since ae-comm agents have to communicate their learned representation, the presence of representation learning task loss means that ae-comm agents are intrinsically optimized for positive signaling (i.e., sending messages that are related to their observation or action). In Table 2, we report the increase in average reward when adding a communication channel, comparing ae-comm with its baselines; this metric is suggested by [27] to
be a sufficient metric for positive listening (i.e., communication messages influence the behavior of agents). We observe that agent task performance improves most substantially in ae-comm with the addition of communication channel.

heading: 6 Discussion and Societal Impacts
text: We present a framework for grounding multi-agent communication through autoencoding, a simple self-supervised representation learning task. Our method allows agents to learn non-differentiable communication in fully decentralized settings and does not impose constraints on input structures (e.g., state inputs or pixel inputs) or task nature (e.g., referential or non-referential). Our results demonstrate that agents trained with the proposed method achieve much better performance on a suite of coordination tasks compared to baselines.
We believe this work on multi-agent communication is of importance to our society for two reasons. First, it extends a computational framework under which scientific inquiries concerning language acquisition, language evolution, and social learning can be made. Second, unlike works in which agents can only learn latent representations of other agents through passive observations [45, 46], it opens up new ways for artificial learning agents to improve their coordination and cooperative skills, increasing their reliability and usability when deployed to real-world tasks and interacting with humans.
However, we highlight two constraints of our work. First, our method assumes that all agents have the same model architecture and the same autoencoding loss, while real-world applications may involve heterogeneous agents. Second, our method may fail in scenarios where agents need to be more selective of the information communicated since they are designed to communicate their observed information indiscriminately. We believe that testing the limits of these constraints, and relaxing them, will be important steps for future work.
We would also like to clarify that communication between the agents in our work is only part of what true “communication” should eventually entail. We position our work as one partial step toward true communication, in that our method provides a strong bias toward positive signaling and allows decentralized agents to coordinate their behavior toward common goals by passing messages amongst each other. It does not address other aspects of communication, such as joint optimization between a speaker and a listener. In the appendix section, we highlight the difficulty of training emergent communication via policy optimization alongside empirical results.
Another limitation of this work, which is also one concern we have regarding its potential negative societal impact, is that the environments we consider are fully cooperative. If the communication method we present in this work is to be deployed to the real world, we need to either make sure the environment is rid of adversaries, or conduct additional research to come up with robust counterstrategies in the face of adversaries, which could use better communication policies as a way to lie, spread misinformation, or maliciously manipulate other agents.

heading: Acknowledgements
text: We sincerely thank all the anonymous reviewers for their extensive discussions on and valuable contributions to this paper. We thank Lucy Chai and Xiang Fu for helpful comments on the manuscript. We thank Jakob Foerster for providing inspiring advice on multi-agent training. Additionally, TL would like to thank Sophie and Sofia; MH would like to thank Sally, Leo and Mila; PI would like to thank Moxie and Momo. This work was supported by a grant from Facebook.

Review summaries:
The goal of the paper is to show that learning a representation to ground language can improve communication in multi-agent settings.  Their method: each agent encodes an image and the other agents messages (through a small conn-net and mlp, respectively). A GRU-based policy outputs an action. A communication autoencoder predicts a message from the image feature and is trained with an auxiliary construction objective (i.e., to reconstruct the image).  They evaluate their model on a referential game with CIFAR images and two grid environments: one where the agents must reach a goal collaboratively and another where one agent has to open a door for another to pass.  They show that their method with an auto encoder strongly outperforms the non-autoencoder variants on the CIFAR task and door task.

In this paper, the authors present a novel method for learning decentralized communication policies in fully cooperative multi-agent RL. The method is based on autoencoding agent observations and using these as messages, thus sidestepping the usual "chicken and egg" problem for simultaneously learning speaker and listener policies, which is a hard joint-exploration problem. The authors present results in which this method outperforms both non-communication and RIAL baselines, and include some analyses and ablations that shed light on how and why this method is successful.

This paper proposes an approach towards encouraging the development of communication protocols in multi-agent reinforcement learning systems. The authors rightly observe that naive training of a communication policy according to task reward often fails, due to high variability in the initial messages, the difficulty of connecting this communication to reward, and lack of common grounding among the agents.  The authors' idea is to imbue agents with a message-generating mechanism cast as an autoencoder, where the agents aim to generate discrete messages that can reconstruct the observations of the agents. Thus, the message generated for a given timestep is essentially a discretized summary of an agents' observation at that time. This encourages the messages to be maximally informative about an agents' current state—something that is shown by qualitatively.  The authors evaluate this communication idea on a variety of environments, from a reference game variant to a few multi-agent minigrid environments, and show that their autoencoded communication outperforms (1) non-communicative baselines and (2) communication policies trained with standard RL objectives. They then have a few analyses that show that the agents are making use of the generated messages, in that (1) messages are informative about agents' current states (by design) and (2) messages affect other agents' entropy over decisions once they indicate a change in state.

The paper proposes to learn the communication protocol in an emergent communication environment via an autoencoder; i.e. they train an autoencoder to learn a discrete representation of the image part of the observation, and use that as the message that gets broadcasted to all other agents at each timestep. The paper shows that this achieves significant gains over learning the communication protocol from scratch in a decentralized way.