Shaped Rewards Bias Emergent Language
Authors: Brendon Boldt
Yonatan Bisk
David R Mortensen
Venue: ICLR 2022 Submitted (2022)
Keywords: emergent language
reinforcement learning
neural networks

Abstract: One of the primary characteristics of emergent phenomena is that they are determined by the basic properties of the system whence they emerge as opposed to explicitly designed constraints. Reinforcement learning is often used to elicit such phenomena which specifically arise from the pressure to maximize reward. We distinguish two types of rewards. The first is the base reward which is motivated directly by the task being solved. The second is shaped rewards which are designed specifically to make the task easier to learn by introducing biases in the learning process. The inductive bias which reward shaping introduces is problematic for emergent language experimentation because it biases the object of study: the emergent language. The fact that shaped rewards are intentionally designed conflicts with the basic premise of emergent phenomena arising from basic principles. In this paper, we use a simple sender-receiver navigation game to demonstrate how reward shaping can 1) explicitly bias the semantics of the learned language, 2) significantly change the entropy of the learned communication, and 3) mask the potential effects of other environmental variables of interest.

Under review as a conference paper at ICLR 2022
SHAPED REWARDS BIAS EMERGENT LANGUAGE
Anonymous authors
Paper under double-blind review
ABSTRACT
One of the primary characteristics of emergent phenomena is that they de- termined by the basic properties of the system whence they emerge as opposed explicitly designed constraints. Reinforcement learning is often used to elicit such phenomena which specifically arise from the pressure to maximize reward. We distinguish two types of rewards. The first is the base reward which mo- tivated directly by the task being solved. The second is shaped rewards which are designed specifically to make the task easier to learn by introducing biases in the learning process. The inductive bias which shaped rewards introduce is problematic for emergent language experimentation because it biases the object of study: the emergent language. The fact that shaped rewards intentionally designed conflicts with the basic premise of emergent phenomena arising from basic principles. In this paper, we to demonstrate how shaped rewards can 1) explicitly bias the semantics of the learned language, 2) significantly change the entropy of the learned language, and 3) mask the potential effects of other environmental variables of interest.
simple sender-receiver navigation game
INTRODUCTION
In emergent language research, the goal is study language as it emerges from the inherent proper- ties of the enviromment, language, and agents. One pitfall for such experiments, though, is that the language simply mirrors some of design choices of the environment or experimental setting more generally. there is a risk that the language simply mirrors the design choices of the environment. For example, [Bullard et al. (202I) introduce a method for discovering optimal languages for commu- nication between independently trained agents, yet rather than emerging from basic principles, the learned language is the result of an intentionally designed search algorithm. Reinforcement learning is common tool in this field for observing the emergence of language out of reward maximiza- tion pressure. One such design choice which can obscure these emergent properties is adding shaped rewards on top of the base reward of the reinforcement learning environment (Wiewiora) 2010).
The base reward of the environment derives directly from succeeding at the task in question. The difficulty with relying solely on the base rewards is that if the task is especially long or complicated, the agent may only receive a reward infrequently which makes for a difficult learning problem. In such case, base reward is considered sparse. This motivates shaped rewards which are inserted at intermediate steps based on domain knowledge in order to introduce an inductive bias towards good solutions. For example, the base reward in chess would simply be winning or losing the game. A shaped reward could then be taking the opponents material while not losing your own. While this shaped reward often good heuristic, it can lead local optima; for example, discourages strategies which would sacrifice individual pieces in order to win the whole game. While local optima present a problem for maximizing reward, the biases introduced by shaped rewards present unique problem for emergent language which we will highlight in this paper.
For emergent language research, the inductive bias which shaped rewards introduce especially problematic because it exerts significant influence on the learned language whose emergent prop- erties are the object of study. This influence can comprise 1) biasing the semantics of the language, 2) changing a property of the whole language (e.g., language entropy), or 3) masking the influence of some other environmental parameter on the language. For example, some emergent language work incorporates shaped rewards into their environment without accounting for the the biases it may introduce (Mordatch & Abbeel 2018:) [Brandizzi et al., 2021 Harding Graesser et al., 2019). From an engineering and design perspective, tweaking the system to achieve desired result standard

---

Under review as a conference paper at ICLR 2022
Gufbel-Softmax
S-agent (perceptron)
R-agent (perceptron)
Goal
(x. location
y) action
(a) the centerward environment, the receiver must navigate to the center of the world.
(b) The architecture of our two-agent, asymmetric system.
Figure 1
practice, but from scientific and experimental perspective, these additional shaped rewards serve as potential confounding factors and hinder accurate observations of the emergent phenomena.
We study this by introducing simple navigation game with continuous states and actions where a sender passes one-word message to the receiver (illustrated in Figure [b and Tak. Within this environment, our experiments look the entropy and semantics of the emergent language in the absence and presence of shaped rewards. In the course of these experiments, we find that rein- forcement learning algorithm's experience buffer size has a significant impact on the entropy of the learned language and potentially explains our experimental findings. To this end we introduce mathematical model based on the Chinese restaurant process for understanding the effect of experi- ence buffer size emergent language more generally. We highlight the following contributions in our paper:
Demonstrating basic ways in which shaped rewards can undesirably influence emergent language experiments
Presenting a mathematical model for understanding the role of experience buffer size in the entropy of emergent language
RELATED WORK
2.1 TAXONOMY
We intentionally design environment study shaped rewards and the entropy of language, which requires it to differ from prior art in specific ways. To elucidate this, we create a taxonomy of emergent language environments based on whether or the environment has multi-step episodes and the presence of a frivially optimal language (defined below). The taxonomy is given in Table and brief description of each environment is given in Appendix
Generally speaking, the motivation for shaped rewards in a given environment is sparsity of the base reward which requires multi-step, multi-utterance environment. Thus, our experiments naturally require a multi-step environment.
We consider an environment have trivially optimal language if the information which needs be communicated from sender to receiver can be perfectly represented in the emergent language. Such language most frequently arises when the communicated information is derived from small number of categorical variables encoded in the observation. For example, in an environment where the sender must specify an element of the set (red, green, blue) ) (square, circle) using messages from the set {r,g, b} {s, c), trivial language where color maps to a unique letter in the first position and shape maps to a unique letter in the second position. Other environments have no trivially optimal languages. For example, if the sender must communicate using messages from the set (a, b, c). there no trivially optimally language since the sender can at best partition the set of integers to reach optimal but imperfect solution.
element of 21,2,... 100)
Kharitonov et al. (2020) gives evidence that there an entropy minimization in pressure inherent in emergent language settings. Building this, Chaabouni et al. (2021) explicitly look the
2

---

Under review as a conference paper at ICLR 2022
Max Reward
Max Reward
Entropy
Entropy
(a) Trivially optimal language
(b) trivially optimal language
Figure 2: Maximum reward for given entropy. When trivially optimal language exists (a), the reward plateaus at a global maximum where further increases in entropy do not increase reward. The environment in this paper has such trivial language and is more similar (b).
Paper
Task
Trivial?
Multi-step?
Havrylov & Titov (2017)
image signalling
no
no
Kottur al. (2017)
dialog Q&A
yes
yes
Mordatch & Abbeel
2018
goal specification
yes yes
Lazaridou et al.
image signalling
yes no
Kharitonov et a.
2202
vector signalling
yes
no
Chaabounr et
202TT
color signalling
no no
This paper
navigation
yes
Table 1: Summary of related work in terms of emergent language configurations used.
tradeoff between entropy and reward higher entropy languages have the potential to convey more information at the expense of being more difficult to learn. This tradcoff disappears if trivially optimal language is learned since there is no further reward maximization pressure. Language tropy greater than this minimum, then, does not emerge from the the reward maximization pressure. Although such environment does not preclude studying entropy choose where the information being communicated is fully continuous so an increase in entropy can always translate increase in reward. This leads to smooth tradeoff between entropy and reward which is illustrated in Figure
environment
2.2 SPECIFIC APPROACHES
The environment and agent configuration of this paper are most closely related to Chaabouni et al. (202)) who test the balance between entropy and semantic precision in a two-agent color discrim- ination task. Although the color space comprises 330 distinct colors, the environment facilitates languages which cover (learned) region of nearby colors. In this way, there is no trivially optimal, one-to-one language as the task is inherently fuzzy. HHvrrylov Titov (2017) use signalling game with natural images which lacks a trivially optimal language, but using natural images results in many uncontrolled variables, lessening the ability of the experiments make basic, tritcpprnncipes claims.
Studying shaped rewards has been explored previously in emergent language primarily as inductive bias for encouraging some property, such compositionality, to emerge (Hazra et al. [2021] Jaques) al., 2019: Ficcles et al., 2019). This paper, instead, focuses on the negative aspects induc- tive biases introduced by shaped rewards regarding how they can hinder empirical investigation of emergent languages.
Superficially, the environment used in this paper bears resemblance to Mordatch & Abbeel (2018) and Kajić et al. (2020) as these both deal with navigation. In both of these environments, the agents communicate about discrete set of actions goal locations, whereas the agents in this paper communicate about a continuous action. Neither of the papers specifically investigates the effects of shaped rewards the languages learned.
3

---

Under review as a conference paper at ICLR 2022
3 METHODS
3.1 ENVIRONMENT
In this paper we use a simple 2-dimensional navigation environment with two closely related tasks. A sender agent observes the position of a receiver agent, sends a message the receiver, and the receiver takes an action. In the centerward task (illustrated in Figure la the receiver is initialized uniformly at random within a circle and must navigate towards a circular goal region at the center. In the edgeward task, the receiver is initialized uniformly at random within a circle and must navigate to a goal region comprising the entire area outside of the circle. The centerward environment is more realistic environment. The edgeward environment, on the other hand, admits greater variety of languages, since moving consistently in any one direction eventually gets you to the edge; therefore, learning move in variety of evenly spaced direction solves the task more quickly but is not strictly necessary. There are no obstacles or walls in the environment. The receiver's 's location and action are continuous variables stored as floating ceatiig--ointttoo values. If the receiver does not reach the goal region within a certain number of steps, the episode ends with no reward given.
3.2 AGENT ARCHITECTURE
Our architecture comprises two agents, conceptually speaking, but in practice, they are single neural network. The sender is a disembodied agent which observes the location of the receiver and passes message in order to guide it towards the goal. The receiver is agent which receives the message as its only input and takes an action solely based on that message (i.e., it is "blind"). The sender and receiver are randomly initialized at the start of training, trained together, and tested together. The architecture of the agents is illustrated in Figure lb
The observation of the sender is pair of floating-poin values representing the receiver's location. The sender itself is a 2-layer perceptron with tanh activations. The output of the second layer is passed to a Gumbel-Softmax bottleneck layer (Maddison et al.) 2017 Jang et al. 2017) which en- ables learning a discrete, one-hot representation, as an information bottleneck (Tishby et al. 22000). The activations of this layer can be thought of as the "words" or "lexicon" of the emergent language. At evaluation time, the bottleneck layer functions deterministically an argmax layer, emitting one- hot vectors. The receiver is 1-layer perceptron which receives the output of the Humbel-Softmat layer as input. The output is pair of floating-point values which determine the action of the agent. The action is clamped to a maximum step size.
3.3 OPTIMIZATION
Optimization is performed using stochastic gradient descent as part of proximal policy optimiza- tion (PPO) (Schulman et al., 2017). Specifically, we use the implementation of PPO provided by Stable Baselines 3 with our neural networks implemented in PyTorch (Raffin et al., 2019; Paszke jet al., 2019). Using a Gumbel-SSftmas bottleneck layer allows for end-to-end hackpropagation, making optimization faster and more consistent than using backpropagation-fiee method like RE- INFORCE (Kharitonov et al., 2020; Williams 1992).
We will give basic explanation of how PPO (and related algorithms) works as it is necessary to connect to the mathematical model presented in Section PPO has two stages sampling and optimizing. During the sampling stage, the algorithm runs the agents in the environment and stores the states, actions, and rewards in a experience buffer (or rollout buffer). The optimization stage then comprises performing gradient descent the agents (i.e., neural network policy) using the data from experience buffer. The next iteration starts with the updated agents and an empty experience buffer.
3.4 REWARDS
We make use of two different rewards in our configuration, base reward and an shaped reward. The base reward is simply a positive reward of given if the receiver reaches to the goal region before the episode ends and reward otherwise. The shaped reward, given at every timestep, is the decrease in distance to the goal. If the goal region is centered at (0, 0), the standard shaped reward for the centerward environment is given by Equation [] also trivially biased version of the
4

---

Under review as a conference paper at ICLR 2022
reward which only takes into account horizontal distance specified in Equation 2
V 22 + vi-
x vi
(1)
7/ 27 Vz2
For the edgeward environment, we use the opposite of rt as the goal is to move away from the center.
The interplay between base and shaped rewards is important to understand in the larger context of how reinforcement learning problems are structured. The base rewards are well-motivated and directly correspond to the ultimate aim of the task, but their sparsity can make it difficult for the agents to learn to succeed. Shaped rewards facilitate learning by using expert knowledge to form an inductive bias, yet they present drawback for traditional reinforcement learning and emergent language. Within reinforcement learning where the goal is to train the best performing agent, shaped rewards can lead to the agent finding local optima if better solutions are excluded by the inductive bias. Within emergent language, the problem more nuanced as the goal is primarily to study a wide range of emergent properties of the language learned within the environment. While base rewards have a natural connection to the environment and task, the shaped rewards introduce a reward signal which is not intrinsically connected with the task and environment, even if it is good heuristic.
4 EXPLANATORY MODEL
We argue that the results presented in Sections 5.2 and 5.3 can be explained by an effective change in the size of PPO's experience buffer. In order illustrate this, we introduce a simple mathematical model based on the Chinese restaurant process (Blii, 2007 Aldous 1985p. While the model does not exactly match our experimental setup, the key simi himilriites allow us to reason about our results as well as potential future experiments.
The Chinese restaurant process is an iterative stochastic process which yiclds a probability distribu- tion over the positive integers. The key similarity between the Chinese restaurant process and our learning setup is that it they are self-reinforcing processes, that is, when given value is selected in one iteration of the process, the probability that that value is chosen again in subsequent iterations increases. We generalize one aspect of the Chinese restaurant process in Section 4.2 to better match the sampling-optimizatin procedure of PPO. The primary simplification which this model makes is that it does not take into account the "meaning" of actions and the effects they have within the environment. For example, every successful agent in the centerward environment will use least three distinct nearly equiprobably so as to span the 2-dimensional space whereas as no such lower bound exists in the stochastic process.
4.1 CHINESE RESTAURANT PROCESS
As the name suggests, useful analogy for the Chinese restaurant process starts with a restaurant with infinitely many empty tables, indexed by the positive integers, which can hold an unbounded number of customers. As each customer walks in, they sit populated table with probability proportional to the number of people already at that table. The customer will sit at a new table proportional a hyperparameter a which modulates the concentration of the final distribution. The decision the customer makes is equivalent to sampling from a categorical distribution where the unnormalized weights are the customer counts along with the weight, a, for the new table. The pseudocode for the Chinese restaurant process is given in Algorithm to the neural networks representing our agents, we can view the tables as bottleneck units and the customers choosing table as parameter updates which reinforce the use of that unit in accordance with the reward. MModdatch & Abbeel (2018) implicitly assume this when they introduce a reward corresponding to the probability that the emergent lexicon is generated by Chinese restaurant process (a Dirichlet process, in their words).
| for 8 = 1. By analogy
The self-reinforcing property can be expressed informally as: more popular tables get more new customers, keeping them popular. A higher a means that customers are more likely to sit at new table, so the distribution over tables will be more spread out in expectation. The distribution stabi- lizes as the number of iterations goes to infinity as an individual new customers has a diminishing effect the relative size of the weights.
5

---

Under review as a conference paper at ICLR 2022
Algorithm Expectation Chinese Restaurant Process
assert type (alpha) is float and alpha assert type (n_iters) is int and __iters >=
3 assert type (beta) is int and beta 0
4
5 def cample_ papp (weights):
w x_alpha weights.copp ()
k num non.ero (weights)
8
w ak 1| = alpha
9
return sample_categorical (w [w_appha / sum (w_alpha))
10
11
weights array ([1, 0, C, ...1)
12
for in range (n (n_iters):
13
addend array ...])
14
for range (beta):
15
http:/aaaattorrcaaaaallllteleeee (weighttss
16
addend[i] += beta
17
weights += addend
18
return weights sum (weightt)))
4.2
EXPECTATION CHINESE RESTAURANT PROCESS
The key difference between how the Chinese restaurant process and PPO works is the relationship between sampling (i.c., simulating episodes) and updating the weights/paramettess. In each iteration, the regular Chinese restaurant process draws sample based on its weights and updates those weight immediately. PPO, the agent will populate the experience buffer with number of steps (on the order of 100 to 1000) in the environment before performing gradient descent with that buffer to update the parameters. As a result, the parameter update is performed based on weighting across multiple bottleneck units based on how often they were used in the episodes recorded in the experience buffer.
Thus, to the appropriately generalize the Chinese restaurant process, introduce the expectation Chinese restaurant process. In this process, we add a hyperparameter which is a positive integer describing how many samples take from the distribution before updating the weights; the updates are normalized by 3 so the sum all weights still only increases by per iteration. The restaurant analogy breaks down here and independently make a decision, get shrunk to 1 their size, and then sit at their table of choice. The pseudocode for the expectation Chinese restaurant process is given in Algorithm
would have say that in each iteration, 3 customers simultaneously
EXPERIMENTS
Each run of experiment starts by training sender and receiver for fixed number of timesteps for a range of independent variable values. The trained models are then evaluated by initializing 3000 episodes evenly distributed locations using Vogel's method (Vogell 1979). In most settings, the agents are able to achieve a 100% success rate during training and evaluation; we remove any models which do not from consideration. All model for our experiments 26 64 bottleneck units which translates to maximum entropy of 6 bits. Hyperparameters are given in Appendix B
5.1 BIASED SEMANTICS
In our first experiment we demonstrate how shaped rewards which are trivially biased directly distort the semantics of the language, that is, the action associated with each bottleneck unit. We compare three environments, no shaped rewards, the standard shaped reward, and the trivally biased shaped reward. We visualize the semantics of the language with so-called "sea star plots" in Figure Each arm of the sea star is the action taken by the receiver in response to a single bottleneck unit with opacity representing the frequency of use.

---

Under review as a conference paper at ICLR 2022
X
(a) No Shaped Rewards
(b) Standard Shaped Reward
(c) Biased Shaped Reward
Figure 3: Sea star plots for three different settings in the edgeward navigation environment. Each
"sea star" corresponds to an independent language learned in the given setting.
Shaped
No Sheped
Entropy (bits)
5
Entropy
3-
3
2
3
2
16
32
4
8
16
32
Entropy (bits)
World Radius
World Radius
(a) Entropy Histogram
(b) No Shaped Rewards
(c) Shaped Reward
Figure 4
In the setting with no shaped rewards, see actions (i.e., the meanings of the messages) learned featuring 2 to 4 arms pointing in a variety of directions. Since the standard shaped reward takes both dimensions into account, do
any bias in the direction of the learned actions. With the trivially biased reward, though, we see the the learned languages exclusively favor actions near the horizontal axis. In this setting, nothing explicitly prevents the agents from learning vertical actions, but the fact that horizontal dimensions receive the shaped reward makes those actions easier learn.
5.2 CHANGING THE DISTRIBUTION OF ENTROPY
Naturally, a shaped reward which favors certain actions others will bias the semanties of the language. Thus, our second experiment investigates more closely the effect that shaped rewards without this explicit bias can have. Specifically, we investigate the distribution of language en- tropies in the two environments. By entropy we are specifically referring to the Shannon entropy (measured in bits) of the bottleneck units as used in the trained agents' language (as averaged over 3000 quasirandomly initialized episodes).
Entropy an important aspect of language as it represents the upper bound on the information that the language can convey. Languages with higher entropy can yield more precise meaning in their utterances, yet this comes at the cost of being more difficult to learn or acquire as they need a greater variety of training example to be learned.
To investigate the distribution of language entropies, we look at a histogram showing the Shannon entropy of languages belonging to environments with and without shaped rewards. The distributions computed from 2000 independent runs for each reward setting. This is shown in Figure 4p. The presence of shaped rewards shifts the distribution upwards, demonstrating that even shaped rewards which is free trivial bias can still bias the emergent language. A potential explanation these results is discussed and illustrated in Section 5.4

---

Under review as a conference paper at ICLR 2022
Entropy (bits)
Entropy (bits)
5
Entropy (bits)
5-
4-
4-
4-
3
3
2-
1
10
100
1000
40
300
3000
40
300
3000
8
Rollout Buffer Size
Rollout Buffer Size
(a) ECRP
(b) No Shaped Rewards
(c) Shaped Reward
Figure 5
5.3 MASKING ENVIRONMENTAL PARAMETERS
In our final primary experiment, we demonstrate how shaped rewards can mitigate the influence of environmental parameters the entropy of the learned language. This is issue insofar as the presence of shaped rewards make it difficult to observe an emergent property of interest. Specifi- cally, we look how the standard shaped rewards hide the effect of world radius entropy in centerward environment.
In Figures Ap and Ap. we plot the language entropies against different world radii. In both settings, we observe that entropy decreases as the world radius increases, but the setting with no shaped rewards shows much more rapid decrease in entropy. We offer one possible explanation for this effect in Section 5.4] When the only reward signal agents have access to is the base reward, they can only accomplish the task by finding the goal randomly at first; as the size of the environment increases, the chance of finding the goal with random movements decreases and the agent pair often fails to learn a fully successful language at the highest world radii.
5.4 EXPERIENCE BUFFER SIZE
As an explanatory experiment, we demonstrate how changing the size of the PPO experience buffer has significant impact of the entropy of the emergent language. We compare this with the effects we would expect to see according to the model presented in the next section, i.c., the expectation Chinese Restaurant process. In turn use this to explain one mechanism by which shaped rewards can have the observed effects on entropy shown by the previous experiments.
In Figure we show the effect of logarithmic sweep of 3 on entropy of the expectation Chinese restaurant process. We first observe that increasing reduces the variance between distributions yielded by the process since, as 3 increases, the individual updates made in each iterations also reduced in variance. In fact, in the limiting case as 3 - co, the process will always yield the same distribution the update will just be the expectation of sampling from the categorical distribution described by the (normalized) weights (plus a). The second effect is that increasing 3 will decrease the concentration (i.e., increase the entropy), on average, of the distribution yielded from the process. The intuition behind this is that the since each update is less concentrated, the distribution as a whole will be less concentrated as the probability mass will be spread out.
These results can be used to explain. in part, both the effect of shaped rewards and world radius on entropy. First, though, we must establish an analogous correspondence between the expectation Chinese restaurant process and the PPO-learning process. An iteration of the process described in Algorithm [ consists of sampling from the modified categorical distribution (Line 15) and incre- menting the weights (Line 17 In PPO, the sampling corresponds to populating the experience buffer with steps associated with a reward, and the increment operation analogous to PPO per- forming gradient descent the agents using the buffer. Thus, , is analogously increased for PPO when the number of successful episodes per iteration which is dependent both on the size of the experience buffer as well as the environmental factors affecting frequency of success.
8

---

Under review as a conference paper at ICLR 2022
In Figures P and p.. directly vary the size of the experience buffer in environments with and without shaped rewards. Both environments replicate the correlation between //buffer size and entropy, though the decrease in variance is less distinct as buffer size increases.
Having established this correlation, we can offer a potential explanation to the experiments involving world radius well as the distribution of entropies between the environments with and without shaped rewards. Shaped rewards effectively increase 3 since it assigns reward signal to every step whereas the base reward-only environment requires successful episode. This effect is exacerbated when the world radius is increased, the base reward-only environment yields rewards less frequently in the beginning because randomly finding the goal is less likely, This effectively decreases 5 which corresponds to a lower entropy and higher variance.
6 CONCLUSION
We have, then, demonstrated the pitfalls that shaped rewards present for emergent language research: directly biasing the learned semantics of the language. changing the distribution of an emergent property of the language (i.e., entropy), and masking the emergent effects of other environmental variables. These experiments were performed with with novel navigation environment. This environment allows allows for shaped rewards through multi-step episodes and avoids trivially optimal language by employing a continuous state and action space. In addition to this, we introduced the expectation Chinese restaurant process both to explain our own experimental results and provide foundation for future models of emergent language.
emergent language
The limitations of this work can be illustrated through the future directions that could be taken. First, studying a variety of environments would further characterize what biases shaped rewards introduce into emergent languages. Second, increasing the complexity of environments would greatly expand the range of both emergent properties and types of shaped rewards which could be studied. For example, a rich environment like chess presents many emergent strategies such as the valuation of positions, balance between offense and defense, and favoring certain pieces; shaped rewards could then take the form of rewarding stochastic strategies, evaluating positions with traditional chess engines, or assigning pieces explicit point values. Furthermore, while we only use the expecta- tion Chinese restaurant process as an explanatory model, further work could design experiments to demonstrate its predictive power.
The studies presented in this paper are both exploratory and anticipatory in nature since emergent language research has yet to tackle environments difficult enough require shaped rewards. Nev- ertheless, the field will follow reinforcement learning in tackling progressively more difficult tasks which will present further opportunities for or even require shaped rewards. When this occurs, ac- counting for the biases which are inherent to shaped rewards is imperative to preserving the integrity of emergent language experiments. This work, then, prepares researchers in the field for these future challenges.
REPRODUCIBILITY STATEMENT
The code used in association with this paper is located httts::/exampli com/ repo-namel (in a ZIP file for review). Reproduction instructions are located in the README.. mc performed on 20-thread Intel Core 19-9900X server which they take less than 24 hours to run. No random seeds are recorded or provided as the training process stable and similar results should be observed for any random seed.
file. Experiments
REFERENCES
David J. Aldous. Exchangeability and related topics. In P. L. Hennequin (ed.), École d' d'Éré de Probabilités de Saint-Flour XIII 1983, pp. 1-198, Berlin, Heidelberg, 1985. Springer Berlin Heidelberg. ISBN 978-3-540-3931164
David Blei. The chinese restinsueersstttuataat www.ssstttuuuuuuutuocacccc 2007. BR007,URLLEE /tt.teeeeee.t.........o.oooooonr http:/aww.cccccaeaaaa//////////.

---

Under review as a conference paper at ICLR 2022
Nicolo' Brandizzi, Davide Grossi, and Luca locchi. Rlupus: Cooperation through emergent com- munication in the werewolf social deduction game, 2021.
Kalesha Bullard, Douwe Kiela, Franziska Meier, Joelle Pineau, and Jakob Foerster. Quasi-
equivalence discovery for zero-shot emergent communication, 2021.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences, 118(12). Mar 2021. ISSN 0027-8424, 1091-6490. doi: 10. 10..oss/p 2212//22166669111
2016569118. URL URthttps:: //ww. pnas org/ cerg//tntet/////ee
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Grae- pel. Biases for Emergent Communication in Multi-agent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Asso-
ciates. Inc., 2019. 35557cc60bbdbeedd... 01144444618999..20 849c3880 Abst html
URL hatps::/papprs. wttp::/ss..isss.cppppaeeeeep wwtpe//ppee/e////aaaeeee
Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3700-3710. Association for Computational Linguistics. Nov aaaalcaam ant www.cnnnoooogggg hology. org D1 9-1384
2019. doi: 10. 10118655/1/DD11-1 1384. URL https: /
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learn- ing to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 2149-2159. Curran Asso-
ciates, Inc., 2017. URL http:://wpccccaaa...eeeeeeee..ps www.ips.ccppppeee httpssccc//apere2///1eee
pdf
Rishi Hazra, Sonu Dixit, and Sayambhu Sen. Zero-shot generalization using intrinsically moti- vated compositional emergent protocols. Visually Grounded Interaction and Language Workshop, NAACL, 2021.
Eric Jang, Shixian Gu, and Ben Poole. Categorical reparameterization with gumbel-softtax. In Proceedings of the 2017 International Conference Learning Representations (ICLR), 2017.
URL httpttttt////pereeeeeeee
net /ett/foornm ?itimmid==reee 1d=rke3y85ee
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, Dj Strouse, Joel Z. Leibo, and Nando De Freitas. Social influence as intrinsic mo- tivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049. PMLR, May 2019. 9a. html ">http://proceedii
URI. <ahref="http:
mtre mlr. wwt.ppess/v/// http:///ww9//aauussssssaaaaaa
Ivana Kajić, Eser Aygün, and Doina Precup. Learning to cooperate: Emergent communication in multi-agent navigation. In 42nd Annual Meeting of the Cognitive Science Society, pp. 1993-1999, Toronto, ON, 2020. Cognitive Science Society.
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimiza- tion in emergent languages. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 5220-5230. PMLR, 13-18 Jul 2020. URL http: httt:://proceeding mlr. 3ress//119//harrttooov
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge "nat- urally" in multi-agent dialog. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017. doi: 10.18653/11/111 1321 URL http: //d: doi.
org/100 (tgg//0.1663//1/111--01
Angelkki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations, 2018. URL httpt::tt///// stttps://openzeview net net/forum?? /ff.rmoorriiiooo
10

---

Under review as a conference paper at ICLR 2022
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the 2017 International Conference on Learning Representations (ICLR), 2017. URL atttp:://operreeee http://wwewee..eeeoooootaaaaaaa net/ forum?ide
$1 11
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 0024-8035.. Curran Associates, Inc., 2019. URL http: http://ppprrssuuuureeeee wttppuurrpss...pprr 9015 -----pyooooc----ippe--- wnw.annimmmaaiiieee senttivve-tyyy high htw.hhpperrrormmneeeee-ee---eee-
pdf
Antonin Raffin, Ashley Hill, Maximilian Emestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3, https::/ggitii ctm./D......oc..ee.....
RM/ stwble-ablleeeee www.saaabbbbbeeeeeeeeeeeaeeeeeee baselines3 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/00044577, 2000.
Helmut Vogel. A better way to construct the sunflower head. Mathematical Biosciecces, 44(3):179- 189, 1979. ISSN 0025-5564. http://///oaaoooo/oooo0000025500 7/www.co
URL https:
Eric Wiewiora. Reward Shuping, pp. 863-865, Springer US, Boston, MA, 2010. ISBN 978-0- 387-30164-8. doi: 10. 10.1007/788-0-337-333------
URL htttt:tt///do ortp/00.1//000077700
778-0-387-30144-
731
Ronald J Williams. Simple statistical gralient-following algorithms for connectionist reinffrcemetiit learning. Machine learning, 8(3):229-266, 1992.
A ENVIRONMENTS OF RELATED WORK
Havryloy & Titov (017)) Standard signalling game where the observations are natural images from Microsoft COCO. There is trivially optimal language since the information being commu- nicated is simply which image is being shown. There are natural image classes (e.g., cat vs. dog), but they are not necessarily the features of the images which the agents need to communicate. The standard signalling game is single-step.
Kottur et al. (0117) "Task and talk" dialog game where one agent must ask questions of the other agent to learn the attributes of an object. Specifically, the questioner has a set of attributes it must learn (the objective) about the object that only the answerer can see. The desired language has the questioner using a unique message to specify property and the answerer responding with unique message to specify the value for that property such that messages are always in one-to-one correspondence what they are communicating. Such language is trivially optimal. Multi-round dialog environments are inherently multi-step.
Mordatch & Abbeel (0188) Collaborative navigation game where multiple agents tell each other whither to move in a 2D environment. The environment consists agents and landmarks distin- guisted by color. Each agent is given "goal"; ; for example, the blue agent might receive "red agent go to green landmark" (represented as a categorical feature vector) which the blue agent then has to communicate to the red agent. The agents have vocabulary large enough to assign unique "word" to each concept being expressed in one-to-one correspondence, yielding a trivially optimal language. Each episodes consists of multiple timesteps which can each have their own utterance.

---

Under review as a conference paper at ICLR 2022
Lazaridou et al. (2018) An image-based signalling game using rendered images from MuJoCo. The information being communicated is the shape and color of an object where set of shapes and colors are both small. Since the number of words available is at least big as the cardinalities of the possible to construct a trivially
sets and the sender is able to use multiple words per utterance, optimal langauge. The standard signalling game is single-step.
Kharitonov et al. 2020) A binary vector-based signalling game with shared context. The goal of the signalling game is communicate the bits of binary vector which are not shared by the sender and receiver. The messages consist of a single symbol where the number of unique symbols greater than the combination of bits communicated, thus there trivially optimal langauge where one unique symbol is assigned to each possible combination of unshared bits. The standard signalling game is single-step.
Chaabouni et al. (2021) A signalling game with 330 different colors represented real vector in CIELAB color space. The game is set up so that colors which are nearby never appear in the same distractor set which encourages solutions which
some arbitrary region of colors. Due this "fuzzy" concept of solution (i.e., not using 330 distinct words for each color), we consider this environment have a trivially optimal solution. The standard signalling game is single-step.
B HYPERPARAMETERS
B. DEFAULT CONFIGURATION
Environment
Type: centerward
World radius: 9
Goal radius: 1
Max steps per episode: x world.radius
Agent Architecture
Bottleneck size: 64
Architecture; sender is 1-3 and receiver 5; bottleneck size is
1. Linear w/ bias: in, 32 out
Tanh activation
Linear bias: in, out
Gumbel-Softmax: N in, N out
Linear w/ bias: N in, out (action) and 1 out (value)
Bottleneck GGumbel-SSfftaxy temperature: 1.5
Weight initialization: 24 (-V (-)) where n the input size the layer (Py 1.10 default)
Optimization
Reinforcement learning algorithm:
proximal policy optimization
Default hyperparameters used unless otherwise
noted:
https://
he wttp://add.....oe///.oo.ooo///// http:///ww..mmmmm//////////////e /po. htm
readthedocs
Training steps: 1 x 105
Evaluation episodes: x 103
Learning rate: 3 x 10-3
Experience buffer size: 1024
12

---

Under review as a conference paper at ICLR 2022
Batch size: 256
Temporal discount factor (y): 0.9
B.2 EXPERIMENT-SPECING CONFIGRATTONS
Note that we define a logarithmic sweep from x to (inclusive) with n steps to be defined by Equation
n-1
E (0, ..... 1}
(3)
Biased Semantics
Type: edgeward
World radius:
Goal Radius: 8
Experience buffer size: 256
Batch size: 256
Changing the Distribution of Entropy
Number of independent runs per configuration: 2000
Experience buffer size: 256
Batch size: 256
Masking Environmental Parameters
World radius: logarithmic sweep from 2 40 with 300 steps
Experience Buffer Size
Experience Buffer Size: logarithmic sweep from 32 to 4096 with 200 steps (floor function applied experience buffer size is integer-valued)
Training steps: 2 x 105
Expectation Chinese Restaurant Process
This is not emergent language experiment and just consists of running the mathematical model.
5
3: logarithmic sweep from 1 to 1000 with 10 000 steps (floor function applied, though plotted without floor function)
Number of iterations per run: 1000
13


Reviews:
[{"review_id": "pzX2ws-zrs3", "reviewer": "Reviewer_5smG", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The question of how inductive biases can affect the solutions of emergent communication algorithms seems interesting and potentially impactful. However, I think the current manuscript has too many fundamental issues that prevent it from making progress on this problem. \n1. The contributions are unclear. The current study is conducted in a very informal manner---without a detailed problem description or an explanation of the empirical methodology---which makes it difficult to identify the main claims, understand the connection between the intended claims and the empirical conclusions, and ultimately leads me to question the correctness and significance of the whole study.  \n2. It is unclear whether the presented data contains a positive result that would make a significant contribution to research community. The experiment section presents data that seem mostly the same between baseline and comparator.\n3. The manuscript does not contain enough details to reproduce its results. Several plots are presented in figures 3, 4, and 5 without a detailed explanation of how they were generated or what their implications may be. It feels like readers are supposed to trust they are correct without question.\n\n#### Questions\n* What are the specific research questions this work intends to address?\n* How is the presented algorithm used in the experiments?\n* What purpose does the Chinese Restaurant Process serve in this work?\n* How exactly are rewards shaped in the experiments? \n* How are the experiments supportive of the general claims about reward shaping and prior information that you wish to make?\n", "summary_of_the_paper": "This paper aims to investigate the negative implications of shaped rewards in an emergent communication learning setting. The authors attempt to view PPO as a Chinese Restaurant Process, and although this link has little relevance to their empirical study, various differences with and without the use of shaped rewards are presented from experience in a single, two-dimensional navigation domain.\n", "summary_of_the_review": "I think after addressing these questions in earnest, and giving the study a major overhaul, there could be some very interesting findings from this line of questioning, of interest to both the RL and Emergent Communication communities. But given the major issues it currently has, I will not reccomend for this paper for acceptance. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "PTdnPdzdnu2", "reviewer": "Reviewer_2p5d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main issue with this paper is that the core contributions may not be particularly useful to people working in emergent communication for different environments and tasks. The general claim that \"shaped rewards bias emergent language\" seems well known in the literature. Many papers have been using shaped rewards in some form or another to try to bias the language towards certain results—usually the target is trying to increase the systematicity and compositionality of the language, e.g. with auxiliary losses ([Chaabouni et al., 2019](https://arxiv.org/abs/1905.12561); [Luna et al., 2020](https://arxiv.org/abs/2004.03868); [Mihai and Hare, 2021](https://arxiv.org/abs/2106.02067), among many others). This work does not pretend that the shaped rewards are not biasing the language—indeed, **the goal is to bias the language**, and the central question is *what shaped rewards lead to the kinds of qualitative language behavior we desire in emergent communication*.\n\nThe authors suggest that shaped rewards obscure our ability to study how emergent languages arise from \"basic principles.\" It's not precisely clear to me what these \"basic principles\" are (every study likely has some sort of contentious design choice somewhere), but there also seems to be some consensus that languages developed with minimal guidance/shaping are successful, but are generally uninteresting and unhelpful (e.g. [Kottur et al., 2017](https://arxiv.org/abs/1706.08502)), both for people interested in computational simulations of linguistic evolution and for those ultimately interested in connecting emergent languages to real ones.\n\nThe authors show that, for a simple multi-agent navigation environment, adding shaped rewards (i.e. auxiliary signals) to the environment bias the emergent language. Granted, since this environment is more slightly more complex than some previous work (insofar as it involves communication over multiple timesteps), so there may be some takeaways for researchers looking at emergent communication in multi-step settings. Such shaped rewards look different than the kinds of regularizers and auxiliary losses used in non-multi-step games, and perhaps the PPO experience buffer observation will be useful for people who use such experience buffers in other studies. Despite this, however, my main concern is that the analyses in this paper seem to have limited utility and applicability outside of the specific navigation environment explored.\n\nIt would have been good to see a wider variety of environments/tasks explored, including those that have already been explored in the literature. Otherwise it's unclear what other researchers will get from this paper, unless they're using the exact same navigation task used in this paper (assuming that the insight that \"shaped rewards bias emergent language\" is already known to them, which I believe is fairly well-established). I believe a more useful version of this paper would show how consistent kinds of shaped rewards (e.g. denser rewards) consistently bias the language across a variety of tasks, thereby substantiating more general claims and considerations that may be useful to people studying multi-agent communication regardless of the environment.\n\n# Strengths\n\n- Some insights into how auxiliary rewards and training parameters bias emergent communication, esp. in RL, may be useful considerations for future researchers. It does paint a worrying picture that something as simple as PPO experience buffer size has dramatic effects on the entropies of the learned languages, which calls into question how stable conclusions drawn from studies of RL communication can be. Though it would be nice to see other environments where these hypotheses also hold, especially environments that have previously been used in multi-agent communication (of which there are now several RL environments have that been explored).\n\n# Weaknesses\n\n- Limited evaluation—no other environments explored.\n- The investigation of how shaped rewards change the entropy/semantics of the language is interesting, but seems limited to this environment (especially, for example, with world radii). I don't think it's particularly surprising that shaping rewards change the dynamics of the language, particularly with respect to their sensitivity to features that are specific to the environment (e.g. world radii). In general we must watch out for confounding factors when modifying our environment across all studies in emergent communication. It's not clear to me how the confounding factors in this environment are teaching us something new about how to approach emergent communication studies across all environments.\n- I feel like the term \"basic principles\", \"first principles\" is thrown around a lot and it's unclear to me what precisely this means. Some more time could be spent explaining what the authors mean, e.g. end of section 2.2 p1, explanation of Bullard et al. 2021 in intro p1,. In fact I expect many design choices made in previous studies on emergent communication argue that their particular reward/choice of game/etc are already \"first principles\".\n- The sea star plots are confusing to me, and more time should be spent explaining them. Why are there only 3-6 arms per plot if there are 64 bottleneck units? Does this mean the teacher does not use the other units to communicate, or do the other units not have interpretable effects on the receiver's trajectory? Does the receiver **always** take an action in a deterministic direction upon seeing? Surely there's some spread/variance—in which case that should be evident in the graph somewhere. Or rather, are all 6 sea star plots responses to different units in a single language? But then the caption \"Each sea star corresponds to an independent language\" wouldn't be correct...\n- Writing seems incomplete in some places, with obvious omissions. Figures should have captions. Intro: \"there is a risk that the design choice simply mirrors\" repeated.\n- \"We remove any models which do not [achieve a 100% success rate] from consideration.\" Without any additional details, this seems highly suspect; ideally you'd find a setting where the agents consistently reach 100% success rate (else the reproducibility statement claim that \"the training process is stable\" is false); at a minimum, you should describe what proportion of agents fail to converge, and whether the languages in this discarded portion differ qualitatively from the included languages in a way that might bias the results examined here.\n- I think some of the specifications in Table 1 are incorrect. E.g. Lazaridou 2018, task is considered \"trivial\" but Lazaridou et al. also explore a real pixel reference game where the task is to communicate over real images. Authors argue this task is trivial since one can learn the underlying features (of shape and color), but extracting these features is very much a non-trivial task. In the same way, then, the Havrylov and Titov paper can also be considered trivial, in that one should just be able to learn the underlying MSCOCO objects that the dataset is sampled from and those objects.\n- The organization of this paper seems off. I feel like section 4 (the CRP) should be introduced around section 5.4, since it is completely ignored until then. The relation between section 5.4 and the rest of the paper could also be made more clear—this investigation seems completely tangential to the question of how shaped rewards bias the environment, and the experiment is not mentioned in abstract.\n- Figure 4: the connection between \"lower entropy at higher world radii\" needs to be better connected with the claim that \"agent pair often fails to learn a fully successful language at the highest world radii\". What does \"fully successful language\" mean? Don't agents get 100% task accuracy on this task---then why isn't the language \"successful\"?\n- Following the prev. question, in terms of actually exploring the emergent language, the only metric really investigated here is entropy. In any experiment I could likely find some metric for the resulting language that changes wildly if I modify training hyperparam or environment reward. Authors should therefore more clearly justify why entropy is the only relevant part of the story here. Why is it good or bad that shaped rewards may lead to a language with higher entropy than is expected from \"basic principles\"? There are a lot of other metrics, e.g. related to compositionality ([Brighton and Kirby, 2006](https://pubmed.ncbi.nlm.nih.gov/16539767/)), or positive signaling/listening; ([Lowe et al., 2021](https://arxiv.org/abs/1903.05168)), that we might care about, and would be good to measure here. Why should we be so concerned/worried about changes to our environment affecting the entropy of the resulting languages?\n\n# Questions\n\n- What does it mean to \"emerge from basic principles?\"\n- Are the experiments in Section 5 done in the edgeward environment, centerward environment, or both? Are there qualitative differences in the behavior of both settings, and if not, what is the motivation for exploring both settings?\n- If i'm understanding this correctly, It's interesting that the agent messages encode horizontal commands (Figure 3c) even though horizontal information is given in the biased reward. I would expect that agents learning to encode vertical actions would help fill in the deficiencies of the environment reward. Though presumably the vertical actions are harder to learn.\n- How often does the sender send messages to the receiver - at every timestep, or just once at the beginning of each episode?\n- Why is the PPO experience buffer experiment an \"alternative explanation\" for the differing entropies observed in the paper? Couldn't experience buffer size and reward shaping affect entropy (or other language metrics) in orthogonal ways?\n\n# Minor\n\n- 5.4 \"model presented in the next section\"...what next section?", "summary_of_the_paper": "This paper studies the effect of auxiliary rewards in an environment for a multi-agent communication task. The task is for a sender to navigate a receiver to the center of some environment, in worlds with varying sizes, and must develop a discrete language (consisting of 1 of 64 symbols) to solve this task.\n\nThe authors argue that seemingly innocuous (\"unbiased\") shaped rewards in the environment have significant, measurable effects on the learned languages, both qualitatively and quantitatively. The authors both visualize the effect of shaped rewards on the emergent language, and measure the change in entropy to the languages caused by the denser shaped rewards. For example, adding denser rewards increases the entropy (and presumably information) communicated by the language; denser rewards prevent a decrease in entropy for languages generated for larger (more difficult) tasks, suggesting that information bandwidth remains high in the shaped reward setting.\n\nFinally, the authors suggest an alternative hypothesis for the differential entropies observed by language in this game: the size of the experience replay buffer in their PPO algorithm, which roughly corresponds to the frequency of updating the semantics of the learned language. There is some connection drawn here to a batched CRP with varying batch size, though it's not super clear to me how strong this connection is to PPO learning and why this CRP is the right way of thinking about experience buffer sizes.\n\nThe core argument of the paper, then, is to more carefully consider how environmental rewards might change the semantics of the learned language and mask other factors of interest in studies of emergent communication.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KAkBp7l0XjZ", "reviewer": "Reviewer_uFB4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary\n\nI really liked this paper. There are a bunch of typos /repeated words, such as 'allow allow', but I felt the structure and the paper was good, the pacing worked well for me, the explanations were clear. I did have a slight hiccup over the use of both PPO and Gumbel, which I eventually realized is because the sender-receiver network as a unit has to operate in an environment which itself gives a stochastic reward, so a small diagram illustrating the relationship between the sender agent, the receiver agent, and the environment/reward might faciltitate this point slightly.\n\nI felt a little odd that the paper both introduces PPO as a way to handle stochastic reward, I mean, relative to other emergent communication papers, and then draws conclusions based only on using PPO. But PPO is a common approach to stochastically rewarding environments, and the conclusions, experimental, and theoretical work, seemed solid to me.\n\nI felt the analogy between ECRP and PPO was slightly stretched to me, but I could more or less see the connection betweeen the two. I think the point I felt least confortable with was comparing positively rewarded episodes in the PPO buffer with the individual 1/beta samples in ECRP. I am unfamiliar with PPO, and would have appreciated some clarification why we are only considering positively-rewarded episodes in this comparison. Does PPO only store positively rewarded episodes in the buffer? Or are considering that only the positively-rewarded episodes will modify the weights, in a particular direction? So, some clarification on your exact thinking on this point, and exactly how PPO works, on this point, would be appreciated please.\n\n## Good points\n\n- use of PPO in emergent communication is new\n- empirical exploration of effect of reward shaping on the entropy of the emergent language is new\n- theoretical analysis of why the reward shaping affects the entropy of the emergent language is new\n\n## Bad points\n\n- a few typos/repeated words (e.g. 'allow allow')\n- could do with a diagram showing the relationship between the sender model, the receiver model, and the task/reward (and in particular how Gumbel and PPO fit into this)\n\n## Questions for author\n\nNone. The work seemed rather complete to me.\n\n## Notes\n\nNotes I made whilst reading the paper:\n\n### Abstract\n\nby the end of the abstract I'm excited to see what the paper is going to show.\n\n### Introduction\n\nseems rather specialized (experience buffer size), but if it does provide both empirical evidence and theory, then, ok :)\n\n### 2.1 taxonomy\n\nconcept and analysis of 'trivially optimal language' is interesting\n\ntable 1: nit: can we make it so we have both columns to be 'yes', rather than one aiming for 'no' and one for 'yes'.\n- maybe a way of phrasing trivial in the other way around. ('non-trivial'? :) )", "summary_of_the_paper": "The paper examines the effect of reward shaping in the context of emergent communication, and specifically with respect to the Shannon entropy of the emerged language. The paper showed that shaping the rewards can lead to higher entropy than training with just the base reward. We care about the entropy because this relates to how expressive the language is. There are existing works, such as Kharatinov et al 2020, which examine the relationship between task/environment and the resulting language entropy.\n\nThe paper uses PPO as its main outer-level optimizer, which it uses because the task is a multi-step task which might be challenging to learn with other algorithms such as REINFORCE.\n\nThe paper provides theoretical analysis of the effect of reward shaping on entropy by considering an analogy between PPO learning and a process they call the 'Extended Chinese Restaurant Process' (ECRP), where at each step, the customer can be divided into beta parts, and each of the beta bits of customers (which are now each of size 1/beta) takes a seat. This is presented as being analogous to how PPO first store multiple episodes over a buffer, and then updates the weights of the network, using the results of the whole buffer; then rinses and repeats.\n\nExperimental results are presented which show that making the task harder/longer (by increasing the world radius) results in a fall of the entropy of the resultant emergent language, but not in the presence of reward shaping.\n\nThe paper asserts that whilst multi-step tasks, and PPO, are not much used in emergent communication literature currently, but it seems plausible that this will become the case in the future, and this paper is a small exploratory step in that direction.", "summary_of_the_review": "Like the paper\n- use of PPO in emergent communication is new\n- empirical exploration of effect of reward shaping on the entropy of the emergent language is new\n- theoretical analysis of why the reward shaping affects the entropy of the emergent language is new", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2r7fw3hZNzR", "reviewer": "Reviewer_oZZ8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe key contribution of the paper is the findings on the impacts of shaped rewards to the the emergent language:\n1. shaped rewards can explicitly bias the semantics of the learned language;\n2. shaped rewards change the entropy of the learned language;\n3. shaped rewards masks the potential effects of other environmental variables of interest.\n\nWeaknesses\n\nThe findings are only confirmed in a specific setting, a simple sender-receiver navigation game with two specific bias rewards, Euclidean distance and horizontal distance.\n\nI find the use of Chinese restaurant processes as an analogy not necessary and confusing. In some places, the analogy breaks down as stated \"The restaurant analogy breaks down here as we would have to say that in each iteration, β customers simultaneously\nand independently make a decision ...\"\n", "summary_of_the_paper": "This paper investigates emergent language research, whose goal is to study language as it emerges from the inherent properties of the environment, language, and agents. This is typically in the context of reinforcement learning such that the phenomena arises from the pressure to maximize reward. This paper considers two types of rewards. One directly comes from the task. The other is shaped rewards which makes learning easier. This paper mainly investigates the impact of shaped reward on the emergent language. Using a simple sender-receiver navigation game, the paper shows that shaped rewards can explicitly bias the semantics of the learned language, change the entropy of the learned language, and mask the potential effects of other environmental variables of interest.", "summary_of_the_review": "The paper studies an interesting topic, the impacts of rewards on the properties of emergent language. However, I find the contribution is very light and limited in a very specific setting. \n\nIt considers only a simple sender-receiver navigation game with two specific bias rewards, Euclidean distance and horizontal distance. The results are also very straightforward and specific to the setting:\n1. \"Since the standard shaped reward takes both dimensions into account, we do not see any bias in the direction of the learned actions. With the trivially biased reward, though, we see the the learned languages exclusively favor actions near to the horizontal axis. In this setting, nothing explicitly prevents the agents from learning vertical actions, but the fact that horizontal dimensions receive the shaped reward makes those actions easier to learn.\"\n\n2. \"we plot the language entropies against different world radii. In both settings, we observe that entropy decreases as the world radius increases, but the setting with no shaped rewards shows a much more rapid decrease in entropy. We offer one possible explanation for this effect in Section 5.4. When the only reward signal agents have access to is the base reward, they can only accomplish the task by finding the goal randomly at first; as the size of the environment increases, the chance of finding the goal with random movements decreases and the agent pair often fails to learn a fully successful language at the highest world radii.\"\n\nThe current results can be summarized in a workshop paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2R1N5cf19Tn", "reviewer": "Reviewer_jcC3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well written and motivated, the experiments in the paper are performed with rigor. All the concepts are explained in the paper making the work easily understandable.\n\nWeakness:\nThe argument for using CRP as a way to explain the RL behavior is not convincing for me, since in the former case the action is picking a table, and every action of picking a table that has already been picked leads to a \"reward\" but in the case of RL, the action doesn't directly correspond to rewards. And the action and state space for both are different.\n\nThe analysis and the results obtained with the agent’s entropy and behavior differences can be said for RL problems in general and not just to emergent language learning. I am not familiar if such analysis has been performed on RL in general. Thus similar analysis on the effects could be carried out in RL in general. The related work section also doesn't include literature on analyzing reward shaping [1] .\n\nThe paper has some minor grammatical mistakes that can be fixed with revisions.\n\nThe description of the model is confusing in the appendix, the author should use the standard method deep learning community describes a network.\n\n[1] M. Grzes and D. Kudenko, \"Theoretical and Empirical Analysis of Reward Shaping in Reinforcement Learning,\" 2009 International Conference on Machine Learning and Applications, 2009, pp. 337-344, doi: 10.1109/ICMLA.2009.33.", "summary_of_the_paper": "The paper looks into finding effects of reward shaping on emergent language learning with RL. The work shows the difference via analyzing entropy and the behavior of the learned agent. It draws similarities between the Chinese restaurant process and RL. And shows that some of the differences between the behavior of the agent learned with and without reward shaping can be explained by experience buffer size used in the RL learning algorithm. The work uses a new simple navigation task for the study.", "summary_of_the_review": "- The work has not taken into consideration the existing literature on analyzing reward shaping on RL. \n- The validity of CRP as a way to explain the behaviors is well-motivated and supported in the paper.\n- Since the experiments are performed in a simple environment, experiments on a complex task should make that paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers eventually agreed on rejection. The highest scoring reviewer agreed their interpretation of the framing of the paper caused their initial high-score, where as the other reviewers took a totally different view on the papers contribution. The authors agreed that the text of the paper was not clear in this regard. And the high scoring reviewer downgraded their score and suggested a different pitch.

Much of the reviews focused on how the paper includes a single handcrafted environment for empirical evaluation, and missing related work on reward shaping. In the AC's view (and several of the reviewers said this too) the simple observation "non-obvious shaped rewards bias language" indeed begs of a broader study across a variety of environments.

Whether more experiments are needed or if this work can be reshaped such that one existence proof experiment is enough does not need to be resolved here; the paper in its current form needs significant changes.
Decision: Reject