Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper looks into finding effects of reward shaping on emergent language learning with RL. The work shows the difference via analyzing entropy and the behavior of the learned agent. It draws similarities between the Chinese restaurant process and RL. And shows that some of the differences between the behavior of the agent learned with and without reward shaping can be explained by experience buffer size used in the RL learning algorithm. The work uses a new simple navigation task for the study.
main_review: Strengths:
The paper is well written and motivated, the experiments in the paper are performed with rigor. All the concepts are explained in the paper making the work easily understandable.

Weakness:
The argument for using CRP as a way to explain the RL behavior is not convincing for me, since in the former case the action is picking a table, and every action of picking a table that has already been picked leads to a "reward" but in the case of RL, the action doesn't directly correspond to rewards. And the action and state space for both are different.

The analysis and the results obtained with the agent’s entropy and behavior differences can be said for RL problems in general and not just to emergent language learning. I am not familiar if such analysis has been performed on RL in general. Thus similar analysis on the effects could be carried out in RL in general. The related work section also doesn't include literature on analyzing reward shaping [1] .

The paper has some minor grammatical mistakes that can be fixed with revisions.

The description of the model is confusing in the appendix, the author should use the standard method deep learning community describes a network.

[1] M. Grzes and D. Kudenko, "Theoretical and Empirical Analysis of Reward Shaping in Reinforcement Learning," 2009 International Conference on Machine Learning and Applications, 2009, pp. 337-344, doi: 10.1109/ICMLA.2009.33.
summary_of_the_review: - The work has not taken into consideration the existing literature on analyzing reward shaping on RL. 
- The validity of CRP as a way to explain the behaviors is well-motivated and supported in the paper.
- Since the experiments are performed in a simple environment, experiments on a complex task should make that paper stronger.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigates emergent language research, whose goal is to study language as it emerges from the inherent properties of the environment, language, and agents. This is typically in the context of reinforcement learning such that the phenomena arises from the pressure to maximize reward. This paper considers two types of rewards. One directly comes from the task. The other is shaped rewards which makes learning easier. This paper mainly investigates the impact of shaped reward on the emergent language. Using a simple sender-receiver navigation game, the paper shows that shaped rewards can explicitly bias the semantics of the learned language, change the entropy of the learned language, and mask the potential effects of other environmental variables of interest.
main_review: Strengths

The key contribution of the paper is the findings on the impacts of shaped rewards to the the emergent language:
1. shaped rewards can explicitly bias the semantics of the learned language;
2. shaped rewards change the entropy of the learned language;
3. shaped rewards masks the potential effects of other environmental variables of interest.

Weaknesses

The findings are only confirmed in a specific setting, a simple sender-receiver navigation game with two specific bias rewards, Euclidean distance and horizontal distance.

I find the use of Chinese restaurant processes as an analogy not necessary and confusing. In some places, the analogy breaks down as stated "The restaurant analogy breaks down here as we would have to say that in each iteration, β customers simultaneously
and independently make a decision ..."

summary_of_the_review: The paper studies an interesting topic, the impacts of rewards on the properties of emergent language. However, I find the contribution is very light and limited in a very specific setting. 

It considers only a simple sender-receiver navigation game with two specific bias rewards, Euclidean distance and horizontal distance. The results are also very straightforward and specific to the setting:
1. "Since the standard shaped reward takes both dimensions into account, we do not see any bias in the direction of the learned actions. With the trivially biased reward, though, we see the the learned languages exclusively favor actions near to the horizontal axis. In this setting, nothing explicitly prevents the agents from learning vertical actions, but the fact that horizontal dimensions receive the shaped reward makes those actions easier to learn."

2. "we plot the language entropies against different world radii. In both settings, we observe that entropy decreases as the world radius increases, but the setting with no shaped rewards shows a much more rapid decrease in entropy. We offer one possible explanation for this effect in Section 5.4. When the only reward signal agents have access to is the base reward, they can only accomplish the task by finding the goal randomly at first; as the size of the environment increases, the chance of finding the goal with random movements decreases and the agent pair often fails to learn a fully successful language at the highest world radii."

The current results can be summarized in a workshop paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper aims to investigate the negative implications of shaped rewards in an emergent communication learning setting. The authors attempt to view PPO as a Chinese Restaurant Process, and although this link has little relevance to their empirical study, various differences with and without the use of shaped rewards are presented from experience in a single, two-dimensional navigation domain.

main_review: The question of how inductive biases can affect the solutions of emergent communication algorithms seems interesting and potentially impactful. However, I think the current manuscript has too many fundamental issues that prevent it from making progress on this problem. 
1. The contributions are unclear. The current study is conducted in a very informal manner---without a detailed problem description or an explanation of the empirical methodology---which makes it difficult to identify the main claims, understand the connection between the intended claims and the empirical conclusions, and ultimately leads me to question the correctness and significance of the whole study.  
2. It is unclear whether the presented data contains a positive result that would make a significant contribution to research community. The experiment section presents data that seem mostly the same between baseline and comparator.
3. The manuscript does not contain enough details to reproduce its results. Several plots are presented in figures 3, 4, and 5 without a detailed explanation of how they were generated or what their implications may be. It feels like readers are supposed to trust they are correct without question.

#### Questions
* What are the specific research questions this work intends to address?
* How is the presented algorithm used in the experiments?
* What purpose does the Chinese Restaurant Process serve in this work?
* How exactly are rewards shaped in the experiments? 
* How are the experiments supportive of the general claims about reward shaping and prior information that you wish to make?

summary_of_the_review: I think after addressing these questions in earnest, and giving the study a major overhaul, there could be some very interesting findings from this line of questioning, of interest to both the RL and Emergent Communication communities. But given the major issues it currently has, I will not reccomend for this paper for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the effect of auxiliary rewards in an environment for a multi-agent communication task. The task is for a sender to navigate a receiver to the center of some environment, in worlds with varying sizes, and must develop a discrete language (consisting of 1 of 64 symbols) to solve this task.

The authors argue that seemingly innocuous ("unbiased") shaped rewards in the environment have significant, measurable effects on the learned languages, both qualitatively and quantitatively. The authors both visualize the effect of shaped rewards on the emergent language, and measure the change in entropy to the languages caused by the denser shaped rewards. For example, adding denser rewards increases the entropy (and presumably information) communicated by the language; denser rewards prevent a decrease in entropy for languages generated for larger (more difficult) tasks, suggesting that information bandwidth remains high in the shaped reward setting.

Finally, the authors suggest an alternative hypothesis for the differential entropies observed by language in this game: the size of the experience replay buffer in their PPO algorithm, which roughly corresponds to the frequency of updating the semantics of the learned language. There is some connection drawn here to a batched CRP with varying batch size, though it's not super clear to me how strong this connection is to PPO learning and why this CRP is the right way of thinking about experience buffer sizes.

The core argument of the paper, then, is to more carefully consider how environmental rewards might change the semantics of the learned language and mask other factors of interest in studies of emergent communication.
main_review: My main issue with this paper is that the core contributions may not be particularly useful to people working in emergent communication for different environments and tasks. The general claim that "shaped rewards bias emergent language" seems well known in the literature. Many papers have been using shaped rewards in some form or another to try to bias the language towards certain results—usually the target is trying to increase the systematicity and compositionality of the language, e.g. with auxiliary losses ([Chaabouni et al., 2019](https://arxiv.org/abs/1905.12561); [Luna et al., 2020](https://arxiv.org/abs/2004.03868); [Mihai and Hare, 2021](https://arxiv.org/abs/2106.02067), among many others). This work does not pretend that the shaped rewards are not biasing the language—indeed, **the goal is to bias the language**, and the central question is *what shaped rewards lead to the kinds of qualitative language behavior we desire in emergent communication*.

The authors suggest that shaped rewards obscure our ability to study how emergent languages arise from "basic principles." It's not precisely clear to me what these "basic principles" are (every study likely has some sort of contentious design choice somewhere), but there also seems to be some consensus that languages developed with minimal guidance/shaping are successful, but are generally uninteresting and unhelpful (e.g. [Kottur et al., 2017](https://arxiv.org/abs/1706.08502)), both for people interested in computational simulations of linguistic evolution and for those ultimately interested in connecting emergent languages to real ones.

The authors show that, for a simple multi-agent navigation environment, adding shaped rewards (i.e. auxiliary signals) to the environment bias the emergent language. Granted, since this environment is more slightly more complex than some previous work (insofar as it involves communication over multiple timesteps), so there may be some takeaways for researchers looking at emergent communication in multi-step settings. Such shaped rewards look different than the kinds of regularizers and auxiliary losses used in non-multi-step games, and perhaps the PPO experience buffer observation will be useful for people who use such experience buffers in other studies. Despite this, however, my main concern is that the analyses in this paper seem to have limited utility and applicability outside of the specific navigation environment explored.

It would have been good to see a wider variety of environments/tasks explored, including those that have already been explored in the literature. Otherwise it's unclear what other researchers will get from this paper, unless they're using the exact same navigation task used in this paper (assuming that the insight that "shaped rewards bias emergent language" is already known to them, which I believe is fairly well-established). I believe a more useful version of this paper would show how consistent kinds of shaped rewards (e.g. denser rewards) consistently bias the language across a variety of tasks, thereby substantiating more general claims and considerations that may be useful to people studying multi-agent communication regardless of the environment.

# Strengths

- Some insights into how auxiliary rewards and training parameters bias emergent communication, esp. in RL, may be useful considerations for future researchers. It does paint a worrying picture that something as simple as PPO experience buffer size has dramatic effects on the entropies of the learned languages, which calls into question how stable conclusions drawn from studies of RL communication can be. Though it would be nice to see other environments where these hypotheses also hold, especially environments that have previously been used in multi-agent communication (of which there are now several RL environments have that been explored).

# Weaknesses

- Limited evaluation—no other environments explored.
- The investigation of how shaped rewards change the entropy/semantics of the language is interesting, but seems limited to this environment (especially, for example, with world radii). I don't think it's particularly surprising that shaping rewards change the dynamics of the language, particularly with respect to their sensitivity to features that are specific to the environment (e.g. world radii). In general we must watch out for confounding factors when modifying our environment across all studies in emergent communication. It's not clear to me how the confounding factors in this environment are teaching us something new about how to approach emergent communication studies across all environments.
- I feel like the term "basic principles", "first principles" is thrown around a lot and it's unclear to me what precisely this means. Some more time could be spent explaining what the authors mean, e.g. end of section 2.2 p1, explanation of Bullard et al. 2021 in intro p1,. In fact I expect many design choices made in previous studies on emergent communication argue that their particular reward/choice of game/etc are already "first principles".
- The sea star plots are confusing to me, and more time should be spent explaining them. Why are there only 3-6 arms per plot if there are 64 bottleneck units? Does this mean the teacher does not use the other units to communicate, or do the other units not have interpretable effects on the receiver's trajectory? Does the receiver **always** take an action in a deterministic direction upon seeing? Surely there's some spread/variance—in which case that should be evident in the graph somewhere. Or rather, are all 6 sea star plots responses to different units in a single language? But then the caption "Each sea star corresponds to an independent language" wouldn't be correct...
- Writing seems incomplete in some places, with obvious omissions. Figures should have captions. Intro: "there is a risk that the design choice simply mirrors" repeated.
- "We remove any models which do not [achieve a 100% success rate] from consideration." Without any additional details, this seems highly suspect; ideally you'd find a setting where the agents consistently reach 100% success rate (else the reproducibility statement claim that "the training process is stable" is false); at a minimum, you should describe what proportion of agents fail to converge, and whether the languages in this discarded portion differ qualitatively from the included languages in a way that might bias the results examined here.
- I think some of the specifications in Table 1 are incorrect. E.g. Lazaridou 2018, task is considered "trivial" but Lazaridou et al. also explore a real pixel reference game where the task is to communicate over real images. Authors argue this task is trivial since one can learn the underlying features (of shape and color), but extracting these features is very much a non-trivial task. In the same way, then, the Havrylov and Titov paper can also be considered trivial, in that one should just be able to learn the underlying MSCOCO objects that the dataset is sampled from and those objects.
- The organization of this paper seems off. I feel like section 4 (the CRP) should be introduced around section 5.4, since it is completely ignored until then. The relation between section 5.4 and the rest of the paper could also be made more clear—this investigation seems completely tangential to the question of how shaped rewards bias the environment, and the experiment is not mentioned in abstract.
- Figure 4: the connection between "lower entropy at higher world radii" needs to be better connected with the claim that "agent pair often fails to learn a fully successful language at the highest world radii". What does "fully successful language" mean? Don't agents get 100% task accuracy on this task---then why isn't the language "successful"?
- Following the prev. question, in terms of actually exploring the emergent language, the only metric really investigated here is entropy. In any experiment I could likely find some metric for the resulting language that changes wildly if I modify training hyperparam or environment reward. Authors should therefore more clearly justify why entropy is the only relevant part of the story here. Why is it good or bad that shaped rewards may lead to a language with higher entropy than is expected from "basic principles"? There are a lot of other metrics, e.g. related to compositionality ([Brighton and Kirby, 2006](https://pubmed.ncbi.nlm.nih.gov/16539767/)), or positive signaling/listening; ([Lowe et al., 2021](https://arxiv.org/abs/1903.05168)), that we might care about, and would be good to measure here. Why should we be so concerned/worried about changes to our environment affecting the entropy of the resulting languages?

# Questions

- What does it mean to "emerge from basic principles?"
- Are the experiments in Section 5 done in the edgeward environment, centerward environment, or both? Are there qualitative differences in the behavior of both settings, and if not, what is the motivation for exploring both settings?
- If i'm understanding this correctly, It's interesting that the agent messages encode horizontal commands (Figure 3c) even though horizontal information is given in the biased reward. I would expect that agents learning to encode vertical actions would help fill in the deficiencies of the environment reward. Though presumably the vertical actions are harder to learn.
- How often does the sender send messages to the receiver - at every timestep, or just once at the beginning of each episode?
- Why is the PPO experience buffer experiment an "alternative explanation" for the differing entropies observed in the paper? Couldn't experience buffer size and reward shaping affect entropy (or other language metrics) in orthogonal ways?

# Minor

- 5.4 "model presented in the next section"...what next section?
summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper examines the effect of reward shaping in the context of emergent communication, and specifically with respect to the Shannon entropy of the emerged language. The paper showed that shaping the rewards can lead to higher entropy than training with just the base reward. We care about the entropy because this relates to how expressive the language is. There are existing works, such as Kharatinov et al 2020, which examine the relationship between task/environment and the resulting language entropy.

The paper uses PPO as its main outer-level optimizer, which it uses because the task is a multi-step task which might be challenging to learn with other algorithms such as REINFORCE.

The paper provides theoretical analysis of the effect of reward shaping on entropy by considering an analogy between PPO learning and a process they call the 'Extended Chinese Restaurant Process' (ECRP), where at each step, the customer can be divided into beta parts, and each of the beta bits of customers (which are now each of size 1/beta) takes a seat. This is presented as being analogous to how PPO first store multiple episodes over a buffer, and then updates the weights of the network, using the results of the whole buffer; then rinses and repeats.

Experimental results are presented which show that making the task harder/longer (by increasing the world radius) results in a fall of the entropy of the resultant emergent language, but not in the presence of reward shaping.

The paper asserts that whilst multi-step tasks, and PPO, are not much used in emergent communication literature currently, but it seems plausible that this will become the case in the future, and this paper is a small exploratory step in that direction.
main_review: ## Summary

I really liked this paper. There are a bunch of typos /repeated words, such as 'allow allow', but I felt the structure and the paper was good, the pacing worked well for me, the explanations were clear. I did have a slight hiccup over the use of both PPO and Gumbel, which I eventually realized is because the sender-receiver network as a unit has to operate in an environment which itself gives a stochastic reward, so a small diagram illustrating the relationship between the sender agent, the receiver agent, and the environment/reward might faciltitate this point slightly.

I felt a little odd that the paper both introduces PPO as a way to handle stochastic reward, I mean, relative to other emergent communication papers, and then draws conclusions based only on using PPO. But PPO is a common approach to stochastically rewarding environments, and the conclusions, experimental, and theoretical work, seemed solid to me.

I felt the analogy between ECRP and PPO was slightly stretched to me, but I could more or less see the connection betweeen the two. I think the point I felt least confortable with was comparing positively rewarded episodes in the PPO buffer with the individual 1/beta samples in ECRP. I am unfamiliar with PPO, and would have appreciated some clarification why we are only considering positively-rewarded episodes in this comparison. Does PPO only store positively rewarded episodes in the buffer? Or are considering that only the positively-rewarded episodes will modify the weights, in a particular direction? So, some clarification on your exact thinking on this point, and exactly how PPO works, on this point, would be appreciated please.

## Good points

- use of PPO in emergent communication is new
- empirical exploration of effect of reward shaping on the entropy of the emergent language is new
- theoretical analysis of why the reward shaping affects the entropy of the emergent language is new

## Bad points

- a few typos/repeated words (e.g. 'allow allow')
- could do with a diagram showing the relationship between the sender model, the receiver model, and the task/reward (and in particular how Gumbel and PPO fit into this)

## Questions for author

None. The work seemed rather complete to me.

## Notes

Notes I made whilst reading the paper:

### Abstract

by the end of the abstract I'm excited to see what the paper is going to show.

### Introduction

seems rather specialized (experience buffer size), but if it does provide both empirical evidence and theory, then, ok :)

### 2.1 taxonomy

concept and analysis of 'trivially optimal language' is interesting

table 1: nit: can we make it so we have both columns to be 'yes', rather than one aiming for 'no' and one for 'yes'.
- maybe a way of phrasing trivial in the other way around. ('non-trivial'? :) )
summary_of_the_review: Like the paper
- use of PPO in emergent communication is new
- empirical exploration of effect of reward shaping on the entropy of the emergent language is new
- theoretical analysis of why the reward shaping affects the entropy of the emergent language is new
review_title: null

Meta-review: All reviewers eventually agreed on rejection. The highest scoring reviewer agreed their interpretation of the framing of the paper caused their initial high-score, where as the other reviewers took a totally different view on the papers contribution. The authors agreed that the text of the paper was not clear in this regard. And the high scoring reviewer downgraded their score and suggested a different pitch.

Much of the reviews focused on how the paper includes a single handcrafted environment for empirical evaluation, and missing related work on reward shaping. In the AC's view (and several of the reviewers said this too) the simple observation "non-obvious shaped rewards bias language" indeed begs of a broader study across a variety of environments.

Whether more experiments are needed or if this work can be reshaped such that one existence proof experiment is enough does not need to be resolved here; the paper in its current form needs significant changes.
Decision: Reject