Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of using population-based training to learn more "natural" languages for communicating between agents.  When two agents (a "sender" and a "receiver") are trained to solve a communication task together, the communication protocol, or language, that they learn often lacks structure, and simply maps arbitrary strings of symbols to different objects.  Surprisingly, training populations agents, and randomly pairing senders and receivers, does not lead to the agents converging to a more significantly structured language than training a single pair of agents.  This paper argues that this is due to the fact that no matter how large the population, the population-level training objective can be maximized by a single receiver policy that simply memorizes the (arbitrary) languages of each sender.

To address this issue, this work modifies the population-based training procedure so that while senders are rewarded for being able to communicate with multiple receivers, each receiver is only required to understand communication from a single sender.  This prevents receivers from directly optimizing their behavior with respect to the average behavior of the senders, while requiring that the senders be interpretable by many distinct receivers.  The experimental results demonstrate that the under this "partitioned" training paradigm, receiver agents are able to correctly interpret communication from senders that they were never paired with during training (which is not observed without partitioning).  Perhaps more importantly, the languages learned by the senders demonstrate significantly more structure (as measured by the topological similarity between the language and the underlying object space), than those learned without this partitioning mechanism.
strength_and_weaknesses: The main contribution of this work is that it provides a convincing explanation for the somewhat counterintuitive phenomenon that population-based training does not more structured and less arbitrary synthetic languages in cooperative communication tasks.  They also present a straightforward solution to this issue through a simple modification of the training process, and demonstrate that this recovers the behavior that was originally expected from population-based training.  There are potential applications of these ideas to other topics such as zero-shot coordination in RL or human-AI interaction more generally.

The main weakness of the work is that the value of learning more "structured" languages in this setting is not fully articulated.  Practical use cases for this training setup, even if only hypothetical, would be helpful in understanding how we should evaluate emergent languages relative to one another.  One example might be to use an emergent language to evaluate to train an agent designed to adapt to existing languages (eg. a particular human language) in an online fashion.  Another might be to design visual communication protocols for user interfaces that are structure in a way that makes it easy for humans to interpret them.  The lack of such motivation means that we cannot determine how useful metrics such as topological similarity really are as proxies for performance in realistic applications.
clarity,_quality,_novelty_and_reproducibility: Overall the work is technically correct, and well presented.  They indicate that source code for their experiments will be available upon publication.  While the work builds heavily on existing methods and results, it presents novel and useful insights about the problem.

A few suggestions for potential improvements:
- the training objectives form general-sum game between the senders and receivers, and so some discussion of whether behaviors reliably converge, or whether cyclic behavior was observed, would be useful.
- Examples of the languages actually learned by the agents would help the reader understand the practical significance of the topological similarity measure.
- a more detailed analysis of whether there is a trade-off between the topological similarity and the accuracy of the learned languages in the ImageNet task would be helpful in understanding whether the partitioned approach to training might be detrimental in some cases.
- A minor point, but using pi to denote both a sender's strategy, and the posterior over objects induced by that strategy is somewhat confusing when reading Section 3, so perhaps a different notation should be considered.
summary_of_the_review: Overall the work provides useful, and somewhat surprising, insights that have not been previously published.  It also demonstrates that a simple modification to population-based training can substantially change the nature of the solutions learned in cooperative communication tasks, which raises the question of whether similar changes might be useful for population based training in other multi-agent settings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper revisits the population-based training method in emergent multi-agent communication. The authors analyze the standard population-based training protocol and point out that the overall objective is the same as that of a single-pair training protocol. The co-adaptation of receivers to senders in the standard method is believed to limit the effect of population. To better make use of population, the paper proposes to partition agents into pairs and one receiver is only trained with its associated sender. The authors claim that such partitioning enhances zero-shot communication with new partners as well as the compositionality of the emergent language. Empirical results in two referential games verify the statements and the effectiveness of the proposed method.
strength_and_weaknesses: Strength：
- This paper is well-written and the figures are demonstrative, making it easy to follow. 
- The proposed method is quite simple yet effective. I appreciate the intuition of reducing co-adaptation to enhance the compositionality of the emergent language. 
- The main results and the ablation study clearly show the effectiveness of the method and the influence of all the components and factors.

Weakness:
- Two communication graphs are adopted in the experiments, as shown in figure 2. However, the fully-connected graph and the circular graph are the two extreme cases. Maybe it is necessary to add a randomly sampled graph as a reference.
- The experiments are all conducted on the referential games, where the receiver only needs to pick a candidate from a set as the prediction of the original object. What if we extend the game into a sequential decision problem? For example, the sender may send messages as the instructions, and the receiver needs to achieve some goals (such as navigation to the goals) by following such instructions. I wonder whether the proposed method can be applied to these games.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: The overall paper is well-written and easy to follow. The figures are clear.
Novelty: The main idea is original but the potential impact maybe is limited.
Reproducibility: The implementation details are available.
summary_of_the_review: The paper is interesting and solid. But the experiments are not convincing enough, requiring comparing with different patterns of graphs, e.g., random graphs. Besides, demonstrating the feasibility of extending the method to a sequential decision-making setting will help the method be more general and impactful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The contribution of the paper is twofold: The paper explores if the “standard training protocol” in multi agent communication exhibits features observed in natural language, namely, if agents are able to communicate with new partners within the same population, and if lager populations tend to develop more structured languages. Authors show that, it is not the case. Moreover, they show that objective function of the standard training approach for multi agents is the same as that of a single pair of agents due to receivers co-adapting to their training partners. Second, authors propose an alternative training approach which partitions sender-receiver pairs which limits co-adaptation of receiver agents. They show that this alternative approach maximizes an objective function that involves two parts:
one function to maximize the “internal communication” between pair of agents and another function to maximize the “mutual intelligibility” across agents. They empirically show that this partitioning approach increases the effect of population size on compositionality and allows emergence of a more structured language, thus allowing to exhibit those features in natural language.



strength_and_weaknesses: 
The authors study an interesting and motivating problem. Analysis is done to explore the objective functions in both cases: standard and  partitioned. They have performed number of experiments to analyze different hypothesis and to showcase the claims and the results are compelling. Provides how the work is related to other work.

I like the organization of the paper (sections containing individual task.) However, the structure can be improved. Here are some suggestions.

In the introduction, adding what you do in each section. This will help the readers to better understand the flow of the paper and your results.
Adding a discussion explaining your results and what the broader connections are. You have explained that "standard procedure" do not showcase the features of the reality. You could consider elaborating on that and explain more on how partitioning allows those features (of course you have explained them- you could consider elaborating and being more precise ), how scaling up to populations can be beneficial and how the way it is done matters. What are the implications and broader connections to natural language? Consider explaining how "simple design choices can have a great impact on the training dynamics and ultimately the effect of population on the emergent language"

Conclusion looks a bit vague. " exhibit some of the properties","Motivated by an analysis of populations at the equilibrium"-be a bit explicit.

Add future work.

All the points made above are suggestions, and I would defer to the authors who best know their work and are in the best position to decide how to make their work more accessible.




clarity,_quality,_novelty_and_reproducibility: The paper could be more clear. 
summary_of_the_review: I believe the authors have obtained some interesting results to explore and answer an interesting problem. However the flow and the explanations could be improved. In my opinion, how the results can be connected at large could be discussed more.
I have made some suggestions to improve the understandability of the paper. After revision paper can be accepted for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper studies emergent communication in multiagent populations. First it shows a surprising negative result: when all agents communicate with each other, the equilibrium communication strategy is independent of population size. Moreover, this strategy need not have the properties of a structured language. However, the paper goes on the demonstrate, if senders and receivers are partitioned in a particular way, then structured language does emerge, and all agents converge on the same mutually intelligible structure.

Strengths:
* Well motivated with interesting background psychology
* Surprising findings that shed light on prior failures of scaling up multiagent communication
* Good theoretical analysis
* Interesting empirical results

Weaknesses:
* The experiments are limited to just a few referential games


justification_for_why_not_higher_score: All reviewers advocate accepting this paper, finding it to be informative and sound. No major weaknesses were identified, but the reviewers do provide many suggestions that could improve the paper. I concur with the consensus and find the paper very interesting. I think poster is the appropriate level given that, although the paper will potentially be impactful, the results are not quite there yet to show if the method is broadly applicable and useful in practice.
justification_for_why_not_lower_score: See above.
Decision: Accept: poster