Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors study the zero-shot emergent non-verbal communication in this paper. Different from most papers on emergent communication. this paper uses the motion of three-joint agents. The agents meet partners that they have never seen in the training phase, presenting the challenge of the universal protocol. To make a universal protocol possible, the authors study intents sampled from Zipf distribution and energy regulation. The authors conducted experiments on tasks with 2, 5, 10 intents. The results show that providing latent energy feature is essential for zero-shot coordination. To achieve better than chance accuracy on tasks with 10 intents, a torque curriculum is needed. 

Pros:
1. The setting of ZS coordination with non-verbal emergent communication is novel and interesting. 
2. The use of energy regulation and Zipf distribution is natural and intuitive. The results also support the intuition. 

Cons:
What is the aim of using motion as a communication protocol? In human communication, actions are often used as an iconic gesture to describe intents that are not well described by words. In this sense, humans can perform zero-shot non-verbal emergent communication since the iconic gestures are always universal. In other words, a grounded world should be used to study ZS coordination. This paper seems to study a problem that is not realistic. 
review_title: Zero-shot emergent non-verbal communication

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper deals with agents that communicate non-verbally via actuating their joints in a 3D environment. The authors show that the agents should be able to learn protocols that can generalize to novel partners. Furthermore, the authors find that the current training approaches are brittle, and they propose and evaluate approaches to address this challenge.

##########################################################################

Reasons:

Overall, I vote for accepting. The authors develop and evaluate an algorithm that allows for emergent communication which generalizes to novel partners. Furthermore, the authors openly communicate and address the brittleness of the training.

##########################################################################

Pros:

* Novel algorithm to perform emergent communication via joint actuations
* Algorithm allows for generalization to novel partners
* Suggestions to improve training stability

##########################################################################

Cons:

* It would be desirable to train the agents N times with different initializations and report mean and std of the performance
* More experiments and evaluation results with various joint numbers, intents, potentially different communication strategies would be helpful 

##########################################################################

Misc:

The text in Figure 1 is too small.

review_title: Interesting field of research

final_score: rating: 5: Marginally below acceptance threshold
review_content: *** Summary ***

The paper investigates emergent gesture-based communication in Embodied Multi-Agent Populations. A noticeable feature of the paper is that it investigates emergent communication in the case of non-uniform distribution of intents and costly communication (i.e. agents are penalized for effort). The authors find that in certain scenarios, these conditions may lead to communication generalization of learned communication strategies to new, previously unseen agents.

*** Relevance, Originality, Novelty *** 

I believe that the paper meets ICLR conference standards when it comes to the importance of the problem considered, the originality of the approach and the novelty of the proposed ideas.

*** Clarity ***

The paper is, in general, clearly written and is a pleasure to read. Figure 4 is deeply confusing, however. As mentioned in the appendix, the actual number of timesteps is equal to 5 and is upsampled during visualization. I believe that it is important to mention it right away (i.e. in the image description).
Otherwise, the reader can easily misunderstand the setup after seeing this picture.

*** Quality ***

The quality of the experimental support is, unfortunately, the weakest part of the contribution.

The authors introduce a number of exciting ideas, all of which make the communication learning setting more realistic. Specifically, the authors are studying communication in an embodied setting, Zipfian intent distribution, and the idea of effort-based action cost.

At the same time, while the ideas sound very promising in theory, in practice, the complications introduced by the embodied setting complicate the interpretation of the obtained results when it comes to the effects of Zipfian intent distribution and effort-based action costs.

In order to mitigate the difficulties, authors propose additional measures: providing explicit feature information to the outside observer (i.e. the observer directly gets the action effort); using "torque pretraining" to make the model more familiar with the reward landscape, dramatically reducing the number of actions.
These measures, in my opinion, are significantly detrimental to the potential impact of the paper. Providing effort information directly they makes the setting less realistic, defeating the purpose of the embodied approach.  Torque curriculum makes the results very specific to the considered setting.

There are other significant limitations of the experimental results that are probably indirectly induced by the embodied setting. Specifically, figure 2 and table 1 report no variability measures for the obtained result, and it's not clear, how many runs were made to obtain these results. Moreover, the agent population considered in these experiments is 10 (mentioned in the appendix). With such a small population, it is not surprising that the observer can simply memorize the individual patterns, and is not pressured to infer the underlying structure.  It seems very natural, therefore, to increase the number of agents in the simulation.
I suspect that these limitations are a consequence of the fact that training in the multidimentional setting is very costly and time-consuming.

Moreover, some of the limitations I describe in the "technical soundness" section could have been controlled for in a simpler setting.

Overall, all things considered, it's not clear what are the insights gained from the embodied approach.

*** Technical soundness ***

There are certain technical problems with the contribution:

Firstly, the justification for why Zipfian intents, together with effort-based penalty should lead to generalization is not rigorous (paragraph 1, section 4.1), and not completely convincing.

One of the sentences in the intuitive justification raises particularly serious concerns. The authors say "Coupled, [Zipfian intents and Energy regularization] incentivize an inverse relationship between energy exertion and intent frequency, assigning minimum energy trajectories to maximally occurring intents.". This assumes that energy levels are "occupied" by a limited number of actions. Why that would be true is entirely unclear to me.

For example, for any action with a certain energy level, there is a mirror action (with all joint torques multiplied by -1) with exactly the same energy consumption. Even if for some reason such specific pair of distinct actions with exactly the same effort 
is impossible in a given setting, given how high the degree of freedom of the system is, it is still easy to imagine that every energy level offers a plethora of actions to choose from.

This raises a question of why then we see good performance in 2-action scenario. There is an alternative explanation that was not explored: there is a unique action with zero effort: doing nothing.
I suspect that it may be the only special case, where energy level is mapped uniquely onto an action (i.e. there is no different action with the same (0) energy penalty).

Another alternative explanation that must be addressed is the exposure bias. Rare intents are sampled less frequently, therefore the model has less time to optimize its actions for that intent. This may explain some of the differences in energy costs between the rare and frequent actions.

Lastly, the architecture choice and problem setting formalization also raise certain concerns. Specifically, the sender only receives joint positions and angles at time t, with no history. This does not seem to be an appropriate state specification. Coupled with the fact that the network is a simple feedforward architecture (no recurrence), this 
limits the number of possible jestures that can be generated.

*** Conclusion ***

The paper addresses a highly relevant problem and proposes a number of interesting ideas. Unfortunately, in my opinion, 
some of the proposed ideas (studying communication in an embodied setting, Zipfian intent distribution, costly actions), when introduced together, are not interacting well and are hindering result interpretation.

As authors mention, using an embodied setting with multidimentional continuous action and observation spaces makes the problem extremely challenging. Therefore, when we see only partial success or no success in learning, it becomes difficult to identify the true source of the problem. That is, it is unclear if we are hitting a fundamental limit of what can be achieved by the combination of the Zipfian intent + costly actions, or is whether it a just a limitation of the architectures/algorithms that the authors used, when those algorithms are applied in a challenging multidimentional setting.

The latter concern is exacerbated by some technical questions. In particular, the choice of a feedforward architecture seems extremely limiting, when combined with the state description that the authors used. The state space for the policy (sender) agent only includes the current joint configuration, without the history. This renders a large number of actions impossible (i.e. the "pendulum-like" wave of a hand becomes impossible, since when the hand returns to the middle after the first half-swing, the policy would act as if the action just started).

Lastly, the logic behind the main hypothesis is not fully theoretically supported, and the alternative explanations are not addressed.

Overall, I believe that the contribution is extremely promising, but it still requires some polishing and potential restructuring in order to be up to the standards of the ICLR conference. I do deeply hope to see this paper improved and published in the future, as I believe that after some improvements, it will be of interest to a large portion of the community.

*** Suggestions ***

I believe that the paper can be made much stronger if the authors take a step back and start with a simplified communication game setting. E.g. instead of actually producing an action trajectory, agents could pick among a number of actions with pre-defined effort levels, or they can separately pick the gesture and the amount of effort to allocate to that gesture (i.e. a subtle hand-wave vs flailing one's arms in the air).
This would allow to explore the benefits and limitations of the proposed Zipfian intent+Costly actions approach. In that case, demonstrating that the approach also works in an emboddied setting would be a beautiful cherry on top. Alternatively, it is possible to "boost" the embodiment part of the paper, 
e.g. by pre-training the agents on some other tasks, thus making certain actions more natural, etc. Currently, it just seems that embodiment itself is not adding additional insights, but complicating the experiments.

Adding noize (mentioned in the appendix) is a very good idea as intuitively, it seems that it is the only reason why the agents don't converge on an array of extremely subtle actions with near-zero effort. I believe that it was not discussed enough in the paper and maybe its importance is somewhat overlooked.

*** Questions *** 

- "Furthermore, with an increase in task complexity to 10 concepts, the external observer never performs better than this baseline strategy" I thought the baseline accuracy was 0.34, so it seems that with curriculum the external observer does outperform the baseline.
- Apart from the zero-effort action, why can't the agents converge on minimally distinguishable actions with the same effort level? 

*** Typos and other minor suggestions ***

"execution decentralized" -> "decentralized execution"
"Agent reward" verbally described as a "function of state and actions taken", but the formula is written as if that it is a function of observation and actions taken.

Table 1 is slightly confusing to read, it may be better to make it broader so that the phrase "Train Input" does not break into two lines.

Conflicting notation: N is used as the number of agents on page 3, but later N refers to the number of concepts. 10 agents - not enough to infer the latent variable, especially if no randomness is added to the agents.

I personally believe that the use of the term "zero-shot" in this setting is not ideal. The observer is trained on a sample of communication protocols and is then tested on a held-out set. 
In my view, it's analogous to "generalization performance", not "zero-shot" performance. I understand that the authors are referring to another study that did use "zero-shot", so this consideration did not affect my score.
Before the name for this problem setting is thoroughly established, it may be possible to use "out of sample coordination" or some other option.

*** Update after rebuttal ***

I have read the rebuttal and I deeply appreciate the detailed response by the authors. I especially appreciate the introduction of a number of experiments on a simple domain that help to illustrate the main point of the paper.

At the same time, I believe that some serious issues still remain unresolved. For example, my main concern remains: I believe that wrapping the problem in the embodied setting is not introducing additional insights. I must clarify that I fully agree that studying how embodiment affects cognition/behavior is an extremely important and exciting area of research. But in the present paper, the models have no chance to benefit from embodiment (since there is no prior / shared embodied experience), but rather have to solve the task despite being placed in an embodied setting.

The main insight (in my opinion) is the observation that Zipfian intent distribution together with energy costs could be a good basis for zero-shot communication. I think that additional experiments that the authors introduced help to strengthen this point, although more experiments could still be beneficial (e.g. systematically varying the population size), as well as a more thorough theoretical discussion. At the same time, the limitations of the main "embodied" experiment remain (most importantly, the fact that fairly high accuracy can be achieved because of the unique "do nothing" action trajectory). 

In short, a large part of the paper (on embodiment) contributes relatively little in terms of its impact and conclusions that can be drawn from it, which necessarily limits the extent to which the main insightful point can be explored. The main point is truly interesting, however, which makes the paper borderline.

Overall, I believe that the paper is extremely promising and I would love to see an expanded version published. I feel very torn about the decision, but at the moment, I believe that the paper is still below the acceptance threshold, although only marginally. I am happy to adjust the score up, and I regret that I can not switch it to an "accept" recommendation.

As a minor aside - the newly introduced Colab Notebook does not fully run and crashes at the cell #4 (model loading), so I can not fully explore the newly introduced experiments. That being said, I think that after fixing, this resource can be very useful in the future. This minor issue did not affect my evaluation.
review_title: Introduces two interesting ideas at once, but this makes interpretation difficult

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies how gesture-based (non-verbal) communication can emerge in embodied multi-agent population.
In their problem setting, there is a set of agents, and an observer. Given an intent sampled from a Zipf distribution, the agents generate energy-efficient trajectories, and the observer predicts the intent given these trajectories. Zero-shot coordination is enabled by combining energy cost with Zipf distribution (which is monotonic), since energy cost is grounded in the environment. The authors evaluate zero-shot coordination performance by evaluating a third-party observer trained and tested on two separate subsets of agents from the population respectively.

Pros: 
1. The idea of using energy cost and combining energy cost with a non-uniform distribution for emergent communication is new.
2. The idea of exploiting common-knowledge grounded in the environment for zero-shot coordination is new and interesting. 

Cons:
1. The paper is sometimes hard to follow with missing details.
2. The authors failed to discuss options other than energy cost for ZS coordination, and why energy cost is preferred compared with other options.
3. The assumption of a monotonic ordering for intent distribution is strong. The assumption that energy cost is available during training/testing is also strong.

I like the idea of exploiting a universal (common-knowledge) cost grounded in the environment for zero-shot coordination.
However, is energy cost the only option here? what about trajectory length (spatial), cumulative angular velocity change, etc. I understand energy cost may be a better option, but the authors should discuss and compare with other options. Meanwhile, in many multi-agent environment, the only available inputs are visual inputs and energy cost may not available. Finally, even if the learned policy achieved zero-shot coordination with a third-party observer, it is not human interpretable since humans cannot directly observe energy cost. Any thoughts on learneing human-interpretable gesture-based language?

While Figure 1 is "Overview of Learning System", it does not really helps the readers to get the key idea of the paper. It is better to have a more detailed figures explaining your problem setting and motivation.

The assumption that the intent is from a monotonic distribution seems a little bit strong to me. It is better to have experiments with other distributions (not uniform). e.g. a subset of intents has equal probabilities to be selected.

In Section 3 Problem Setting, you mentioned that "After T steps, the episode terminates, and the sequence of ...". So the number of steps per trajectory is fixed? Why?

In 5.1 where are the "unseen observer agents in the population" come from? My understanding is that there is one observer and a set of agents during the training. But here it implies that there are more than one observer during the training?

Algorithm 1 in A.2 is not clear. For example, Line 2 says "Let A<-population of agents". However, "A" never showed up again in the remaining of the algorithm; The term discriminator (line 3), receiver (line 5), and observer (line 15) are used interchangeably in Algorithm 1.
review_title: The idea of using energy cost for zero-shot coordination is interesting, but why it is preferred is missing.

Meta-review: This paper received borderline scores, R1, R3, R4 gave a score of 6 and recommended a borderline acceptance. R2 provided by far the most detailed review and recommended a score of 5 (i.e., borderline reject). After the rebuttal, R2 comments, "I believe that the paper is still below the acceptance threshold, although only marginally". Overall, I concur with R2. The reasons are detailed below: 

The paper proposes a method for communication between two agents, wherein one agent actuates its joints to communicate intent. Intuitively, this resembles making a gesture. The paper considers the setting of a discrete number of intents. The sender agent is modeled as a neural network that takes as input the intent and outputs a trajectory of joints. The receiver observes a noisy version of the trajectory and outputs the intent. The parameters of the sender policy and receiver discrimination network are optimized to maximize classification accuracy. It is shown that if the intents are sampled from Zipf distribution and trajectories are penalized based on their energy, then a receiver agent initialized from scratch is better at inferring the intent from a pre-trained speaker agent, as opposed to when the distribution is uniform or when the energy regularization is not used (Figure 2).

Further, section 5.2 shows that when the listener is provided with the energy of the trajectory then it is better at recognizing the intent as opposed to being provided with the entire trajectory when a number of intents are small. With a larger number of intents (N=10), the performance is at chance accuracy.
 
The biggest challenge with the paper is that it is very poorly written. Large parts of the method and experimental setup are in the appendix (A.2 / A.3), which makes it hard to understand the paper. Section 4.2 is rather confusing because the ideas introduced are not used for training, but simply for evaluation. Further, the authors point out in the rebuttal that torque curriculum is not required, but it is still there in the paper and makes it more confusing. I recommend the authors to substantially rewrite the paper and focus on relevant parts instead of philosophical arguments. Lastly, I am confused by results in Table 2 -- the authors mention in the text that with 10 intents, intent identification is at chance (i.e. 34% accuracy), but the table shows 56% accuracy. A clarification would be helpful here. 

The problem of communicating intents via gestures, when the agents are unaware of mapping from intents to gestures is an exciting area of research. From the perspective of emergent gestures, this paper has a novel contribution. However, the settings are toy and even in such a setup, the results are underwhelming. The assumptions that make the setup toy are: the listener agent knows about all joint locations of the sender (with some noise) and also has access to the energy exerted by the agent. Without access to energy, the performance is poor. In real-world scenarios, these are big assumptions. Furthermore, even when the energy is known For instance, even when the number of intent is small (i.e.,  N=10,) the performance is bad. The authors argue that is due to local minima in the optimization -- but that's exactly where the contribution could have been. 

I will reiterate, that the authors claim their contribution is in using energy minimization + Zipf intent distribution as a mechanism for communicating intent -- which I agree to. However, as pointed out earlier, the paper is not well executed or written and therefore I recommend rejection. 


Decision: Reject