Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes to learn to communicate while balancing three different losses. The effectiveness of communication (utility as usual), the informativeness of communication as measured by autoencoding loss, and the complexity as measured by the entropy of the communication distribution. Furthermore, the authors propose a new algorithm VQ-VIB which integrates the information bottleneck with a discrete gradient estimator VQ-VAE. The authors demonstrate how to learn to communicate with their novel method and how to integrate all three losses into their method. 

The authors test the three losses and their novel algorithm on two environments, a particle world where a sender must give a location to a receiver and the World Colour Survey, following Chaabouni et al, where agents must learn a vocabulary to communicate colours. In the former, the use MADDPG and compare to regular discrete, continuous, and proto communication. In the latter they compare to just discrete communication using the gumbel-softmax estimator. They find that the autoencoding loss improves performance, and the complexity loss can be used to reduce complexity. In the particle world, VQ-VIB can outpeform a regular discrete method for the same vocabulary size but not a continuous communication method. In the WCS, agents can learn an optimal language (complexity for effectiveness) as in Chaabouni et al and the complexity loss can vary across the range of complexities. In some configuration, VQ-VIB learns a language that better corresponds to the space of colours than regular gumbel-softmax.
strengths_and_weaknesses: 
The paper is both interesting and well-written. It contains many novel insights and experiments but overall, it is not well organized and the overall message and novelty are not clear. If the authors could add some experiments listed in the questions section and perhaps formulate their contribution more clearly, I would be likely to recommend acceptance.

To begin, the paper proposes two separate ideas. The first is the three axes (or two auxiliary losses) utility, informativeness, and complexity. Utility is a given in emergent communication, and the latter two have been investigated separately before (informativeness is autoencoding in AEComm, complexity has been penalized as far back as Mordatch and Abbeel, 2017). The second idea is VQ-VIB which is a clever combination of VQ-VAE and VIB and does seem quite interesting. The issue is that these two contributions are quite separate as far as I can tell and neither contribution is very strong.

For VQ-VIB, although the algorithm is interesting it is not empirically verified in depth. The authors claim it achieves greater utility and informativeness for the same complexity but this is only true in the particle envs and not in the WCS (and could depend on hyperparamters). Furthermore, they do not do an exhaustive comparison of baselines (REINFORCE instead of GS, trying a wider range of vocabulary sizes, etc..). Where the results are better, they are not hugely better and for a novel algorithm it should be investigated on more environments or more in depth.

For the tradeoff of utility, informativeness, and complexity, the analysis is quite good and the results are interesting. Sadly, the novelty is not quite there. Demonstrating that a complexity loss can reduce complexity is not very novel and does not require so much attention (both Fig 4b and 5). Chaabouni et al's work on WCS was novel because it showed that the natural inductive bias of neural networks was similar to that of humans (low complexity, high informativeness). This work explicitly adds those factors as losses and shows it can find the optimal curve, which is again not very novel. Still, the auto-encoding loss is shown to be effective and this is quite exciting but it is not investigated in depth and no previous benchmarks are improved upon (e.g. MADDPG's particle envs)

Simply showing that these losses work as intended is not particularly interesting and as a reviewer, I am not fully convinced of the value of VQ-VIB. Because of the quantity of new ideas here, none of them seem to be investigated in enough depth either and many results are relegated to the appendix with only one or two sentence summaries. This is not very convincing and the overall paper feels like it brushes over key contributions. 

Overall, a stronger contribution would be to focus on one of the two proposed ideas or demonstrate that both ideas are crucially necessary towards some goal. E.g. if the authors were working towards human-AI communication and demonstrate that both techniques (VQ-VIB, and losses) improve zero-shot human understanding of self-play messages. In that case, contributions feel like two separate but important tools towards a single goal. In the current case, there is an interesting investigation but none of the results are sufficiently convincing or novel.
questions: why use VQ-VIB after if it generates a continuous message and is worse than simple continuous messages?

for 5.1, you haven't demonstrated that VQ-VIB is better than one-hot, please test vocab size > 100
- if performance increases as vocab size increases, why not increase vocab size more?
- you can make the argument that VQ-VIB outperforms one-hot *at the same vocab size* but not at the sample complexity. If one-hot can reach the same performance/complexity at a larger vocab size then you can't say it completely outperforms it
- furthermore, neither agent is reaching optimal performance!
- if you can show than VQ-VIB outperforms one-hot accuracy/complexity for *any* one-hot hyperparameters, then your method becomes much more interesting and your results are stronger

related to the above, Fig 4b shows that continuous communication (cont) clearly outperforms discrete communication (one-hot) but the only difference between the two channels should be the bandwidth
- for a fairer comparison, you should increase the discrete vocabulary size (or use a channel of length > 1)

why does Fig 4b show unconverged protocols? 
- ``complexity and reward measured at each increment of 1,000 training episodes from 3,000 to 10,000''
- why are you measuring unconverged protocols during training? they are obviously going to be sub-optimal? 

for 5.2, please test REINFORCE for discrete communication
    - Chaabouni et al found that REINFORCE learns less complex protocols than Gumbel-Softmax which is what you use
    - REINFORCE is as common (or more common) than GS for emergent communication, so it would be the better baseline for complexity

Fig 5a is suspicious
- either 5a is too small and therefore we can't see the variance of results
- or the learned protocols are at the exact optimal accuracy / complexity tradeoff and $lambda_I = 1$ is sufficient to achieve this (despite Chaabouni et al's results which show a variance when $lambda_I = 0$)
- these results could imply the choice of a vocabulary size (or other hyperparameters) that guaranteed optimality (see Natural Language does not Emerge Naturally (Kottur et al)) and this graph should preferably me more zoomed in and likely should stick with the original vocabulary size 1024 used in Chaabouni et al


### citation suggestions

your auxiliary loss for informativeness seems the same as AEComm (Learning to Ground Communication with Autoencoders, Lin et al, 2021) and may be worth a citation
- notably, their experiments found that adding informativeness (AE) to utility (RIAL) reduces performance so your results are interesting and novel from that perspective!

please cite Concrete Distribution (Maddison et al, 2016) on top of Jang et al for the Gumbel-Softmax estimator (they are concurrent work)


### clarification
is the entropy regularization done in place of $\lambda_c I(X,C)$ or on top of that loss? (I understood it to be in place of that loss)

is the entropy regularization in one-hot equivalent to regular entropy regularization done in previous RL works (e.g. Mnih et al 2016, Lazaridou et al, 2018) or is it the opposite? Are you penalizing entropy or encouraging entropy?


in 5.2.1 you meantion that VQ-VIB learn more complex communication, did you mean the opposite since we are trying to learn less complex, more effective communication
limitations: n/a
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper analyzes the influence of three factors on languages emerging in communication games between simple VQ-VAE based agents. Specifically, the authors contrast the effect of promoting utility and "informativeness" versus reducing the complexity of the language. The authors demonstrate that informativeness generally improves success at the game, but at the detriment of complexity. An adequate tradeoff between these two factors seem to lead to "human-like" languages, at least as measured by a comparison to color-naming schemes.
strengths_and_weaknesses: While the paper touches on interesting issues (the interplay between informativeness and complexity), I am having a hard time understanding what I should make out of it. The extreme simplicity of the tasks and languages considered, coupled with a debatable definition of complexity make it hard to understand the relevance of the results.


### Strengths

1. **Clear exposition**: the paper is rather clear throughout. The writing reads well and a number of well designed figures illustrate the author's points (in particular figure 1)
2. **Well laid out experimental section**. I enjoyed the structure of the experiments section, first spelling out hypotheses, experimental settings and then discussing results.

### Weaknesses
1. **Notion of complexity**: In the paper, the complexity of a language is defined as the mutual information between "inputs" (what is communicated about) and signals (what is communicated). I am not entirely convinced by this notion of complexity for several reasons
    - It seems to me that a language can have a high mutual information by virtue of being unambiguous: indeed if we write $I(X;C) = H(X) - H(X\mid C)$, then for a fixed distribution over inputs, the mutual information is maximal for languages where $H(X\mid C)=0$, in other words when any given message $c$ refers unambiguously to a single input $x$. To me this seems unrelated to complexity. For example emergent languages might be equally unambiguous, yet more "complex" in a different sense (eg. less compositional, using longer words...).
    - More generally this is an odd definition in the context of human languages. Much ink has been spilled in linguistics on the topic of language complexity: syntactical complexity vs morphological complexity, are all human languages equally complex, is there a meaningful measure of language complexity, etc... There is very little mention of this literature which is a bit of a shame given that one of the objectives of the paper is to "[demonstrate] how fundamental principles that are believed to characterize human language evolution may inform emergent communication in artificial agents." I recommend "On the Feasibility of Complexity Metrics" by Miestamo (2004) as a starting point for a discussion. It should also be mentioned that previous work in emergent communication has looked at other metrics which are related to complexity: length of the messages ("“LazImpa”: Lazy and Impatient neural agents learn to communicate efficiently", Rita et al. 2020), ease of cultural transmission ("Emergence of Compositional Language with Deep Generational Transmission", Cogwell et al. 2019, "Compositional languages emerge in a neural iterated learning model", Ren et al. 2020).

    Overall it seems to me that the notion of complexity advocated by the author is more or less the entropy of the distribution of the messages, which I understand to be a proxy for the size of the lexicon (the number of utterances in the language). I am not sure if this is an interesting metric: arguably a key feature of human languages is that they allow for generating a potentially infinite number of utterances, and what differentiate them is how they are able to do so by the use of finite means (phonemes, words...)
2. **Unclear takeaway**: What do the results of the paper mean? The two takeaways I read from the papers are that (1) "principles that characterize language evolution can inform emergent communication" and (2) "penalizing complexity is necessary to avoid idiosyncratic languages". I would contest that (1) is a novel insight because there is already plenty of work porting ideas from cognitive science and language evolution to emergent communication (see work on iterated learning, zipf's law in emergent languages, etc...). As for (2) I am not entirely convinced due to my issues with the definition of complexity used in the paper (outlined above).
3. **Informativeness vs utility**: The distinction between informativeness and utility does not seem very useful in the context of the paper: indeed the two are not really contrasted in experiments. In a lewis signaling game (such as the color naming experiment) they are essentially equivalent. It would have been interesting to also vary along this dimension in experiments. Alternatively, I think the key claims of the papers could have been made by focusing on the opposition between informativeness and complexity (and leaving utility out of the picture).
questions: - What is semantic communication? Or rather what would the opposite of semantic communication be? Communication without meaning?
- What is the purpose of VQ-VIB after? It seems to me that randomness in VQ-VIB after emulates a perturbation in the environment (on which the agent has no control), whereas randomness in VQ-VIB before models uncertainty from the agent (via a reparametrization trick). It is not clear to me why the former is interesting in the context of this work (in fact it is used much less than VQ-VIB before in the experiments.
- What are $m$ and $\hat m$ in section 3.1? Are they typos for $x$ and $\hat x$?
limitations: I think there could be at the very least a more thorough discussion of the notion of complexity of a language (and the difficulty of defining a metric)
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors introduced a new discretization method for endowing neural agents with a communication channel in a cooperative multi-agent framework. By adding auxiliary losses, each penalizing the utility, informativeness, and complexity of the generated language, testing on two MARL benchmark and a referential task, the authors study the conditions that led the communication protocol to be successful and to resemble properties of natural language as described by the IB (information bottleneck) principle.
strengths_and_weaknesses: I believe the main strengths of this paper are in the introduction of VQ-IB and the analysis of the contribution of the additional losses in terms of complexity and informativeness.  The experimental setup is solid and the results support the idea that the introduced terms can steer the emergent languages to different complexity classes and make them similar to natural languages.

I could not find any major weaknesses in this work. Useful addition to the paper would be a qualitative analysis for the navigation environments. The quantitative results are informative and support the main claim of the paper but a qualitative analysis might help shed a light on better understanding the language protocol. Interesting questions would be of this sort: "are there specific messages the systematically mapped to actions of the environment e.g. go right, the target is in the top right corner, go there etc.".  

While I believe that the tested environments correctly corroborates the introduced hypothesis, it would be interesting to know whether the discretization method can scale to larger scale setup involving complex input like natural images. Havrylov and Titov 2017 and Dessi et al. 2021 showed that agents can learn a protocol describing natural images when trained from pixel input and without any supervised pretraining. VQ-IB is a drop in replacement in their methods (they both used gumbel softmax sampling) so it would interesting to see wether your results hold at that scale. Given that the way "complexity" is penalized in previous work is usually by changing vocabulary sizes it could be useful to test whether an auxiliary loss term would be more effective instead.


Finally, I consider the introduced method and the presented results interesting, the analysis to be thorough and informative and believe this paper should be included in NeurIPS since it could be beneficial for the emergent communication community.

-----

Minor:

- A brief description of the Proto architecture would be useful to understand its differences with the other architectures

questions: Lin et a. 2021 reports how difficult it can be to train fully decentralized agents for tasks that involve learning a communication protocol. Sharing gradients between agents is a form of communication that could not used in a fully decentralized setups. Have you considered or tried training your agents with some form of RL algorithms that would not involve any gradient passing between agents? Note that this is not to be intended as a missing experiment for this work but rather as something to be intended as future work.
limitations: The authors have adequately addressed limitations and negative societal impact of their work
review_title: null

Meta-review: metareview: From the ratings alone this paper appears borderline leaning towards acceptance, however, I want to highlight to the authors that in discussion with reviewers and my own reading of the paper there are aspects that shifted this even closer to the decision boundary. In the end, my own conflicted views of the work and the lack of further discussion from the more negative reviewer led to a recommendation of accept.

I'll briefly review some of the strengths and weakness in the latest revision as I see them.

+ The work truly studies the effect of controlling multiple objectives of communication in the emergent communication setting and how these affect trade-offs in complexity, informativeness*, and utility. This scientific approach to the experimental work is in my view a clear strength of the work.

+ There is a clear motivation for the choice of objectives based upon existing work in related fields. Although I have reservations about how well the realization, building on the Zaslavsky et. al.'s work and within emergent communication is something I think will benefit the community.

+ The writing itself is clear and easy to read. Figures in the main text and appendix were informative and quite interesting.

- Important details, especially around the math and experimental setup, are lacking. Some of thing examples that bothered me were: ambiguity on definition of terms in the objective (all three terms could be stated more precisely but for U(X, Y) we are not told what Y is and for I(X, C) it is not clear if this is coming from 3.2.2 or 3.2.4); lacking clarity around gradient flow (passing gradients back to the sender in this type of setup should be stated very clearly and upfront).

- Connection between I(X, C) and complexity. As I was reading I interpreted this to be (as in 3.2.2) "the KL divergence of μ(x) and σ(x) from a unit Gaussian", and found this is to be a very strange choice (not as an objective but as a measure of the complexity of the language). This is, I believe, a different concern than what was raised by one of the reviewers. The issue I saw there was that the codebook need not be uniformly distributed in the continuous space, and therefore the implications on message complexity of a particular variance in the continuous latent space may be inconsistent. For some regions of latent space a unit variance could imply a single message with high probability, while other areas of the latent space may be more closely clustered causing the same unit variance to be nearly uniform over several different messages. Moving on to 3.2.4, we see (perhaps) evidence that the authors encountered the repercussions of this choice "simply penalizing this term in training was insufficient to train VQ-VIB agents to use fewer unique discrete embedding". The motivation is great, but I strongly suspect that there are better ways of realizing it and that this particular choice is not capturing what the author(s) intended.

- Limited novelty of the method. The combination of VIB and VQ-VAE seemed like something that would have already been published, since regularizing the prior of the latent distribution is such a well used (maybe even well understood) method, but I looked and must grant that this does seem to be a novel combination. That said, it is not so novel that it would be able to stand on its own without the specific setting itself and connection with models of complexity in human languages to support it.

- (Minor) The author(s) made several additional changes to address reviewer concerns around prior work and putting their design choices into context, but there remain some issues in this space. I found the discussion of related work to be fairly shallow and at times even dismissive. It reads as though the work was undertaken entirely devoid of consideration for related work except for that which directly motivated the approach, and then was added defensively but without making real connections between this work and others. I mark this as minor because it is unfortunately somewhat common and because it is a more subjective evaluation.

I hope this and the other reviews will help author(s) to understand how the work may be experienced by readers and potentially make further refinements. Overall, despite limitations, I do believe this work will be of interest to researchers in emergent communication and potentially more broadly due to common underlying questions around trading off complexity and other primary learning objectives.

confidence: Less certain
Decision: Accept