Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose some extensions to existing work on VQ-VIB architecture for modeling EC with a more explicitly information theoretic framework.  Namely, they extend the number of tokens which are chosen, and in doing so combine two threads of EC research -- existing VQ-VIB, and more traditional RNN/sequential type architectures.  In a set of experiments on English, the authors vary aspects of the model such as informativeness, showing among other things that increased informativeness is associated with better test time generalization, up until the point where it matches or exceeds that of the communication partner's language.
strength_and_weaknesses: The main contributions of this work are likely (1) the extensions to VQ-VIB, and (2) a sort of evaluation paradigm that involves combining learned EQ representations with word embeddings into the same communication game.  The VQ-VIB extension, generalizing the model from a single "word" to multiple words, is straightforward, but important.  It is easy to imagine how this could lead to more direct comparisons with traditional EC models, or how the decoder could been sequential instead of fully connected, such that there would have been more incentive for the model to learn a more compositional protocol.  At any rate, it is a necessary extension for moving towards any real form of human-like EC, where a message is composed of multiple words, where meaning is conveyed not just in the words chosen but by their order.


The second contribution, the experimental setup, is harder to evaluate because there are so many unorthodox choices going on, and there is not consensus in what assumptions are aligned with the goals of EC (or constitute EC).  In my mind, many of these choices make this work more about grounding between two models, or more like unsupervised machine translation, than proper EC.  This makes it hard for me to evaluate the proposed findings as such.

Another comment on the modeling -- I was surprised to see the decoder is trained in a reconstruction style objective using an MSE loss, rather downstream from the classification loss on the signaling game.  What was the motivation here?  Is it reasonable in an EC setting to assume a connection between the speaker observation and the decoder on the other side of a learned protocol?  Doesn't this reduce the EC task substantially?

I found some of the formal descriptions to be a bit messy.  m as a meaning and not a message / c as a communication and not a context/content, as is the case for many EC presentations, was slightly disorienting.  The KL is denoted with D and not KL.  Then D serves double duty as the decoder.  The lines are blurred between minimizing objectives over single m/c and the set of Ms/Cs (not defined).  Then c does double duty as indexes into the set of candidates.  Overall there is a lot of poorly chosen terminology in Section 3.

I would also argue that some of the preliminary discussion dealing with the complexity of the English naming system have less to do with a general property of English, and more specifically with properties of the dataset and the granularity of the naming used there.  It was not clear to me if the authors agree with this point, since some of the latter discussion then seemed to explicitly mention the dataset when talking about this point.  
clarity,_quality,_novelty_and_reproducibility: The presentation is generally well-written but the formalization is less so, and there is a lot of abuse of notation (though this does not to my knowledge hinder readability).  Reproducibility would be difficult from the paper alone, however the authors intend to release their code.  Novelty in modeling is limited.  The experimental design is quite novel, but difficult to understand the full consequences of this design and the implications of its findings on EC more generally.
summary_of_the_review: Overall, I am still undecided on the overall merit of this paper.  On one hand, there is modest modeling contributions, and the experimental setup is unlike any I recall seeing before in EC.  The connections to the information theoretic complexity of English (of this dataset) is surprising in that it closely approximates previous attempts to calculate this number.

On the other hand, the formal presentation is messy, and the experiment setup seems to deviate strongly from standard EC assumptions.  Is it EC if one half of the protocol is derived from word embeddings of image labels?  Or if direct supervision is injected between the speaker and decoded output?  I have a hard time finding takeaway lessons that are both nonobvious and apply to the field more broadly.  The discussion contained within the paper does not go a long way towards reconciling this design with most existing EC work.  Similarly, this task seems setup specifically towards noun communication, but to what extent would many of these experiment choices work on a more general task with multiple parts-of-speech?  

At a minimum, for a paper claiming that most existing EC methods fall short of achieving desired properties in ways that this work presumably does not, it could obviously benefit from direct comparison with such methods.  Do existing works not also manipulate the same factors?  Informativeness is increased here through means like scaling factors on loss terms or increases in the vocab size and number of words communicated, but existing work also explores the effect of varying such things, albeit more removed from a formal information theoretic formulation.  I'm not sure a comparison to prototype style EC is sufficiently representative of EC methods in general (conclusions are phrased wrt to these more general methods)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper shows that explicitly controlling the trade-off between complexity, informativeness, and game utility in the emergent communication setup allows agents to discover languages with better generalization to harder/OOD tasks and better alignment with natural language. In order to demonstrate that, the authors study performance of a multi-symbol modification of VQ-VIB (which allows to control this trade-off) in a series of experiments, focusing on two setups: (a) referential games, and (b) human-agent translation.

The experiments performed on the ManyNames dataset demonstrate that:
* higher communication complexity correlates with higher utility;
* higher communication complexity correlates with better generalization to harder (more distractors) and OOD (unseen categories) settings;
* up to a bound, increasing communication complexity allows higher utility in a simulated translation task.
strength_and_weaknesses: Strengths:
* The paper is clearly written and well-motivated, the findings are novel;
* The experiments are convincing (taking into account Appendix C);
* The simulated English translation task is very interesting.

Weaknessess (mostly minor):
* I believe the abstract can be more specific and indicate that the "human-agent interaction" is "simulated human-agent interaction" and "English" is eg "English-based tags system";
* I am curious if there is a way for controlling the IB trade-off w/o imposing such strong architecture constraints. Eg, much of prior work operated on more standard architectures containing LSTM/GRU/Transformer blocks and producing sequentially generated multi-symbol messages. Are those compatible?
* The last paragraph of S6.2 mentions that using more candidates at training-time also allows better generalization - however, at the expense of lower performance, e.g. training with C=16 is twice faster than training with C=32. Is it possible to elaborate on where the difference is coming from? Since the image encoder is pretrained and frozen, the image embeddings can be precalculated and the difference between selecting between 16 or 32 candidates should be small comparing to the rest of the computations?
* While controlling the trade-off coefficient might be more efficient than increasing the number of candidates, the latter approach is universally applicable to any agent architecture. Is this worth highlighting?
* I believe that the proposed adaptation of VQ-VIB to producing multiple tokens independently is akin to the widely used product quantization scheme (Product Quantization for Nearest Neighbor Search, Jégou et al; VQ-WAV2VEC: Self-supervised Learning of Discrete Speech Representations, Baevski et al). Perhaps it is worth having a reference.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, the experiments are high-quality and interesting. The authors shared the code.
summary_of_the_review: A very nice paper with interesting findings and high-quality experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work explores the relationship between emergent communication informativeness and complexity, hyperparameters of the vector-quantized variational information bottleneck (VA-VIB) method of training EC agents, and downstream metrics like generalizability and translatability with human language.
It finds that as the informativeness and complexity of emergent communication increases, it correlates with increased generalizability and translatability.
In support of these experiments, the paper extends VQ-VIB to multi-component messages and introduces a method for translating between embedding spaces of different languages (viz., English and an emergent language).

strength_and_weaknesses: ### Strengths
- [major] Empirical evaluation generally supports the hypotheses.
- [minor] The extensions to VQ-VIB (translation and multi-component messages) are simple and appear to be effective.
### Weaknesses
- [major] The paper makes too many claims which means that each claim receives only a small amount of empirical evaluation, analysis, and discussion.
  - I think the primary direction forward would be to take one of the claims, and take the whole paper to address it thoroughly.
    Give a robust explanation of why that claim is important and how important future work will be enabled by the contributions of the paper (i.e., do not just say "we could extend this with X, Y, and Z").
  - Example: "increasing informativeness ... allows EC agents to better generalize to novel settings and more challenging tasks" -- this is an interesting claim that would require a whole battery of experiments to prove convincingly.
    Increasing the number of distractors in a signalling game does not convincingly count as a novel setting or more challenging task.
    Instead, it would be best to see multi-step, multi-agent environments with different objectives that are not just a simple extension of the training environment.
  - Example: "increasing EC informativeness only improves team performance up to a certain threshold, corresponding to English informativeness-complexity tradeoff" -- again, very interesting, but one would need to provide a robust characterization of what the trade-off is in English in the first place.
    Furthermore, how do we evaluate this tradeoff beyond a specific environment and dataset; it would be good to first show this on synthetic data or in a theoretical setting before validating that it works the same way on real data.
  - Example: "the structure of emergent VQ-VIB communication vectors encodes some semantic similarities and facilitates open-domain communication, similar to word embeddings in natural language" -- a very worthwhile direction that only receives a diagram and a paragraph rather than thorough treatment analyzing the structure of embedding spaces from multiple directions.

clarity,_quality,_novelty_and_reproducibility: ### Clarity
- Although writing itself is clear, the organization of high-level concepts is a little difficult to follow.
  Presenting a clear roadmap would help fix this.
  - Between sections 2 and 3, a little over two pages are spent revisiting prior work -- this can be compressed.
- The visualizations in the work are helpful and aid in understanding the results.
  - Figure 1 is especially informative
- The contributions and hypotheses are clear at a basic level, but they are not explained in enough detail to make clear what the criteria for success are.
- The results seemed to be making a lot of distinct but related claims in rapid succession.
  As a result, I had difficulty incorporating the result into the big picture of the paper.
### Quality
- The empirical result generally corroborate what the hypotheses and intuitions would suggest.
  - Nevertheless, since there are not clear criteria for positive results (since the hypotheses are not well-defined), it is not easy to determine to what degree the empirical data justifies the claims.
  - The number and nature of the experiments has the wrong balance between quantity and quality: many different experiments off smaller bits of evidence for the claims rather than many related experiments supporting one big claim.
  - More quantitative analysis of the results would make them more convincing.
- The signalling game is a relatively simple/easy environment which limits the robustness of the claims.
  Would we the see the same relationships in more complex environments?
- The multi-component message extension to VQ-VIB and the translation method are simple and effective.
### Novelty
- This work furthers understanding of a newer framework within EC, namely VQ-VIB.
- The topic and approach are not particularly innovative.
  Papers of the form "we explore the relationship between X and Y" have characterized emergent language research for years, and work needs to either go beyond that basic recipe or explain why it is important to know the relationship between X and Y and to do so thoroughly.
### Reproducibility
- Looked at (did not run) repository with code.
  `README` present with most information necessary for running the code.
  Only notable item missing with a specification of the environment used (e.g., `requirements.txt`, `environment.yml`).

summary_of_the_review: This paper shows a handful of positive empirical results in support of the intuitions presented about hyperparameters in the VQ-VIB framework.
Nevertheless, the treatment of each contribution is relatively brief which does not communicate a sufficiently robust understanding of what is going or impress the importance of the contribution to emergent communication research as a whole.
As a result, the paper falls into a common category of emergent communication paper which presents an array of experiments and claims which, while interesting, are unlikely to be impactful.


### Misc. Notes
- Please use $\mathbb{R}$ (`\mathbb{R}`) for the real numbers
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how the informativeness and complexity of the emergent communication protocol in two-agent referential games affects the generalization performance and how these messages align with natural language. 

In particular, they use the recently proposed framework of VQ-VIB, a variational inference approach using a discrete set of vectors (vector-quantitized) as possible messages for the agents. They extend this framework to allow passing $N$ vectors rather than just one as messages, and show that this allows the communication protocol to have more complexity.  They show that the \lambda parameter in the VQ-VIB objective (associated with the **********informativeness********** of the message) also leads to the communication channel being more complex. Overall, the results suggest that greater complexity in protocol (with either multiple code-vectors or higher \lambda) leads agents solving the referential game with higher performance. 

To analyze the impact of the communication channel on out-of-domain generalization, they train agents for the same referential but using, at test time, a higher number of candidate images than trained for and/or candidates belonging to the same category of images. Overall they find that higher protocol complexity leads to agents that generalize better.

Finally they test how aligned this emergent communication are with natural language by building a simple model that “translates” english words to one of the quantitized vectors and passing the associated image category word through this model to get a code and having the receiver agent predict based on this. They find that overall that this simple translation protocol works and that higher complexity in the channel helps but that, unlike with pure vector-based protocols, improvements with complexity plateu earlier.
strength_and_weaknesses: - The paper is well-written and has a nice narrative. The idea of allowing multiple vectors is quite simple but works well in pratice, and the overall analysis of how informativeness and complexity in the channel affects performance is insightful.
- The experiments with (simulated) human-agent are really interesting, and are nice approach to interpreting the "abstract" protocols these models learn.

My main criticism of this work, given the simplicity of the approach (that mostly just builds on top of the previously proposed VQ-VIB) I would like to see if it generalizes to multiple benchmarks and settings that emergent communication could be applied to. The authors only study one task/referential game for images, so it’s unclear if the findings surrounding the importance of channel complexity / applicability of their proposed approach generalize to more settings. 
clarity,_quality,_novelty_and_reproducibility: Overall the paper is well-written and the experiments included are well-motivated and explained. The novelty (in terms of the new proposed method for training EC agents) is somewhat limited, but works well in-practice and the analysis are interesting. To cement the importance of this work, I would only like to see this method and analysis applied to other settings/referential games.
summary_of_the_review: This paper proposes an extension of VQ-VIB to allow for more complex communication protocols and show that complexity is intrinsically correlated with how well the agents will do at the game. The analysis and experiments are convincing for the single task explored in the paper, but I would like to see this methodology applied more extensively to other tasks and settings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: There is one weak reject and one strong reject for the paper. By reading the paper by myself, I tend to agree with their raised weaknesses.  The two reviewers haven't been convinced by the rebuttal.

Moreover, besides the clarity/paper organization issues raised by the two reviewers, I found the technical contribution of the paper is limited. Firstly, the proposed extension to VQ-VIB is a minor contribution. As there exist many EC models that do not rely on VQ, this contribution only affect a small portion of EC research. Secondly, the contribution of using translation to analyze and evaluate emergent language is not novel (e.g., Yao et al., 2022. Linking Emergent and Natural Languages via Corpus Transfer). This limits the second technical contribution, and weakened Reviewer 2's (the one who champions the acceptance) claim of the paper's strength.

On the other hand, I do agree with the reviewers that the paper demonstrates novel findings, which is an important direction in the EC field. As mentioned by Reviewer 3 "the paper makes too many claims which means that each claim receives only a small amount of empirical evaluation, analysis, and discussion", which I agree, I think the paper can be made stronger to focus on the analysis. Hope the reviews are useful for the authors to improve the paper in the future.
justification_for_why_not_higher_score: There is one weak reject and one strong reject for the paper. By reading the paper by myself, I tend to agree with their raised weaknesses.  The two reviewers haven't been convinced by the rebuttal.
Decision: Reject