Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper explores the use of pre-training models on corpora of emergent languages to improve performance on downstream tasks (in this case language modeling and image captioning).  The authors compare against using other types of pre-training corpora, like synthetic or natural languages.  While not always optimal, the authors demonstrate that there is clearly an advantage to using emergent languages in this way, if one is otherwise considering training from scratch.  They propose a new metric for evaluating whether languages have structure, doing so by training a translation model to map emergent language to natural language on the same data, and validate the measure by showing it corresponds closely to task accuracy. 
main_review: The paper's very clearly written, well-organized, and much of the discussion was grounded in a broad range of references to previous work.  My concerns with this paper are in regards the experimental design, the strength of the conclusions, and how the authors are choosing to interpret them.

Perhaps the best way to discuss this paper is through the authors' goals, to understand: unsolved: "(i) can emergent languages be used outside the game? (ii) if so, would these metrics predict the usefulness of emergent languages for downstream tasks? We tackle these two questions in Sections 3 and 4 respectively."

## can emergent languages be used outside the game?

To question (1), it is of course important to note that this has already been shown in Li et al. 2020 in the context of pre-training for MT.  There is plenty of opportunity to broaden the scope of their findings, but as it pertains to this question, it no longer needs to be asked and should be presented truthfully in this way.  This language persists throughout, and a reader not aware of Li's work would easily go through most of the paper believing that this paradigm is being attempted here for the first time.  For instance, even at the end of the paper "We present a new perspective for studying emergent communication by linking the corpora of emergent and natural languages in two ways."  One of these seems like an established perspective, and what sets this work apart is primarily the choice of tasks: language modeling and image captioning.  

Returning to the question, can emergent languages be used outside the game for language modeling or image captioning?  First, it is also important to note that this EC setup is only one possibly game, and the properties of the EC are hugely shaped by the properties of the game.  So again, for accuracy, it's important to clearly define the scope of the work as pertaining to this one particular scenario.

For language modeling, the authors show that indeed the EC pre-training is useful for the downstream language modeling task, but the results are not very convincing that this would ever be useful in a practical sense.  Pre-training on EC data is rarely the best option, often outperformed by simple synthetic language data, and is never the best option as more data is observed.  And since pre-training on Spanish is still the most effective strategy even in low-resource languages, and Spanish is more easily acquirable than EC data, in what situation would one ever opt for using this strategy?  Back to the research question, yes, EC languages *can* be used outside the game, but it's not apparent that it would ever be best choice nor the easiest.

The case is similar for the captioning task, except the margins (even those between the best choice -- natural language -- and the worst choice baselines) is exceedingly small.  I think it throws into question whether MS-COCO was ever a good testbed for this research question, as (as the authors note) the captions are very regular and structured.  It seems like a reasonable attempt, but it's difficult to find a take-away.

When it comes to understanding the properties of the emergent languages, how these properties are translating into useful biases for downstream tasks, it is also left quite open.  The ablation experiments compare against random scenarios, and I personally wasn't able to gain much insight into where the performance comes from.  Personally I would have liked more qualitative analysis into the real correlations and differences of the EC and natural languages as it pertains to the downstream tasks, but even barring that I imagine there are also some purely quantitative metrics which could have been more targetted, and in doing so, be more informative.

## if so, would these metrics predict the usefulness of emergent languages for downstream tasks?

Another thread of this work is criticism of some existing EC metrics, like topographical similarity, which may be failing to correspond well to the types of structure and compositionality found in NLs.  The authors propose a new method of evaluating ECs by training (briefly) a translation model to translate to natural language captions from EC language generated from the same images.  The metric corresponds better to downstream task accuracy than topographic similarity.  

The authors argue that "intuitively, a higher translation score means the emergent and natural sentences are closer in structure and semantics, similar to how French-English translation might be easier than Chinese-English".  But isn't the purpose of a such a linguistically-focused EC metric to evaluate how well EC reflects the types of structures found in natural (big-L) Language?  i.e., not any specific language.  It seems that in closely tracking the language used in the caption data, it is only natural that this relates to higher downstream task accuracy, but isn't that at the cost of low similarity to another natural language?  Criticisms of existing metrics are likely warranted, but I fail to see how this metric really improves upon it.  An EC learns a Chinese-like structure from a caption dataset, the translation model performs poorly at mapping it to English (vs a German-like EC source language) and now we have a plausible, NL-like EC with low scores under the metric.  I think ideally a good EC metric should score all human languages highly, or one should answer the question of which of two human languages contains more "human-like" language structure.  A metric that targets specific aspects of human languages cuts around this issue, but I think in this case the authors are burdened with providing some answer to this question.

So overall the story of the paper feels like a bit of circuitous reasoning: EC languages might be more helpful to downstream tasks than natural languages if they somehow capture a greater structural similarity to the target (presumably low-resource) language than available natural languages.  But the closer the language to existing natural language, the less useful it is, since it doesn't provide a unique advantage over simply using that natural language.  And in the space of all human languages, only the one closest in structure to the particular fine-tuning task achieves high marks.  So I'm curious how the authors reconcile this?


## Conclusion

Overall the work is tackling important problems with an interesting method, but in its current state I believe it's lacking clear insights into the problem, does not provide a practical usefulness, and doesn't dig sufficiently deep in understanding why models behave the way they do.  It also feels almost like two disjoint papers -- a paper could explore pre-training on ECs but with more focus given on understanding the properties of the ECs, and broadening it to more games to ensure lessons are more generally applicable.  Another paper could target metrics like topographical similarity, but in this case it is important to discuss the seemingly obvious failure cases of the proposed measure.

I think this is summed up well in the conclusion:
"Our methodology and results open up possibilities of establishing a synergy between the research of language emergence and natural language processing, where we could potentially better understand and improve emergent communication through the lens of natural language, and in turn improve upon low-resource natural language tasks with the help of emergent languages"

While the phrasing is again speculative/potential, I find it hard to view the paper experiments as strongly supporting either of these cases: I don't think we've learned much about the structure of ECs, even the limited examples used here in the paper, and I don't know of situations where EC would reasonably be used for improving low-resource monolingual NLP over NLs.

That's not to say the paper doesn't have a lot going for it, but each experiment felt like it only scratched the surface of the problem, and more thorough research feels necessary.



Typos / Grammar:
((Figure 3(a)))
"Our research in based on"


summary_of_the_review: The paper is well-structured and the experiments seem generally sound, but they're too brief and at too high of a level to provide real insight into EC language structure and how we should identify and measure it.  However, I do call into question really how sound the ablation experiments are, since they are contrasted with such trivial baselines that it again hinders the paper from providing novel insights.  There evaluation measure is I think the closest thing to a novel contribution, and the authors seem correct in pointing out the shortcomings of existing measures, but the proposed solution doesn't seem a suitable replacement (it seems like it violates some core assumptions of what the community would want from an EC metric, and is also language and task specific).  Overall it seems like good preliminary work, and I hope to see a deeper exploration using these same methods and motivations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new way of using techniques from the emergent communication literature, where agents are trained to develop languages for communication on some shared task, to improve more typical supervised learning tasks in NLP, namely language modeling and image captioning. This is a question that is only just beginning to be addressed in some work (e.g. Lowe et al., 2021; Lazaridou et al., 2020), so this area is ripe for exploration. This paper proposes an interesting and novel method of using EC to improve supervised learning: generate emergent language from an image-based referential game with simple speaker/listener models. Then, using a much larger transformer, pretrain on this corpus before doing fine-tuning on the supervised task of interest, a la Papadimitriou and Jurfasky, 2020.

Both language modeling and image captioning tasks are explored, and results convincingly show that emergent communication corpora are surprisingly effective. Some ablation studies help elucidate where the benefits arise, although there are some questions here (see Weaknesses).

Finally the authors propose a new metric for evaluating the naturalness and usefulness of a corpus of emergent language: if you have natural language messages generated for some task, as well as true human messages, the authors literally just build a model for translating emergent languages into the corresponding human messages. The ROUGE score obtained by such a model is thus a "transferability" metric. This metric is not only useful for identifying whether an EC corpus will be useful for transfer for some task (more similar = more useful), but is possibly also a measure of humanness/compositionaliy of a language in general, and so this contribution will be useful even for those in EC who are not interested in supervised learning.

Overall, I very much like the philosophy of this paper. I think it helps fill an important gap in the literature, which is namely what to do with the recent deluge of emergent communication studies. The big elephant in the room with these studies is whether the languages are actually useful for some reason besides being studies of linguistic evolution with limited generality. This paper proposes a simple and interesting way of using the emergent languages for something more productive and more of interest to people actually working on real NLP applications. I do however see some weaknesses and lack of comparison which I would love to see answered and/or addressed in the final version.
main_review: # Strengths

- This is a great idea, and one of those ideas that I (and others) will wish they had come up with. The idea of pretraining on emergent corpora neatly brings together new ideas about transfer learning and modern studies of emergent communication.
- Significant experiments in both grounded (captioning) and ungrounded (LM) settings convincingly demonstrate the efficiency of EC pretraining. The ablation studies are also very useful, but also open up several questions (see Weaknesses).
- There is a very rich space of future experiments to try in this area. I imagine it will inspire plenty of follow-up work, exploring how different aspects of the EC corpus do or do not result in better transfer. This may also become a typical evaluation w.r.t. how useful/systematic a language corpus is—how useful is it for downstream transfer?
- The natural language translation metric is extremely clever, and is a natural extension of similar ideas for "translating" emergent sentences (Andreas et al., 2018).

# Weaknesses

- Poor comparison to parameter transfer approaches
    - Perhaps the biggest issue I have with the paper is the lack of comparison to parameter transfer approaches (Li et al., 2020). The GRU transfer line in Table 2 is the right idea, but is woefully insufficient: it makes no sense to compare a 1-layer GRU perplexity to transformer complexity, and there's no reason that you can't generate the EC corpus with transformers and therefore more cleanly compare corpus transfer to parameter transfer. I would be really interested in these results. To be clear, I don't think it's necessary that corpus transfer has to do "better" than parameter transfer for the paper to be useful—there are some potential advantages of corpus transfer: (1) learning from EC corpora generated by heterogenous models, (2) cheaper to generate EC corpora from a cheaper model. But I still really want to see this comparison.
    - I don't think this is a fatal weakness, and perhaps it illustrates follow up experiments that this work inspires, which is to vary the architecture/capability of the EC agents and seeing what kinds of agents produce EC corpora that are more useful for transfer, and how those EC corpora differ using the various tools we have for analyzing emergent languages (e.g. topographic similarity).
    - In fact, I think this experiment is nonsensical and misleading enough (every other line in the table uses a Transformer), that, without rerunning modified experiments, I would encourage authors to just remove this number entirely.
- Limited evaluation of non-EC pretraining for image captioning.
    - The experimental evaluation is overall quite comprehensive, except in the image captioning studies. Authors say "We note that such a transfer scheme cannot work for other synthetic corpora such as music, code, or paren-zipf as they are not grounded on perceptual stimuli." While these other corpora indeed are not grounded in perceptual stimuli, you could still imagine pretraining on the ungrounded data, then finetuning while introducing grounding, or even arbitrarily associating different strings from a corpus with images. I would really like to see this comparison, as part of the strengths of Sec 3.1 are in the careful comparison of EC pretraining w/ other choices of pretraining corpora.
- Results on permuted EC corpora could be explained further
    - One outstanding question-issue I have with this paper is the results on permuted EC corpora. As I understand it, this is for the ungrounded language modeling task for ro and he languages. The permuted EC corpus does not seem substantially worse than EC pretraining (e.g. 195 vs 211 perplexity is not a huge difference compared with 211 vs 266), so I'm somewhat suspicious of the author's claim that "speaker develops a coding scheme with sentence structures beyond bag of words."  A significance test would be helpful to measure the effect here. But regardless, it's clear that training on permuted EC results in impressive gains. Then what benefit does the EC corpus actually provide? The permutation experiments show that it's not syntax/sequential structure. And it's not learning a grounding between objects in images and emergent language tokens, since these numbers are for the ungrounded LM task. Maybe it's just token colocation statistics then?
    - As an aside, I would really love to see the same experiments made with the image captioning task as well (e.g. even if the language ignores sequential structure, the bag of words annotation of relevant objects/features in the grounded input that the permuted EC corpus provides may be just as good as the original EC corpus)
- Connections to related work.
    - There have been a few attempts ta combining emergent communication and supervision, e.g. via multitask training ([Lowe et al., 2021](https://arxiv.org/abs/2002.01093), also [Lazaridou et al., 2020](https://arxiv.org/abs/2005.07064)) which are missing from related work. The connection could be made more clear. Why might we expect corpus transfer to do better or worse than multitask training? Perhaps these methods could be combined, e.g. multitask training on EC corpus and real corpus?

# Questions/Minor

- How does changing the vocabulary size and bandwidth of the EC pretraining corpus change its utility for pretraining? There is surely a point at which EC corpora are too simple (e.g., imagine just a single token and a very small vocab). Would be interesting to quantify the benefits gained not just by varying corpus size, but by varying emergent language complexity. Figure 3 partially answers this question (in that untrained EC corpora are not helpful for transfer), but I'd be interested, for example, in seeing a plot of (V, T) and transfer efficacy.
- One of the central issues in studies of emergent communication is that agents often develop non-compositional, unintuitive communication protocols. The EC setup described in 2.1 doesn't seem to have any of the guards against degenerate languages that more modern studies of EC have investigated, e.g. different ImageNet categories (Lazaridou et al., 2017) or SimCLR views ([Dessi et al., 2021](https://arxiv.org/abs/2106.04258)), or even other ways of regularizing a language (e.g. [Luna et al., 2020](https://arxiv.org/abs/2004.03868)) Yet the EC languages still seem useful for transfer. So questions are, (1) do we still see degeneracy in the languages used in the EC corpora, and (2) if so, does trying to reduce degeneracy/improve systematicity of the language result in better transfer performance?
- Why are there no standard deviations for EC pretrain/from scratch in Table 2?
- Table 3 - how many data points actually go into the computation of correlations here? Is it 5 x 4 = 20? I would really love to see the full scatterplots rather than just correlation, e.g. in the appendix, and to actually see how much variance we see in the different EC corpora, as measured by the metrics.
- The details of the training steps expt in Figure 3 are a little underexplained, and I tried my best to reconstruct, though I invite the authors to clarify if I've misunderstood anything. You take checkpoints of the EC corpora generated from EC agents at 0, 200, 400, ... 1k training steps, and at each step measure the various EC langauge metrics, and also do the pretrain/finetune experiment with transformers, starting with the EC corpora, to get plot d - perplexity. Is that correct?
summary_of_the_review: Overall, I think this is a creative, technically sound, and very interesting paper with valuable contributions to the emergent communication literature (notably, the big question of how to make EC useful for the rest of the community), and the paper really made me think. It's a shame that there are some unanswered questions remaining in the paper (see Weaknesses). I hope the authors might consider some of these follow-up questions and experiments during the rebuttal phase, but I think even despite these flaws, the paper has enough contributions to stand on its own. If all of the issues in the weaknesses section are addressed I would consider raising my score to a 10.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies whether output from emergent communication systems might be useful as pre-training data for natural language tasks. The authors train an EC agent, generate from it, and use the generated IDs as data for language model pre-training. Compared to pre-training on Spanish wikipedia or a simple bracket language, pre-training on EC-generated data is better when you have 2M examples, but quickly becomes worse as the amount of data increases. In experiments on image captioning, the authors find that pre-training on either EC-generated or natural language captions improve over not pre-training, but that there is not much of a difference between the results when pre-training on EC-generated text and natural language. Finally, the authors propose EC -> natural language translation as a way of evaluating the quality of EC agents.
main_review: Strengths:
- I think this research question is important and interesting, and that the study of what EC agents output and how they might relate to natural language is important for both EC and NLP
- The setting was made very clear. Figure 1 is excellent.
- The claims, particularly those about language modeling, are evaluated on a variety of settings

Weaknesses:
- Some of the claims don't seem entirely justifiable. It's true that pretraining on EC is better than pre-training on ES or parens at 2M tokens, but it's not entirely clear to me that this is an interesting setting because _all_ models perform poorly. It's more interesting to me that ES is similar to parens
- It's not clear to me how specific to this particular experimental setup the proposed evaluation technique is. It seems to correlate better with LM performance in this setting, but how about others? Using a particular method for evaluation requires some degree of confidence that it holds generally, and I'm not sure that the results here convince me of that.
summary_of_the_review: I think this work addresses an interesting question, but I disagree with a few of their claims based off of the experimental evidence provided. The broader research question is important enough that I feel like even initial work in this direction is perhaps of interest to the ICLR community and worth accepting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to use corpora generated from _emergent communication_ as a fine-tuning signal for NLP tasks (language modeling and image captioning in particular).  They show that, especially in small-data regimes, pre-training on an emergent language can yield significant performance boosts in both tasks.  (In particular, pre-training on emergent language performs on average better than synthetic hierarchical data, but not quite as well as a different natural language, and all of these pre-training methods do better than training from scratch.)  This is the newest of a small but growing body of literature that seeks to connect emergent communication with genuine NLP tasks.  The results are intriguing and promising and should be of interest both to the emergent communication community as well as to the broader community working on low-resource NLP.
main_review: # Summary

The authors propose to use corpora generated from _emergent communication_ as a fine-tuning signal for NLP tasks (language modeling and image captioning in particular).  They show that, especially in small-data regimes, pre-training on an emergent language can yield significant performance boosts in both tasks.  (In particular, pre-training on emergent language performs on average better than synthetic hierarchical data, but not quite as well as a different natural language, and all of these pre-training methods do better than training from scratch.)  This is the newest of a small but growing body of literature that seeks to connect emergent communication with genuine NLP tasks.  The results are intriguing and promising and should be of interest both to the emergent communication community as well as to the broader community working on low-resource NLP.

Strengths:
* Novel method of using emergent language for pre-training (as opposed to transferring an entire artificial agent)
* Some good ablations to identify what contributes to successful transfer
* A new evaluation metric (emergent --> NL translation performance) that best correlates with fine-tuning performance

Weaknesses:
* Some parameter choices and the design of some ablations are not completely justified
* Some additional related works could be included


# Minor comments / questions

* "However, this metric is too rigid in its definition of compositionality, ignoring aspects like argument structure, context or morphology which play a key role in determining the combination of word semantics (Goldberg, 2015)."  Steinert-Threlkeld (2020) "Towards the Emergence of Non-trivial Compositionality" makes similar points and could be cited here as well.

* Why are the lines for "from scratch" flat in Figure 2?  I would have thought this was training an LM on progressively larger portions of the relevant data (being used for fine-tuning the others), in which case I'd also expect a downward trend in perplexity.  Are these rather the ppl resulting from training an LM on the full dataset?  It should be stated more clearly in the text what this means.

* Why does the EC pre-training use |V| = 4035, as opposed to 50 that's used in the other tasks and the fine-tuning corpora?  Do the authors expect better results with a larger vocab size?

* Ablation: the model vs corpus transfer comparison seems unfair to me.  In particular, you are comparing a small GRU LM to a larger transformer LM, where the latter is, as you mention, a much more powerful model.  While it is also true that corpus transfer _enables_ this divergence in model types, to really test whether transferring the whole model versus using the corpus works or not, I would think you would want to compare fine-tuning the sender GRU on the LM data vs. starting from scratch _with a GRU of the same type_ and pre-training on the EC corpus before fine-tuning.  Did you do an experiment of that type?

* Related work: I would also include a discussion of this Lazaridou et al paper where they compare ways of combining EC with non-EC learning signals (e.g. image caption training): https://aclanthology.org/2020.acl-main.685.pdf

* Ethics statement: I appreciate this statement and agree with the possible positive impacts.  Do the authors see any potential negative impacts?  If not, that should also be explicitly stated.


# Typographic comments

* p 1, "the input out of detractors" --> "the input out of distractors"
 
* p 2: "transferable benefits for downstream natural language tasks" the single hyphens surrounding the subsequent list should be em dashes (three hyphens in TeX)

* p 3: "uses another GRU layer to decode the message m into a hidden vector hl" I would use "encode" instead of "decode" here, since text-->representation is usually what an encoder does

* p 3: "The most straightforward metric is the accuracy of playing the referential game p(guess = Ii)."  Given the way p(guess=Ii) is used above, I think this should be more like E[argmax(p(guess=Ii)) = i].  Or if they are measuring the probability assigned to the true image and not just accuracy, the name shoudl be changed from accuracy.

* p 4: "es should set a upper bound" --> "es should set an upper bound"

* p 5: " ec perform better than or " --> " ec performs better than or "
summary_of_the_review: The paper presents a novel method and evaluation of using emergent communication for pre-training models on real NLP tasks.  This has the potential to help in low-resource settings, and demonstrates the value of emergent communication for NLP in addition to theoretical interest in language evolution.
review_title: null

Meta-review: This paper explores ways in which *emergent communication* (EC) methods from representation learning can be evaluated extrinsically, by hooking them into downstream NLP tasks. Reviewers agree that the paper is thorough, and finds encouraging results.

This paper is borderline, and difficult to evaluate, even after very substantial discussion (some of it private). From my reading of the reviews and pieces of the paper, I'm very sympathetic to wvqW's concern that none of the present-day applications under study seem likely to benefit from this kind of emergent communication pretraining: *Natural* language pretraining, even transferring across natural languages, is for too strong a baseline, and it's not even conceptually clear how one could substantially outperform that baseline. I'm very concerned that the results in this paper will be—misleadingly—cited as proof that EC research is already contributing to downstream progress in NLP.

However, the narrow claims in the paper itself seem to be sound, and two confident reviewers whom I trust argue strongly that the ideas results here are surprising and novel, and that the paper could be the starting point for productive discussion and future work in this area. I'm recommending spotlight presentation in the hope that the paper will provoke a nuanced discussion in that setting.
Decision: Accept (Spotlight)