Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a study of "computational language acquisition", where a speaker agent learns to use natural language to play Lewis-style referential games. These games are played with a fixed, listener "teacher" agent which makes decisions based on the referring expressions generated by the speaker, optionally giving "natural language feedback" in terms of the ground-truth caption.

This setting has been explored in existing work; the authors' main contribution is to outfit the speaker with a "theory of mind" listener model, which is a learned model of the listener given listener feedback over interactions. Then the speaker model looks like a pragmatic speaker similar to the Rational Speech Acts literature, where it optimizes a weighted objective of (1) truthfulness (i.e. probabilty under the image captioning model) and (2) listener utility under the learned ToM model. Authors show that, for some combination of ToM speaker hyperparameters, ToM speakers are able to indeed generate utterances at higher quality than a speaker without ToM.

Some secondary analyses are also conducted on whether the difficulty of distractors sampled from reference games changes the learned languages; authors find they do, in that more difficult games result in better languages according to various metrics (e.g. utterance complexity, length, closeness to ground-truth caption).
strength_and_weaknesses: # Strengths

This paper presents a number of interesting results in a simulated language learning environment and explores a number of sensible hypotheses about how modeling and dataset choices change the learned languages (though several of these have been explored already - see Weaknesses below). I think the most interesting contribution of this paper (which is highlighted by the title) is the learned ToM-style internal listener model.

Authors are correct that in a lot of the rational speech acts literature, the speaker is allowed to explicitly optimize against the known parameters/probabilities of the partner listener model. Showing that speakers can build internal model of listeners from a weaker supervision signal (just listener choices in certain contexts), and still use this to improve generation, is an interesting finding and likely to be useful to the broader community.

# Weaknesses

The weaknesses I see in the paper have to do with missing contextualization with related work (most salient), a confusion about some of the modeling details (equation 10), as well as some lesser concerns about the broader applicability of this work as a whole.

## Missing contextualization with related work

I believe this paper has more similarities to related work than it lets on. One clear lacuna is discussion with Rational Speech Act models of pragmatic language use. In particular, the idea of an internal listener model that is used to rerank utterances is precisely the "pragmatic speaker" formulation in RSA which optimizes utility of an internal literal listener. This idea was precisely implemented in [Andreas and Klein, 2016](https://arxiv.org/abs/1604.00562) which authors correctly note. But the similarities are much more than that: Andreas and Klein precisely propose "sample and rerank RSA" where candidate utterances are sampled from a speaker model, then rereanked with an internal listener module. This process is *identical* to the ToM speaker, the only difference being that the ToM listener is learned. Andres and Klein also define the same tradeoff between optimizing for the listener probability vs optimizing the speaker probability of utterance. I think these similarities should be highlighted more beyond just "we draw inspiration from." Another relevant paper is [Amortized RSA](https://arxiv.org/abs/2006.00418) which trains a model directly to maximize likelihood via gumbel-softmax (similar to the strictly communicative objective in section 3.1 of this paper, though here PPO is used as the optimizer). Again this is with a fixed listener model; I think the current paper should more clearly highlight the idea of using a learned listener model as the key differentiator over existing work in data-driven RSA.

The idea of increasing difficulty of reference games by making distractors more similar is also a common idea in the literature, e.g. this experimental condition has been explored in [Monroe et al. 2017](https://aclanthology.org/Q17-1023/), [Achlioptas et al., 2019](https://arxiv.org/abs/1905.02925) etc, and indeed show positive effects on the pragmatics and complexity of language when trained in "harder" games.

Another relevant paper which should be discussed is [Lazaridou et al., 2020](https://arxiv.org/abs/2005.07064) which examines the idea of initializing a pretrained image captioning model, then finetuning it with a communicative reward, similar to the present study, as well as using the listener model (jointly learned with the speaker) to improve generations. Note that they (and many in the literature) observe this phenomenon of semantic drift, i.e. speakers begin to use language in ways not implied by the original static training dataset. Did authors find this phenomenon here?

## What's happening in Equation 10?

Equation 10 (ToM objective) and its description are confusing to me. Equation 9 shows $P_{ToM}(I_t \mid u^j)$ playing the role of the listener, i.e. a probability distribution over images $I_t$ given utterances $u^j$. But here in Eq 10 we are evaluating $P_{ToM}(w_u^n)$. What does this mean? What is $w_u^n$? (Seems overloaded with the ToM hyperparameter $w_l$). What is the summation index variable? Description says the ToM objective is "defined as the cross-entropy loss between the distribution of the ToM listener and that of the speaker". Should "speaker" be "true listener" here? This is more aligned with "untrained listener trained to emulate the actual listener's outputs over time."

I have a mental model of what is going on here (that we are training the ToM listener to mimic the actual listener, whenever we actually get listener choices), but I'm not sure if this is actually happening as I found this section difficult to parse. It's possible I'm misunderstanding something here, and I can't be too confident in my recomendation until this is cleared up.

## Unclear whether this is a realistic setting for either NLP/CogSci communities

This paper certainly has some interesting experiments and modeling contributions, but one weakness I see (and I think this is true of many computational simulations of language acquisition, e.g. emergent communication) is the degree to which this is a particularly useful simulation for either (1) giving us better NLP systems, or (2) teaching us about language development. Re: (1), it would be great if authors gave us more concrete settings or potential applications where training a speaker model in this kind of fashion, with natural language feedback, might be useful for building better NLP systems. Re: (2), there are a lot of ad-hoc hyperparameter choices in this simulation environment (e.g. the thresholds at which listeners choose to abstain from providing feedback, or provide the ground-truth caption; the various hyperparameter tradeoffs between speaker objectives) that could drastically change results. If results are fairly sensitive to such ad hoc hyperparameter choices, then we should be careful to make broad sweeping conclusions about what this tells us about human and/or machine language development.

Moreover, it's a little unclear how authors intend, in the conclusion, for this study to support further investigations into the "similarity of the learning process between human learners and our computational models." Babies learn in naturalistic environments, certainly aren't initialized with ResNets, and do far more than just play referring expression games.

I don't view this as a fatal weakness of the paper - I think these are fundamental problems that a lot of papers in this kind of line have, and this debate need not be resolved here.

## Other minor considerations

- It would be great to have an explicit measure of the ToM model success, i.e. how well does the ToM listener approximate the real listener throughout training? I'm interested in a more isolated metric for measuring this beyond just "the full ToM speaker does better". If we get very close to the true listener then it makes perfect sense that ToM modeling would improve generation.
- How much is the "learning from feedback" objective required to get learning off the ground in this setting? Assuming the LSTM is not pre-initialized (which I don't think it is), It seems extremely difficult to learn a good NLG policy from scratch given the massive complexity of the natural language search space. What happens when this objective is turned off? How often is natural language feedback given?
- I find the use of "learning from feedback" to be a bit misleading compared to how the term "feedback" has been traditionally used in the literature (e.g. [1](https://arxiv.org/abs/2204.14146), [2](https://arxiv.org/abs/2009.14715)) - "feedback" implies some language corrections or explanation for why a referring expression is correct/incorrect, but in reality this is just providing the ground-truth label to the agent (which you then optimize for explicitly).
- Notation is a little convoluted at times, e.g. eq 7 $w_{u^j_1}$, is it really necessary to have these multiple nested subscripts?
- I think the conclusion statement that "psychological modelling communities...consider the further incorporation of ToM into language learning simulations" may be slightly uncouth, given that theory of mind is a central idea in an extremely long line of literature in both computational and non-computational pragmatics dating all the way back to Grice.
- I would advise authors to be careful about what constitutes "better" language, and claims that the speakers trained with harder distractors have strictly higher obejctive measures of "utterance quality". If we adopt a fully pragmatic view of language, then the best language should be as concise as possible given the context, and with simpler distractors, it makes sense that the language is simpler - that is the world they are trained on, after all. It'd be interesting to see whether speakers trained on a mixture of context difficulties can modulate the complexity of their utterances when the context demands it.
clarity,_quality,_novelty_and_reproducibility: Seems clear enough - I hope authors will release code and data if accepted.
summary_of_the_review: To summarize, I think this paper presents a technically sound set of experiments, although many of the findings here are quite similar to findings that already exist in the computational pragmatics literature. Another weakness is the several seemingly ad-hoc hyperparameter design choices that go into the environment on which the authors base their conclusions.

However, the idea of learning an internal listener model for a pragmatic speaker, is, to my knowledge, new, and is likely to be of interest to researchers in this space. Based on this, I am marginally in favor of acceptance, though I don't (at the moment) feel particularly strongly and look forward to the other reviewers and the author response.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is inspired by cognitive science that young children actively acquire language through interactions with their surrounding environment and caretakers. Specifically one critical mechanism of language learning is the ability to infer the mental states of other agents in social environments, referred to as Theory of Mind (ToM) by Premack & Woodruff (1978). Departing from current approaches of language models,  the paper presents language-learning agents equipped with ToM.  It models ToM by giving the speaker agent an internal listener model that is trained alongside the speaker and using this ToM model to rerank potential utterances. The paper also experimented with varying task difficulty assuming that stronger environmental pressures will promote the development of more complex language. The paper shows that speakers trained with a ToM listener component have higher accuracies than those trained without in our image referential game setting. It further observes that increasing task difficulty in the training process results in more fluent, higher-quality utterances in evaluation. 
strength_and_weaknesses: Strength
1. Inspired by theory of mind in cognitive science, a critical mechanism of language learning is the ability to infer the mental states of other agents in social environments, the paper presents language-learning agents equipped with ToM.  

2. The paper finds that incorporating ToM into the speaker models leads to improvements in speaker performance and incorporating harder distractors leads to the development of more complex and fluent languages.

Weaknesses
1. The problem setting is very limited with vocabulary of 200 words and the max utterance length is 20 tokens. Yet the paper makes very strong claims on answering two fundamental questions:
RQ1. Does the inclusion of ToM in language acquisition models improve the performance and learned language of said models? (internal ToM mechanism)
RQ2. How do our models’ learned languages adapt to discriminate between more visually and semantically similar images? (external environmental pressure)

2. The model design is presented as it is. There is no discussion on alternative designs and whether the design choice achieves the desired goals. For example, why LSTM is used. Would transformer or pretrained language model be a better choice.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written until section 3.2. Eqn. 5. O_CG is not defined. 

It seems to a straightforward application of adding ToM to the speaker and listener formulations defined in Zhu et al. (2022).
summary_of_the_review: As it current stands, the paper added theory of mind to the standard speaker and listener formulations. The model design is also very straight forward. The evaluation is on a very limited environment. Why the very limiting image referential game environment is the most relevant for computational language acquisition with theory of mind? What are the difficulties to add ToM to big multi-modal language models such as Flamingo?

In its current form, I do not find the results appealing to the ICLR community.

== post authors' response, and reviewer and area chair discussion
The authors' response is very defensive in nature and not convincing. For example "Additionally, we believe that the image referential environment strikes a balance between ease of investigation and realisticness." Why is the image referential environment strikes a balance and why is it realistic? What other environments are you comparing with?

However, after discussing with other reviewers and the area chair in a virtual zoom meeting, I am a bit more positive on the limited contribution of this paper. I am willing to upgrade my rating to 5: marginally below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes an image captioning task to represent language acquisition through its structure.  The learner or "speaker" is the image captioner.  The feedback is provided via a "listener" that provides feedback in the form of a ground truth caption. The innovation comes from the inclusion, in the speaker model of a representation of the listener, the "theory of mind" (ToM) listener.

The authors find that 1) the model learns better with a ToM listener and 2) the outputs are more complex when the feedback is based on difficult distractors rather than easy ones.
strength_and_weaknesses: Strengths
* Clear connection from the problem statement to the empirical experimentation
* Clearly written. Relatively easy to understand results and modeling formulation

Weaknesses
* While the findings related to the inclusion of the ToM component are obvious from Table 1, the effects of distractor difficulty in Table 2 are less obvious. It's not obvious how the Model Distractors are connected to the claims of distractor difficulty.  However, the distinction to the Gold Standard are very clear.
clarity,_quality,_novelty_and_reproducibility: Clarity
* Mostly very clear.  One question about the material in Table 2

Quality
* High quality focused contribution.

Novelty
* Sufficiently Novel. Tackles a compelling research question from a novel direction.

Reproducibility
* Reasonably described to promote reproducibility. Authors state code and data will be releated.
summary_of_the_review: This paper describes an approach to evaluate claims of language acquisition through image captioning.  The experimental setting is well motivated and reasonably evaluated.  With the exception of the claims about linguistic complexity being influenced by difficulty of distractors, the findings are clearly evidenced by the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
This paper shows that inserting a Theory of Mind module into an image captioning agent playing a referential game leads that agent to learn more effective and fluent language.

(motivation)
Some work in developmental psychology says that human can quickly learn and adapt language because they model the mental states of their social companions -- because they have a Theory of Mind (ToM). Recent work has studied image captioning agents that learn or adapt language while playing referential games. This work asks whether adding a ToM component into these agents improves the language they learn.

(approach)
The speaker sees a target image and captions that image. The listener sees the speaker's caption and must use it to pick the correct image from a list of distractor images. The listener also returns the ground truth caption for its selected image when it is confident, but not highly confident. These agents are initialized by pre-training on image-caption pairs. The speaker is finetuned to
1) make the listener guess the correct target (PPO objective) and
2) mimic feedback (GT captions) when given by the listener.

The speaker is also augmented with a ToM component. When the listener gives feedback on an image (potentially not the target) the ToM component is trained to

3) assign high probability to that image given the speaker's caption for it.
During inference potential speaker outputs are re-ranked to choose the utterance which the ToM speaker assigns highest probability.

(evaluation)
Evaluation focuses on the quality of the speaker's language, showing that
1) The listener picks the correct image (Accuracy) more often with the ToM speaker than without. ("Baseline" and "Zero" ToM Weight)
2) The speaker's language is more fluent with ToM (GPT-2 assigns it higher probability)
3) The speaker's language is more similar to ground truth captions (adpositions, nouns, verbs usage; sentence length)
4) Using hard distractors (image choices that are similar to the target) results in better language fluency, though not better Accuracy.

The paper concludes that adding the ToM component improves speaker fluency and effectiveness.

strength_and_weaknesses: 
Strengths
===

* I really appreciate the motivation for this work. Thinking about language acquisition from an evolutionary perspective seems important and the role of Theory of Mind in evolutionary language acquisition in AI has not been explored. This has also been specifically identified as in need of further study in the cited work (Lazaridou et. al. 2020).

* Futhermore, this particular computational relization of Theory of Mind seems new and the evidence points in the direction

* The idea of the listener giving feedback only in certain circumstances, and using the ground truth caption to do that is interesting and potentially novel.


Weaknesses
===

These are roughly ordered by importance.

* (quality) One thing that's still not clear to me is whether the improvement is really because the speaker is modeling the behavior of the listener or whether the ToM listener component is just making it better at image captioning in general. While the improvement of the "Zero" ToM Weight models over the "Baseline" can only be due to a general improvement in image captioning ability, the additional improvement of the "High" ToM Weight models could still be purely due to an additional increase in general image captioning ability. This could be measured by looking at the general captioning quality of the various models. If caption quality stayed the same but listener choice accuracy increased then that would indicate the improvement is really due to the speaker's model of the listener. Even if caption quality went up some component could still be due to the speaker's model of the listener, so I don't think the issue is straightforward to resolve. I think the available evidence points toward some contribution from the ToM component, but more work is needed to resolve this with high confidence.
    * (suggestion) It would certainly help to have a better measurement of the speaker's overall captioning quality. The Part Of Speech F1 and Average Length metrics give some sense of this, suggesting the "High" models are better captioners, but it should be measured with a more established metric like BLEU, SPICE, or CLIPScore[2].

* (clarity, quality, novelty) The connection between ToM and distractor difficulty isn't very clear, and distractor difficulty isn't very well motivated for studying for its own sake. This makes it hard to see why the content of section 6.3 was important to investigate.

* (quality) The listener feedback mechanism is interesting, but the paper does not address how it affects performance in the experiments. How often does the ToM listener receive GT feedback? How does that change over the course of training? This matters because we expect NNs not to be well calibrated by default, and over-confidence could result in this mechanism simply being ignored (with P_max always greater than theta_2 in eq. (2)).

* (clarity) The notation is confusing at times. The reranking weight and caption words both use the letter w with different subscripts. The w_{n^u} notation is never explicitly defined. (Is n the summation variable in eq. 10? Isn't P_ToM a conditional distribution?). I can't understand how the ToM listener is trained because I don't understand eq. 10. Is it supposed to assign high probability to all possible choice images given the caption that the speaker assigns them? Or is it suppose to assign the same label as the actual listener given the choice already made by the listener and the caption the speaker assigns to that choice?

* (quality) The paper does not argue (either empically or rhetorically) whether the diverse choices used in re-ranking are diverse enough for re-ranking to be very meaningful. Often LM samples can be rather redundant and may not always include good choices. Is that the case here?

* (novelty) This work does not cite [1] even though it is very similar. Both works take an image captioning speaker agent and add an internal component which makes the speaker produce utterances that are more discriminative between target and distractor images. I think this work is still different because it is motivated by ToM and thus incorporates feedback from the listener to direct the discriminativity, unlike [1].

* (clarity) RQ1 and RQ2 are not very clear as stated. In RQ1 I see "the performance" of "said models" as referring to their ability to get the other agents to perform well. If it is the listener then perhaps this makes sense, but at this point its not clear whether the listener or the speaker is referred to as "said models" This is also the first mention of RQ2, or at least I'm not immediately sure of how it relates to the intro having been read so far or how it relates to RQ1. A clear definition of what the environment is would be helpful here. Later on there is also a disconnect in flow between section 6.2 and section 6.3, and this would help improve that.

* (clarity) In section 2.1 I is initially referred to as a _set_ of images, while I^N seems to be referencing the same set later. For consistency I^N should be used in the first case or some other adjustment should be made.

* (clarity) "However, the language network parameters are additionally trained [...] to optimize the network values": I don't understand how the listener is trained. Is the COCO train dataset is used?

* (clarity) What exactly are the "easy" distractors? I'm guessing they are randomly selected COCO images, but that should be verified and made clear in the text.

* (clarity) The "Return" metric is not defined or referenced in the results section.



[1]: Vedantam, Ramakrishna et al. “Context-Aware Captions from Context-Agnostic Supervision.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 1070-1079.
[2]: Hessel, Jack et al. “CLIPScore: A Reference-free Evaluation Metric for Image Captioning.” ArXiv abs/2104.08718 (2021): n. pag.

clarity,_quality,_novelty_and_reproducibility: 
Quality: The evidence provides support for the conclusions about the impact of adding a ToM component. However there are some shortfalls in the approach that would make the support more clear if addressed.
Clarity: The paper is easy to read, but some pieces of information are missing or vague. The main ideas can be understood.
Novelty: The idea of adding a ToM component to improve langauge adaptation in referential games is novel.
Significance: This may inspire others to improve their language models by adding ToM components. Some may be watching for this sort of experiment (Lazaridou et. al. 2020).
Reproducibility: Some missing details may prevent reproducibility.

summary_of_the_review: 
The novelty and potential significance of the work make it a relevant contribution to this community. While the quality and clarity of the work have some flaws, in their current state they are good enough to comprise a meaningful contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper develop language-learning agents equipped with a theory-of-mind (ToM) module. They consider referential games, where a speaker agent learns to use language to communicate to a listener that has to pick a target image only known to the speaker. As pointed out by reviewer eJ14, this setting has been explored in prior work, but the key contribution of this paper is to add a ToM listener simulator within the speaker, where the simulated listener model is learned using listener feedback in interactions. This allows the speaker to optimize both truthfulness and listener utility and generate appropriate utterances. 

Experiments are mixed but some variants of the method work better than baselines that do not learn a listener model using ToM. The paper also performs some analyses that may inform future work in this direction. The main criticisms of this paper were the fact that the vocabulary is quite limited (200 words) and a max utterance length of 20 words (Z8Jq) and missing contextualization with related work (eJ14). The authors promised to resolve the latter with revisions to the paper. If accepted, I strongly encourage the authors to tone down the claims in the abstract and introduction to better reflect scope of the actual empirical results in the paper (and mention upfront the limited scale of the experiments).
justification_for_why_not_higher_score: See reviewer discussion above
justification_for_why_not_lower_score: I think overall the paper presents interesting ideas and although there are concerns around the experimental setup being too simple, I think the findings could help future work in this direction. If there is space in the proceedings, I recommend acceptance but the decision can be bumped down.
Decision: Accept: poster