Computational Language Acquisition with Theory of Mind
Authors: Andy Liu
Hao Zhu
Emmy Liu
Yonatan Bisk
Graham Neubig
Venue: ICLR 2023 poster (2023)
Keywords: language acquisition
theory of mind
referential games
natural language processing

Abstract: Unlike current state-of-the-art language models, young children actively acquire language through interactions with their surrounding environment and caretakers. One mechanism that has been argued to be critical to language learning is the ability to infer the mental states of other agents in social environments, coined Theory of Mind (ToM) by Premack & Woodruff (1978). Drawing inspiration from the modern operationalized versions of ToM implemented in Rabinowitz et al. (2018) and Zhu et al. (2021), we build language-learning agents equipped with ToM, and measure its effects on the learning process. We model ToM by giving the speaker agent an internal listener model that is trained alongside the speaker and used to rerank potential utterances. We experiment with varying task difficulty, hypothesizing that models will acquire more complex language to adapt to stronger environmental pressures. We find that training speakers with a highly weighted ToM listener component leads to performance gains in our image referential game setting. We also find some evidence that increasing task difficulty in the training process results in more fluent and precise utterances in evaluation. This suggests the potential utility of further incorporating ToM, as well as other insights from child language acquisition, into computational models of language acquisition.

Published as a conference paper at ICLR 2023
COMPUTATIONAL LANGUAGE ACQUISITION WITH THEORY OF MIND
Andy Liu Harvey Mudd College Claremont, CA, USA {ajliu}eg hmc. edu
Hao Zhu, Emmy Liu, Yonatan Bisk, Graham Neubig Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, USA {zhuhao, mengyan3, ybisk, gneubig)@cs. Jomss..mmmmmu. edu
ABSTRACT
Unlike current state-of-the-art language models, young children actively acquire language through interactions with their surrounding environment and caretakers. One mechanism that has been argued to be critical to language learning is the abil- ity infer the mental states of other agents in social environments, coined Theory of Mind (ToM) by Premack & Woodruff (1978). Drawing inspiration from the modern operationalized versions of ToM implemented in Rabinowitz al. (2018) and Zhu et al. (202I). we build Language-learingg
agents equipped with ToM, and measure its effects on the learning process. We model ToM by giving the speaker agent an internal listener model that is trained alongside the speaker and used rerank potential utterances. We experiment with varying task difficulty, hy- pothesizing that models will acquire more complex language to adapt to stronger environmental pressures. We find that training speakers with a highly weighted ToM listener component leads to performance gains in our image referential game setting. We also find some evidence that increasing task difficulty in the training process results in more fluent and precise utterances in evaluation. This suggests the potential utility of further incorporating ToM, as well as other insights from child language acquisition, into computational models of language acquisition
INTRODUCTION
Human languages are fundamentally shaped by nocial-ommuuicative goals in the grounded world. Modern theories from developmental psychology often attribute humans' unique ability to quickly acquire and adapt language to their ability to ascribe mental states to other agents (Tomasello) 2005). an ability also known as Theory of Mind (ToM).
Some previous studies have attempted to perform computational modeling of ToM. For instance, ToM-like mechanisms have been demonstrated to allow models to better predict the behavior of future agent (Rabinowitz et al., 2018), model agents' beliefs in negotiation (Cao et al. 2018) or cooperative game (Bard et al. 2020), choose good utterances based the listener's 's linguistic abilities (Zhu et al. 2021). However, the effects ToM have not yet been studied in the higher-level context of computational language acquisition.
In this paper, we study how an internal ToM mechanism and external environmental pressure con- tribute to language learning. We training episodes between a speaker, which represents language learner (Zhu et al.) 2022). and listener, which represents fluent teacher. When presented with a set of images, one of which the target referent, the speaker must learn to generate an English utterance that the listener can use to select the target. The speaker rewarded for generating utterances that are used to correctly guess the target image. Additionally, the speaker may be given feedback depending listener has the selection. This setting provides an attractive test-bed for testing the effects of various reward signals or model designs on the speaker's learned language; previous studies of pragmatics in language acquisition, such as Andreas & Klein (2066), have used similar settings.
image referential game setting consisting of series of
the confidence
"ooee and data found at wttpthtttsttt//tt wwwptttt....ccccbcbeeeeeaaaat htwp:/aaaaboooooaaaaaaaaeeae anguage Acquisitio

---

Published as a conference paper at ICLR 2023
Target Image
Distractor Image
1. man frisbee
2. Frisbee front trees
1. Three people play frisbee
3. Three people play frisbee
2. Frisbee front of frees
3. Yellow shirt man throw frisbee
yellow shirt man throw frisbee
Candidate Utterance Sampling
ToM Listener Reranking
Output Utterance
An example of how ToM is used by our speaker models in our implementation. The speaker first generates candidate utterances before reranking them according to the probability given to each target image, utterance pair by the internal ToM listener. It then selects either the highest- scoring or a random utterance from the candidate pool to output.
Figure 1:
Within this setting, we seek to better understand how models with and without ToM adapt to their environments. We focus on two specific research questions in this area:
RQ1. How does the inclusion of ToM in language acquisition speaker models affect their perfor- mance and learned language? (internal ToM mechanism)
RQ2. How do our models adapt to more difficult referential game environments during the lan- guage acquisition process? (external environmental pressure)
We study the impact ToM (RQ1) by modeling an internal listener module within our speakers that aims predict which utterances most likely result in the desired listener behavior. By incor- porating the probabilities given to the target image by the ToM listener into the utterance reranking process, shown in Fig. I select for more pragmatic utterances sampled from the speaker's distribution. To study the impact of environmental pressure (RQ2) on speakers, we create ref- erential games with different difficulties by sampling distractors from different distributions. These distributions are based on the similarity between images calculated by various image representation models, including CLIP (Radford al. 202T), RoBERTa (Liu et al. 2020), and TF-IDF variants.
In experiments, we find that (RQ1) speaker models including ToM components generally outper- form those that do not in terms of fluency and final accuracy. We also find that (RQ2) training with more visually and semantically similar distractor referents causes the speaker model to develop longer, more fluent, and more precise utterances to distinguish between potential referents, although these do not always translate to gains in referential game performance. These results suggest tributions to language acquisition from both ToM and environmental pressures in this setting. We still find significant gaps between the language that speaker model acquires and human captions. Additionally, we restrict both the vocabulary and the maximum length of our speakers' utterances. These both suggest that there is still room for improvement in this class of models. However, hope our results still hint at both better training methods for pragmatic language models and deeper computational understanding of human language acquisition.
IMAGE REFERENTIAL GAME ENVIRONMENT
Following Lazaridou et al. 2016): Lowe et al. 2019 Zhu et al. 2022). we consider image ref- erential games with real world images, where speaker and listener collaborate identifying target image x E C among distractors randomly sampled from a set of candidates C. The identity of the target is known only to the speaker. The speaker generates English-language utterance
2

---

Published as a conference paper at ICLR 2023
that describes the target image, which is then passed the listener. The listener will either select an image &, in the case where they understand the utterance with a high enough probability or refuse to act, in the case where they do not. The listener can also determine whether not to provide feedback (linguistic input in the form of the ground-truth caption of the target image) to the speaker.
Speaker training is motivated by reward function that seeks model the communicative goal of directing the listener to the correct referent. The speaker gets a positive reward of if the listener chooses the target image and penalty of -1 if the listener chooses the wrong target image. Ad- ditionally, a smaller penalty Whoop is applied to the speaker if the listener chooses to not choose an image; this done to penalize the speaker for generating unclear utterances.
2.1 AGENT FORMULATION
We follow the speaker and listener formulations defined in Zhu et al. (2022). The speaker is model f I + Σ" that generates utterances (sequences of tokens from its vocabulary) that correspond to provided image. We define E as the speaker vocabulary and the of candidate images.
The listener is defined on Σ' and 1 which is a set of N candidate images sampled from 1. Given an utterance 2n and an observation space E IN, the listener will either return the index of its predicted image or noop, if it cannot predict an image. It may also return the ground-truth caption from its vocabulary. We define it with the model g Σ" IN ((0. 1... 1] noop) x Σ.
2.2 SPEAKER DESIGN
The speaker that we use is a captioning model composed of pretrained ResNet (He et al. 2016) model and an LSTM-based utterance generation model. After generating an embedding vector of the target image using the ResNet model, the speaker autoregressively generates an utterance = {u: h using the LSTM network:
P(u: d1, .2...... 11
oc exp(w" ("p LSTM(W WW,, Wazs..., -1,-1: hg = ResNet(z))):
where Wu, E Rd is the word embedding of u4. The speaker generates sequence of tokens to either the maximum length, which is set to 20 in our experiments, or an end-of-sequence token. In experiments, we use vocabulary size of 200 limit the size of the action space and more realistically mimic the vocabulary a language learner.
2.3 LISTENER DESIGN
Given a set of candidate images I, (j E 1,2...n. 1), where n is the number of distractors, and an utterance u, the listener computes embeddings for each image, I(Tī), for the utterance, I(u). It then takes the dot product of each L((,,) 1,) with T(u). and the softmax of the dot products to compute the probability P(4ylu) x exp(L(4))L(u)) of the utterance referring image 15 for each image.
The listener also has rule-based component to control when to give linguistic input in the form of the ground-truth caption C'xx). We introduce the most likely target image, t = arg max P(I,lu), [),, and the probability given to this choice, Puax max P(I u). it returns its choice and input flistenex according to the following logic:
thresholds, 02 and 0z. After the listener computes
(noop, 0) Pmax 01 (t, (t))
(i, fiistenez)
81 < Pumxx < 02
(t, Pmax 02
This control strategy mimics caregiver who only gives feedback when they understand what the language learner is referring to, but wants to direct it to generate nigher-quality utterances. When the listener's confidence is very low, it will neither select target image nor provide linguistic input, as the speaker utterance is too low-quality, Meanwhile, when the listener's 's confidence is very high, it will stop giving linguistic input, further improvement is deemed unnecessary.
3

---

Published as a conference paper at ICLR 2023
We use LSTM-based model learn embeddings of speaker utterances combined with ResNet model to derive image features. Because we want our listener to at least model a competent user of the language that the speaker is trying to acquire, some pretraining is required before initializing listener in a new environment. The controller values 0,, 02 and the ResNet parameters are set after preliminary experimentation the dataset. The listener's language network parameters then trained with mini-batch stochastic gradient descent to optimize the network values
1 N
Oistener arg max R
REC
log Pistener U, C)0)
(3)
B
3 SPEAKER TRAINING PROCESS
3.1 COMMUNICATIVE GOALS
We model communicative goals and learning from linguistic input as two separate learning objec- tives for our speaker network, which we combine with an adjustable coefficient X. The communica- tive goal reward Ocg is simply the average reward (defined in 2 over each game in the training episode. The space of possible output utterances is discrete and non-differentiable, so speaker learns its policy π via the reinforcement learning method PPO (Schulman et al. 2017).
3.2 LEARNING FROM LINGUISTIC INPUT
We model the learning from linguistic input objective the maximum likelihood of the listener input in the speaker models and optimize this continuous function with stochastic gradient descent.
OLI Ex, (un((l))()l))) log (U (a, x C)
(4)
We then combine the separate learning objectives for each task using coefficient E [0, 1]:
O juint = XOcG +1 (1-1))01
(5)
Next, we introduce ToM Modeling, and an additional task objective, OTOM. This is cross-entropy objective that represents how accurate the speaker's 's internal listener model is. Similar to the objec- tive above, we combine these into single learning objective for the entire speaker network.
INTRODUCING ToM MODELING
We incorporate into the speaker architecture LSTM-based ToM listener. This listener model is trained alongside of the rest of the speaker network to learn the "mental state" of the actual listener the neural network parameters that allow it to best replicate the probabilities that the listener would assign to image-utterance pairs. It uses a similar architecture to the actual listener, combining its own learned sentence embeddings from LSTM network with pretrained image embeddings using ResNet. In other words, our ToM listener seeks to learn the sentence embedding model @roM that will maximize the probabilities assigned the listener's choice given an utterance.
BroM arg max ProM (u.(a) exp(0# (u:) ResNet(())
We introduce ToM into the speaker model by having our speaker sample candidate utterances and rerank them with the help of the ToM listener. Our ToM speaker first samples N candidate utterances W, from its distribution. Next, we generate a speaker and a listener score for each:
P Papeaker (1) x II P(us (u1, u2, 11,1
Prox eXp(LSTM (M) (w))esnet(2:)
(8)
4

---

Published as a conference paper at ICLR 2023
Here, the probability P(ug/ug, Ug..... u. is computed according We then combine these scores using a listener weight hyperparameter wi to get the overall score assigned to each utterance. Finally, we select the best utterance, u², as the argmax of this score:
u arg max (ara((((/))))) m Papeaker (u²))
In our experiments, we train models with three different settings of wy. We train models with wy = to isolate the effects of the ToM listener from the effects of the new model architecture. We train models with wi = which weigh the speaker and ToM listener input equally. Finally, we train models where wt is the arbitrarily high constant 1000. In this case, the listener score dominates, and our speaker essentially seeks to maximize P istener ( which it approximates with the ToM listener score. If we replace the learned ToM listener with copy of the external listener, this utterance sampling process is close to that of the learned Rational Speech Act (RSA) model in Andreas Klein (2016), which also trains pragmatic speakers on referential game environments. We compare our ToM speakers to these "RSA" speakers to evaluate the impact of using our learned approximation rather than the actual listener.
Finally, we introduce hyperparameter a: our speaker outputs a random utterance with probability a and u² with probability - a. a is set to decay linearly over time; this randomization is done to promote early exploration.
By default, we begin with an untrained listener that will be trained to emulate the actual listener's outputs over time. To train the ToM listener, we introduce third training objective in addition to Our and Oca. the ToM objective OToM, defined as the cross-entropy loss between the distribution of the ToM listener and that of the true listener. Thus, to the listener's choice & based on the speaker's 's utterances. Formally,
training it to give higher probabilities
OтoM- = { -llg ProM (+)u P Pinax > 0
(10)
Pinax < 01
Note that this requires choice be made by the listener, so we mask the case where the actual listener does not choose an image by setting it to zero. This is done using the same parameter 0 and listener confidence Pmax that were used control linguistic input in We add this the joint
learning objective previously defined in to compute our combined objective for the ToM speaker system as whole:
Ojoint XOCG + (1-))) + OToM
(II)
However, an untrained listener can introduce significant noise to the utterance generation process. To counteract this, we anneal the influence of the listener, by linearly increasing wi from 0 to the final wy value over fixed number of steps. This allows our speaker to only begin using its ToM listener when the listener at least somewhat capable.
INCREASING DISTRACTOR DIFFICULTY
Previous studies on language acquisition have found that infants initially look at familiar objects when hearing semantically related labels for different objects, but adapt to more difficult tasks over time by learning more complex semantic representations of objects (Bergelson Aslin 2017a). Ad- ditionally, Yuksekgonul et al, (202)) showed that contrastive training on harder distractors improved visiolinguistic models' performance on tasks involving compositionality. We hypothesize that the usage of more similar distractor images might similarly force our speaker generate more complex utterances due to the need to further distinguish between images. This motivated us to generate more similar distractor images in the training process in order to achieve such effect. To do so, we computed a similarity rank between images based on visual and semantic similarity. Then, after selecting "target" image in the training process, sampled images with high similarities to the target image to use as distractors during training.
We experiment with three options for calculating image/caption similarity:
5

---

Published as a conference paper at ICLR 2023
Original Image
Random Distractor
Image Similarity
Caption Similarity
Hybrid Similarity
a little girl holding kitten next blue fence.
two sheep standing each other the snow.
little dog sitting on wooden bench.
woman fugs a gray to her chest.
a kitten sitting down by door.
Figure 2: An example image-caption pair, and distractors ranked as similar by various metrics. Caption and Hybrid similarities are computed with TF-IDF- weighted RoBERTa embeddings.
Visual Cosine Similarity. We compute the most visually similar images based on the cosine similarity of image embeddings from pretrained ResNet model. For each image, we save a similarity ranking of the next 1000 most visually similar images, and then select distractors from this set during training whenever the original image is selected as a target.
Textual Cosine Similarity. We calculate the textual similarity of captions by taking the cosine similarity of vector representations of the text. We experiment with both (1) dense vectors using embeddings calculated using either the pretrained RoBERTa or the pretrained CLIP mean pooled models from the uuntence-transfomee
library (Reimers & Gurevych 2019), and (2) sparse vec- tors using one-hot vectors each word. We also experimented with using term frequency inverse document frequency (TF-IDF) to weight the vector for each token in captions when computing caption similarity as a way to upweight rarer content words.
Visual+ Textual Similarity. In cases where both image and caption similarity were used, added a caption weight parameter we E [o, 1] and used it to compute weighted average of image and caption similarity. We hoped that using hybrid methods (w, 0.5) would capture distractors with both visual and semantic similarities.
We also train models on randomly selected, or "easy", distractors create a baseline study the effects of distractor difficulty. Examples of a selection of the distractor settings are shown in Fig,
EXPERIMENTAL SETUP AND RESULTS
6.1 EXPERIMENTAL SETUP
We focus two main results: A speaker's ability to use language to correctly discriminate between the target and distractor images, as well as the quality of the language speakers use to do so.
To evaluate the former, primarily consider average accuracy, how often the listener selects the correct referent. We evaluate the quality of our speakers' learned languages using fluency, average length, Fl score for salient parts of speech, and caption quality scores. We a fluency score that measures the grammatical quality of output utterances (Kann et al.) 2018). This is defined as the average gain log probability when moving from unigram model a pretrained language model.
fluency =
(In(pM(4)) (n))
(12)
[u]
We train unigram model, pv, and fine-tune GPT-2 large (Radford et al. 2019), PMr on training set.
In addition to fluency and accuracy, consider two additional performance metrics. To better evaluate the overall caption quality of the system, we compute BLEU score (Papineni et al.) 2002) using the implementation found in Bird et al. (2009). Additionally, to evaluate the accuracy of the learned listener model, we compute ToM Accuracy, defined as how often the candidate image that the ToM listener believes has the highest probability of being chosen is the actual listener's 's choice.
We also consider pfrt-of-speech statistics to evaluate utterance quality. Parts of speech for words in speaker utterances and ground-truth captions tagged with spaCy (Honnibal et al.) 2020). We use

---

Published as a conference paper at ICLR 2023
Target
Distractor
Distractor
a group of men on a field playing
person on some skis.
white building with clock the front and side of it.
baseball.
Base Speaker: baseball player holding holding umbrella.
Speaker Trained on Hard Distractors: baseball player holding bat a field.
ToM Speaker Trained on Hard Distractors: group men are playing baseball outside.
Figure 3: Examples output utterances generated by various speaker models. Here, the "Hard Distractors" referenced selected with hybrid metrics computed with CLIP embeddings. Both training on more difficult distractors and training with ToM leads to more accurate, fluent utterances.
Table 1: Performance and language features of various ToM speakers.
Model
Performance Fluency
POS FI NOUN
ToM Weight
Distractors
Acc
BLEU
ToM
ADJ
ADP
VERB
Baseline (No ToM)
Easy
0.81
0.20
1.50
N/A
0.16
0.52
0.41
0.38
Baseline (No ToM)
Hard
0.81
0.24
1.87
N/A
0.24
0.58
0.46
0.45
Gold Standard
N/A
0.92
1.00
2.52
N/A
1.00
1.00
1.00
1.00
Zero
Hard
0.83
0.26
1.99
0.81
0.22
0.64
0.49
0.47
Normal
Hard
0.85
0.26
2.25
0.88
0.22
0.65
0.52
0.49
High
Hard
0.88
0.27
2.23
0.89
0.22
0.66
0.52
0.50
High RSA
Hard
0.87
0.28
2.26
0.93
0.23
0.65
0.50
0.49
Zero
Easy
0.85
0.25
1.73
0.85
0.21
0.57
0.48
0.49
Normal
Easy
0.88
0.26
2,09
0.91
0.21
0.64
0.50
0.52
High
Easy 0.88
0.27
2.07
0.91
0.22
0.65
0,51
0.50
High RSA
Easy 0.89
0.29
0.65
0.52
0.49
this to compute F1 scores between speaker utterances and ground-truth captions over each part of speech. This allows us to evaluate how effectively our speakers identify referents within the target image that human captioner would consider salient. Higher Fl scores over a part of speech would suggest that speaker more capable of identifying concepts that fall under that part speech.
Images and captions are drawn from the MS COCO dataset introduced in Lin et al. (2014). The tran-val-test split given by the MS COCO 2017 dataset is extended to our experimental setup.
6.2 EFFECTS OF ToM
We train models equipped with ToM listeners with normal weight (i.e. equally weighting the speaker and the listener scores when reranking output utterances) and with high weight (where the listener weight is arbitrarily 1000 and the speaker essentially optimizes for ((x(wx)). In order to isolate the effects of using ToM listener from the effects of modifying the utterance selection process, we also train model equipped with a ToM listener that it does not give any weight to. Additionally, train a model without any ToM component for comparison. We do this for both easy and hard distractors, where the hard distractors were those that were chosen by the hybrid similarity between visual and semantic (CLIP) features outlined in 5 Finally, captions over the test set to compute gold-standard metries of accuracy and fluency.
give the listener the ground-truth
We find significant performance improvements in Table I when speaker models are trained to rerank utterances solely by ToM listener score. Such "high-weight ToM" speaker models achieve accuracy gains of 3.0% and 4.6% on easy and hard distractors, respectively. This suggests that the inclusion of a sufficiently influential ToM reranker during the speaker training process improves speaker per- formance, although the relative gains appear to be much higher when training easy distractors.

---

Published as a conference paper at ICLR 2023
Table 2:
Performance and language features of speakers trained on various distractors. We only
show the most performant variants of Caption and Hybrid similarity, with the others shown in A.1
Model
Performance Ace
POS F1
Average Length
Distractors
Fluency
ADJ
ADP
NOUN
VERB
Base
0.81
1.50
0.16
0.52
0.41
0.38
8.97
Gold Standard
0.92
2.52
1.00
1.00
1.00
1.00
10.79
Image 0.80
2.09
0.19
0.61
0,45
0.49
10.33
Caption 0.86
2.01
0.19
0.62
0.49
0.48
10.04
Hybrid 0.85
2.19
0.20
0.62
0.49
0.47
10.18
However, we find that speaker models that rerank utterances using a combined speaker- Took ToM score generally fail outperform models that do not use their ToM listener in training.
We also find that the usage of a highty ToM listener leads to significant fluency gains when training on both easy (15.6% relative increase in fluency score) and hard (11.6%) distractors. We also see longer and more complex utterances when using normally or highly weighted ToM listeners. Additionally, we find limited gains in general captioning ability between baseline and high-weight models, as measured by BLEU score. However, these effects are more subtle, and do not always lead significant accuracy gains, suggesting that the main driver of ToM accuracy gains increased pragmatic ability. We conclude that usage of highly influential ToM listener during the training process leads significant performance and fluency gains. We are also able qualitatively observe the improvement in model performance from ToM. As seen in one representative example in Fig. 3 our ToM Speaker is able to identify two elements that clearly distinguish the target image from distractors (i.e. that there are multiple men who are playing baseball) in fluent utterance.
Finally, we find that the ToM listener successfully approximates the external listener. Models with learned listeners and RSA models with the pretrained listener perform comparably in accuracy and fluency. Because the RSA models represent the upper bound of how good a speaker's listener model can be, this suggests that our learned listeners very beneficial to the speakers. This is also shown through the high ToM accuracies reported, especially in the most performant models, those with high listener weight. These qualitative and quantitative results provide computational evidence that ToM can play an important role in simulated language acquisition, similarly to how it has been hypothesized play critical role in human language acquisition.
6.3 EFFECTS OF DISTRACTOR DIFFICULTY
As shown in Table [] we generally find significant improvements in language quality in models trained on more difficult distractors. The largest gains are those seen in the fluency score, where difficult distractors achieve gains ranging between 25% to 46%. We also find that models trained on more difficult distractors more similar vocabulary to the ground-true truth captions, measured by Fl score in the ground-trute captions and utterances produced. This is significantly higher over adpositions, nouns, and verbs models trained with more difficult distractors. Finally, all speakers trained on difficult distractors generate more complex utterances compared to the base speaker, with utterances that are at least one word longer on average. This supports our hypothesis that when confronted with increased environmental pressure, the speaker adapts by becoming more precise, fluent, and complex with its language. These can be seen qualitatively one representative example in Fig. 3 which shows that a speaker trained on hard distractors is able to generate more fluent utterances that more precisely describe the image (in this example, correctly identifying an object in the image as a baseball bat, as opposed to an umbrella).
We find smaller differences between the language of models trained with various types of hard dis- tractors. Speaker models trained with visually similar distractors achieve the highest fluency, at 2.094, and form the longest utterances. They also have more precise verb selection, as measured by Fl. Speakers trained on distractors that were selected with hybrid or caption similarity, achieved high noun Fl scores of 0.49 compared to 0.41 for the base speaker model, indicating that semanti- cally similar distractors in training may be better for identifying salient nouns. We find that training on more difficult distractors does not consistently improve model performance when evaluating on easier distractors. This suggests some disconnect between language's 's fluency and its suitability to
8

---

Published as a conference paper at ICLR 2023
the image referential game environment. However, speakers that train more semantically simi- lar distractors still achieve up to 5 percent higher accuracy than the base speaker, indicating some benefits to performance from training on certain harder distractors.
RELATED WORK
Parallels in Human Language Acquisition. The concept of learning language through repeated exposure to referents is a popular model within the psychology community. Smith & Yu (2008) found that infants resolve the uncertainty of determining which referent in scene word refers to by statistical learning over word-scene pairings. Yu & Ballard (2007) incorporated a social element into models by considering "social-cognitive" capacities, such as attention reading, that act as a form of ToM. Yu & Smith (2012) studied caregivers' social impact on early language acquisition using head- mounted cameras, finding that caregiver feedback during moments where referent dominant directly led to language learning in the infant. [Bergelson & Aslin (201 found that unrelated images were easier for infants to differentiate between than semantically similar images.
visually
Pragmatic Modelling. Andreas & Klein (016)) used pragmatic models to jointly train neural speaker and listener models to play a referential game that involved describing contrasting scenes. They also use a sample-and-rerank method for selecting utterances in this setup. However, they use the actual listener, rather than learned listener, rerank utterances. Additionally, we apply this process to a computational model of language acquisition. Nematzadeh et al. (2018) created dataset to evaluate question-answerre answering models' ability to keep track of inconsistent worldviews They found that state-of-the-an ofrtte-rrttttttt- neural models lack this ability, indicating that they cannot solve tasks with ToM. Monroe et al. (017)) studied the effects of visually similar distractors on a prag- matic model of color identification, finding that pragmatic models had the largest gains in the most difficult settings. Vedantam et al. (2017) also consider the effects of including pragmatic components in image captioning models, namely internal module to better discriminate between images.
Emergent Language. Lazaridou & Baroni (2020) surveyed recent progress in emergent language from multi-agent communication, claiming that further progress can help deep networks become more interactive and interpretable. Lazaridou et al. (2020) place a generally trained language model in multi-agent environment with task-specific rewards. Similarly
our work, this results in cask-conditional language model which the authors claim can better communicate with humans over visual tasks. Chaabouni al. (2022) analyzed the effects of task difficulty on emergent communi- cation tasks by varying the number of distractors, leading to negative effects on model performance during evaluation. Mu & Goodman (202T) attempted to improve interpretability of learned lan- guages in referential games by forcing speakers to communicate over sets of objects representing abstract visual concepts, and analyzed the compositionality of the ensuing emergent languages.
CONCLUSION AND FUTURE WORK
In this paper, we extend existing computational framework that models the language acquisition process in an image referential game environment. Most notably, we add a ToM component to our speaker models, allowing our speakers to pragmatically rerank candidate utterances with learned internal listener. We also experiment with increasing distractor difficulty by upweighting more se- mantically and visually similar distractor images. We find that incorporating ToM into our speaker models leads to improvements in speaker performance. We also find that incorporating harder dis- tractors leads the development of more complex and fluent languages.
Future work could measure the similarity of the learning process between human learners and our models, and whether the changes implemented this paper lead to a more humanlike learning process. Further work could also consider the implications of more dynamic difficulty adjustment or curriculum design for instance, studying whether models trained on a variety of distractor difficulties are able to adjust their utterances to fit context. Finally, we can study these effects in more complex environments by varying the listener architecture or by considering more difficult settings, such object referential games. We encourage the machine learning and psychological modelling communities to consider the further incorporation of ToM into computational models of language acquisition, which could help develop more pragmatically-awan models.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1: 1173-1182, 2016.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Elika Bergelson and Richard Aslin. Semantic specificity in one-year-olds' word comprehen- sion. Language Learning and Development, 13:1-21, 06 2017a. doi: 10.1080/15455441.2011 1324308.
Elika Bergelson and Richard N. Aslin. Nature and origins of the lexicon in 6-mo-olds. Pro- ceedings of the National Academy of Sciences, 111649)12116-1222,, 2017b. doi: 10.
1073/pnas.111296611 1712966114,
URL https: http://ww.ppaac org wwwddoiiabbbs.. 1t1.:////.1111111/aeee.mm pnas..
1712966114
Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O' O'Reilly, Beijing, 2009. ISBN 378-0-99--51649-9, doi:
htt.rr/ooosooiiiecccco/////56111 cow. 16499. URL wwwp://wwwwwwom. wwt.:/wwwkk.oo... www.:/ot....o.oooooooooooo
Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Leiboo Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. arXiv preprint arXiv: 804.03980 2018.
Rahma Chaabouni, Florian Strub, Florent Altché, Fugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent com- munication at scale. In International Conference on Learning Representations, 2022. URL
net forum? wtwp:uummm=-UUUA LeA.:uumIILLAAUUUUIIVVVVVE wow.enrrreeeeeeetooouuumm
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spacy: Industrial- strength natural language processing in python. 2020. doi: 00.2281/2emodo.12123000
Katharina Kann, Sascha Rothe, and Katja Filippova. Sentence-level fluency evaluation: References help, but can be spared! In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 313-323, 2018.
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communicatiin in the deep learning era. CoRR, abs/2006.02419, 2020.
00..000600221999 02419.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv: 1612.07788 2016.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. CoRR, abs/2005.0700 0.0000 2020. URL http://wwwaaa....////222005000.0
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. be A robustly optimized (bert) pre- training approach, 2020. CtLLaatt:://// httttp::///penrevvieee net "ww.eeoiimmmsSSai0
Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In International Conference on Learning Representations, 2019.
10

---

Published as a conference paper at ICLR 2023
Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 3:325-338, 2017. doi: 10.1 a 00064. URL https:
//acl antaanntooollo ogy. www.llyyo..ooggggo otg/p17-112/.
Jesse Mu and Noah Goodman. Emergent communication of generalizations. zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17994-18007. Curran Asso- ciates, Inc., 2021. URL
M. Ran-
neurips CC pttppppr//22///1eee.
Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. Evaluating theory of mind in question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pp. 2392-2400, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/1//D18-11 1261. URL httpss//aclamnhoooooy.oo///////
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02. pp. 311-318, USA, 2002. Association for Computa- tional Linguistics. doi: 10.3 (0.3115//073383311711333 URL https: / //cii. wtt.:/i...r110.11.......
1073083 1073135
David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 144):515-526, 1978.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pp. 4218- 4227. PMLR, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv org/ abs /ttp:////1..0000
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. URL htttps:/arrtt
org/ abs//ab//it awwaaa//...// 1908. 10084
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. https:://aaat..... http://wwwvv..r///////111..1111.
Linda Smith and Chen Yu. Infants rapidly learn word-referent mappings via cross-situational statistics. Cognition, 106(3):1558 1568, 2008 ISSN 0010-0277. doi: https//doi.org/ URL httpp: / http://www////ecc http://wwwweeeeeeeeeeeeeeeeeeeee
10. arricleep111500002277000
Michael Tomasello.
Constructing a Language: A Usage-Based Theory of Language Acquisition, volume 1. Harvard University Press, 2005.
Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. Context- aware captions from context-agnostic supervision. CoRR, foss/1701.0250 2017. (tLhttp:
//arxiv.org/abst
02870
Chen Yu and Dana H. Ballard. A unified model of early word learning: Integrating statistical and social cues. Neurocomputing, 70(13):2149-211 2165, 2007. ISSN 0925-2312. doi: https://doi.onr org/ 10.1016/jaeuccmm220660.0.000
URL https: //t:://wwww sci eeceeineiiee weeeedrrreeee.... com/ saann/ccence///// articllepii//025331111 50 Selected papers from the 3rd International Confer- ence on Development and Learning (ICDL 2004) Time series prediction competition: the CATS benchmark.
11

---

Published as a conference paper at ICLR 2023
Chen Yu and Linda B. Smith. Embodied attention and word learning by toddlers. Cog-
nition, 125(2):244-22 262, 2012. ISSN 0010-0277. doi:
2012.06.016. URL https:/ https:/www.sceenceeee
S0C1 0127000271111 200771200110 369
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-worddh and what do about it?, 2022. URL
https://aat http://axxvv... wtw.://gg/abbbaaooooooooooss
abs/ 2210..2210... 01936
Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In International Conference on Machine Learning, pp. 12901-12911. PMLR, 2021.
Hao Zhu, Yonatan Bisk, and Graham Neubig. Simulated language learning through communicative goals and linguistic input. Proceedings of the Annual Meeting of the Cognitive Science Society, 44:1351-1358, 2022.
A APPENDIX
A.1 FULL EFFECTS DISTRACTOR DIFFICULTY
In Table 2
we select the caption and hybrid distractors involved in the training of the most perfor- mant models to be reported. Here, as promised, we report the results experiments over fuller range of distractor variants.
Table 3: Performance and language features of speakers trained all distractor variants.
Model
Performance Acc
POS F1
Average Length
Distractors
Fluency
ADJ
ADP
NOUN
VERB
Base
0.81
1.50
0.16
0.52
0.41
0.38
8.97
Gold Standard
0.92
2.52
1.00
1.00
1.00
1.00
10.79
Image
0.80
2.09
0.19
0.61
0.45
0.49
10.33
Caption CLIP
0.83
1,87
0.18
0.57
0.46
0.43
9.56
Caption RoBERTa
0.83
1.93
0.20
0.60
0.48
0.47
10.06
Caption RoBERTa TFIDF
0.86
2.01
0.19
0.62
0.49
0.48
10.04
Caption TFIDF
0.83
2.05
0.21
0.63
0.46
0.45
10.30
Hybrid CLIP
0.81
1.87
0.24
0.58
0.46
0.45
9.78
Hybrid RoBERTa
0.83
1.94
0.20
0.61
0.48
0.47
10.09
Hybrid RoBERTa TFIDF
0.84
2.09
0.20
0.65
0.48
0.46
10.29
Hybrid TFIDF
0.85
2.19
0.20
0.62
0.49
0.47
10.18
12


Reviews:
[{"review_id": "soEEke1CU6H", "reviewer": "Reviewer_eJ14", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a study of \"computational language acquisition\", where a speaker agent learns to use natural language to play Lewis-style referential games. These games are played with a fixed, listener \"teacher\" agent which makes decisions based on the referring expressions generated by the speaker, optionally giving \"natural language feedback\" in terms of the ground-truth caption.\n\nThis setting has been explored in existing work; the authors' main contribution is to outfit the speaker with a \"theory of mind\" listener model, which is a learned model of the listener given listener feedback over interactions. Then the speaker model looks like a pragmatic speaker similar to the Rational Speech Acts literature, where it optimizes a weighted objective of (1) truthfulness (i.e. probabilty under the image captioning model) and (2) listener utility under the learned ToM model. Authors show that, for some combination of ToM speaker hyperparameters, ToM speakers are able to indeed generate utterances at higher quality than a speaker without ToM.\n\nSome secondary analyses are also conducted on whether the difficulty of distractors sampled from reference games changes the learned languages; authors find they do, in that more difficult games result in better languages according to various metrics (e.g. utterance complexity, length, closeness to ground-truth caption).", "summary_of_the_review": "To summarize, I think this paper presents a technically sound set of experiments, although many of the findings here are quite similar to findings that already exist in the computational pragmatics literature. Another weakness is the several seemingly ad-hoc hyperparameter design choices that go into the environment on which the authors base their conclusions.\n\nHowever, the idea of learning an internal listener model for a pragmatic speaker, is, to my knowledge, new, and is likely to be of interest to researchers in this space. Based on this, I am marginally in favor of acceptance, though I don't (at the moment) feel particularly strongly and look forward to the other reviewers and the author response.\n", "strength_and_weaknesses": "# Strengths\n\nThis paper presents a number of interesting results in a simulated language learning environment and explores a number of sensible hypotheses about how modeling and dataset choices change the learned languages (though several of these have been explored already - see Weaknesses below). I think the most interesting contribution of this paper (which is highlighted by the title) is the learned ToM-style internal listener model.\n\nAuthors are correct that in a lot of the rational speech acts literature, the speaker is allowed to explicitly optimize against the known parameters/probabilities of the partner listener model. Showing that speakers can build internal model of listeners from a weaker supervision signal (just listener choices in certain contexts), and still use this to improve generation, is an interesting finding and likely to be useful to the broader community.\n\n# Weaknesses\n\nThe weaknesses I see in the paper have to do with missing contextualization with related work (most salient), a confusion about some of the modeling details (equation 10), as well as some lesser concerns about the broader applicability of this work as a whole.\n\n## Missing contextualization with related work\n\nI believe this paper has more similarities to related work than it lets on. One clear lacuna is discussion with Rational Speech Act models of pragmatic language use. In particular, the idea of an internal listener model that is used to rerank utterances is precisely the \"pragmatic speaker\" formulation in RSA which optimizes utility of an internal literal listener. This idea was precisely implemented in [Andreas and Klein, 2016](https://arxiv.org/abs/1604.00562) which authors correctly note. But the similarities are much more than that: Andreas and Klein precisely propose \"sample and rerank RSA\" where candidate utterances are sampled from a speaker model, then rereanked with an internal listener module. This process is *identical* to the ToM speaker, the only difference being that the ToM listener is learned. Andres and Klein also define the same tradeoff between optimizing for the listener probability vs optimizing the speaker probability of utterance. I think these similarities should be highlighted more beyond just \"we draw inspiration from.\" Another relevant paper is [Amortized RSA](https://arxiv.org/abs/2006.00418) which trains a model directly to maximize likelihood via gumbel-softmax (similar to the strictly communicative objective in section 3.1 of this paper, though here PPO is used as the optimizer). Again this is with a fixed listener model; I think the current paper should more clearly highlight the idea of using a learned listener model as the key differentiator over existing work in data-driven RSA.\n\nThe idea of increasing difficulty of reference games by making distractors more similar is also a common idea in the literature, e.g. this experimental condition has been explored in [Monroe et al. 2017](https://aclanthology.org/Q17-1023/), [Achlioptas et al., 2019](https://arxiv.org/abs/1905.02925) etc, and indeed show positive effects on the pragmatics and complexity of language when trained in \"harder\" games.\n\nAnother relevant paper which should be discussed is [Lazaridou et al., 2020](https://arxiv.org/abs/2005.07064) which examines the idea of initializing a pretrained image captioning model, then finetuning it with a communicative reward, similar to the present study, as well as using the listener model (jointly learned with the speaker) to improve generations. Note that they (and many in the literature) observe this phenomenon of semantic drift, i.e. speakers begin to use language in ways not implied by the original static training dataset. Did authors find this phenomenon here?\n\n## What's happening in Equation 10?\n\nEquation 10 (ToM objective) and its description are confusing to me. Equation 9 shows $P_{ToM}(I_t \\mid u^j)$ playing the role of the listener, i.e. a probability distribution over images $I_t$ given utterances $u^j$. But here in Eq 10 we are evaluating $P_{ToM}(w_u^n)$. What does this mean? What is $w_u^n$? (Seems overloaded with the ToM hyperparameter $w_l$). What is the summation index variable? Description says the ToM objective is \"defined as the cross-entropy loss between the distribution of the ToM listener and that of the speaker\". Should \"speaker\" be \"true listener\" here? This is more aligned with \"untrained listener trained to emulate the actual listener's outputs over time.\"\n\nI have a mental model of what is going on here (that we are training the ToM listener to mimic the actual listener, whenever we actually get listener choices), but I'm not sure if this is actually happening as I found this section difficult to parse. It's possible I'm misunderstanding something here, and I can't be too confident in my recomendation until this is cleared up.\n\n## Unclear whether this is a realistic setting for either NLP/CogSci communities\n\nThis paper certainly has some interesting experiments and modeling contributions, but one weakness I see (and I think this is true of many computational simulations of language acquisition, e.g. emergent communication) is the degree to which this is a particularly useful simulation for either (1) giving us better NLP systems, or (2) teaching us about language development. Re: (1), it would be great if authors gave us more concrete settings or potential applications where training a speaker model in this kind of fashion, with natural language feedback, might be useful for building better NLP systems. Re: (2), there are a lot of ad-hoc hyperparameter choices in this simulation environment (e.g. the thresholds at which listeners choose to abstain from providing feedback, or provide the ground-truth caption; the various hyperparameter tradeoffs between speaker objectives) that could drastically change results. If results are fairly sensitive to such ad hoc hyperparameter choices, then we should be careful to make broad sweeping conclusions about what this tells us about human and/or machine language development.\n\nMoreover, it's a little unclear how authors intend, in the conclusion, for this study to support further investigations into the \"similarity of the learning process between human learners and our computational models.\" Babies learn in naturalistic environments, certainly aren't initialized with ResNets, and do far more than just play referring expression games.\n\nI don't view this as a fatal weakness of the paper - I think these are fundamental problems that a lot of papers in this kind of line have, and this debate need not be resolved here.\n\n## Other minor considerations\n\n- It would be great to have an explicit measure of the ToM model success, i.e. how well does the ToM listener approximate the real listener throughout training? I'm interested in a more isolated metric for measuring this beyond just \"the full ToM speaker does better\". If we get very close to the true listener then it makes perfect sense that ToM modeling would improve generation.\n- How much is the \"learning from feedback\" objective required to get learning off the ground in this setting? Assuming the LSTM is not pre-initialized (which I don't think it is), It seems extremely difficult to learn a good NLG policy from scratch given the massive complexity of the natural language search space. What happens when this objective is turned off? How often is natural language feedback given?\n- I find the use of \"learning from feedback\" to be a bit misleading compared to how the term \"feedback\" has been traditionally used in the literature (e.g. [1](https://arxiv.org/abs/2204.14146), [2](https://arxiv.org/abs/2009.14715)) - \"feedback\" implies some language corrections or explanation for why a referring expression is correct/incorrect, but in reality this is just providing the ground-truth label to the agent (which you then optimize for explicitly).\n- Notation is a little convoluted at times, e.g. eq 7 $w_{u^j_1}$, is it really necessary to have these multiple nested subscripts?\n- I think the conclusion statement that \"psychological modelling communities...consider the further incorporation of ToM into language learning simulations\" may be slightly uncouth, given that theory of mind is a central idea in an extremely long line of literature in both computational and non-computational pragmatics dating all the way back to Grice.\n- I would advise authors to be careful about what constitutes \"better\" language, and claims that the speakers trained with harder distractors have strictly higher obejctive measures of \"utterance quality\". If we adopt a fully pragmatic view of language, then the best language should be as concise as possible given the context, and with simpler distractors, it makes sense that the language is simpler - that is the world they are trained on, after all. It'd be interesting to see whether speakers trained on a mixture of context difficulties can modulate the complexity of their utterances when the context demands it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rkFVXYdm0hs", "reviewer": "Reviewer_Z8Jq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is inspired by cognitive science that young children actively acquire language through interactions with their surrounding environment and caretakers. Specifically one critical mechanism of language learning is the ability to infer the mental states of other agents in social environments, referred to as Theory of Mind (ToM) by Premack & Woodruff (1978). Departing from current approaches of language models,  the paper presents language-learning agents equipped with ToM.  It models ToM by giving the speaker agent an internal listener model that is trained alongside the speaker and using this ToM model to rerank potential utterances. The paper also experimented with varying task difficulty assuming that stronger environmental pressures will promote the development of more complex language. The paper shows that speakers trained with a ToM listener component have higher accuracies than those trained without in our image referential game setting. It further observes that increasing task difficulty in the training process results in more fluent, higher-quality utterances in evaluation. ", "summary_of_the_review": "As it current stands, the paper added theory of mind to the standard speaker and listener formulations. The model design is also very straight forward. The evaluation is on a very limited environment. Why the very limiting image referential game environment is the most relevant for computational language acquisition with theory of mind? What are the difficulties to add ToM to big multi-modal language models such as Flamingo?\n\nIn its current form, I do not find the results appealing to the ICLR community.\n\n== post authors' response, and reviewer and area chair discussion\nThe authors' response is very defensive in nature and not convincing. For example \"Additionally, we believe that the image referential environment strikes a balance between ease of investigation and realisticness.\" Why is the image referential environment strikes a balance and why is it realistic? What other environments are you comparing with?\n\nHowever, after discussing with other reviewers and the area chair in a virtual zoom meeting, I am a bit more positive on the limited contribution of this paper. I am willing to upgrade my rating to 5: marginally below the acceptance threshold.", "strength_and_weaknesses": "Strength\n1. Inspired by theory of mind in cognitive science, a critical mechanism of language learning is the ability to infer the mental states of other agents in social environments, the paper presents language-learning agents equipped with ToM.  \n\n2. The paper finds that incorporating ToM into the speaker models leads to improvements in speaker performance and incorporating harder distractors leads to the development of more complex and fluent languages.\n\nWeaknesses\n1. The problem setting is very limited with vocabulary of 200 words and the max utterance length is 20 tokens. Yet the paper makes very strong claims on answering two fundamental questions:\nRQ1. Does the inclusion of ToM in language acquisition models improve the performance and learned language of said models? (internal ToM mechanism)\nRQ2. How do our models’ learned languages adapt to discriminate between more visually and semantically similar images? (external environmental pressure)\n\n2. The model design is presented as it is. There is no discussion on alternative designs and whether the design choice achieves the desired goals. For example, why LSTM is used. Would transformer or pretrained language model be a better choice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6fm48ZRTxp6", "reviewer": "Reviewer_J8M3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper shows that inserting a Theory of Mind module into an image captioning agent playing a referential game leads that agent to learn more effective and fluent language.\n\n(motivation)\nSome work in developmental psychology says that human can quickly learn and adapt language because they model the mental states of their social companions -- because they have a Theory of Mind (ToM). Recent work has studied image captioning agents that learn or adapt language while playing referential games. This work asks whether adding a ToM component into these agents improves the language they learn.\n\n(approach)\nThe speaker sees a target image and captions that image. The listener sees the speaker's caption and must use it to pick the correct image from a list of distractor images. The listener also returns the ground truth caption for its selected image when it is confident, but not highly confident. These agents are initialized by pre-training on image-caption pairs. The speaker is finetuned to\n1) make the listener guess the correct target (PPO objective) and\n2) mimic feedback (GT captions) when given by the listener.\n\nThe speaker is also augmented with a ToM component. When the listener gives feedback on an image (potentially not the target) the ToM component is trained to\n\n3) assign high probability to that image given the speaker's caption for it.\nDuring inference potential speaker outputs are re-ranked to choose the utterance which the ToM speaker assigns highest probability.\n\n(evaluation)\nEvaluation focuses on the quality of the speaker's language, showing that\n1) The listener picks the correct image (Accuracy) more often with the ToM speaker than without. (\"Baseline\" and \"Zero\" ToM Weight)\n2) The speaker's language is more fluent with ToM (GPT-2 assigns it higher probability)\n3) The speaker's language is more similar to ground truth captions (adpositions, nouns, verbs usage; sentence length)\n4) Using hard distractors (image choices that are similar to the target) results in better language fluency, though not better Accuracy.\n\nThe paper concludes that adding the ToM component improves speaker fluency and effectiveness.\n", "summary_of_the_review": "\nThe novelty and potential significance of the work make it a relevant contribution to this community. While the quality and clarity of the work have some flaws, in their current state they are good enough to comprise a meaningful contribution.\n", "strength_and_weaknesses": "\nStrengths\n===\n\n* I really appreciate the motivation for this work. Thinking about language acquisition from an evolutionary perspective seems important and the role of Theory of Mind in evolutionary language acquisition in AI has not been explored. This has also been specifically identified as in need of further study in the cited work (Lazaridou et. al. 2020).\n\n* Futhermore, this particular computational relization of Theory of Mind seems new and the evidence points in the direction\n\n* The idea of the listener giving feedback only in certain circumstances, and using the ground truth caption to do that is interesting and potentially novel.\n\n\nWeaknesses\n===\n\nThese are roughly ordered by importance.\n\n* (quality) One thing that's still not clear to me is whether the improvement is really because the speaker is modeling the behavior of the listener or whether the ToM listener component is just making it better at image captioning in general. While the improvement of the \"Zero\" ToM Weight models over the \"Baseline\" can only be due to a general improvement in image captioning ability, the additional improvement of the \"High\" ToM Weight models could still be purely due to an additional increase in general image captioning ability. This could be measured by looking at the general captioning quality of the various models. If caption quality stayed the same but listener choice accuracy increased then that would indicate the improvement is really due to the speaker's model of the listener. Even if caption quality went up some component could still be due to the speaker's model of the listener, so I don't think the issue is straightforward to resolve. I think the available evidence points toward some contribution from the ToM component, but more work is needed to resolve this with high confidence.\n    * (suggestion) It would certainly help to have a better measurement of the speaker's overall captioning quality. The Part Of Speech F1 and Average Length metrics give some sense of this, suggesting the \"High\" models are better captioners, but it should be measured with a more established metric like BLEU, SPICE, or CLIPScore[2].\n\n* (clarity, quality, novelty) The connection between ToM and distractor difficulty isn't very clear, and distractor difficulty isn't very well motivated for studying for its own sake. This makes it hard to see why the content of section 6.3 was important to investigate.\n\n* (quality) The listener feedback mechanism is interesting, but the paper does not address how it affects performance in the experiments. How often does the ToM listener receive GT feedback? How does that change over the course of training? This matters because we expect NNs not to be well calibrated by default, and over-confidence could result in this mechanism simply being ignored (with P_max always greater than theta_2 in eq. (2)).\n\n* (clarity) The notation is confusing at times. The reranking weight and caption words both use the letter w with different subscripts. The w_{n^u} notation is never explicitly defined. (Is n the summation variable in eq. 10? Isn't P_ToM a conditional distribution?). I can't understand how the ToM listener is trained because I don't understand eq. 10. Is it supposed to assign high probability to all possible choice images given the caption that the speaker assigns them? Or is it suppose to assign the same label as the actual listener given the choice already made by the listener and the caption the speaker assigns to that choice?\n\n* (quality) The paper does not argue (either empically or rhetorically) whether the diverse choices used in re-ranking are diverse enough for re-ranking to be very meaningful. Often LM samples can be rather redundant and may not always include good choices. Is that the case here?\n\n* (novelty) This work does not cite [1] even though it is very similar. Both works take an image captioning speaker agent and add an internal component which makes the speaker produce utterances that are more discriminative between target and distractor images. I think this work is still different because it is motivated by ToM and thus incorporates feedback from the listener to direct the discriminativity, unlike [1].\n\n* (clarity) RQ1 and RQ2 are not very clear as stated. In RQ1 I see \"the performance\" of \"said models\" as referring to their ability to get the other agents to perform well. If it is the listener then perhaps this makes sense, but at this point its not clear whether the listener or the speaker is referred to as \"said models\" This is also the first mention of RQ2, or at least I'm not immediately sure of how it relates to the intro having been read so far or how it relates to RQ1. A clear definition of what the environment is would be helpful here. Later on there is also a disconnect in flow between section 6.2 and section 6.3, and this would help improve that.\n\n* (clarity) In section 2.1 I is initially referred to as a _set_ of images, while I^N seems to be referencing the same set later. For consistency I^N should be used in the first case or some other adjustment should be made.\n\n* (clarity) \"However, the language network parameters are additionally trained [...] to optimize the network values\": I don't understand how the listener is trained. Is the COCO train dataset is used?\n\n* (clarity) What exactly are the \"easy\" distractors? I'm guessing they are randomly selected COCO images, but that should be verified and made clear in the text.\n\n* (clarity) The \"Return\" metric is not defined or referenced in the results section.\n\n\n\n[1]: Vedantam, Ramakrishna et al. “Context-Aware Captions from Context-Agnostic Supervision.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 1070-1079.\n[2]: Hessel, Jack et al. “CLIPScore: A Reference-free Evaluation Metric for Image Captioning.” ArXiv abs/2104.08718 (2021): n. pag.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3wmNQCIKiH8", "reviewer": "Reviewer_NqTJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an image captioning task to represent language acquisition through its structure.  The learner or \"speaker\" is the image captioner.  The feedback is provided via a \"listener\" that provides feedback in the form of a ground truth caption. The innovation comes from the inclusion, in the speaker model of a representation of the listener, the \"theory of mind\" (ToM) listener.\n\nThe authors find that 1) the model learns better with a ToM listener and 2) the outputs are more complex when the feedback is based on difficult distractors rather than easy ones.", "summary_of_the_review": "This paper describes an approach to evaluate claims of language acquisition through image captioning.  The experimental setting is well motivated and reasonably evaluated.  With the exception of the claims about linguistic complexity being influenced by difficulty of distractors, the findings are clearly evidenced by the experiments.", "strength_and_weaknesses": "Strengths\n* Clear connection from the problem statement to the empirical experimentation\n* Clearly written. Relatively easy to understand results and modeling formulation\n\nWeaknesses\n* While the findings related to the inclusion of the ToM component are obvious from Table 1, the effects of distractor difficulty in Table 2 are less obvious. It's not obvious how the Model Distractors are connected to the claims of distractor difficulty.  However, the distinction to the Gold Standard are very clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster