Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
Authors: Yilun Hua
Yoav Artzi
Venue: COLM (2024)
Keywords: Interaction, Communication Efficiency, LLM, In-context learning, Linguistic Convention
TL;DR: We introduce a framework to evaluate and analyze multimodal LLMs' abilities to improve communication efficiency through in-context adaptation.

Abstract: Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showing properties of human language that go beyond relaying intents. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions, and what mechanisms they may adopt for this purpose. 
We introduce ICCA, an automated framework to evaluate such conversational adaptation as an in-context behavior in MLLMs. We evaluate several state-of-the-art MLLMs, and observe that while they may understand the increasingly efficient language of their interlocutor, they do not spontaneously make their own language more efficient over time. This latter ability can only be elicited in some models (e.g., GPT-4) with heavy-handed prompting. This shows that this property of linguistic interaction does not arise from current training regimes, even though it is a common hallmark of human language.

Published as a conference paper at COLM 2024
Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
Yilun Hua and Yoav Artzi
Department of Computer Science and Cornell Tech Comell University {yilunhua, yoav]
Abstract
Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showing properties human language that go beyond relaying intents. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions, and what mechanisms they may adopt for this purpose. We introduce ICCA, an automated framework to evaluate such conversational adaptation MLLMs. We evaluate several state-of-the-art MLLMs, and observe that while they may understand the increasingly efficient language of their interlocutor, they do not spontaneously make their own language more efficient over time. This latter ability can only be elicited in some models (e.g., GPT-4) with heavy-handed prompting. This shows that this property of linguistic interaction does not arise from current training regimes, even though it is common hallmark of human language.
in-context behavior in
1 Introduction
Human interlocutors adapt to each other during interactions, developing increasingly efficient ways to refer to concepts and objects. Hawkins et al. (2020b) exemplify this via communication between a nurse and bed-ridden patient home. Initially, the patient may refer to a medicine with the medicine for my back pain in a small blue medicine bottle ..., but after week of care, they are likely to just ask for their back meds. This increase in efficiency relies on the interlocutors forming ad-hoc linguistic conventions: the mutually understood, concise phrases communicate referential content. This phenomenon has been repeatedly observed and characterized in controlled studies using repeated reference games (Figure
e.g., Krauss & Weinheimer, 1964 Brennan & Clark 1996
Hawkins al. 2020a)
We study this ability in multimodal large language models (MLLMs). LLMs and MLLMs are well positioned acquire this behavior and display it spontaneously in interactions. They are trained on large amounts of human language data, in which this behavior is common
and the history of an ongoing interaction is often retained, thereby explicitly keeping the information needed at hand. Beyond the scientific question, such ad-hoc adaptation has significant application impacts: enabling more natural interactions, reducing the costs involved in conversations (e.g., using shorter utterances to communicate the same amount of information), and increasing the accuracy of relaying intent.
We propose ICCA an automated framework to evaluate and characterize the ability of models to form ad-hoc conventions. ICCA uses a corpus of human-human reference game interactions, allowing for completely automated evaluation, which does not require further human interaction, making it easy to deploy for the analysis of new models. The interaction follows the standard repeated reference game setup (Clark & Wilkes a speaker refers to an image within shared context of images, and listener resolves the
1986), where
ICCA stands for In-context Conversational Adaptation.

---

Published as a conference paper at COLM 2024
Context
Repetition
Repetition 6
Photo with bowd of banarias with pokadot
black
Image
Image A
A
B
A bous full mixed fruit, black back backgronnnd
pokandot
C
D
Image
Image B
Figure Illustration of a reference game. The speaker (blue) and listener (orange) observe shared set of images 2 The interaction progresses in six repetitions, each includes a trial for every context image. In each trial, the speaker describes target image, and the listener has to select the correct target given the description only. For simplicity, this figure omits the feedback on listener actions. This interaction illustrates some of the effects of convention formation: the descriptions become shorter as the interaction progresses, and lexical choices converge subset of the words used in earlier repetitions.
reference to select one of the images, ideally the one originally referred to. Figure Illlsstaaees the scenario. We focus on in-context adaptation as the interaction progresses, the entire history is retained in-context. Core to our approach is comparing the changes in model behavior, either as speaker or listener, throughout an interaction to the changes observed in humans. We measure different properties that have been shown to be influenced by convention formation: utterance length, lexical convergence, and selection accuracy.
We apply our approach to five representative MLLMs: IDEFICS (Huggingface 2023), LLaVa- 1.5 (Liu et al. 2023a), , GPT4-vision (OpenAI et al. 20224), Gemini 1.0 Vision Gooole 2023) and Claude 3 opus (Anthropic) 2024). We find that all models struggle to spontaneously introduce conventions and adapt as speakers. Prompt engineering an explicit in-context instruction specific to the reference game scenario can address this to some degree. The strongest models (GPT4, Gemini, and Claude) can then gradually use shorter messages (gaining lexical efficiency) but still struggle with convergence or stability, which hinders the emergence of truly efficient communication. When acting as a listener, GPT4 displays adaptation trends close to humans, improving its accuracy as the interaction progresses, while other models show this behavior setups. Overall, we show that while today's MLLMs may passively understand the evoly- ing language of their interlocutor, the ability to adapt their own language for efficient communication does not naturally emerge from their training instruction-tuning This outlines important future research problems. We release ICCA under the MIT license at https://github... com com////
lesser degree or only under some simplified
Background and Related Work
Repeated Reference Games A reference game is an interaction where speaker and listener (i.e., a dyad) interact over a shared context. The shared context is a set of images.
The speaker describes target image. The target designation is only revealed to the speaker. The listener has to select an image following the speaker's 's description. Each participant sees the images in different order, they cannot use position information to communicate the referent. Reference games have been used extensively in the study of computational models, including recently evaluate visual abstraction (li et al., 2022 and conversational aptitude (Chalamalasetti et al. 2023).
A repeated reference game (Figure 1) includes multiple repetitions. Each repetition has one trial for each image in the shared context. The listener receives feedback after every
2nn Figure we show the shared context only once for compactness. In our experiments, we shuffle and show the context for each trial, but also experiment with showing it only once.
2

---

Published as a conference paper at COLM 2024
trial, which indicates the correct selection. The repetition and feedback allow the dyad to form message agreements over the repeating stimuli (i.e,, the speaker would naturally use gradually shorter but related messages acToss repetitions, and the listener learns what they refer to). ICCA's 's repeated reference games are developed based on the setup and data from Hawkins et al. (2020b). The shared context includes four images, and there are six repetitions, giving total of 24 trials. The order of images is shuffled across the trials. We refer to this as the standard setup. Hawwins et al. (2020b) only uses this standard setup. Beyond this standard setup, we design variants to further disentangle the types and causes of model failures for in-context LLM adaptation (Section 4 and Section 5).
Ad-hoc Adaptation in Interactions Existing literature shows that humans are inclined to reduce the effort needed to convey their intended information and for their audience to comprehend it, leading to efficient communication (e.g., Zipti (ipi 1949; Gibson et al. 2019 Yin et al. 2024). When human individuals interact through dialogue, this is manifested by developing and using ad-hoc linguistic conventions. This phenomenon has been observed
with repeated reference games (Krauss & Weinheimer 1964 1966 Clark& & Wilkes-Gibbe
1986: Hawkins et al. 2020a), and related interaction scenarios (Haber et al. 2019). Studies have also shown various properties of these conventions, such as arbitrariness, stability,
stickiness, and convergence (Lewis, 1969: BBrenaan Clark 1996 Markman & Makin 1998
Hawkins et al. 22200a Eliav et al. 2023). This adaptation was modeled with the pragmatic rational speech act model (RSA; Goodman & Frank 2016), leading development of
models that replicate this behavior in reference games (egg, Monroe et al. [211 McDowell & Goodman 2019 White et al. (2020) and use it improve model performance on other tasks (e.g., Andreas & Klein 2016 Fried et al.) 2018). Adaptation was studied beyond reference games, showing how the complexity of the scenario influences how conventions manifest in the language (Effenberger et al., 2021).
Ad-hoc conventions are a particular instantiation of the broader phenomenon of common ground, which is defined as the mutually recognized shared information between the par- ticipants of a conversation (Clark & Brennan 1991 Lewis 1969: Stalnaker 2002). Common
ground has been studied extensively, with focus on both human cognition (Clark) 1996
Horton & Gerrig 2016) and machine reasoning (Cohen & Levesque Grosz & Sidner] Traum
1994 Del Tredici et al. 2022 Shaikh et al. 2024 Andukuri et al. 2024
Testoni & Fernández
2024
Model Adaptation Adapting models during an interaction improve communication efficiency or success is relatively understudied. Hawkins et al. (2020b) proposes continual learning method for CNN-RNÑ models to gain communication efficiency in repeated refer- ence games through continual weight updates. /Zhu et al. (2021) proposes explicitly training models for ad-hoc adaptation through meta-learning. We focus on the in-context capabilities of I.LMs and MILMs,, which offer an update-free route for adaptation that is particularly compelling given the costs of updating large models. Our use of in-context learning differs from how this mechanism is usually used either by providing instructions (Ouyang et al., 2022) or few-shot examples (Brown et al. 2020). While reference games can be seen as related to the few-shot approach, ad-hoc adaptation is not about replicating patterns, but showing change and adaptation over time.
3 The ICCA Framework
ICCA uses a dataset of human-human interactions, and allows to easily customize different parts of the interaction. This flexibility enables different research questions. For example, in /Section 5) we customize the interaction structure to analyze how well models handle long interactions with multiple images interleaved in them. ICCA supports studies with the model acting either as speaker or listener, and includes several metrics to track different properties of adaptation during the interaction.
ICCA is fully automated and easily applicable to new MLLMs. Our design does not require collecting new data or human studies, but instead uses Hawkins et al. (202bb)ss 's human-human interaction data to simulate human interacting with an MLLMM Each
3

---

Published as a conference paper at COLM 2024
interaction in the dataset was collected under the standard setup (Section 2) and uses visually challenging reference context, consisting of four similar images (Fgure 1). The dataset contains 54 human-human interactions, which we incorporate into ICCA.
A repeated reference game interaction R is a sequence of tuples ((C; Ci, Sp, 4, fi)) where C; is the set of images forming the reference context, Cj is the index of the target image, s; is the speaker utterance, I; is the listener selection, and fi is the feedback based on the listener's s selection. At every trial with the evaluated model as either the listener or speaker, ICCA constructs the MLLM prompt from an instruction text I, the history (i.e., all prior trials R|: f)), and the stimuli for the current trial with a user-defined pre-processing function F. Upon receiving the model's response, ICCA computes the feedback fi to help the next trial.
The function prepares the prompt depending on the experiment, and key to the flexibility of ICCA. For example, when evaluating model as listener under the standard setup, the function input would be F(I, Ri: f), St),, and it would format and concatenate all the elements in order. Figure in the appendix shows an example prompt. ICCA allows to easily modify this standard setup, for example by processing the data such that F drops all reference contexts except C, thereby creating simpler input where the images appear only once at the beginning of the interaction.
ICCA simulates the interlocutor of the model evaluated either with deterministic counter- part or by having another model take the role. A deterministic speaker outputs the messages from recorded human interactions dataset, showing predetermined, realistic trajectories of message shortening over time, but it does not adapt its language based on the listener's 's selections. It can be considered as "convention comprehension" task, potentially more challenging due to the non-adapting speaker messages. We use this simulated speaker for our model-as-listenee experiments (Section 5) because it exposes the model listener to behaviors and linguistic conventions naturally occurring in human interactions. Inversely, for our speaker experiments (Section 4), we use a high-perforrance model listener (GPT4), which we observe to have performance similar to human listeners (Section 5)
We evaluate model behavior with adaptations of the metrics used in human studies with reference games (Hawkins et al. [2020a b). For listener experiments, we follow Hawkins et al. (2020b) and report the average accuracy in each repetition. Speaker experiments are evaluated using several metrics. We report the average message length and the listener's ss accuracy in each repetition. Additionally, we evaluate the similarity between corresponding messages from consecutive repetitions. While this was done using GloVe embeddings (Pen- nington et al. 2014) in past work (Hawkins et al., 2020a), we design a new metric called Word Novelty Rate (WNR), which is sensitive to exact word choices. WNR is modified word error rate that only counts insertions and substitutions, and ignores deletions. It is motivated by how people naturally drop words from their messages the interaction progresses (Hawkins et al.) 2020a), whereas additions and substitutions of words often reflect important changes in information based on our observations. Compared to GloVe, WNR is more sensitive to lexical inconsistencies that can increase the listener's 's cognitive load. [Appendix Al presents more details of our metrics, including a comparison of WNR and simbedding-based similarity metrics and a variant of WNR that is not normalized by message length (referred to as Word Novelty Distance).
4 Model-as-speaker Experiments
We study model behavior as speaker with five state-of-the-art vision MLLMs: IDEFICS-
80b-instruct 3 LLaVa-1.5-13b, GPT4-vision, Gemini 1.0 Pro Vision, and Claude 3 opus. Throughout all speaker experiments, we customize the data to only show the referential context once at the beginning of the interaction, so there is no shuffling of context through- out the interaction. We engineer prompts for each model individually to best evaluate its capability. We use GPT4 as the listener. It exhibits high performance in our listener experiments (Section 5), especially when the context appears only once at the beginning,
*IDEFICS an open-source reproduction of Flamingo (layrrac et al. 2022
4

---

Published as a conference paper at COLM 2024
IDEFICS
LLaVa
CPT4
Gemini
Claude
Human (Standard Setup)
Explicit Instruc-
S1: Standard Speaker
S2: Gricran Instruction
S3: Explicit Instruction
Honn-Consistency Request
100
80
Acc %
60
40
20
20
Msg (tokens)
15
10
5
1.0
0.8
WNR
0.6
0.4
0.2
2
5
2
5
2 3
5
3
5
Repetition
Figure 2: Speaker experiments. Margins of errors are bootstrapped 95% Cls.
so it is a suitable substitute for a human listener in this study. Appendix A.3 provides implementation details.
We design four speaker variants by modifying the instruction I. The variants were developed throughout our experiments, by observing the difficulty of models to present patterns similar to human linguistic behavior. The variants instruct the model to display the convention formation behavior observed in human speakers increasingly explicit and specific ways:
S1: Standard Speaker The standard speaker setup (Seciion 2).
The model speaker only receives the basic game instruction, with mention of communication efficiency. Figure 6 in the appendix shows an example prompt.
S2: Gricean Instruction A relatively light-handed and general way to introduce the expected convention formation behavior is to explicitly instruct the model to follow the Gricean quantity maxim. This kind of instruction is not specific to reference games, and does not explicitly mention message length. Its focus is information, and it entails that cooperative interlocutors would provide enough information to identify the referent but would not make the message more informative than necessary. We add additional instructions based on the maxim and further instruct the model think about how the amount of information needed may change more trials are completed and based the listener's performance in previous trials. 41
S3: Explicit Instruction We instruct the model to explicitly reduce message length as the interaction progresses. Unlike S1 and S2, this instruction is specific to reference games, as language adaptation in other scenarios not necessarily accompanied by length reduction (Effenberger et al. 2021) We add to S1 an explicit instruction to
4hhe models did not show substantial improvement without this additional instruction. Not the exact prompt used for experiments; shortened and revised for illustrative purposes.

---

Published as a conference paper at COLM 2024
reduce utterance length: as more trials are completed and as the listener understands you better, gradually condense your messages, making them shorter and shorter every trial
S4: Explicit Instruction + Consistency Request Convention formation in reference games is not only characterized by reduction in utterance length, but also by lexical consistency. This variant explicitly instructs the model to follow this pattern. Similar to S3, it is specific to the repeated reference game setup and its use of a repeating context. We add to S3 the instruction: when creating a shorter message for an image, try to extract salient tokens from the previous messages for this image rather than introducing new words. The short messages should still allow the listener to choose the target correctly. For each image, when you reach a message that cannot be further shortened, you should keep using that message for the rest of the gamel
Figure 2] showw the results for all variants, along with properties of the human messages from Hawkins et al. (2020b), which were collected using the standard setup. We report mean message length, WNR, and listener accuracy for each repetition. Overall, all models fail to spontaneously improve communication efficiency. It is only with fairly heavy-handed instruction that GPT4, Gemini, and Claude show adaptation trends similar to humans.
Variant S1 shows that without any explicit instruction, the models show trends that are far from human behavior. GPT4 Gemini, and Claude generate longer utterances in later repetitions. IDEFICS and LLaVa maintain consistent message lengths. But, upon close inspection, we observe they simply tend to repeat previously used messages for the same image. This explains their very low and almost constant WNR. We analyze this further in Section 6 Also, the messages of IDEFICS and LLaVa are less effective in distinguishing the target images,
shown by the lower listener accuracies. This demonstrates the inability of these models to correct their behavior based on feedback. No models show WNR trends similar to humans. GPT4, Gemini, and Claude constantly introduce new words as the game progresses, as shown by higher WNR curves, even though we do not use token sampling for decoding. Gemini shows downward trend, but achieves this by the undesirable practice of making its messages longer every trial while making a relatively constant number of word insertions or changes, so a bigger portion of the message is maintained! 6
Gricean instruction (S2) leads GPT4 and Claude to reduce message length over time, though GPT4's reduction is far from humans'. Both models have WNR curves sigmificantly higher than humans. As their messages shorten, they still frequently introduce new words and do not stabilize the messages, a behavior adverse to communication efficiency. We further discuss this issue with S3, where more models display this issue.
S3's explicit instruction has no impact on IDEFICS and LLaVa. Both continue repeating messages, failing to follow the instructions. Gemini and GPT4 show decreasing message length trends similar to humans but still have longer messages than humans throughout. Claude eventually produces messages as short as humans but starts with much longer messages than other models. All models showing message shortening frequently introduce new words as they shorten the messages. Even when the message is too short to be condensed, the models may adopt new words next time without changing the message length. Figure 8 in the appendix exemplifies these behaviors. Such behaviors deviate from the stability property of conventions and the observation that human messages show high consistency and convergence. The gap between the WNR curves of these models and humans illustrates this issue. While these models show increasing lexical efficiency, the use of new words reduces communication efficiency, burdening the listener to reason about the words that did not appear the last time the image was referenced. We further discuss this issue in (Section 6
54 addresses the consistency issue but requires further explicit instruction. The final prompt elicits from 4PT4, Gemini, and Claude both length reduction and message convergence, as observed with humans. However, the S4 prompt is very specific to ICCA's setup and does not generalize beyond reference games. Heavy-handed prompt interventions like this are
6This is due WNR's -ength-normmaiizatinnn which critical to reflect similarity Appendix A.2
6

---

Published as a conference paper at COLM 2024
also known to cause unintended model behaviors is not likely to be the solution.
(Shaikh et al.
2024). Prompt engineering
Model-as-listener Experiments
Listener experiments follow a setup similar to the speaker experiments as far as models and prompt optimization (Section 4). Gemini, LLaVa, and Claude cap the number of input images, limiting their use in some of our listener variants. Overall, we design four main variants, each implemented through the pre-processing function F. Our design process is iterative, with some variants designed based on the behavior observed with earlier ones. Throughout the listener variants, we keep the instruction largely constant and about the role of the listener. We vary how we display the referential context.
The listener action space is more limited, simply requiring the model to select the referenced image. We focus evaluating model accuracy, similar to how listener behavior is charac- terized in human studies. Figure 3 visualizes the behaviors we observe. We also include human listener accuracy trends as reference to model accuracies.
The starting point for the listener study is the standard reference game setup (Section 2):
L1: Standard Listener Images are shuffled and re-displayed for each trial, so each image will potentially have new label relative to previous trials.
L1 requires a growing number of images in the prompt as the interaction progresses. With six repetitions of four trials and context of four images, the maximum number of images in the prompt at the end of the interaction is 96. Gemini, LLaVa, and Claude can take at most 16, 4, and 20 images, so we only use GPTT and IDEFICS with L1.
We expect an effective model to exploit the conversation history to reason about the human speaker's 's conventionalized ways of speaking. Even the model starts with low accuracy, has the opportunity to improve because the prompt at later stages includes feedback for its choices, and as the messages conventionalize, later messages for an image are often exact repetitions, albeit with the referential context shuffled.
Both GPT4 and IDEFICS do significantly worse than humans (Figure left). As expected, humans demonstrate strong performance to start with, and show upward trend in ac- curacy, as the interlocutors adapt to each other. GPT4 is significantly worse than humans, though performing fairly well. It shows a marginal improvement trend (88.9% +92.5% in repetition 5), but is not significant, and weakens in the last repetition (91.2%). IDEFICS is much worse immediately in the beginning (46.8)), and rather than improving, its per- formance deteriorates as the interaction progresses, reaching random chance in the later trials. This happens even though it is receiving increasing amount of information that should allow it to improve its performance. A possible cause for this trend is the dramatic increase in the prompt size, especially as more and more images are added, as the interaction progresses. We further discuss this issue and its potential causes in Section 6
5.1 History and Context Impact
Following the observations with L1, we design three variants with simplified referential contexts and history to better understand how well the models handle the interaction history and the referential context:
L2: No History Each of the 24 trials is given to the model in isolation, without any history. The model input includes the context of four images and the speaker utterance, as well the basic game instruction. This variant reveals the extent to which the model can reason about the ad-hoc conventions formed in repeated human-human interaction without access the history in which they were formed.
L3: Images Once A potential challenge of L1 is the large number of images in the prompt. A model's architecture or training may not be suitable for handling large number of images, and I.I. prompt length is known to adversely in influence
7

---

Published as a conference paper at COLM 2024
IDEFICS
Va
GPT4
Gemini
Claude
Human (Standard Setup)
Standard Listener
L2: No History
Images Once
LA: Shuffle
75
Acc %
50
25
2
5
1
3
4
6
2
4
5
1
3
4
5
6
Repetition
Figure 3: Listener experiments. Margins of Error are 95% bootstrapped Cls.
performance (Liu et al. 2023b). L.3 uses shorter history, by only providing the referential context (four images) to the model once in the first trial. Image labels are persistent across all trials, which also avoids the impact shuffling, Unlike L2, this variant includes the complete message, selection, and feedback history.
L4: No Shuffle The four images appear every trial similar to L1 but are not shuffled across trials. L4 shows the effects of image shuffle if compared with L1. also shows the effect of image quantity if compared with L3, which does not involve shuffling either.
1.2 and 1.3 need only four images in the prompt, allowing us to test all the models.
The no-history variant (L.2) reveals different performance trends for IDEFICS and GPT4 compared to the standard setup (L1). IDEFIĆS is still not doing great (45.8% on the first repetition), but performance largely remains consistent across repetitions, indicating that possibly the complexity of the prompt is at the root of its downward trend in the L1 scenario. GPT4 starts with similar performance (87.5%) to its results in L1, but then shows slight downward trend (83.8% at the end), in contrast to the initial upward trend in L1. This indicates that the gradually conventionalized, shorter messages a human speaker uses tend to be more difficult for the model to resolve and that the conversation history can be an effective remedy. Gemini 300.2+78.8% LL (72.7+71.33), and Claude (57.4~+5..5%) show similar trends to GP'T from Repetition 1 to 6, with overall lower accuracies.
The benefit of history becomes more conspicuous when the referential context only given once (L3). This variant dramatically simplifies the prompt, but retains the information needed for convention formation (ie., prior message, selection, and feedback). All the models show upward trajectory as the interactions progress. GPT4 and Claude are the strongest, eventually reaching 100% accuracy, matching human listeners' performance (99.54%). This suggests that all models can associate the current message with the relevant prior messages, thereby increasing their prediction accuracy as the interaction progresses. Admittedly, because an image always has the same label throughout the game under this setup, the models may do well by simply drawing associations between an image's 's label and the messages that have referred to that image (i.e., mubel-message associations). Under this mechanism, the model can improve its accuracy without reasoning about the actual visual input. We explore this possible mechanism with more game variants Appendix D Nonetheless, this experiment shows that MLLMs possess some capabilities for increasingly efficient communication with humans when acting as the listener.
The no-shuffle (L4) experiment also provides key insights. GPT4's accuracy is similar to that in L.3 and higher than that in L1. This demonstrates sensitivity to image shuffling and the constantly changing image labels. GPT4 in L3 and L4 may be relying to some degree text similarity between repetitions by exploiting the label-mese message associations, rather than grounding to the visual input. We study this further in Appendix D IDEFICS' performance
8

---

Published as a conference paper at COLM 2024
on the other hand is different from both L3 and L1, showing almost unnoticeable trend up. This shows that it is not only the shuffling but also the increase in the number of images that IDEFICS cannot seem to handle well. We further discuss this issue in Section 6
Discussion
Our studies point to various issues that likely hinder specific models from displaying communication efficiency gains, and point out directions for future works.
Tendency to Repeat Messages In the speaker study, IDEFICS and LLa Va tend to repeat the first message they use for each image, showing no adaptation. To further study how much these models prefer patterns of repetition, we design a test that uses these models for language modeling rather than text generation. We construct two transcripts for each of the 54 human-human interactions in our original dataset. One is the original transcript from human- interactions, showing the natural evolution of messages. The other is a manipulated transcript where the speaker repeats the messages from Repetition in all the later repetitions. We calculate the log probability and perplexity IDEFICS and LLa Va assign to these transcripts, count the number of times one type of transcript has better log- probability/ / /perplexity than the other, and apply sign test. We find that the manipulated transcript showing message repetitions consistently receives higher log probability and lower perplexity for all 54 interactions, showing the models' significant tendency towards repeated patterns (sign test p-values are near zero). Unfortunately, this experiment cannot be done with GPT4, Gemini, or Claude due to API limitations.
Lexical Efficiency A Communication Efficiency The convergence of human speaker mes- sages for a particular image to a short, stable convention often takes the form of extracting salient tokens from the previous message and sticking to the same message once it becomes very short (Hawkins et al. 2020a). Unless directly instructed to do so through a highly engineered prompt (S4), 4 Gemini, and Claude often introduce new words when shortening their messages or even when the messages cannot be further shortened, as shown in the S3 explicit instruction variant. Such inconsistency with human behaviors is problematic. When messages for the same image do not converge, no conventions can form and the listener will likely need additional cognitive effort to process the previously unseen words. Intuitively, even if a new message is semantically similar to previous one by using synonyms, resolving still likely entail greater cognitive load than an exact repetition. Moreover, when new words describing new aspect of an image are introduced after few rounds of relatively similar messages, they violate the human listener' expectation, potentially leading to miscommunication and slower response (Metzing & Brennan, 2003).
Performance Degradation with Many-image Inputs Among the models that support large number of images, IDEFICS performs much worse as the number of images increases. Even though the images in L4 are not shuffled across trials, which could have allowed the model to exploit label-message associations as an efficient way to gain high accuracy, the model still had much lower accuracy than when the images only appear once (L3) (Figure 3). When the history contains a growing number of images that are shuffled between trials (LT), IDEFICS shows an even worse accuracy trend.
A likely hypothesis is that a greater number of images creates challenges for capturing the dependency between specific visual input and textual cues, which can manifest as failures to associate an image's 's label with the actual content of the image. In a qualitative experiment, we supply sequence of images and their labels as input to IDEFICS, and instruct to describe Image [X]. We observe that IDEFICS can describe the correct image easily when we give up to four labeled images, but often makes mistakes as the number increases (Figure in the appendix). Therefore, even though IDEFICS is designed and trained to support multi-image inference its multi-image capabilities do not generalize beyond few images.
The Flamingo architecture behind IDFFICS supports arbitrary number of images as input and it is trained interleaved texts and multiple images (Huggingface) 2023

---

Published as a conference paper at COLM 2024
Another potential cause is the Flamingo architecture behind IDEFICS. Each token's cross- modal attention only applies to the visual features of the last image that precedes it, rather than all the images. Information about other images can only be indirectly accessed through self-attention on the hidden states of their respective <image> tokens in the text sequence. This architecture may degrade the ability of the model to reason about images, depending on their locations in the input. When the target is not the last image, Image D, the message tokens will not have direct cross-modal attention to the target's visual features, which may hurt IDEFICS' prediction. In L3, where IDEFICS did perform well, this limitation might have been mitigated by the strong textual cues and label-message associations, whereas having more images may distract IDEFICS from these cues and thus manifests this limitation.
Conclusion
ICCA provides perspective into the performance of today's MLLMs that is missing in existing benchmarking, and can be easily applied to new MLLMs without collecting new human data. We observe that of-te-of- models lack the in-context abilities to adapt their own language for efficient communication, even though they may sometimes perform better while passively receiving increasingly efficient language from their interlocutor. This issue is fundamental because, unlike humans, the models do not perceive the effort or cost needed for communication, thus having no inherent reason to reduce them. It is still surprising though, given that LLMs/ MLLMs have successfully displayed many other LLMs/MLLMs
human behaviors and impressive abilities in various applications, by learning from the large amounts of human data, where adaptation for efficiency is common. Overall, the current paradigm for creating LLMs fails to address the need for conversational adaptation and future research is needed on improving their abilities to spontaneously improve language efficiency, maintain language consistency for the same referent, avoid excessive tendency for repetitions, and handle more images in single query.
Acknowledgments
This research was supported by ARO W911NF21-1-0106, NSF under grant No. 1750499, gift from Open Philanthropy, and a gift from Apple. We thank Robert Hawkins and Marten van Schijndel for insightful discussions. We thank the anonymous reviewers and the area chair for their valuable feedback.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Ne- matzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Proceedings of Advances in Neural Information Processing Systems, 2022, URL httpps:/pproceeddingg neurips www.nips.cc.ppprr..ee..... 990a122cchh777777c00441188887889
httppperriillespppp///////////// pdf
Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. http:///www....llbe////////notto
www.wl.wwwbbc..cc
Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions. arXiv, 2024. URL
Anthropic. Introducing the next generation of Claude, 2024. URL comm/news///laude--ffaaan
10

---

Published as a conference paper at COLM 2024
Susan E. Brennan and Herbert H. Clark. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 1996.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini
Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems, vol- ume 33, 2020. URL httpp::/pproceedinnng http:/ppooccednnnn..eiirrrssssss https/sccppppp/////eee/ Sitaaeerfillesppperr/222////
pdf
Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. Clembench: Using game play to evaluate chat-optimized languag models conversational agents. arXiv, 2023. URL.https///axxx..o/r/////////// 13455
Herbert H. Clark. Common ground. Cambridge University Press, 1996.
Herbert H. Clark and Susan E. Brennan. Grounding communication, In Perspectives on Socially Shared Cognition. American Psychological Association, 1991.
Herbert H. Clark and Deanna Wilkes-Gibbs. Referring as a collaborative process.
Cognition, 22, 1986. URL
0010027786900107
Philip Cohen and Hector J. Levesque. Rational Interaction as the Basis for Communiccttinn. The MIT Press. URL httpp:://do..rrg//....
0029339.000... 0030000000100000 0014
Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. From rewriting to remembering: Common ground for conversational qa models. 2022. URL
Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi Analysis of language change in collaborative instruction following. In Findings of the Association for Computa- tional Linguistics: EMNLP, 2021. URL httts:://aalaantoology .rg.lloo..ooorrooooooeoo 2021. http:/2221.....nnfg.eeee. findings-emnlpp
239 3pf
Ron Eliav, Anya Ji, Yoav Artzi, and Robert Hawkins. Semantic uncertainty guides the extension of conventions to new referents. arviv, 2023. UBLL/tttp///axxv.orr/abb// 06539
Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018. URL
https:: //wp:://wwww.. ac awwweb......c.................lb org/ orgg/aatooooooooo Bttp//ntoooogy/N--11111
Edward Gibson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. How efficiency shapes human language. Thends in Cog- nitive Sciences, 23, 2019. URL httpp://www.sieeeedireett.. httpr//eeeetttttoosssseeeeerrrrr
$1354661319300588
Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic probaaabiiiiiopoooibibbbiiiiiiie inference. Trends in Cognitive Sciences, 20, 2016. URL httpps///ppi /ttp::://ai..smmatiiccccci sww.tiiscccccnicncccccc CorpusID:3632786
Google. Introducing Gemini: Our largest and most capable Al model, 2023. URL https
//blog. /gg.ggg././eccoooooo////oggggggo
Barbara J. Grosz and Candace Sidner. Plans for discourse. In Intentions in Communication. The MIT Press. URL httph:ttpt:/oo org.0/...rr//1.00 http:///.7755..mmtteeeeesee 003. 0022
11

---

Published as a conference paper at COLM 2024
Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Ranued Fernández. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019. htw.htttsss//allantollog.. www.aannhooooooogggoooo.o.. org/ P1P-191111 11844.....pp. pdf
Robert Hawkins, Michael C.
Frank, and Noah D. Goodman. Characterizing the dynamics of learning in repeated reference games. Cognitive Science, 44, 2020a. URL https:// onlinelibrary www.wwwwwwwww.leeeeooooooooo
com/ http://oo/a//////e 100 1111/cogs. 12845
Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. Continual adaptation for efficient machine communication. In Proceedings of the Conference on Compulational Natural Language Learning, November 2020b.
William S. Horton and Richard J. Gerrig. Revisiting the memory-based processing approach to common ground. Topics in Cognitive Science, 8, 2016. URL httpsss//onnnnnllbrraa wi wiley. httpy.coom/oo//uu/1111111///////
12216
Huggingface. FlagghngfacceM//dddffff------s-s wuwgginf--cce///ddfffcc--------
2023. URL https://hugginngacceccm
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert Hawkins, and Yoav Artzi. Abstract visual reasoning with tangram shapes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, December 2022. URL
https://ac..anthooog...orr. orgg////.oooy..//2222222/ www.eemalpppm.i.....m
Robert M. Krauss and Sidney Weinheimer. Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1, 1964. URL htttp::://iikksppiingeee
http:///o/oottceeeoo/////////331 3758//F333////1
Robert M. Krauss and Sidney Weinheimer. Concurrent feedback, confirmation, and the encoding of referents in verbal communication. Journal of Personality and Social Psychology, 4, 1966.
David Kellogg Lewis. Convention: A Philosophical Study, Wiley-Blackwell, 1969.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. Xiv, 2023a. URL htttttss/aarriv.or////////33
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2023b. 2024. 1.
org/
Arthur B. Markman and Valerie S. Makin. Referential communication and category acquisi- tion. Journal of Experimental Psychology: General, 127, 1998.
Bill McDowell and Noah D. Goodman. Learning from omission. In Proceedings of the Annual Meeting of the Association for Compulational Linguistics, 2019. URL https:://clannthologg
org/P19-1059, pdf
Charles Metzing and Susan E. Brennan. When conceptual pacts are broken: Partner-specific effects on the comprehension of referring expressions. Journal of Memory and Language, 49, 2003. URL wtt.:httpp///.
Will Monroe, Robert Hawkins, Noah D. Goodman, and Christopher Potts. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5, 2017. URL htttp:://aclatthol org/ Q17- Q17- pdf
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- -hhapeet-Shhapro,
12

---

Published as a conference paper at COLM 2024
Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanítscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schul- man, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Iseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Fe- lipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPTT-4 Technical Report. arviv, 2024. URL http:///rxiiv.or/aab///3g3
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishknn, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In Proceedings of the Conference on Neural Information Processing Systems, 2022. URL https: https::/pproceedinggs... netpeeeedddnnnneneeiiereeee ccppaperrrppppceeeppeeeeeeee. files/ ille//pperr/ pdf
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove:
Global vectors
for word representation. In Proceedings of the Conference on Empirical Methods in Natural I Language Processing, 2014. URL http::
13

---

Published as a conference paper at COLM 2024
Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding gaps in language model generations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2024. URL https: http://aaaaaatoogy...oooo www.:ogg///220 .oooll-oonn.3....
Robert Stalnaker. Common ground. Linguistics and Philosophy, 25, 2002. URL https
com/article//0..... 10tp://10.11/0:1:100000:::::00 102887999022887166999
Alberto Testoni and Raquel Fernández. Asking the right question the right time: Human
and model uncertainty guidance to ask clarification questions. In Proceedings of the Conference of the European Chapler of the Association for Computational Linguistics, 2024. URL
httts::/aallattooooo...r////
eacl-long. 16. pdf)
David Traum. A Computational Theory of Grounding in Natural Language Conversation. PhD thesis, University of Rochester, 1994.
Julia White, Mu, and Noah D. Goodman. Learning to refer informatively by amortizing pragmatic reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2020. URL
Kayo Yin, Terry Regier, and Dan Klein. American I haw.guaagggeennsssspsseeeeeeeeee pressures for communicative efficiency. arviv, 2024. URL htKiittss///ariii.r/////////// 0402/
04024
Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In Proceedings of the International Conference Machine Learning, 2021.
U mlr wwwps//vw///eeeeeeeeeee hhtp:///22222222h211121 pdf
George Kingsley Zipf. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Sadissn-Wesley Press, 1949.
14

---

Published as a conference paper at COLM 2024
A Implementation Details
A.1 Message Length Measurement
We measure the length of the generated messages by counting the number of tokens. Because different MLLMs have different tokenizers, we choose to only use IDEFICS's tokenizer for message length calculations for all models.
A.2 Word Novelty Rate
Word Novelty Rate a modified Word Error Rate, which only counts insertions and substi- tutions, and ignores deletions. The number of insertions and substitutions is normalized by the length of the reference message, as done in the standard Word Error Rate calculation. For two messages from Repetition N-1 and N, we use the message from Repetition N-1 as the reference and the one from Repetition et al. (2020a)'s 's metric design and drop most function words to only consider open-class content words (nouns, adjectives, verbs, and adverbs) as well as pronouns, numbers, and adpositions.
as the hypothesis. We follow Hawkins
WNR addresses the limitation of metrics using cosine similarity between averaged embed- dings (e.g., GloVe), which operate in the semantic space, for example as in Hawkins et al.) (2020a). Semantic similarity between messages is not sensitive to some lexical changes, ignoring the importance of exact word choice in convention formation. For example, once convention is formed to use the car to refer to an image, changing it to semantically similar message, the aufomobile, violates the stability property of conventions, which may increase the listener's 's cognitive load. Empirically, we find that WNR produces results consistent with Hawkins et al. (2020a)'s 's averaged GloVe embedding similarity, as shown in Figure (WNR moves in the opposite direction to the Glo similarity because WNR directly measures dissimilarity).
GloVe-Sim
WNR
0.8
0.6
0.4
0.2
0
1
2
3
4
5
6
Repetition #
Figure 4: GloVe embedding similarity and WNR between messages from consecutive repetitions. Every increase GloVe embedding similarity is captured by a corresponding decrease in WNR, and vice versa. Margins of Error are 95% bootstrapped Cls.
We also report a variant of WNR, which is not normalized by length. We refer to this variant as Word Novelty Distance (WND). WND is sometimes more interpretable because each
15

---

Published as a conference paper at COLM 2024
unit difference in it directly corresponds to word insertion or deletion. However, because WND is small between any two short messages, it can be harder to interpret when messages are short. Figure 5 reports WND for speaker experiments.
IDEFICS
LLava
Gemini
Claude
Human (Standard Setup)
$4: Explicit Instruc-
S1: Standard Speaker
52: Gricean Instruction
Explicit Instruction
tion+Consistency Request
7
6
5
WND
4
3
2
1
0
1
4
6
2
4
6
2
4
2
4
Repetition
Figure 5: Word Novelty Distance for speaker experiments. bootstrapped Cls.
Margins of Error are 95%
A.3 MLLM Implementation Details
The exact versions of the MLLMs used are idefics-S0biinstruut,, Ilava-1.5-13b, gpt-4-1106- vision-preview, Gemini 1.0 Pro Vision, and claude33-ppus22240223 For IDEFICS, use 8-bit quantization to fit the 80b model into 3 A6000 GPUs. For all MLLMs, we use a decoding tem temperature of 0 to avoid the uncertainty caused by sampling, which for IDEFICS and LL.Va. We use the models' default values for other hyperparameters.
also the default
The GPT4 listener used to evaluate model speakers follows the listener interaction setup in L3: Images Once. L3 is the scenario where GPTT4 shows its best performance and almost perfectly matches human performance.
LLaVa only supports taking 1 image as input. To bypass this constraint, we merge the images in the referential context into image, using 2-by-2 grid. Instead of using image labels (A, B, C, D), experiments with LLaVa refer to an original image using its location in the merged image (top right, top left, bottom right, and bottom left).
We conduct prompt engineering for each model individually to find the most suitable phrasing of the instructions. The prompt engineering was done over the pilot study dataset released in Hawkins et al. (2020b)'s official Github repository. The pilot study dataset is distinct from the 54 human interactions ICCA that we use for evaluation. This pilot study dataset contains Humman-humant interactions easy referential contexts, where the target images are more easily distinguishable to humans with very short messages. For this reason, the human interactions in this pilot study dataset show less language adaptation, and we only use it for prompt engineering and not evaluation.
16

---

Published as a conference paper at COLM 2024
B
Example Prompts
Model-as-speaker Prompt:
[System] Play game with a listener. This game consists of multiple trials and 4 images (labeled A, B, C, D). You will act as the speaker in this game. In each trial, one of the images given the target. You will generate message to tell the listener which image the target without mentioning any image label. The listener will try choose the target correctly based on your message. You will know which image the listener guesses, so you may adjust your messages based on the listener's 's accuracy. Your reply should only contain the message and be shorter than 20 words. Do not mention any image's label (A, B, C, D) in your message.
Image A: <imgl>
Image B: <img2>
Image C: <img3>>
Image <img4>
Trial 1, the target is Image B.
[Speaker] Message: Two bananas and apples in a white bowl on polka dot cloth. [System] The listener correctly answered Image B.
Trial 2, the target Image A.
[Speaker] Message: Bananas on a floral cloth with an Amazon box. [System] The listener correctly answered Image B.
Trial 3, the target Image
[Speaker] Message: Bananas and apples in a bowl, striped cloth underneath. [System] The listener mistakenly answered Image C.
Trial 4, the target Image C.
[Speaker] Message: Assorted fruit painted ceramic bowl.
[System] The listener correctly answered Image C.
Trial 5, the target Image B.
[Speaker] Message:
Figure 6: Example prompt for model standard setup. The prompt is shortened and revised for illustrative purposes.
17

---

Published as a conference paper at COLM 2024
Model-as-listener Prompt:
[System] Play game with multiple trials involving the same set of images. In each trial, will refer to one of the images with message. You will guess which image I'm referring If present, the history the previous trials may help you better understand how refer specific images.
Trial
Image A: <img4>
Image <img2>
Image C: <img3>
Image D: <img1>
Which image is this message referring to: The photo with 'amazon' on the back, with 2 bananas.
[Listener] Image D
[System] Correct. was referring to Image
Trial 2 Image A: <imgl>
Image <img3>
Image C: <img2>
Image D: <img4>
Which image this message referring to: Photo with a bowl of bananas with pokadot background.
[Listener] Image B
[System] Wrong. was referring Image C.
Trial 3
Image A: <img3>
Image <imgl>
Image <img2>
Image D: <img4>
Which image is this message referring to: A bowl full of mixed fruit, black background.
[Listener] Image A
[System] Correct, was referring to Image
Trial
Image A: <img3>
Image B: <img4>
Image <img2>
Image D: <imgl>
Which image is this message referring to: Photo with bowl of 3 bananas with stripes and other
patterns in the background.
[Listener] Image B
[System] Correct. was referring to Image B.
Trial
Image A: <img3>
Image <img2>
Image C: <img4>
Image D: <ingl>
Which image is this message referring to: Amazon in back with 2 bananas.
[Listener] Image
Figure 7: Example prompt for model-as-listener standard setup. The prompt is shortened and revised for illustrative purposes.
18

---

Published as a conference paper at COLM 2024
C Figures for Error Analysis
Messages
dirty truck going towards the bridge
2. dirly truck going to bridge
3. how is this even guessing, it's so casy, dirty truck going to bridge
4. dirty truck
5. dirty truck
dirty truck
GPT4 Messages
1. construction bridge with concrete mixer track underneath
2. bridge construction with white truck
bridge work, dirty below bridge construction overpass work
6. concrete mixer
Figure 8: Evolution of an image's corresponding messages across 6 repetitions in S3. Both the human speaker and the GPT4 speaker show length reduction, but GPT4's messages are more inconsistent.
Image A:
Carro
Image Bo
Image C:
Image D:
TOM
Image
Image F:
Image
Image II:
Camya
User: describe Image
IDEFICS: truck is parked in a parking lot.
Figure Example of IDEFICS' issue with associating images with their correct labels. The input is sequence of images interleaved with their labels and prompt to ask the model to describe Image [X]. We tried different formats to label the images, such as <img> is Image and Image A: <ing> but none helped IDEFICS produce reliable results as the number of images increases.
19

---

Published as a conference paper at COLM 2024
D Image Grounding Impact
IDEFICS
Ilava
GPT4
Gemini
Claude
Human (Standard Setup)
L7: Images Mis-
Images Masked
Images Misleading
leading Rep.
75
50
25
2
3
5
1
2
3
4
6
2
3
5
Repetition
Figure 10: Average accuracy for variants L5-7. Margins of Error are 95% bootstrapped Cls.
The results with L3 (Section 5.1) raised concerns about whether the models are using images effectively or just exploiting label-message association. We develop three variants to study this:
L5: Images Masked Each image is replaced by an image mask, where all pixels have the black color, (0, 0, 0). The image order is persistent, the same label refers always to the same underlying image that is masked. For the first four trials in Repetition 1, the model has to make guess. Regardless of the guess, the correct target is given by the system feedback as usual. Theoretically, the model can use the feedback for later repetitions. For a model succeed in this variant, it must associate the messages with the image labels since no actual images are given. This variant tests the extent which model can exploit label-message associations through in-context learning as way of convention understanding.
L6: Images Misleading This variant complements L55 to further study the impact of text signals and images. The setup here follows L3, showing the referential context once, except that we manipulate the images to be misleading. For the manipulation, we shuffle the images when presenting them to the model listener at the beginning the game, but we do not change the gold image labels in the system. Therefore, Image [X] from the speaker and the system's 's perspective is likely different from the Image [X] from the listener's 's perspective, and so on. For example, using the context in Figure 11 the model may see the message Photo with bowl of bananas with pokadot and choose Image C but the system and the speaker would always think the image that features bananas and polka dot Image B, since we shuffled the images without updating the gold labels accordingly. Then the system feedback would be wrong, the correct answer is Image B. To succeed under this setting, the model must learn to ignore the images and just associate all the descriptions related to polka dot (for example) with the label Image B.
L7: Images Misleading in Rep. This variant further tests if the models' previous promising performance in L3 comes from just exploiting the textual signals (the label-message associations) and ignoring the visual input. We show the context once similar to L3 at the beginning, but then show shuffled versions during each trial in the last repetition, without updating the gold labels (same manipulation as L6). We hypothesize that if the model tends to ignore the image input and rely textual associations in the conversation history, this manipulation will have little impact on its prediction or the accuracy calculated based on the old gold label. This variant requires 20 images in the last trial so it cannot be used with LLa Va or Gemini.
20

---

Published as a conference paper at COLM 2024
When models cannot utilize the image input (L5 and L6), all models start with around or below random chance accuracies (25%). As the interaction progresses, models exploit past messages and feedback via label-message associations to improve performance and display a trend of improvement. I.5 and L6 results together demonstrate that exploiting label-message associations while ignoring the images easily emerge as effective mechanism for MLLMs' convention understanding behavior, provided that the image referent has consistent textual label.
Injecting misleading images in the last repetition (L7) leads to a significant drop in per- formance relative to previous repetitions for IDEFICS (87.5-+69.4%) 4%), GPT4 (99.5+277. .%)), and Claude (98.6-22.22)) The drop is particularly conspicuous for GPT4 and Claude, where performance goes down to around random chance. This suggests that GPT4 and Claude do not overly exploit the consistent textual associations from the history, because otherwise they would not have been 'misled' by the manipulations in the last repetition. This is desirable because it shows that the visual input matters to these models' output. On the other hand, while IDEFICS also shows drop in performance, it is not as strong as what the other two models show. This indicates that although IDEFICS is not completely ignoring the visual input, does exploit the textual associations beyond what is desired.
Speaker's Perspective
Listener's Perspective
Trial 1
Speaker: Photo with bowl of 3 bananas with pokadot
B
B
Listener: Image C
Speaker: wrong, was referring to Image B.
D
D
Figure 11: Example Trial from L6: Images Misleading
21


Reviews:
[{"review_id": "irvGImFhFF", "reviewer": "Reviewer_XRWU", "summary": "In order to understand the Multi-modal LLM (MLLM)'s capability of In-context Conversational Adaption (ICCA). This paper defined ICCA as a framework to evaluate. \n\nICCA includes \n(1) A human-human interaction dataset that focus on repeated reference game interaction\n(2) A few success metrics including accuracy, message length, and Word Novelty Distance (WND)\n(3) Two types of set up: model-as-speaker and model-as-listener with different types of prompt settings\n\nWith such a framework, a series of experiments has been conducted and the results suggested the SOTA MLLMs are not good at in-context adaptation yet.\n\nThe paper is structured well, and quite easy to follow. The experiment design looks good to me. The result itself isn't surprising. Thus I am on the borderline leaning towards accepting the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you! We are glad you appreciated our novelty, clarity, and soundness.\n\n**Significance of the results**\n\nAre our results surprising? The progress of LLMs has been impressive, with new results often surpassing human performance. LLMs are trained on human language data, which is rich in adaptation and convention formation/use. Given these models’ success, one would expect them to acquire these characteristics of human language as well. Our results show the opposite: while some aspects of language are well-addressed by current paradigms, others are not. Beyond the fundamental learning problem, our work’s importance was made clear by the recent conversational use cases LLM companies have been demoing (e.g., OpenAI’s latest demo). Overall, this illustrates that the problem is both important and underdiagnosed by current research. ICCA is a framework to drive important future development, in a way that was not previously on the agenda or easy to diagnose. We will improve the presentation of the significance in the paper. \n\n**Breadth & depth of the results**\n\nIndeed, we observe MLLMs do not display in-context conversational adaptation. Our analysis and ICCA itself go deeper though, including:\n* Even with explicit instruction on length reduction (Variant S3), the models display lexical efficiency but miss critical aspects of true communication efficiency (convergence and consistency). This finding is unexpected (contrary to human behaviors). It emphasizes that communication efficiency is not only about length reduction. ICCA gives future researchers the tools to look beyond length. \n* Some models can display seemingly good listener accuracy by memorizing textual information without forming conventions on visual signals. Our different interaction variants make ICCA an effective tool for such diagnosis. This is not possible using the standard evaluation from the prior work. We discuss this in Appendix D, but will expand the discussion in the paper.\n* Associative recall, the ability to recall information mentioned in-context is believed to be predictive of in-context learning quality [1]. We observe underexplored implications of this ability. Throughout an interaction, IDEFICS and LLaVa assign high probability to the exact repetition of initially used phrases, marking a good associative recall. But this hinders the development of more concise phrases. We will further elaborate on this issue when we revise the paper.  \n\n[1] https://arxiv.org/pdf/2312.04927", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Novelty**\nBuilding metrics to evaluate LLMs on in-context adaptation is a nice thing to have. When LLMs are more capable of getting answers right, the next step might be getting the communication more efficient during multimodal settings.\n\n**Clarity**\nThe paper is clear about what it does and how things are being done. \n\n**Sounds Approach**\nThe approach of the paper is sound. Defining", "reasons_to_reject": "**Significance**\nThe finding of the paper isn't surprising. Establishing the metrics and building a baseline about MLLMs on one task itself is a contribution, I don't feel excited about the result and the work.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xNhVxxR9FY", "reviewer": "Reviewer_MJyu", "summary": "The manuscript explores the ability of MLLMs to engage in reference games, which involve a duo of a speaker and a listener examining similar images. The speaker's task is to choose and describe one image for the listener to pinpoint. While simple for humans, the game's purpose is to uncover tactics to reduce the length of descriptions needed over time through implicit agreements among the participants. The core experiment involves MLLMs playing these games to see if they can evolve such communicative strategies. The MLLM prompts are specifically designed to encourage this behavior. The study notes that the state-of-the-art MLLM tested did not effectively develop these strategies for more efficient communication, although it could comprehend them when presented.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! \n\n\n>the term adaption is way too vague to describe the phenomenon taking place during reference games. \n\nWe use “adaptation” following established conventions in cognitive science that describe the phenomena humans show in reference games: spontaneously and gradually adopting more efficient language and understanding the interlocutor better. These are exactly the behaviors we are interested in. Our abstract and introduction clarify that our goal is to create tools to evaluate whether MLLMs display such adaptations as humans do, which are essential for communication efficiency. We will make sure the intro relays the term \"adaptation\" in a concrete and clear way. \n\nAlso, such adaptations are broad phenomena, and not just about reference games. But, they are known to be well-demonstrated, measurable, and extensively studied in cognitive science through reference games. If LLMs acquire linguistic abilities similar (or better) to humans, reference games make for an ideal testbed.\n\n\n>And even more it can let think that the LLMs may be adapted to reach such specific objectives.\n\nThe title specifically focuses on “in-context” behaviors of MLLMs, so it clearly states the kind of adaptation we are studying, meaning: in contrast to tuning parameters or changing the architectures. In-context learning is well established in the study of LLM as a way to modify (i.e., adapt) LLM behavior. We also discuss the relatedness of in-context adaptation and few-shot in-context learning in the related work section. We will further clarify this distinction as early in the paper as possible.\n\n\n>one may straightforwardly think of means to accompany these prompts and re-enforce their effects...\n\nOur focus is evaluating if LLMs without further modifications display this behavior, to see if it arises from their training. This is key to displaying this behavior in a truly general way. Our goal here is not just to understand the abilities of LLMs, but to contribute to the fundamental understanding of what arises (and what does not) from the way LLMs are trained. This is why we made prompting efforts specifically to elicit behavior that is already baked into the model, but maybe doesn’t show for one reason or another. Building on top of LLMs to further address their deficiency in adaptation is a good direction for future work, one that is enabled and supported by the ICCA framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall the paper present a decent piece of work. The presentation is clear and the concepts well introduced and recalled. The study can be a first step on the line to train MLLM with better capacity of adapting or forming ad-hoc conventions when communicating with humans.", "reasons_to_reject": "The title is misleading: the term adaption is way too vague to describe the phenomenon taking place during reference games. And even more it can let think that the LLMs may be adapted to reach such specific objectives. But the study boils down to pure observation of MLLMs' behaviors when used in reference games. Proposing different prompts with customized instructions (\"Gricean\" etc) is a good start, but one may straightforwardly thing of means to accompany these prompts and re-enforce their effects (for instance depending on the type of exemples proposed in the context, or using MLLMs in several epochs; including explicit summarisation/simplification taks). Here's what we might like to read.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4piVgOfqdz", "reviewer": "Reviewer_eDRe", "summary": "The paper proposes a framework to evaluate the conversational adaptation of MLLMs. The paper then describes evaluations of a number of MLLMs. The results show that the generations do not become more efficient over time, as would be desirable. However, more careful prompting somewhat mitigates this. These findings have implications for the development of MLLMs.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! \n\n**Comparison with Hawkins et al (2020ab)**\n\nWe are inspired by Hawkins’ works, rely on their data, and discuss them extensively. We will enhance the discussion of how our work differs. First, we take their data and the reference game scenario in a very different direction, both our object of study and research questions are different. Hawkins presents an approach of adapting through tuning model parameters during an interaction. We focus on diagnosing the behavior of LLMs (a class of models very different from the models Hawkins studied) and on in-context learning/adaptation. Additionally, our evaluation differs. Hawkins focuses only on the standard setup. When a model fails under the standard setup, there can be many contributing factors to the failure. We design variants specifically to diagnose the types and causes of model failures for in-context LLM adaptation. As you suggest, we will communicate these differences more clearly and explicitly.\n\n**Adding related works**\n\nThese are great suggestions! We will improve the paper to discuss [1-4] (if the paper is accepted, we get an extra page for the camera ready). \n\nIt’s particularly relevant that [3] shows the unintended consequences of using explicit prompts to help form common grounds, which is consistent with our findings and intuition that explicit prompt intervention cannot solve adaptation failures. Discussing [3] will provide a more comprehensive picture of this issue. The synthesis of our work with [3] and [4] also highlights interesting connections. Both [3] and [4] study scenarios where common ground requires explicit acts (e.g., clarification/followup questions) and LLMs presume such common grounds too easily without adequately using those acts. In our case, ad-hoc conventions form without explicit actions in humans, whereas LLMs fail to do so. These different types of common grounds bring about different challenges. This highlights both the richness and the importance of the space. \n\n\n>Anthropomorphizing language should be avoided\n\nWe completely agree with the importance of avoiding such language! We will revise the writing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an important problem, and the paper describes a valuable contribution to addressing it. The setup is simple and clear, which I see as a strength. I also found the introduction well-motivated and the background sections very detailed.", "reasons_to_reject": "Some limitations and issues I wanted to raise include:\n\n-The contribution appears somewhat iterative compared to existing setups and measurements by Hawkins et al (2020ab), extending to evaluating MLLMs. The novelty is not very clearly communicated to readers.\n\n-The paper is very relevant to, but does not engage with the concepts and findings on conversational grounding and building common ground through dialogue. Please see related work [1-4] on some relevant seeds for background and existing studies regarding evaluating and improving conversational adaptation of LLMs (especially 3 and 4, which characterize similar limitations in similar settings).\n\n-Anthropomorphizing language should be avoided, especially when the claims are unfounded (\"they (MLLMs) may understand the increasingly efficient language of their interlocutor\"; this study doesn't aim to measure understanding); \"MLLMs may passively understand\" (see [5,6])\n\n1. Clark, Herbert H., and Susan E. Brennan. \"Grounding in communication.\" (1991).\n2. Andukuri, Chinmaya, et al. \"Star-gate: Teaching language models to ask clarifying questions.\" arXiv preprint arXiv:2403.19154 (2024).\n3. Shaikh, Omar, et al. \"Grounding Gaps in Language Model Generations\" NAACL 2024.\n4. Testoni, Alberto, and Raquel Fernández. \"Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions.\" EACL 2024.\n5. Inie, Nanna, et al. \"From\" AI\" to Probabilistic Automation: How Does Anthropomorphization of Technical Systems Descriptions Influence Trust?.\" arXiv preprint arXiv:2404.16047 (2024).\n6. Abercrombie, Gavin, et al. \"Mirages: On anthropomorphism in dialogue systems.\" arXiv preprint arXiv:2305.09800 (2023).", "questions_to_authors": "", "extra_scores": {"ethics_concerns_details": "N/A"}}, {"review_id": "plRkZwk1l0", "reviewer": "Reviewer_8pZ2", "summary": "This paper presents a series of experiments designed to probe the ability of LLMs to form conceptual pacts / to act on precedent in repeated reference games. Given that LLMs have been tested on reference games before, this is a natural next step to test, and it is good to see it realised. The authors find that there is very little evidence of LLMs treating the conversational context in the same way as the presence of conversational (and hence shared) context would be by natural interlocutors. Only when coaxed through very suggestive prompting, some small effect can be found.\nThe paper is well written and easy to follow, and could be useful if only to introduce the relevant literature to a field where talk about \"mental models\" and \"theory of mind\" is often very cheap.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments!\n\n>I think the results are stronger than the authors let on...\n\nWe will improve and expand the discussion (if accepted, with the extra camera-ready page). The framing you suggest (agent vs next-word predictor) nails it: LLMs have no reason to communicate efficiently (no cost/effort). We hypothesized that they may still acquire it by training on human data. Our results clearly show this is not the case, which can be considered surprising given how much human data is used and how well the models imitate it. \n\n>Experiments on using prompts to induce… bit unnecessary\n\nWhile we hope adaptation to arise naturally, we need to answer questions about inducing it (e.g., as raised by R-XRWU), and prompting is often considered promising. We also wanted to see if MLLMs do have the ability for Gricean behavior, and maybe just not show it without being “prompted”. Our results show this is not the case, and prompting is not sufficient for such pragmatic behaviors. \n\nMore profoundly, these variants reveal underexplored causes of failures. For example, the gain in lexical efficiency without message convergence under some prompts is a surprising finding, which will caution future researchers not to rely on length reduction as the sole indication of efficiency. Overall, to offer a framework for evaluating MLLMs, we wanted ICCA to be thorough and address the different ways people may use MLLMs (e.g., different prompt intervention/interaction formats). \n\n**Engineering prompts for each model**\n\nWe wanted to give models the best chance, even at the cost of extra development. For example, a prompt that worked well for one model might lead another model to sometimes output invalid outputs (e.g. “I’m not sure about the answer”), whereas additional instructions against these trivial but adverse behaviors may hurt the models not displaying these behaviors (verbosity of the instruction can hurt LLM performance [1]). By adopting model-specific prompts, we can mitigate these issues. With this being said, all model-specific prompts under the same experiment setup are based on the same instruction. Only the prompts across different interaction variants (different experiment setups) have a substantial difference. \n\n[1] https://arxiv.org/pdf/2307.03172\n\n**Related work**\n\nThank you for suggesting the related works! We will discuss the photobook dataset, Chalamalasetti et al 2023, and the broad picture of using self-play to evaluate LLMs as conversational agents.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Very timely experiment, well conducted\n- Well and very clearly written\n- Good command of the relevant literature", "reasons_to_reject": "- Discussion somewhat disappointing. I think the results are stronger than the authors let on: The fact that the models fail so completely to exhibit precedent effects indicates that they, despite what many people seem to hope, have not transcended beyond being text completion models and towards being agent models. The models simply have no reason to communicate efficiently -- among other things, because they are not communicating.\n\n- Related to that, I found the experiments on using prompts to induce the desired behaviour a bit unnecessary. With these prompts, the focus moves from investigating what kind of entity has been learned to less interesting engineering questions. (Although I do understand the rationale that users of these models might even be confused if apparent conceptual pacts are broken.)", "questions_to_authors": "- Can you say more about why you engineered promts for each model? This is a question that arises in all kinds of between-model evaluations, and people come to very different conclusions. But it seems to me that most often, people decide on \"no prompt engineering is fairest\".\n\n- Not a question, but a comment: You could mention the PhotoBook dataset in the related work, which was built to study exactly your main phenomenon. (Haber et al 2019)\n\n- Not a question, but a comment, part II: You could mention as related the model-model self-play frameworks that came out last year, which pioneered the setup that you are also using. One of which, (Chalamalasetti et al 2023) even includes a reference game as one of the test dialogue games. (But they are not focussing on repeated episodes.)\n\n\nKranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2023. clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11174–11219, Singapore. Association for Computational Linguistics.\n\nJanosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910, Florence, Italy. Association for Computational Linguistics.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework called ICCA (In-context Conversational Adaptation) to evaluate the ability of Multi-modal Large Language Models (MLLMs) to engage in reference games and develop efficient communication strategies over time. The authors include experiments on the human-human repeated reference game dataset from Hawkins et al (2020), and compare different setup variants. Experiments using the ICCA framework suggest that current state-of-the-art MLLMs do not effectively adapt their communication strategies in-context: they are relatively good "listeners" but not human-like "speakers".

The majority of reviewers agree that this paper addresses an important problem and makes a valuable contribution to understanding the in-context adaptation capabilities of MLLMs. The ICCA framework provides a clear and well-motivated approach to evaluate these models' ability to develop efficient communication strategies over time, drawing inspiration from human behaviors in reference games. I personally loved this paper.

Key strengths highlighted by the reviewers include the novelty of the evaluation framework, the clarity of the paper's presentation, and the soundness of the experimental approach. The findings, while perhaps not surprising, are significant in showing that current MLLMs do not effectively display the kind of in-context adaptation that humans exhibit in these scenarios. This has important implications for the development and deployment of these models in conversational settings.

The authors provide thoughtful responses to the reviewers' comments and suggestions in their rebuttal. They clarify the differences between their work and prior related research and propose to enhance the discussion of these differences in the paper. The rebuttal also addresses concerns about the use of model-specific prompts and the inclusion of prompt variants in the experiments, providing convincing justifications for these choices.

In summary, the ICCA framework and the insights gained from its application to MLLMs make this paper accept. The work advances our understanding of the strengths and limitations of these models in conversational settings and lays the groundwork for future research in this area. With some revisions to clarify the novelty and significance of the contributions, as well as to incorporate additional related work and refine the discussion, this paper is likely to have a positive impact on the field.
Decision: Accept