Bridging semantics and pragmatics in information-theoretic emergent communication
Authors: Eleonora Gualdoni
Mycal Tucker
Roger P. Levy
Noga Zaslavsky
Venue: NeurIPS 2024 poster (2024)
Keywords: semantics
pragmatics
emergent communication
information theory
artificial agents
Primary area: neuroscience_and_cognitive_science
TL;DR: Co-evolution of semantics and pragmatics in artificial agents

Abstract: Human languages support both semantic categorization and local pragmatic interactions that require context-sensitive reasoning about meaning. While semantics and pragmatics are two fundamental aspects of language, they are typically studied independently and their co-evolution is largely under-explored. Here, we aim to bridge this gap by studying how a shared lexicon may emerge from local pragmatic interactions. To this end, we extend a recent information-theoretic framework for emergent communication in artificial agents, which integrates utility maximization, associated with pragmatics, with general communicative constraints that are believed to shape human semantic systems. Specifically, we show how to adapt this framework to train agents via unsupervised pragmatic interactions, and then evaluate their emergent lexical semantics. We test this approach in a rich visual domain of naturalistic images, and find that key human-like properties of the lexicon emerge when agents are guided by both context-specific utility and general communicative pressures, suggesting that both aspects are crucial for understanding how language may evolve in humans and in artificial agents.

Bridging semantics and pragmaties in information-theoretic emergent communication
Eleonora Gualdoni Apple MLR Universitat Pompeu Fabra wwwplddnniapple.... com
Mycal Tucker MIT mycal@mit.edd
Roger P. Levy MIT rplevy@mit.edu
Noga Zaslavsky NYU nogazOnyu.edu
Abstract
Human languages support both semantic categorization and local pragmatic inter- actions that require context-sensitive reasoning about meaning. While semantics and pragmatics are two fundamental aspects of language, they are typically studied independently and their co-evolution is largely under-explored. Here, aim bridge this gap by studying how a shared lexicon may emerge from local pragmatic interactions. To this end, we extend recent coormatiiontheoretii framework for emergent communication in artificial agents, which integrates utility maximiza- tion, associated with pragmaties, with general communicative constraints that are believed to shape human semantic systems. Specifically, we show how to adapt this framework to train agents via unsupervised pragmatic interactions. and then evaluate their emergent lexical semantics. We test this approach in rich visual domain of naturalistic images, and find that key human-like properties of the lex- icon emerge when agents are guided by both context- speccfcc utility and general communicative pressures, suggesting that both aspects are crucial for understanding how language may evolve in humans and in artificial agents.
Introduction
Languages evolve through repeated interactions in rich contexts, where various communicative and non-communicative goals co-exist. The conveyed meaning is often shaped by the local conversational context of utterances (Figure I), as captured by the pragmatic behavior of interlocutors [1]. For example, the word 'player' can be interpreted as baseball batter, catcher, or a guitar player, depending on the conversational context that shapes the listener's 's beliefs about the speaker's 's state of mind. That is, understanding meaning in context requires pragmatic reasoning about other agents' intentions and beliefs [1-3]. At the same time, words are associated with con-contextualized meanings, as captured by lexical semantics. For example, we have shared idea of what 'player' means, regardless of any specific context in which this word might appear, and we can use it to communicate with new conversational partners in new contexts. While semantics and pragmatics are widely studied, their interface and co-evolution is largely under-explored and not well understood. In this work, we focus on key open question at the interface of lexical semantics and pragmatic reasoning: How can shared human-like lexicon emerge from local context-sensitine pragmatic interactions?
To address this question, we build on framework for information theoretic -hehorttt emergent communication in multi-agent reinforcement learning systems [4]. This framework is particularly relevant for addressing our question because integrates task-specific utility maximization in well-established models of human pragmatic reasoning [2, 3, 5], which has also been center-stage the literature on emergent communication artificial agents [6-8] - with general information- theoretic constraints that are believed to shape human lexical semantic systems [9-13] and have
central component
Work finished prior to joining Apple
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Semantics
"player"
"batter"
Pragmatics
An example of an image from the ManyNames dataset illustrating both semantic and pragmatic communication. Top (semantic setting): A speaker communicates a target object at (red box) by naming it regardless of the context induced by the image. The ManyNames dataset includes such responses from native English speakers (in this example, 10 speakers produced 'man', 10 'batter', 5 'baseball player', 4 'player', and 3 'person"). Bottom (pragmatic setting): A speaker and listener observe two objects in an image (red and blue boxes). Only the speaker knows which one is the target z; and which one is the distractor T.. The speaker's goal is to communicate given this shared context, and the listener's goal is to discriminate the target from the distractor.
Figure 1:
recently also been applied to human pragmatic reasoning [14]. These information-theoree constraints are derived from rate-distortion theory [15, 16), and more specifically, the information bottleneck (IB) principle [17], characterizing semantic systems in terms of efficient compression of meanings into words by optimizing the IB tradeoff between the complexity and informativeness of the lexicon [9]. Tucker al. [4] developed framework for training agents in emergent communication settings by integrating utility maximization with the IB objective, yielding a multi-objective optimization problem that trades off maximizing task-specific utilities with maximizing task-agnostic communicative informativeness and minimizing communicative complexity.
Here, we extend that framework to explicitly model the co-evolution of semantics and pragmatics. In our setup, artificial neural agents learn to communicate in a pragmatic setting, i.c., in the presence of shared conversational context that may alter the meaning of their communication signals (illustrated intuitively in the bottom image of Figure 1). Importantly, agents are trained via self-play, without any supervision or exposure human languages. (i.e., utility) of the agents as well as the they converged on, by invoking it in a lexical semantic setting (illustrated intuitively in the top image of Figure 1, and see Figure 2 for a full description of our model's architecture). Given the substantial empirical evidence that human semantic systems are pressured to optimize the IB tradeoff, predict that shared human-like lexicon will not emerge when agents are guided solely by utility maximization but rather when they are guided by a non-trivial tradcoff between optimizing utility, informativeness, and complexity. Our goal in this work is to test this prediction, and more generally, to characterize the landscape of emergent communication systems induced by our novel cermmation-theoretie framework for the co-evolution of semantics and pragmatics.
test time, we assess the pragmatic competence of the shared emergent lexicon that
To this end, we consider a rich visual domain of naturalistic images provided by the ManyNames dataset [18]. which also contains free-naming human data generated by native English speakers. This domain allows us to train agents across many different conversational contexts and then evaluate their emergent lexicon with respect to the English naming data. In support of our prediction, we find that nuuann-like properties of the lexicon (its size, complexity, and alignment with English speakers), together with high pragmatic competence, emerge when agents are guided by both context-specific utility and general communicative pressures as derived from the IB principle. Interestingly, pressure for communicative informativeness, rather than (non-communicative) task utility, appears the main driver of emergent communication, but weaker pressures to minimize complexity and maximize utility are still crucial for achieving human-like properties of the lexicon.
Our work significant both from a cognitive science perspective and from machine learning perspective. From cognitive science perspective, our work provides novel computational frame- work for studying the under-explored interface between lexical semantics and pragmatic reasoning, and our findings suggest that human languages may evolve under pressure to optimize tradcoff

---

between task- seeek---eificcc utilities and general communicative constraints. From a machine learning perspective, our work demonstrates how cognitively motivated optimization principles, implemented in neural network agents as intrinsic training objectives, can facilitate the emergence of interpretable human-like communication systems without any human supervision.
2 Related work
Our work integrates cognitively information-theorele principles that are believed to underlie human language evolution, with deep learning tools for studying emergent communication in artificial agents, in order to develop a computational account of the co-evolution of lexical semantics and pragmatic reasoning. Semantics and pragmatics constitute two subfield in cognitive linguistics. While both focus on meaning in language, they capture largely complementary aspects of meaning. Lexical semantics generally concerned with word meanings [19, 20], independent of any specific conversational context, whereas pragmatics is concerned with language use in context, typically assuming a known shared lexicon [1, 21, 3]. Our work focuses on the underexplored interface between these two aspects of language, departing from the traditional assumption that the lexicon is given a-priori and shared among pragmatic interlocutors.
Contemporary cognitive approaches to lexical semantics argue that word meanings are shaped by pressure for efficient communication [22. 23, 10]. Most relevant to our work, is the information- theoretic framework for semantic systems, proposed by Zaslavsky et al. [9]. that predicts that human semantic systems evolved to optimize the information bottleneck (IB) trade-off [17] between the complexity of the lexicon (roughly, how many bits are allocated for communication) and its informativeness (roughly, how well listener can understand speaker, regardless of context). This framework has shown to account for the structure of semantic systems across hundreds of languages and multiple domains [9, 24, 11, 12], as well as semantic change over time [13]. Here, we leverage this empirically-suppore theoretical framework to guide our interactive agents.
Research on pragmatic communication focuses how speakers' / lexical choices and listener's interpretations are affected by their local conversational context [1. 3, 25, 26). Prominent models of pragmatics, such as the Rational Speech Act (RSA) framework [3], are grounded in game theory (see also [2]). In this view, pragmatic behavior is formulated within a cooperative reference game, where agents pragmatically reason about each other's communicative intentions with the goal of maximizing the game's 's utility. While these models enjoy broad empirical support [3], they assume that the underlying lexicon is given and shared across speakers and listeners, even when applied in reinforcement learning settings [27]. Recently, a theoretical link between the RSA framework and rate-distortion theory (RDT) has been derived [14), connecting pragmatic reasoning with general informational constraints that are closely related to the aforementioned IB trade-off. However, the RDT approach to pragmatics has also assumed given lexicon [14], and more generally, the implied information-thcoretic link between semanties and pragmatics has not yet been explored. Our work explored this potential link by relaxing the assumption that the lexicon is given to our agents, and instead studying how they may develop their own near-optimal human-like lexicon via a training objective that takes into account informational constraints.
Relatedly, Brochhagen et al. [28] developed a game theoretic model for the evolution of the division of labor between semantics and pragmatics, and tested the model in a relatively small domain. In comparison, we consider here different learning dynamics and objective function, employ state-of-the- art deep-learning tools for training agents at scale, and evaluate our approach with respect to actual human data in rich domain of naturalistic images [18]. McDowell and Goodman [29] considered the role of pragmatics in learning lexical meanings in deep-lcarning agents. However, they trained their agents in a supervised manner with respect to human- -ennrratd pragmatic data. In contrast, we are interested in how pragmatics and lexical semantic may emerge without any human supervision.
The emergent communication literature focuses on how agents may learn to communicate in rein- forcement learning settings, without exposure to human- -eneratee linguistic data [6. 7, 30]. While this body of literature largely focuses on utility (or reward) maximization, Chaabouni et al. [8] showed that -ased emergent communication can lead to IB-efficient systems. In their settings, however, tradeoffs can only be determined by external environmental factors. Tucker et al. [4] showed how to directly integrate the IB objective function with utility maximization in emergent communication and demonstrated the advantages of this framework for
different
3

---

utility
informativeness
speaker
listener
VAE
m2
S
D
mh
VAE
Mid
mask
complexity
shared context
IF0
21
Figure 2: Communication model for the co-evolution of semantics and pragmatics (see Section 3). Agents are trained in a pragmatic setting, where both observe inputs z0, 21 shared context; one input is randomly selected as target 3e for the speaker. After training, the agent's emergent communication systems are evaluated in a lexical semantics setting, without shared context; the speaker observes only 2; while za is masked, and both inputs are masked for the listener.
faster convergence rates and better out-of-distribution generalization [31]. However, they did not study the co-emergence of semantics and pragmatics. Our work directly builds on the framework of Tucker et al. [4, 31] and extends it for studying the interface between semantics and pragmatics.
3 A unifying model for the co-evolution of semantics and pragmatics
The information-theonm framework for emergent communication that we build on [4, 31] consists of an objective function that integrates utility maximization with the IB principle, and communication model that specifies the agents' architectures. Our model (Figure 2) builds on Tucker et al. [4]'s proposed method, called vector-quantized variational information bottleneck (VQ-VIB), which modify in order to be able to model the co-evolution of semantic and pragmatics; that is, to be able to train agents in a pragmatic setting and evaluate their communication, at test time, in both pragmatic and lexical semantic settings. The initial VQ-VIB architecture includes a speaker and a listener tasked to jointly solve reference game. In our case, the speaker is defined by (i) a pre-trained variational autoencoder [VAE, 32] that provides visual representation module for mapping input to 'mental' representation m; and (ii) an encoder module S that generates communication signal w given the speaker's 's mental state. The listener is defined by (i) a decoder D that observes and solving a downstream task.
generates a reconstruction m, and (ii) a policy
In our pragmatic setting both agents observe a shared context (40. x1), while the speaker also observes which referent the target and which is a distractor d (more details about this step are provided in Section 4). The speaker then aims to communicate the target zt. The VAE representation model applied 2g and Ia independently, generating m,, the speaker's mental representations for the target, and ma, the speaker's ' mental representations for the distractor. The listener's task to guess the target based on y L(m, (Nr, .1)). We wish to emphasize that in contrast to the standard approach in pragmatic models, in our setup the agents are not given a shared lexicon but rather implicitly learn it through their local, context-sensitiee pragmatic interactions.
Agents are trained by optimizing a tradeoff between maximizing expected utility, maximizing informativeness, and minimizing complexity. The utility term, U U(x,, y), is task-specific, and here take to be the accuracy of the listener's downstream decisions. Informativeness and complexity are task-agnostic communicative objectives. derived from the Information Bottleneck (IB) framework for semantic systems [9]. In this framework, the speaker's 's and listener's mental representations, m, and m, are defined as belief states (i.c., probability distributions) over features environment. Informativeness is related to the distortion between m, and m. defined by the Kullback-Leibler (KL) divergence D(mel] [p]] such that low expected distortion amounts high informativeness. Complexity corresponds to the mutual information between speaker's 's meanings and words, [(m,; w). which roughly the number of bits that the agents will need for communication. Following [4, 31), we optimize a bound on the IB terms for practical reasons, as direct optimization of those terms does not scale. For informativeness, we treat m4 and mn as the means of the agents' mental distributions and then approximate informativeness by their MSE. For complexity, we consider known variational

---

bound on the mutual information, denoted here for simplicty by I [for more details and full derivation of the objective function, see 4]. Overall, the training objective is to maximize
Ay [U (II, y)] X [Ime mell m Ac [(T);
(I)
where the As non-negative tradeoff weights that sum 1, and f(T) can be seen as function that extracts two objects (a target and a distractor) from an image.
Optimizing only the utility term in the objective function, without any pressure for alignment between the speaker's 's and listener's representations (i.e., high informativeness)) is expected to lead to lexical systems that are biased towards success in the specific training downstream task, and such systems are likely to depart from human lexical systems. On the other hand, maximizing informativeness alone will lead to highly complex and task-agnostic systems, and minimizing complexity alone will lead to no communication. None of these extremes seems human-like, and we therefore expect that human-like systems will emerge when all tradeoff' parameters active (i.e., non-zero).
Our semantic setting is designed to evaluate the emergent lexicon at test time, lending a window into how the agents use their words irrespectively of any local context. In this setting, only the target is shown to the speaker (the distractor is masked) and then the listener reconstructs mn based on the speaker's 's word w, without any additional context downstream task. This task resembles in its nature the task with which annotations for ManyNames -our dataset of interest [18], see below- gathered, i.e. naming task where annotators were asked to freely produce names for target objects appearing in natural images, identified with bounding boxed. Thus, we can use these naming annotations at test time evaluate the agents' emergent semanties.
4 Experimental setup
4.1 The ManyNamee domain
We train and test agents on the ManyNames dataset [18], which contains 25K naturalistic images (see Figure 1 for example), each with one target object, appearing in a bounding box, annotated with ~ 36 names provided by English native speakers asked freely produce a name (one word) to describe the object, 2 The name produced by the majority of the annotators for a target object is called the topname. In the case of Figure 1, top image, 'man' and 'batter' are equally probable topnames. Objects in the ManyNames dataset are also annotated according to their high-level semantic domain, which can be people, animals- plaanss, buildings, vehicles, clothing. food, or home. Importantly, our agents only observe the images in the dataset. They are not trained on any of the linguistic labels, which are used only for evaluation.
4.2 Data selection for pragmatic training
With respect to the choice of the target-distractor pairs for our pragmatic training setup (Figure 1, bottom image), our intuition is that identifying competing objects appearing in the same image, e.g.. a batter and a baseball player in the same field, or a car and a taxi on the same street, instead of using random objects cropped from different images, should encourage the emergence of more natural semantic partitions, since agents will need to create lexical entries to solve naturally occurring ambi- guities. For each image in the dataset, we consider the target object highlighted in the Many annotation; and an additional object that we individuate through automatic object detection and few filtering filtering steps. We used the Bottom-Up object detection model proposed by Anderson et al. [33], which incorporates a Faster R-CNN architecture [34] for object detection, and ResNet- 101 architecture [35] for feature extraction. Since Anderson et al. [33]'s model fine-tuned on the VisualGenome image dataset [36]. of which ManyNames images are a sample, we are guaranteed that this model can make good quality predictions on ManyNames. After running the object detector model on our images, we filter the detected bounding boxes keeping only those with Intersection over Union smaller than 0.1 with the ManyNames target, and that did not have size smaller than 10% of the We then computed the similarity between the candidates' visual features -automatically target size.3
CCrettive Commons Attribution 4.0 International License.
'For 2.5K images, typically depicting only one large object as the ManyNames target, we do not find any detected object with the desired properties. Being unable to use these images in the pragmatic setting, we exclude them from our data sample.
5

---

Model
AComp.
ALexSize
NID
Utility
MSE
A =
1.76 (+0.31)
559 (#:82)
0.84 (=0.02)
0.95 (#0.00)
20 (87.18)
A = 1
2.80 (10.06)
1687 (#107)
0.60 (=0.00)
0.83 (#0.00)
0.13 (=0.00)
c =
5.21 (=0.01)
381 (=0.98)
0.99 (=0.00)
0.58 (+0.00)
0.32 (#0.00)
Fng
1.48 (=0.08)
22 (+:35
0.55 (=0.00)
0.72 (=0.01)
0.23 (=0.00)
Table 1: Evaluation of the model that is best aligned with English, Xon (Au: = 0.005, X = 0.98,
Àc = 0.015), in comparison with three baselines: Au = 1, corresponding to utility maximization without any informational constraints, which is the most common objective function in the emergent communication literature; and X and Ac 1 which correspond to the other two extremes of either maximizing informativeness or minimizing complexity. ACompl. and ALexSize are the absolute value of the differences between the complexity or lexicon size of the emergent system and English. NID measures the misalignment between the emergent system and English. For all three measures, lower values reflect a better fit to the human data. Urility and MSE correspond to the agent's pragmatic competence and reconstruction error, respectively. Each cell shows the mean value across three random seeds + SEM.
extracted by the ResNet-101 incorporated in Anderson al. [33]'s model- and the ManyNames target's ' features, choosing the detected object with the highest similarity value. This final selection step based visual similarity aims providing our agents with some challenging cases to solve, like the one shown in Figure I, where communication needs to allow the discrimination between two objects from the same semantic category, e.g. batter and another baseball player.
At the end this procedure, we obtained two objects per image: the ManyNames target; and the additional object detected by us. We used these two objects to train our agents, choosing our target's identity randomly, making sure that in 50% of the cases the target would be the larger object, and in the other 50% the distractor would be larger.
4.3 Multi-agent simulations
We trained 270 agent pairs with a range of combinations of Au Ac and λ₁. We used parameter annealing which, besides being computationally efficient alternative training agents from random initializations, has been shown to capture at least some aspects of the evolution of human semantic systems [9]. We used a pretrained ResNet18 [35] to extract 512 dimensional features from target and distractor objects before passing them to the agents.* 4 In all our experiments, we trained agents on 70% of the images (randomly sampled) in self self-play, with no human supervision, batch size of 128, and codebook initialized with 3000 trainable communication vectors (see Appendix A for further
details).
Results
5.1
Quantitative evaluation
We report quantitative results for the 5 evaluation metrics shown in Figure 3, for each set of λs, at test time. The top panel shows the three components of the agents' training objective: (a) the agents' pragmatic competence, pragmatic setting: (b) the reconstruction loss between the speaker's is and listener's ' mental representa- tions, measured by the negative MSE (values closer to 0 corresponds to higher informativeness); and (c) the complexity of the emergent lexicon. The bottom panel of Figure shows two measures for assessing the liman-likeness, of the emergent systems: (d) the lexicon size of the artificial systems,
measured by their test-time expected utility on unseen images, in the
"We extract visual features with ResNet18 model, that is standard choice for object classification,, and choose not use the visual features from Anderson et al. [33]'s model, which we employ for the object detection phase. This because this model, being specialized for object detection, produces visual features for objects in bounding boxes that carry some degree of information about the image context. This is desirable property when performing object detection, hut not for our experiments.
"Our code is available https: // /github. com/ wwt.://ooooooonnnfeeeooo intiiaa//iiioeeee sem- prag- neurips2024.

---

M
14
Eng
67
-.40
Au
Ac
Au
(a) Utilty
(b) Negative MSE
Complexity
A
NI
1.0
1993
Eng
07
7ay
1:c
(d) Lexicon size
(o) Semantic alignment
Figure 3: The intoommationtthorree landscape of emergent communication. Each simplex shows a test-time evaluation metric using either pragmatic (a) or semantic (b-e) settings, for the range of models spanned by the values of À (Au, År, Ac). Overall, none of the extremes yield human- like communication, and the best alignment with English is achieved for À > O near the top of the simplex. (a) Utility, reflecting the agents' pragmatic competence in solving the discrimination task. (b) Reconstruction loss, measured by the negative MSE between the speaker's and listener's target representations; values closer to 0 correspond
more informative communication (for ease of visualization, the model trained with Ay = 1, with negative MSE 200 is ignored in this plot). (c) Complexity, measured by the mutual information between the speaker's inputs and communication signals. The red arrow indicates the complexity of English as estimated from the Many dataset. (d) The effective lexicon size of the emergent systems (i.e., the number of signals that are used with non-zero probabilities). The red arrow indicates the number of unique English terms in the ManyNames dataset. (e) Semantic (mis)alignment between the emergent systems and English, measure by Normalized Information Distance (NID). Lower values correspond to better alignment. (d-e) Insets show the top of the simplex with higher resolution, revealing the importance of À in attaining numan--ike lexicon sizes and better alignment with English.
in comparison to the number of English words used in the Many dataset; and (e) the semantic amiinanit between the emergent systems and the English naming system, measured by the Nor- malized Information Distance [NID 37j. NID takes into account for full meaning-to-fore mappings and is bounded between [0, 1). with lower values corresponding to better alignment. The measures in (b-e) are evaluated in the semantic setting, which was not used during training, across the full dataset.
As expected, agents trained with large weights on one of the three Às develop communication systems skewed towards one component of the objective, yielding non-human like solutions at the extremes of the À simplex. Next, characterize the landscape of the emergent systems and discuss the human-like tradcoff's that best capture the linguistic behavior of English speakers.
High AD regime. High values of À (bottom left corner) allow agents to achieve very high pragmatic competence (utility > 0.9, Figure 3a), maximizing the listener's success solving the downstream task (i.e., distinguishing the target from the distractor), but result in very poor alignment between the emergent lexicon and the English lexicon (NID values > 0.8, Figure 3e). This finding resonates well with the cognitive science literature evolution of human semantic systems [10], as well as with empirical findings that suggest that humans do not achieve maximal utility in task when restricted to using only lexicalized items [26]. 6 In other words, in this regime, which focuses primarily on utility maximization without information
the role of informational constraints in the
'In our settings, agents can communicate only using lexicalized items. The usage of more complex structures, such syntactic constructions, beyond the scope of the present work and is left for future research.
7

---

constraints, agents are likely to develop communication strategies that do not align with human- like lexical semantics as they are pressured to over-compen for the lack of syntactic constructions.
High Xy regime. High values of 'r (top corner) lead to highly informative communication (negative MSE > -0. -0.20, Figure 3b). However, as expected, this comes at the cost of large lexicon sizes and learn to use thousands of words for the ManyNames domain (> 1.5K vs. 400 topnames used by English speakers for A = 1. Figure 3d), and too complex communication systems (> bits vs, 5 bits in English, Figure 3c). Compared to Ay, high values of x seem to generally favour semantic alignment (NID ~ 0.6, Figure 3c). although is important to note that the best NID not achieved with X 1, but rather with a non-trivial combination of all tradeoffs, as discussed below. Finally, it is noteworthy that highly informative lexical systems also yield high utility (e.g., utility= > 0.9 and negative MSE > -0.18 for Au < 0.3, X > 0.7, À = 0.0). This is in line with findings from Tucker et al. [38]. Indeed, in our framework, utility and informativeness are not in complete competition, but rather capture complementary aspects of successful communication that are only partially aligned.
high complexity. Agents trained with Ar values close
High Ac regime. High values of λo (bottom right corner) encourage minimal complexity and lead to unsuccessful communication, both in terms of utility and informativeness, well to very small lexicon sizes and low semantic alignment. In general, λe seems to act at very small scale, with important effects on the lexicon already perceivable at very small values (see, for instance, the large decrease in lexicon size for λo ~ 0.02 in Figure 3d, as well as the corresponding increase in semantic alignment in Figure 3e).
Human-like tradeoff's (Ai 0). A combination of pressures fosters the emergence of natural solutions. Table 1 summarizes key properties of the landscape that have explored. As expected, the three extremes on the simplex lead to unnatural, non-human-lide solutions. Moving away from the extremes, we identify non-trivial tradeoff, À (Au 0.005, \ 0.98, A 0.015), that achieves the best fit with respect to the English data. We thus refer to this model as Xing. This model achieves the best semantic alignment with English (Figure 3e), and roughly matches the English lexicon size (Figure 3d) and complexity rate (Figure 3c). It also achieves good reconstruction (Figure 3b) and high pragmatic competence (Figure 3a).
These quantitative findings support our predictions and demonstrate how our framework can advance understanding of the co-evolution of semanties and pragmatics. To further understand the communication strategies learned by our agents, we next turn to a qualitative exploration of the emergent communication systems.
5.2 Qualitative evaluation
Figure 4 offers visualization of the agents' communication. We plot dots the visual features (reduced to 2D via PCA) of 500 objects randomly sampled from 3 categories in ManyNames ('woman', 'giraffe', and 'train', i.c. the most frequent names in the semantic domains of 'people', 'animals/plants', and 'vehicles'), ) identified by color. White crosses correspond to the listeners' reconstructions (m,, Figure 2) in the semantic setting, which roughly represent word meanings.
Figure 4a, b, and с illustrate the solutions learnt by the models the edges of our simplex. When trained with A 1 (Fig. 4a), agents are only driven by the task-related utility, i.e., maximizing success in pragmatic interactions. In this scenario, the listener does not learn to reconstruct a mental representation of the object, and the solution lacks a robust, non-contextual semantics. When trained with xy = 1 (Fig. 4b), in order to maximize informativeness, agents learn highly complex solutions, with large lexicons mapping words to small sets of objects, and not identifying human-like categories. When trained with Ac- (Fig. 4c), agents compress their lexicons at the cost of losing important distinctions, and achieve a solution where the same word describes all the objects. This solution does not enable successful communication.
In contrast, our Ain model (Fig. 4d), trained with a tradeoff between utility, complexity, and informativeness, starts approaching a natural solution, learning word meanings that roughly map to the human categories. This solution is simple, yet it allows for informative communication and successful pragmatic interactions. Still, the agents seem to have learnt additional words, capturing spurious distinctions, especially for the peripheral areas of each category cluster, and for the category
8

---

Au =
(b)
2, = 1
graaffe
10-
o0
-2
-30
-30
(C)
c 1
(c)
2.
Eng
grafe
-5
-30
-10
Figure 4: Visualization of the emergent communication systems for different combinations of As (best seed for each). In each plot, the colored dots correspond to 2D PCA of the input features of 500 randomly sampled objects from distinct high-frequency categories the ManyNames dataset ('woman', 'train", "giraffe".. The white crosses correspond to the listener's reconstructions (m,, see Figure given the speaker's communication signal, which roughly captures the meaning of each signal. (a) When agents are trained only with respect to utility, the listener does not learn to reconstruct a mental representation of the object.
(b) When agents are trained only to maximize informativeness, the emergent communication system is highly complex, essentially assigning unique signals to much of the input space. (c) When agents are trained only to minimize complexity, trivial non-informative system emerges that employs only one signal. (d) When agents trained with respect to a non-trivis tradeoff between utility, complexity, and accuracy, more human- like communication system emerge. These systems simple possible while maintaining sufficient utility and informativeness.
'woman' (the most frequent one ManyNames). Besides reasons related the data distribution, our hypothesis is that the agents may have developed human-like categorizations for prototypical members of the human categories (i.e. those with visual features near the center of the category cluster), but also additional, non human-like categories for atypical objects. Indeed, atypical objects are harder to categorize for humans as well, and may not clearly belong to one single semantic category [39, 40]. This hypothesis may also explain why our Xing model has an NID score of 0.55, showing moderate, but not perfect, alignment with human semantics. We gather support for this explanation by computing the NID score only with objects prototypical for their category: on this set of objects, the NID for X decreases to 0.45 (#0.02), suggesting higher alignment sse Appendix for further details.
Conclusion
Words in the human lexicon are associated with non-contextual meanings, well shaped by the local conversational context. this work, we have addressed key open question for language evolution: How can shared lexicon emerge from local context-sensitive interactions? We modeled the pemantis--raggmatics interface by building on framework for information-theoree emergent communication in neural agents. We trained agents to interact in self-play in the presence of shared

---

conversational context, guiding them with combinations of lognitively-motivtte pressures. We then tested their pragmatic competence, as well as the -imannlikeliness of their emergent semantics. By exploring the landscape of emerging artificial languages, we demonstrate that, if trained with pressures for both context-specific utility and general communicative constraints, agents learn systems with key human-like properties and that allow for successful pragmatic interactions. Our findings inform current theories of language evolution, and show that cognitively-motivane optimization principles can facilitate the emergence of human-like communication strategies in neural networks.
7 Limitations
Our work aims to better understand the computational principles that underlie language evolution, in humans and artificial agents, with focus the interface between semantics and pragmatics. While we presented an important first step towards this goal, we were only able to evaluate our model on English data and focused the lexicon only. An important direction for future work is the evaluation of our model on a larger, more diverse set of languages. In addition, our work has focused only on the of lexical items in communication. Therefore, another important direction for future research is to extend our framework to more complex communication structures, such as syntax, morphology, and compositional meaning.
One potential concern about our model is that it employs a pre-trained object classification model [35] to extract visual features. This pre-trained model was trained with classification labels. Thus, one might worry that our agents were implicitly exposed to some linguistic knowledge. We have several reasons believe that this exposure is negligible. First, the classification labels are coarse while we evaluate the agents with respect to fine-grained naming data. Second, our agents are trained in pragmatic setting, whereas the classification labels non-contextual. Third, the majority of our agents develop non-human-like lexical systems, suggesting that the pretrained vision component is not sufficient for alignemnt with English. Having said that, in further work we intend to explore the influence of other types of visual features.
Finally, the NID score achieved by our agents, while encouraging, suggests that even our best- performing agents are not yet fully aligned with (English speaking) humans. Therefore, further research is needed to understand how to close this gap and guide agents toward more human-likk -ike communication systems.
Acknowledgements
This research was partially by supported by grant PID2020-1120022I-00000IC AEI/10. 10000.1039//0 3309//501 1000 1033 from the Ministerio de Ciencia e Innovación and the Agencia Estatal de Investigación (Spain) and the European Union's Horizon 2020 research and innovation programme (grant agreement No. 715154). We thank the COLT group from Universitat Pompeu Fabra for feedback on this work.
References
[1] Herbert P. Grice. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Speech Acts, volume 3, pages 41-58. Academic Press, 1975. URL http: //www. wwww..ull.... wtt.aaacc.kk/ssssseeeee.ss https//stdddypcc///rrieeee
[2] Michael Franke. Signal to act: Game theory in pragmatics. 01 2009.
[3] Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(1 11:81888889) 2016.
[4] Mycal Tucker, Roger P. Levy, Julie Shah, and Noga Zaslavsky. Trading off utility, informative- ness, and complexity in emergent communication. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: / /ttp////pp///eee//////ee/ee/eeee net www..eee/ooummree-5aaaaaaee
[5] Anton Benz and Jon Stevens. Game-theoretic approaches to pragmatics. Annual Review of Linguistics, 4, 02 2018. doi: 10. 0ttp://muuuu--iinnnisiss--00-110
10

---

[6] Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[7] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-oompettive environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages 6382-6393, Red Hook, NY, USA, 2017. Curran Associates Inc.
[8] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings the National Academy of Sciences, 1 11812))e20165561111 2021. doi: 10. 10.1073/pnas. 2tt3:paas.20165666 201656911111 18. URL
https:://ww. wttp:/wwwppp..p..ptt.aat......ap wwwpppan......pppppoooopppoppppp wwrgg/di/abb///... 1073 10aa/pnas...... 2016569118.
[9] Noga 7aslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. PNAS, 1 115(31);7937-794, 2018. ISSN 0027-8424. doi: 10. 10.3/0//p
http://aas..800211111 1800521115. wwwphhttp//wwww.. pnas wwtp://paas...r/oorrooooee orgg//onteet/1//3///1955
[10] Noga Zaslavsky. Information-Theooe Thesis, The Hebrew University of Jerusalem, 2020.
Principles in the Evolution of Semantic Systems. Ph.D.
[11] Noga Zaslavsky, Mora Maldonado, and Jennifer Culbertson. Let's talk (efficiently) about us: Person systems achieve near-optimal compression. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society, 2021.
[12] Francis Mollica, Geoff Bacon, Noga Zaslavsky, Yang Xu, Terry Regier, and Charles Kemp. The forms and meanings grammatical markers support efficient communication. Proceedings of the National Academy of Sciences, 118(49), 2021. ISSN 0027-8424. doi: 10. 10. 2025993118.
[13] Noga Zaslavsky, Karee Garvin, Charles Kemp, Naftali Tishby, and Terry Regier. The evolution of color naming reflects pressure for efficiency: Evidence from the recent past. Journal of Language Evolution, 04 2022. ISSN 2058-458X. Izac001.
[14] Noga Zaslavsky, Jennifer Hu, and Roger Levy. A Rate- Distorriitiin view of human pragmatic rea- soning. 2020. URL /attps://aaitt org/ /205 06641.
[15] Claude E. Shannon. Coding theorems for discrete source with fidelity criterion. IRE Nat. Conv. Rec, 4(142-163):1, 1959.
[16] Toby Berger. Rate distortion theory; a mathematical basis for data compression. Prentice-Han Hall, Englewood Cliffs, NJ, 1971.
[17] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc, of the 37-th Annual Allerton Conference Communication, Control and Computing, pages 368-377, 1999. htLpstp::// //axxiv oww/oogg///oooooooooooooooooeoss http//aapphsscsc////a/a
[18] Carina Silberer, Sina ZarrieB, and Gemma Boleda. Object naming in language and vision: A survey and new dataset. In Proceedings of LREC, pages 5792-5801, Marseille, France, 2020. European Language Resources Association.
[19] Eleanor Rosch and Carolyn B Mervis. Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7(4):573-608 1975. ISSN 0010-0285.
[20] Gregory Murphy. The Big Book of Concepts. MIT Press, 2004.
[21] Stephen C. Levinson. Presumptive Meanings: The Theory of Generalized Conversational Implicature. MIT Press, Cambridge, 2000.
[22] Terry Regier, Charles Kemp, and Paul Kay. Word Meanings across Languages Support Efficient Communication, pages 237-263. 01 2015. ISBN 0ON199880301177 doi: 10. 1100099781118-0000 chll.

---

[23] Charles Kemp, Yang Xu, and Terry Regier. Semantic typology and efficient communication. Annual Review of Linguistics, 128, 01 2018.
[24] Noga Zaslavsky. Terry Regier, Naftali Tishby, and Charles Kemp. Semantic categories of artifacts and animals reflect efficient coding. In Ashok K. Goel, Colleen M. Seifert, and Christian Freksa, editors, Proceedings of the 41th Annual Meeting of the Cognitive Science Sociery, CogSci 2019: Creativity + Cognition Computation, Montreal, Canada, July 24-27, 2019, pages 1254-1260. cogmitiessiiencessciityoooo 2019. URL https: http://mmmmmmdddme/eeeam http///ggcc22000/ppppss////22//9 /nde////indx.../e html.
[25] Caroline Graf, Judith Degen, Robert D. Hawkins, and Noah D. Goodman. Animal, dog. or dalmatian? level of abstraction in nominal referring expressions. Cognitive Science, 2016. URL https://apl htapss//app.eemmttiicchhh..
www.nti
:ete/CorpuII::::0 90667477.
[26] Andreas Mädebach, Torubarova Ekaterina, Eleonora Gualdoni, and Gemma Boleda. Effects of task and visual context on referring expressions using natural scenes. Jn Culhertson, A. Perfors, and V. Rabagliati, H. & Ramenzoni, editors, Proceedings of the 44th Annual Conference the Cognitive Science Society. Cognitive Science Society, 2022.
[27] Julia White, Jesse Mu, and Noah D. Goodman. pragmatic reasoning. CoRR, abs/2006.00418, 2020. URL https://arxiv wwwp:aawww.....raaaaaaaaaaooeera wwtp://../g////////e////ee abs/a/bs/.0.. 00418.
Learning to refer informatively by amortizing
[28] Thomas Brochhagen, Michael Franke, and Robert van Rooij. Coevolution of lexical meaning and pragmatic use. Cognitive Science, 4208)22777- 2018. doi: https:///oi. hrg::////oooooo////// ortg//.11/1111 10.1111/coogs 12681. URL
http://llnnnibrrrrrwrrarraia wiley wawelleyycomdddee coom/doiaabbe 1tt/:/.....11////1 tog1/cogs...... 12681.
[29] Bill McDowell and Noah D. Goodman. Learning from omission. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 619-628, Florence, Italy, July 2019. Association for Computational Linguistics. URL https:/ /tt////wwwww./ aclweb. org/an org/anthology////// httphhooogy/P9---0055500 1P19/19999199
[30] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
[31] Mycal Tucker, Roger P. Levy, Julie Shah, and Noga Zaslavsky. Generalization and translatability in emergent communication via informational constraints. In NeurIPS 2022 Workshop on Information--hhoreet Principles in Cognitive Systems, 2022. URL https: openreview. net/forumiid Bisuuuf.................
[32] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[33] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of CVPR, 2018.
[34] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 39, 06 2015.
[35] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pages 770-778, 2016.
[36] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32 73, 2016. URL. https: / /apppp..mmaiicccoolaaa.... arg/CorpusID:44990
12

---

[37] Nguyen Vinh, Julien Epps, and James Bailey. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. JMLR, 11:2837- 2854, 10 2010.
[38] Mycal Tucker, Roger P. Levy, Julie Shah, and Noga Zaslavsky. Trading off utility, informa- tiveness, and complexity in emergent communication. In RSS Workshop: Social Intelligence in Humans and Robots, 2022. URL httpp:://oocaal-nneellllle hwwaneeeennnnnne--e human ai github. io/ docs/ camready 4. pdf.
[39] Joan Gay Snodgrass and Mary Vanderwart. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of experimental psychology. Human learning and memory, 2:174-215, 1980.
[40] Eleonora Gualdoni, Thomas Brochhagen, Andreas Mädebach, and Gemma Boleda. What's in name? A large-scale computational study on how competition between names affects naming variation. Journal of Memory and Language, 133:104459, 2023.
13

---

A Details about agents' training
We started by training our agents with Au 1 until convergence, i.c. for 10K epochs. After that, we first annealed models by keeping λe fixed at 0, while gradually decreasing the value of A and increasing the value of A,, until reaching X
Then, for each trained value of AU we gradually annealed Àc. For each annealing step, we trained until reaching variance in the training objective lower than 0.0001 for the latest 1K epochs (as a criterion for convergence). We followed non-uniform annealing schedule, re-fined after an initial exploration phase, aimed at identifying regions in the simplex showing interesting patterns with respect to our metries. In these regions, we sampled the combinations of parameters more densely, e.g. with steps of 0.002, while in the other regions we sampled with steps of 0.1. Agents were trained with batch size 128, hyperparameter B of the VQ architecture set at 1. See Tucker et al. [38] for further details about the architecture and hyperparameters.
Experiments were run on cluster with 12 nodes with 5 NVIDIA A30 GPUs and 48 CPUs each. Training the Aσ 1 model took around 30 minutes. Training one annealed model could take up to 15 minutes, often less. Computing evaluation metrics took total of 10 hours. We estimate the overall time required to run this analysis to be around 4 days. Considering our exploration phase and failed experiments, we estimate the total runtime required by this paper to have been around 20 days.
B Identifying prototypical objects
To identify in ManyNames objects that are prototypical members of their categories, we took the following approach: for each topname appearing at least 30 times in ManyNames, selected the 15 most probable images based on the human annotations (in general, visual typicality for a name correlates with name probability [39, 40]). This process resulted in 69 words and total of 1035 images. These images are, at worst, the 50% most typical images for their human name. On this set, Aing achieves NID of 0.45 (=0.0)).
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state research question, and how our findings contribute to answer that question.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not contain theoretical results in terms of theorems and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our code is available info-sem Sero---e------------ prag- tettt-rraggnuiria neurips2024. We describe in detail our model (Section 3 and Appendix A), and reference the original paper where the model's core components were first presented. We provide information about our data, data selection strategies, and models' training.
https: / http:://githbbb com/InfoCogL.ab/
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our code is available at httpps///gttuub..mm///tccoggg// info-sem sem www-ssae--------eeeee neurips2024
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: https/ccccbbcccgcciiceaeeeecpppa Celeee//ooSSSubmeeeeeee ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: see Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: yes, when we report the numerical results reported in Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: see Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: / http///eerrpp/eeeeeeeeeeee cc httppub.ic/ttiiictttoelele...ee.
didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:[Yed [es]
Justification:
No potential harms are identified.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: we consider this paper foundational research, with no risk of malicious use.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not train models at high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: do not release any new dataset. The experiments' code will be released upon acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:[NA]
Justification:
The paper does not involve crowdsourcing
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "ZUU7oKQGvi", "reviewer": "Reviewer_zhL9", "summary": "This paper investigates emergent communication in artificial agents and how properties of the emergent language compares to properties of human language. Specifically, it aims to present a framework to study pragmatic language emergence in systems with different learning objectives, in order to determine constraints that lead to human-like linguistic systems. For training, they use a reference-game setup based in naturalistic images. Based on the closest similarity in complexity, lexicon size and lowest normalized information distance compared to the human baseline, they argue that agents need to optimize for all: utility, informativeness, and complexity.", "questions": "- Could you elaborate on what naturalistic images afford compared to more controllable environments (e.g., more interesting lexicon size?)\n- What is the basis for the claim that utility and informativeness are only partially aligned? (line 245)\n- There is an argument brought forth that the agents are pressured to overcompensate because of the lack of syntax (lines 231-233). Could you elaborate on this a bit more? Why can syntax not emerge and wouldn't that then be a fundamental constraint on the comparability of the system to human language (especially when it comes to lexicon size as a metric)?\n- In the utility-only condition, the agents learn to solve the task well, but the authors argue that the listener doesn't learn to reconstruct robust and non-contextual semantics. How is it possible that the model still generalizes so well?\n- How do the pragmatic and semantic condition map onto the results? I understood the paper to say that the agents were trained in the pragmatic setting where it's the listener's goal to pick out the intended target based on two options. The utility evaluation seems to measure the listener's success to pick out the correct target. What is the semantic setup used for?", "limitations": "What questions does this leave unanswered? What simplifying assumptions are being made about language learning (no syntax or social learning)? The English language limitation is certainly true but not particularly meaningfully discussed -- why would we potentially expect variation across languages?", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well thought-through dataset preprocessing and task formulation\n- Relevant and interesting topic for the computational linguistics and computation cognitive science community\n- Thoughtful setup and analyses that are informative for understanding what the agents learn", "weaknesses": "- The paper overall argues for the necessity of a trade-off of different communicative pressures in order to learn a human-like emergent language -- as measured by the system's similarity to human language in complexity, lexicon size and Normalized Information Distance (NID). Based on these parameters, the authors argue that utility, informativeness, and complexity need to be jointly optimized for in order for a human-like language system to emerge. However, the best-performing system (according to the  complexity, lexicon size, and NID metrics) only achieves a final task performance of 72% -- If I understand the task setup correctly, the baseline chance performance is 50% and people should be expected to achieve nearly 100% in performance. The utility-only objective achieves an accuracy of 95% but does so with a larger lexicon size, worse NID and differently looking complexity. So why do the complexity/lexicon size/NID metrics hold so much more value than the actual task performance? **It seems to me that this argument can really just be made when all systems succeed on the task equally and conditioned on that, we can inspect which learning constraints appear to lead to the most human-like system.** And learning constraints that don't lead to task success while others do should already disqualify from further inspection. I find this methodologically problematic and to me this is a fundamental flaw in the argument. (It also severely lacks discussion in the paper.)\n- Minor: The paper writing could generally be improved. Firstly, I find Figure 2 quite confusing as to what the labels mean and when something is masked. Secondly, there are various typos in the paper (e.g., lines 90, 153, 158, 170, 179) and confusing enumeration (line 125-129).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses:**\n\n1. Major: The reviewer assumed that “people should be expected to achieve nearly 100% in performance” and based on that, argued that we should disqualify systems that do not achieve near-perfect task utility. We would like to clarify that the reviewer’s claim is inconsistent with prior work showing empirical evidence that **people cannot achieve near-perfect performance in our task** (see additional comment for detail). This also makes sense intuitively: our study focuses on the lexicon, and therefore our task focuses on the use of **lexical items** (i.e., single words/signals). Humans cannot achieve near-perfect performance with lexical items, but rather need to employ syntactic constructions for that. This stems from the ambiguity of the human lexicon, an important property that is believed to facilitate efficient communication [e.g., 1; 2]. Therefore, we find it encouraging that our approach has identified a regime in which human-like behavior arises both at the lexical level — as captured by NID, lexicon size, and informational complexity — as well as at the pragmatic level, as captured by the (expectedly bounded) task performance. We briefly explained this in lines 226-233 in our submission, but thanks to the reviewer’s thoughtful comments we realize that we should elaborate more on this point and explain it more clearly. We will certainly do this for the final version.\n\n   Re the reviewer’s question on task performance vs other metrics: due to space limitation, we addressed this in the additional comment.\n\n2. Minor: we will easily fix the typos and confusing enumeration. Furthermore, we will do a thorough English proofreading of the entire paper. The reviewer also noted that the labels in Fig2 are confusing, but they didn’t explain what is confusing about these labels (they seem rather intuitive to us). We therefore ask the reviewer for a clarification on this potential issue, to allow us to fully address all the concerns.\n\n\n**Questions:**\n\n1. Our chosen dataset of naturalistic images has several advantages: (a) richer lexicons, as the reviewer suggested; (b) using images of naturalistic scenes is particularly crucial in our case, because the structure of the emergent communication is sensitive to the statistics of co-occurring objects in the environment; (c) controllable environments have been considered in prior work with the utility-complexity-informativeness tradeoff [3], we thus extend this framework to more naturalistic settings;\n\n2. Optimizing utility and optimizing informativeness are only partially aligned because they are different objectives that can lead to different solutions, but at the same time, they don’t necessarily compete and can sometimes support each other. For example, high informativeness can facilitate high utility, as seen in Table 1 ($\\lambda_I = 1$). The intuition for this is that if the listener can accurately reconstruct the speaker’s representation of the target object, then it could perform reasonably well in many downstream tasks. At the same time, optimizing utility alone can be achieved with very poor feature reconstruction (i.e., low informativeness), as can be seen in Table 1 ($\\lambda_U = 1$) and Figure 4a. The intuition for this is that when there is no pressure for informativeness, the listener can learn arbitrary representation spaces (e.g., random projections) that don’t resemble what the speaker has in mind. While this may yield good task performance for in-distribution inputs, it’s unlikely to generalize well to out-of-distribution (see [4] for preliminary results).\n\n3. As noted above, our work focuses on lexical systems, i.e., the meanings of single words, and not syntax. We therefore allow our speaker agents to send only one word/signal per interaction. Respectively, the English ManyNames data that we use is also based on single-word responses from people. Our measures of complexity — the lexicon size and the informational complexity term — are both measures of lexical complexity, not syntactic complexity. The reviewer is correct that syntax is also an important feature of language; it is beyond the scope of this work and we will  add a note on this matter in the limitation section.\n\n4. This is best illustrated in Fig. 4a, which shows that in the utility-only condition the listener’s reconstructions are misaligned with the speaker’s representations of the target visual features, in contrast to other solutions (Fig. 4b and Fig. 4d) where the listener is better able to recover the speaker’s representations. We believe that the listener achieves high task performance in the utility-only condition because it learns random projections instead of useful reconstructions. The random representations could perform well for in-distribution inputs, but they are unlikely to generalize well to out-of-distribution inputs (see [4] for preliminary results).\n\n5. The semantic setup is designed to recapitulate the human naming experiment with our artificial agents. In a naming experiment (such as the ManyNames experiment), participants are shown a single object and are asked to name it (using a single word). In contrast to the pragmatic task, there is no distractor and no downstream task. We use this setup to evaluate the emergent lexical systems with respect to the human naming data. NID, lexicon size, and complexity are all based on this setup.\n\n**Limitations:**\n\nIndeed, as the reviewer noted, this work doesn’t address the evolution of syntax. We focus on the evolution of the lexicon, which provides the building-blocks for syntax, and it is far from being well understood. We agree that this is a simplifying assumption and adding syntax is an important direction for future work. We will add a discussion on this in the limitations section.\n\nAs for the question about linguistic variation: it is well documented in the literature that languages vary widely in the ways they structure the environment into words, see for example [5-7].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TjSZSa0Prp", "reviewer": "Reviewer_Cm9N", "summary": "This paper looks at aligning emergent communication with human language through the joint optimization of the utility, informativeness, and complexity of a communication channel.\nThis is done in the context of a \"pragmatic\" signalling game where a speaker agent must refer to an object in an image contrasting it with a distractor object in the image.\nThe resulting emergent communication protocol is evaluated in terms of its lexical semantics such as lexicon complexity and size.", "questions": "Although I think the contributions of this paper are worthwhile, I think it is somewhat caught between different framings.  From the introduction (and title), I got the sense that the paper was about semantics and pragmatics, but from the rest of the paper I got more the sense that the paper was about validating the informativity-complexity-utility tradeoff using emergent communication techniques with the ManyNames dataset.\nI think if the framing of pragmatic/semantics is primary, then there need to be more explicit definitions of what semantics and pragmatics is, why the distinction is important, and what the empirical investigation is telling us about it; I understand the difference between the \"pragmatic\" and \"semantic\" goals in the environment, but I do not get an explicit sense of what the relationship is between these two and why it is important. Also, some of the framing seems to make pragmatics seem wholly independent of semantics when really pragmatics is more a pressure on semantics.\nOn the other hand, if the framing is to be more consistent with the empirical work as it stands currently, I think there ought to be more talk of why the IUC-tradeoff is important/useful both generally and specifically with relation to the experiments.\nAddressing this issue adequately would likely lead to me raising my Rating/Presentation score.\n\n\n### Comments\n- Figure 2: \"Agent are trained\" -> \"Agents are trained\"\n- Line 156: \"in its nature to\" -> \"in it nature\"\n- Not a requirement at all but it would be informative to compare the EC results to the recently released Mandarin Chinese data in ManyNames (assuming the data is already in the right format for analysis).", "limitations": "N/A.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "In conjunction with standard criteria, there are three characteristics that are particularly important for emergent communication research: reusability (how easily can another researcher use the products of this research), generalizability (how much do the findings of this research apply broadly to our knowledge of emergent communication), and directedness (does this research contribute concretely to particular questions in emergent communication research).\n\n### Quality\n- (major) The experiments are reasonably designed and demonstrate informative results.\n### Clarity\n- (major) The experiments are easy to follow and are accompanied with informative visualizations.\n### Reusability\n- Nothing of note.\n### Generalizability\n- (major) The UIC framework utilized in this work is a general way to think about emergent communication systems with support in psycholinguistic theories.  As a result, providing evidence of the utility of this framework as well as an example of how to use it makes such a framework more effective down the road for for emergent communication research.\n### Directedness\n- (major) As mentioned above, the UIC framework makes progress towards multiple goals within emergent communication research: developing more formal models to describe system behavior as well as aligning emergent language agents (and the languages themselves) with human languages behavior.", "weaknesses": "### Quality\n- Nothing of note.\n### Clarity\n- (major) It is difficult to tell if the paper is primarily about relationship between semantics and pragmatics in a given EC system or about empirically verifying the utility-complexity-informativeness framework.\n### Reusability\n- (major) The code is not available to be reviewed.\n- (major) Although the code is available for VQ-VIB, the details on how the code (or some other codebase) was adapted to this paper are sparse.\n### Generalizability\n- (minor) There are not many details on how the UCI framework could be applied to other emergent communication systems or questions.  I think the framework is already pretty general, so it would be too difficult to expand on this for a paragraph or two.\n### Directedness\n- Nothing of note.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Clarity + Questions (major)**\n\nThe main goal of the paper is to study the interface between semantics and pragmatics, and specifically, to address the question: how can a shared human-like lexicon emerge from task-specific, context-sensitive pragmatic interactions? As the reviewer noted in the “Questions” section of their review, this framing is clear in the title and introduction (we also believe it is clear in the abstract, related work, and conclusions), but less so in other parts of the paper. We thank the reviewer for noticing this, and believe that this shortcoming is due to our focus on the technical details in the more technical parts of the paper, such as the experimental setup and results.\n\nTo address this issue, the reviewer suggested that “there need to be more explicit definitions of what semantics and pragmatics is, why the distinction is important, and what the empirical investigation is telling us about it”. We agree and are grateful for this suggestion. Here we address the reviewer’s questions, and in the global rebuttal we explain how we will revise the paper to address this concern.\n\nThe term **lexical semantics** refers to the study of word meanings [1, 2], independently of any specific conversational context. An example we mention in the paper (lines 23-25) is that we have a shared idea of what the word ‘player’ means, regardless of any context or scene in which this word might appear. The structure and evolution of word meanings has been an active research area in linguistics and cognitive science for several decades [3-5], and it has recently been characterized using the information-bottleneck (IB) complexity-informativeness tradeoff  [6, 7]. However, the vast majority of studies in this area do not take into account how words are used in context, which can alter their meaning.\n\nThe term **pragmatics** refers to the study of how the local conversational context may alter meaning in real-time, as speakers and listeners reason about each other’s intentions and beliefs [8, 9]. For example, a speaker may choose either ‘player’ or ‘batter’ to describe the target person in the red bounding box in Fig. 1. In scenes with only one player, the more frequent word ‘player’ may be easier to produce, but in scenes with more than one player this word would not be informative about the speaker’s intended meaning. Therefore, if the speaker takes into account how a rational listener would interpret its words, then in the scene shown in Fig. 1 the speaker should use the word ‘batter’ instead of ‘player’. This type of reasoning, called pragmatic reasoning, has been widely studied both experimentally and with computational models that typically use tools from Bayesian inference and game theory [e.g., 9-11], that emphasizes only utility maximization. However, the vast majority of studies in this area assume that the lexicon is given and fixed.\n\nThe **distinction** between semantics and pragmatics is important for several reasons: First, they capture two different aspects of language: semantics captures how meaning is structured into words regardless of context, while pragmatics captures the context-sensitive meaning. Second, they capture two different time-scales: word meanings are stable building-blocks of language that change relatively slowly, while pragmatic reasoning applies in real-time, everyday language use. Finally, these two aspects of linguistic meaning constitute two subfields of linguistics, and therefore much of the relevant literature, including in computational linguistics and cognitive science, is organized around this distinction. \n\nAt the same time, it is widely agreed that semantics and pragmatics are related, as the reviewer noted as well; however the interface between them has been largely underexplored. As noted above, these two aspects of language are typically studied independently (but see our related work section for a few exceptions). As we state in the abstract, **our computational and empirical investigation aims to bridge this gap in the literature**, by providing a unified computational framework for studying semantics and pragmatics simultaneously. To this end, we extend a framework that integrates the IB complexity-informativeness tradeoff that characterizes lexical semantics with utility maximization that has been used to model pragmatic reasoning. In addition, in our setup, we do not assume that the lexicon is given a-priori, in contrast to most models of pragmatics, but rather investigate how it may naturally emerge from context-sensitive interactions. Our results show that in order to understand how a human-like lexicon may emerge from pragmatic interaction, it is crucial to consider all three terms in our objective function: utility, informativeness, and complexity.\n\n\n**Reusability (major)**\n\nTo address the reviewer’s concern about the availability of our code, we made our code available by providing a link to an anonymized repository. Following the NeurIPS instructions we received, this link was posted in a separate comment to the AC. We included in the repository a demo notebook, useful for exploring the ManyNames dataset and identifying target and distractor objects in the images, which we hope will facilitate further extensions of our work.  \n\n\n**Generalizability (minor)**\n\nWe completely agree with the reviewer that our framework is general and could be used to study additional environments and open questions in the literature. We will happily add a short discussion on this in the paper.\n\n\n**Comments (minor)**\n\nThanks for noting the typos, we will certainly fix them. We are also very excited about considering the Mandarin Chinese data in ManyNames in future work! We would have loved to do so in this paper, but given the scope of a single NeurIPS paper, we won’t be able to do justice to a cross-linguistic study in addition to our current set of results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iS9pRlnWE0", "reviewer": "Reviewer_F8cg", "summary": "This paper addresses the co-evolution of pragmatics, which must be interpreted according to context, and lexical semantics, the common meaning independent of context. In this paper neural agents are trained in the pragmatic setting to select the target object from two objects, and evaluated in the semantic setting to reconstruct mental representation of the target.\n\nThe authors used the vector quantize variation information bottleneck (VQ-VIB) to evaluate the trade-off of utility, informativeness, and complexity. The results showed that human-like outcomes emerged when not solely pursuing one aspect among utility, informativeness, and complexity, but rather making a non-trivial trade-off among them.", "questions": "- What is the significance of training in a pragmatic setting and testing in a semantic setting?\n- Since the training loss includes utility, informativeness, and complexity, isn't semantics considered during training?", "limitations": "n/a", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea of learning emergent communication in a pragmatic setting and evaluating it in a semantic setting is intriguing and meaningful.\n\nThis paper effectively conducts experiments on naturalistic images. Through quantitative and qualitative evaluation, the authors show that human-like lexicon emerges when optimizing both context-specific and general objectives.", "weaknesses": "The explanation for the significance of the scenario that training in a pragmatic setting and testing in a semantic setting is insufficient.\nThe paper's contributions are somewhat unclear. Except for the point that the communication model is trained in a pragmatic setting, it is similar to [1] and [2]. It uses VQ-VIB as presented in [1], and the overall communication model is similar to [2].\n\nThe novelty is somewhat unclear in the claim that optimizing both task-specific and general objectives yields human-like results. Although there is a difference in that emergent communication is learned in a pragmatic setting, the trade-off between task-specific utility and general information is already presented in [1].\n\n[1] Tucker, Mycal, R. Levy, J. Shah, and Noga Zaslavsky. 2022. “Trading off Utility, Informativeness, and Complexity in Emergent Communication.” *Advances in Neural Information Processing Systems*.\n\n[2] Tucker, Mycal, Roger P. Levy, Julie Shah, and Noga Zaslavsky. 2022. “Generalization and Translatability in Emergent Communication via Informational Constraints.” https://openreview.net/pdf?id=yf8suFtNZ5v.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The explanation for the significance of the scenario that training in a pragmatic setting and testing in a semantic setting is insufficient.\n\n> What is the significance of training in a pragmatic setting and testing in a semantic setting?\n\nWe believe that this concern is somewhat related to the framing concern of reviewer Cm9N. As we explain in the related work section, and in our response to Cm9N, semantics and pragmatics are two key aspects of linguistic meaning that have so far been mostly studied separately from each other. While they are related, the interface between them has been largely under-explored and is not well understood. Most computational models of pragmatics assume a given lexicon, and most of the work of lexical semantics ignores the context-sensitive aspect of meaning that is captured by pragmatics. As we note in the abstract: “we aim to bridge this gap by studying how a shared lexicon may emerge from local pragmatic interactions.”\n\nWe believe that clarifying the framing throughout the paper, as we explain in the global rebuttal, will address this concern.\n\n> The paper's contributions are somewhat unclear.\n\nOur contributions are stated in the abstract (lines 9-15), introduction (lines 68-75), and conclusion section (lines 294-304). We would appreciate it if the reviewer could explain what is unclear about these lines in the paper, so that we could address this concern. Otherwise, this comment by the reviewer is too vague.\n\n> Except for the point that the communication model is trained in a pragmatic setting, it is similar to [1] and [2]. It uses VQ-VIB as presented in [1], and the overall communication model is similar to [2].\n\nWhile our model builds on the work by Tucker et al. that the reviewer mentioned, we extend that work in several important ways. First, not only is our training procedure different, but also our communication model is different. To see this, compare Fig. 2 in our paper with Fig. 1 in each one of the papers the reviewer cites. The evident differences are crucial for our work: we adjust the model to incorporate a shared context for the speaker and listener and implement a masking mechanism that allows us to invoke the model in two different ways (pragmatics and semantics settings). Second, in contrast to Tucker et al., who considered a discrimination task between whole images, our task focuses on discrimination between co-occurring objects within single images. This is crucial in our case, as we aim to capture communication in naturalistic scenes. Finally, our work addresses a major open question that has not been addressed by Tucker et al. That is, we focus on the under-explored interface between semantics and pragmatics, while Tucker et al. have focused on establishing the utility-informativeness-complexity tradeoff, and on the generalization and translatability afforded by this new framework.\nThanks to the reviewer’s comment, we have noticed that these important differences were not stated clearly enough in the related work section. We will adjust that section accordingly to address the reviewer’s point.\n\n> Since the training loss includes utility, informativeness, and complexity, isn't semantics considered during training?\n\nThe key point in this context is that semantics is not built into the agents and we do not assume that a shared lexicon between the speaker and listener is given a-priori. This is in contrast to the vast majority of studies in computational pragmatics that assume a given lexicon (for review see: Goodman, & Frank, (2016). Pragmatic language interpretation as probabilistic inference. *Trends in Cognitive Sciences*). Semantics is considered during training only implicitly, in the sense that we incorporate in our training objective a general optimality principle, the information bottleneck, that is believed to guide the evolution of human semantic systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "brdBkUluoX", "reviewer": "Reviewer_CiKA", "summary": "This study explores how language, combining meaning and context, might evolve. It examines how a shared vocabulary can emerge from interactions that consider the situation.  By training agents in an unsupervised fashion to consider both context-specific utility and general communication pressures, the research reveals these aspects are key for understanding language evolvement.", "questions": "-  The paper's approach is interesting.  To just confirm my understanding: in this specific setting, pragmatic representations will be always more specific (like hyponyms) than semantic representations, rather than dealing with connotations. This seems similar to the challenge of finding the most accurate hyponyms to describe something. Is my understanding correct?\n\n- What do you think would be the potential real-world applications of this work?", "limitations": "The paper addressed limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Pragmatics is under-explored in NLP, but it is an important topic to study and model language. This work is an novel and interesting effort towards that direction. \n- The approach seems reasonable, and the evaluation shows the potential of using cognitively-inspired optimization principles for the communication strategies akin to human interaction.", "weaknesses": "- I don't see major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and encouraging feedback!\n\nTo address the reviewer’s questions:\n\n> The paper's approach is interesting. To just confirm my understanding: in this specific setting, pragmatic representations will be always more specific (like hyponyms) than semantic representations, rather than dealing with connotations. This seems similar to the challenge of finding the most accurate hyponyms to describe something. Is my understanding correct?\n\nThis is partially correct. While agents could learn the strategy described by the reviewer, they may also learn a mix of strategies that humans employ. For example, a related phenomenon is known as scalar implicatures, in which a weaker characterization, such as ‘dog’, may implicitly rule out references that correspond to stronger characterizations, such as ‘Pomeranian’ (imagine a scene with a Labrador and a Pomeranian; both are types of dogs but Labrador is considered a more typical dog compared to Pomeranian). Humans also employ strategies with non-hierarchical alternatives (Silberer et al., 2020). That is, different conceptualizations for the same object can be produced by speakers to highlight different aspects of the referent. For example, we can call someone “teacher” irrespectively of any context, but also refer to them as “tennis player” when the context requires us to do so, even though “teacher” and “tennis player” are not in a hypernym-hyponym relationship. Our agents may also converge on solutions of this kind. \n\n> “What do you think would be the potential real-world applications of this work?”\n\n\nWhile our work focuses on the science of intelligence, rather than on engineering AI applications, we see at least one important direction in which our work could potentially lead to improvement in real-world applications. Recall that our agents are trained without any human-generated linguistic data. This stands in sharp contrast to LLMs that require massive amounts of training data, which is unavailable in low-resource languages. Therefore, our work could help understand how to leverage cognitively-motivated optimization principles for training more data-efficient language models, with the hope of advancing more equitable language technologies. We thank the reviewer for raising this question and we will address this point in the paper.\n\n\n**References**\n\nSilberer, C., Zarrieß, S., and Boleda, G. (2020). Object naming in language and vision: A survey and a new dataset. In *Proceedings of the 12th Language Resources and Evaluation Conference*, pages 5792–5801, Marseille, France. European Language Resources Association.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their helpful and thoughtful comments. We are excited that the reviews are generally positive and in favor of publication. As we explain in our detailed response to each reviewer and in the summary below, we believe that all the concerns raised by the reviewers can be easily addressed with minor revisions to the paper that will clarify a few points about the framing and significance of our work.

Below is a summary of our responses to the key points raised in the reviews, as well as the revisions we intend to implement to address all the reviewers’ concerns.

**1. zhL9’s major concern**

We believe that the major concern raised by zhL9, which seems to be the grounds for this reviewer’s negative rating, is based on a misinterpretation of our work and does not take into account prior work on how humans would perform in our task. Specifically, the reviewer assumed that humans perform near-perfectly in our task, without providing any references to support that claim, and based on that, argued that we should have excluded systems that do not solve the discrimination task perfectly. However, as we explain in our detailed response to zhL9, the reviewer’s assumption is unsupported by prior work that provides empirical evidence in direct support of our findings. That is, humans cannot solve the task perfectly, similar to our model that is best aligned with the English lexicon. Therefore, the fact that we are able to identify, in a principled way, a regime in which human-like behavior emerges both at the lexical semantics level — as captured by the resemblance between the emergent lexicon and the English lexicon in our domain — and at the pragmatics level — as captured by the bounded task performance — is not a design flaw but rather a major contribution of our work.

We suspect that this confusion may have stemmed from misinterpreting our work as focusing on language as a whole, including syntax, rather than focusing specifically on the semantics-pragmatics interface as highlighted throughout the paper (including in the title and abstract).

We hope that we addressed zhL9’s major concern, both here and in our detailed response to the reviewer. We greatly appreciate zhL9’s comments and feel that clarifying this point would improve the presentation of the paper. Specifically, we intend to address this issue by:

(a) Further elaborating on the empirical evidence suggesting that humans do not solve our task perfectly.

(b) Clarifying that achieving bounded task performance is a desired (human-like) property for lexical systems (this is not the case for syntax, but that is not within the scope of our work).

(c) Clarify that our work does not consider syntax, and discuss the important extension to syntax in the limitations section of the paper.


**2. Clarity of framing**

Reviewers Cm9N and F8cg raised related concerns about the framing of the paper. Specifically, Cm9N noted that while the title and introduction capture the correct framing, other parts of the paper seem to reflect a somewhat different framing. As explained in our response, we believe that the abstract, conclusions, and related work sections also capture the correct framing, but we agree with the reviewer that the framing in the more technical parts of the paper could be improved. The reviewer also offered a constructive suggestion for addressing this issue, by elaborating on what semantics and pragmatics are, why their distrintinction is important, and how our work is significant in this context. We are grateful for this suggestion and will address it along the lines of our detailed response to Cm9N, as follows: 

(a) Extend the first paragraph of the introduction to clarify what we mean by semantics and pragmatics.

(b) Adjust the related work section to clarify the distinction between semantics and pragmatics, in addition to the significance of bridging them.

(c) Revise the technical sections to ensure that they are also contextualized w.r.t. the correct framing of the paper.

Relatedly, F8cg wondered about the significance of training in a pragmatic setting and testing in a semantic setting. We believe that the revisions described above will address this concern as well, as we explain in our detailed response to F8cg.

**3. Differentiation from prior work**

F8cg raised a concern that our work uses the same communication model from Tucker et al. (“Generalization and Translatability in Emergent Communication via Informational Constraints.”). As we explain in our detailed response to F8cg, this is factually inaccurate and there are three key differences between our model and the Tucker et al model. Most importantly, our new setting is designed to address a major open question at the interface between semantics and pragmatics, which Tucker et al. have not considered and their communication model would not support. Having said that, we agree with the reviewer that these key differences w.r.t. Tucker et al. were not conveyed clearly enough in the paper. To address this issue, we will clarify this point in the related work section, around lines 113-117, where we discuss the prior work by Tucker et al.


**4. Other questions and concerns**

- Cm9N’s reusability concern: to fully address this concern, we have provided an anonymized link to our code (in a comment to the AC, following the NeurIPS instructions).


- We will fix all the typos and will do a thorough English proofreading of the entire manuscript.

- The reviewers also raised several valuable questions. We intend to incorporate our clarifications in response to these questions in the revised version of the paper.

Meta-review: This paper on emergent communication systems explores the co-evolution of pragmatics and lexical semantics. While it presents interesting methodology and results, it is somewhat borderline. The most expert reviewer raised significant concerns about the trade-off between pragmatic objectives and task performance, questioning the validity of the main argument. Despite this concern, which I assess the authors have responded satisfactorily to and to which the reviewer did not completely respond to, and given the paper's potential contributions to the field and the positive aspects noted by this reviewer and by other reviewers, I recommend acceptance. The authors should address the raised issues in the camera-ready version to strengthen the paper.
Decision: Accept (poster)