Bridging semantics and pragmatics in information-theoretic emergent communication

Authors: eleonora  gualdoni
apple  mlr
mycal  tucker
roger p levy
noga  zaslavsky
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:17:06

Abstract: Human languages support both semantic categorization and local pragmatic interactions that require context-sensitive reasoning about meaning. While semantics and pragmatics are two fundamental aspects of language, they are typically studied independently and their co-evolution is largely under-explored. Here, we aim to bridge this gap by studying how a shared lexicon may emerge from local pragmatic interactions. To this end, we extend a recent information-theoretic framework for emergent communication in artificial agents, which integrates utility maximization, associated with pragmatics, with general communicative constraints that are believed to shape human semantic systems. Specifically, we show how to adapt this framework to train agents via unsupervised pragmatic interactions, and then evaluate their emergent lexical semantics. We test this approach in a rich visual domain of naturalistic images, and find that key human-like properties of the lexicon emerge when agents are guided by both context-specific utility and general communicative pressures, suggesting that both aspects are crucial for understanding how language may evolve in humans and in artificial agents.

Main paper: 
 introduction: 
 Languages evolve through repeated interactions in rich contexts, where various communicative and non-communicative goals co-exist. The conveyed meaning is often shaped by the local conversational context of utterances (Figure 1 ), as captured by the pragmatic behavior of interlocutors [1] . For example, the word 'player' can be interpreted as a baseball batter, catcher, or a guitar player, depending on the conversational context that shapes the listener's beliefs about the speaker's state of mind. That is, understanding meaning in context requires pragmatic reasoning about other agents' intentions and beliefs [1] [2] [3] . At the same time, words are associated with non-contextualized meanings, as captured by lexical semantics. For example, we have a shared idea of what 'player' means, regardless of any specific context in which this word might appear, and we can use it to communicate with new conversational partners in new contexts. While semantics and pragmatics are widely studied, their interface and co-evolution is largely under-explored and not well understood. In this work, we focus on a key open question at the interface of lexical semantics and pragmatic reasoning: How can a shared human-like lexicon emerge from local context-sensitive pragmatic interactions?To address this question, we build on a framework for information-theoretic emergent communication in multi-agent reinforcement learning systems [4] . This framework is particularly relevant for addressing our question because it integrates task-specific utility maximization -a central component in well-established models of human pragmatic reasoning [2, 3, 5] , which has also been center-stage in the literature on emergent communication in artificial agents [6] [7] [8] -with general informationtheoretic constraints that are believed to shape human lexical semantic systems [9] [10] [11] [12] [13] and have Figure 1 : An example of an image from the ManyNames dataset illustrating both semantic and pragmatic communication. Top (semantic setting): A speaker communicates a target object x t (red box) by naming it regardless of the context induced by the image. The ManyNames dataset includes such responses from native English speakers (in this example, 10 speakers produced 'man', 10 'batter', 5 'baseball player', 4 'player', and 3 'person'). Bottom (pragmatic setting): A speaker and a listener observe two objects in an image (red and blue boxes). Only the speaker knows which one is the target x t and which one is the distractor x d . The speaker's goal is to communicate x t given this shared context, and the listener's goal is to discriminate the target from the distractor. recently also been applied to human pragmatic reasoning [14] . These information-theoretic constraints are derived from rate-distortion theory [15, 16] , and more specifically, the information bottleneck (IB) principle [17] , characterizing semantic systems in terms of efficient compression of meanings into words by optimizing the IB tradeoff between the complexity and informativeness of the lexicon [9] . Tucker et al. [4] developed a framework for training agents in emergent communication settings by integrating utility maximization with the IB objective, yielding a multi-objective optimization problem that trades off maximizing task-specific utilities with maximizing task-agnostic communicative informativeness and minimizing communicative complexity.Here, we extend that framework to explicitly model the co-evolution of semantics and pragmatics. In our setup, artificial neural agents learn to communicate in a pragmatic setting, i.e., in the presence of a shared conversational context that may alter the meaning of their communication signals (illustrated intuitively in the bottom image of Figure 1 ). Importantly, agents are trained via self-play, without any supervision or exposure to human languages. At test time, we assess the pragmatic competence (i.e., utility) of the agents as well as the 'human-likeness' of the shared emergent lexicon that they converged on, by invoking it in a lexical semantic setting (illustrated intuitively in the top image of Figure 1 , and see Figure 2 for a full description of our model's architecture). Given the substantial empirical evidence that human semantic systems are pressured to optimize the IB tradeoff, we predict that a shared human-like lexicon will not emerge when agents are guided solely by utility maximization but rather when they are guided by a non-trivial tradeoff between optimizing utility, informativeness, and complexity. Our goal in this work is to test this prediction, and more generally, to characterize the landscape of emergent communication systems induced by our novel information-theoretic framework for the co-evolution of semantics and pragmatics.To this end, we consider a rich visual domain of naturalistic images provided by the ManyNames dataset [18] , which also contains free-naming human data generated by native English speakers. This domain allows us to train agents across many different conversational contexts and then evaluate their emergent lexicon with respect to the English naming data. In support of our prediction, we find that human-like properties of the lexicon (its size, complexity, and alignment with English speakers), together with high pragmatic competence, emerge when agents are guided by both context-specific utility and general communicative pressures as derived from the IB principle. Interestingly, pressure for communicative informativeness, rather than (non-communicative) task utility, appears as the main driver of emergent communication, but weaker pressures to minimize complexity and maximize utility are still crucial for achieving human-like properties of the lexicon.Our work is significant both from a cognitive science perspective and from a machine learning perspective. From a cognitive science perspective, our work provides a novel computational framework for studying the under-explored interface between lexical semantics and pragmatic reasoning, and our findings suggest that human languages may evolve under pressure to optimize a tradeoff between task-specific utilities and general communicative constraints. From a machine learning perspective, our work demonstrates how cognitively-motivated optimization principles, implemented in neural network agents as intrinsic training objectives, can facilitate the emergence of interpretable human-like communication systems without any human supervision. 
related work: 
 Our work integrates cognitively-motivated information-theoretic principles that are believed to underlie human language evolution, with deep learning tools for studying emergent communication in artificial agents, in order to develop a computational account of the co-evolution of lexical semantics and pragmatic reasoning. Semantics and pragmatics constitute two subfield in cognitive linguistics. While both focus on meaning in language, they capture largely complementary aspects of meaning. Lexical semantics is generally concerned with word meanings [19, 20] , independent of any specific conversational context, whereas pragmatics is concerned with language use in context, typically assuming a known shared lexicon [1, 21, 3] . Our work focuses on the underexplored interface between these two aspects of language, departing from the traditional assumption that the lexicon is given a-priori and shared among pragmatic interlocutors.Contemporary cognitive approaches to lexical semantics argue that word meanings are shaped by pressure for efficient communication [22, 23, 10] . Most relevant to our work, is the informationtheoretic framework for semantic systems, proposed by Zaslavsky et al. [9] , that predicts that human semantic systems evolved to optimize the information bottleneck (IB) trade-off [17] between the complexity of the lexicon (roughly, how many bits are allocated for communication) and its informativeness (roughly, how well a listener can understand a speaker, regardless of context). This framework has shown to account for the structure of semantic systems across hundreds of languages and multiple domains [9, 24, 11, 12] , as well as semantic change over time [13] . Here, we leverage this empirically-supported theoretical framework to guide our interactive agents.Research on pragmatic communication focuses on how speakers' lexical choices and listener's interpretations are affected by their local conversational context [1, 3, 25, 26] . Prominent models of pragmatics, such as the Rational Speech Act (RSA) framework [3] , are grounded in game theory (see also [2] ). In this view, pragmatic behavior is formulated within a cooperative reference game, where agents pragmatically reason about each other's communicative intentions with the goal of maximizing the game's utility. While these models enjoy broad empirical support [3] , they assume that the underlying lexicon is given and shared across speakers and listeners, even when applied in reinforcement learning settings [27] . Recently, a theoretical link between the RSA framework and rate-distortion theory (RDT) has been derived [14] , connecting pragmatic reasoning with general informational constraints that are closely related to the aforementioned IB trade-off. However, the RDT approach to pragmatics has also assumed a given lexicon [14] , and more generally, the implied information-theoretic link between semantics and pragmatics has not yet been explored. Our work explored this potential link by relaxing the assumption that the lexicon is given to our agents, and instead studying how they may develop on their own a near-optimal human-like lexicon via a training objective that takes into account informational constraints.Relatedly, Brochhagen et al. [28] developed a game-theoretic model for the evolution of the division of labor between semantics and pragmatics, and tested the model in a relatively small domain. In comparison, we consider here different learning dynamics and objective function, employ state-of-theart deep-learning tools for training agents at scale, and evaluate our approach with respect to actual human data in a rich domain of naturalistic images [18] . McDowell and Goodman [29] considered the role of pragmatics in learning lexical meanings in deep-learning agents. However, they trained their agents in a supervised manner with respect to human-generated pragmatic data. In contrast, we are interested in how pragmatics and lexical semantic may emerge without any human supervision.The emergent communication literature focuses on how agents may learn to communicate in reinforcement learning settings, without exposure to human-generated linguistic data [6, 7, 30] . While this body of literature largely focuses on utility (or reward) maximization, Chaabouni et al. [8] showed that utility-based emergent communication can lead to IB-efficient systems. In their settings, however, different complexity-informativeness tradeoffs can only be determined by external environmental factors. Tucker et al. [4] showed how to directly integrate the IB objective function with utility maximization in emergent communication and demonstrated the advantages of this framework for Agents are trained in a pragmatic setting, where both observe inputs x 0 , x 1 as shared context; one input is randomly selected as target x t for the speaker. After training, the agent's emergent communication systems are evaluated in a lexical semantics setting, without shared context; the speaker observes only x t while x d is masked, and both inputs are masked for the listener.faster convergence rates and better out-of-distribution generalization [31] . However, they did not study the co-emergence of semantics and pragmatics. Our work directly builds on the framework of Tucker et al. [4, 31] and extends it for studying the interface between semantics and pragmatics.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: see Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
a unifying model for the co-evolution of semantics and pragmatics: 
 The information-theoretic framework for emergent communication that we build on [4, 31] consists of an objective function that integrates utility maximization with the IB principle, and a communication model that specifies the agents' architectures. Our model (Figure 2 ) builds on Tucker et al. [4] 's proposed method, called vector-quantized variational information bottleneck (VQ-VIB), which we modify in order to be able to model the co-evolution of semantic and pragmatics; that is, to be able to train agents in a pragmatic setting and evaluate their communication, at test time, in both pragmatic and lexical semantic settings. The initial VQ-VIB architecture includes a speaker and a listener tasked to jointly solve a reference game. In our case, the speaker is defined by (i) a pre-trained variational autoencoder [VAE, 32] that provides a visual representation module for mapping an input x to a 'mental' representation m; and (ii) an encoder module S that generates a communication signal w given the speaker's mental state. The listener is defined by (i) a decoder D that observes w and generates a reconstruction m, and (ii) a policy L for solving a downstream task.In our pragmatic setting both agents observe a shared context (x 0 , x 1 ), while the speaker also observes which referent is the target t and which is a distractor d (more details about this step are provided in Section 4). The speaker then aims to communicate the target x t . The VAE representation model is applied to x t and x d independently, generating m t , the speaker's mental representations for the target, and m d , the speaker's mental representations for the distractor. The listener's task is to guess the target based on y = L( m, (x 0 , x 1 )). We wish to emphasize that in contrast to the standard approach in pragmatic models, in our setup the agents are not given a shared lexicon but rather implicitly learn it through their local, context-sensitive pragmatic interactions.Agents are trained by optimizing a tradeoff between maximizing expected utility, maximizing informativeness, and minimizing complexity. The utility term, U (x t , y), is task-specific, and here we take it to be the accuracy of the listener's downstream decisions. Informativeness and complexity are task-agnostic communicative objectives, derived from the Information Bottleneck (IB) framework for semantic systems [9] . In this framework, the speaker's and listener's mental representations, m t and m, are defined as belief states (i.e., probability distributions) over features in environment. Informativeness is related to the distortion between m t and m, defined by the Kullback-Leibler (KL) divergence D[m t ∥ m], such that low expected distortion amounts to high informativeness. Complexity corresponds to the mutual information between speaker's meanings and words, I(m t ; w), which is roughly the number of bits that the agents will need for communication. Following [4, 31] , we optimize a bound on the IB terms for practical reasons, as direct optimization of those terms does not scale. For informativeness, we treat m t and m as the means of the agents' mental distributions and then approximate informativeness by their MSE. For complexity, we consider a known variational bound on the mutual information, denoted here for simplicty by Ĩ [for more details and full derivation of the objective function, see 4]. Overall, the training objective is to maximizeEQUATIONwhere the λs are non-negative tradeoff weights that sum to 1, and f (I) can be seen as a function that extracts two objects (a target and a distractor) from an image.Optimizing only the utility term in the objective function, without any pressure for alignment between the speaker's and listener's representations (i.e., high informativeness), is expected to lead to lexical systems that are biased towards success in the specific training downstream task, and such systems are likely to depart from human lexical systems. On the other hand, maximizing informativeness alone will lead to highly complex and task-agnostic systems, and minimizing complexity alone will lead to no communication. None of these extremes seems human-like, and we therefore expect that human-like systems will emerge when all tradeoff parameters are active (i.e., non-zero).Our semantic setting is designed to evaluate the emergent lexicon at test time, lending a window into how the agents use their words irrespectively of any local context. In this setting, only the target is shown to the speaker (the distractor is masked) and then the listener reconstructs mt based on the speaker's word w, without any additional context or downstream task. This task resembles in its nature the task with which annotations for ManyNames -our dataset of interest [18] , see beloware gathered, i.e. a naming task where annotators were asked to freely produce names for target objects appearing in natural images, identified with bounding boxed. Thus, we can use these naming annotations at test time to evaluate the agents' emergent semantics.4 Experimental setupQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
the manynames domain: 
 We train and test our agents on the ManyNames dataset [18] , which contains 25K naturalistic images (see Figure 1 for example), each with one target object, appearing in a bounding box, annotated with ∼ 36 names provided by English native speakers asked to freely produce a name (one word) to describe the object. 2 The name produced by the majority of the annotators for a target object is called the topname. In the case of Figure 1 , top image, 'man' and 'batter' are equally probable topnames. Objects in the ManyNames dataset are also annotated according to their high-level semantic domain, which can be people, animals-plants, buildings, vehicles, clothing, food, or home. Importantly, our agents only observe the images in the dataset. They are not trained on any of the linguistic labels, which are used only for evaluation.With respect to the choice of the target-distractor pairs for our pragmatic training setup (Figure 1 , bottom image), our intuition is that identifying competing objects appearing in the same image, e.g., a batter and a baseball player in the same field, or a car and a taxi on the same street, instead of using random objects cropped from different images, should encourage the emergence of more natural semantic partitions, since agents will need to create lexical entries to solve naturally occurring ambiguities. For each image in the dataset, we consider the target object highlighted in the ManyNames annotation; and an additional object that we individuate through automatic object detection and a few filtering filtering steps. We used the Bottom-Up object detection model proposed by Anderson et al. [33] , which incorporates a Faster R-CNN architecture [34] for object detection, and a ResNet-101 architecture [35] for feature extraction. Since Anderson et al. [33] 's model is fine-tuned on the VisualGenome image dataset [36] , of which ManyNames images are a sample, we are guaranteed that this model can make good quality predictions on ManyNames. After running the object detector model on our images, we filter the detected bounding boxes keeping only those with Intersection over Union smaller than 0.1 with the ManyNames target, and that did not have size smaller than 10% of the target size. 3 We then computed the similarity between the candidates' visual features -automatically 1 : Evaluation of the model that is best aligned with English, λ * Eng (λ U = 0.005, λ I = 0.98, λ C = 0.015), in comparison with three baselines: λ U = 1, corresponding to utility maximization without any informational constraints, which is the most common objective function in the emergent communication literature; and λ I = 1 and λ C = 1 which correspond to the other two extremes of either maximizing informativeness or minimizing complexity. ∆Compl. and ∆LexSize are the absolute value of the differences between the complexity or lexicon size of the emergent system and English. NID measures the misalignment between the emergent system and English. For all three measures, lower values reflect a better fit to the human data. Utility and MSE correspond to the agent's pragmatic competence and reconstruction error, respectively. Each cell shows the mean value across three random seeds ± SEM.extracted by the ResNet-101 incorporated in Anderson et al. [33] 's model-and the ManyNames target's features, choosing the detected object with the highest similarity value. This final selection step based on visual similarity aims at providing our agents with some challenging cases to solve, like the one shown in Figure 1 , where communication needs to allow the discrimination between two objects from the same semantic category, e.g. a batter and another baseball player.At the end of this procedure, we obtained two objects per image: the ManyNames target; and the additional object detected by us. We used these two objects to train our agents, choosing our target's identity randomly, making sure that in 50% of the cases the target would be the larger object, and in the other 50% the distractor would be larger.We trained 270 agent pairs with a range of combinations of λ U , λ C and λ I . We used parameter annealing which, besides being a computationally efficient alternative to training agents from random initializations, has been shown to capture at least some aspects of the evolution of human semantic systems [9] . We used a pretrained ResNet18 [35] to extract 512 dimensional features from target and distractor objects before passing them to the agents. 4 In all our experiments, we trained agents on 70% of the images (randomly sampled) in self-play, with no human supervision, batch size of 128, and codebook initialized with 3000 trainable communication vectors (see Appendix A for further details). 55 Results 
quantitative evaluation: 
 We report quantitative results for the 5 evaluation metrics shown in Figure 3 , for each set of λs, at test time. The top panel shows the three components of the agents' training objective: (a) the agents' pragmatic competence, as measured by their test-time expected utility on unseen images, in the pragmatic setting; (b) the reconstruction loss between the speaker's and listener's mental representations, measured by the negative MSE (values closer to 0 corresponds to higher informativeness); and (c) the complexity of the emergent lexicon. The bottom panel of Figure 3 shows two measures for assessing the human-likeness of the emergent systems: (d) the lexicon size of the artificial systems, As expected, agents trained with large weights on one of the three λs develop communication systems skewed towards one component of the objective, yielding non-human like solutions at the extremes of the λ simplex. Next, we characterize the landscape of the emergent systems and discuss the human-like tradeoffs that best capture the linguistic behavior of English speakers.High λ U regime. High values of λ U (bottom left corner) allow agents to achieve very high pragmatic competence (utility > 0.9, Figure 3a ), maximizing the listener's success in solving the downstream task (i.e., distinguishing the target from the distractor), but result in very poor alignment between the emergent lexicon and the English lexicon (NID values > 0.8, Figure 3e ). This finding resonates well with the cognitive science literature on the role of informational constraints in the evolution of human semantic systems [10] , as well as with empirical findings that suggest that humans do not achieve maximal utility in our task when restricted to using only lexicalized items [26] . 6 In other words, in this regime, which focuses primarily on utility maximization without information constraints, agents are likely to develop communication strategies that do not align with human-like lexical semantics as they are pressured to over-compensate for the lack of syntactic constructions.High λ I regime. High values of λ I (top corner) lead to highly informative communication (negative MSE > -0.20, Figure 3b ). However, as expected, this comes at the cost of large lexicon sizes and high complexity. Agents trained with λ I values close to 1 learn to use thousands of words for the ManyNames domain (> 1.5K vs. ∼ 400 topnames used by English speakers for λ I = 1, Figure 3d ), and too complex communication systems (> 6 bits vs. ∼ 5 bits in English, Figure 3c ). Compared to λ U , high values of λ I seem to generally favour semantic alignment (NID ∼ 0.6, Figure 3e ), although it is important to note that the best NID is not achieved with λ I = 1, but rather with a non-trivial combination of all tradeoffs, as discussed below. Finally, it is noteworthy that highly informative lexical systems also yield high utility (e.g., utility > 0.9 and negative MSE > -0.18 for λ U < 0.3, λ I > 0.7, λ C = 0.0). This is in line with findings from Tucker et al. [38] . Indeed, in our framework, utility and informativeness are not in complete competition, but rather capture complementary aspects of successful communication that are only partially aligned.High λ C regime. High values of λ C (bottom right corner) encourage minimal complexity and lead to unsuccessful communication, both in terms of utility and informativeness, as well as to very small lexicon sizes and low semantic alignment. In general, λ C seems to act at a very small scale, with important effects on the lexicon already perceivable at very small values (see, for instance, the large decrease in lexicon size for λ C ∼ 0.02 in Figure 3d , as well as the corresponding increase in semantic alignment in Figure 3e ).Human-like tradeoffs (λ * Eng > 0). A combination of pressures fosters the emergence of natural solutions. Table 1 summarizes key properties of the landscape that we have explored. As expected, the three extremes on the simplex lead to unnatural, non-human-like solutions. Moving away from the extremes, we identify a non-trivial tradeoff, λ = (λ U = 0.005, λ I = 0.98, λ C = 0.015), that achieves the best fit with respect to the English data. We thus refer to this model as λ * Eng . This model achieves the best semantic alignment with English (Figure 3e ), and roughly matches the English lexicon size (Figure 3d ) and complexity rate (Figure 3c ). It also achieves good reconstruction (Figure 3b ) and high pragmatic competence (Figure 3a ). These quantitative findings support our predictions and demonstrate how our framework can advance our understanding of the co-evolution of semantics and pragmatics. To further understand the communication strategies learned by our agents, we next turn to a qualitative exploration of the emergent communication systems.Figure 4 offers a visualization of the agents' communication. We plot as dots the visual features (reduced to 2D via PCA) of 500 objects randomly sampled from 3 categories in ManyNames ('woman', 'giraffe', and 'train', i.e. the most frequent names in the semantic domains of 'people', 'animals/plants', and 'vehicles'), identified by color. White crosses correspond to the listeners' reconstructions ( mt , see Figure 2 ) in the semantic setting, which roughly represent word meanings. Figure 4a, b, and c illustrate the solutions learnt by the models at the edges of our simplex. When trained with λ U = 1 (Fig. 4a ), agents are only driven by the task-related utility, i.e., maximizing success in pragmatic interactions. In this scenario, the listener does not learn to reconstruct a mental representation of the object, and the solution lacks a robust, non-contextual semantics. When trained with λ I = 1 (Fig. 4b ), in order to maximize informativeness, agents learn highly complex solutions, with large lexicons mapping words to small sets of objects, and not identifying human-like categories. When trained with λ C = 1 (Fig. 4c ), agents compress their lexicons at the cost of losing important distinctions, and achieve a solution where the same word describes all the objects. This solution does not enable successful communication.In contrast, our λ * Eng model (Fig. 4d ), trained with a tradeoff between utility, complexity, and informativeness, starts approaching a natural solution, learning word meanings that roughly map to the human categories. This solution is simple, yet it allows for informative communication and successful pragmatic interactions. Still, the agents seem to have learnt additional words, capturing spurious distinctions, especially for the peripheral areas of each category cluster, and for the category 'woman' (the most frequent one in ManyNames). Besides reasons related to the data distribution, our hypothesis is that the agents may have developed human-like categorizations for prototypical members of the human categories (i.e. those with visual features near the center of the category cluster), but also additional, non human-like categories for atypical objects. Indeed, atypical objects are harder to categorize for humans as well, and may not clearly belong to one single semantic category [39, 40] . This hypothesis may also explain why our λ * Eng model has an NID score of 0.55, showing moderate, but not perfect, alignment with human semantics. We gather support for this explanation by computing the NID score only with objects prototypical for their category: on this set of objects, the NID for λ * Eng decreases to 0.45 (±0.02), suggesting higher alignment -see Appendix B for further details. 
conclusion: 
 Words in the human lexicon are associated with non-contextual meanings, as well as shaped by the local conversational context. In this work, we have addressed a key open question for language evolution: How can a shared lexicon emerge from local context-sensitive interactions? We modeled the semantics-pragmatics interface by building on a framework for information-theoretic emergent communication in neural agents. We trained agents to interact in self-play in the presence of a shared conversational context, guiding them with combinations of cognitively-motivated pressures. We then tested their pragmatic competence, as well as the human-likeliness of their emergent semantics. By exploring the landscape of emerging artificial languages, we demonstrate that, if trained with pressures for both context-specific utility and general communicative constraints, agents learn systems with key human-like properties and that allow for successful pragmatic interactions. Our findings inform current theories of language evolution, and show that cognitively-motivated optimization principles can facilitate the emergence of human-like communication strategies in neural networks. 
limitations: 
 Our work aims to better understand the computational principles that underlie language evolution, in humans and artificial agents, with focus on the interface between semantics and pragmatics. While we presented an important first step towards this goal, we were only able to evaluate our model on English data and focused on the lexicon only. An important direction for future work is the evaluation of our model on a larger, more diverse set of languages. In addition, our work has focused only on the use of lexical items in communication. Therefore, another important direction for future research is to extend our framework to more complex communication structures, such as syntax, morphology, and compositional meaning.One potential concern about our model is that it employs a pre-trained object classification model [35] to extract visual features. This pre-trained model was trained with classification labels. Thus, one might worry that our agents were implicitly exposed to some linguistic knowledge. We have several reasons to believe that this exposure is negligible. First, the classification labels are coarse while we evaluate the agents with respect to fine-grained naming data. Second, our agents are trained in a pragmatic setting, whereas the classification labels are non-contextual. Third, the majority of our agents develop non-human-like lexical systems, suggesting that the pretrained vision component is not sufficient for alignemnt with English. Having said that, in further work we intend to explore the influence of other types of visual features.Finally, the NID score achieved by our agents, while encouraging, suggests that even our bestperforming agents are not yet fully aligned with (English speaking) humans. Therefore, further research is needed to understand how to close this gap and guide our agents toward more human-like communication systems. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not train models at high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: see Section 4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: we do not release any new dataset. The experiments' code will be released upon acceptance. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We started by training our agents with λ U = 1 until convergence, i.e. for 10K epochs. After that, we first annealed models by keeping λ C fixed at 0, while gradually decreasing the value of λ U and increasing the value of λ I , until reaching λ I = 1. Then, for each trained value of λ U , we gradually annealed λ C . For each annealing step, we trained until reaching variance in the training objective lower than 0.0001 for the latest 1K epochs (as a criterion for convergence). We followed a non-uniform annealing schedule, re-fined after an initial exploration phase, aimed at identifying regions in the simplex showing interesting patterns with respect to our metrics. In these regions, we sampled the combinations of parameters more densely, e.g. with steps of 0.002, while in the other regions we sampled with steps of 0.1. Agents were trained with batch size 128, hyperparameter β of the VQ architecture set at 1. See Tucker et al. [38] for further details about the architecture and hyperparameters.Experiments were run on a cluster with 12 nodes with 5 NVIDIA A30 GPUs and 48 CPUs each. Training the λ U = 1 model took around 30 minutes. Training one annealed model could take up to 15 minutes, often less. Computing evaluation metrics took a total of 10 hours. We estimate the overall time required to run this analysis to be around 4 days. Considering our exploration phase and failed experiments, we estimate the total runtime required by this paper to have been around 20 days.To identify in ManyNames objects that are prototypical members of their categories, we took the following approach: for each topname appearing at least 30 times in ManyNames, we selected the 15 most probable images based on the human annotations (in general, visual typicality for a name correlates with name probability [39, 40] ). This process resulted in 69 words and a total of 1035 images. These images are, at worst, the 50% most typical images for their human name. On this set, λ * Eng achieves NID of 0.45 (±0.02).1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We clearly state our research question, and how our findings contribute to answer that question. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Creative Commons Attribution 4.0 International License.For 2.5K images, typically depicting only one large object as the ManyNames target, we do not find any detected object with the desired properties. Being unable to use these images in the pragmatic setting, we exclude them from our data sample.We extract visual features with a ResNet18 model, that is a standard choice for object classification, and choose not use the visual features from Anderson et al.[33]'s model, which we employ for the object detection phase. This is because this model, being specialized for object detection, produces visual features for objects in bounding boxes that carry some degree of information about the image context. This is a desirable property when performing object detection, but not for our experiments.Our code is available at https://github.com/InfoCogLab/info-sem-prag-neurips2024.In our settings, agents can communicate only using lexicalized items. The usage of more complex structures, such as syntactic constructions, is beyond the scope of the present work and is left for future research. 


Decision: This paper on emergent communication systems explores the co-evolution of pragmatics and lexical semantics. While it presents interesting methodology and results, it is somewhat borderline. The most expert reviewer raised significant concerns about the trade-off between pragmatic objectives and task performance, questioning the validity of the main argument. Despite this concern, which I assess the authors have responded satisfactorily to and to which the reviewer did not completely respond to, and given the paper's potential contributions to the field and the positive aspects noted by this reviewer and by other reviewers, I recommend acceptance. The authors should address the raised issues in the camera-ready version to strengthen the paper.

Reviews: review: paper_summary: paper_summary: This paper investigates emergent communication in artificial agents and how properties of the emergent language compares to properties of human language. Specifically, it aims to present a framework to study pragmatic language emergence in systems with different learning objectives, in order to determine constraints that lead to human-like linguistic systems. For training, they use a reference-game setup based in naturalistic images. Based on the closest similarity in complexity, lexicon size and lowest normalized information distance compared to the human baseline, they argue that agents need to optimize for all: utility, informativeness, and complexity.
strength_weakness: strength_weakness: - The paper overall argues for the necessity of a trade-off of different communicative pressures in order to learn a human-like emergent language -- as measured by the system's similarity to human language in complexity, lexicon size and Normalized Information Distance (NID). Based on these parameters, the authors argue that utility, informativeness, and complexity need to be jointly optimized for in order for a human-like language system to emerge. However, the best-performing system (according to the  complexity, lexicon size, and NID metrics) only achieves a final task performance of 72% -- If I understand the task setup correctly, the baseline chance performance is 50% and people should be expected to achieve nearly 100% in performance. The utility-only objective achieves an accuracy of 95% but does so with a larger lexicon size, worse NID and differently looking complexity. So why do the complexity/lexicon size/NID metrics hold so much more value than the actual task performance? **It seems to me that this argument can really just be made when all systems succeed on the task equally and conditioned on that, we can inspect which learning constraints appear to lead to the most human-like system.** And learning constraints that don't lead to task success while others do should already disqualify from further inspection. I find this methodologically problematic and to me this is a fundamental flaw in the argument. (It also severely lacks discussion in the paper.)
- Minor: The paper writing could generally be improved. Firstly, I find Figure 2 quite confusing as to what the labels mean and when something is masked. Secondly, there are various typos in the paper (e.g., lines 90, 153, 158, 170, 179) and confusing enumeration (line 125-129).
questions: questions: - Could you elaborate on what naturalistic images afford compared to more controllable environments (e.g., more interesting lexicon size?)
- What is the basis for the claim that utility and informativeness are only partially aligned? (line 245)
- There is an argument brought forth that the agents are pressured to overcompensate because of the lack of syntax (lines 231-233). Could you elaborate on this a bit more? Why can syntax not emerge and wouldn't that then be a fundamental constraint on the comparability of the system to human language (especially when it comes to lexicon size as a metric)?
- In the utility-only condition, the agents learn to solve the task well, but the authors argue that the listener doesn't learn to reconstruct robust and non-contextual semantics. How is it possible that the model still generalizes so well?
- How do the pragmatic and semantic condition map onto the results? I understood the paper to say that the agents were trained in the pragmatic setting where it's the listener's goal to pick out the intended target based on two options. The utility evaluation seems to measure the listener's success to pick out the correct target. What is the semantic setup used for?
limitations: limitations: What questions does this leave unanswered? What simplifying assumptions are being made about language learning (no syntax or social learning)? The English language limitation is certainly true but not particularly meaningfully discussed -- why would we potentially expect variation across languages?
paper_summary: paper_summary: This paper looks at aligning emergent communication with human language through the joint optimization of the utility, informativeness, and complexity of a communication channel.
This is done in the context of a "pragmatic" signalling game where a speaker agent must refer to an object in an image contrasting it with a distractor object in the image.
The resulting emergent communication protocol is evaluated in terms of its lexical semantics such as lexicon complexity and size.
strength_weakness: strength_weakness: ### Quality
- Nothing of note.
### Clarity
- (major) It is difficult to tell if the paper is primarily about relationship between semantics and pragmatics in a given EC system or about empirically verifying the utility-complexity-informativeness framework.
### Reusability
- (major) The code is not available to be reviewed.
- (major) Although the code is available for VQ-VIB, the details on how the code (or some other codebase) was adapted to this paper are sparse.
### Generalizability
- (minor) There are not many details on how the UCI framework could be applied to other emergent communication systems or questions.  I think the framework is already pretty general, so it would be too difficult to expand on this for a paragraph or two.
### Directedness
- Nothing of note.
questions: questions: Although I think the contributions of this paper are worthwhile, I think it is somewhat caught between different framings.  From the introduction (and title), I got the sense that the paper was about semantics and pragmatics, but from the rest of the paper I got more the sense that the paper was about validating the informativity-complexity-utility tradeoff using emergent communication techniques with the ManyNames dataset.
I think if the framing of pragmatic/semantics is primary, then there need to be more explicit definitions of what semantics and pragmatics is, why the distinction is important, and what the empirical investigation is telling us about it; I understand the difference between the "pragmatic" and "semantic" goals in the environment, but I do not get an explicit sense of what the relationship is between these two and why it is important. Also, some of the framing seems to make pragmatics seem wholly independent of semantics when really pragmatics is more a pressure on semantics.
On the other hand, if the framing is to be more consistent with the empirical work as it stands currently, I think there ought to be more talk of why the IUC-tradeoff is important/useful both generally and specifically with relation to the experiments.
Addressing this issue adequately would likely lead to me raising my Rating/Presentation score.


### Comments
- Figure 2: "Agent are trained" -> "Agents are trained"
- Line 156: "in its nature to" -> "in it nature"
- Not a requirement at all but it would be informative to compare the EC results to the recently released Mandarin Chinese data in ManyNames (assuming the data is already in the right format for analysis).
limitations: limitations: N/A.
paper_summary: paper_summary: This paper addresses the co-evolution of pragmatics, which must be interpreted according to context, and lexical semantics, the common meaning independent of context. In this paper neural agents are trained in the pragmatic setting to select the target object from two objects, and evaluated in the semantic setting to reconstruct mental representation of the target.

The authors used the vector quantize variation information bottleneck (VQ-VIB) to evaluate the trade-off of utility, informativeness, and complexity. The results showed that human-like outcomes emerged when not solely pursuing one aspect among utility, informativeness, and complexity, but rather making a non-trivial trade-off among them.
strength_weakness: strength_weakness: The explanation for the significance of the scenario that training in a pragmatic setting and testing in a semantic setting is insufficient.
The paper's contributions are somewhat unclear. Except for the point that the communication model is trained in a pragmatic setting, it is similar to [1] and [2]. It uses VQ-VIB as presented in [1], and the overall communication model is similar to [2].

The novelty is somewhat unclear in the claim that optimizing both task-specific and general objectives yields human-like results. Although there is a difference in that emergent communication is learned in a pragmatic setting, the trade-off between task-specific utility and general information is already presented in [1].

[1] Tucker, Mycal, R. Levy, J. Shah, and Noga Zaslavsky. 2022. “Trading off Utility, Informativeness, and Complexity in Emergent Communication.” *Advances in Neural Information Processing Systems*.

[2] Tucker, Mycal, Roger P. Levy, Julie Shah, and Noga Zaslavsky. 2022. “Generalization and Translatability in Emergent Communication via Informational Constraints.” https://openreview.net/pdf?id=yf8suFtNZ5v.
questions: questions: - What is the significance of training in a pragmatic setting and testing in a semantic setting?
- Since the training loss includes utility, informativeness, and complexity, isn't semantics considered during training?
limitations: limitations: n/a
paper_summary: paper_summary: This study explores how language, combining meaning and context, might evolve. It examines how a shared vocabulary can emerge from interactions that consider the situation.  By training agents in an unsupervised fashion to consider both context-specific utility and general communication pressures, the research reveals these aspects are key for understanding language evolvement.
strength_weakness: strength_weakness: - I don't see major weaknesses.
questions: questions: -  The paper's approach is interesting.  To just confirm my understanding: in this specific setting, pragmatic representations will be always more specific (like hyponyms) than semantic representations, rather than dealing with connotations. This seems similar to the challenge of finding the most accurate hyponyms to describe something. Is my understanding correct?

- What do you think would be the potential real-world applications of this work?
limitations: limitations: The paper addressed limitations.

Comments: title: Responses to follow-up questions - Part 2
Responses to follow-up questions - Part 1
Further details and references
References
comment: Thank you for your rebuttal.  I believe that the proposed changes will sufficiently address my concerns about the framing and make the findings of the empirical clear both in the context of the paper and in the broader context of computational approaches to linguistics.  I will update the review's scores accordingly.
I thank the authors for their detailed responses. I am going to stick with my current score.
> how are you thinking about comparing a system that fundamentally has one more dimension for reducing referential ambiguity to a system that is aimed to be stripped from it?

First, we would like to emphasize that we are comparing the agents’ lexicon with the human lexicon. That is, the reviewer's claim is factually inaccurate in the sense that our analysis does not include the additional syntactic dimension of English, but rather focuses only on the lexical dimension of English. 

Second, while it is certainly true that the human lexicon evolved together with syntax, in contrast to the emergent lexicon in our agents, this simplifying assumption is very common in the emergent communication literature, and more generally, in agent-based and game-theoretic approaches to language evolution. Furthermore, this kind of simplifying assumption – i.e., studying only one key aspect of human cognition, even though it has evolved together with many other cognitive functions – applies to every cognitive model that we are aware of and seems inevitable, at least given the current state of the field.

Finally, we would like to highlight that we have already acknowledged in the rebuttal that this simplifying assumption should be addressed more explicitly in the limitations sections of the paper and we certainly intend to do so if given the opportunity to revise the paper.

> Could you clarify how restricting an emerging language system to single words necessarily disables learning syntactic structure? 

In our model, agents learn a codebook of k communication vectors, similar conceptually to the idea of word embeddings. As explained above, at each communication round the speaker can generate only a single communication vector, which rules out the possibility of syntactic structures that emerge from combining communication signals. As for the possibility of a complex morphological structure in the vector embeddings, this is unlikely to emerge because the agents cannot reuse subparts of the communication vectors, but rather only the vectors as a whole. To address this, Tucker et al. (2022) proposed an extension of the model that does support some degree of combinatorial structure within the communication signals, but we have not used that extension in our work.

> You also stated that "High task performance (utility) can be trivially achieved with very non-human-like communication systems." -- I agree that utility is not a sufficient condition but what I raised in my review is the concern that a system that's not achieving task performance seems like it's missing a necessary condition to be an interpretable cognitive model.

Thank you for clarifying your point. We hope that our rebuttal and response to your first follow-up question here have convinced you that this is not a necessary condition. Human lexical systems do not seem to afford near-perfect task performance, and therefore it would not make sense to consider only emergent systems that achieves near-perfect task performance.

> Do you have estimates on how stable the learned systems are and how generalizable the results? I'm aware that the tested model is a ResNet model which has been shown in the past to have promising representational alignment with humans. However, there are still many ad-hoc decisions that are part of training and evaluating such models as cognitive models which might significantly affect the results. For instance, is there evidence that the stopping criterion used for training is a reasonable stopping criterion for finding human alignment? If you were to continue training would the learned system significantly change?

We have verified that our results are robust across random seeds (see Table 1). The stopping criteria we used is convergence of the training objective, which is one of the most common, generic stopping criteria in the literature. We have not considered a stopping criteria that specifically targets better alignment with humans, but that presumably could only improve our results. For other hyperparameters, we have built on the prior work by Tucker et al. (2022) that established the VQ-VIB framework.
We thank the reviewer for engaging in an interesting discussion about our work. Before turning to our detailed response (below), we would like to highlight that none of the points that the reviewer raised seem to be grounds for rejection. We have already addressed the concerns that were raised in the review, and the reviewer’s main follow-up questions about the syntax-semantics interface are not specific to our work. These questions touch upon some of the deepest open challenges in cognitive science and linguistics. Our work does not attempt to address these challenges. Instead, we follow common practices in the field and make well-established simplifying assumptions in order to advance the understanding of how lexical systems may emerge from contextual pragmatic interactions. Furthermore, the reviewer’s first follow-up question, on estimating the expected human performance, has actually helped us to further strengthen the support for our results and conclusions. We therefore hope that the reviewer will positively reconsider the rating of our paper.

Detailed response to the reviewer's questions:

> Firstly, could you quantify the expected human performance on the task and how you infer that number? 

While the Mädebach et al. data does not allow us to quantify precisely the human performance in our task (our point in the paper, lines 226-233, and in our rebuttal was qualitative rather than quantitative), we can use that data to roughly estimate an upper and lower bound on the human performance. For the upper bound, notice that Figure 2 in Mädebach et al. shows that, across all context conditions (no-competitor, lexicon-sufficient, and syntax-necessary), the performance is around 80%, and recall that Mädebach et al. did not restrict the participants’ responses to lexical items. Therefore, it is unlikely that in our task, which is restricted to lexical items, the performance would be better than 80%. For the lower bound, notice that in the most favorable condition, i.e., the lexicon sufficient condition, which could in principle be solved using the lexical items, the proportion of responses that correspond to lexical items without syntactic construction is only ~42% (‘no specification’ and ‘lexical’ response type). In ~39% of the response participants use syntactic construction even though there is a lexical item that could unambiguously describe the target. One possible explanation for this behavior is that in some cases lexical retrieval is harder than syntactic constructions (e.g., for infrequent words). Therefore, while we expect the proportion of lexical responses to be higher than ~42% in our version of the task, we also expect the error rate to be higher. While we don’t have a way to quantify this further, we do agree with the reviewer that the task performance would probably be higher than 50%. 
To summarize: the Mädebach et al. data suggests that the human performance in our task would be somewhere between 50%-80%. In comparison, our model achieves ~72%, which is well within the reasonable range of human performance. We find this estimation very helpful to further support our results and conclusions, and thank the reviewer for suggesting it!

> where do you draw the line between syntax and semantics in this particular setup? 

Syntax corresponds to how single words can be combined into larger linguistic structures, such as phrases or sentences. Lexical semantics corresponds to word meanings. These two subfields of linguistics are largely separable, but indeed, as the reviewer notes, there are gray areas in between. However, these gray areas are irrelevant to our work because our agents simply cannot learn them. Recall that in our setup communication signals cannot be combined and in each communication act only a single signal can be generated. Therefore borderline cases like the noun compound, which require combining words, cannot emerge in our agents. Instead of lexicalized compounds, our agents would simply use a single communication signal, thus bypassing the issue that Mädebach et al. had to address in the human data.
Thank you for the response.
Thank you for the details and clarifications. I have a few follow-up questions based on your response.

Firstly, could you quantify the expected human performance on the task and how you infer that number? I'm assuming it's definitely higher than 50% because it's unlikely that there are more than two objects with the same lexical label (on average). Is it closer to 60%, 70%, 80%, 90%, 95%? 
(To get further insights on the performance, could you maybe leverage the data split that Mädebach et al. 2022 introduced into no-competitor, lexicon-sufficient, and syntax-needed? I think this might potentially lead to interesting observations on error patterns and whether the cases where the model fails might be cases where people would employ syntactic specification.)

About the claim that *syntax is excluded from this analysis:* You specifically highlight that this work's contribution aims to only consider the semantics/pragmatics interface and tries to abstract away from syntax. I would like some clarifications on this point.
 - Firstly, where do you draw the line between syntax and semantics in this particular setup? Mädebach et al. 2022 specifically discusses that they had to make some fairly ad-hoc decisions about what qualifies as lexical and syntactic modification. A prime example are compound nouns, where Mädebach et al. for instance chose a specific frequency threshold to determine whether they classify a noun compound as syntactic or lexical modification (making "tennis player" lexical specification and "front court player" syntactic specification). This is simply to illustrate that the syntax/semantic divide isn't as clear cut in this setting as you make it seem in your response and I would like some clarification on this point.
- Secondly, human language has developed alongside syntactic complexity, so the English language has fundamentally one major additional dimension of system complexity that actively negotiates semantic complexity. As you pointed out in your response, in Mädebach et al's work, they find that people choose syntactic structures over lexical specification quite frequently. The option to have syntactic variation at a speaker's disposal for a reference game or reference disambiguation task can then be reasonably expected to decrease the lexical variation that will emerge in a learning system. In fact, there is ample of evidence in the cognitive science literature that different languages indeed optimize for a distinct syntactic complexity/semantic complexity trade-off (see, e.g., Reali et al., 2018 [1], for an interesting exploration of this trade-off). And even in an already established language, like English, recent work suggests that people trade off lexical and syntactic complexity against each other in production (Rezaii et al., 2022 [2]). Given this, how are you thinking about comparing a system that fundamentally has one more dimension for reducing referential ambiguity to a system that is aimed to be stripped from it?
- This is a more minor point, but you highlight in your responses that you restricted the model learning setup to single words and that that prohibits learning of syntactic structures. Could you clarify how restricting an emerging language system to single words necessarily disables learning syntactic structure? Since the definition of what qualifies as a token is sort of arbitrary, couldn't the system in principle map to sophisticated morphosyntax? (To clarify, I don't believe that that's the case here but I would like clarification on how the single-word constraint relates to emerging-language syntax.)

Other points:
- You also stated that "High task performance (utility) can be trivially achieved with very non-human-like communication systems." -- I agree that utility is not a *sufficient* condition but what I raised in my review is the concern that a system that's not achieving task performance seems like it's missing a *necessary* condition to be an interpretable cognitive model. 
- Do you have estimates on how stable the learned systems are and how generalizable the results? I'm aware that the tested model is a ResNet model which has been shown in the past to have promising representational alignment with humans. However, there are still many ad-hoc decisions that are part of training and evaluating such models as cognitive models which might significantly affect the results. For instance, is there evidence that the stopping criterion used for training is a reasonable stopping criterion for finding human alignment? If you were to continue training would the learned system significantly change?

Citations:

[1] Reali, Chater, Christiansen (2018), "Simpler grammar, larger vocabulary: How population size affects language", Proceedings of the Royal Society B

[2] Rezaii, Mahowald, Ryskin, Dickerson, Gibson (2022), "A syntax–lexicon trade-off in language production", PNAS
**Empirical evidence suggesting that humans do not achieve near-perfect performance:**

First, recall that our work focuses on understanding the emergence of the lexicon, and therefore our task setup only considers the use of lexicalized items (i.e., single signals or words) in communication. This kind of experimental paradigm is very common in the cognitive science of language, and is often used for studying the emergence of the lexicon. Second, while we do not have human data for our specific task, prior work has collected behavioral data for closely related tasks in this domain:

(a) The ManyNames dataset itself [9], which contains non-contextualized free naming data (i.e., with no distractor) from native English speakers, shows that the English lexicon contains unresolved ambiguities. This is not surprising given what we know about the lexicon and the probabilistic nature of many semantic categories [e.g., 1, 2, 5].

(b) Mädebach et al. (2022) [8] conducted a human experiment in a pragmatic referential task similar to ours (with distractors), but in contrast to our setup, they did not restrict participants to single-word responses. Their findings suggest that English speakers often cannot use the lexicon alone to resolve visual ambiguities in the ManyNames images. For example, the lexicon does not allow speakers to unambiguously distinguish between two similar chairs in the same scene. Instead, in such cases participants would need to employ syntactic constructions that go beyond the lexicon itself (e.g., “the chair on the left”). 

While studying syntactic constructions is an important direction for future extensions of our work, our current paper focuses on how the semantic structure of the lexicon emerges, and this aspect of human language does not support perfect task performance.


**On the importance of task performance vs other metrics for evaluation**

> “The utility-only objective achieves an accuracy of 95% but does so with a larger lexicon size, worse NID and differently looking complexity. So why do the complexity/lexicon size/NID metrics hold so much more value than the actual task performance?” 

High task performance (utility) can be trivially achieved with very non-human-like communication systems. For example, by assigning a unique signal to each object in the dataset, or by using the same signal for entirely unrelated objects that never appear in the same context. Therefore, focusing only on task performance is unlikely to explain the emergence of human-like lexicons. Taken together with our previous point, that the English lexicon does not afford near-perfect task performance, we actually find it very encouraging that the emergent system that is most aligned with humans also achieves bounded task performance. In other words, we value all metrics for evaluation, including task-performance, and when comparing them with human behavior they all consistently support our conclusions.


**References:**

[1] Piantadosi, S.T., Tily, H.J., & Gibson, E. (2011). The communicative function of ambiguity in language. *Cognition*, 122, 280-291.

[2] Zaslavsky, N., Kemp, C., Regier, T., & Tishby, N. (2018). Efficient compression in color naming and its evolution. *Proceedings of the National Academy of Sciences*, 115(31), 7937-7942.

[3] Tucker, M., Levy, R., Shah, J., & Zaslavsky, N. (2022a). Trading off utility, informativeness, and complexity in emergent communication. *Advances in Neural Information Processing Systems*.

[4] Tucker, M., Levy, R., Shah, J., & Zaslavsky, N. (2022b). Generalization and translatability in emergent communication via informational constraints. In *NeurIPS 2022 Workshop on Information-Theoretic Principles in Cognitive Systems*.

[5] Malt, B. C. (1995). Category coherence in cross-cultural perspective. *Cognitive Psychology*, 29(2), 85–148.

[6] ​​Zaslavsky, N., Regier, T., Tishby, N., & Kemp, C. (2019). Semantic categories of artifacts and animals reflect efficient coding. *Proceedings of the Annual Meeting of the Cognitive Science Society*, 41

[7] He, Y., X. Liao, J. Liang, G. Boleda. 2023. The Impact of Familiarity on Naming Variation: A Study on Object Naming in Mandarin Chinese. *CoNLL 2023*. 

[8] Mädebach, A., Torubarova, E., Gualdoni, E., & Boleda, G. (2022). Effects of task and visual context on referring expressions using natural scenes. In *Proceedings of the Annual Meeting of the Cognitive Science Society*, 44.

[9] Silberer, C., Zarrieß, S., and Boleda, G. (2020). Object naming in language and vision: A survey and a new dataset. In *Proceedings of the 12th Language Resources and Evaluation Conference*, pages 5792–5801, Marseille, France. European Language Resources Association.
[1] Dowty, D. R., Wall, R. E., & Peters, S. (1981). *Introduction to Montague Semantics*. Springer.

[2] Murphy, G. (2002). *The Big Book of Concepts*. The MIT Press.

[3] Rosch, E. (1975). Family resemblances: Studies in the internal structure of categories. *Cognitive Psychology*, 7(4), 573-605.

[4] Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., & Boyes-Braem, P. (1976). Basic objects in natural categories. *Cognitive Psychology*, 8(3), 382-439.

[5] Regier, T., Kemp, C., & Kay, P. (2015). Word meanings across languages support efficient communication. In B. MacWhinney & W. O'Grady (Eds.), *The Handbook of Language Emergence* (pp. 237-263). Wiley-Blackwell.

[6] Zaslavsky, N., Kemp, C., Regier, T., & Tishby, N. (2018). Efficient compression in color naming and its evolution. *Proceedings of the National Academy of Sciences*, 115(31), 7937-7942.

[7] ​​Zaslavsky, N., Regier, T., Tishby, N., & Kemp, C. (2019). Semantic categories of artifacts and animals reflect efficient coding. *Proceedings of the Annual Meeting of the Cognitive Science Society*, 41.

[8] Grice, Paul (1975). "Logic and conversation". In Cole, P.; Morgan, J. (eds.). Syntax and semantics. Vol. 3: Speech acts. New York: Academic Press. pp. 41–58.

[9] Goodman, N., & Frank, M. (2016). Pragmatic language interpretation as probabilistic inference. *Trends in Cognitive Sciences*, 20.

[10] Monroe, W., Hawkins, R. X. D., Goodman, N. D., & Potts, C. (2017). Colors in context: A pragmatic neural model for grounded language understanding. *Transactions of the Association for Computational Linguistics*, 5, 325–338

[11] Franke, M. (2012). Signal to Act: Game Theory in Pragmatics. PhD thesis, University of Amsterdam.