Question: Why is it okay that we define unpredictability in terms of 1 step (e --> e+1) rather than all of the previous trials, in the order the trial occurred?
Evidence: Language games were first introduced by to explore the meanings of language utterances. Instantiating this concept with the signalling game design from enables linguists to explore the emergence of linguistic structure where artificial languages are represented as symbolic systems. The success of deep learning (DL) models on complicated cognitive tasks then inspired researchers to apply DL-based models to language games to investigate the agents' ability to invent communication protocols without preset rules (e.g..
In the existing works (e.g., there are usually two types of agents, speakers that emit messages based on their observations (i.e. input target objects) and listeners that receive messages and act accordingly, as illustrated in Figure. Based on the goals for listeners, we can categorise most of the games into the following three types: 1) referential games in which listeners need to select the target object observed by the speaker among a set of candidate objects (candidates for short) (e.g.; 2) reconstruction games in which listeners need to reconstruct the speaker's observation (e.g. ; and 3) navigation games in which listeners need to go to the location specified by speakers (e.g.. We focus on referential games, illustrated in Figure, which have been well investigated in both the linguistic community (e.g. and emergent communication community (e.g..
It is reasonable to assume that listeners in referential games can differentiate the target object from the other distractors in the context as long as the speaker's message encodes some unique feature of the target. Therefore, the speaker's messages must convey a different amount of information for listeners to complete their task in games where candidates are similar in different degrees. For example, to distinguish two very similar candidates, e.g. blue wide-stripped shirt and blue narrow-stripped shirt, it requires more information about the target to be encoded by the speaker's message than to distinguish a shirt from a cup. Experiments with human participants show that emergent languages are sensitive to the requirements of the communicative tasks for which they are used, with languages developing in which only the necessary discriminatory information is encoded.
˚Correspondence author, e-mail address: s.guo@ed.ac.uk   Following, we refer to such ability to encode information about input space (e.g. structure, dimensionality, distribution of samples) as expressivity. In this work, we explore the factors that influence expressivity in the framework of DL-based referential games, and we argue that it is important to understand these factors such that agents could develop languages that are effective enough for completing various tasks.
Our contribution in this work is threefold. First, we propose and verify a hypothesis about the determining factors of expressivity of emergent languages under the DL-based framework. Following, we show that context size (i.e. the number of candidates the listener must select among when interpreting the speaker's message) influence both the complexity and unpredictability of that context. Complexity refers to the similarity between the items in the context: in order for the listener to select the target, more information needs to be encoded in the speaker's signal if the context contains more similar distractors for a given target. Unpredictability refers to the extent to which the information that needs to be encoded by the speaker is stable across contexts in successive episodes of communication: in games where the information required by the listener differs from trial (training epoch for DL agents) to trial, the speaker needs to encode more information on every trial in order to be sure to encode the information the listener needs. As we show in Section 3.1, complexity and unpredictability are affected differently by the context size: as context size increases, complexity increases but unpredictability decreases. Therefore, we propose and verify the following hypothesis about the determining factors of expressivity:
[HYPOTHESIS] The expressivity of emergent languages is a trade-off between the complexity and the unpredictability of context in language games.
[1]
Our second contribution is a novel measure of expressivity based on partial ordering of languages in terms of their generalisation performances across tasks. Although expressivity is related to the amount of information encoded in a language, we illustrate that mutual information (MI) is not an appropriate measurement for expressivity in Section 3.2. Considering that one of our objectives is to facilitate a universally useful emergent language, we propose to measure expressivity of a language based on the generalisation performance of listening agents trained with that language across different games. Since it is more challenging to quantitatively and directly measure expressivity, we focus on the partial ordering between languages in this first step towards understanding expressivity.
Our final contribution is the discovery of message type collapse, i.e. the number of unique messages is significantly lower than the size of input space in relatively easy games, which can lead to highly ambiguous emergent languages. To overcome the technical limitations imposed by GPU memory size on large-scale referential games, we introduce the contrastive loss proposed by in referential games. While comparing the behaviour of the contrastive loss and the loss function used in previous works (e.g, we find that this contrastive loss can greatly alleviate the collapse of message types, leading to more disambiguous and potentially more expressive emergent languages.
As we show in Section 3.1,complexity and unpredictability are affected differently by the context size: as context size increases,complexity increases but unpredictability decreases.
Answer: Like we said in our previous reply, no matter how many timesteps we use to define the unpredictability, the unpredictability would still decrease as the size of candidate set increases. Therefore, our conclusion still holds with a different number of time steps.