Question: The paper does not fully motivate why Definition 3.1 is a good quantifiable metric for studying expressivity in emergent languages. How does generalization that is studied in emergent languages like [2] relate to this particular definition of expressivity?
Evidence: Following the intuition that the different types of games require messages to convey different amount of information about input space X , MI may seem to be a good candidate measure for expressivity. However, we found that even for two emergent languages that are both deterministic and bijective mapping functions, i.e. they have the same MI value logp|X |q, their expressivity can still be different. Further discussion about MI can be found in Appendix D.
Since it is more challenging to directly quantify the expressivity of emergent languages, we instead focus on the partial orders between them in this work. Considering that the purpose of the emergent communication is to complete tasks, we therefore propose to measure the partial order between expressivity based on game performance. This leads us to a common practice in the transfer learning community (e.g. Pan & Yang, 2009), i.e. train models on source domain but evaluate them on different target domains. Inspired by this practice, here we propose a definition of the partial order between the expressivity of two emergent languages L A and L B , i.e. E L A and E L B :
Let g be an instance of language game, and G be a set of language game instances. Then, given G, if the generalisation performance of L A is better than L B on some language games G 1 with statistically significant difference, i.e. @g P G 1 , P g L A ą P g L B , where ∅ Ă G 1 Ď G and P denotes the generalisation performance, while there is no statistically significant difference between the converged generalisation performance of L A and L B on the remaining games, i.e. P g 1 L A « P g 1 L B @g 1 P GzG 1 4 , we then say that expressivity of L A is higher than
The metric of generalisation performance could be varied according to different types of games.
Briefly speaking, given a set of games, if one language performs better than the other on some games while they perform approximately the same on the remaining games, we then say the expressivity of that language is higher than the other. Following this definition, there are several possible relationships between the expressivity of two languages L A and
Following the intuition that the different types of games require messages to convey different amount of information about input space X , MI may seem to be a good candidate measure for expressivity. However, we found that even for two emergent languages that are both deterministic and bijective mapping functions, i.e. they have the same MI value logp|X |q, their expressivity can still be different. Further discussion about MI can be found in Appendix D. Since it is more challenging to directly quantify the expressivity of emergent languages, we instead focus on the partial orders between them in this work. Considering that the purpose of the emergent communication is to complete tasks, we therefore propose to measure the partial order between expressivity based on game performance. This leads us to a common practice in the transfer learning community (e.g. Pan & Yang, 2009), i.e. train models on source domain but evaluate them on different target domains. Inspired by this practice, here we propose a definition of the partial order between the expressivity of two emergent languages LA and LB, i.e. ELA and ELB :
Answer: We updated Section 3.2 to better motivate the definition of expressivity based on generalisation performance.
Regarding the notion of generalisation, the authors of [2] evaluate the generalisation performance on a single kind of target game, while we use generalisation performance on a set of games as a metric for the expressivity of emergent languages.
At the same time, [2] shows that our input space (containing 10,000 samples) is large enough for agents to achieve a reasonable generalisation performance (>99%). We use this as evidence to support the set-up of our language games.
Referecne:
[2] Chaabouni, R., Kharitonov, E., Bouchacourt, D., Dupoux, E., & Baroni, M. (2020). Compositionality and generalization in emergent languages. arXiv preprint arXiv:2004.09124