SlowGuess/NAACL_2022_OCR huggingface dataset:
measures for mini-ImageNet. Despite only reaching a signaling accuracy of  $33.328 \pm 19.324$ , the highest Top.Sim. and Dis.Sim. scores (0.450) are realized by mismatched ProtoPNet-CW, thus compositionality was able to emerge despite the absence of generalization ability, as originally reported by Chaabouni et al. (2020). CW-CW and ProtoPNet-ProtoPNet scored similarly, obtaining  $85.585 \pm 5.476$  and  $67.542 \pm 3.001$  best epoch signaling accuracy, respectively. The best mixed-encoder system in Figure 4 was ConvNetProtoPNet ( $74.929 \pm 3.079$  accuracy), with a similarly high topographic similarity. We thus conclude:
A2: Heterogeneous systems evidence lower signaling performance compared to same-encoder baselines on novel objects, although the former are capable of improving the language compositionality. Holistically, disentangled networks either matched or outperformed the ConvNet baselines.
(b)
Figure 4: Comparison of communication success rate on mini-ImageNet few-shot testing data for permutations of a) ConvNet senders, b) ProtoPNet senders, and c) CW senders. Shaded regions denote standard deviation over five random seeds (each swapping in new sender encoder and receiver encoder checkpoints), with  $L = 10$ . Agents in this experiment have never seen the testing objects before.
Sender Arch.	Top. Sim.	Dis. Sim.	Recv. Acc.
ProtoPNet	0.523	0.523	81.625
z' Utterance	±0.006	±0.006	±1.031
ProtoPNet	0.362	0.362	80.343
±0.078	±0.078	±0.681
ProtoPNet+LSA	0.383	0.383	81.496
±0.033	±0.033	±1.665
ConvNet	0.391	-	86.753
±0.055	±0.651
Table 3: Comparison of LSA module, 1-Length utterance baseline, and normal RNNs on CUB10 testing data using ProtoPNet disentangled structure to weight the vocabulary  $(|V| = 10)$ .
Informing design (Q3): Tables 3 and 4 examine the aggregate effect on CUB10 test data after integrating concept-aligned latent vectors into the agent utterance, using our LSA module described in Section 3.1. We report statistics using each system's best signaling checkpoint. Since  $|V| = k$ , for ease of comparison we set  $|V| = k = 10$  for both ProtoPNet and CW, and constrain all agents to communicate with fixed message lengths  $L = 10$ . In addition to the normal variants, we compare against a baseline which uses the latent vector  $\mathbf{z}'$  directly as the vocabulary logits to produce a single utterance, establishing the potential usefulness of the disentangled vector for communication. In the case of ProtoPNet in Table 3, we observe that the LSA variant is competitive with the ConvNet baseline, and slightly improves over the regular ProtoPNet baseline. In fact, the 1-Length baseline was sufficient for generalization (81.625 ± 1.031 signaling accuracy). In Table 4 for CW models, we observe that the LSA module had significant impact, increasing signaling accuracy from 52.523 ± 17.600
to  $79.859 \pm 2.417$ . Notably, the 1-Length utterance was scored  $75.670 \pm 1.820$ , indicating that the learned concepts were more useful than the proto-language created by the regular CW variant. Despite this, the result with  $\mathrm{CW + LSA}$  evidences that incorporating both sources of information is beneficial for signaling performance.
Apart from incorporating new information into the sender's message decoding process, our proposed LSA module may enable better diversity in the agent utterances. Recall from Section 3.1 that the disentangled representation is linearly recombined and multiplied with message logits at each time-step, thus we posit that the LSA module encourages agents to use vocabulary that is otherwise ignored, which may explain the higher topographic similarity scores observed in Tables 3 and 4. We study this idea for CW in Figure 5 (and ProtoPNet in Appendix A.5.1) by taking the top-performing multi-agent system for ConvNet (top row), CW (middle row), and  $\mathrm{CW + LSA}$  (bottom row). Each inset heat map is the activated vocabulary usage (x-axis) across input data classes (y-axis) in the CUB10 test set at a given time step for each figure column (i.e., columns  $t = 0$  to  $t = 10$ ), with the cumulative sum over time displayed in the right-most column. We quantify the vocabulary usage by calculating normalized area ( $A$ ) of shaded heatmap pixels, shown below each cumulative matrix. ConvNet and CW variants (top and middle row, respectively) evidence a tendency to re-use vocabulary at each time-step and do not explore different combinations of vocabulary across time steps. LSA variants (bottom row) instead have a higher distribution of vocabulary across time, which can be seen qualitatively by comparing early time-steps to
Figure 5: Heat-maps of the best-performing agent's vocabulary usage across time and data classes over the entire CUB10 test set for CW ( $|V| = k = 10$ ), where agents communicate for ten time-steps (with end-of-sentence symbol '0' at the final time-step). The final column shows the cumulative sum over all time steps, subtitled by the normalized area of non-zero cells.
Sender Arch.	Top. Sim.	Dis. Sim.	Recv. Acc.
CW	0.325	0.503	75.670
z' Utterance	±0.031	±0.006	±1.820
CW	0.278	0.339	52.523
±0.079	±0.044	±17.600
CW+LSA	0.266	0.439	79.859
±0.087	±0.076	±2.417
ConvNet	0.391	-	86.753
±0.055	±0.651
Table 4: Comparison of LSA module, 1-Length utterance baseline, and normal RNNs on CUB10 testing data using CW disentangled representation to weight the vocabulary  $\left( {\left| V\right|  = {10}}\right)$  .
later time-steps. For example, the CW+LSA variant (bottom row) uses the vocabulary space [6, 10) across classes early in the message, whereas this space is less used in later parts of the message (e.g., compare space [6, 10) at  $t = 2$  against  $t = 9$ ). Notably, this space-swapping behavior can be seen in the ConvNet variant (top row), which causes higher cumulative area than the regular CW variant (middle row). In this analysis, CW+LSA obtains the highest cumulative area, so it can be said that this variant leverages the most tokens in the vocabulary, which may help explain the high disentangled similarity (0.439) in Table 4. We thus conclude:
A3: Incorporating the aligned subspace information into the vocabulary logits can have substantial impact on disentangled agents, remaining competitive with the non-disentangled ConvNet baseline accuracy, and improving or matching concept alignment (Dis. Sim.) compared to the baseline CW or ProtoPNet variant.
Categorization power and communication (Q4): Throughout the evaluation, we have compared against ProtoPNet ( $k = 100$ ) and CW ( $k = 10$ ). To better isolate the concept realization implied by  $k$ , we train ProtoPNet encoders for  $k$  values in  $\{1, 10, 100\}$ , then play signaling games with mixed ProtoPNet variations (denoting each as  $S_{k}$  or  $R_{k}$ ) using fixed message length  $L = 10$ . We report the communication success rate over epochs in Figure 6, where each sub-figure is a fixed  $S_{k}$ . For  $S_{k} = 10$  in Figure 6a, it is capable of generalization when paired with  $R_{k} \in \{10, 100\}$ , and suffers with  $R_{k} = 1000$ . Although  $R_{k} = 1000$  under-performs in each case (Figures 6a-c), we observe that ProtoPNet encoders with  $k = 1000$  have sufficient generalization in the upstream classification task, scoring at least  $91\%$  accuracy on CUB10 (we report all upstream scores in Appendix Table 13). This finding is nuanced by considering additional model architectures, such as DenseNet-161 in Appendix Section A.4.1, which shows that the complexity penalty observed in Figures 6a-c is mitigated by using the larger DenseNet-161 architecture (illustrated in Appendix Figure 10). Thus certain model architectures have unexpected benefits when deploying more complicated sender/receiver pairs. We thus conclude:
A4: Better concept realization does not always equate to better reception, despite performance on an upstream task such as classification. Otherwise, mixed- $k$  agents have a tendency to generalize similarly, despite differences in the number of concepts.
(b)
Figure 6: Modulating the sender's categorization ability over realized concepts  $k$  for a)  $S_{k} = 10$ , b)  $S_{k} = 100$ , and c)  $S_{k} = 1000$ , when paired with receivers of varying categorization ability.
The properties of language acquisition and emergence of its structural properties (e.g., compositionality and recursion) were investigated early on by Kirby (1998), who proposed that natural language can evolve to possess an internal structure as a result of observational learning. Due to the availability of complex real-world data and the scalability of deep learning, recent studies in language evolution have focused on artificial, yet realistic analogues of language evolution (Lazaridou et al., 2017; Havrylov and Titov, 2017; Lazaridou and Baroni, 2020). Notably, artificial language emergence has been shown to exhibit an inner structure as originally investigated by Kirby (1998), e.g., compositionality (Lazaridou et al., 2018; Chaabouni et al., 2020) and encoding preference (Chaabouni et al., 2019). During artificial language formation, agents modulate the structure of the language in service of jointly solving the learning task. For example, by negotiating symbols in the artificial language, Hagiwara et al. (2019) evidenced the ability for same-encoder agents to realize data categories in an unsupervised way, leading to a form of emergent categorization. The most related work to ours is by Chaabouni et al. (2020), who study concept formation in same-encoder artificial agents through the lens of disentanglement. However, their study focuses on proposing new metrics of compositionality in small-scale, categorical data. We instead investigate the case of mixed-encoder agents, each realizing a different level of disentangled categorization ability on large-scale RGB image data.
Our results suggest that agents in a cooperative game setting are capable of communication despite differences in concept realization. Despite this,
we find that heterogeneous agents can be at odds with each other, often under-performing compared to their same-encoder variants. In this way, the reasoning process of an agent can largely determine its interaction with other agents. On novel objects, there is a notable advantage to using disentangled networks, and in some cases mismatched agents, evidenced by higher signaling success and language compositionality. Our proposed Latent Self-attention module illustrates the advantage of incorporating an agent's categorization power into utterances, evidenced by higher disentangled similarity in experiments. The interplay between disentanglement and interpretability of the emerged language will be investigated in future work.
This work was funded by the Air Force Research Laboratory's Early Career Award program in conjunction with the AFOSR Center of Excellence on Assured Autonomy (AFRL-FA9550-19-1-0169) and NSF grant CNS-2055123.
Henry Brighton and Simon Kirby. 2006. Understanding linguistic evolution by visualizing the emergence. Artificial Life, 12(2):229-242.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. 2020. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4427-4442, Online. Association for Computational Linguistics.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. 2019. Anti-efficient encoding in emergent communication. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing
Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 6290-6300.
Daniel Chandler. 2007. Semiotics: The basics, 2nd edition. Basics (Routledge (Firm). Routledge, London; New York. OCLC: ocm70864411.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. 2019. This looks like that: Deep learning for interpretable image recognition. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8928-8939.
Zhi Chen, Yijie Bei, and Cynthia Rudin. 2020. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782. ArXiv: 2002.01650.
Dedre Gentner. 2010. Bootstrapping the mind: Analogical processes and symbol systems. Cognitive Science, 34(5):752-775.
Yoshinobu Hagiwara, Hiroyoshi Kobayashi, Akira Taniguchi, and Tadahiro Taniguchi. 2019. Symbol Emergence as an Interpersonal Multimodal Categorization. Frontiers in Robotics and AI, 6:134.
Serhii Havrylov and Ivan Titov. 2017. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2149-2159.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Douglas R. Hofstadter. 2007. I am a strange loop. Basic Books, New York. OCLC: ocm64554976.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261-2269. IEEE Computer Society.
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. 2019. EGG: a toolkit for research on emergence of lanGuage in games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 55-60, Hong Kong, China. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Simon Kirby. 1998. Learning, bottlenecks and the evolution of recursive syntax. In In E. Briscoe (Ed.), Linguistic. University Press.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962-2967, Copenhagen, Denmark. Association for Computational Linguistics.
Angeliki Lazaridou and Marco Baroni. 2020. Emergent multi-agent communication in the deep learning era. arXiv:2006.02419 [cs]. ArXiv:2006.02419.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. 2018. Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. 2018. Emergent translation in multi-agent communication. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
David Lewis. 1969. Convention. Cambridge, MA.
Marvin Minsky. 1988. Society Of Mind. Simon and Schuster. Google-Books-ID: bLDLlIfRpdkC.
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M. Dai, and Kyunghyun Cho. 2020. Capacity, bandwidth, and compositionality in emergent language learning. arXiv:1910.11424 [cs, stat]. ArXiv: 1910.11424.
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. 2022. Interpretable Machine Learning: Fundamental principles and 10 grand challenges. Statistics Surveys, 16:1-85.
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. 2015. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems 28: Annual
Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3528-3536.
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Brian Skyrms and Oxford University Press. 2010. Signals: Evolution, learning, and information. Signals: Evolution, learning, & information. OUP Oxford.
Endel Tulving. 2002. Episodic memory: From mind to brain. Annual Review of Psychology, 53(1):1-25.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3630-3638.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. 2018. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9):2131-2145.
Bolei Zhou, Aditya Khosla, Ågata Lapedriza, Aude Oliva, and Antonio Torralba. 2015. Object detectors emerge in deep scene CNNs. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
A Appendix
A.1 Training details
In our experiments, every pair of sender and receiver is trained with distinct seeds, thus they never share the same checkpoint, only the training data. We use the publicly available code to train encoders from the respective authors (Chen et al., 2019, 2020). The underlying model for CW and ProtoPNet encoders is a standard ResNet-50
CNN (He et al., 2016) trained on the full-size ImageNet dataset<sup>3</sup>, and fine-tuned on the respective dataset (CUB10 or mini-ImageNet) for five epochs over ten random seeds (i.e., unique sender/receiver seeds for five systems).
The Gumbel-Softmax estimator samples from the Gumbel distribution with a temperature term, which controls how similar the relaxation is to the true argmax distribution. As recommended by Havrylov and Titov (2017), we allow the sender agent to learn the inverse of the temperature in order to avoid manual tuning. Agent parameters are tuned by stochastic gradient descent (SGD) using the Adam optimization algorithm (Kingma and Ba, 2015). The recurrent sender decoder  $g$  and receiver encoder  $h$  are instantiated in practice with recurrent neural networks (RNNs) using LSTM cells. We leverage the NVIDIA DALI library to accelerate data loading through the use of GPGPU hardware.
A.2 Hyperparameter selection
In order to achieve the best performance for agents and encoders, we performed a grid-search over the following hyperparameter combinations:
- Number of epochs in  $\{30, 50, 100\}$ ,
- Hidden dimension of LSTM hidden state in  $\{64, 128, 256\}$ ,
- Number of distractors in  $\{1,3,5\}$ ,
- Message length  $L$  in  $\{10, 100\}$ ,
- ConvNet classifier fine-tuning learning rate in  $\{1^{-2}, 1^{-3}\}$ ,
- ProtoPNet dense representation network  $f$  learning rate in  $\{1^{-4}, 1^{-5}\}$ ,
- ProtoPNet interface layer learning rate in  $\{3^{-5}, 3^{-4}, 3^{-3}\}$ ,
- ProtoPNet prototype layer learning rate in  $\{3^{-5}, 3^{-4}, 3^{-3}\}$ ,
- ProtoPNet classifier learning rate in  $\{1^{-5}, 1^{-4}\}$ ,
- CW classifier and whitening layer learning rate  $\{1^{-2}, 1^{-3}\}$ ,
3https://pytorch.org/vision/stable/ models.html
<sup>4</sup>https://github.com/NVIDIA/DALI
- Sender learning rate in  $\{1^{-5}, 1^{-4}, 1^{-3}, 1^{-2}\}$ , and
- Receiver learning rate in  $\{1^{-5}, 1^{-4}, 1^{-3}, 1^{-2}\}$ .
As a result of the grid-search, we selected the best performing hyperparameter combination which balanced training time and communication success rate performance (e.g., receiver signaling accuracy). This combination is shown in Table 5.
A.3 Hardware description
All experiments were performed on an NVIDIA DGX-2 compute node equipped with 16 NVIDIA Tesla V100 Tensor Core GPUs, high-speed NVMe flash storage, 1TB main system memory, and 40 Intel Xeon Platinum CPU cores.
A.4 Supplemental Results
A.4.1 Results on VGG-16 and DenseNet-161 variants
The disentangled representations discussed in the main text are not restricted to residual networks such as ResNet-50, in fact they are flexible enough to interact with other network designs such as VGG-16 (Simonyan and Zisserman, 2015) and DenseNet-161 (Huang et al., 2017). To expand the empirical contribution, we provide additional results addressing Q1, Q3, and Q4 of the main text using disentangled image encoders built on top of VGG-16 and DenseNet-161, as well as their non-disentangled baselines.
Q1 (VGG-16 & DenseNet-161). In Figures 7 and 8, we see the same general tendency for same-encoder systems to either match or outperform the mixed-encoder versions. This is evidenced on VGG-16 (Figure 7, red line) across each sender variant, with CW being a notable exception due to performing similarly between CW-CW and CW-ConvNet variants. On DenseNet-161, we observe a failure mode for CW (Figure 8c) due to low accuracy on the upstream task (Table 16). We experimentally observed that CW causes training instability for DenseNet-161 on CUB10. In this failure mode, there is little difference between same- and mixed-encoder agent systems, which may indicate that the performance discrepancies are tied to success of the upstream task.
As in the main text, we provide measures of the language compositionality under different encoder variants for VGG-16 (Table 6) and DenseNet-161
(Table 7). In this context, topographic similarity or disentangled similarity for mixed-encoder systems can be higher in some cases (e.g., VGG-16 CWConvNet Dis. Sim. of 0.435 compared to 0.401 of CW-CW in Table 6, or VGG-16 ConvNet-CW Top.Sim. of 0.312 compared to 0.305 of ConvNetConvNet). In general, same-encoder variants evidence a tendency to outperform the mixed-encoder variants, invariant to the choice of underlying architecture. Returning to the failure mode of DenseNet161 CW seen previously, we observe that the topographic similarity scores for DenseNet-161 CW variants in Table 7 are competitive with models that did not experience failure, e.g., the CW-CW system scored Top.Sim. of 0.391 whereas ProtoPNetProtoPNet scored 0.369. This is further evidence of the effect observed by Chaabouni et al. (2020), which is that compositionality can emerge despite lack of generalization ability.
Q3 (VGG-16 & DenseNet-161). We investigated Q3 under the same constraints of Section 4, but instead using VGG-16 (Tables 8 and 9) and DenseNet-161 (Tables 10 and 11) instead of ResNet-50. We observe the general tendency from Section 4 for RNN+LSA variants to either match or outperform the standard RNN variants, whilst remaining competitive with the non-disentangled ConvNet baseline. The exception to this trend is the failure mode of DenseNet-161 CW variants (Table 11), despite evidencing higher disentangled similarity with low generalization ability (e.g., Dis.Sim of 0.785 for RNN+LSA with  $45.487\%$  receiver accuracy).