SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ease and transfer learning (ETL). ETL [10] captures how fast and well the emergent language is transmitted to new listeners performing a different task. We focus on the emerged language's transferability between reasoning tasks with different attribute values,  $N$ . Specifically, we first dump the deterministic languages (choosing the token greedily with the highest probability) of well-trained speakers under each attribute value setting. We then map deterministic languages in the source task setting and reasoning problems in the target task settings based on the rules. Finally, we train new randomly initialized listeners using mapped language and reasoning problems and observe the convergence speed and accuracy of the listeners. Our source task setting contains attribute values  $N \in \{20, 30, 40\}$ , and the target task setting contains  $N \in \{20, 30, 40\}$ , and an extremely large  $N = 80$ .
To show the benefits and effectiveness of emergent communication through the Reasoning Game, we answer the following three questions by the experimental results.
5.1 What do the emerged language tend to describe?
Emerged language prefers to describe rules rather than context panels. We demonstrate this issue from two aspects. 1) Compare which input (rules or context panels) has a stronger correlation (measured by Topsim) with the message space. As Table 1 shows, in all three diversities of attribute values (i.e.,  $N = 20, 30, 40$ ), the Topsim between rules and messages (i.e., Rule-Message) is always substantially higher than that between context panels and messages (i.e., Panel-Message). This directly proves the conclusion, and the emerged languages are somewhat compositional (because Topsim is also used to measure compositionality). 2) Another indirect evidence comes from the relationship between ID generalization performance (measured by Listeners' accuracy) and attribute value diversities. Specifically, since the capacity of the communication channel is not enough to describe the context panels ( $15^4 < 10^{10}$ ), more diverse context panels are expected to exacerbate this shortage, leading to a rapid decrease in accuracy. However, paradoxically, the data in each column of Table 2 shows that the accuracy is almost equal as the attribute value diversity increases (from 20 to 30 or 40).
Table 1: Topsim (mean ± standard error) between rules/panels and messages on ID data split with p-value<  ${10}^{-4}$  .
Attribute values	Rule-Message	Panel-Message
20	0.2382 ± 0.0117	0.1148 ± 0.0122
30	0.2540 ± 0.0130	0.1127 ± 0.0080
40	0.2381 ± 0.0289	0.0937 ± 0.0124
Figure 4: Train accuracy under the message-blocked scene. The results are averaged across the performance of 16 randomly initialized listeners on 4 randomly generated dataset instances. The shaded region represents the standard deviation.
Figure 5: Train accuracy w/ and w/o two-stage curriculum training. The results are averaged across 8 seeds. The shaded region represents the standard deviation.
5.2 Why are the proposed benchmark and baseline effective and necessary?
Rule-RAVEN dataset. We quantitatively compare our rule-based candidate panels generation algorithm (i.e., Algorithm 1) in rule-RAVEN with the I-RAVEN. Specifically, by separately training the listener under a message-blocked scene (i.e., the speaker's message is forced to a constant), we compared the overfitting degree of rule RAVEN and I-RAVEN (the higher the accuracy, the less dependent the listener is on the speaker's reasoning about rules, i.e., the higher the overfitting degree). Figure 4 shows that I-RAVEN suffers from overfitting severely (lines with the legend 'I-RAVEN_20/30/40'), for the listener's train accuracy still achieves  $\sim 0.9$  even if the speaker's message is completely ignored. While rule-RAVEN (lines with the legend 'rule-RAVEN_20/30/40') can effectively control the degree of overfitting, the listener's train accuracy is only  $\sim 0.6$ , which is significantly different from when receiving the speaker's message normally ( $\sim 0.6$  v.s.  $\sim 0.9$ ). So it is enough to distinguish whether communication is effective or not. Note that controlling overfitting to an ideal level (8 candidate panels,  $\approx 0.125$  message-free accuracy) is unlikely because the listener can always guess the answer by memorizing the patterns corresponding to specific rules. Besides, naive methods, such as increasing attribute values, do not alleviate overfitting (as shown in lines with the legend 'I-RAVEN,' from 20 to 30 or 40).
Two-stage curriculum training method. We compare the training curves with and without the two-stage curriculum training method. Figure 5 shows that the two-stage curriculum training method leads to successful communication in all three attribute value diversities (lines with the legend 'w_ct_20/30/40'), for agents reaches higher accuracy ( $\sim$  0.9). In contrast, in the case without the two-stage curriculum training method (lines with the legend 'w/o_ct_20/30/40'), agents only converge to a similarly low accuracy as under the message-blocked scene, which implies that agents can not communicate successfully. An intuitive explanation for this phenomenon is that, the speaker's training in the first stage helps him acquire some reasoning ability to cope with the drifting context, which helps stabilize the semantic production in the joint training stage.
Table 2: Accuracy (mean  $\pm$  standard error) of 4 generation levels.
Attribute values	ID	Inpo-ood	Expo-ood-L1	Expo-ood-L2
20	0.9492 ± 0.0034	0.9473 ± 0.0039	0.6590 ± 0.0371	0.4921 ± 0.0658
30	0.9369 ± 0.0073	0.9295 ± 0.0083	0.5880 ± 0.0349	0.4684 ± 0.0630
40	0.9194 ± 0.0190	0.9210 ± 0.0178	0.5741 ± 0.0238	0.4597 ± 0.0379
Table 3: Train accuracy (mean ± standard error) of task transferring.
Target
20	30	40	80
Source	20	-	0.9257 ± 0.0059	0.9238 ± 0.0051	0.9199 ± 0.0081
30	0.9098 ± 0.0110	-	0.9124 ± 0.0087	0.9096 ± 0.0083
40	0.9022 ± 0.0171	0.9043 ± 0.0159	-	0.8974 ± 0.0136
5.3 What are the benefits of communicating extracted rules?
Beneficial for generalizing to unseen rule combinations. As shown in Table 2, the high generalization performance at the ID level  $(\approx 0.92)$  shows that communicating abstract rules can generalize to different reasoning problems with the same rule combination. Moreover, the generalization performance of the Inpo-ood level is similar to that of the ID level, which indicates that communicating abstract rules can also generalize to unseen rule combinations, e.g., based on (add, min) and (progression, max) on (number, size) generalize to (add, max) on (number, size).
Beneficial for agents apply rules to other attributes. As shown in Table 2, we observe that the performance of level Expo-ood-L1 is higher than that of Expo-ood-L2(≈ +0.13), which suggests that communicating abstract rules can help the listener to apply rules extracted from one attribute to others, e.g., applying rule (progression) extracted from (number) to (size). This characteristic is similar to the human ability to draw inferences from one instance to another. Similar to the observation in Chaabouni et al. [10], complicating the task of language emergence (reasoning task in our work) does not improve the systematicness of emergent language (from average Inpo-ood ≈ 0.92, to Expo-ood-L1 ≈ 0.58, and to Expo-ood-L2 ≈ 0.46), possibly because reasoning about unseen rules is highly challenging in our game.
Beneficial for transferring to a different task setup. As shown in Table 3, all transfers in the table have relatively high train accuracy and are close to the upper bound (ID and Inpo-ood accuracy). Even languages that emerged on the simplest task (20 values of each attribute) can be transferred to the most challenging task (80 values of each attribute). Figure 6 shows the training curve of task transferring grouped by target attribute values. Emerged language (lines with the legend 'agent') has a similar convergence speed compared with the ideal language that directly uses rule encoding as the language (lines with the legend 'rule') and only hundreds of optimization steps (less than 10 epochs) to converge. Besides, emerged language outperformed the random group (lines with the legend 'random'), which randomly maps source languages and the target reasoning problems. Good cross-task transferability also indirectly confirms that the emerged languages tend to describe rules more than context panels.
(a) Transferring to 20.
(b) Transferring to 30.
Figure 6: Training curve of task transferring grouped by target attribute values. The results are averaged across 8 seeds, and the shaded region represents the standard deviation.
(c) Transferring to 40.
(d) Transferring to 80.
In this work, we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules. Based on our rule-RAVEN dataset and two-stage curriculum agent training method, we successfully observe the emergence of a language that can describe abstract rules and further reveal the language's impressive generalization ability and transferability.
Our research may inspire the following directions: 1) Emerging communication on more realistic-stimuli-based reasoning datasets [32, 41, 47] to study the co-evolution of perception, cognition, and language system in agents; 2) Debiasing and avoiding overfitting on the procedurally generated reasoning dataset from a more fine-grained perspective; 3) Designing pre-training tasks to converge in environments with multiple variable drifts.
We summarized the limitation of our works as follow: 1) Our work only focuses on language emergence on a clean symbolic-based reasoning dataset, lacking the exploration of more realistic stimuli-based (e.g., synthetic or natural images) reasoning datasets; 2) The reasoning task (RAVEN) adopted in our work only requires the agent to complete the reasoning via a single round of interaction, simplifying the natural reasoning process; 3) Our work only analyzes the semantics of the emerged languages at the message level, lacking fine-grained structural (e.g., grammar) and semantic analysis at the token level (e.g., the degree of polysemy and ambiguity between tokens).
This work is partially supported by the NSF of China (under Grants 61925208, U22A2028, 62102399, 62222214, 62102398, U19B2019), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
[1] Michal Auersperger and Pavel Pecina. Defending compositionality in emergent languages. arXiv preprint arXiv:2206.04751, 2022.
[2] David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pages 511-520. PMLR, 2018.
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009.
[4] Yonatan Bisk, Kevin Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial operations in a rich 3d blocks world. Proceedings of the AAAI Conference on Artificial Intelligence, 32, 12 2017.
[5] Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life, 12(2):229-242, 2006.
[6] Patricia Carpenter, Marcel Just, and Peter Shell. What one intelligence test measures: A theoretical account of the processing in the raven progressive matrices test. Psychological review, 97:404-31, 08 1990.
[7] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4427-4442, Online, July 2020. Association for Computational Linguistics.
[8] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems, 32, 2019.
[40] Luc Steels. The synthetic modeling of language origins. Evolution of communication, 1(1):1-34, 1997.
[41] Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton Hengel. V-prom: A benchmark for visual reasoning using visual progressive matrices. Proceedings of the AAAI Conference on Artificial Intelligence, 34:12071-12078, 04 2020.
[42] Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
[43] Kyle Wagner, James A. Reggia, Juan Uriagereka, and Gerald S. Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69, 2003.
[44] Ke Wang and Zhendong Su. Automatic generation of raven's progressive matrices. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, page 903-909. AAAI Press, 2015.
[45] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229-256, may 1992.
[46] Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arXiv:2007.04212, 2020.
[47] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[48] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
A Details of rule-RAVEN dataset
A.1 Rules in rule-RAVEN
In the joint training stage, a specific rule contains 4 elements on different attributes, e.g., (Number:add, Color:progression_2, Size:constant, Shape:max). These rules meet the structural and functional requirements stated in Section 3.2 The embodiment of each rule on each attribute are shown in Figure 7. For example, the 'Color:progression_2' means: in the same row of panels, the image color gradually darkens by 2 degrees from left to right.
No.	Rules	Attributes
Number	Color	Size	Shape
1	constant	→ → → →	→ → → →	→ → → →	→ → → →
2	add	→ → + →	→ → + →	→ → + →	Δ → ∅ → ∅
3	minus	→ → - →	→ → - →	→ → - →	→ ∅ → - Δ
4	min	→ → min →	→ → min →	→ → min →	→ ∅ → min →
5	max	→ → max →	→ → max →	→ → max →	→ ∅ → max →
6	progression_2	→ +2 → +2 →	→ +2 → +2 →	→ +2 → +2 →	Δ → +2 → ∅ → +2
7	varprogression_1	→ -1 → -2 →	→ -1 → -2 →	→ -1 → -2 →	→ -1 → ∅ → -2
8	varprogression_2	→ +2 → +1 →	→ +2 → +1 →	→ +2 → +1 →	Δ → +2 → ∅ → +1
Figure 7: The embodiment of all 8 rules involved in the joint training stage on the 4 attributes, i.e., (Number, Color, Size, Shape).
In the speaker pre-training stage, the rules involved in each attribute are shown in Figure 8, which do not overlap with those in Figure 7, and have similar semantics to rules No.6, No.7, and No.8 in Figure 7. Therefore, pre-training the speaker with these rules does not provide agents with prior knowledge about the rules in the joint training stage, which only helps the speaker acquire reasoning capabilities to cope with the drifting context.
A.2 A RPM case in rule-RAVEN
Figure 9 shows one RPM case of the rule-RAVEN dataset used in our experiments. The form of this case is consistent with Figure2. Figure2 is just for the convenience of illustration, only 1 attribute of the rule is drawn, but the actual rule consists of 4 attributes, i.e., (Number, Color, Size, Shape).
Figure 8: The embodiment of 4 extra rules involved in the speaker pre-training stage on the 4 attributes, i.e., (Number, Color, Size, Shape). These rules involved in the pre-training phase do not overlap with rules in the joint training stage.
Figure 9: An example problem of the rule-RAVEN dataset with 4 attributes (i.e., (Number, Color, Size, Shape)).
A.3 Details of generalization data splits
Figure 10 shows an example of four levels of generalization data splits described in Section 4.2.
For ID, Inpo-ood, and Expo-ood-L2 generalization levels, we first get  $7^4 = 2401$  rule combinations based on rules No.1 to 7 in Figure 7 on all 4 attributes. We then randomly sample 300 rule combinations from these 2401 rule combinations and generate 10 problems for each rule combination as the Inpo-ood data split (with a total of  $300 \times 10 = 3000$  problems). For each of the remaining  $7^4 - 300 = 2101$  rule combinations, we generate 10 problems as the train data split (for ID, Inpo-ood, and Expo-ood-L2) and 10 problems as the ID data split (with a total of  $2101 \times 10 = 21010$  problems for training and 21010 for ID). We finally combine rule No.8, excluded in the previous process, with the rules on other attributes to get  $8^4 - 7^4 = 1695$  rule combinations and generate 10 problems for each rule combination as Expo-ood-L2 data split (with a total of  $1695 \times 10 = 16950$  problems).
For the Expo-ood-L1 generalization level, we exclude rule No.6 on attribute 1, rule No.3 on attribute 2, rule No.4 on attribute 3, and rule No.8 on attribute 4, getting 2401 rule combinations remaining. By comparing the accuracy difference between Expo-ood-L1 and Expo-ood-L2 levels (both Expo-ood-L1 and Expo-ood-L2 exclude one rule on each attribute, but on Expo-ood-L1 level, rules excluded on one attribute appear on other attributes), we can check whether the language describing the rules can help the listener apply the rules across attributes. To compare fairly with Expo-ood-L2 level, we randomly exclude 300 rule combinations and generate 10 problems for each of the remaining 2101 rule combinations as the training set for Expo-ood-L1 level (with a total of  $2101 \times 10 = 21010$  problems). We combine the rules excluded in the previous process with the rules on other attributes to get  $8^4 - 7^4 = 1695$  rule combinations and generate 10 problems for each rule combination as Expo-ood-L1 data split (with a total of  $1695 \times 10 = 16950$  problems).
Figure 10: An example (with 2 attributes, (attr-1, attr-2)) to illustrate 4 generalization levels.
B Details of Agent Model
The model diagram of the speaker and listener is shown in Figure 11. Some hyperparameters of the model are listed in Table 4. Please refer to the source code in the supplementary materials for more implementation details.
Figure 11: The Speaker's and Listener's Model.
Table 4: Some hyperparameters of the model.
Hyperparameter	Value of N ∈ {20, 30, 40, 80}
fS and fL output dim	[80, 120, 160, 240]
Groups of gS andgL	[80, 120, 160, 240]
Experts of gS andgL	5
gS andgL output dim	[400, 600, 800, 1200]
hS and hL hidden dim	[400, 600, 800, 1200]
C Results of Diverse Language Sizes
The proposed two-stage training method does not introduce constraints on the size of the emerged language (i.e., message_length and vocabulary_size). The reason is that the speaker updates  $f^S$ ,  $g^S$ , and  $h^S$  in the first stage of training, while only load parameters of  $f^S$ ,  $g^S$  (without  $h^S$ ) in the second stage. Therefore, even if the  $\mathbb{1}(r_i, m_i)$  operator on message encoder  $h^S$  constrains the equal size between message  $m_i$  and rule  $r_i$  in the first stage, we can the training a new  $h^S$  with reconfigurable language size in the second stage (joint) training.
We also provide results with diverse language sizes (Table 5) in our reasoning game and get similarly high generalization performance (accuracy  $\sim 0.95$ ).
Table 5: Generalization accuracy with different language sizes (message_length  $M$ , vocabulary_size  $V$ ) and attribute values  $N$ .
(M, V, N)	ID	Inpo-ood
(6, 15, 20)	0.9539 ± 0.0018	0.9528 ± 0.0015
(6, 30, 20)	0.9522 ± 0.0015	0.9513 ± 0.0031
(6, 15, 30)	0.9429 ± 0.0053	0.9385 ± 0.0045
(6, 30, 30)	0.9411 ± 0.0022	0.9362 ± 0.0014
D Token-level Analysis of the Emerged Language
We compute the probability distribution  $P(message|attribute, rule)$  of a randomly selected seed and provide the most probable tokens for a given attribute and rule, as shown in Table 6.
Table 6: Given attributes and rules, the most probable tokens at each position.
Rules	Color	Number	Size	Shape
add	K, C, K, H	B, C, B, C	K, L, K, K	B, K, B, C
minus	B, L, B, B	K, O, G, O	G, G, G, H	J, J, J, J
min	K, C, K, C	J, L, J, C	K, O, K, K	B, B, B, C
max	B, B, B, K	K, O, K, K	K, G, J, B	F, B, K, C
constant	K, B, K, K	B, K, C, K	K, C, K, C	K, B, K, C
progression_2	B, L, B, B	K, O, K, O	G, G, H, H	J, O, J, J
varprogression_-1	K, C, K, C	B, J, J, C	K, O, K, C	B, K, B, C
The results indicate that tokens exhibit regular patterns (i.e., language-like syntax and compositionality) for different attributes and rules. For example, almost all rules related to attribute 'color' start with tokens 'K' or 'B', and attribute 'size' start with tokens 'K' or 'G'. On another dimension, the rule 'varprogression_1' across all attributes ends with token 'C'.