Emergent Communication for Rules Reasoning
Authors: Yuxuan Guo
Yifan Hao
Rui Zhang
Enshuai Zhou
Zidong Du
Xishan Zhang
Xinkai Song
Yuanbo Wen
Yongwei Zhao
Xuehai Zhou
Jiaming Guo
Qi Yi
Shaohui Peng
Di Huang
Ruizhi Chen
Qi Guo
Yunji Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Emergent communication
Multi-agent communication
Raven's Progressive Matrices
Representation learning
TL;DR: We study emergent communication in the cognition-oriented environment (i.e., RPM-based Reasoning Game) and show the emerged language's generalization and transfer ability.

Abstract: Research on emergent communication between deep-learning-based agents has received extensive attention due to its inspiration for linguistics and artificial intelligence. 
  However, previous attempts have hovered around emerging communication under perception-oriented environmental settings,
  that forces agents to describe low-level perceptual features intra image or symbol contexts.
  In this work, inspired by the classic human reasoning test (namely Raven's Progressive Matrix), we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts.
  Moreover, we propose 1) an unbiased dataset (namely rule-RAVEN) as a benchmark to avoid overfitting, 2) and a two-stage curriculum agent training method as a baseline for more stable convergence in the Reasoning Game,
  where contexts and semantics are bilaterally drifting.
  Experimental results show that, in the Reasoning Game, a semantically stable and compositional language emerges to solve reasoning problems.
  The emerged language helps agents apply the extracted rules to the generalization of unseen context attributes, and to the transfer between different context attributes or even tasks.

Emergent Communication for Rules Reasoning
Yuxuan Guol,23
Yifan Hao²
Rui Zhang²
Enshuai Zhou's 1,3,3
Zidong 2
Xishan Zhang
Xinkai Song?
Yuanho Wen
Yongwei Zhao²
Xuchai Zhou!
Jiaming Guo
Yi1.2,3
Shaohui Peng
Di Huang² Yunji Chen
Ruizhi Chen
Guoz
UUniversity of Science and Technology of China State Key Lab of Processors, Institute of Computing Technology, CAS Cambricon Technologies *University of Chinese Academy of Sciences SShanghai Innovation Center for Processor Technologies 6 Intelligeett Software Research Center, Institute of Software, CAS
syx_201708180mail. ustc. edu. en, (hacyifan, cyj)@ict.ac.cu
Abstract
Research emergent communication between deep-learning- braaning--basdd agents has received extensive attention due to its inspiration for linguistics and artificial intelli- gence. However, previous attempts have hovered around emerging communication under perception-oorenteed environmental settings, that forces agents to describe low-level perceptual features intra image symbol contexts. In this work, inspired by the classic human reasoning test (namely Raven's Progressive Matrix), we propose the Reasoning Game, inggitioon-oientted environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Moreover, we propose 1) unbiased dataset (namely rule-RAVEN) a benchmark to avoid overfitting, 2) and two-stage curriculum agent training method as a baseline for more stable convergence in the Reasoning Game, where contexts and semantics are bilaterally drifting. Experimental results show that, in the Reasoning Game, semantically stable and compositional language emerges solve reasoning problems. The emerged language helps agents apply the extracted rules the generalization of unseen context attributes, and to the transfer between different context attributes or even tasks.
Introduction
Research on emergent communication has received extensive attention in recent years due to its inspiration to the fields of linguistics and artificial intelligence (AI). From the linguistic perspective, the study of emergent communication may help understand the origin and evolution of natural language [9, 13, 14, 28, 40]. From the Al perspective, introducing communication mechanism into the deep-learning- based multi-agents system is promising way to make interpretable [11], human-interactable [25], and more importantly, generalizable [1, 7, 37].
Lewis-style Game [27] is widely used in previous studies to simulate emergent communication between two agents, speaker, and listener. In this game, the speaker sends a message based on perceived contexts (usually images or symbols), and the listener completes the perception task (usually discriminating or reconstructing the target object) based on the message. As two agents
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Peseeivod contents
Pereeired ocsteut jummber ciniles:
Resoning Ruies
Number
clices or lines incoaing
Nutieer of polygous 1. C Numberaa 3, 4,2,
!
1,4.4
grey edaes
Commuriatt
3,4,0
sumber of lince
Nimber Incrasing
3,4,1,
2
3,4,3,
Speaker
Lisena
Spesker
Panepliee-ctitreel anvincement
Cogniäun-orimaall cnvinement
Figure 1: Perception-oriented environment (Left, i.e., speak by seeing) V.S. environment (Right, i.e., speak by reasoning) for emergent communication.
cognition-oriented
gradually endow messages with precise semantics describe the target object more accurately, machine language emerges with communication.
However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forcing agents to describe low-level perceptual features (e.g., shape, color and combinations of them) intra image [11, 14, 15, 23, 24, 31] symbol [28, 35-37] contexts. Unfortunately, such cnneeption-orientce environments (i.e., speak by seeing) are unable to prompt agents emerge cognitive- oggnitive-based communication (i.e.. speak by reasoning), which is considered the foundation of human language and intelligence evolution by research in linguistics [12] and cognitive psychology [6]. Therefore, how emerge communication from perception cognition remains an unsolved issue.
In this work, we propose the Reasoning Game, -oriene environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Figure shows the comparison between the perception-oriented environment and cognition-orrenned environ- ment. The Reasoning Game requires two agents to cooperatively solve a classic human reasoning test Raven Progressive Matrix [34] (RPM). Specifically, the speaker reasons underlying rules from two rows of contexts (consisting of visually simple attributes), and sends a message to the listener. Mcanwhile, the listener must correctly parse the received message and grasp the extracted rules, as to select a candidate option that matches the rules (and fills the blank in the last row of the contexts matrix). In the process of solving RPM by agents, the rules communicated are not related to perceived features intra a single context but rather stem from abstract and structural logical relationships inter multiple contexts [39], such as gradually deepening colors and increasing numbers. According to this characteristic of RPM, agents can only complete the Reasoning Game by reasoning implicit rules at the cognitive level, instead of explicit visual features at the perceptual level. Such cognition-oriented environmental settings are considered to characterize the defining feature of high-level human-like intelligence [6. 18, 48].
Morcover, to conveniently evaluate the cognitive ability of agents through the Reasoning Game, we propose 1) an unbiased rule-RAVEN dataset benchmark, and 2) a two-stage curriculum agent training method as a baseline. For the former, the RAVEN dataset contains numerous RPM problems with different combinations of rules, and each problem consists of context panels with multiple attributes. The dataset is debiased by our rule- based candidate panel generation algorithm to avoid overfitting in agent communication. For the latter, the two-stage curriculum agent training method ensures agents converge more stably in the Reasoning Game, where contexts and semantics are bilaterally drifting.
We summarize the contributions of this paper with experimental results:
(Sec 5.1) The Reasoning Game successfully enables the emergent communication for reasoning rules between agents, and the emerged language is semantically stable and compositional.
(Sec 5.2) The proposed benchmark (i.e., the rule-RAVEN dataset) and baseline (i.e., the two- stage curriculum agent training method) in the Reasoning Game effective expected and necessary for agent evaluation.
(Sec 5.3) We verify the improvement of agent cognitive ability through the emergent commu- nication for reasoning rules from three aspects: 1) generalize to unseen rule combinations, 2) apply the extracted rules to other context attributes, 3) and transfer to different tasks.

---

Question panels
(g), 92)
Rule
?
R
(Number Comber-IIccrassinn Increasing)
Context Pamels
C= (e.....,
baca" is "number increasing"
Cundidate panels
Messuge
A (a.....,as) as)
M [M71. mss
baca
Speaker
Listener
Target panel dg y
Figure 2: The Reasoning Game
2 Related Works
Emergent Communication. Using computational models to simulate emergent communication has a long history [21, 22, 43] and has recently been revived with the popularity of deep learning. A. Lazaridou et al. [24] studied emergent communication
the basis of reinforcement learning and deep neural network agents for the first time. Starting from this, dozens of studies focus on exploring the properties of the emerged language from the perspective of linguistics and cognitive psychology, such as alignment of agents [11], efficiency of language [8], capacity of communication channel [23], community interaction [15], ease of teaching [28], culture transmission [14], population heterogeneity [31], and environment scale [10]. In general, these studies adopt perception-oriene environmental settings, restricting the emerged languages to describing low-level perceptual features intra image or symbol contexts. Specifically, previous works [11, 15, 24] emerge communication of intra-context object attributes (e.g., color, shape) and their combinations (e.g., blue OR/AND triangle) [31]. On the contrary, our work focuses on emergent communication in a cognition-orient orienter environment (i.e., Reasoning Game), where agents reason inter-context high-level rules and apply them, so to cooperatively solve reasoning tasks (i.e., RPM).
Raven's Progressive Matrices (RPM). RPM [34] is widely studied human IQ test, which has recently been revived the AI community after can be procedurally generated [44]. Compared with other reasoning benchmarks, such Visual Question Answering [4, 19, 29] and Arithmetic Reasoning [38], RPM focuses on relational reasoning inter multiple contexts, reflecting humans' core cognitive abilities [6]. Based RPM, many synthetic [2, 48] and natural [41] datasets and model designs [17, 46] are proposed to boost the high-level reasoning capability of deep learning models. In this work, are the first to study emergent communication through cooperatively solving RPM problems. We find that such communication scenarios need to ensure clear semantics and avoid overfitting that imposing new requirements the rules design and candidate panels generation of the RPM dataset. Motivated by this, we propose the rule-R mile-RAVER AVEN dataset as a benchmark, and its corresponding two-stage curriculum agent training method as a baseline.
3 Method
3.1
Reasoning Game
The Reasoning Game is a cognition- oriennted environment that requires two agents to solve RPM problems by reasoning and communicating high-level rules. The game can be formalized as 5-tuple: (C, Q, A, R. y). where = (cI...., ci) is set of context panels, Q = (g....., 9m) is a set m question panels, A = (a1.... a,,) is set of candidate panels with 1 target panel and 1 distract panels, R = (ri...., rk)) is a vector that encodes the rules implicit in C' (i.e., hidden to the speaker and listener), and y E {1, {1,...,7) n indicates the index of the target panel, the only panel together with the question panels that match the rule R. Each panel contains several attributes (such as number, color, et al.) T .d,..... (s), and the values of all attributes within a pre-defined range.
Figure shows the scheme of the Reasoning Game. In this scheme, the speaker first reasons the embedding, which represents the rules R implicit in context pancls C, then encodes the embedding
3

---

Contest punels
Question panels
Candidate punelfgeenctteed by method) (ot)...,
Candidate punelogemettdd (-RAVEN)
C- (6))
Q (qt.0)
(8,00 ep 022) a ().4)
=- (3,3)
Ty distat nicins,, peogronsion)
medile imanberl counch
ik X beger pirel mooh
(6, 7' tagal peaci no modily ranch nai
(3.8))
(3, Y% (mta, progewantio)
- (ad4, progresston)
Stabox
Oolor
O
D.3) demernte mtn)
(3.5) modiy curnber onler, no rale morch
Figure 3: (left) A reasoning problem example in the rule- ule-RAVEN dataset, where the candidate panels are generated by introducing distract rules, shown in Algorithm 1. (right) Previous based datasets (e.g., I-RAVEN) generate candidates by hierarchically modifying some attribute values of the target panel, making it easy for the listener to distinguish without considering communication from the speaker.
into a message M = (m1...., m.,) (each token m, is derived from pre-defined vocabulary V) for sending to the listener. Finally, according to the received message M, the listener chooses the target panel from candidate panels A to fill the blank behind the question panel Q. If the reasoning question correctly solved, the listener and the speaker will be rewarded simultaneously.
3.2 Rule-RAVEN dataset
A reasoning problem (e.g., RPM) can be naturally divided into two steps: extracting rules and applying rules. It seems that the reasoning dataset (e.g.. RAVEN) can be straightforwardly used to play the Reasoning Game by assigning the former step (i.e., extracting rules) to the speaker and the latter step (i.e., applying rules) to the listener. However, the premise for such a scheme of Reasoning Game could proceed successfully is that the speaker and listener and understand the extracted rules by emergent communication. To this end, crafting a reasoning dataset as a benchmark still has requirements on the rules set design and candidate panels generation.
effectively transmit
Regarding the rules set design, the mapping relationship between implicit rules and context attributes must be unambiguous emerge semantically clear communication and avoid imbalanced prior probability distribution of rules' frequency in the dataset. Specifically, the design principle is subdivided into two aspects: 1) The rule corresponding each set of attribute values in panels is unique. For example, the attribute (e.g., size) value in context panels (C1, C2 C2) ) is 3), 3),
which matches both rules 'sum" (1 then the attribute value should be filtered out from the dataset. 2) The meanings expressed between different rules do not overlap. For example, when any set of attribute value (a, b) matches the rule "progression ierressson? (a + const = h, + const = c). it also matches the rule 'average" (b atc). In this case, only one of the rules should be retained in the dataset.
3) and 'progression +1 (1+1) = 2,
Regarding the candidate panels generation, distract panels should be generated carefully so as to represent close but not quite correct rules that force the speaker and listener communicate precisely the correct rules. Specifically, the design principle is that each distractor in candidate panels should match a 'distract rule' (as shown in figure 3 "our method'), ), to prevent the listener from inferring the target solely through the existence of regular rules between the question panels and candidate panels (a negative example is shown in figure (right) 'I-RAVEN [17]'), Otherwise, it will cause the speaker to lose the motivation to extract and communicate rules correctly (we will demonstrate this in detail by experiments in Sec 5.2).
Following the above requirements, we propose a symbolic dataset 2 RAVEN, for the Reasoning Game. Figure 3 (leff) shows an example case in the rule-RAVEN dataset (See Appendix A for more details.).
3.3 Model and Training
We build the speaker's and listener's model in the Reasoning Game as follows.
'The RAVER dataset is based on symbols rather than images, mainly because visual recognizability limits the attribute values.

---

Algorithm 1: Rule-based candidate panels generation algorithm
Input: Q, R. distract_panee Bacts_ppaeii____e_ee num
Output: distract_panede
// generate distract rule space
possible_rulee Bulllleruuleeeee =]]
for i=0 to [R| do tmp_rule
for r E rrul_ssere do
if check saack__saatss(((f: iJ. r) then imp imp_nule.append(f)
end end
possible_rulee mnles.append(tmp_nu
end
11 saale_spaceea = cartesian proouut((possibble_mui surbbbleeereee_eeeeeeeeeee R
// generate distract panels
12 distract_panels
for raadaannnmaaae
www.ddoow_wamppeeddssstttteeeses distract pacilllttppplls__aeee num) do
14
tmp_panel=
i=0 |r|| do
16
tmp_attr = reason(rs[i], Q i))
17
wwpp_aanel.append(mm)
end
19
20
end
return distract_panch,
Speaker's model. The speaker consumes context panels and outputs message M. The speaker's policy can be formalized by S(M 8) with trainable parameters 6. Given context panels C, an ML.P-based perception module fs first encodes each panel into an embedding (c))) Then, a reasoning module g extracts the row-level rule embedding of = =gS((f (c))) and r = ( (c))) from multiple context embeddings. We follow the state-of-art RAVEN solver SCL [46]'s core module, shared group MLPs, as the reasoning module. The row-level rule embeddings are average-aggregated to get the rule embedding S mean(rf: rr)) of context panels C. Finally, LSTM [16] message encoder S takes the rule embedding S as initial state = and special token mg <SOS> as start token to generate message M. each timestep / 1, based on current state 2n S (z--1, m)-1), message my sample (soplesssotaaas ftmaz(g((a)) (x)))) where g is linear transformation to project the hidden state space to the vocabulary space. Token generation terminates until the message reaches fixed length special token <EOS> is generated.
Listener's model. The listener consumes message M, question panels Q, and candidate panels A and outputs prediction ÿ. The listener's 's policy can be formalized by M, Q, A; ø) with trainable parameters c. Given question panels Q and candidate panels A. perceptual module fz. first encodes Q and A into corresponding embeddings (f²())) and (a=(a) Meanwhile, an LSTM message decoder h takes a special state zá as the start state to decode message M. At each timestep 2 current state af h- (= my). Message decoding terminates until it reaches the maximum message length |M| or <EOS>, We use the final state z!. as the decoded message embedding, representing the rules implied in context panels C. Then, each candidate embedding is concatenated with the question embedding and fed into the reasoning module g obtain multiple candidate rule embeddings of gf (g) fL(2)) (a,))) e (1.....8, 8). Finally, the message embedding and the candidate rule embedding perform an inner product and softmax operation to obtain the probability distribution so aofttaa(()) = ))) of candidate panels, which is further used compute the loss and the prediction ý.
To ensure that the cognitive capabilities of the speaker and the listener are similar, we build their perception and reasoning module architecture identically. No parameters are shared between the
5

---

speaker and listener to avoid introducing artificial priors about their consensus. More details of the model can be found in Appendix B.
In our game, both rules reasoning and language emergence are highly challenging. We find that jointly training the speaker and listener directly from scratch leads communication failures. The reason is that, in the early stage of joint training, the reasoning context panels and the message semantics are drifting simultaneously, leading the speaker cannot produce helpful messages the listener, and the listener tends to ignore the speaker's 's message. To help agents jump out of the local optimum, we propose a two-stage curriculum training [3] method.
In the first stage, the speaker is mre-trained on a held-out rule set (See Appendix A for more details.) generated dataset acquire some reasoning capabilities to cope with the drifting context. This stage can be considered a supervised learning task with the ground-truth rule vector R as the label and only updating the speaker's parameters 0. Note that this does not impose artificial priors on emerged language since the rule encodings predicted at this stage are absent in the joint training stage. The gradients of the objective function J1 the first stage are:
VaJ1 E 1(r,, log S(m, E R,m, M
Note that although the ((-,)) operator exists in Va./1, this does not introduce constraints on the size of emerged language (i.e., gesssage_lengtie and vocabulary_siae size) in the second stage of training. See Appendix C for more details.
In the second stage, the pre-trained speaker and listener train jointly emerge language. Since discrete communication breaks the end-to-end differentiability. we employ the REINFORCE[45] algorithm. The gradients of the objective function J2 in the second stage are:
VoJ2 = E[1 (9. y) V log S(M[C) + VH[S(MIC))
V J2 (8 y) V log (() 4)
Where X is a hyperparameter controlling entropy regularization.
4 Exprimental Settings
4.1
Basic Settings
Implementation. We use Python3 [42] implement the rule-RAVEN dataset. Our model imple- mentation is based on Pytorch [33] and EGG [20] toolkit. The code is available in supplementary materials.
Dataset preparation. We instantiate our rulc-RAVEN dataset (introduced See 3.2). For each case of reasoning problem, there are 6 context panels (i.e., C' (ci...., )). 2 question panels (i.e., Q = (41, 42, and 8 cadidate panels (i.e., A = (az,..., .g))). Each panel has 4 attributes (i.e., T (number, color, size, shape)), and each attribute have N f20, 30, 40) values (N4 in total). The larger N (more diverse the attribute values) means it is more difficult to identify contexts and reason rules. There are 8 rules that apply to each attribute respectively, resulting in a total of 81 4096 different rule combinations. For each rule combination, We randomly generated 20 different reasoning problem cases (sampling from N' attribute values) with Algorithm (4096 x 20 = 81 920 in total), and half of these cases for training, half for testing. Besides, we use held-out rule to additionally generate 2000 cases for pre- pre-training the speaker in the first stage of agent training (introduced in Sec 3.3).
Communication Channel. We set the maximum message length |M| 4 and the vocabulary size |V] = 15. With this setup. the capacity of the communication channel is 154, sufficient to describe all rule combinations (> 84), and preserve redundant tokens for two-stage curriculum training, and much lower than vocabularies required to directly describe the context panels (~ 10'
Optimization. Agents' parameters are optimized by AdamW [30], with a learning rage of 10 weight decay of 0.01, 31 = 0.99, 32 0.999 and batch size of 512. For the speaker, we set the hyperparameter of entropy regularization À 0.01.

---

4.2 Language Metrics of emergent communication
Generalization abilities. To test the generalization ability of the emerged language, we further create 4 data splits, corresponding to 4 levels of generalization ability: 1) In-distribution generalization (ID). The training and test sets share the same rule combinations but consist of different problems. The ID level tests the language's is ability to generalize to different problems with the same rules. 2) Interpolated out-of- distiriibnt generalization (Inpo-ood). Each rule in the rule set appears in each attribute the train set, but some rule combinations do not appear the train set and form the test set. The Inpo-ood level tests the language's 's ability to generalize to unseen rule combinations based on components. 3) Extrapolated out-of-distribution generalization level-1 (oxpo-ooddL1) Each attribute has a unique rule that does not appear in the train set, and these rules are combined with other rules that appear in the train set to form test set. The Expo-ood-Le level tests the language's ability to describe rules across attributes. 4) Extrapolated out-of-distribution generalization level-2 (Expo-ood-L22 A rule does not appear on any attribute on the train set, and it is combined with other rules present in the train set to form the test set. The Expo-ood- level tests the language's s ability to generalize to new rules.
Topographic similarity (Topsim). Topsim [5] is widely used in previous emergent language studies measure language compositionality. It is defined as the Spearman correlation coefficient of pairwise distances between input and message space. In the input space, we use the normalized Hamming distance between rules and the cosine distance between panels. In the message space, we adopt the Levenshtein [26] distance with the same removal, insertion, and substitution costs. Computing Topsim involves pairwise distances and thus has a computational complexity of O which computationally prohibitive in our environment. In the experiment, we randomly sample 1000 samples per run to calculate Topsim and take the average over 20 runs the final result.
Ease and transfer learning (ETL). ETL [10] captures how fast and well the emergent language is transmitted to new listeners performing different task. We focus the emerged language's transferability between reasoning tasks with different attribute values, N. Specifically, we first dump the deterministic languages (choosing the token greedily with the highest probability) of well-trained speakers under each attribute value setting. We then map deterministic languages in the source task setting and reasoning problems in the target task settings based the rules. Finally, train new randomly initialized listeners using mapped language and reasoning problems and observe the convergence speed and accuracy of the listeners. Our source task setting contains attribute values N E {20, 30, 40), and the target task setting contains N E {20, 30, 40), and an extremely large N 80.
5 Exprimental Results
To show the benefits and effectiveness of emergent communication through the Reasoning Game, we answer the following three questions by the experimental results.
5.1 What do the emerged language tend to describe?
Emerged language prefers to describe rules rather than context panels. We demonstrate this issue from two aspects. 1) Compare which input (rules or context panels) has stronger correlation (measured by Topsim) with the message space. As Table shows, in all three diversities of attribute values (i.e..A N 20, 30, 40), the Topsim between rules and messages (i.e., Rule Message) is always substantially higher than that between context panels and messages (i.c., Panel -Message. This directly proves the conclusion, and the emerged languages are somewhat compositional (because Topsim is also used to measure compositionality) 2) Another indirect evidence comes from the relationship between ID generalization performance (measured by Listeners' accuracy) and attribute value diversities. Specifically, since the capacity of the communication channel is not enough to describe the context panels (154 < 1019), more diverse context panels are expected to exacerbate this shortage, leading to rapid decrease in accuracy. However, paradoxically, the data in each column of Table 2 shows that the accuracy is almost equal as the attribute value diversity increases (from 20 to 30 or 40).
7

---

Table 1: Topsim (mean standard error) between rules/panels and messages on data split with
p-value< 10"
Attribute values
Rule-Message
Panel-Message
20
0.2382 + 0.0117
0.1148 + 0.0122
30
0.2540 0.0130
0.1127 0.0080 0.0937 + 0.0124
40
0.2381 + 0.0289
3.0
02
0.8
TN4h ACAEW
0.5
RWTH 30
A0. 2
DoVER 60
0.4
NO. Pt.
ALK 40
30
Figure 4: Train accuracy under the message- blocked scene. The results are averaged across the performance of 16 randomly initial- ized listeners on 4 randomly generated dataset instances. The shaded region represents the standard deviation.
Figure 5: Train accuracy w/ and w/o two- stage curriculum training. The results are av- eraged across 8 seeds. The shaded region represents the standard deviation.
5.2 Why are the proposed benchmark and baseline effective and necessary?
Rule-RAVEN dataset. We quantitatively compare rule-based candidate panels generation algorithm (i.e., Algorithm 1) in rule-RAVEN with the I-RAVEN, Specifically, by separately training the listener under message-blocked scene (i.e., the speaker's message is forced to constant), we compared the overfitting degree of rule RAVEN and I-RAVEN (the higher the accuracy, the less dependent the listener on the speaker's reasoning about rules, i.e., the higher the overfitting degree). Figure shows that I-RAVEN suffers from overfitting severely (lines with the legend 'I-RAVEN 20/30/40'), for the listener's train accuracy still achieves 0.9 even if the speaker's message is completely ignored. While rule-RAV (lines with the legend 'rule-RAVEN RAVEN_20/3000000 can effectively control the degree of overfitting, the listener's train accuracy is only 0.6, which is significantly different from when receiving the speaker's 's message normally (~~ 0.6 v.s. ~ 0.9). So it enough to distinguish whether communication effective not. Note that controlling overfitting to an ideal level (8 candidate panels, R 0.125 message-free accuracy) is unlikely because the listener can always guess the answer by memorizing the patterns corresponding to specific rules. Besides, naive methods, such as increasing attribute values, do not alleviate overfitting (as shown in lines with the legend I-RAVEN," from 20 to 30 or 40).
Two-stage curriculum training method. We compare the training curves with and without the two-stage curriculum training method. Figure 5 shows that the two-stage curriculum training method leads to successful communication in all three attribute value diversities (lines with the legend "w_pt__a Ww_pt_20/30000 20/30//4// for agents reaches higher accuracy (~ 0.9). In contrast, in the case without the two-stage curriculum training method (lines with the legend "w/o_pt ae/o_pt_20/330 2ot___20/3//0'), agents only converge to a similarly low accuracy as under the message-blocked scene, which implies that agents can not communicate successfully. An intuitive explanation for this phenomenon is that, the speaker's training the first stage helps him acquire some reasoning ability to cope with the drifting context, which helps stabilize the semantic production in the joint training stage.

---

Table 2: Accuracy (mean standard error) of generation levels. 0.4597 + 0.0379
Attribute values
ID
Inpo-ood
Expo-ood-L1
Expo-ood-12
20
0.9492 + 0.0034
0.9473 + 0.0039
0.6590 0.0371
0.4921 + 0.0658
30
0.9369 + 0.0073
0.9295 = 0.0083
0.5880 + 0.0349
0.4684 + 0.0630
40
0.9194 = 0.0190
0.9210 0.0178
0.5741 0.0238
Table 3: Train accuracy (mean + standard error) of task transferring.
Target
20
30
40
80
20
0.9257
0.0059
0.9238 + 0.0051
0.9199 + 0.0081
Source
30
0.9098 + 0.0110
0.9124 + 0.0087
0.9096 + 0.0083
40
0.9022 + 0.0171
0.9043 0.0159
0.8974 0.0136
5.3
What are the benefits of communicating extracted rules?
Beneficial for generalizing to unseen rule combinations. As shown in Table 2, the high generaliza- tion performance the ID level(>> 0.92) shows that communicating abstract rules can generalize different reasoning problems with the same rule combination. Moreover, the generalization perfor- mance of the Inpo-ood level similar that of the ID level, which indicates that communicating abstract rules can also generalize to unseen rule combinations, c.g,, based on (add, min) and (progression, max) on (number, size) generalize to (add, max) on (number, size).
Beneficial for agents apply rules to other attributes. As shown in Table 2, we observe that the performance of level Expo-ood-L1 is higher than that of Expo-ood-L22 +0.13), which suggests that communicating abstract rules can help the listener to apply rules extracted from one attribute to others, e.g., applying rule (progression similar to the human ability to draw inferences from one instance to another. Similar to the observation in Chaabouni et al. [10], complicating the task of language emergence (reasoning task in our work) does not improve the systematicness of emergent language (from average Inpo-oods 0.92, to Expo-ood-a L1 x 0.58, and to Expo-ood-L2 0.46), possibly because reasoning about unseen rules is highly challenging in our game.
extracted from (number) (size). This characteristic is
Beneficial for transferring to different task setup. As shown in Table all transfers in the table have relatively high train accuracy and are close to the upper bound (ID and Inpo-ood accuracy). Even languages that emerged the simplest task (20 values of each attribute) can be transferred to the most challenging task (80 values of each attribute). Figure 6 shows the training curve of task transferring grouped by target attribute values. Emerged language (lines with the legend 'agent') has a similar convergence speed compared with the ideal language that directly uses rule encoding as the language (lines with the legend 'rule') and only hundreds of optimization steps (less than 10 epochs) to converge. Besides, emerged language outperformed the random group (lines with the legend 'random') " which randomly maps source languages and the target reasoning problems. Good cross-task transfcrability also indirectly confirms that the emerged languages tend to describe rules more than context panels.
(a) Transferring to 20.
(b) Transferring to 30.
(c) Transferring to 40.
(d) Transferring to 80.
Figure Training curve of task transferring grouped by target attribute values. The results are averaged across 8 seeds, and the shaded region represents the standard deviation.

---

Discussion and Conclusion
In this work, we propose the Reasoning Game, cognition-oriented environment that encourages agents to reason and communicate high-level rules. Based on our rule-RAVEN dataset and two- stage curriculum agent training method, we successfully observe the emergence of language that can describe abstract rules and further reveal the language's 's impressive generalization ability and transferability.
Our research may inspire the following directions: 1) Emerging communication on more realistic- stimuli-bass baseld reasoning datasets [32, 41, 47] to study the co-evolution of perception, cognition, and language system in agents; 2) Debiasing and avoiding overfitting on the procedurally generated reasoning dataset from more fine-graines perspective; 3) Designing pre-training tasks to converge in environments with multiple variable drifts.
Limitations
We summarized the limitation of our works follow: 1) Our work only focuses language emergence on a clean symbolic-based reasoning dataset, lacking the exploration of more realistic stimuli-based (e.g., symthetic natural images) reasoning datasets; 2) The reasoning task (RAVEN) adopted in our work only requires the agent to complete the reasoning via single round interaction, simplifying the natural reasoning process; 3) Our work only analyzes the semantics of the emerged languages at the message level, lacking fine-graneed structural (e.g., gramma) and semantic analysis the token level (e.g., the degree of polysemy and ambiguity between tokens).
Acknowledgments and Disclosure of Funding
This work is partially supported by the NSF of China (under Grants 61925208, U22A2028, 62102399, 62222214, 62102398, U19B2019), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
References
[1] Michal Auersperger and Pavel Pecina. Defending compositionality in emergent languages. arXiv preprint arXiv:2206.0475: 2022.
[2] David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning, pages 511-520 PMLR, 2018
[3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009.
[4] Yonatan Bisk, Kevin Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial operations in a rich 3d blocks world. Proceedings of the AAAI Conference Artificial Intelligence, 32, 12 2017.
[5] Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life, 12(2):229-242 2006.
[6] Patricia Carpenter, Marcel Just, and Peter Shell. What one intelligence test measures: A theoretical account of the processing in the raven progressive matrices test. Psychological review, 97:404 31, 08 1990.
[7] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4427 4442, Online, July 2020. Association for Computational Linguistics.
[8] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems, 32, 2019.
10

---

[9] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient noming systems. Proceedings of the National Academy of Sciences, 1 (18(12):e20163))) 65691 18, 2021.
[10] Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International Conference on Learning Representations, 2022.
[11] Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional communi- cation learning from raw visual input. In International Conference on Learning Representations, 2018.
[12] Noam Chomsky, Language and mind: current thoughts on ancient problems. Variation and universals in biolinguistics, pages 379-405. Brill, 2004.
[13] Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of compositional language with deep generational transmission, 2019.
[14] Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Co-evolution of language and agents in referential games. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 2993-3004, Online, April 2021. Association for Computational Linguistics.
[15] Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 3700-3710, Hong Kong, China, November 2019. Association for Computational Linguistics.
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1735 1997.
[17] Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 1567-1574, 2021.
[18] Susanne M. Jaeggi, Martin Buschkuehl, John Jonides, and Walter J. Perrig. Improving fluid intelligence with training on working memory. Proceedings the National Academy of Sciences, 105(19):6829-66) 6833, 2008
[19] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[20] Eugene Kharitonov, Roberto Dessi, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG; toolkit for research on Emergence of lanGuage in Games. https://gi thub. com/ facebookresearch/FOn 2021.
[21] Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the labora- tory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences, (1033)):10681106663 2008
[22] Simon Kirby, Kenny Smith, and Henry Brighton. From ug universals: Linguistic adaptation through iterated learning. Studies in Language, v.28, 587-607 (2004), 28, 09 2004.
[23] Satwik Kottur, Jose Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge naturally in multi-agent dialog. In Proceedings of the 20/7 Conference on Empirical Methods in Natural anguage Processing, pages 2962-2967, Copenhagen, Denmark, Sep 2017. A Association for Computational Linguistics.

---

[24] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. International Conference on Learning Representations, 2017.
[25] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674, Online, July 2020. Association for Computational Linguistics.
[26] Vladimir losifovich Levenshtein. Binary codes capable of correcting deletions, insertions and reversals. Soviel Physics Doklady, 10(8):707-710, 1966. Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.
[27] David Lewis. Convention: philosophical study. John Wiley & Sons, 2008
[28] Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. Advances in neural information processing systems, 32, 2019.
[29] Zechen Li and Anders Søgaard. QLEVR: A diagnostic dataset for quantificational language and elementary visual reasoning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 980-996, Seattle, United States, July 2022, Association for Computational Linguistics.
[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[31] Jesse Mu and Noah Goodman. Emergent communication of generalizations. In A. Beygelzimer,
Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[32] Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Animashree Anandkumar. Bongard-logo: A new benchmark for human-level concept learning and reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc., 2019.
[34] J. C. Raven and Australian Council for Educational Research. Raven's progressive matrices (1938): sets A, B, C, D, E. Australian Council for Educational Research Melbourne, 1938.
[35] Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In International Conference on Learning Representations, 2020.
[36] Mathieu Rita, Florian Strub, Bean-Bastien Grill, Olivier Pietquin, and Emmanuel Dupoux. On the role of population heterogeneity in emergent communication. In International Conference on Learning Representations, 2022.
[37] Mathieu Rita, Corentin Tallec, Paul Michel, Lean-Bastien Grill, Olivier Pietquin, Emmanuel Dupoux, and Florian Strub. Emergent communication: Generalization and overfitting in lewis games. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[38] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
[39] E Snow, C Kyllonen, and Marshalek. The topography of learning and ability correlations. Advances in the psychology of human intelligence, 2:47-103, 1984.
12

---

[40] Luc Steels. The synthetic modeling of language origins. Evolution of commmiication, 1(1):1-34, 1997.
[41] Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton Hengel. V-prom: A benchmark for visual reasoning using visual progressive matrices. Proceedings of the AAAI Conference on Artificial Intelligence, 10:11001112078 04 2020.
[42] Guido Van Rossum and Fred L.. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009.
[43] Kyle Wagner, James A. Reggia, Juan Uriagereka, and Gerald Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37-69, 2003,
[44] Ke Wang and Zhendong Su. Automatic generation of raven's progressive matrices. In Proceed- ings of the 24th International Conference on Artificial Intelligence, IJCAI'15, page 903-909. AAAI Press, 2015.
[45] Ronald J. Williams. Simple statistical gradient- following algorithms for connectionist reinforce- ment learning. Mach. Learn., 8(3-4):229225 256, may 1992.
[46] Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint 2rXiv:2007.04212, 2020.
[47] Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A bench- mark for compositional visual reasoning. In Thirty-sixie sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[48] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
13

---

A Details of rule-RAVEN dataset
A.1 Rules in rule-RAVEN
In the joint training stage, specific rule contains 4 elements on different attributes, c.g.. (Number: add, Color: progression_2, Size: constant, Shape max)).max) These rules meet the structural and functional requirements stated in Section 3.2 The embodiment of each rule on each attribute are shown Figure 7. For example, the 'Color: progress ion_2 means: the same row of panels, the image color gradually darkens by 2 degrees from left to right.
Attributzs
No.
Rales
Sbape
add
atone
TAX
progresston_a
rarpregrensten_.t
varprogreecion..
Figure 7: The embodiment of all rules involved in the joint training stage on the attributes, i.c., (Number, Color, Size, Shape).
In the speaker pre-training stage, the rules involved in each attribute are shown in Figure 8, which do overlap with those Figure and have similar semantics rules No.6, No.7, and No.8 in Figure 7. Therefore, pre-training the speaker with these rules does not provide agents with prior knowledge about the rules in the joint training stage, which only helps the speaker acquire reasoning capabilities to cope with the drifting context.
A.2 A RPM case in rule-RAVEN
Figure shows RPM of the rule-RAVEN dataset used in our experiments. The form of this case is consistent with Figure2. Figure2 is just for the convenience of illustration, only attribute of the rule drawn, but the actual rule consists of 4 attributes. i.e., (Number, Color, Size, Shape).
14

---

Rales
Bumber
Color
Bize
Shape
progreazion_
progresstio_-1
progressico_-2
vaprogreasiou..
The embodiment of 4 extra rules involved in the speaker pre-training stage on the attributes, i.e., (Number, Color, Size, Shape). These rules involved in the pre-training phase do not overlap with rules in the joint training stage.
Figure 8:
Oedees pands
Canddaac pands (())))))))
Torge nadd tegget vilu
(LA84)
"L
paral
Qaodieipards
(5,41-0)))
4e 0.4A%
Q
Distect pand, runch dismer rile fadd, ma, sia,
Distoct perel meah fitrot nic
Intzes, SAZ, Baz, 2tt)
(5.7.500
Doarci pand, ranch deme
Dbtttc parel reacc dotract nil
famber:
(ali, Aeggragreeeeen... aMs, mal)
(etwa,, vwppe aee mas, aaA)
Figure 9; An example problem of the nile-AVEEN dataset with 4 attributes (i.e., (Number, Color, Size, Shape)).
15

---

A.3 Details of generalization data splits
Figure 10 shows an example of four levels of generalization data splits described in Section 4.2.
For ID, Inpo-ood, and Expo-ood-L2 generalization levels, we first get 74 = 2401 rule combina- tions based on rules No.1 to 7 in Figure 7 on all 4 attributes. We then randomly sample 300 rule combinations from these 2401 rule combinations and generate 10 problems for each rule combination the Inpo-ood ood data split (with total of 300 10 3000 problems). For each of the remaining 71-300 = 2101 rule combinations, we generate 10 problems as the train data split (for ID, Inpo-ood, and ood-L2 and 10 problems as the ID data split (with total of 2101 10 21010 problems for training and 21010 for ID). We finally combine rule No.8, excluded in the previous process, with the rules on other attributes to get 8 74 1695 rule combinations and generate 10 problems for each rule combination as Expo-ood- ood-12 L2 data split (with total of 1695 10 =
16950 problems).
For the Expo-oode attribute 2, rule No.4 on attribute 3, and rule No.8 on attribute 4, getting 2401 rule combinations remaining. By comparing the accuracy difference between Expo-ood-L1 and Expo-ood-L2 levels (both Expo-ood-L1 and Expo-ood-12 exclude one rule on each attribute, but on Expo-ood-Li level, rules excluded describing the rules can help the listener apply the rules across attributes. To compare fairly with Expo-ood-L2 level, randomly exclude 300 rule combinations and generate 10 problems for each of the remaining 2101 rule combinations as the training set for expo-ood-L1 level (with total of 2101 10 21010 problems). We combine the rules excluded in the previous process with the rules on other attributes to get 8+ 7 = 1695 rule combinations and generate 10 problems for each rule combination as Expo-oor sxppo-oddlle L1 data split (with a total of 1695 x 10 16950 problems).
generalization level, exclude rule No.6 attribute rule No.3 on
attribute appear other attributes),
check whether the language
Train
Test
Train
Test
attr-2
attr-2
attr-2
attr-2
attr-1
attr-1
attr-1
attr-1
Inpo-ood
Train
Test
Train
Test
attr-2
attr-2
j
attr-1
attr-1
attr-1
==
Expo-ood-Li
Expo-ood-12
Figure 10: An example (with attributes, (attr-1, attr-2)) illustrate generalization levels.
16

---

B Details of Agent Model
The model diagram of the speaker and listener is shown in Figure 11. Some hyperparameters of the model are listed in Table 4. Please refer to the source code in the supplementary materials for more implementation details.
Row-keve nile embeddingg
Contect embeddinngs
Contest
LSTM Message Inceder
(
((s)
(aa
(oa)
<808>
(ma
Speaker's
Message Decodar
Question pancla
f'a
(
(a:)
emboedding
Start abate
(et.os)
(w)
(oa)
((ax)
A
Candidate panels
(a))....da)
(ea]
mi
Listene''s Model
Figure 11: The Speaker's and Listener's Model.
Table 4: Some hyperparameters of the model.
Hyperparameter
Value of V E {20, 30, 40, 80)
S and f output dim
80, 120, 160, 240]
Groups of و and gt Experts of gd and g² go and g" output dim h and h- hidden dim
[80, 120, 160, 240] 5 [400, 600, 800, 1200]
[400, 600, 800, 1200]
C Results of Diverse Language Sizes
The proposed two-stage training method does not introduce constraints on the size of the emerged language (i.e., message message_length and pocabulary_sizer The reason is that the speaker updates fs, g".gs, and h in the first stage of training, while only load parameters of fs, g (without h5) in the second stage. Therefore, even the 1(ri, m;) operator on message encoder h constrains the equal size between message m, and rule in the first stage, we can the training new h with reconfigurable language size in the second stage (joint) training.
We also provide results with diverse language sizes (Table 5) in our reasoning game and get similarly high generalization performance (accuracy 0.95).
Table 5: Generalization accuracy with different language sizes (message__engse M, vocabulary_size V) and attribute values N.
(M,V.N)
ID
Inpo-ood
(6, 15,
0.9539 + 0.0018
0.9528 = 0.0015
(6, 30, 20)
0.9522 = 0.0015
0.9513 = 0.0031
(6. 15.30)
0.9429 + 0.0053
0.9385 = 0.0045
(6, 30)
0.9411 0.0022
0.9362 = 0.0014
17

---

D Token-level Analysis of the Emerged Language
We compute the probability distribution P(message attribute, rule) randomly selected seed and provide the most probable tokens for given attribute and rule, as shown in Table 6.
Table 6: Given attributes and rules, the most probable tokens each position.
Rules
Color
Mumber
Size
Shape
add
K, K.
B. C
K. B, C J.J.J,J B.B.
minus
B
O,O.G.
G.G.G.H
min
K, C.
J,L.J.C
K.O.K.K
max constant
B, B. K
K,O.K,
K,G.J.B C,B.K,C K.C.K.C G.G.H. J,O.J,J
B, K
K, K,O,K,O
K,B.K.C
progression_2
L,
varprogreasion.-1
C,
B,J.J.C K,O,K,C B,K.B.C
The results indicate that tokens exhibit regular patterns (i.e., language-like syntax and compositional- ity) for different attributes and rules. For example, almost all rules related to attribute 'color' start with tokens 'K' or BB', and attribute 'size' start with tokens 'K' or 'G', On another dimension, the rule vvrrpogrrsssoon_. across all attributes ends with token 'C'.
18


Reviews:
[{"review_id": "R01MptBg95", "reviewer": "Reviewer_LuwD", "summary": "This paper proposes an emergent communication game over abstract visual concepts, inspired by Raven's progressive matrices tests. The basic idea is to evaluate neural speakers and listeners on a communication game, where the speaker sees a collection of images encoding some abstract rule (e.g. \"number of objects in the image is increasing\"); the speaker must then generate a message that allows a listener to complete an unseen sequence. The authors show that agents trained to play this game indeed seem to learn to communicate the abstract rules for which they are trained for, as measured by intrinsic measures of language compositionality and ease of transfer to harder tasks.", "questions": "- Did authors try varying the number of context panels given to the speaker? Or even show the partial sequence given to the listener? Wonder how this affects the languages' propensity to communicate abstractions; e.g. if the speaker sees the listener's partial sequence, does the language still communicate the abstract rule, or does the speaker internally learn the abstract rule but nevertheless convey the perceptual features (e.g. \"single triangle\")?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- This is an interesting dataset and interesting problem in emergent communication which may be useful to the community. It indeed explores more abstract visual concepts than in existing work (though note that novelty over the existing EC literature is overclaimed; see Weaknesses).\n- Careful controls for dataset difficulty (ensuring one distinct feature that can be used to solve each task; ensuring \"hard negative\" rules) show the authors' care to making sure this is a well-constructed dataset, including an analysis of to what extent existing \n- Interesting experimental analysis shows that models seem to be (to some extent) communicating abstract rules, rather than superficial input features.", "weaknesses": "- Only a synthetic dataset consisting of clean symbolic inputs is evaluated. One could imagine more realistic settings requiring communication of rules at least over synthetic visual inputs, if not more realistic visual concepts. Similarly, there is no exploration of downstream transfer to other tasks that perhaps don't involve emergent communication, e.g. instruction following or visual reasoning. While this does not preclude publication, there's not a lot one can gain from this paper as it relates to actual realistic ML tasks. If the paper were to be rejected, IMO it would likely be because the experiments are just a little too synthetic/marginal to be useful to the broader NeurIPS community.\n- The claim that existing work in EC does not at all care about expresing abstract generalizations or rules is a bit overblown. Separating inputs given to the student and teacher, so as to facilitate communication of abstract concepts, was introduced as early as Lazaridou (2017), recurs in Choi et al., Kiela et al., etc. Mu and Goodman (2022) also propose generalizations over abstract visual concepts involving multiple visual inputs, which is very similar to the task presented here. I do think the present work makes some interesting contributions over the existing literature, in that it is even more abstract, but the relation to existing work needs to be made more clear. Many of these papers are not discussed in detail and simply bucketed as  \"forcing agents to descrie low-level features of images\" (L31-32) which I believe is false. Section 2 Emergent Communication also completely neglects to discuss such efforts in the EC community.\n- I think it's important for footnote 1 to be made more clear in the text, i.e. that this is not a grounded communication game over real images, despite many of the introductory figures seemingly suggesting this.\n\n## Minor\n\n- Title of paper and title on OpenReview do not match\n- spaces between text and citations would be ideal\n- It'd be interesting to see how pretraning agents on such visual reasoning communication tasks might improve performance on downstream visual reasoning tasks such as ARC (Chollet et al., ?)\n- The description of the paragraphs in L119 and L128 as \"structural\" and \"functional\" requirements is a little confusing and nonstandard to me—it's not clear what structural and functional mean here. It might be appropriate for example to refer to the \"functional requirement\" as sampling \"hard negative distractors\", as is used in the terminology for contrastive learning for example. In other words, distractors should be sampled carefully so as to represent close but not quite correct rules that force the speaker and listener to communicate precisely the right rules.\n- L171 \"directly from the sketch\" -> \"directly from scratch\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments. We would like to clarify the concerns as follows:\n\n1. > Downstream transfer to visual reasoning tasks, and set more realistic environments requiring communication of rules over visual (image) inputs.\n\n   \n\n   We demonstrate the emerged language's transfer performance on image-based downstream tasks. Specifically, we first generate symbolic data with $Number \\in \\\\{1, \\dots 9\\\\}$, $Color \\in \\\\{1, \\dots 9\\\\}$, $Shape \\in \\\\{3, \\dots 9\\\\}$ (triangle, square, ..., nonagon), and $Size \\in \\\\{1, \\dots, 9\\\\}$ using the rule-RAVEN dataset. We then implemented a render to draw the panel's symbol as a $320\\times320$ grayscale image. Finally, we train new listeners using the message from the symbolic environment and question-candidate panel images (we replace $f^L$ with a 5-layer ConvNet to process the image input). After 20 epochs of training, the training accuracy of the listener is in Table 1.\n\n   \n\n   **Table 1**: Transfer accuracy of language emerged on symbolic reasoning task to image-based downstream reasoning tasks ('agent': language from a well-trained speaker; random:  a random language).\n\n   | Attribute values |        agent        |       random        |\n   | :--------------: | :-----------------: | :-----------------: |\n   |        20        | $0.9193 \\pm 0.0029$ | $0.3519 \\pm 0.0245$ |\n   |        30        | $0.9168 \\pm 0.0084$ | $0.3030 \\pm 0.0182$ |\n   |        40        | $0.8996 \\pm 0.0109$ | $0.3406 \\pm 0.0618$ |\n\n   The results show that languages emerging in symbolic environments can transfer (with ~ 0.9 accuracy) to downstream visual reasoning tasks. \n\n   \n\n   In fact, our symbolic rule-RAVEN dataset is sufficient to encourage agents to reason and communicate high-level rules. The reason is that, without changing the semantic information, the format of input data (e.g., visual or symbolic) only affects agents' perception but does not affect agents' cognition ability for rules reasoning.\n\n   \n\n   Furthermore, we tend to investigate the emergence of language on more realistic (e.g., V-Prom [1]) and complex (e.g., ARC [2]) image-based reasoning datasets in future work. \n\n   \n\n   Refs:\n\n   [1] Teney, D., & van den Hengel, A. (2020). V-prom: A benchmark for visual reasoning using visual progressive matrices.\n\n   [2] Chollet, F. (2019). On the measure of intelligence.\n\n   \n\n2. > Relation to existing work, which also 'communicate abstract concepts' (e.g., [3-6]), needs to be made more clear.\n\n   \n\n   The original claim in related work does indeed lead to misunderstandings, and we do not deny that the previous work also 'communicate abstract concepts'. Based on the different definitions of \"abstract concepts\", we would do a more fine-grained comparison with previous work and revise the paper. \n\n   We take the several papers you mentioned ([3-6]) as examples:\n\n   - The 'abstract concepts' refers to inner-context object attributes (e.g., color, shape) in [3-5] or combinations of them (e.g., blue OR/AND triangle) in [6]. \n   - While in our RPM-based reasoning game, the 'abstract concepts' refers to extracting and using inter-context rules to handle reasoning problems rather than selecting which object has specified attributes.\n\n   Refs:\n\n   [3] Lazaridou, A. (2016). Multi-agent cooperation and the emergence of (natural) language.\n\n   [4] Choi, E. (2018). Multi-agent compositional communication learning from raw visual input.\n\n   [5] Graesser. (2019). Emergent linguistic phenomena in multi-agent communication games.\n\n   [6] Mu, J., & Goodman, N. (2021). Emergent communication of generalizations.\n\n   \n\n3. > Footnote 1 (i.e., that this is not a grounded communication game over real images) needs to be made more clear in the text.\n\n   \n\n   We point out that rule-RAVEN is a symbolic dataset in line 136 of the paper (last paragraph in section 3.2) and just give the reason in footnote 1. Thanks for pointing out misunderstandings that may arise here. We will more clearly illustrate that rule-RAVEN is a symbolic dataset and revise the paper.\n\n   \n\n4. > Try varying the number of context panels given to the speaker or listener, and research how the setting affects the languages' propensity to communicate abstractions.\n\n   \n\n   We tried a new game setting: 1) the speaker receives the 6 context panels and 2 question panels, reasons the answer, and sends messages to the listener, and 2) the listener selects the target panel from the 8 candidate panels, only referring to the speaker's message.\n\n   \n\n   Experimental results (Table 2) show that such a setting will lead to a slight decrease in generalization accuracy because this setting compels the speaker not only reasoning but also to describe the target panel accurately to the listener.\n\n   \n\n   **Table 2**: Generalization accuracy on ID and Inpo-ood data splits.\n\n   | Attribute values |         ID          |      Inpo-ood       |\n   | :--------------: | :-----------------: | :-----------------: |\n   |        20        | $0.7273 \\pm 0.0728$ | $0.7299 \\pm 0.0725$ |\n   |        30        | $0.7996 \\pm 0.0185$ | $0.7952 \\pm 0.0172$ |\n\n   Moreover, there is evidence (Table 3) showing that such a setting leads the speaker tends to learn the abstract rule internally but convey the perceptual features of the target panel (Topsim(panel, message) > Topsim(rule, message)).\n\n   \n\n   **Table 3**: Topsim between rules/target-panel and messages.\n\n   | Attribute values |    Rule-Message     |    Panel-Message    |\n   | :--------------: | :-----------------: | :-----------------: |\n   |        20        | $0.1776 \\pm 0.0137$ | $0.2249 \\pm 0.0683$ |\n   |        30        | $0.1571 \\pm 0.0272$ | $0.2280 \\pm 0.0306$ |\n\n   \n\n5. > About minor questions.\n\n   \n\n   (1) Align to the title on the Openreview site, we will revise the paper's title.\n\n   (2) Real visual inputs related see the answer of Question 1.\n\n   (3) We will revise text format and typos, and improve writing in specified paragraphs (L119 and L128)."}, {"review_id": "IXRyWJcKYy", "reviewer": "Reviewer_3wvB", "summary": "This paper introduces a novel setting for abstract reasoning (i.e., RAVEN problems) by proposing a speaker-listener framework for communicating higher-level abstract rules. The authors propose an unbiased dataset (rule-RAVEN) to overcome overfitting in the original RAVEN-family datasets (I-RAVEN), and propose a two-stage curriculum agent training method for successful communication. Experiments have shown the efficacy of the curriculum training for solving the rule-RAVEN and out-of-distribution generalization.", "questions": "See the weakness section. I am open to changing my score, so I hope the authors can address these concerns.", "limitations": "The authors did not address the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I like the idea of both: i). introducing communicative game settings to abstract reasoning tasks, and ii). see how higher-level relational abstractions (instead of low-level perceptual features) emerge in communicative games. The limited capacity communication channel formulation can lead to emergent abstractions for problem-solving, including more powerful representations for abstract reasoning and concept learning. Previous attempts in drawing are good cases but not complex enough to depict the importance of abstraction and emergent language. This preliminary trial on RAVEN tests sets a suitable problem formulation for emergent communication in abstract reasoning.\n\n- The rule-RAVEN dataset effectively mitigates the existing bias in the I-RAVEN dataset, making the speaker-listener communication valid.\n\n- The paper is well-written and easy to read. The flow of writing in section 5 is also appropriate for addressing potential concerns for readers.\n", "weaknesses": "Although I like the task settings in this paper, the experiment and proposed methods appear to have some weaknesses. I list them as follows:\n\n- The communicative formulation is very similar to Mu & Goodman, 2021. It seems this work (communicative RAVEN) is a special case of generalization, shifting from learning object-centric, attribute-level concepts (e.g., shape red or blue) to learning relational concepts (number-increasing). Authors should address more comparisons to these existing formulations.\n\n- The evaluation for emergent language is still quite limited. For example, can you probe the learned language to see if it can be linearly projected to some algebraic representations for relational concepts (e.g., the \"number increasing\" concept can be described as a multiplication matrix in Zhang et al., 2022) or just explicitly manipulate them and see if they have some language-like syntax or compositionality emerged.\n\n- The use of symbolic RAVEN and two-stage curriculum training (with the first stage supervised learned) made me doubt the applicability of this communicative method to more complex or real-world tasks. For example, Mu & Goodman, 2021 used a real-world dataset, pixel input, and end-to-end training.\n\nrefs:\n1. Mu, J., & Goodman, N. (2021). Emergent communication of generalizations. Advances in Neural Information Processing Systems, 34, 17994-18007.\n\n2. Zhang, C., Xie, S., Jia, B., Wu, Y. N., Zhu, S. C., & Zhu, Y. (2022, October). Learning algebraic representation for systematic generalization in abstract reasoning. In European Conference on Computer Vision (pp. 692-709). Cham: Springer Nature Switzerland.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments. We would like to clarify the concerns as follows:\n\n1. > More comparisons to existing formulations [1].\n\n   \n\n   Similar to many previous works, the agent communicative formulation (i.e., the definition of 'abstract concepts') in [1] identifies the **inner-context** object attributes (e.g., shape, color). While in our RPM-based reasoning game, the agent communicative formulation is extracting and using **inter-context** rules to handle reasoning problems. We will classify previous work based on different communicative formulations and revise the related work in our paper.\n\n   \n\n2. > Explore if the emergent language has some language-like syntax or compositionality.\n\n   \n\n   Inspired by [2], we compute the probability distribution $P(token\\_sequence|attribute, rule)$ under randomly selected seeds and give the most probable tokens for a given attribute and rule, as shown in Table 1.\n\n   **Table 1**: Given attributes and rules, the most probable tokens at each position.\n\n   |       Rules       |       color        | number         | size               | shape          |\n   | :---------------: | :----------------: | -------------- | ------------------ | -------------- |\n   |        add        |   **K**, C, K, H   | B, C, B, C     | **K**, L, K, K     | B, K, B, C     |\n   |       minus       |   **B**, L, B, B   | K, O, G, O     | **G**, G, G, H     | J, J, J, J     |\n   |        min        |   **K**, C, K, C   | J, L, J, C     | **K**, O, K, K     | B, B, B, C     |\n   |        max        |   **B**, B, B, K   | K, O, K, K     | **K**, G, J, B     | F, B, K, C     |\n   |     constant      |   **K**, B, K, K   | B, K, C, K     | **K**, C, K, C     | K, B, K, C     |\n   |   progression_2   |   **B**, L, B, B   | K, O, K, O     | **G**, G, H, H     | J, O, J, J     |\n   | varprogression_-1 | **K**, C, K, **C** | B, J, J, **C** | **K**, O, K, **C** | B, K, B, **C** |\n\n   The results indicate that tokens exhibit regular patterns (i.e., language-like syntax and compositionality) for different attributes and rules. For example, almost all rules related to attribute 'color' start with tokens 'K' and 'B', and attribute 'size' start with tokens 'K' and 'G'. On another dimension, the rule 'varprogression_-1' across all attributes ends with token 'C'.\n\n   \n\n3. > Apply the two-stage curriculum training method to real-world tasks [1] with pixel input, and verify its effectiveness.\n\n   \n\n   First, to verify that our two-stage training method can be applied to real-world datasets, we demonstrate the transferability of language produced by two-stage trained agents on image-based (i.e., pixel input) downstream tasks. Specifically,  we firstly generate symbolic data with $Number \\in \\\\{1, \\dots 9\\\\}$, $Color \\in \\\\{1, \\dots 9\\\\}$, $Shape \\in \\\\{3, \\dots 9\\\\}$ (triangle, square, ..., nonagon), and $Size \\in \\\\{1, \\dots, 9\\\\}$ using the rule-RAVEN dataset. We then implemented a render to draw the panel's symbol as a $320\\times320$ grayscale image. Finally, we train new listeners using the message from the symbolic environment and question-candidate panel images (we replace $f^L$ with a 5-layer ConvNet to process the image input). After 20 epochs of training, the training accuracy of the listener is in Table 2.\n\n   \n\n   **Table 2**: Transfer accuracy of language emerged on symbolic reasoning task to image-based downstream reasoning tasks ('Agent' represents using the language emerged by a well-trained speaker, and 'random' represents using a random language).\n\n   | Attribute values |        agent        |       random        |\n   | :--------------: | :-----------------: | :-----------------: |\n   |        20        | $0.9193 \\pm 0.0029$ | $0.3519 \\pm 0.0245$ |\n   |        30        | $0.9168 \\pm 0.0084$ | $0.3030 \\pm 0.0182$ |\n   |        40        | $0.8996 \\pm 0.0109$ | $0.3406 \\pm 0.0618$ |\n\n   The results show that languages emerging in symbolic environments can transfer (with ~ 0.9 accuracy) to downstream visual reasoning tasks.\n\n   \n\n   Second, we qualitatively analyze why the two-stage training method (especially the first stage) is still effective for real-world datasets. The first stage (i.e., supervised training) aims to warm up the speaker, enabling it to generate higher-quality messages during early epochs. This stage is crucial in preventing communication from getting trapped in a local optimum. For instance, in our work, we aim to avoid situations where the listener completely disregards the speaker’s message or where the speaker only conveys partial information.\n\n   \n\n4. > About limitation.\n\n   We summarized the limitation of our works as follow:\n\n   - Our work only focuses on language emergence on a clean symbolic-based reasoning dataset, lacking the exploration of more realistic stimuli-based (e.g., synthetic or natural images) reasoning datasets. We supplemented the transfer performance in image-based downstream reasoning tasks during the rebuttal (related results in reviewer LuwD Q1), and further investigation of the language emergence of image stimuli is still needed.\n\n   - The reasoning task (RAVEN) adopted in our work only requires the agent to complete the reasoning via a single round of interaction, simplifying the natural reasoning process.\n\n   - Our work only analyzes the semantics of the emerged languages at the message level, lacking fine-grained structural (gramma) and semantic analysis at the token level. We supplemented with coarse-grained token-level semantic analysis experiments during rebuttal (related results in Q2), and we would do more systematic analyzes further (e.g., the similarity of languages when given attributes and rules, and the degree of polysemy and ambiguity between tokens).\n\nRefs:\n\n[1] Mu, J., & Goodman, N. (2021). Emergent communication of generalizations.\n\n[2] Zhang, C., Xie, S., Jia, B., Wu, Y. N., Zhu, S. C., & Zhu, Y. (2022, October). Learning algebraic representation for systematic generalization in abstract reasoning."}, {"review_id": "7CcuogRWEj", "reviewer": "Reviewer_Ntdv", "summary": "This paper proposed a new environment along with the training framework for emergent communication of abstract rules. They designed a context generation pipeline rule-RAVEN to avoid overfitting and a two-stage curriculum training method for more stable convergence. They evaluated the emerged language from the perspectives of generalization and transfer learning.\n", "questions": "In Sec 5.1, how distance(rule) and distance(panel) are computed? Just want to clarify whether these two distances are comparable.\n\n", "limitations": "1. Though the goal of the task is to emerge the language for abstract rules, it will also be interesting to know whether the receiver can learn to induce rules after the communication (instead of applying rules during the communication, not required experiments). Similar to ETL, you can test the accuracy of the reasoning problem on the communicated receiver without further training.\n2. As the author mentioned, the current context input is a structured symbol. It will strongly encourage compositional language. It will be interesting to know how agents can emerge languages in a raw pixel input.\n3. Can the emerged languages generalize to contexts with different attributes?\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a new research angle of abstract rule reasoning for emergent communication. The context requires the agent to go beyond the low-level perceptual features and communicate more abstract rules. \n2. The candidate pool is smartly designed to motivate agents to extract rules from the context.\n3. A suite of comprehensive evaluations is designed to measure the generalization of the emerged languages. \n", "weaknesses": "1. Structural requirements of the new benchmark may need to be further explained: I am not clear about why the rules must be unambiguous. From my understanding, though the multiple rules can be applied to the current context, as long as the agents can communicate either of the rules, the receiver should capture the correct candidate? Though Figure 4 demonstrates the receiver can select the candidate correctly without the sender’s message trained with rule-RAVEN, further experiments/explanations are still needed to show that:\n\n    a. The communication training benefits from the structural or functional requirement or both.\n\n    b. The language that emerged using the I-RAVEN dataset is not/less generalizable/compositional/transferable. \n\n2. Sender’s rule reasoning and perception encoding are entangled. In the first stage, both $g^S$ and $f^S$ are trained. Though the training data is not shown in the communication stage, it will still introduce structural information because of the term $\\mathbb{1}(r_i, m_i)$. \n\n    a. does that require the length of the messages to equal the size of the rules? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments. We would like to clarify the concerns as follows:\n\n1. > Experiments/explanations about 1a and 1b.\n\n   \n\n   (1a) The communication training process benefits from both the structural and functional requirements.\n\n   \n\n   From the listener's perspective, structural requirements (unambiguous) are necessary. Specifically, when considering a dataset comprising multiple RPM problems, the occurrence of ambiguity (i.e., attribute values and rules are not one-to-one correspondence) would result in an imbalanced *prior probability distribution* of rules' frequency in the dataset. This imbalance implies the inequality significance between rules, inducing inductive bias in the dataset, thus posing a higher risk of listener's overfitting.\n\n   \n\n   From the perspective of the speaker, functional requirements are also necessary. Specifically, our rule-based candidate-panel generation algorithm deliberately confuses target rules between question-candidate panel pairs. Such deliberate confusion forces the speaker to initiate meaningful communication to help the listener.\n\n   \n\n   (1b) The language that emerged using the I-RAVEN dataset is not generalizable, transferable, and compositional. \n\n   For generalizable: We first emerge language using the I-RAVEN dataset, then test the language effectiveness on the rule-RAVEN dataset (these two datasets share the same rules but differ in how they generate candidate panels). Experimental results (Table 1) show low generalization performance of the language (i.e., ID/Inpo-ood accuracy only ~0.6, close to the message-blocked scene).\n\n   \n\n   **Table 1**: Generalization accuracy with different attribute values N.\n\n   |  N   |         ID          |      Inpo-ood       | Message-blocked scene |\n   | :--: | :-----------------: | :-----------------: | :-------------------: |\n   |  20  | $0.6312 \\pm 0.0017$ | $0.6270 \\pm 0.0024$ |  $0.6220 \\pm 0.0019$  |\n   |  30  | $0.6271 \\pm 0.0034$ | $0.6247 \\pm 0.0025$ |  $0.6215 \\pm 0.0033$  |\n\n   \n\n   For transferable: The experimental results (Table 2) also show low language transfer performance (accuracy only ~0.6, close to random).\n\n   \n\n   **Table 2**: Transfer accuracy of (source S, target T) attribute values ('agent': language from a well-trained speaker; random:  a random language).\n\n   |  (S, T)  |        agent        |       random        |\n   | :------: | :-----------------: | :-----------------: |\n   | (20, 30) | $0.6262 \\pm 0.0057$ | $0.6255 \\pm 0.0036$ |\n   | (20, 40) | $0.6246 \\pm 0.0002$ | $0.6248 \\pm 0.0025$ |\n   | (30, 40) | $0.6236 \\pm 0.0007$ | $0.6209 \\pm 0.0034$ |\n\n   \n\n   For compositional: Further analysis of the language shows that, when using I-RAVEN, the speakers describe the rules of different reasoning problems with constant messages (the listener completely ignores the speaker's message) on all random seeds, which indicates that the language is not compositional.\n\n   \n\n2. > Does the proposed training method have a limit on the message size?\n\n   \n\n   The training method does not introduce constraints on the length of messages. Specifically, the speaker updates  $f^S$, $g^S$, and $h^S$ in the first stage of training, while only load parameters of $f^S$, $g^S$ (without $h^S$) in the second stage. Therefore, even if the $1(r_i,m_i)$ operator on message encoder $h^S$ constrains the equal size between message $m_i$ and rule $r_i$ in the first stage, we can the training a new $h^S$ with reconfigurable message size in the second stage (joint) training.\n\n   \n\n   To demonstrate this claim, we tried diverse message sizes (Table 3) in our reasoning game, and get similarly high generalization performance (accuracy ~0.95).\n\n   \n\n   **Table 3**: Generalization accuracy with different message sizes (message length M, vocabulary size V) and attribute values N.\n\n   |  (M, V, N)  |         ID          |      Inpo-ood       |\n   | :---------: | :-----------------: | :-----------------: |\n   | (6, 15, 20) | $0.9539 \\pm 0.0018$ | $0.9528 \\pm 0.0015$ |\n   | (6, 30 ,20) | $0.9522 \\pm 0.0015$ | $0.9513 \\pm 0.0031$ |\n   | (6, 15, 30) | $0.9429 \\pm 0.0053$ | $0.9385 \\pm 0.0045$ |\n   | (6, 30, 30) | $0.9411 \\pm 0.0022$ | $0.9362 \\pm 0.0014$ |\n\n   \n\n3. > How distance of rules/panels are computed?\n\n   \n\n   - For rules, different rules belong to different categories, so we first compute rule vectors one-hot encoding. Then, we use the cosine distance (in this case, cosine distance is similar to the normalized hamming distance) to calculate the distance between rules.\n   - For panels, according to 4 attributes（number, shape, color, size）value, each context panel can be represented as a 4-dim integer vector. Within one RPM problem, we directly concatenate all 6 context panels into a 24-dim integer vector. Then, we use the cosine distance to calculate the distance between context panels of RPM problems.\n\n   \n\n4. > Whether the listener can learn to induce rules after the communication.\n\n   \n\n   We infer that the listener can learn to induce rules after the communication because the key to cooperative reasoning is that the speaker and listener have a similar ability to reason and agree on the message's linguistics (rules mapping). We check this by testing the accuracy of the reasoning problem after exchanging the parameters of the reasoning modules ($g^S$ and $g^L$) of the well-trained speaker and listener.\n\n   \n\n5. >  Emerge language in a raw pixel input, and show compositionality.\n\n   \n\n   We supplement the transfer performance of emerged language to image-based downstream reasoning task experiments in reviewer LuwD(Q1) and show the compositionality of emerged language in Reviewer 3wvB(Q2).\n\n   \n\n6. >  Languages generalize to contexts with different attributes.\n\n   \n\n   The emerged language can generalize to contexts with different attributes as long as ensuring the quality of the representations produced by the perception module (no effect on the cognition module)."}, {"review_id": "FBEelP2aAX", "reviewer": "Reviewer_M8eN", "summary": "This paper takes the ever-popular Lewis signalling game for emergent\ncommunication and studies experiments on rule-focused communication as opposed\nto the perception-focused communication of prior work.  In particular, it uses\na modified version of Raven's progressive matrices to formulate a signalling\ngame directly on attribute-value vectors which requires pattern\nrecognition/reasoning to complete.  The authors find that agents can learn to\ncommunicate when using a two-stage curriculum which essentially pretrains the\nspeaker for more stable communication at the beginning of the communication\nstage.  The experiments demonstrate that the resulting emergent language\ncorrelates better with the underlying rules of observations rather than the\nindividual observations themselves.\n", "questions": "- How does this work compare with prior art which also uses categorical\n  variables for a signalling game, even if there is no reasoning element to the\n  game?\n- What are the particular effects on an emergent language of\n  a reasoning-focused game versus one that is perception-focused?  What sort of\n  inductive biases are present in the task that has a downstream effect on the\n  language?\n\n## Minor Comments/Questions\n- `Line 46` \"inner\": typo?\n- `Line 46, 86` \"inter\": typo?\n- `Line 122` Do not use curly braces for ordered sequences; use parentheses.\n- `Paragaph @ 119` Why is it necessary for there to be no ambiguity?  This sort of ambiguity shows up frequently in human communication.\n- `Line 251` \"cause\" -> \"because\"\n- `Paragaph @ 246` I would use \"shows\" or \"demonstrates\" instead of \"proves\" since it is not a formal, mathematical proof.\n- Use the default LaTeX placement of tables/figures instead of `[h]`; the former is less distracting.\n", "limitations": "N/A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "## Originality\n- `[major]` Addresses the signalling game from a new perspective, i.e.,\n  reasoning instead of perception.\n- `[minor]` Introduces a new dataset.\n## Quality\n- `[major]` Presents good variety of empirical evaluation with clear results\n- `[minor]` Presents different levels and senses of \"generalization\".\n## Clarity\n- `[minor]` Details for implementation are presented without being overwhelming\n## Significance\n*See Originality.*\n", "weaknesses": "The paper is relatively complete, but what keeps my rating from being higher is\nthat there is relatively sparse comparison with prior work.  Such comparison\nwould better contextualize the results and increase its significance.  For\nfurther details, see the *Questions* section of the review.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments. We would like to clarify the concerns as follows:\n\n1. > Comparison with previous work using symbolic dataset.\n\n   We compare with previous emergent language work using symbolic datasets  (cited in our paper) from multiple angles: research goals, task orientation, input complexity, and agent training.\n\n   \n\n   **Table 1**: Comparison of previous emergent language work using symbolic datasets.\n\n   |                  | Ours                                                         | Others                                                       |\n   | ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n   | Research Goals   | Emergent cognition-based communication for rules reasoning.  | Efficiency of Language Coding [1], Ease-of-teaching [2], population heterogeneity [3], and promote structured, generalizable languages [4]. |\n   | Task Orientation | Extracting and using **inter-context** rules to handle reasoning problems. | Identifying the **inner-context** object attributes for discrimination or reconstruction task [1-4]. |\n   | Input Complexity | Multiple symbolic vectors with attributes.                   | One symbolic vector with attributes [1-4].                   |\n   | Agent Training   | Two-stage training due to the contexts-and-semantics bilaterally drifting task. | End-to-end training due to the training simplistic task [1-3], iterated training [4]. |\n\n   We would supplement more related work and revise our paper.\n\n   \n\n   Refs: \n\n   [1] Chaabouni, R., Kharitonov, E., Dupoux, E., & Baroni, M. (2019). Anti-efficient encoding in emergent communication.\n\n   [2] Li, F., & Bowling, M. (2019). Ease-of-teaching and language structure from emergent communication.\n\n   [3] Rita, M., Strub, F., Grill, J. B., Pietquin, O., & Dupoux, E. (2022). On the role of population heterogeneity in emergent communication.\n\n   [4] Ren, Y., Guo, S., Labeau, M., Cohen, S. B., & Kirby, S. (2020). Compositional languages emerge in a neural iterated learning model.\n\n   \n\n2. > Semantic effects and inductive biases of a reasoning-focused game versus one that is perception-focused.\n\n   \n\n   **Table 2**: Comparison of reasoning-focused and perception-focused games from both perspectives of semantic effects and inductive biases.\n\n   |                      | Reasoning-focused game (ours)                                | Perception-focused game                                      |\n   | -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n   | **Semantic effects** | The emerged language describes changing patterns (i.e., rules) between *multiple* input contexts. | The emerged language describes perceptual features of objects (or attributes) within a *single* input context. |\n   | **Inductive biases** | The game settings force agents to cooperative reason by conveying abstract rules *implicit* in contexts. | The game settings force agents to discriminate or reconstruct targets by conveying perceptual features of contexts. |\n\n   \n\n3. > `Paragraph @ 119` Why is it necessary for there to be no ambiguity?\n\n   \n\n   From the listener's perspective, structural requirements (unambiguous) are necessary. Specifically, when considering a dataset comprising multiple RPM problems, the occurrence of ambiguity (i.e., attribute values and rules are not one-to-one correspondence) would result in an imbalanced *prior probability distribution* of rules' frequency in the dataset. This imbalance implies the inequality significance between rules, inducing inductive bias in the dataset, thus posing a higher risk of listener's overfitting.\n\n   \n\n4. > About other minor questions. \n\n   \n\n   We will revise the text format and typos, and modify inappropriate expressions in line 122 and paragraph at 246."}, {"review_id": "9AIURD8k6D", "reviewer": "Reviewer_9tUw", "summary": "This work investigates the emergent communication framework for reasoning rules. That is, unlike prior studies that focus on communication about perceived low-level contexts, this paper proposes a cognition-oriented environment to encourage agents to reason and communicate about high level-rules. To this end, it introduces a new interesting and unbiased benchmark, rule-RAVEN. This benchmark, as opposed to the original one (I-RAVEN) avoids overfitting and pushes the agents to have an actual communication protocol.\n\nThe authors show, with different experiments, that agents are able to succeed in the reasoning tasks and develop a compositional and semantically stable language.", "questions": "Can you explain further the experiments of the paragraph \"Rule-RAVEN dataset\"  (line 260)? In particular, I don't understand why we have an \"unsuccessful\" communication game with \"rule-RAVEN\" dataset if the speaker was already trained (in a two-stage setting). That is, in this training regime, the speaker's language is not random, and the listener should be able to succeed in the communication game (or at least have a good enough accuracy) without modifying the speaker's language). Maybe you can elaborate more on the stage 1 training?\n\nAlso, you state that \"for the listener’s train accuracy still achieves ~0.9 even if the speaker’s message is completely ignored\". How do you check if the speaker's message is completely ignored?", "limitations": ".", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The authors introduce a well-thought benchmark that could be beneficial for future works to analyze the content of emergent languages. This benchmark, which is a modification of I-RAVEN, forces agents to develop an actual communication protocol. They perform the needed ablation to show its benefit compared to I-RAVEN.\n\nFurthermore, this paper is well-written, and a detailed description of the setting, and hyper-parameters are provided (on top of the code).", "weaknesses": "The main weakness of this work is its motivation. As stated in the paper, the goal of the emergent communication framework is to:\n- either study the origin of the human languages\nand/or\n- develop intelligent communicating artificial agents\n\nIt is unclear what this work's position is. If the former, is there a theory that our language emerged to communicate about a high-level reasoning task? If so, can this line of work be clarified in the paper? If the goal is to develop communicating agents, communicating about visual inputs is more practical for human-agent interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments. We would like to clarify the concerns as follows:\n\n1. > Clarify this work's position: study the origin of the human languages or develop intelligent communicating artificial agents.\n\n   \n\n   Our work's position closer to the later, i.e., developing intelligent communicating artificial agents, cause we mainly focus on verifying the generalizable and transferable ability of agents in this paper. \n\n   \n\n   In addition,  now we supplement experiments for demonstrating the emerged language's transferability performance on image-based downstream real-world tasks. Specifically, we first generate symbolic data with $Number \\in \\\\{1, \\dots 9\\\\}$, $Color \\in \\\\{1, \\dots 9\\\\}$, $Shape \\in \\\\{3, \\dots 9\\\\}$ (triangle, square, ..., nonagon), and $Size \\in \\\\{1, \\dots, 9\\\\}$ using the rule-RAVEN dataset. We then implemented a render to draw the panel's symbol as a $320\\times320$ grayscale image. Finally, we train new listeners using the message from the symbolic environment and question-candidate panel images (we replace $f^L$ with a 5-layer ConvNet to process the image input). After 20 epochs of training, the training accuracy of the listener is in Table 1.\n\n   \n\n   **Table 1**: Transfer accuracy of language emerged on symbolic reasoning task to image-based downstream reasoning tasks ('agent': language from a well-trained speaker; random:  a random language).\n\n   | Attribute values |        agent        |       random        |\n   | :--------------: | :-----------------: | :-----------------: |\n   |        20        | $0.9193 \\pm 0.0029$ | $0.3519 \\pm 0.0245$ |\n   |        30        | $0.9168 \\pm 0.0084$ | $0.3030 \\pm 0.0182$ |\n   |        40        | $0.8996 \\pm 0.0109$ | $0.3406 \\pm 0.0618$ |\n\n   The results show that languages emerging in symbolic environments can transfer (with ~ 0.9 accuracy) to downstream visual reasoning tasks. \n\n   \n\n   Moreover, for the former (i.e., study the origin of the human languages), cognitive literature [1] provides the theory: \n\n   > *The ability to use linguistic signs to express freely-formed thoughts marks \"the true distinction between man and animal\" or machine.*\n\n    We will clarify in the next version of the paper.\n\n   \n\n   Refs: \n\n   [1] Chomsky, N. (2004). \"Chapter 15 Language and Mind: Current Thoughts on Ancient Problems\".\n\n   \n\n2. > Clarify the paragraph 'Rule-RAVEN dataset' at line 260.\n\n   This paragraph compares the impact of different panel generation methods in two datasets (i.e., rule-RAVEN and I-RAVEN) on agents' overfitting. \n\n   Specifically, an undeniable fact is that, in a message-blocked scene (where the speaker is bypassed, and the message is always a constant, making it unusable by the listener), due to the complete ineffectiveness of the message, a higher accuracy (i.e., the likelihood of the listener correctly selecting the target) implies a greater degree of inductive bias, which in turn suggests a larger overfitting risk.\n\n   In this paragraph, based on the experiments under such message-blocked scenes, we prove that the candidate panels generated by the I-RAVEN-style method lead to a more severe overfitting, i.e., the listener achieves ~0.9 accuracies by just simply analyzing the question-candidate answer panels without any information from the speaker (lines with the legend 'I-RAVEN_20/30/40' in Figure 4, significantly higher than rule-RAVEN.)\n\n   We will revise this paragraph to make it clearer and less confusing.\n\n   \n\n   \n\n3. > How do you check if the speaker's message is completely ignored?\n\n   The message-blocked scenes (where the speaker is bypassed, and the message is always a constant, making it unusable by the listener) are equivalent to the listener completely ignoring the message from the speaker, which is the lower bound of the accuracy of the communication task."}]

Author rebuttal:

Meta-review: The paper presents an intriguing approach to emergent communication, focusing on reasoning and communication of high-level abstract rules using the rule-RAVEN benchmark. This novel perspective offers an interesting direction in the field, and the reviewers' assessments highlight both the strengths and potential areas for improvement of the paper.

The introduction of the rule-RAVEN benchmark, which encourages agents to develop communication protocols for abstract reasoning, is commended by the reviewers. The careful construction of the dataset and the two-stage curriculum training approach showcase the authors' attention to detail in experimental design. The exploration of higher-level rule-based reasoning through emergent communication aligns well with the broader goals of studying language emergence and developing intelligent communicating agents.

While the motivation behind the work is noted as a potential weakness, the reviewers appreciate the paper's contributions in introducing a well-designed benchmark and the detailed experimental analysis. Some reviewers have suggested a need for further comparisons with existing formulations, both perception-focused and reasoning-focused, to contextualize the results.

In conclusion, the innovative approach of reasoning-focused communication using the rule-RAVEN benchmark holds promise and presents a valuable contribution to the field. With some addressing of the reviewers' concerns, the paper should be accepted to share its insights and methodologies with the community.
Decision: Accept (poster)