Title: Emergent Communication for Rules Reasoning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Emergent communication

Multi-agent communication

Raven's Progressive Matrices

Representation learning
Abstract: Research on emergent communication between deep-learning-based agents has received extensive attention due to its inspiration for linguistics and artificial intelligence. 
  However, previous attempts have hovered around emerging communication under perception-oriented environmental settings,
  that forces agents to describe low-level perceptual features intra image or symbol contexts.
  In this work, inspired by the classic human reasoning test (namely Raven's Progressive Matrix), we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts.
  Moreover, we propose 1) an unbiased dataset (namely rule-RAVEN) as a benchmark to avoid overfitting, 2) and a two-stage curriculum agent training method as a baseline for more stable convergence in the Reasoning Game,
  where contexts and semantics are bilaterally drifting.
  Experimental results show that, in the Reasoning Game, a semantically stable and compositional language emerges to solve reasoning problems.
  The emerged language helps agents apply the extracted rules to the generalization of unseen context attributes, and to the transfer between different context attributes or even tasks.

Emergent Communication for Rules Reasoning

Abstract

Research on emergent communication between deep-learning-based agents has received extensive attention due to its inspiration for linguistics and artificial intelligence. However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forces agents to describe low-level perceptual features intra image or symbol contexts. In this work, inspired by the classic human reasoning test (namely Raven's Progressive Matrix), we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Moreover, we propose 1) an unbiased dataset (namely rule-RAVEN) as a benchmark to avoid overfitting, 2) and a two-stage curriculum agent training method as a baseline for more stable convergence in the Reasoning Game, where contexts and semantics are bilaterally drifting. Experimental results show that, in the Reasoning Game, a semantically stable and compositional language emerges to solve reasoning problems. The emerged language helps agents apply the extracted rules to the generalization of unseen context attributes, and to the transfer between different context attributes or even tasks.

1 Introduction

Research on emergent communication has received extensive attention in recent years due to its inspiration to the fields of linguistics and artificial intelligence (AI). From the linguistic perspective, the study of emergent communication may help understand the origin and evolution of natural language [9, 13, 14, 28, 40]. From the AI perspective, introducing communication mechanism into the deep-learning-based multi-agents system is a promising way to make it interpretable [11], human-interactable [25], and more importantly, generalizable [1, 7, 31, 37].

Lewis-style Game [27] is widely used in previous studies to simulate emergent communication between two agents, a speaker, and a listener. In this game, the speaker sends a message based on perceived contexts (usually images or symbols), and the listener completes the perception task (usually discriminating or reconstructing the target object) based on the message. As two agents

Speaker

?

Number

increasing

Communicate

circles or lines

increasing

Reasoning

Rules

Number of polygons: 1, 1, ? ×

Number of edges: 3, 4, ? √

Perceived contexts

number of circles: 1,2,3

number of lines: 2,3,4

Number of edges

3, 4, 4, ×

3, 4, 0, ×

3, 4, 5, √

3, 4, 3, ×

grey

3 edges

Speaker

Perceived contexts

Perception-oriented environment

Cognition-oriented environment

gradually endow messages with precise semantics to describe the target object more accurately, a machine language emerges with communication.

However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forcing agents to describe low-level perceptual features (e.g., shape, color and combinations of them) intra image [11, 14, 15, 23, 24, 31] or symbol [28, 35-37] contexts. Unfortunately, such perception-oriented environments (i.e., speak by seeing ) are unable to prompt agents to emerge cognitive-based communication (i.e., speak by reasoning ), which is considered the foundation of human language and intelligence evolution by research in linguistics [12] and cognitive psychology [6]. Therefore, how to emerge communication from perception to cognition remains an unsolved issue.

In this work, we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Figure 1 shows the comparison between the perception-oriented environment and cognition-oriented environment. The Reasoning Game requires two agents to cooperatively solve a classic human reasoning test Raven Progressive Matrix [34] (RPM). Specifically, the speaker reasons underlying rules from two rows of contexts (consisting of visually simple attributes), and sends a message to the listener. Meanwhile, the listener must correctly parse the received message and grasp the extracted rules, so as to select a candidate option that matches the rules (and fills the blank in the last row of the contexts matrix). In the process of solving RPM by agents, the rules communicated are not related to perceived features intra a single context but rather stem from abstract and structural logical relationships inter multiple contexts [39], such as gradually deepening colors and increasing numbers. According to this characteristic of RPM, agents can only complete the Reasoning Game by reasoning implicit rules at the cognitive level, instead of explicit visual features at the perceptual level. Such cognition-oriented environmental settings are considered to characterize the defining feature of high-level human-like intelligence [6, 18, 48].

Moreover, to conveniently evaluate the cognitive ability of agents through the Reasoning Game, we propose 1) an unbiased rule-RAVEN dataset as a benchmark, and 2) a two-stage curriculum agent training method as a baseline. For the former, the rule-RAVEN dataset contains numerous RPM problems with different combinations of rules, and each problem consists of context panels with multiple attributes. The dataset is debiased by our rule-based candidate panel generation algorithm to avoid overfitting in agent communication. For the latter, the two-stage curriculum agent training method ensures agents converge more stably in the Reasoning Game, where contexts and semantics are bilaterally drifting.

We summarize the contributions of this paper with experimental results:

(Sec 5.1) The Reasoning Game successfully enables the emergent communication for reasoning rules between agents, and the emerged language is semantically stable and compositional.

(Sec 5.2) The proposed benchmark (i.e., the rule-RA VEN dataset) and baseline (i.e., the twostage curriculum agent training method) in the Reasoning Game are effective as expected and necessary for agent evaluation.

(Sec 5.3) We verify the improvement of agent cognitive ability through the emergent communication for reasoning rules from three aspects: 1) generalize to unseen rule combinations, 2) apply the extracted rules to other context attributes, 3) and transfer to different tasks.

"baca" is "number increasing"

baca

Speaker

?

Question panels

Context Panels

Message

Candidate panels

Rule

Target panel

,

?

2 Related Works

Emergent Communication. Using computational models to simulate emergent communication has a long history [21, 22, 43] and has recently been revived with the popularity of deep learning. A. Lazaridou et al. [24] studied emergent communication on the basis of reinforcement learning and deep neural network agents for the first time. Starting from this, dozens of studies focus on exploring the properties of the emerged language from the perspective of linguistics and cognitive psychology, such as alignment of agents [11], efficiency of language [8], capacity of communication channel [23], community interaction [15], ease of teaching [28], culture transmission [14], population heterogeneity [31], and environment scale [10]. In general, these studies adopt perception-oriented environmental settings, restricting the emerged languages to describing low-level perceptual features intra image or symbol contexts. Specifically, previous works [11, 15, 24] emerge communication of intra-context object attributes (e.g., color, shape) and their combinations (e.g., blue OR/AND triangle ) [31]. On the contrary, our work focuses on emergent communication in a cognition-oriented environment (i.e., Reasoning Game), where agents reason inter-context high-level rules and apply them, so as to cooperatively solve reasoning tasks (i.e., RPM).

Raven's Progressive Matrices (RPM). RPM [34] is a widely studied human IQ test, which has recently been revived in the AI community after it can be procedurally generated [44]. Compared with other reasoning benchmarks, such as Visual Question Answering [4, 19, 29] and Arithmetic Reasoning [38], RPM focuses on relational reasoning inter multiple contexts, reflecting humans' core cognitive abilities [6]. Based on RPM, many synthetic [2, 48] and natural [41] datasets and model designs [17, 46] are proposed to boost the high-level reasoning capability of deep learning models. In this work, we are the first to study emergent communication through cooperatively solving RPM problems. We find that such communication scenarios need to ensure clear semantics and avoid overfitting so that imposing new requirements on the rules set design and candidate panels generation of the RPM dataset. Motivated by this, we propose the rule-RAVEN dataset as a benchmark, and its corresponding two-stage curriculum agent training method as a baseline.

3 Method

3.1 Reasoning Game

The Reasoning Game is a cognition-oriented environment that requires two agents to solve RPM problems by reasoning and communicating high-level rules. The game can be formalized as a 5-tuple: ( C, Q, A, R, y ) , where C = ( c 1 , . . . , c l ) is a set of l context panels, Q = ( q 1 , . . . , q m ) is a set of m question panels, A = ( a 1 , . . . , a n ) is a set of n candidate panels with 1 target panel and n -1 distract panels, R = ( r 1 , . . . , r k ) is a vector that encodes the rules implicit in C (i.e., hidden to the speaker and listener), and y ∈ { 1 , . . . , n } indicates the index of the target panel, the only panel together with the question panels that match the rule R . Each panel contains several attributes (such as number, color, et al.) T = ( t 1 , . . . , t k ) , and the values of all attributes are within a pre-defined range.

Figure 2 shows the scheme of the Reasoning Game. In this scheme, the speaker first reasons the embedding, which represents the rules R implicit in context panels C , then encodes the embedding

Context panels

Question panels

row 2

row 1

row3

, distract rule

, distract rule

, target panel, match rule

, distract rule

Candidate panels(generated by our method)

Candidate panels(generated by I-RAVEN)

, modify (number), no rule match

, target panel, no modify, match rule

, modify (color), no rule match

, modify (number, color), no rule match

into a message M = ( m 1 , . . . , m s ) (each token m i is derived from a pre-defined vocabulary V ) for sending to the listener. Finally, according to the received message M , the listener chooses the target panel from candidate panels A to fill the blank behind the question panel Q . If the reasoning question is correctly solved, the listener and the speaker will be rewarded simultaneously.

3.2 Rule-RAVEN dataset

A reasoning problem (e.g., RPM) can be naturally divided into two steps: extracting rules and applying rules. It seems that the reasoning dataset (e.g., RA VEN) can be straightforwardly used to play the Reasoning Game by assigning the former step (i.e., extracting rules) to the speaker and the latter step (i.e., applying rules) to the listener. However, the premise for such a scheme of Reasoning Game could proceed successfully is that the speaker and listener can effectively transmit and understand the extracted rules by emergent communication. To this end, crafting a reasoning dataset as a benchmark still has requirements on the rules set design and candidate panels generation.

Regarding the rules set design , the mapping relationship between implicit rules and context attributes must be unambiguous to emerge semantically clear communication and avoid imbalanced prior probability distribution of rules' frequency in the dataset. Specifically, the design principle is subdivided into two aspects: 1) The rule corresponding to each set of attribute values in panels is unique. For example, if the attribute (e.g., size) value in context panels ( c 1 , c 2 , c 3 ) is (1 , 2 , 3) , which matches both rules 'sum' ( 1 + 2 = 3 ) and 'progression +1' ( 1 + 1 = 2 , 2 + 1 = 3 ), then the attribute value should be filtered out from the dataset. 2) The meanings expressed between different rules do not overlap. For example, when any set of attribute value ( a, b, c ) matches the rule 'progression' ( a + const = b, b + const = c ), it also matches the rule 'average' ( b = a + c 2 ). In this case, only one of the rules should be retained in the dataset.

Regarding the candidate panels generation , distract panels should be generated carefully so as to represent close but not quite correct rules that force the speaker and listener to communicate precisely the correct rules. Specifically, the design principle is that each distractor in candidate panels should match a 'distract rule' (as shown in figure 3 'our method'), to prevent the listener from inferring the target solely through the existence of regular rules between the question panels and candidate panels (a negative example is shown in figure 3 ( right ) 'I-RAVEN [17]' ). Otherwise, it will cause the speaker to lose the motivation to extract and communicate rules correctly (we will demonstrate this in detail by experiments in Sec 5.2).

Following the above requirements, we propose a symbolic dataset 2 , rule-RAVEN, for the Reasoning Game. Figure 3 ( left ) shows an example case in the rule-RAVEN dataset (See Appendix A for more details.).

3.3 Model and Training

We build the speaker's and listener's model in the Reasoning Game as follows.

Speaker's model. The speaker consumes context panels C and outputs message M . The speaker's policy can be formalized by S ( M | C ; θ ) with trainable parameters θ . Given context panels C , an MLP-based perception module f S first encodes each panel into an embedding ( f S ( c i )) 6 i =1 . Then, a reasoning module g S extracts the row-level rule embedding r S 1 = g S (( f S ( c i )) 3 i =1 ) and r S 2 = g S (( f S ( c i )) 6 i =4 ) from multiple context embeddings. We follow the state-of-art RAVEN solver SCL [46]'s core module, shared group MLPs, as the reasoning module. The row-level rule embeddings are average-aggregated to get the rule embedding r S = mean ( r S 1 , r S 2 ) of context panels C . Finally, an LSTM [16] message encoder h S takes the rule embedding r S as initial state z S 0 and a special token m 0 = <SOS> as start token to generate message M . At each timestep t ≥ 1 , based on current state z t = h S ( z t -1 , m t -1 ) , message m t = sample ( softmax ( q ( z t ))) , where q is a linear transformation to project the hidden state space to the vocabulary space. Token generation terminates until the message reaches a fixed length or a special token <EOS> is generated.

Listener's model. The listener consumes message M , question panels Q , and candidate panels A and outputs a prediction ˆ y . The listener's policy can be formalized by L (ˆ y | M,Q,A ; ϕ ) with trainable parameters ϕ . Given question panels Q and candidate panels A , a perceptual module f L first encodes Q and A into corresponding embeddings ( f L ( q i )) 2 i =1 and ( f L ( a i )) 8 i =1 . Meanwhile, an LSTM message decoder h L takes a special state z L 0 as the start state to decode message M . At each timestep t ≥ 1 , current state z L t = h L ( z L t -1 , m t ) . Message decoding terminates until it reaches the maximum message length | M | or <EOS> . We use the final state z L as the decoded message embedding, representing the rules implied in context panels C . Then, each candidate embedding is concatenated with the question embedding and fed into the reasoning module g L to obtain multiple candidate rule embeddings r L i = g L ( f L ( q 1 ) , f L ( q 2 ) , f L ( a i )) , i ∈ { 1 , . . . , 8 } . Finally, the message embedding and the candidate rule embedding perform an inner product and softmax operation to obtain the probability distribution softmax (( r ′ i · z T | M | ) 8 i =1 ) of candidate panels, which is further used to compute the loss and the prediction ˆ y .

To ensure that the cognitive capabilities of the speaker and the listener are similar, we build their perception and reasoning module architecture identically. No parameters are shared between the

speaker and listener to avoid introducing artificial priors about their consensus. More details of the model can be found in Appendix B.

In our game, both rules reasoning and language emergence are highly challenging. We find that jointly training the speaker and listener directly from scratch leads to communication failures. The reason is that, in the early stage of joint training, the reasoning context panels and the message semantics are drifting simultaneously, leading the speaker cannot produce helpful messages to the listener, and the listener tends to ignore the speaker's message. To help agents jump out of the local optimum, we propose a two-stage curriculum training [3] method.

In the first stage, the speaker is pre-trained on a held-out rule set (See Appendix A for more details.) generated dataset to acquire some reasoning capabilities to cope with the drifting context. This stage can be considered a supervised learning task with the ground-truth rule vector R as the label and only updating the speaker's parameters θ . Note that this does not impose artificial priors on emerged language since the rule encodings predicted at this stage are absent in the joint training stage. The gradients of the objective function J 1 in the first stage are:

Note that although the ✶ ( · , · ) operator exists in ∇ θ J 1 , this does not introduce constraints on the size of emerged language (i.e., message_length and vocabulary_size ) in the second stage of training. See Appendix C for more details.

In the second stage, the pre-trained speaker and listener train jointly to emerge language. Since discrete communication breaks the end-to-end differentiability, we employ the REINFORCE[45] algorithm. The gradients of the objective function J 2 in the second stage are:

∇ θ J 2 = E [ ✶ (ˆ y, y ) ∇ log S ( M | C )] + λ · ∇ H [ S ( M | C )] ∇ ϕ J 2 = E [ ✶ (ˆ y, y ) ∇ log L (ˆ y | M,Q,A )] Where λ is a hyperparameter controlling entropy regularization.

4 Exprimental Settings

4.1 Basic Settings

Implementation. We use Python3 [42] to implement the rule-RAVEN dataset. Our model implementation is based on Pytorch [33] and EGG [20] toolkit. The code is available in supplementary materials.

Dataset preparation. We instantiate our rule-RAVEN dataset (introduced in Sec 3.2). For each case of reasoning problem, there are 6 context panels (i.e., C = ( c 1 , . . . , c 6 ) ), 2 question panels (i.e., Q = ( q 1 , q 2 ) ), and 8 cadidate panels (i.e., A = ( a 1 , . . . , a 8 ) ). Each panel has 4 attributes (i.e., T = ( number,color, size, shape ) ), and each attribute have N ∈ { 20 , 30 , 40 } values ( N 4 in total). The larger N (more diverse the attribute values) means it is more difficult to identify contexts and reason rules. There are 8 rules that apply to each attribute respectively, resulting in a total of 8 4 = 4096 different rule combinations. For each rule combination, We randomly generated 20 different reasoning problem cases (sampling from N 4 attribute values) with Algorithm 1 ( 4096 × 20 = 81920 in total), and half of these cases for training, half for testing. Besides, we use a held-out rule set to additionally generate 2000 cases for pre-training the speaker in the first stage of agent training (introduced in Sec 3.3).

Communication Channel. We set the maximum message length | M | = 4 and the vocabulary size | V | = 15 . With this setup, the capacity of the communication channel is 15 4 , sufficient to describe all rule combinations ( > 8 4 ), and preserve redundant tokens for two-stage curriculum training, and much lower than vocabularies required to directly describe the context panels ( ∼ 10 10 ).

Optimization. Agents' parameters are optimized by AdamW [30], with a learning rage of 3 · 10 -3 , a weight decay of 0 . 01 , β 1 = 0 . 99 , β 2 = 0 . 999 and a batch size of 512. For the speaker, we set the hyperparameter of entropy regularization λ = 0 . 01 .

4.2 Language Metrics of emergent communication

Generalization abilities. To test the generalization ability of the emerged language, we further create 4 data splits, corresponding to 4 levels of generalization ability: 1) In-distribution generalization ( ID ). The training and test sets share the same rule combinations but consist of different problems. The ID level tests the language's ability to generalize to different problems with the same rules. 2) Interpolated out-of-distribution generalization ( Inpo-ood ). Each rule in the rule set appears in each attribute on the train set, but some rule combinations do not appear on the train set and form the test set. The Inpo-ood level tests the language's ability to generalize to unseen rule combinations based on components. 3) Extrapolated out-of-distribution generalization level-1 ( Expo-ood-L1 ). Each attribute has a unique rule that does not appear in the train set, and these rules are combined with other rules that appear in the train set to form a test set. The Expo-ood-L1 level tests the language's ability to describe rules across attributes. 4) Extrapolated out-of-distribution generalization level-2 ( Expo-ood-L2 ). A rule does not appear on any attribute on the train set, and it is combined with other rules present in the train set to form the test set. The Expo-ood-L2 level tests the language's ability to generalize to new rules.

Topographic similarity (Topsim). Topsim [5] is widely used in previous emergent language studies to measure language compositionality. It is defined as the Spearman correlation coefficient of pairwise distances between input and message space. In the input space, we use the normalized Hamming distance between rules and the cosine distance between panels. In the message space, we adopt the Levenshtein [26] distance with the same removal, insertion, and substitution costs. Computing Topsim involves pairwise distances and thus has a computational complexity of O ( n 2 ) , which is computationally prohibitive in our environment. In the experiment, we randomly sample 1000 samples per run to calculate Topsim and take the average over 20 runs as the final result.

Ease and transfer learning (ETL). ETL [10] captures how fast and well the emergent language is transmitted to new listeners performing a different task. We focus on the emerged language's transferability between reasoning tasks with different attribute values, N . Specifically, we first dump the deterministic languages (choosing the token greedily with the highest probability) of well-trained speakers under each attribute value setting. We then map deterministic languages in the source task setting and reasoning problems in the target task settings based on the rules. Finally, we train new randomly initialized listeners using mapped language and reasoning problems and observe the convergence speed and accuracy of the listeners. Our source task setting contains attribute values N ∈ { 20 , 30 , 40 } , and the target task setting contains N ∈ { 20 , 30 , 40 } , and an extremely large N = 80 .

5 Exprimental Results

To show the benefits and effectiveness of emergent communication through the Reasoning Game, we answer the following three questions by the experimental results.

5.1 What do the emerged language tend to describe?

Emerged language prefers to describe rules rather than context panels. We demonstrate this issue from two aspects. 1) Compare which input (rules or context panels) has a stronger correlation (measured by Topsim ) with the message space. As Table 1 shows, in all three diversities of attribute values (i.e., N = 20 , 30 , 40 ), the Topsim between rules and messages (i.e., Rule-Message ) is always substantially higher than that between context panels and messages (i.e., Panel-Message ). This directly proves the conclusion, and the emerged languages are somewhat compositional (because Topsim is also used to measure compositionality). 2) Another indirect evidence comes from the relationship between ID generalization performance (measured by Listeners' accuracy) and attribute value diversities. Specifically, since the capacity of the communication channel is not enough to describe the context panels ( 15 4 < 10 10 ), more diverse context panels are expected to exacerbate this shortage, leading to a rapid decrease in accuracy. However, paradoxically, the data in each column of Table 2 shows that the accuracy is almost equal as the attribute value diversity increases (from 20 to 30 or 40).

0

5

10

15

20

25

30

35

40

Epochs

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Train Accuracy

I-RAVEN_20

I-RAVEN_30

I-RAVEN_40

rule-RAVEN_20

rule-RAVEN_30

rule-RAVEN_40

0

25

50

75

100

125

150

175

200

Epochs

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Train Accuracy

w/_pt_20

w/o_pt_20

w/_pt_30

w/o_pt_30

w/_pt_40

w/o_pt_40

5.2 Why are the proposed benchmark and baseline effective and necessary?

Rule-RAVEN dataset. We quantitatively compare our rule-based candidate panels generation algorithm (i.e., Algorithm 1) in rule-RA VEN with the I-RA VEN. Specifically, by separately training the listener under a message-blocked scene (i.e., the speaker's message is forced to a constant), we compared the overfitting degree of rule RAVEN and I-RAVEN (the higher the accuracy, the less dependent the listener is on the speaker's reasoning about rules, i.e., the higher the overfitting degree). Figure 4 shows that I-RAVEN suffers from overfitting severely (lines with the legend 'I-RAVEN_20/30/40'), for the listener's train accuracy still achieves ∼ 0 . 9 even if the speaker's message is completely ignored. While rule-RAVEN (lines with the legend 'rule-RAVEN_20/30/40') can effectively control the degree of overfitting, the listener's train accuracy is only ∼ 0 . 6 , which is significantly different from when receiving the speaker's message normally ( ∼ 0 . 6 v.s. ∼ 0 . 9 ). So it is enough to distinguish whether communication is effective or not. Note that controlling overfitting to an ideal level (8 candidate panels, ≈ 0 . 125 message-free accuracy) is unlikely because the listener can always guess the answer by memorizing the patterns corresponding to specific rules. Besides, naive methods, such as increasing attribute values, do not alleviate overfitting (as shown in lines with the legend 'I-RAVEN,' from 20 to 30 or 40).

Two-stage curriculum training method. We compare the training curves with and without the two-stage curriculum training method. Figure 5 shows that the two-stage curriculum training method leads to successful communication in all three attribute value diversities (lines with the legend 'w_pt_20/30/40'), for agents reaches higher accuracy ( ∼ 0 . 9 ). In contrast, in the case without the two-stage curriculum training method (lines with the legend 'w/o_pt_20/30/40'), agents only converge to a similarly low accuracy as under the message-blocked scene, which implies that agents can not communicate successfully. An intuitive explanation for this phenomenon is that, the speaker's training in the first stage helps him acquire some reasoning ability to cope with the drifting context, which helps stabilize the semantic production in the joint training stage.

5.3 What are the benefits of communicating extracted rules?

Beneficial for generalizing to unseen rule combinations. As shown in Table 2, the high generalization performance at the ID level( ≈ 0 . 92 ) shows that communicating abstract rules can generalize to different reasoning problems with the same rule combination. Moreover, the generalization performance of the Inpo-ood level is similar to that of the ID level, which indicates that communicating abstract rules can also generalize to unseen rule combinations, e.g., based on (add, min) and (progression, max) on ( number , size ) generalize to ( add , max ) on ( number , size ).

Beneficial for agents apply rules to other attributes. As shown in Table 2, we observe that the performance of level Expo-ood-L1 is higher than that of Expo-ood-L2 ( ≈ +0 . 13 ), which suggests that communicating abstract rules can help the listener to apply rules extracted from one attribute to others, e.g., applying rule ( progression ) extracted from ( number ) to ( size ). This characteristic is similar to the human ability to draw inferences from one instance to another. Similar to the observation in Chaabouni et al. [10], complicating the task of language emergence (reasoning task in our work) does not improve the systematicness of emergent language (from average Inpo-ood ≈ 0 . 92 , to Expo-ood-L1 ≈ 0 . 58 , and to Expo-ood-L2 ≈ 0 . 46 ), possibly because reasoning about unseen rules is highly challenging in our game.

Beneficial for transferring to a different task setup. As shown in Table 3, all transfers in the table have relatively high train accuracy and are close to the upper bound ( ID and Inpo-ood accuracy). Even languages that emerged on the simplest task (20 values of each attribute) can be transferred to the most challenging task (80 values of each attribute). Figure 6 shows the training curve of task transferring grouped by target attribute values. Emerged language (lines with the legend 'agent') has a similar convergence speed compared with the ideal language that directly uses rule encoding as the language (lines with the legend 'rule') and only hundreds of optimization steps (less than 10 epochs) to converge. Besides, emerged language outperformed the random group (lines with the legend 'random'), which randomly maps source languages and the target reasoning problems. Good cross-task transferability also indirectly confirms that the emerged languages tend to describe rules more than context panels.

0

100

200

300

400

Optimization Steps

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Train Accuracy

agent_30_to_20

agent_40_to_20

random_30_to_20

random_40_to_20

rule_20

(a) Transferring to 20.

0

100

200

300

400

Optimization Steps

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Train Accuracy

agent_20_to_30

random_20_to_30

random_40_to_30

rule_30

(b) Transferring to 30.

0

100

200

300

400

Optimization Steps

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Train Accuracy

agent_20_to_40

agent_30_to_40

random_20_to_40

random_30_to_40

rule_40

(c) Transferring to 40.

0

100

200

300

400

Optimization Steps

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Train Accuracy

agent_20_to_80

agent_30_to_80

random_30_to_80

random_40_to_80

rule_80

(d) Transferring to 80.

6 Discussion and Conclusion

In this work, we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules. Based on our rule-RAVEN dataset and twostage curriculum agent training method, we successfully observe the emergence of a language that can describe abstract rules and further reveal the language's impressive generalization ability and transferability.

Our research may inspire the following directions: 1) Emerging communication on more realisticstimuli-based reasoning datasets [32, 41, 47] to study the co-evolution of perception, cognition, and language system in agents; 2) Debiasing and avoiding overfitting on the procedurally generated reasoning dataset from a more fine-grained perspective; 3) Designing pre-training tasks to converge in environments with multiple variable drifts.

7 Limitations

We summarized the limitation of our works as follow: 1) Our work only focuses on language emergence on a clean symbolic-based reasoning dataset, lacking the exploration of more realistic stimuli-based (e.g., synthetic or natural images) reasoning datasets; 2) The reasoning task (RA VEN) adopted in our work only requires the agent to complete the reasoning via a single round of interaction, simplifying the natural reasoning process; 3) Our work only analyzes the semantics of the emerged languages at the message level, lacking fine-grained structural (e.g., gramma) and semantic analysis at the token level (e.g., the degree of polysemy and ambiguity between tokens).

Acknowledgments and Disclosure of Funding

This work is partially supported by the NSF of China (under Grants 61925208, U22A2028, 62102399, 62222214, 62102398, U19B2019), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.

References

Michal Auersperger and Pavel Pecina. Defending compositionality in emergent languages. arXiv preprint arXiv:2206.04751 , 2022.

David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference on machine learning , pages 511-520. PMLR, 2018.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning , 2009.

Yonatan Bisk, Kevin Shih, Yejin Choi, and Daniel Marcu. Learning interpretable spatial operations in a rich 3d blocks world. Proceedings of the AAAI Conference on Artificial Intelligence , 32, 12 2017.

Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life , 12(2):229-242, 2006.

Patricia Carpenter, Marcel Just, and Peter Shell. What one intelligence test measures: A theoretical account of the processing in the raven progressive matrices test. Psychological review , 97:404-31, 08 1990.

Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4427-4442, Online, July 2020. Association for Computational Linguistics.

Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems , 32, 2019.

Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences , 118(12):e2016569118, 2021.

Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International Conference on Learning Representations , 2022.

Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional communication learning from raw visual input. In International Conference on Learning Representations , 2018.

Noam Chomsky. Language and mind: current thoughts on ancient problems. In Variation and universals in biolinguistics , pages 379-405. Brill, 2004.

Michael Cogswell, Jiasen Lu, Stefan Lee, Devi Parikh, and Dhruv Batra. Emergence of compositional language with deep generational transmission, 2019.

Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Co-evolution of language and agents in referential games. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 2993-3004, Online, April 2021. Association for Computational Linguistics.

Laura Harding Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3700-3710, Hong Kong, China, November 2019. Association for Computational Linguistics.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.

Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , volume 35, pages 1567-1574, 2021.

Susanne M. Jaeggi, Martin Buschkuehl, John Jonides, and Walter J. Perrig. Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences , 105(19):6829-6833, 2008.

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , July 2017.

Eugene Kharitonov, Roberto Dessì, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on Emergence of lanGuage in Games. https://github.com/ facebookresearch/EGG , 2021.

Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the laboratory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences , 105(31):10681-10686, 2008.

Simon Kirby, Kenny Smith, and Henry Brighton. From ug to universals: Linguistic adaptation through iterated learning. Studies in Language, v.28, 587-607 (2004) , 28, 09 2004.

Satwik Kottur, Jose Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge naturally in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 2962-2967, Copenhagen, Denmark, Sep 2017. Association for Computational Linguistics.

Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations , 2017.

Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7663-7674, Online, July 2020. Association for Computational Linguistics.

Vladimir Iosifovich Levenshtein. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady , 10(8):707-710, 1966. Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.

David Lewis. Convention: A philosophical study . John Wiley & Sons, 2008.

Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. Advances in neural information processing systems , 32, 2019.

Zechen Li and Anders Søgaard. QLEVR: A diagnostic dataset for quantificational language and elementary visual reasoning. In Findings of the Association for Computational Linguistics: NAACL 2022 , pages 980-996, Seattle, United States, July 2022. Association for Computational Linguistics.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Jesse Mu and Noah Goodman. Emergent communication of generalizations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Animashree Anandkumar. Bongard-logo: A new benchmark for human-level concept learning and reasoning. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

J. C. Raven and Australian Council for Educational Research. Raven's progressive matrices (1938): sets A, B, C, D, E . Australian Council for Educational Research Melbourne, 1938.

Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In International Conference on Learning Representations , 2020.

Mathieu Rita, Florian Strub, Jean-Bastien Grill, Olivier Pietquin, and Emmanuel Dupoux. On the role of population heterogeneity in emergent communication. In International Conference on Learning Representations , 2022.

Mathieu Rita, Corentin Tallec, Paul Michel, Jean-Bastien Grill, Olivier Pietquin, Emmanuel Dupoux, and Florian Strub. Emergent communication: Generalization and overfitting in lewis games. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations , 2019.

R E Snow, P C Kyllonen, and B Marshalek. The topography of learning and ability correlations. Advances in the psychology of human intelligence , 2:47-103, 1984.

Luc Steels. The synthetic modeling of language origins. Evolution of communication , 1(1):1-34, 1997.

Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton Hengel. V-prom: A benchmark for visual reasoning using visual progressive matrices. Proceedings of the AAAI Conference on Artificial Intelligence , 34:12071-12078, 04 2020.

Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual . CreateSpace, Scotts Valley, CA, 2009.

Kyle Wagner, James A. Reggia, Juan Uriagereka, and Gerald S. Wilkinson. Progress in the simulation of emergent communication and language. Adaptive Behavior , 11(1):37-69, 2003.

Ke Wang and Zhendong Su. Automatic generation of raven's progressive matrices. In Proceedings of the 24th International Conference on Artificial Intelligence , IJCAI'15, page 903-909. AAAI Press, 2015.

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. , 8(3-4):229-256, may 1992.

Yuhuai Wu, Honghua Dong, Roger Grosse, and Jimmy Ba. The scattering compositional learner: Discovering objects, attributes, relationships in analogical reasoning. arXiv preprint arXiv:2007.04212 , 2020.

Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

A Details of rule-RA VEN dataset

A.1 Rules in rule-RAVEN

In the joint training stage, a specific rule contains 4 elements on different attributes, e.g., (Number:add, Color:progression_2, Size:constant, Shape:max) . These rules meet the structural and functional requirements stated in Section 3.2 The embodiment of each rule on each attribute are shown in Figure 7. For example, the ' Color:progression_2 ' means: in the same row of panels, the image color gradually darkens by 2 degrees from left to right.

Rules

No.

1

2

3

4

5

6

7

8

=

=

=

=

=

=

=

=

+

+

+

+

-

-

-

-

min

min

min

min

+2

+2

+2

+2

+2

+2

+2

+2

-1

-2

-1

-2

-1

-2

-1

-2

+2

+1

+2

+1

+2

+1

+2

+1

Attributes

In the speaker pre-training stage, the rules involved in each attribute are shown in Figure 8, which do not overlap with those in Figure 7, and have similar semantics to rules No.6, No.7, and No.8 in Figure 7. Therefore, pre-training the speaker with these rules does not provide agents with prior knowledge about the rules in the joint training stage, which only helps the speaker acquire reasoning capabilities to cope with the drifting context.

A.2 A RPM case in rule-RAVEN

Figure 9 shows one RPM case of the rule-RAVEN dataset used in our experiments. The form of this case is consistent with Figure2. Figure2 is just for the convenience of illustration, only 1 attribute of the rule is drawn, but the actual rule consists of 4 attributes, i.e., (Number, Color, Size, Shape) .

Rules

No.

1

2

3

4

5

Attributes

+1

+1

+1

+1

+1

+1

+1

+1

-1

-1

-1

-1

-1

-1

-1

-1

-2

-2

-2

-2

-2

-2

-2

-2

+1

+2

+1

+2

+1

+2

+1

+2

-2

-1

-2

-1

-2

-1

-2

-1

Question panels

row 2

row 1

Candidate panels

Context panels

row 3

Target panel, match target rule

Distract panel, match distract rule

Distract panel, match distract rule

Distract panel, match distract rule

Distract panel, match distract rule

Distract panel, match distract rule

Distract panel, match distract rule

Distract panel, match distract rule

?

Choose

from

A.3 Details of generalization data splits

Figure 10 shows an example of four levels of generalization data splits described in Section 4.2.

For ID , Inpo-ood , and Expo-ood-L2 generalization levels, we first get 7 4 = 2401 rule combinations based on rules No.1 to 7 in Figure 7 on all 4 attributes. We then randomly sample 300 rule combinations from these 2401 rule combinations and generate 10 problems for each rule combination as the Inpo-ood data split (with a total of 300 × 10 = 3000 problems). For each of the remaining 7 4 -300 = 2101 rule combinations, we generate 10 problems as the train data split (for ID , Inpo-ood , and Expo-ood-L2 ) and 10 problems as the ID data split (with a total of 2101 × 10 = 21010 problems for training and 21010 for ID ). We finally combine rule No.8, excluded in the previous process, with the rules on other attributes to get 8 4 -7 4 = 1695 rule combinations and generate 10 problems for each rule combination as Expo-ood-L2 data split (with a total of 1695 × 10 = 16950 problems).

For the Expo-ood-L1 generalization level, we exclude rule No.6 on attribute 1, rule No.3 on attribute 2, rule No.4 on attribute 3, and rule No.8 on attribute 4, getting 2401 rule combinations remaining. By comparing the accuracy difference between Expo-ood-L1 and Expo-ood-L2 levels (both Expo-ood-L1 and Expo-ood-L2 exclude one rule on each attribute, but on Expo-ood-L1 level, rules excluded on one attribute appear on other attributes), we can check whether the language describing the rules can help the listener apply the rules across attributes. To compare fairly with Expo-ood-L2 level, we randomly exclude 300 rule combinations and generate 10 problems for each of the remaining 2101 rule combinations as the training set for Expo-ood-L1 level (with a total of 2101 × 10 = 21010 problems). We combine the rules excluded in the previous process with the rules on other attributes to get 8 4 -7 4 = 1695 rule combinations and generate 10 problems for each rule combination as Expo-ood-L1 data split (with a total of 1695 × 10 = 16950 problems).

Train

Test

Train

Test

Train

Test

Train

Test

B Details of Agent Model

The model diagram of the speaker and listener is shown in Figure 11. Some hyperparameters of the model are listed in Table 4. Please refer to the source code in the supplementary materials for more implementation details.

Context panels

Context embeddings

MLP

Mean

LSTM Message Encoder

Softmax

Sample

Row-level rule

embeddings

Question panels

Candidate panels

MLP

Message

embedding

LSTM Message Decoder

Start state

Speaker's Model

C Results of Diverse Language Sizes

The proposed two-stage training method does not introduce constraints on the size of the emerged language (i.e., message_length and vocabulary_size ). The reason is that the speaker updates f S , g S , and h S in the first stage of training, while only load parameters of f S , g S (without h S ) in the second stage. Therefore, even if the ✶ ( r i , m i ) operator on message encoder h S constrains the equal size between message m i and rule r i in the first stage, we can the training a new h S with reconfigurable language size in the second stage (joint) training.

We also provide results with diverse language sizes (Table 5) in our reasoning game and get similarly high generalization performance (accuracy ∼ 0 . 95 ).

D Token-level Analysis of the Emerged Language

We compute the probability distribution P ( message | attribute, rule ) of a randomly selected seed and provide the most probable tokens for a given attribute and rule, as shown in Table 6.

The results indicate that tokens exhibit regular patterns (i.e., language-like syntax and compositionality) for different attributes and rules. For example, almost all rules related to attribute ' color ' start with tokens 'K' or 'B', and attribute ' size ' start with tokens 'K' or 'G'. On another dimension, the rule ' varprogression_-1 ' across all attributes ends with token 'C'.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the emergent communication framework for reasoning rules. That is, unlike prior studies that focus on communication about perceived low-level contexts, this paper proposes a cognition-oriented environment to encourage agents to reason and communicate about high level-rules. To this end, it introduces a new interesting and unbiased benchmark, rule-RAVEN. This benchmark, as opposed to the original one (I-RAVEN) avoids overfitting and pushes the agents to have an actual communication protocol.\n\nThe authors show, with different experiments, that agents are able to succeed in the reasoning tasks and develop a compositional and semantically stable language.\n\nSTRENGTHS:\nThe authors introduce a well-thought benchmark that could be beneficial for future works to analyze the content of emergent languages. This benchmark, which is a modification of I-RAVEN, forces agents to develop an actual communication protocol. They perform the needed ablation to show its benefit compared to I-RAVEN.\n\nFurthermore, this paper is well-written, and a detailed description of the setting, and hyper-parameters are provided (on top of the code).\n\nWEAKNESSES:\nThe main weakness of this work is its motivation. As stated in the paper, the goal of the emergent communication framework is to:\n- either study the origin of the human languages\nand/or\n- develop intelligent communicating artificial agents\n\nIt is unclear what this work's position is. If the former, is there a theory that our language emerged to communicate about a high-level reasoning task? If so, can this line of work be clarified in the paper? If the goal is to develop communicating agents, communicating about visual inputs is more practical for human-agent interactions.\n\nQUESTIONS:\nCan you explain further the experiments of the paragraph \"Rule-RAVEN dataset\"  (line 260)? In particular, I don't understand why we have an \"unsuccessful\" communication game with \"rule-RAVEN\" dataset if the speaker was already trained (in a two-stage setting). That is, in this training regime, the speaker's language is not random, and the listener should be able to succeed in the communication game (or at least have a good enough accuracy) without modifying the speaker's language). Maybe you can elaborate more on the stage 1 training?\n\nAlso, you state that \"for the listener’s train accuracy still achieves ~0.9 even if the speaker’s message is completely ignored\". How do you check if the speaker's message is completely ignored?\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper takes the ever-popular Lewis signalling game for emergent\ncommunication and studies experiments on rule-focused communication as opposed\nto the perception-focused communication of prior work.  In particular, it uses\na modified version of Raven's progressive matrices to formulate a signalling\ngame directly on attribute-value vectors which requires pattern\nrecognition/reasoning to complete.  The authors find that agents can learn to\ncommunicate when using a two-stage curriculum which essentially pretrains the\nspeaker for more stable communication at the beginning of the communication\nstage.  The experiments demonstrate that the resulting emergent language\ncorrelates better with the underlying rules of observations rather than the\nindividual observations themselves.\n\nSTRENGTHS:\n## Originality\n- `[major]` Addresses the signalling game from a new perspective, i.e.,\n  reasoning instead of perception.\n- `[minor]` Introduces a new dataset.\n## Quality\n- `[major]` Presents good variety of empirical evaluation with clear results\n- `[minor]` Presents different levels and senses of \"generalization\".\n## Clarity\n- `[minor]` Details for implementation are presented without being overwhelming\n## Significance\n*See Originality.*\n\nWEAKNESSES:\nThe paper is relatively complete, but what keeps my rating from being higher is\nthat there is relatively sparse comparison with prior work.  Such comparison\nwould better contextualize the results and increase its significance.  For\nfurther details, see the *Questions* section of the review.\n\nQUESTIONS:\n- How does this work compare with prior art which also uses categorical\n  variables for a signalling game, even if there is no reasoning element to the\n  game?\n- What are the particular effects on an emergent language of\n  a reasoning-focused game versus one that is perception-focused?  What sort of\n  inductive biases are present in the task that has a downstream effect on the\n  language?\n\n## Minor Comments/Questions\n- `Line 46` \"inner\": typo?\n- `Line 46, 86` \"inter\": typo?\n- `Line 122` Do not use curly braces for ordered sequences; use parentheses.\n- `Paragaph @ 119` Why is it necessary for there to be no ambiguity?  This sort of ambiguity shows up frequently in human communication.\n- `Line 251` \"cause\" -> \"because\"\n- `Paragaph @ 246` I would use \"shows\" or \"demonstrates\" instead of \"proves\" since it is not a formal, mathematical proof.\n- Use the default LaTeX placement of tables/figures instead of `[h]`; the former is less distracting.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new environment along with the training framework for emergent communication of abstract rules. They designed a context generation pipeline rule-RAVEN to avoid overfitting and a two-stage curriculum training method for more stable convergence. They evaluated the emerged language from the perspectives of generalization and transfer learning.\n\nSTRENGTHS:\n1. This paper proposed a new research angle of abstract rule reasoning for emergent communication. The context requires the agent to go beyond the low-level perceptual features and communicate more abstract rules. \n2. The candidate pool is smartly designed to motivate agents to extract rules from the context.\n3. A suite of comprehensive evaluations is designed to measure the generalization of the emerged languages.\n\nWEAKNESSES:\n1. Structural requirements of the new benchmark may need to be further explained: I am not clear about why the rules must be unambiguous. From my understanding, though the multiple rules can be applied to the current context, as long as the agents can communicate either of the rules, the receiver should capture the correct candidate? Though Figure 4 demonstrates the receiver can select the candidate correctly without the sender’s message trained with rule-RAVEN, further experiments/explanations are still needed to show that:\n\n    a. The communication training benefits from the structural or functional requirement or both.\n\n    b. The language that emerged using the I-RAVEN dataset is not/less generalizable/compositional/transferable. \n\n2. Sender’s rule reasoning and perception encoding are entangled. In the first stage, both $g^S$ and $f^S$ are trained. Though the training data is not shown in the communication stage, it will still introduce structural information because of the term $\\mathbb{1}(r_i, m_i)$. \n\n    a. does that require the length of the messages to equal the size of the rules?\n\nQUESTIONS:\nIn Sec 5.1, how distance(rule) and distance(panel) are computed? Just want to clarify whether these two distances are comparable.\n\nLIMITATIONS:\n1. Though the goal of the task is to emerge the language for abstract rules, it will also be interesting to know whether the receiver can learn to induce rules after the communication (instead of applying rules during the communication, not required experiments). Similar to ETL, you can test the accuracy of the reasoning problem on the communicated receiver without further training.\n2. As the author mentioned, the current context input is a structured symbol. It will strongly encourage compositional language. It will be interesting to know how agents can emerge languages in a raw pixel input.\n3. Can the emerged languages generalize to contexts with different attributes?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel setting for abstract reasoning (i.e., RAVEN problems) by proposing a speaker-listener framework for communicating higher-level abstract rules. The authors propose an unbiased dataset (rule-RAVEN) to overcome overfitting in the original RAVEN-family datasets (I-RAVEN), and propose a two-stage curriculum agent training method for successful communication. Experiments have shown the efficacy of the curriculum training for solving the rule-RAVEN and out-of-distribution generalization.\n\nSTRENGTHS:\n- I like the idea of both: i). introducing communicative game settings to abstract reasoning tasks, and ii). see how higher-level relational abstractions (instead of low-level perceptual features) emerge in communicative games. The limited capacity communication channel formulation can lead to emergent abstractions for problem-solving, including more powerful representations for abstract reasoning and concept learning. Previous attempts in drawing are good cases but not complex enough to depict the importance of abstraction and emergent language. This preliminary trial on RAVEN tests sets a suitable problem formulation for emergent communication in abstract reasoning.\n\n- The rule-RAVEN dataset effectively mitigates the existing bias in the I-RAVEN dataset, making the speaker-listener communication valid.\n\n- The paper is well-written and easy to read. The flow of writing in section 5 is also appropriate for addressing potential concerns for readers.\n\nWEAKNESSES:\nAlthough I like the task settings in this paper, the experiment and proposed methods appear to have some weaknesses. I list them as follows:\n\n- The communicative formulation is very similar to Mu & Goodman, 2021. It seems this work (communicative RAVEN) is a special case of generalization, shifting from learning object-centric, attribute-level concepts (e.g., shape red or blue) to learning relational concepts (number-increasing). Authors should address more comparisons to these existing formulations.\n\n- The evaluation for emergent language is still quite limited. For example, can you probe the learned language to see if it can be linearly projected to some algebraic representations for relational concepts (e.g., the \"number increasing\" concept can be described as a multiplication matrix in Zhang et al., 2022) or just explicitly manipulate them and see if they have some language-like syntax or compositionality emerged.\n\n- The use of symbolic RAVEN and two-stage curriculum training (with the first stage supervised learned) made me doubt the applicability of this communicative method to more complex or real-world tasks. For example, Mu & Goodman, 2021 used a real-world dataset, pixel input, and end-to-end training.\n\nrefs:\n1. Mu, J., & Goodman, N. (2021). Emergent communication of generalizations. Advances in Neural Information Processing Systems, 34, 17994-18007.\n\n2. Zhang, C., Xie, S., Jia, B., Wu, Y. N., Zhu, S. C., & Zhu, Y. (2022, October). Learning algebraic representation for systematic generalization in abstract reasoning. In European Conference on Computer Vision (pp. 692-709). Cham: Springer Nature Switzerland.\n\nQUESTIONS:\nSee the weakness section. I am open to changing my score, so I hope the authors can address these concerns.\n\nLIMITATIONS:\nThe authors did not address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an emergent communication game over abstract visual concepts, inspired by Raven's progressive matrices tests. The basic idea is to evaluate neural speakers and listeners on a communication game, where the speaker sees a collection of images encoding some abstract rule (e.g. \"number of objects in the image is increasing\"); the speaker must then generate a message that allows a listener to complete an unseen sequence. The authors show that agents trained to play this game indeed seem to learn to communicate the abstract rules for which they are trained for, as measured by intrinsic measures of language compositionality and ease of transfer to harder tasks.\n\nSTRENGTHS:\n- This is an interesting dataset and interesting problem in emergent communication which may be useful to the community. It indeed explores more abstract visual concepts than in existing work (though note that novelty over the existing EC literature is overclaimed; see Weaknesses).\n- Careful controls for dataset difficulty (ensuring one distinct feature that can be used to solve each task; ensuring \"hard negative\" rules) show the authors' care to making sure this is a well-constructed dataset, including an analysis of to what extent existing \n- Interesting experimental analysis shows that models seem to be (to some extent) communicating abstract rules, rather than superficial input features.\n\nWEAKNESSES:\n- Only a synthetic dataset consisting of clean symbolic inputs is evaluated. One could imagine more realistic settings requiring communication of rules at least over synthetic visual inputs, if not more realistic visual concepts. Similarly, there is no exploration of downstream transfer to other tasks that perhaps don't involve emergent communication, e.g. instruction following or visual reasoning. While this does not preclude publication, there's not a lot one can gain from this paper as it relates to actual realistic ML tasks. If the paper were to be rejected, IMO it would likely be because the experiments are just a little too synthetic/marginal to be useful to the broader NeurIPS community.\n- The claim that existing work in EC does not at all care about expresing abstract generalizations or rules is a bit overblown. Separating inputs given to the student and teacher, so as to facilitate communication of abstract concepts, was introduced as early as Lazaridou (2017), recurs in Choi et al., Kiela et al., etc. Mu and Goodman (2022) also propose generalizations over abstract visual concepts involving multiple visual inputs, which is very similar to the task presented here. I do think the present work makes some interesting contributions over the existing literature, in that it is even more abstract, but the relation to existing work needs to be made more clear. Many of these papers are not discussed in detail and simply bucketed as  \"forcing agents to descrie low-level features of images\" (L31-32) which I believe is false. Section 2 Emergent Communication also completely neglects to discuss such efforts in the EC community.\n- I think it's important for footnote 1 to be made more clear in the text, i.e. that this is not a grounded communication game over real images, despite many of the introductory figures seemingly suggesting this.\n\n## Minor\n\n- Title of paper and title on OpenReview do not match\n- spaces between text and citations would be ideal\n- It'd be interesting to see how pretraning agents on such visual reasoning communication tasks might improve performance on downstream visual reasoning tasks such as ARC (Chollet et al., ?)\n- The description of the paragraphs in L119 and L128 as \"structural\" and \"functional\" requirements is a little confusing and nonstandard to me—it's not clear what structural and functional mean here. It might be appropriate for example to refer to the \"functional requirement\" as sampling \"hard negative distractors\", as is used in the terminology for contrastive learning for example. In other words, distractors should be sampled carefully so as to represent close but not quite correct rules that force the speaker and listener to communicate precisely the right rules.\n- L171 \"directly from the sketch\" -> \"directly from scratch\"?\n\nQUESTIONS:\n- Did authors try varying the number of context panels given to the speaker? Or even show the partial sequence given to the listener? Wonder how this affects the languages' propensity to communicate abstractions; e.g. if the speaker sees the listener's partial sequence, does the language still communicate the abstract rule, or does the speaker internally learn the abstract rule but nevertheless convey the perceptual features (e.g. \"single triangle\")?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}