SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuxuan Guo $^{1,2,3}$  Yifan Hao $^{2}$  Rui Zhang $^{2}$  Enshuai Zhou $^{1,2,3}$  Zidong Du $^{2,5}$   
Xishan Zhang $^{2,3}$  Xinkai Song $^{2}$  Yuanbo Wen $^{2}$  Yongwei Zhao $^{2}$  Xuehai Zhou $^{1}$   
Jiaming Guo $^{2}$  Qi Yi $^{1,2,3}$  Shaohui Peng $^{6}$  Di Huang $^{2}$  Ruizhi Chen $^{6}$   
Qi Guo $^{2}$  Yunji Chen $^{2,4*}$
<sup>1</sup>University of Science and Technology of China
$^{2}$ State Key Lab of Processors, Institute of Computing Technology, CAS
$^{3}$ Cambricon Technologies  $^{4}$ University of Chinese Academy of Sciences
5Shanghai Innovation Center for Processor Technologies
$^{6}$ Intelligent Software Research Center, Institute of Software, CAS
gyx_20170818@mail.ustc.edu.cn, {haoyifan, cyj}@ict.ac.cn
Research on emergent communication between deep-learning-based agents has received extensive attention due to its inspiration for linguistics and artificial intelligence. However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forces agents to describe low-level perceptual features intra image or symbol contexts. In this work, inspired by the classic human reasoning test (namely Raven's Progressive Matrix), we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Moreover, we propose 1) an unbiased dataset (namely rule-RAVEN) as a benchmark to avoid overfitting, 2) and a two-stage curriculum agent training method as a baseline for more stable convergence in the Reasoning Game, where contexts and semantics are bilaterally drifting. Experimental results show that, in the Reasoning Game, a semantically stable and compositional language emerges to solve reasoning problems. The emerged language helps agents apply the extracted rules to the generalization of unseen context attributes, and to the transfer between different context attributes or even tasks.
Research on emergent communication has received extensive attention in recent years due to its inspiration to the fields of linguistics and artificial intelligence (AI). From the linguistic perspective, the study of emergent communication may help understand the origin and evolution of natural language [9, 13, 14, 28, 40]. From the AI perspective, introducing communication mechanism into the deep-learning-based multi-agents system is a promising way to make it interpretable [11], human-interactable [25], and more importantly, generalizable [1, 7, 31, 37].
Lewis-style Game [27] is widely used in previous studies to simulate emergent communication between two agents, a speaker, and a listener. In this game, the speaker sends a message based on perceived contexts (usually images or symbols), and the listener completes the perception task (usually discriminating or reconstructing the target object) based on the message. As two agents
Figure 1: Perception-oriented environment (Left, i.e., speak by seeing) v.s. cognition-oriented environment (Right, i.e., speak by reasoning) for emergent communication.
gradually endow messages with precise semantics to describe the target object more accurately, a machine language emerges with communication.
However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forcing agents to describe low-level perceptual features (e.g., shape, color and combinations of them) intra image [11, 14, 15, 23, 24, 31] or symbol [28, 35-37] contexts. Unfortunately, such perception-oriented environments (i.e., speak by seeing) are unable to prompt agents to emerge cognitive-based communication (i.e., speak by reasoning), which is considered the foundation of human language and intelligence evolution by research in linguistics [12] and cognitive psychology [6]. Therefore, how to emerge communication from perception to cognition remains an unsolved issue.
In this work, we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Figure 1 shows the comparison between the perception-oriented environment and cognition-oriented environment. The Reasoning Game requires two agents to cooperatively solve a classic human reasoning test Raven Progressive Matrix [34] (RPM). Specifically, the speaker reasons underlying rules from two rows of contexts (consisting of visually simple attributes), and sends a message to the listener. Meanwhile, the listener must correctly parse the received message and grasp the extracted rules, so as to select a candidate option that matches the rules (and fills the blank in the last row of the contexts matrix). In the process of solving RPM by agents, the rules communicated are not related to perceived features intra a single context but rather stem from abstract and structural logical relationships inter multiple contexts [39], such as gradually deepening colors and increasing numbers. According to this characteristic of RPM, agents can only complete the Reasoning Game by reasoning implicit rules at the cognitive level, instead of explicit visual features at the perceptual level. Such cognition-oriented environmental settings are considered to characterize the defining feature of high-level human-like intelligence [6, 18, 48].
Moreover, to conveniently evaluate the cognitive ability of agents through the Reasoning Game, we propose 1) an unbiased rule-RAVEN dataset as a benchmark, and 2) a two-stage curriculum agent training method as a baseline. For the former, the rule-RAVEN dataset contains numerous RPM problems with different combinations of rules, and each problem consists of context panels with multiple attributes. The dataset is debiased by our rule-based candidate panel generation algorithm to avoid overfitting in agent communication. For the latter, the two-stage curriculum agent training method ensures agents converge more stably in the Reasoning Game, where contexts and semantics are bilaterally drifting.
We summarize the contributions of this paper with experimental results:
- (Sec 5.1) The Reasoning Game successfully enables the emergent communication for reasoning rules between agents, and the emerged language is semantically stable and compositional.
- (Sec 5.2) The proposed benchmark (i.e., the rule-RAVEN dataset) and baseline (i.e., the two-stage curriculum agent training method) in the Reasoning Game are effective as expected and necessary for agent evaluation.
- (Sec 5.3) We verify the improvement of agent cognitive ability through the emergent communication for reasoning rules from three aspects: 1) generalize to unseen rule combinations, 2) apply the extracted rules to other context attributes, 3) and transfer to different tasks.
Figure 2: The Reasoning Game
Emergent Communication. Using computational models to simulate emergent communication has a long history [21, 22, 43] and has recently been revived with the popularity of deep learning. A. Lazaridou et al. [24] studied emergent communication on the basis of reinforcement learning and deep neural network agents for the first time. Starting from this, dozens of studies focus on exploring the properties of the emerged language from the perspective of linguistics and cognitive psychology, such as alignment of agents [11], efficiency of language [8], capacity of communication channel [23], community interaction [15], ease of teaching [28], culture transmission [14], population heterogeneity [31], and environment scale [10]. In general, these studies adopt perception-oriented environmental settings, restricting the emerged languages to describing low-level perceptual features intra image or symbol contexts. Specifically, previous works [11, 15, 24] emerge communication of intra-context object attributes (e.g., color, shape) and their combinations (e.g., blue OR/AND triangle) [31]. On the contrary, our work focuses on emergent communication in a cognition-oriented environment (i.e., Reasoning Game), where agents reason inter-context high-level rules and apply them, so as to cooperatively solve reasoning tasks (i.e., RPM).
Raven's Progressive Matrices (RPM). RPM [34] is a widely studied human IQ test, which has recently been revived in the AI community after it can be procedurally generated [44]. Compared with other reasoning benchmarks, such as Visual Question Answering [4, 19, 29] and Arithmetic Reasoning [38], RPM focuses on relational reasoning inter multiple contexts, reflecting humans' core cognitive abilities [6]. Based on RPM, many synthetic [2, 48] and natural [41] datasets and model designs [17, 46] are proposed to boost the high-level reasoning capability of deep learning models. In this work, we are the first to study emergent communication through cooperatively solving RPM problems. We find that such communication scenarios need to ensure clear semantics and avoid overfitting so that imposing new requirements on the rules set design and candidate panels generation of the RPM dataset. Motivated by this, we propose the rule-RAVEN dataset as a benchmark, and its corresponding two-stage curriculum agent training method as a baseline.
3.1 Reasoning Game
The Reasoning Game is a cognition-oriented environment that requires two agents to solve RPM problems by reasoning and communicating high-level rules. The game can be formalized as a 5-tuple:  $(C, Q, A, R, y)$ , where  $C = (c_{1}, \ldots, c_{l})$  is a set of  $l$  context panels,  $Q = (q_{1}, \ldots, q_{m})$  is a set of  $m$  question panels,  $A = (a_{1}, \ldots, a_{n})$  is a set of  $n$  candidate panels with 1 target panel and  $n - 1$  distract panels,  $R = (r_{1}, \ldots, r_{k})$  is a vector that encodes the rules implicit in  $C$  (i.e., hidden to the speaker and listener), and  $y \in \{1, \ldots, n\}$  indicates the index of the target panel, the only panel together with the question panels that match the rule  $R$ . Each panel contains several attributes (such as number, color, et al.)  $T = (t_{1}, \ldots, t_{k})$ , and the values of all attributes are within a pre-defined range.
Figure 2 shows the scheme of the Reasoning Game. In this scheme, the speaker first reasons the embedding, which represents the rules  $R$  implicit in context panels  $C$ , then encodes the embedding
Figure 3: (left) A reasoning problem example in the rule-RAVEN dataset, where the candidate panels are generated by introducing distract rules, shown in Algorithm 1. (right) Previous RPM-based datasets (e.g., I-RAVEN) generate candidates by hierarchically modifying some attribute values of the target panel, making it easy for the listener to distinguish without considering communication from the speaker.
into a message  $M = (m_{1},\ldots ,m_{s})$  (each token  $m_{i}$  is derived from a pre-defined vocabulary  $V$ ) for sending to the listener. Finally, according to the received message  $M$ , the listener chooses the target panel from candidate panels  $A$  to fill the blank behind the question panel  $Q$ . If the reasoning question is correctly solved, the listener and the speaker will be rewarded simultaneously.
3.2 Rule-RAVEN dataset
A reasoning problem (e.g., RPM) can be naturally divided into two steps: extracting rules and applying rules. It seems that the reasoning dataset (e.g., RAVEN) can be straightforwardly used to play the Reasoning Game by assigning the former step (i.e., extracting rules) to the speaker and the latter step (i.e., applying rules) to the listener. However, the premise for such a scheme of Reasoning Game could proceed successfully is that the speaker and listener can effectively transmit and understand the extracted rules by emergent communication. To this end, crafting a reasoning dataset as a benchmark still has requirements on the rules set design and candidate panels generation.
Regarding the rules set design, the mapping relationship between implicit rules and context attributes must be unambiguous to emerge semantically clear communication and avoid imbalanced prior probability distribution of rules' frequency in the dataset. Specifically, the design principle is subdivided into two aspects: 1) The rule corresponding to each set of attribute values in panels is unique. For example, if the attribute (e.g., size) value in context panels  $(c_{1}, c_{2}, c_{3})$  is  $(1, 2, 3)$ , which matches both rules 'sum'  $(1 + 2 = 3)$  and 'progression +1'  $(1 + 1 = 2, 2 + 1 = 3)$ , then the attribute value should be filtered out from the dataset. 2) The meanings expressed between different rules do not overlap. For example, when any set of attribute value  $(a, b, c)$  matches the rule 'progression'  $(a + const = b, b + const = c)$ , it also matches the rule 'average'  $(b = \frac{a + c}{2})$ . In this case, only one of the rules should be retained in the dataset.
Regarding the candidate panels generation, distract panels should be generated carefully so as to represent close but not quite correct rules that force the speaker and listener to communicate precisely the correct rules. Specifically, the design principle is that each distractor in candidate panels should match a 'distract rule' (as shown in figure 3 'our method'), to prevent the listener from inferring the target solely through the existence of regular rules between the question panels and candidate panels (a negative example is shown in figure 3 (right) 'I-RAVEN [17]'). Otherwise, it will cause the speaker to lose the motivation to extract and communicate rules correctly (we will demonstrate this in detail by experiments in Sec 5.2).
Following the above requirements, we propose a symbolic dataset  ${}^{2}$  ,rule-RIVEN,for the Reasoning Game. Figure 3 (left) shows an example case in the rule-RIVEN dataset (See Appendix A for more details.).
3.3 Model and Training
We build the speaker's and listener's model in the Reasoning Game as follows.
2The rule-RAVEN dataset is based on symbols rather than images, mainly because visual recognizability limits the attribute values.
Algorithm 1: Rule-based candidate panels generation algorithm
Input:  $Q,R$  , distractpanels_num Output: distractpanels // generate distract rule space   
1 possibleRules  $\equiv$  []   
2 for  $i = 0$  to  $|R| - 1$  do   
3 tmp_rule  $=$  []   
4 for  $r\in$  rule_set do   
5 if check_satisfy(Q[;, i], r) then   
6 tmp_rule.append(r)   
7 end   
8 end   
9 possiblerules.append(tmp_rule)   
10 end   
11 distract_rule_space  $=$  cartesian_product(possible_rule)- R // generate distract panels   
12 distractpanels  $\equiv$  []   
13 for rs  $\in$  random_sample(distract_rule_space, distractpanels_num) do   
14 tmppanel  $\equiv$  []   
15 for  $i = 0$  to  $|rs| - 1$  do   
16 tmp_attr  $=$  reason(rs[i], Q[;, i])   
17 tmppanel.append(tmp_attr)   
18 end   
19 distractpanels.append(tmppanel)   
20 end   
21 return distractpanels
Speaker's model. The speaker consumes context panels  $C$  and outputs message  $M$ . The speaker's policy can be formalized by  $S(M|C;\theta)$  with trainable parameters  $\theta$ . Given context panels  $C$ , an MLP-based perception module  $f^S$  first encodes each panel into an embedding  $(f^S(c_i))_{i=1}^6$ . Then, a reasoning module  $g^S$  extracts the row-level rule embedding  $r_1^S = g^S((f^S(c_i))_{i=1}^3)$  and  $r_2^S = g^S((f^S(c_i))_{i=4}^6)$  from multiple context embeddings. We follow the state-of-art RAVEN solver SCL [46]'s core module, shared group MLPs, as the reasoning module. The row-level rule embeddings are average-aggregated to get the rule embedding  $r^S = mean(r_1^S, r_2^S)$  of context panels  $C$ . Finally, an LSTM [16] message encoder  $h^S$  takes the rule embedding  $r^S$  as initial state  $z_0^S$  and a special token  $m_0 = <\mathsf{SOS}>$  as start token to generate message  $M$ . At each timestep  $t \geq 1$ , based on current state  $z_t = h^S(z_{t-1}, m_{t-1})$ , message  $m_t = sample(\text{softmax}(q(z_t)))$ , where  $q$  is a linear transformation to project the hidden state space to the vocabulary space. Token generation terminates until the message reaches a fixed length or a special token  $<\mathsf{EOS}>$  is generated.
Listener's model. The listener consumes message  $M$ , question panels  $Q$ , and candidate panels  $A$  and outputs a prediction  $\hat{y}$ . The listener's policy can be formalized by  $L(\hat{y} | M, Q, A; \phi)$  with trainable parameters  $\phi$ . Given question panels  $Q$  and candidate panels  $A$ , a perceptual module  $f^L$  first encodes  $Q$  and  $A$  into corresponding embeddings  $(f^L(q_i))_{i=1}^2$  and  $(f^L(a_i))_{i=1}^8$ . Meanwhile, an LSTM message decoder  $h^L$  takes a special state  $z_0^L$  as the start state to decode message  $M$ . At each timestep  $t \geq 1$ , current state  $z_t^L = h^L(z_{t-1}^L, m_t)$ . Message decoding terminates until it reaches the maximum message length  $|M|$  or <EOS>. We use the final state  $z^L$  as the decoded message embedding, representing the rules implied in context panels  $C$ . Then, each candidate embedding is concatenated with the question embedding and fed into the reasoning module  $g^L$  to obtain multiple candidate rule embeddings  $r_i^L = g^L(f^L(q_1), f^L(q_2), f^L(a_i)), i \in \{1, \ldots, 8\}$ . Finally, the message embedding and the candidate rule embedding perform an inner product and softmax operation to obtain the probability distribution softmax(( $r_i' \cdot z_{|M|}^T$ )_{i=1}^8) of candidate panels, which is further used to compute the loss and the prediction  $\hat{y}$ .
To ensure that the cognitive capabilities of the speaker and the listener are similar, we build their perception and reasoning module architecture identically. No parameters are shared between the
speaker and listener to avoid introducing artificial priors about their consensus. More details of the model can be found in Appendix B.
In our game, both rules reasoning and language emergence are highly challenging. We find that jointly training the speaker and listener directly from scratch leads to communication failures. The reason is that, in the early stage of joint training, the reasoning context panels and the message semantics are drifting simultaneously, leading the speaker cannot produce helpful messages to the listener, and the listener tends to ignore the speaker's message. To help agents jump out of the local optimum, we propose a two-stage curriculum training [3] method.
In the first stage, the speaker is pre-trained on a held-out rule set (See Appendix A for more details.) generated dataset to acquire some reasoning capabilities to cope with the drifting context. This stage can be considered a supervised learning task with the ground-truth rule vector  $R$  as the label and only updating the speaker's parameters  $\theta$ . Note that this does not impose artificial priors on emerged language since the rule encodings predicted at this stage are absent in the joint training stage. The gradients of the objective function  $J_{1}$  in the first stage are:
$$
\nabla_ {\theta} J _ {1} = \mathbb {E} [ \sum_ {i = 1} ^ {k} \mathbb {1} (r _ {i}, m _ {i}) \nabla \log S (m _ {i} | C) ], r _ {i} \in R, m _ {i} \in M
$$
Note that although the  $\mathbb{1}(\cdot ,\cdot)$  operator exists in  $\nabla_{\theta}J_{1}$ , this does not introduce constraints on the size of emerged language (i.e., message_length and vocabulary_size) in the second stage of training. See Appendix C for more details.
In the second stage, the pre-trained speaker and listener train jointly to emerge language. Since discrete communication breaks the end-to-end differentiability, we employ the REINFORCE[45] algorithm. The gradients of the objective function  $J_{2}$  in the second stage are:
$$
\nabla_ {\theta} J _ {2} = \mathbb {E} \left[ \mathbb {1} (\hat {y}, y) \nabla \log S (M | C) \right] + \lambda \cdot \nabla H [ S (M | C) ]
$$
$$
\nabla_ {\phi} J _ {2} = \mathbb {E} [ \mathbb {1} (\hat {y}, y) \nabla \log L (\hat {y} | M, Q, A) ]
$$
Where  $\lambda$  is a hyperparameter controlling entropy regularization.
4.1 Basic Settings
Implementation. We use Python3 [42] to implement the rule-RAVEN dataset. Our model implementation is based on Pytorch [33] and EGG [20] toolkit. The code is available in supplementary materials.
Dataset preparation. We instantiate our rule-RAVEN dataset (introduced in Sec 3.2). For each case of reasoning problem, there are 6 context panels (i.e.,  $C = (c_{1},\ldots ,c_{6})$ ), 2 question panels (i.e.,  $Q = (q_{1},q_{2})$ ), and 8 cadidate panels (i.e.,  $A = (a_{1},\dots ,a_{8})$ ). Each panel has 4 attributes (i.e.,  $T =$  (number, color, size, shape)), and each attribute have  $N\in \{20,30,40\}$  values ( $N^4$  in total). The larger  $N$  (more diverse the attribute values) means it is more difficult to identify contexts and reason rules. There are 8 rules that apply to each attribute respectively, resulting in a total of  $8^{4} = 4096$  different rule combinations. For each rule combination, We randomly generated 20 different reasoning problem cases (sampling from  $N^4$  attribute values) with Algorithm 1 ( $4096\times 20 = 81920$  in total), and half of these cases for training, half for testing. Besides, we use a held-out rule set to additionally generate 2000 cases for pre-training the speaker in the first stage of agent training (introduced in Sec 3.3).
Communication Channel. We set the maximum message length  $|M| = 4$  and the vocabulary size  $|V| = 15$ . With this setup, the capacity of the communication channel is  $15^4$ , sufficient to describe all rule combinations ( $>8^4$ ), and preserve redundant tokens for two-stage curriculum training, and much lower than vocabularies required to directly describe the context panels ( $\sim 10^{10}$ ).
**Optimization.** Agents' parameters are optimized by AdamW [30], with a learning rage of  $3 \cdot 10^{-3}$ , a weight decay of 0.01,  $\beta_{1} = 0.99$ ,  $\beta_{2} = 0.999$  and a batch size of 512. For the speaker, we set the hyperparameter of entropy regularization  $\lambda = 0.01$ .
4.2 Language Metrics of emergent communication
Generalization abilities. To test the generalization ability of the emerged language, we further create 4 data splits, corresponding to 4 levels of generalization ability: 1) In-distribution generalization (ID). The training and test sets share the same rule combinations but consist of different problems. The ID level tests the language's ability to generalize to different problems with the same rules. 2) Interpolated out-of-distribution generalization (Inpo-ood). Each rule in the rule set appears in each attribute on the train set, but some rule combinations do not appear on the train set and form the test set. The Inpo-ood level tests the language's ability to generalize to unseen rule combinations based on components. 3) Extrapolated out-of-distribution generalization level-1 (Expo-ood-L1). Each attribute has a unique rule that does not appear in the train set, and these rules are combined with other rules that appear in the train set to form a test set. The Expo-ood-L1 level tests the language's ability to describe rules across attributes. 4) Extrapolated out-of-distribution generalization level-2 (Expo-ood-L2). A rule does not appear on any attribute on the train set, and it is combined with other rules present in the train set to form the test set. The Expo-ood-L2 level tests the language's ability to generalize to new rules.
Topographic similarity (Topsim). Topsim [5] is widely used in previous emergent language studies to measure language compositionality. It is defined as the Spearman correlation coefficient of pairwise distances between input and message space. In the input space, we use the normalized Hamming distance between rules and the cosine distance between panels. In the message space, we adopt the Levenshtein [26] distance with the same removal, insertion, and substitution costs. Computing Topsim involves pairwise distances and thus has a computational complexity of  $O(n^{2})$ , which is computationally prohibitive in our environment. In the experiment, we randomly sample 1000 samples per run to calculate Topsim and take the average over 20 runs as the final result.