SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zixing Lei $^{1}$ , Yiming Zhang $^{2}$ , Yuxin Xiong $^{1}$ , Siheng Chen $^{1,3*}$
chezacarss@sjtu.edu.cn; yz2926@cornell.edu;
xyx1323@sjtu.edu.cn; sihengc@sjtu.edu.cn
<sup>1</sup> Cooperative Medianet Innovation Center, Shanghai Jiao Tong University,
$^{2}$  Cornell University,  $^{3}$  Shanghai AI Laboratory
Figure 1: In the proposed interactive sketch question answering (ISQA) task, two collaborative players are interacting to answer a question about an image. This task emphasizes multi-round interaction, which is essential in daily human communication.
Vision-based emergent communication (EC) aims to learn to communicate through sketches and demystify the evolution of human communication. Ironically, previous works neglect multi-round interaction, which is indispensable in human communication. To fill this gap, we first introduce a novel Interactive Sketch Question Answering (ISQA) task, where two collaborative players are interacting through sketches to answer a question about an image in a multi-round manner. To accomplish this task, we design a new and efficient interactive EC system, which can achieve an effective balance among three evaluation factors, including the question answering accuracy, drawing complexity and human interpretability. Our experimental results including human evaluation demonstrate that multi-round interactive mechanism facilitates targeted and efficient communication between intelligent agents with decent human interpretability. The code is available at here.
Emergent communication aims for a better understanding of human language evolution and is a promising direction for achieving human-like communication between intelligent agents [1, 2, 3, 4]. An ultimate EC system allows intelligent agents to interact and exchange information with one another, facilitating collaboration in the completion of downstream tasks. EC is expected to play an important role in a wide range of applications, including multi-robot navigation [5], collaborative perception [6] and human-centric AI applications [7].
Previous methods on EC can be divided into language-based EC and vision-based EC. Language-based EC aims to encode information into message vectors or nature language for transmission between intelligent agents [1, 2, 3, 8, 9, 10, 11, 12]. Vision-based EC systems have been developed with the aim of imitating the pictographic systems that were used in early human communication, and have shown promising results in achieving human interpretability [13, 14]. A representative vision-based EC approach [13] is inspired by the popular game "Draw & Guess," where the sender acts as a sketcher while the receiver is responsible for making guesses. Sketch is a visible and
universal form to transmit information, which is suitable for building human-interpretable systems. Following this spirit, this work specifically focuses on sketch-based EC.
In this emerging direction, current works on sketch-based EC show two critical issues. First, from the aspect of task setting, multi-round interaction is underexplored. As a fundamental aspect of human communication, interaction involves ongoing feedback and adjustment, facilitating bidirectional communication between intelligent agents. However, previous works on EC largely focused on single-round communication for the tasks of image classification or Lewis's game [15]. A recent work [14] proposes a multi-round mechanism, but its receiver is still limited to producing binary flags, indicating whether or not it is confident of its decision. This plain feedback cannot provide sufficient feedback for the sender to promote meaningful interaction. Second, in terms of evaluation, previous works on sketch-based EC primarily focus on optimizing for the performances of downstream tasks without sufficient consideration of communication efficiency and quantitative interpretability. This would result in complex and non-interpretable information interchange, which is not suitable for human-centric AI systems.
To address the task setting issue, we propose a new multi-round interactive task, named interactive sketch question answering (ISQA); see an illustration in Fig. 1. In this task, there are two players: a sender and a receiver. An RBG image is provided to the sender and the question related to this image is given to the receiver. Based on the image, the sender can generate a sketch and transmit it to the receiver with certain drawing complexity constraints. According to the sketch, the receiver generates an answer to the question. If the answer is wrong, the receiver can send spatial bounding boxes to the sender as feedback, which indicate which parts of the sketch require more detailed drawing in subsequent rounds. Through multiple rounds of interaction, the receiver gains more information about the image and achieves better question answering performance. Compared to previous tasks, the new task of ISQA calls more necessity of interaction because it creates a game of incomplete information for both participants, where the sender does not have access to the language question and the receiver does not know the original image. This setting prompts them to engage in bidirectional information exchange through a multi-round interaction.
To address the evaluation issue, we develop a comprehensive set of metrics, covering the task performance, drawing complexity and human interpretability, to evaluate the performance in the ISQA tasks. The system's task performance is assessed by measuring its accuracy in performing ISQA tasks. The system's drawing complexity is determined by analyzing the amount of pixels the sender transmits and digits used in the interactive feedback. The system's human interpretability is determined by the CLIP distance metric between sketch and input image. By integrating these three metrics, a comprehensive evaluation of the EC system's effectiveness can be established.
Based on the ISQA task and the corresponding metrics, we propose and evaluate a novel EC system, which consists of a sender module and a receiver module. The sender module is a variable drawing complexity sketcher, and the receiver includes an SQA module and a feedback module which selects region of high relevance about the question in the sketch. The SQA module is specifically trained to adopt sketches and the feedback module provides a gradient analysis based on the answer distribution calculated by the SQA module, enabling it to select high-relevance regions in the sketch and feedback to the sender. Our experiments including human evaluation show that our EC system provides distinct performance enhancements when compared to non-interaction sketch-based EC systems, and also maintains adequate human interpretability across various complexity constraints. This validates that based on sketches, intelligent agents can emerge interactive collaboration within constrained complexity. The proposed method may pave a path to build interactive agents that can work with both human and other agents by considering others' priorities and exchanging crucial information to promote effective collaboration. Our contribution can be summarized as four aspects:
- We propose a novel interactive sketch question answering (ISQA) task for EC.
- We design an efficient and interactive EC system to improve communication quality.
- We propose a novel three-factor evaluation metric, covering question answering performance, drawing complexity and human-interpretability.
- Our experiments show that interactive EC system leverages a targeted and efficient communication.
2.1 Emergent communication
Emergent communication refers to the phenomenon where communication emerges among intelligent agents that share a task or incentive which can be better achieved when intelligent agents cooperate
with each other. One view of EC[16, 17] focuses on utility by framing messages to deliver high performance in downstream task. The other view inspired by a cognitive science and information theory perspective [18, 19, 20] focuses on task-agnostic communicative objectives as major forces shaping human languages [21, 22]. [23] integrates these two views of communication by optimizing a tradeoff among utility, informativeness and complexity.
Sketching is an efficient and direct way for communication. [13] designs a sender and receiver structure which allows two agents to communicate by playing "Draw & Guess" game, which requires the receiver to learn how to associate the sketch drawn by the sender with a corresponding image. However, the game is played in only one round and the receiver does not provide any information to the sender. [14] designs a multi-round communication game which allows the sender and receiver to communicate in multiple rounds to explores the change of iconicity, symbolicity, and semanticity during the iterations of training procedure. During the game, the sketch will be updated by the sender which is built upon a reinforcement learning model [24] each round until the receiver is confident to make a decision. Despite the multi-round mechanism being in place, the receiver is still constrained to output a binary flag that simply indicates whether to continue drawing or not. This cannot provide information to guide the generation of a better sketch in the subsequent round. To tackle this issue, our work focuses on multi-round interaction and the receiver can provide informative feedback.
2.2 Visual question answering
Visual Question Answering [25] task enables the interaction between texts and images, which has been a topic of concern in recent years. Early VQA models [26, 27, 28] often use CNN to extract image feature and RNN to embed the question. Then attention-based models [29, 30, 31] are introduced to focus on specific regions of images and questions. After transformer [32] is proposed, transformer-based models [33, 34, 35] further boost the performance of VQA models. The deep Modular Co-Attention Network (MCAN) [35] is one of the transformer-based models with high performance. It was evaluated on the benchmark VQA-v2 dataset [36] and won the VQA Challenge 2019. Our SQA model uses the structure of MCAN to predict the answer from sketches and questions.
2.3 Sketch generation
Drawing is a direct and efficient way for communication. The endeavor of teaching neural models to generate sketch is started by Sketch-RNN [37]. To extract more features and make the sketch more explainable, the encoder-decoder structure is used in many sketching models [38, 39]. After CLIP [40] is proposed, CLIP-guidance sketching methods [41, 42] are able to generate high-quality sketches, but the efficiency is low due to the iterative methods. [43] uses depth information, CLIP features, and the LSGAN setup [44] to train a sketch model, which achieves a good compromise between sketch quality and efficiency. Our sketch generator in the SQA game is based on [43] model.
3 Interactive Sketch Question Answering
3.1 Task formulation
In the task of interactive sketch question answering (ISQA), two collaborative players, a sender and a receiver, are interacting through sketches to answer a question about an image. The sender does not know the question and the receiver cannot see the image. The sender takes an image as input and generates a sketch with specific drawing complexity. This sketch is then transmitted to the receiver. The receiver's job is to answer a given question about the original image through the sketch from the sender. If the receiver is uncertain about the answer, it will feedback a drawing with several bounding boxes, indicating uncertain areas and requesting more detailed information about the original image from the sender. The sender, in turn, generates a new sketch according on the receiver's feedback to assist the receiver to answer the question.
Mathematically, in the  $i$ th interaction round, let  $\mathbf{X} \in \mathcal{R}^{H \times W \times 3}$  be an RGB image,  $a$  be the scalar value to indicate the human-interpretability level,  $b_i$  be the scalar indicator of the drawing complexity level in the  $i$ th interaction round,  $\mathbf{H}_i$  be the feedback provided by receiver in the  $i$ th round, which is initialized by zeros. Then, the sender obtains the sketch as follows,
$$
\mathbf {S} _ {i} = f _ {\theta_ {s}} \left(\mathbf {X}, \mathbf {H} _ {i - 1}, b _ {i}, a\right) \in [ 0, 1 ] ^ {H \times W}, \tag {1}
$$
where  $f_{\theta_s}(\cdot)$  is the sketch generation network and  $\mathbf{S}_i$  is the sketch generated in  $i$ th interaction round. Note that the value of each pixel in  $\mathbf{S}_i$  is ranging from 0, indicating a black and activated pixel to 1, reflecting a white and unactivated pixel.
Correspondingly, let  $\mathbf{Q}$  be a language-based question. In the  $i$ th interaction round, the receiver works as follow,
$$
\hat {\mathbf {A}} _ {i}, \mathbf {H} _ {i} = f _ {\theta_ {r}} (\mathbf {Q}, \left\{\mathbf {S} _ {\tau} \right\} _ {\tau = 1, 2, \dots , i}), \tag {2}
$$
where  $f_{\theta_r}(\cdot)$  is the sketch question answering network,  $\widehat{\mathbf{A}}_i$  is the predicted answer distribution that reflects the probability of each alternative answer, and  $\mathbf{H}_i$  is a feedback sketch transmitted to sender, which consists of a set of bounding boxes indicating the spatial areas that have a high correlation with the problem.
3.2 Triangular evaluation
To quantify the overall performance of two collaborative players in both training and inference phases, we consider three factors: question answering ability, drawing complexity, and human interpretability.
Question answering ability. To reflect the performance of SQA in the training phase, we consider the binary cross-entropy (BCE) loss between the ground-truth answer  $\mathbf{A}$  and the predicted answer  $\widehat{\mathbf{A}}$ ; that is,  $L_{1} = \mathrm{BCE}(\mathbf{A}, \widehat{\mathbf{A}})$ . Note that the binary cross-entropy(BCE) loss is commonly used in the VQA task. In inference, we consider the answering accuracy following the common VQA methods.
Drawing complexity. To reflect the drawing complexity in both training and inference phases, we count the number of activated pixels in a sketch. When provided with a complexity constraint  $b_{i}$ , and  $N$  denotes the number of pixels in original RGB image, only  $b_{i}N$  pixels are permitted to be activated. In multi-round setting, the sender may not take full usage of  $b_{i}$  in every round and we let  $p_i$  to be the actual pixels transmitted in  $i$ th round and  $p_i \leq b_iN$ . Besides, the feedback complexity is also constrained. The complexity of feedback in  $i$ th round is  $5h_{i}$ , where  $h_i$  denotes the numbers of bounding boxes transmitted in the  $i$ th round since two points and a weight can represent a weighted bounding boxes. Therefore, the total drawing complexity is  $B = \sum_{i}(p_{i} + 5h_{i})$ .
Human interpretability. In the training phase, to maximize the interpretability in human vision, we can minimize the following loss,
$$
L _ {2} = \sum_ {\ell} \| \operatorname {C L I P} _ {\ell} \left(f _ {s} (\mathbf {X})\right) - \operatorname {C L I P} _ {\ell} (\mathbf {S}) \| _ {2} ^ {2} - \cos - \sin \left(\operatorname {C L I P} \left(f _ {s} (\mathbf {X})\right), \operatorname {C L I P} (\mathbf {S})\right), \tag {3}
$$
where  $\mathrm{CLIP}_{\ell}(\cdot)[40]$  is the CLIP visual encoder activation at layer  $\ell$ ,  $\mathrm{CLIP}(\cdot)$  is the CLIP visual encoder that outputs a feature vector,  $\cos -\sin (\cdot)$  is the cosine similarity function and  $f_{s}(\cdot)$  is a pre-trained sketch generation network [43]. The first term of the loss function emphasizes local geometric guidance, while the second term focuses on a semantic perspective to globally guide the sketch generation process.
In the inference phase, we also utilize  $L_{2}$  in (3) as the evaluation criterion for human interpretability, where a lower  $L_{2}$  loss indicates higher interpretability of the generated sketches for human perception. By integrating distance of multilevel geometric features and high-level semantic features, this metric leverages the CLIP model which is trained in 400 million (image, text) pairs and shows great zero-shot performance in vision-related task to imitate human vision.
Overall, we aim to minimize  $L = L_{1} + f_{\mathrm{balance}}(a) \cdot L_{2}$ , under various levels of complexity constraints. The hyper-parameter  $a$  controls the relative weight of the  $L_{1}$  and  $L_{2}$  losses, determining whether the model prioritizes human interpretability or SQA accuracy. To ensure a balanced trade-off between the two loss terms, the function  $f_{\mathrm{balance}}(a)$  is introduced to adjust the weight of  $L_{1}$  with respect to  $L_{2}$ . A default choice for  $f_{\mathrm{balance}}(a)$  is to set it to  $f_{\mathrm{balance}}(a) = 10a$ .
The proposed metric enables quantitative evaluation of both accuracy and human interpretability across various drawing complexity levels. Utilizing a multi-level CLIP loss function, human interpretability can be effectively measured, ranging from shallow geometric to deep semantic perspectives. By optimizing both BCE loss and CLIP loss under different complexity constraints, the ISQA system can achieve a decent tradeoff in the triangular metric.
4 Emergent Communication System
4.1 System overview
Based on the task formulation and the evaluation in the previous section, this section proposes an emergent communication system to accomplish this task; see an overall illustration in Figure 2. The proposed system comprises a sender module and a receiver module. Both are trainable networks to realize the roles of the sender and receiver in this task. We now elaborate the details of each module.
4.2 Sender Module
Sender module is designed to improve the drawing ability given specific requirements, including the drawing complexity and human-interpretability, so that the receiver can easily capture information.
Overall procedure. Here we consider a drawing pipeline with a varying complexity. Let  $\mathbf{X}$  be the input RGB image,  $\mathbf{H}_i$  be the feedback sketch in the  $i$ th round,  $a$  be the human interpretability level that determines the coefficient of  $L_2$  in Eq 3 and  $b_i$  is the drawing complexity of the sketches. In the  $i$ th round, the sketch  $\mathbf{S}_i$  generated by the sender can be obtained as,
$$
\mathbf {F} _ {\text {g e o}} = f _ {\text {g e o}} (\mathbf {X}), \mathbf {F} _ {\text {p r a g}} = f _ {\text {p r a g}} (\mathbf {X}), \tag {4a}
$$
$$
\mathbf {F} _ {\text {f u s i o n}} = f _ {\text {f u s i o n}} \left(\mathbf {F} _ {\text {g e o}}, \mathbf {F} _ {\text {p r a g}}, a\right), \tag {4b}
$$
$$
\widehat {\mathbf {S}} = f _ {\text {s k e t c h}} \left(\mathbf {F} _ {\text {f u s i o n}}, \sum_ {\tau = 1} ^ {i} b _ {\tau}\right), \tag {4c}
$$
$$
\mathbf {S} _ {i} = f _ {\text {s e l e c t}} \left(\widehat {\mathbf {S}}, \mathbf {H} _ {i}, b _ {i}\right), \tag {4d}
$$
where  $\mathbf{F}_{\mathrm{geo}}$ ,  $\mathbf{F}_{\mathrm{prag}}$ ,  $\mathbf{F}_{\mathrm{fusion}}$ , are feature maps,  $\widehat{\mathbf{S}}$  represents the sketch before undergoing the selection procedure to obtain the hard complexity constraints.
Image encoding. In Step (4a), we consider two encoders: a geometric encoder, which extracts geometric features for better sketching, and a pragmatic encoder, which extracts pragmatic features for better semantic information. The geometric encoder  $f_{\mathrm{geo}}(\cdot)$  encodes an RGB image into high-dimensional representations. The geometric encoder  $f_{\mathrm{geo}}(\cdot)$  is a pretrained model with fixed parameters introduced in [43] with a generative adversarial network. This encoder is designed to generate sketches with high geometrical similarity and can extract features that are closely relevant to the process of sketch generation. In addition, another pragmatic encoder,  $f_{\mathrm{prag}}(\cdot)$ , which consists of several ResNet[45] blocks, is trained to minimize loss  $L$  under the game setting. The inclusion of a trainable pragmatic encoder in the proposed model provides the sender with the ability to encode the input image into highly compressed features that are suited to situations where the complexity constraint is set to be extremely low.
Feature fusion. In Step (4b), the feature fusion module  $f_{\mathrm{fusion}}(\cdot)$  is used to fuse geometric features and pragmatic features based on a given different human interpretability level. To implement,  $f_{\mathrm{fusion}}(\cdot)$ , we consider a simple convex combination of geometric and pragmatic features:
$$
\mathbf {F} _ {\text {f u s i o n}} = \operatorname {D e c o n v} \left(a \mathbf {F} _ {\text {g e o}} + (1 - a) \mathbf {F} _ {\text {p r a g}}\right), \tag {5}
$$
where  $a$  is the hyper-parameter that controls the relative weight of the  $L_{1}$ ,  $L_{2}$  losses and Deconv () is a deconvolutional network. When  $a = 1$ , the objective is to minimize the CLIP distance between the generated sketch and the original image, thereby maximizing human interpretability. Conversely, when  $a = 0$ , the objective is to maximize the SQA performance without considering human interpretability. By tuning the hyper-parameter  $a$ , the sender can adjust the weight balance between human interpretability and downstream SQA task performance.
Sketch generation. Step (4c) generates the sketch from the fused feature with the drawing complexity constraints. Initially, the drawing complexity  $b_{i}$  is transformed into an indication vector, which is further encoded using a multi-layer perceptron to produce the drawing-complexity matrix. The drawing-complexity matrix is then concatenated to  $\mathbf{F}_{\mathrm{fusion}}$ , yielding a complexity-aware feature map. This feature map is then fed to the decoder, which is composed of a U-Net[46] and multiple residual blocks to decode the complexity-aware feature map into the reconstructed sketch  $\widehat{\mathbf{S}} \in \mathcal{R}^{H \times W}$ .
Pixel Selection. To ensure that the communication consumption does not exceed the upper bound, all pixels are ranked according to their value, and the top  $b_{i}N$  pixels are selected to display in the final
Figure 2: System overview with two interaction rounds.
sketch, while the other pixels are discarded, where  $N$  denotes the number of pixels in original RGB image. By combining soft complexity encoding and hard selection procedures, we build a network that is guided by complexity constraints to generate suitable sketches for different situations, while the hard selection ensures that the complexity will not be exceeded. The selection mechanism can be naturally extended to select crucial pixels in multi-round settings. After receiving  $\mathbf{H}_i$ , the sender rerank the importance of each pixel according to their value times corresponding weight in  $\mathbf{H}_i$ . After removing previously transmitted pixels, the sender adds the top n remaining pixels to the sketch in subsequent rounds.
4.3 Receiver Module
The duty of the receiver module is to answer the question according to the sender module's sketch and provide feedback. The key of the receiver module is to associate the question with the critical spatial areas in the sketch. Our receiver module consists of two parts: SQA and interactive feedback mechanism. SQA is a modified VQA module that takes questions and sketches as input and select an answer from all the options. Interactive feedback module consists of a reasoning module that couples tightly with the detector-based vision encoder and a selection mechanism.
SQA module. The SQA module is primarily based on MCAN [35], with the addition of our pretrained sketch Faster-RCNN. Let  $\mathbf{Q}$  be the question,  $\mathbf{S}_i$  be the sketch, and  $\hat{\mathbf{A}}_i$  be the predicted answer distribution in the  $i$ th round. The SQA module works as follow,
$$
\mathbf {F} _ {\text {l a n g u a g e}} = f _ {\text {l a n g u a g e}} (\mathbf {Q}), \tag {6a}
$$
$$
\mathbf {P} _ {i}, \mathbf {F} _ {\text {v i s i o n}, i} = f _ {\text {v i s i o n}} \left(\sum_ {\tau = 1} ^ {i} \mathbf {S} _ {\tau}\right), \tag {6b}
$$
$$
\hat {\mathbf {A}} _ {i} = f _ {\text {c o a t t n}} \left(\mathbf {F} _ {\text {l a n g u a g e}}, \mathbf {F} _ {\text {v i s i o n}, i}\right), \tag {6c}
$$