SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathbf{P}_i$ ,  $\mathbf{F}_{\mathrm{vision},i}$  and  $\widehat{\mathbf{A}}_i$  denote the proposals, vision feature and answer distribution in the  $i$ -th round, respectively, and  $\mathbf{F}_{\mathrm{language}}$  is the language feature. In Step (6a), questions are tokenized into words and transformed into vectors using pre-trained word embeddings  $f_{\mathrm{language}}(\cdot)$  [47]. In Step (6b), a Faster-RCNN-based [48] visual encoder  $f_{\mathrm{vision}}(\cdot)$  extracts features from the accumulated sketch and
return corresponding proposals  $\mathbf{P}_i$ . To train this sketch encoder, we create the sketch-VG dataset[49] by utilizing [43] and pretrain the vision encoder on that to replace RGB-based decoder on MCAN [35]. The visual and language features are subsequently passed on to the modular co-attention layers in the MCAN architecture,  $f_{\mathrm{coattn}}(\cdot)$  in Step (6c) to calculate the  $\widehat{\mathbf{A}}_i$ , while the proposals  $\mathbf{P}$  are forwarded to the interactive feedback module for further attention allocation.
Interactive feedback module. To facilitate efficient interactive feedback with human interpretability, it is natural to return a feedback sketch that indicates which parts of the sketch are more relevant to the question. To obtain this feedback sketch, our intuition is to analyze the relevance between the predicted answer distribution and each feature via the well-known Grad-CAM [50] method. In the  $i$ th round, the feedback module works as follow,
$$
\beta_ {i} ^ {k} = \sum_ {j} \frac {\partial \sum_ {v = 1} ^ {l} \widehat {\mathbf {A}} ^ {v}}{\partial \mathbf {F} _ {\text {v i s i o n} , i} ^ {k , j}} \in \mathcal {R}, \tag {7a}
$$
$$
w _ {i} ^ {j} = \operatorname {R e L U} \left(\sum_ {k} \beta_ {i} ^ {k} \mathbf {F} _ {\text {v i s i o n}, i} ^ {k, j}\right) \in \mathcal {R}, \tag {7b}
$$
$$
\mathbf {H} _ {i} ^ {j} = \frac {w _ {i} ^ {j}}{E _ {i} ^ {j}} \mathbf {P} _ {i} ^ {j} \in \mathcal {R} ^ {H \times W}, \tag {7c}
$$
$$
\mathbf {H} _ {i} = \sum_ {j} ^ {n} \mathbf {H} _ {i} ^ {j} \in \mathcal {R} ^ {H \times W}, \tag {7d}
$$
where  $\beta_i^k$  represents the weight of the feature map in the  $k$ th channel,  $v$  denotes the index of top  $l$  possible answer,  $\widehat{\mathbf{A}}^v$  denotes the probability of answer index  $v$ ,  $\mathbf{F}_{\mathrm{vision},i}^{k,j}$  refers to the vision feature of proposal  $j$  in channel  $k$ ,  $E_i^j$  is the area of proposal  $j$  and  $\mathbf{P}_i^j$  is a mask matrix reflecting the spatial area of a proposal, where the element located in the  $j$ -th proposal is 1, and 0 elsewhere,  $\mathbf{H}_i^j$  is a weighted mask matrix associated with proposal  $j$  whose weight reflects its importance and  $\mathbf{H}_i$  is the feedback sketch in the  $i$ th round.
In Step (7a), we calculate the weight  $\beta^k$  of each channel by computing the partial derivative  $\widehat{\mathbf{A}}$  with respect to  $\mathbf{F}_{\mathrm{vision}}$  following the original Grad-CAM procedure. In Step (7b), we utilize  $\beta^k$  to sum up the channel dimension and get weight  $w^j$  of each proposal  $j$ . Combining these two steps, we leverage the gradient flowing to assign importance value to each feature vector of a particular proposal. In Step (7c), the attention matrix for proposal  $j$  is computed by dividing the weight of the proposal by its area, in order to avoid bias towards large proposals that may otherwise dominate the attention. In Step (7d), we aggregate the bounding boxes of each proposal to obtain the final feedback sketch  $\mathbf{H}_i$ . Notably,  $\mathbf{H}_i$  can be simply represented as a bounding box and the corresponding weight, which only needs five numbers, significantly reducing the drawing complexity.
Visual Question Answering (VQA) v2.0 [36] is a dataset that contains open-ended questions about images that require an understanding of vision, language and commonsense knowledge to answer. To fit with our SQA game, we remove all questions that include color-related word since sketch has no information about color. This reduces the size of dataset by about  $15\%$ . Visual Genome (VG) [49] is a dataset that contains 108K images with densely annotated objects, attributes and relationships. Based on VG dataset, we generate the corresponding sketches by the sketching model [43] to train our Faster-RCNN model required by ISQA tasks. The evaluation metrics of VG datasets include the mean average precision(mAP) with IOU threshold 0.5(mAP@50) and the mAP@0.5 weighted by the number of boxes for each class(weighted mAP@50).
5.2 Experimental setting
With aforementioned optimization object  $L$ , we trained three different models with SQA module pretrained under VQAv2 dataset and the images are transferred to sketches with [43]:
1. Pragmatic: trained to minimize  $L_{1}$ , equivalent to minimize  $L$  with  $a = 0$ .
2. Geometric: trained to minimize  $L_{2}$ , equivalent to minimize  $L$  with  $a = 1$ .
3.  $PraGeo$ : trained to minimize  $L$  with  $a = 0.5$ .
Figure 3: Comparison of Pragmatic, Geometric, Prageo(left) and Prageo models with different rounds(right). The  $x$ -axis is  $\sum b_{i}$ .
Question: What are these animals?
Image
Figure 4: Multi-round interactive SQA. From left to right we display RGB image, sketch transmitted in round 1, RGB image masked with  $\mathbf{H}_1$ , extra pixels added in round 2, and the whole sketches in round 2.
Round I sketch
Round I answer: Horse?
Feedback sketch
(included masked Image)
Round2 sketch
Round2 Answer: Giraffes!
Accumulated sketch:
5.3 Quantitative evaluation
Fig 3 compares the ISQA accuracies of Pragmatic, Geometric and PraGeo models as a function of the drawing complexity  $B$ , where  $x$ -axis is  $\sum b_i$  and  $y$ -axis is the answer accuracy. For PraGeo, we consider both one-round and multi-round settings.
Comparison among Pragmatic, PraGeo and Geometric models. 1) When  $B$  is set to be lower than  $0.3N$ , Pragmatic outperforms other methods in overall accuracy and PraGeo outperforms Geometric in complexity situations. 2) When  $B$  is larger than  $0.3N$ , the accuracy of geometric model catches up with other methods'. 3) The peak accuracy of pragmatic model is lower than other models' since Pragmatic is easier to overfit in low-complexity situations.
Effectiveness of interactive feedback mechanism. 1) Multi-round interaction can improve the SQA accuracy. 2) The accuracy gap between one-round  $\text{PraGeo}$  and multi-round  $\text{PraGeo}$  are more significant when complexity  $B \in (0.05N, 0.3N)$  since: when the complexity is too low reasoning module is unable to infer sufficient useful information and interaction cost extra bandwidth consumption; when complexity is appropriate, the bandwidth expended on this extra interaction is relatively negligible and reasoning module generate effective feedback according to the round 1 sketches; when the complexity is too high, most of the crucial information is transmitted in round 1. 3) When comparing two-round interaction to three-round interaction, we see that the difference is minor when compared to the gap observed between one-round and two-round interaction.
Human interpretability evaluation. Table 1 compares the human interpretability of Pragmatic, Geometric and PraGeo by calculating CLIP distance. We see that i) without any constraint in training, Pragmatic has a poor human interpretability, and ii) PraGeo is much closer to Geometric.
Pragmatic	PragGeo	Geometric
Human interpretability Score	0.998 ± 0.002	0.600 ± 0.068	0.478 ± 0.108
CLIP Distance	0.9077	0.1815	0.1193
Table 1: Average CLIP distances and human evaluation of three models.
To verify the consistency between CLIP and manual measurement, we conduct a human evaluation, where 12 participants were tasked with assessing the human-interpretability of three different models. Participants were presented with three corresponding images from each model simultaneously and asked to score them based on judgment criteria ranging from 0 to 1. (lower is better, 0, 1 represent
Figure 5: Sketches under multiple drawing complexities. From left to right, there are RGB-images, sketch without complexity constraint, and sketch generated with PraGeo model, when  $B = 0.5N$ , 0.3N and 0.1N
fully understand and can not understand, respectively). We sampled 3,000 sets of images and each set comprised images generated by the Pragmatic, PragGeo, and Geometric models from the same RGB images. We see that: 1) the results in human experiments and the CLIP distance are consistent to show the order of Geometric, PragGeo Pragmatic in terms of human interpretability; 2) both PragGeo and Geometric models provide an human interpretable sketches while Pragmatic models can not be understood; and 3) Comprehensively considering with Fig.3, the PragGeo achieves a more optimal balance among the task performance and human interpretability.
Besides, to further verify the human interpretability, we sampled 300 images and questions pairs from the dataset and test human substituting as the receiver. Ten people are involved in this experiment and table. 2 shows the average accuracy and variance of people and machine receiver. We see that human achieve better accuracy in number question but failed to outperform machine in Yes-or-No and other question. The experiments show that our architecture can deliver a decent human interpretability and our sender can work with human.
Yes-or-No	Number	Other
Human	66.22 ± 4	39.71 ± 4.33	28.57 ± 11.30
Machine	70	26	38
Table 2: Results of human substituting as the receiver.
5.4 Qualitative evaluation
Multi-round interaction. Fig. 4 illustrates the procedure of multi-round interaction. We see: i) the receiver generates the feedback sketch which contains crucial region according to the preliminary assessment; ii) the sender provides extra pixels when round is more than 1 according to the feedback sketch; and iii) the receiver modifies the answer in accordance with the updated accumulated sketch.
Drawing complexity. Fig. 5 presents the sketches generated by PraGeo model under various drawing complexities. We see that our PraGeo model can preserve human interpretability when the drawing complexity is sufficiently large ( $B \geq 0.1N$ ).
Human interpretability comparison. Fig. 6 shows examples of sketches generated by the three models when  $B = 0.03N$ . We see that i) the sketches generated by the Pragmatic model are difficult for humans to understand; ii) PraGeo model can still capture the outlines of the bowls and chopsticks and outperform Pragmatic model.
Interactive feedback mechanism. Fig. 7 illustrates that interactive feedback mechanism generates  $\mathbf{H}$  according to specific questions. The first row shows the RGB image and corresponding sketch, from left to right. When the question is "What are the people in the background doing?", the interactive feedback module generates  $\mathbf{H}$  that specifically focuses on people to request more information about the area that is likely to contain humans. However, when the question is "What is he on top of?", the interactive feedback module choose to request more information about the area under the man.
Insights brought by ISQA task. First, information disparity is the prerequisite for interaction. Fig. 7 shows the feedback messages for the same image according to two different questions. We see that our feedback message transfers question-related information to the sender, enhancing communication efficiency via a gradient-informed region of interest (can be displayed as sketch as shown in Figure 2, which enables a human-like multi-round interaction for the first time in visual emergent communication. This provides an insight that one of the reasons why interaction emerges might be querying task-aligned information when receiver is more acquainted with the object than sender. Second, the constrain of communication complexity promotes interaction. Fig. 3 shows the ISQA performance comparison between multi-round and one-round communication. We see that the
Geometric
PraGeo
Figure 6: Sketches generated with different models when  $B = 0.03N$ .
What are the people in the background doing?
Figure 7: For the same image, different questions lead to distinct feedback sketches.
'What is he on top of?'
downstream performance can be boosted when interaction is introduced to the visual communication pipeline. This provides an insight that the complexity constrains promote the agents to pursue a more efficient communication leads to interaction. Third, it requires sufficient input information for the feedback to be functional. Fig. 3 shows that functional feedback from the receiver relies on adequate communication complexity. This offers the crucial realization that precise feedback hinges on a foundational shared understanding.
5.5 ISQA study
Due to the differences in input modality between SQA and VQA, the RGB-based model is not suitable for SQA. Two key components, Faster-RCNN vision encoder and MCAN question answering module, thus need to be trained with a sketch-based dataset. Table 3 evaluates two Faster-RCNN models. We see: i) the RGB-based model is not applicable to sketch-based dataset. ii) sketch-based detector works normally on sketches but not as well as RGB-based model on RGB images.
Table 4 evaluates the performance of two MCAN models. We see that i) the RGB-based model suffers significant performance degradation on sketch-based dataset. ii) the performance of sketch-based MCAN is acceptable while 10.16 percent lower than RGB-based model in RGB-based dataset, which also serves as the upper-bound of ISQA task.
model	dataset	mAP@0.5	Weighted mAP@0.5
RGB-based	RGB	0.082	0.149
RGB-based	Sketch	0.003	0.010
Sketch-based	Sketch	0.034	0.085
Table 3: Performance of Fast-RCNN-based models with ResNet50 backbone on VG and sketch-VG. The RGB-based model is trained on color images and the Sketch-based model is trained on sketches.
model	dataset	overall	other	yes-or-no	numbers
RGB-based	RGB	64.74	52.44	83.41	47.28
RGB-based	Sketch	40.86	20.88	67.48	23.19
Sketch-based	Sketch	54.58	40.81	74.81	37.01
Table 4: Accuracy of RGB-based MCAN and sketch-based MCAN on RGB/sketch-based VQAv2 dataset. The RGB-based model is trained on color images and the Sketch-based model is trained on sketches.
This study proposes a new interactive sketch question-answering task along with an interactive emergent communication (EC) system to facilitate bidirectional communication between two players. Our approach employs an interactive feedback mechanism that allows the EC system to provide feedbacks to guide sketch generation. Our experiments reveal that the proposed multi-round EC system can achieve an effective balance among question answering ability, drawing complexity and human interpretability.
Limitation and future work. The current interactive EC system is limited to two collaborative players. In the future, we are going to explore interactive emergent communication among a group of intelligent agents, which is more complex and tricky on the design of task and interactive process. Collective intelligence has shown its capacity in applications such as collaborative perception, UAV cluster technology, social network analysis, etc. We believe that multi-agent interaction is a promising direction for artificial intelligence research in the future.
Acknowledgement. This research is supported by NSFC under Grant 62171276 and the Science and Technology Commission of Shanghai Municipal under Grant 21511100900, 22511106101, and 22DZ2229005