Emergent Communication in Interactive Sketch Question Answering
Authors: Zixing Lei
Yiming Zhang
Yuxin Xiong
Siheng Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Emergent communication
Interactive
Question Answering
TL;DR: In this paper, we introduce a novel Interactive Sketch Question Answering task where two collaborative players are interacting through sketches to answer a question about an image.

Abstract: Vision-based emergent communication (EC) aims to learn to communicate through sketches and demystify the evolution of human communication. Ironically, previous works neglect multi-round interaction, which is indispensable in human communication. To fill this gap, we first introduce a novel Interactive Sketch Question Answering (ISQA) task, where two collaborative players are interacting through sketches to answer a question about an image. To accomplish this task, we design a new and efficient interactive EC system, which can achieve an effective balance among three evaluation factors, including the question answering accuracy, drawing complexity and human interpretability. Our experimental results demonstrate that multi-round interactive mechanism facilitates tar- geted and efficient communication between intelligent agents. The code will be released.

Emergent Communication in Interactive Sketch Question Answering
Zixing Lei', Yiming Zhang", Yuxin Xiong's Siheng Chen 1.3+ chezacarss@s edu. y229268 edu; xyx13238 jtu. edu, en; sihengces jtu. edu en Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Cornell University, 3 Shanghai Al Laboratory
Please dhaw the kwegaroo
Quastion: which offection obss the kwrgeroo's haaeeooooo///////goo go?
Handa maybo crucial
Pease craw haad!
Ol Head coucill
laft!
In the proposed interactive sketch question answering (ISQA) task, two collaborative players interacting to answer question about an image. This task emphasizes multi-round interaction, which is essential in daily human communication.
Figure I:
Abstract
Vision-based emergent communication (EC) aims to learn communicate through sketches and demystify the evolution of human communication. Ironically, pre- vious works neglect multi-round interaction, which is indispensable human communication. To fill this gap, we first introduce a novel Interactive Sketch Question Answering (ISQA) task, where two collaborative players are interacting through sketches to answer question about an image in multi-round manner. To accomplish this task, we design a new and efficient interactive EC system, which can achieve an effective balance among three evaluation factors, including the question answering accuracy, drawing complexity and human interpretability. Our experimental results including human evaluation demonstrate that multi-round interactive mechanism facilitates targeted and efficient communication between intelligent agents with decent human interpretability. The code is available at here.
Introduction
Emergent communication aims for better understanding of human language evolution and is promising direction for achieving human-like communication between intelligent agents [1, 2, 3, 4]. An ultimate EC system allows intelligent agents interact and exchange information with one another, facilitating collaboration in the completion of downstream tasks. EC is expected to play an important role in a wide range applications, including multi-robot navigation [5], collaborative perception [6] and human-centric Al applications [7]-
Previous methods on EC can be divided into language-based EC and vision-based EC. Language- based EC aims to encode information into message vectors or nature language for transmission between intelligent agents [1, 3, 8, 9, 10, 11, 12]. Vision-based EC systems have been developed with the aim of imitating the pictographic systems that were used in early human communication. and have shown promising results in achieving human interpretability [13, 14]. A representative dision-based EC approach [13] is inspired by the popular game "Draw & Guess," where the sender acts as sketcher while the receiver is responsible for making guesses. Sketch is a visible and
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

universal form to transmit information, which is suitable for building human cumaan-nterpretabble systems. Following this spirit, this work specifically focuses on sketch-based EC.
In this emerging direction, current works on sketch-based EC show two critical issues. First, from the aspect of task setting, multi-round interaction is underexplored. As fundamental aspect of human communication, interaction involves ongoing feedback and adjustment, facilitating bidirectional communication between intelligent agents. However, previous works on EC largely focused on single-round communication for the tasks of image classification or Lewis's game [15]. A recent work [14] proposes a multi-round round mechanism, but receiver still limited to producing binary flags, indicating whether or not it is confident of its decision. This plain feedback cannot provide sufficient feedback for the sender to promote meaningful interaction. Second, in terms of evaluation, previous works on sketch-based EC primarily focus on optimizing for the performances of downstream tasks without sufficient consideration of communication efficiency and quantitive interpretability. This would result in complex and non-interpretable information interchange, which is not suitable for human-centric Al systems.
To address the task setting issue, we propose a new multi-round interactive task, named interactive sketch question answering (ISQA); see an illustration in Fig. In this task, there are two players: sender and a receiver. An RBG image is provided to the sender and the question related to this image given to the receiver. Based on the image, the sender can generate a sketch and transmit to the receiver with certain drawing complexity constraints. According to the sketch, the receiver generates an answer to the question. If the answer wrong, the receiver can send spatial bounding boxes to the sender as feedback, which indicate which parts of the sketch require more detailed drawing in subsequent rounds. Through multiple rounds of interaction, the receiver gains more information about the image and achieves better question answering performance. Compared to previous tasks, the new task of ISQA calls more necessity of interaction because it creates a game of incomplete information for both paricipants, where the sender does not have access to the language question and the receiver does not know the original image. This setting prompts them to engage in bidirectional information exchange through a multi-round interaction.
To address the evaluation issue, we develop a comprehensive set of metrics, covering the task performance, drawing complexity and human interpretability, to evaluate the performance in the ISQA tasks. The system's 's task performance is assessed by measuring its accuracy performing ISQA tasks. The system's 's drawing complexity is determined by analyzing the amount of pixels the sender transmits and digits used in the interactive feedback. The system's 's human interpretability is determined by the CLIP distance metric between sketch and input image. By integrating these three metries, a comprehensive evaluation of the EC system's effectiveness can be established.
Based on the ISQA task and the corresponding metrics, we propose and evaluate a novel EC system, which consists of sender module and a receiver module. The sender module a variable drawing complexity sketcher, and the receiver includes an SQA module and a feedback module which selects region of high relevance about the question in the sketch. The SQA module is specifically trained to adopt sketches and the feedback module provides a gradient analysis based on the answer distribution calculated by the SQA module, enabling it to select high-relevance regions in the sketch and feedback to the sender. Our experiments including human evaluation show that our EC system provides distinct performance enhancements when compared to non-interaction sketch-based based EC systems, and also maintains adequate human interpretability across various complexity constraints. This validates that based on sketches, intelligent agents can emerge interactive collaboration within constrained complexity. The proposed method may pave path to build interactive agents that can work with both human and other agents by considering others' priorities and exchanging crucial information to promote effective collaboration. Our contribution can be summarized as four aspects:
We propose a novel interactive sketch question answering (ISQA) task for EC.
We design an efficient and interactive EC system to improve communication quality.
We propose a novel three-factor evaluation metric, covering question answering performance,
drawing complexity and human- intrraaattttr....biittt...
Our experiments show that interactive EC system leverages a targeted and efficient communication.
2 Related work
2.1 Emergent communication
Emergent communication refers to the phenomenon where communication emerges among intelligent agents that share task or incentive which can be better achieved when intelligent agents cooperate

---

with each other. One view of EC[16, 17] focuses on utility by framing messages to deliver high performance in downstream task. The other view inspired by a cognitive science and information theory perspective [18, 19, 20] focuses on task-agnostic communicative objectives as major forces shaping human languages [21, 22]. [23] integrates these two views of communication by optimizing a tradeoff among utility, informativeness and complexity.
Sketching is an efficient and direct way for communication. [13] designs a sender and receiver structure which allows two agents to communicate by playing "Draw & Guess" game, which requires the receiver to learn how to associate the sketch drawn by the sender with a corresponding image. However, the game is played in only one round and the receiver does not provide any information to the sender. [14] designs multi-round communication game which allows the sender and receiver to communicate in multiple rounds to explores the change of iconicity, symbolicity, and semanticity during the iterations of training procedure. During the game, the sketch will be updated by the sender which is built upon reinforcement learning model [24] each round until the receiver is confident to make a decision. Despite the multi-round mechanism being in place, the receiver is still constrained to output binary flag that simply indicates whether to continue drawing or not. This cannot provide information to guide the generation of a better sketch in the subsequent round. To tackle this issue, our work focuses on multi-round interaction and the receiver can provide informative feedback.
2.2 Visual question answering
Visual Question Answering [25] task enables the interaction between texts and images, which has been topic of concern in recent years. Early VQA models [26, 27, 28] often use CNN extract image feature and RNN to embed the question. Then attention-based models [29, 30, 31J are introduced to focus on specific regions of images and questions. After transformer [32] is proposed. transformer- based models [33, 34, 35] further boost the performance of VQA models. The deep Modular Co-Attention Network (MCAN) [35] is one of the transformer-basee models with high performance. It was evaluated on the benchmark VQA-v2 dataset [36] and won the VQA Challenge 2019. Our SQA model uses the structure of MCAN to predict the answer from sketches and questions.
2.3 Sketch generation
Drawing is direct and efficient way for communication. The endeavor of teaching neural models to generate sketch is started by Sketch-RNN -NNN [37]. To extract more features and make the sketch more explainable, the encoder-decoder structure is used in many sketching models [38, 39]. After CLIP [40] is proposed, CLIP-guidance sketching methods [41, 42] are able to generate high-quality sketches, but the efficiency is low due to the iterative methods. [43] uses depth information, CLIP features, and the LSGAN setup [44] to train a sketch model, which achieves a good compromise between sketch quality and efficiency. Our sketch generator in the SQA game is based on [43] model.
3 Interactive Sketch Question Answering
3.1 Task formulation
In the task of interactive sketch question answering (ISQA), two collaborative players, a sender and a receiver, are interacting through sketches to answer a question about image. The sender does not know the question and the receiver cannot see the image. The sender takes an image as input and generates sketch with specific drawing complexity. This sketch is then transmitted to the receiver. The receiver's ' job is to answer a given question about the original image through the sketch from the sender. If the receiver is uncertain about the answer, it will feedback a drawing with several bounding boxes, indicating uncertain areas and requesting more detailed information about the original image from the sender. The sender, in turn, generates new sketch according on the receiver's feedback to assist the receiver to answer the question.
Mathematically, in the ith interaction round, let X E R Hx333 be an RGB image, be the scalar value to indicate the numan-inteppettabiiiyy level, b; be the scalar indicator of the drawing complexity level in the ith interaction round, H, be the feedback provided by receiver in the ith round, which is initialized by zeros. Then, the sender obtains the sketch as follows.
S fe. (X, H. 1,,,,, [0, 1 HXW
(I)
3

---

where fo. (-) is the sketch generation network and S; is the sketch generated in ith interaction round. Note that the value of each pixel in S; ranging from 0, indicating a black and activated pixel reflecting a white and unactivated pixel.
Correspondingly, let be language- based question. In the äth interaction round, the receiver works as follow,
A, H. = So- (0.1S.)r=12.-.1
(2)
where fo. (.) is the sketch question answering network, A; is the predicted answer distribution that reflects the probability of each alternative answer, and H; is feedback sketch transmitted to sender, which consists of a set of bounding boxes indicating the spatial areas that have a high correlation with the problem.
3.2 Triangular evaluation
To quantify the overall performance of two collaborative players in both training and inference phases, we consider three factors: question answering ability, drawing complexity, and human interpretability.
Question answering ability. To reflect the performance of SQA in the training phase, we consider the binary cross-entropy (BCE) loss between the ground-truie -ruth answer and the predicted answer A; that is, L1 BCE(A, (A, A). Note that the binary cross-entropy(BCIL loss commonly used in the VQA task. In inference, we consider the asnwering accuracy following the common VQA methods.
Drawing complexity. To reflect the drawing complexity in both training and inference phases, we count the number of activated pixels in sketch. When provided with complexity constraint b;. and N denotes the number of pixels in original RGB image, only b, pixels are permitted to be activated. In multi-round setting, the sender may not take full usage of b; in every round and we let P: to be the actual pixels transmitted in ith round and p; S N. Besides, the feedhack complexity also constrained. The complexity of feedback in ith round is 5/;, where h; denotes the numbers of bounding boxes transmitted in the ith round since two points and weight can represent a weighted bounding boxes. Therefore, the total drawing complexity is B (p: + 5h.).
Human interpretability. In the training phase, to maximize the interpretability in human vision, we can minimize the following loss
L2 Σ |CLIP& (f. (X)) CLIP; (S)1 cos-sim (CLIP (fa (X)), CLIP(S))
where CLIP, (-)440] is the CLIP visual encoder activation at layer f, CLIP (-) is the CLIP visual encoder that outputs feature vector, cos-sim (-) is the cosine similarity function and fa(-) is pre-trained sketch generation network [43]. The first term of the loss function emphasizes local geometric guidance, while the second term focuses on semantic perspective to globally guide the sketch generation process.
In the inference phase, we also utilize L2 in (3) as the evaluation criterion for human interpretability. where lower L2 loss indicates higher interpretability of the generated sketches for human perception. By integrating distance of multilevel geometric features and high-level semantic features, this metric leverages the CLIP model which is trained in 400 million (image, text) pairs and shows great zero-shot performance in vision-related task imitate human vision.
Overall, we aim to minimize L L1 + fuatanse L2, under various levels of complexity constraints. The hyper-paramettr controls the relative weight of the L1 and La losses, determining whether the model prioritizes human interpretability or SQA accuracy. To ensure balanced trade-off between the two loss terms, the function fbulance (a) is introduced to adjust the weight of L1 with respect to L2.
default choice for ftalance (a) is to set it to fualance (a) = 10a.
The proposed metric enables quantitative evaluation of both accuracy and human interpretability across various drawing complexity levels. Utilizing a multi-level CLIP loss function, human inter- pretability can be effectively measured, ranging from shallow geometric to deep semantic perspectives. By optimizing both BCE loss and CLIP loss under different complexity constraints, the ISQA system can achieve a decent tradeoff in the triangular metric.

---

Emergent Communication System
4.1 System overview
Based on the task formulation and the evaluation in the previous section, this section proposes emergent communication system to accomplish this task; see an overall illustration in Figure 2. The proposed system comprises sender module and a receiver module. Both are trainable networks to realize the roles of the sender and receiver in this task. We now elaborate the details of each module.
4.2 Sender Module
Sender module designed to improve the drawing ability given specific requirements, including the drawing complexity and human--nterpretabilityy, so that the receiver can easily capture information.
Overall procedure. Here consider drawing pipeline with varying complexity. Let X be the input RGB image, H, be the feedback sketch in the ith round, a the human interpretability level that determines the coefficient of L2 in Eq 3 and b; is the drawing complexity of the sketches. In the ith round, the sketch S; generated by the sender can be obtained as,
g00 = fgco (X), F prug Spag (X);
(4a)
Jusion = ftusson (F goo: F prag, a),
(4b)
S Sakrch
F Finsions
bz
(4c)
S: Soler (S,H.. HH,b.)..
(4d)
where F goo F prag: F fusion, are feature maps, S represents the sketch before undergoing the selection procedure to obtain the hard complexity constraints.
Image encoding. In Step (4a), we consider two encoders: a geometric encoder, which extracts geometric features for better sketching, and a pragmatic encoder, which extracts pragmatic fea- tures for better semantic information. The geometric encoder f-uo(-) encodes RGB image into high-dimensional representations. The geometric encoder (-) is a pretrained model with fixed parameters introduced in [43] with a generative adversarial network. This encoder is designed to generate sketches with high geometrical similarity and can extract features that are closely relevant to the process of sketch generation. In addition, another pragmatic encoder, fprug (.). which consists of several ResNet[45] blocks, is trained to minimize loss J. under the game setting. The inclusion of a trainable pragmatic encoder in the proposed model provides the sender with the ability to encode
the input image into highly compressed features that are suited to situations where the complexity constraint is set be extremely low.
Feature fusion. In Step (4b), the feature fusion module frusion (.) is used to fuse geometric features and pragmatic features based on given different human interpretability level. To implement, frusion (-), we consider a simple convex combination of geometric and pragmatic features:
F Fusion Deconv (aF gen (1 a)F pmg).
(5)
where a is the hyper-parametee that controls the relative weight of the L1. L2 losses and Deconv a deconvlutional network. When a = 1, the objective is to minimize the CLIP distance between the generated sketch and the original image, thereby maximizing human interpretability. Conversely, when a = 0, the objective is to maximize the SQÁ performance without considering human inter- pretability. By tuning the hyper-parameter the sender can adjust the weight balance between human interpretability and downstream SQA task performance.
Sketch generation. Step (4c) generates the sketch from the fused feature with the drawing complexity constraints. Initially, the drawing complexity by is transformed into an indication vector, which further encoded using multi-layer perceptron to produce the trawing-complexin
matrix. The wwwwing-complxiigg matrix is then concatenated to F Fusionn yielding a complexity-awar feature map. This feature map is then fed to the decoder, which is composed of U-Net[46] and multiple residual feature map into the reconstructed sketch S R
blocks to decode the complexity
Pixel Selection. ensure that the communication consumption does not exceed the upper bound, all pixels are ranked according to their value, and the top b.N pixels are selected to display in the final
5

---

Drawing complexity
Whih droate Aargoncc learg??
Receiver
Sketcher
SQA
KNd diatte
SQA Module
kongnoo' raad gio
fouture
MCAN
Selection
Feedback
Vision Enc
Introurt
are round
Interactive Feedback
Skescher
SQA
Sender
Receiver
Sender
Feature Fusion
Sketch Generation
Praz
Drowing ixity
Pragmatic
ENC
Decoder
XO1 K0)
Selection
ENC
Figure System overview with two interaction rounds.
sketch, while the other pixels discarded, where N denotes the number of pixels in original RGB image. By combining soft complexity encoding and hard selection procedures, we build a network that is guided by complexity constraints generate suitable sketches for different situations, while the hard selection ensures that the complexity will not be exceeded. The selection mechanism can be naturally extended select crucial pixels multi-round settings. After receiving H, the sender rerank the importance of each pixel according to their value times corresponding weight in H, After removing previously transmitted pixels, the sender adds the top remaining pixels to the sketch in subsequent rounds.
4.3 Receiver Module
The duty of the receiver module is to answer the question according to the sender module's sketch and provide feedback. The key of the receiver module is to associate the question with the critical spatial areas in the sketch. Our receiver module consists two parts: SQA and interactive feedback mechanism. SQA is modified VQA module that takes questions and sketches input and select an answer from all the options. Interactive feedback module consists reasoning module that couples tightly with the detector-based vision encoder and selection mechanism.
SQA module. The SQA module is primarily based on MCAN [35], with the addition our pretrained sketch Faster-RCNN, Let Q be the question, S, be the sketch, and A, be the predicted answer distribution in the ith round. The SQA module works follow,
F Panguage faaguage (Q),
(6a)
P. F Fvision.
fvisson Σ Sr):
(6b)
A feontta languages F Fvision.e):
(6c)
where P;, Fvision. and A, denote the proposals, vision feature and answer distribution the i-th round, respectively, and Fanguugge is the language feature. In Step (6a), questions are tokenized into words and transformed into vectors using pre-trained word embeddings Stangunge (.) [47]. In Step (6b), Raster-RCNN based [48] visual encoder fvision (-) extracts features from the accumulated sketch and

---

return corresponding proposals P,. To train this sketch encoder, we create the sketch- VG dataset[49) by utilizing [43] and pretrain the vision encoder on that to replace RGB-based decoder on MCAN [35]. The visual and language features are subsequently passed on to the modular co-attention layers in the
MCAN architecture, fouat(() (.) in Step (6c) to calculate the A; while the proposals P are forwarded to the interactive feedback module for further attention allocation.
Interactive feedback module. To facilitate efficient interactive feedback with human interpretabilltyy it is natural to return a feedback sketch that indicates which parts of the sketch are more relevant to the question. To obtain this feedback sketch, our intuition is to analyze the relevance between the predicted answer distribution and each feature via the well-known Grad-CAM [50] method. In the ith round, the feedback module works as follow,
E
8
Σ
E R,
(7a)
OF OF visina,
// = ReLU
Σ
F vision,
E R,
(7b)
H p RxW
(7c)
H = Σ H E E "xw
(7d)
where o represents the weight of the feature map in the keth channel, denotes the index of top / Fh4 possible answer, A denotes the probability of answer index v, vision.; refers to the vision feature of
proposal j in channel k, E is the area of proposal and P is mask matrix reflecting the spatial area of proposal, where the element located in the j-th proposal is 1, and 0 elsewhere, H is weighted mask matrix associated with proposal j whose weight reflects its importance and H, is the feedback sketch in the ith round.
In Step (7a), we calculate the weight 3 of each channel by computing the partial derivative À with respect F Viision following the original Grad-CAM procedure. In Step (7b), we utilize 3 to sum up the channel dimension and get weight u> of each proposal j. Combining these two steps, we leverage the gradient flowing to assign importance value to each feature vector of particular proposal. In Step (7c), the attention matrix for proposal j is computed by dividing the weight of the proposal by its area, in order avoid bias towards large proposals that may otherwise dominate the attention. In Step (7d), we aggregate the bounding boxes of each proposal to obtain the final feedback sketch H, Notably. H, can be simply represented as a bounding box and the corresponding weight, which only needs five numbers, significantly reducing the drawing complexity.
5 Experiments
5.1 Dataset
Visual Question Answering (VQA) v2.0 [36] is dataset that containas open-ended questions about images that require an understanding of vision, language and commonsense knowledge to answer. To fit with our SQA game, we remove all questions that include color-related word since sketch has information about color. This reduces the size of dataset by about 15%. Visual Genome (VG) [49] is dataset that contains 108K images with densely annotated objects, attributes and relationships. Based on VG dataset, we generate the corresponding sketches by the sketching model [43] to train Faster-RCNN model required by ISQA tasks. The evaluation metrics of VG datasets include the mean average precision(mAPP) with IOU theshold 0.5(mAP@ 50) and the mAP@0.5 weighted by the number of boxes for each class(weighted mAP@50).
5.2 Experimental setting
With aforementioned optimazation object I., we trained three different models with SQA module pretrained under VQAv2 dataset and the images are transferred to sketches with [43]:
Pragmatic: trained to minimize L1, equivalent minimize with 0.
2. Geometric: trained to minimize L2, equivalent to minimize L with 1.
3. PraGeo: trained to minmmize T. with = 0.5.
7

---

Accuracy (%6
Accuracy (%)
52
50-
46
44
44
Geumetric
Round1
PragGeo
Round2
38
Pragmatic
Round3
36
30
05
64
61
03
6/4
05
Drawing complexity
Drawing complexity
Figure 3: Comparison of Pragmatic, Geometric, Prageo(left) and Prageo models with different rounds(right). The a-axis is 2 b.
Question:Wine What are these animals?
Image
Roundl sketch Round onswer: Horse?
Feedback sketch (icluded masked Image)
Round2 sketch
Accumulated sketch: Round2 Answer: Giraffes!
Figure 4: Multi-roumt interactive SQA. From left to right we display RGB image, sketch transmitted round 1, RGB image masked with H,, extra pixels added in round 2, and the whole sketches in round
5.3 Quantitative evaluation
Fig compares the ISQA accuracies of Pragmatic, Geometric and PraGeo models function of the drawing complexity B where r-axis is > b, and y-axis is the answer accuracy. For PraGeo, we consider both one-round and multi-round settings.
Comparison among Pragmatic, PraGeo and Geometric models. 1) When is set to be lower than 0.3.N, Pragmatic outperforms other methods in overall accuracy and PraGeo outperforms Geometric in complexity situations. 2) When B is larger than 0.3N, the accuracy of geometric model catches up with other methods'. 3) The peak aceuracy of pragmatic model is lower than other models' since Pragmatic is easier to overfit in low-complexity situations.
Effectiveness of interactive feedback mechanism. 1) Multi-round interaction improve the SQA accuracy. 2) The accuracy gap between one-round PraGeo and multi-round PraGeo are more significant when complexity B (0.05N, 0.3N) since: when the complexity is too low reasoning module is unable to infer sufficient useful information and interaction cost extra bandwidth consumption; when complexity is appropriate, the bandwidth expended this extra interaction relatively negligible and reasoning module generate effective feedback according to the round sketches; when the complexity is too high, most of the crucial information is transmitted in round 1. 3) When comparing two-round interaction to three-round interaction, we see that the difference is minor when compared the gap observed between one-round and two-round interaction.
Human interpretability evaluation. Table 1 compares the human interpretability of Pragmatic, Geometric and PraGeo by calculating CLIP distance. We that without any constraint in training, Pragmatic has poor human interpretability, and ii) PraGeo is much closer Geometric.
Pragmatic
PragGeo
Geometric
Human interpretability Score
0.998 1 0.002 0.600 1 0.068
0.478 1 0.
CLIP Distance
0,9077
0.1815
0.1
Table 1: Average CLIP distances and human evaluation of three models.
To verify the consistency between CLIP and manual measurement, we conduct human evaluation, where 12 participants were tasked with assessing the human-interpretabbin Participants were presented with three corresponding images from each model simultaneously and asked to score them based on judgment criteria ranging from 0 to 1. (lower is better, 0, I represent
of three different models.

---

Figure 5: Sketches under multiple drawing complexities, From left to right, there are IGBB-images, sketch without complexity constraint, and sketch generated with PraGeo model, when B 0.5N, a.3.and0....
fully understand and can not understand, respectively). We sampled 3,000 sets of images and each set comprised images generated by the Pragmatic, PragGeo, and Geometric models from the same RGB images. We see that: 1) the results in human experiments and the CLIP distance are consistent to show the order of Geometric, PragGeo Pragmatic in terms of human interpretability; 2) both PragGeo and Geometrie models provide human interpretable sketches while Pragmatic models can be understood; and 3) Comprehensively considering with Fig.3, the PragGeo achieves more optimal balance among the task performance and human interpretability.
Besides, to further verify the human interpretability, we sampled 300 images and questions pairs from the dataset and test human substituting as the receiver. Ten people are involved this experiment and table. 2 shows the average accuracy and variance of people and machine receiver. We see that human achieve better accuracy in number question but failed outperform machine in Yes-or-No and other question. The experiments show that our architecture can deliver a decent human interpretability and our sender can work with human.
Yes-or-No
Number
Other
Human
66.22 4
39.71
= 4.33
28.57 = 11.30
Machine
70
26
38
Table 2: Results of human substituting as the receiver.
5.4 Qualitative evaluation
Multi-round interaction. Fig. 4 illustrates the procedure of multi-round interaction. We see: i) the receiver generates the feedback sketch which contains crucial region according to the preliminary assessment; ii) the sender provides extra pixels when round is more than 1 according to the feedback sketch; and iii) the receiver modifies the answer accordance with the updated accumulated sketch.
Drawing complexity, Fig. presents the sketches generated by PraGeo model under various drawing complexities. We see that our PraGeo model can preserve human interpretability when the drawing complexity is sufficiently large (B 2 O.1N).
Human interpretability comparison. Fig. 6 shows examples of sketches generated by the three models when = 0.03N. We see that i) the sketches generated by the Pragmatic model are difficult for humans to understand; ii) PraGeo model can still capture the outlines of the bowls and chopsticks and outperform Pragmatic model.
Interactive feedback mechanism. Fig. 7 illustrates that interactive feedback mechanism generates H according to specific questions. The first row shows the RGB image and corresponding sketch, from left to right. When the question is "What are the people in the background doing?". the interactive feedback module generates H that specifically focuses on people to request more information about the area that is likely to contain humans. However, when the question is "What is he on top of?", the interactive feedback module choose to request more information about the area under the man.
Insights brought by ISQA task. First, information disparity is the prerequisite for interaction. Fig. shows the feedback messages for the same image according two different questions. We see that our feedback message transfers question-related information to the sender, enhancing communication efficiency via gradient-informen region of interest (can displayed as sketch as shown in Figure 2, which enables a human-like multi-round interaction for the first time in visual emergent communication. This provides an insight that one the reasons why interaction emerges might be querying task-aligned information when receiver is more acquainted with the object than sender. Second, the constrain of communication complexity promotes interaction. Fig. 3 shows the ISQA performance comparison between multi-round and onc-round communication. We see that the

---

RGB
Pragmatic
RGB
Sketch
What are the people background daing?'
What is he top of?
Geometric
PraGeo
Figure Sketches generated different models when 0.03N.
Figure 7: For the same image, different questions lead to distinct feedback sketches.
downstream performance can be boosted when interaction is introduced to the visual communication pipeline. This provides insight that the complexity constrains promote the agents pursue efficient communication leads to interaction. Third, it requires sufficient input information for the feedback to functional. Fig. 3 shows that functional feedback from the receiver relies on adequate communication complexity. This offers the crucial realization that precise feedback hinges on foundational shared understanding.
5.5
ISQA study
Due to the differences in input modality between SQA and VQA, the RGB-based model is not suitable for SQA. Two key components, Faster-RCNN vision encoder and MCAN question answering module, thus need to be trained with sketch-based dataset. Table 3 evaluates two Faster-RCNN models. We see: the RGB-based model is not applicable to sketch-based dataset. ii) sketch-based detector works normally on sketches but not well as
model on RGB images.
Table 4 evaluates the performance of MCAN models. We that the RGB-based model suffers significant performance degradation on sketch-based dataset. ii) the performance of sketch-based MCAN is acceptable while 10.16 percent lower than RGB-based model in RGB-based datset, which also serves as the upper- bound of ISQA task.
model
dataset
mAP@0.5
Weighted mAP@0.55
dataset
other
yes-or-no
RGB-basee
RGB
0.082
0.149
RGB-based
RGB
64.74
52.44
83.41
47.28
0.010
RGB-basod
Sketch
20.88
23.19
Sketch-based
Sketch
0.034
0.085
Sketch-hasen
Sketch
54.58
40.81
74,81
37.01
Table 3:
Performance of Past-RCNN-based mod- els with ResNet50 backbone on and sketch- VG.
Table Accuracy of RGB-based MCAN and sketch-
VQAv2 dataset. The RGB-based model is trained on color images and
based MCAN on RGB/sketch-bass
The RGB-based model is trained on color images and the Sketch-baas based model is trained on sketches.
the Sketch-based model is trained on sketches.
Conclusion
This study proposes a new interactive sketch guestion-answering task along with an interactive emergent communication (EC) system to facilitate bidirectional communication between two players. Our approach employs an interactive feedback mechanism that allows the EC system to provide feedbacks to guide sketch generation. Our experiments reveal that the proposed multi-round EC system can achieve an effective balance among question answering ability, drawing complexity and human interpretability.
Limitation and future work. The current interactive EC system limited to two collaborative players. In the future, we are going to explore interactive emergent communication among a group of intelligent agents, which more complex and tricky on the design of task and interactive process. Collective intelligence has shown its capacity in applications such as collaborative perception, UAV cluster technology, social network analysis, etc. We believe that multi-agent interaction a promising direction for artificial intelligence research in the future.
10

---

Acknowledgement. This research is supported by NSFC under Grant 62171276 and the Science and Technology Commission of Shanghai Municipal under Grant 2151 100900, 2251 1061001, and 22D72222005

---

References
[1] Rahma Chaabouni, Florian Strub, Florent Altché, Fugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International conference on learning representations.
[2] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02417 2020.
[3] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learnnnnng Advances in neural information processing systems, 29, 2016.
[4] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pincau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pages 693-701, 2019.
[5] Ivana Kajić, Eser Aygün, and Doina Precup. Learning to cooperate: Emergent communication in multi- agent navigation. arXiv preprint arXiv:2004.01097 2020.
[6] Yue Hu, Shaoheng Fang. Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication- efficient collaborative perception via spatial confidence maps. In Advances in Neural Information Process- ing Systems.
[7] Safinah Ali, Nisha Devasia, and Cynthia Breazeal. Designing games for enabling co-creation with social agents, arXiv preprint arXiv:210S. 105.00201, 2021.
[8] Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In 6th International Conference on Learning Representutions, ICLR 2018- C018-Conference Truck Proceedings, 2018
[9] Lukasz Kuciáski, Tomasz Korbak, Pawel Kolodziej. and Piotr Milos. Catalytic role of noise and necessity of inductive biases the emergence compositional communication. Advances in Neural Information Processing Systems, 34:23075-23088, 2021.
[10] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference Learning Representations.
[11] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent popula- tions. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[12] Mycal Tucker, Huao Li, Siddharth Agrawal, Dana Hughes, Katia Sycara, Michael Lewis, and Julie Shah. Emergent discrete communication in semantic spaces. Advances in Neural Information Processing Systems, 34:10574-10586, 2021.
[13] Daniela Mihai and Jonathon Hare. Learning to draw: Emergent communication through sketching. volume 34, pages 7153-7166, 2021.
[14] Shuwen Qiu, Sirui Xie, Lifeng Fan, Tao Gao, Jungseock Joo, Song Chun Zhu, and Yixin Zhu. Emergent graphical conventions in a visual communication game. In Advances in Neural Information Processing Systems.
[15] David Lewis. Convention: A philosophical study. John Wiley & Sons, 2008
[16] Jakob Foerster, loannis Alexandros Assacl, Nando de Freitas, and Shimon Whiteson. Learning commu- nicate with deep multi-agent reinforcement learning. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016
[17] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.00 02419, 2020.
[18] Noga Zaslavsky, Charles Kemp, Teryy Regier, and Naftali Tishby. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31)773777992,, 2018.
[19] Naftali Tishby, Fermando C Pereira, and William Bialek. The information bottleneck method. Proceedings of the 37th Annual Allerton Conference Communication, Control and Computing, 1999.
12

---

[20] Judith E Fan, Robert D Hawkins, Mike Wu, and Noah D Goodman. Pragmatic inference and visual abstraction enable contextual flexibility during visual communication. Computational Brain Behavior, 3:86-101, 2020.
[21] Edward Gihson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. Reger.evy. How efficiency shapes human language. Trends in Cognitive Sciences, 23(5):889-407, 2019.
[22] Charles Kemp, Yang Xu, and Terry Regier. Semantic typology and efficient communication. Annual Review of Linguistics, 4(1):109- 128, 2018.
[23] Myeal Tucker, Roger P Levy, Julie Shah, and Noga Zaslavsky. Trading off utility, informativeness, and complexity emergent communication. In Advances in Neural Information Processing Systems.
[24] Zhewei Huang, Wen Heng, and Shuchang Zhou. Learning to paint with model-based deep reinforeement learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Octoher 2019.
[25] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
[26] Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Device-aware progressive search for pareto-optimal neural architectures. In Proceedings of the Europan Conference on Computer Vision (ECCV), September 2018.
[27] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
[28] Mengye Ren, Ryan Kiros, and Richard 7emel. Exploring models and data for image question answering. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information P Processing Systems, volume 28. Curran Associates, Inc., 2015.
[29] Jiasen Lu, Jianwei Yang. Dhruv Batra, and Devi Parikh. Hierarchical question-imann co-attention for visual question answering. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[30] Hyeonseob Nam, Mao Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
[31] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), June 2016.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Nearal Information Processing Syxtems. volume 30, 2017.
[33] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task- task- agnostic visiolinguistic representations for vision-and-language tasks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[34] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encader representations from transformers, 2019.
[35] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[36] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruy Batra, and Devi Parikh, Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6904-6913 6913,, 2017.
[37] David Ha and Douglas Eck. A neural representation of sketch drawings. International Conference on Learning Representations, 2018
13

---

[38] Jifei Song. Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales. Learning to sketch with shortcut cycle consistency. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[39] Alexander Wang, Mengye Ren, and Richard Zemel. Sketchembednet: Learning novel concepts by imitating drawings. Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 10870-10881.
PMLR, 18-24 Jul 2021.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pages 8748 8763. PMLR, 2021.
[41] Yacl Vinker, Ehsan Pajouheshgar, Jessica Y. Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen- Amir Zamir, and Ariel Shamir. Clipasso: Semantically- aware object sketching. ACM Trans. Graph., 41(4), jul 2022.
[42] Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir. Clipascene: Scene sketching with diffcrent types and levels of abstraction. arXiv preprint a/Xiv:221..1255,, 2022.
[43] Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7915-7925, 2022.
[44] Xudong Mao, Qing Li, Haoran Xic, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smollcy, Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[46] Olaf Ronneherger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer. 2015.
[47] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. Proceedings of the 2014 confereence on empirical methods natural language processing (EMNLP), pages 1532-1543, 2014.
[48] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[49] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-74 -3, 2017.
[50] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pages 618 626, 2017.
14


Reviews:
[{"review_id": "a4ASCtOb9b", "reviewer": "Reviewer_ATWv", "summary": "The authors present a new problem setup for sketch-based emergent communication, distinguishing itself from existing work primarily through communication taking place iteratively over multiple rounds.  The authors also argue that the reliance on downstream tasks for evaluations allows for communication protocols to develop which are not necessarily easily interpretable by humans, and thus fail to fulfill this important goal of EC research.  Evaluating the behavior of various design choices, the authors show that they can prioritize different aspects of the problem: performance, but also human interpretability and drawing complexity (automated metrics)\n\n\n\n\n", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of multi-round EC is very exciting!  I would love to see research move in this direction and, given just the difficulty of agents who learn when to talk (and talk with resonable sparsity), I imagine there are plenty of interesting problems to solve in that space.\n- On that topic, the ability for the sketch model to generate very different sketches using the same image (when the question demands it) is demonstrated here and is a perfect example of what I would expect as sort of a main contribution from an EC model in this space.\n- The problem setup and task are novel", "weaknesses": "- I won't dwell on this too much since it's primarily a track issue, and the paper could be considered by other merits, but it is difficult for me to consider this emergent communication at all.  It does however mean that a lot of EC motivation cited here doesn't seem very relevant upon reaching the experimental design section and understanding the learning problem.\n\n- While the authors argue that existing work relied too much on downstream tasks for evaluation, regardless of this point, evaluating downstream did serve an important purpose in that it helped demonstrate some potentially useful application of the learned protocol.  Here the protocal seems rather contrived.  Of course something like a referential game is also rather contrived and I concede that point.  However, I'm willing to accept contrived environments if what emerges in the language is itself interesting and gives us some insight on what sorts of less contrived environments we may consider in future work.\n\n- The drawer is vastly simplified when compared to the existing visual referential game work (cited here).  The authors state, \"Vision-based EC systems have been developed with the aim of imitating the pictographic systems that were used in early human communication\", but a pictographic system tends to abstract important visual features, sometimes caricaturing them for the purpose of clarity in communication.  I'm not convinced that this drawer is an appropriate substitute for this process.  If I want a giraffe drawn in 3 strokes vs. 8 strokes, we see the important visual features that are most characteristic of the giraffe.  If we are essentially revealing areas of an edge/depth-detected version of a real image, it seems very different.  From the examples of the sketches produced by various modes (pragmatic, geometric, prageo), none strike me as very similar in creating some simplified version of the original high-res image of the object, and a case should be made why this process could be considered an imitation of those systems in early human communication.\n\n- While generating very different images from the same image when the question differs\n\n- Is the binary flag model of [14] really that different from what occurs in this work?  Of course, time not considered, the listener would like to continue receiving new information until the end.  That seems like optimal policy.  So whether the listener conveys to the speaker what it would like to see, or if the speaker already has a priority order in which it would reveal / detail more parts of the image, that's not a hugely important distinction in my mind, unless the speaker and listener have very different perceptual abilites, or goals in mind.  It would of course be good to be user-centric in many cases, but how important is it?  I would have liked to see a comparison.\n\n- In comparison to existing work, and bearing in mind the emphasis on human interpretability in the paper narrative, I would have liked to see this method compete with [13]/[14] with a human substituting as the listener, or at least trying to solve the task (and perhaps no communicating).  Without being able to play with the models directly, the previous work seems more interpretable with fewer strokes.  I really find it surprising that humans aren't involved in the measuring of human interpretability, and I think that fact hints that there may be a more suitable name for what is being measured.\n\nOverall I think there is some promising work in how the task is setup, but deviations from the sketch model and the region-based (rather than complexity-based) of existing work seem like steps backwards.  No direct comparisons to previous work, or adaptations of existing work to this, is detrimental both to placing it in the larger research context, and understanding the relative strengths/weaknesses of the proposed approach.\n\nOther comments:\n\nParagraph 1: These claims seem speculative / opinion-based.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1:  \n\nFirst, our work falls into the topic of emergent communication. Emergent communication aims to facilitate human-like communication between intelligent agents, as delineated in [1,2,3,4]. In this work, we propose a novel question answering setting to promote multi-round, bilateral, reciprocal communication between a pair of a sender and a receiver, which clearly follows the definition of emergent communication. See more details in General Response 1.\n\nSecond, our motivation is to promote multi-round interaction, and our experiments can support this motivation from two aspects:\n\n1) Quantatively, Fig 3 shows the downstream performance as a function of the communicaton complexity. We see that the 2-round communication significantly outperforms the 1-round communication across a vast communication complexity. This validates that multi-round interaction promotes higher communication quality.\n\n2) Qualitatively, Fig. 4 and Fig. 7 illustrates the step-by-step process of the interactive communication. We see that our feedback mechanism enables the sender to pay more attention to the specific areas according to different questions from the receiver, guiding the communication to focus on question-critical area. The experimental findings validate that our multi-round setting simulates human-like multi-round conversation.\n\n---\n\nW2: \n\n- We agree with you that the evaluation of downstream tasks is important, but only relying on downstream tasks would cause non-interactive and non-interpretable issues; see more details in General Response 1. In comparison, the triangular evaluation is a more comprehensive evaluation metric, see more details about the triangular evaluation in General Response 1.\n- We are not sure how the reviewer defines whether a protocol is contrived or not. But our task setting is more natural and better at promoting emergent communcation than previous works, because it can deliver a higher resemblance to human communication in the real world. Human interactions go far beyond simple games, such as classifying images; more often, they involve bilateral, interative communication to achieve a complicated task. From this perspective, our multi-round question-answering task is more closely aligned with daily human interactions than the common Lewis' game in previous works.\n- Our work provides three interesting insights: the prerequisite, the promoter and the condition of interaction in communication. see more details in General Response 2.\n\n---\n\nW3: \n\nWe simplified the drawer for two reasons. \n- First, existing stroke-based non-iterative drawing method cannot produce sketches with sufficient quality to satisfy the basic needs in ISQA. The sketches provided by [13] can only contain a blurry outline of the major object and can not contain enough information to solve question like \"What page is displayed on the monitor?\"; see table 1 in [13]. Only iterative methods such as [40], have yielded visually satisfactory results. However, iterative methods are time-consuming, often requiring several minutes to generate a single sketch.\n- Second, our motivation focuses on multi-round interaction, which is an orthogonal direction to drawing. our work emphaisze multi-round interaction, which establises a mechanism that both sender and receiver should be capable to send informative message to each other and guide the direction of the communication. The proposed multi-round interaction framework is also compatible with stroke-based drawing methods. Once a more mature stroke-based drawing method is developed, we can substitute the drawing module in the current framework and test its communication quality. \nBesides, if we consider the AI agents as an intelligence who lives in a discrete world, it is reasonable to consider dot as AI agents' stroke.\n\n---\n\nW4: \n\nI guess you mean \"why generating\". Intuivetively, Different questions would require different visual information to provide the corresponding answers, leading to diverse sketches. This is actually one important insight we learnt from experiments. For example, in Figure 7, a representative image is subjected to two distinct questions. The first inquiry, \"How many people are visible in the image?\", inherently emphasizes the concept of \"people\". Consequently, regions within the sketch depicting individuals take precedence, as they are more pertinent to the posed question, commanding greater weight in the question-answering module based on the attention network. Conversely, when the question shifts to \"Is it still snowing in the image?\", the focus broadens to a more global perspective. This is attributed to the absence of a salient region garnering specific attention within the question-answering module for this particular question.\n\n---\n\nW5:\n\n- Yes, the feedback mechanism between [14] and this work is significantly different; see more details in General Response 3.\n- If we understand the reviewer correctly, the reviewer is considering a comparison between sender-centric and the receiver-centric communication. However, different from these two settings, our setting considers a bilateral, reciprocal communication. The sender and the receiver have equal rights and collaboratively finish the QA task. The sender and the receiver share the same target while neither of them has fully integrated information to reach the target.\n\n---\n\nW6:\n\n- Our motivation is to promote multi-round, bilateral interaction for emergent communication, instead of creating better drawing merely from the sender perspective; also see W3.\n- We cannot directly compare our work with [13,14], because they are targeting different tasks. Even [13] and [14] have not been directly compared. In this case, a forceful comparison is unfair and cannot provide any useful conclusion.\n- Human experiments: We conduct two human experiments to validate 1) human can be a good receiver (but slightly worse than our model); and 2) CLIP distance and human perception are consistent; see General Response 4."}, {"review_id": "OP0RAuRZ03", "reviewer": "Reviewer_APpT", "summary": "In this paper, the authors proposed a new task about emergent communication by tackling visual question answering as an iterative sketch question answering process. The authors proposed a three-factor evaluation metric, including question answering performance, drawing complexity and human-interpretability. A framework consisting of Sender and Receiver is proposed to perform multi-round interaction to tackle the proposed task. VQAv2 is used for empirical evaluation of the proposed framework. ", "questions": "Please check the weakness for details.", "limitations": "Need more discussion on the fundamental questions of emergent communication.", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "1. The problem setting is very interesting.\n2. The proposed method is intuitive and straightforward.\n3. The paper is presented clearly and easy to follow.", "weaknesses": "1. The new insight is limited.\n\na. Visual question answering is indeed a new task compared to classification. But what is unique in terms of emergent communication when visual question answering is used as the target task? From the current manuscript, there is not really a metric and any empirical evidence showing the improve communication quality over [14]. \n\nb. Despite the authors target at multi-round interaction, the two settings evaluated are one-round and two-round.From the visualizations, the sketch used is usually the sketch of the main object. There doesn't seem to be a pattern in terms of communication with only one/two round of communication.\n\nc. More fundamentally, how does communication emerge and how does communication gets better/more efficient when the task is more complex? The reviewer feels these fundamental questions still left unsolved and the current manuscript didn't show any potential of helping solve these problems.\n\n2. Empirically, current evaluation is not sufficient enough. Currently, the communication quality is mainly measured through automatic metric like CLIP-based score. There should be some quantitative analysis verifying the correlation between automatic score and manual measurement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer APpT\n\nThank you for your time and valuable suggestions. Here are our detailed responses.\n\n---\n\n**W1:**  a) Uniqueness of the proposed task? Comparison to classification. Comparison with [14]. b) Pattern with one/two round communication.  c) Communication gets better when the task is more complex?\n\n**a)** We address this point from three aspects: \n  1) The uniqueness of the proposed visual question answering task for emergent communication are three-folds: information disparity, feedback mechanism and  triangle evaluation; see more details in General Response 1.\n  2) The proposed visual question answering task setting is better at promoting emergent communication than the classification task, because it can deliver a higher resemblance to human communication in the real world. Human interactions go far beyond merely classifying images; more often, they involve bilateral communication to achieve a complicated task. From this perspective, our question-answering task is more closely aligned with daily human interactions than the common Lewis' game in previous works. \n  3) It is hard for us to experimentally compare with [14], because the aims and task settings are different and incomparable; see more details in General Response 3.\n  \n**b)** There is one distinct pattern in 2-round communication: for the same image, each question lead to a unique sketch in the second round, reflecting the interaction successfully helps the sender focus on question-related content. As depicted in Figure 7, it is evident that for disparate questions, the weight assigned to different regions varies considerably, leading to a prioritization of pixel transfer from lighter areas. For question \"“How many people can you see in the picture?\", the sketch transferred in the second round will focus on the people while for question \"Is it still snowing in the picture?\", the sketch transferred in the third round will be presented in a more global fashion. The fact shows that the sketch in the second round depends on the question and definitely will have different patterns when compare with round 1.\n  \n**c)** Through making the task more complex, this work shows a lot of potentials to promote emergent communication. From an intuitive aspect, it is natural for a more complex task to promote emergent communication, because there is no need to create a sophisticated linguistic system if the task for collaboration is very simple. This work provides a recipe for designing a complex task through creating  information disparity, designing feedback mechanism and proposing triangle evaluation. From an experimental aspect, we have found a series of interesting patterns; see more details in General Response 2.\n\n---\n\n**W2:**  Verify the correlation between automatic score and manual measurement.\n\nTo verify the correlation between CLIP and manual measurement, we conduct a human evaluation, where 12 participants were tasked with assessing the human-interpretability of three different models. Participants were presented with three corresponding images from each model simultaneously and asked to score them based on judgment criteria ranging from 1 to 5 (higher score indicating better interpretability); see Table.1 in rebuttal PDF. We sampled 3,000 sets of images consistent with the settings in Table 1 of the paper. Each set comprised images generated by the Pragmatic, PragGeo, and Geometric models from the same RGB images.\n\nTable.2 in rebuttal PDF compares human interpretability score and CLIP distance. We see that: 1) the results in human experiments and the CLIP distance are consistent to show the order of Geometric, PragGeo Pragmatic in terms of human interpretability; 2) both PragGeo and Geometric models provide an average human interpretability score between partially understand and mainly understand while the sketches provided by Pragmatic models can not be understood; and 3) Comprehensively considering with Fig.3, the PragGeo achieves a more optimal balance among the task performance, drawing complexity and human interpretability.\n\n---\n\nOverall, we hope that our responses can fully address your concerns. We will be grateful for your further feedback."}, {"review_id": "0CfA2uR8aq", "reviewer": "Reviewer_jxDd", "summary": "This paper proposed a new multi-round visual communication task with an interactive system for emergent communication. During the game, the sender needs to sketch on the canvas to communicate a target image, while the receiver needs to answer a question regarding the target image and give feedback on the sender’s sketch. The training framework balances task accuracy, drawing complexity, and human interpretability. Experimental results show that the agents can communicate successfully using sketches and feedback. The emerged sketches can maintain interpretability and high accuracy with lower complexity. And the feedback given by the receiver can effectively enhance the performance. \n", "questions": "Why is the $\\sum b_\\tau$ given to the sketch model? Would $b_i$ be sufficient?\n", "limitations": "It will be interesting if the agents can control the complexity of the sketch based on the target image and the receiver’s feedback. Similarly for the number of bounding boxes at the receiver’s side.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a novel setting where each of the agents can only observe a partial environment that necessitates the feedback of the receiver. And the feedback is smartly provided in a sender-understandable way (bounding boxes). Compared with the previous work, this environment enables bi-directional communication where both agents can “draw” on the canvas.\n2. The training framework considers triangle optimization – task accuracy, drawing complexity, and human interpretability. \n", "weaknesses": "1. Complexity B: \nfor the complexity in section 5.3, what is the specification for $b_i$ and $h_i$ separately? \nIt will be interesting to know how which agents contribute more to the efficiency – while achieving high accuracy, is the high efficiency due to the sender drawing less or the receiver giving more accurate feedback?\n2. The maximum round: only models trained with two-round are reported. Is there a reason that the maximum round is set to 2? Given more rounds, the performance change can help us understand whether one round of feedback from the receiver is enough to finish the task. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your time and valuable suggestions. Here are our detailed responses.\n\n---\n\n**W1:** Which agents contribute more.\n\nOur setting considers a collaboration between the sender and the receiver. Through the proposed feedback mechanism, both agents have equal rights to affect the interaction and either agent cannot be separately considered. The reason why sender can use less complexity to transmit sufficient information is because the receiver gives it an accurate feedback while the accurate feedback constructs on the decent round1 communication. To illustrate, this can be likened to a teacher identifying key contents for students prior to the final exam. By concentrating efforts within key contents, students are able to achieve commendable results with less time. Meanwhile, the reason why the teacher accurately point out these key contents is because the teacher communicate sufficiently with the students, thereby gaining understanding into students strength and weakness of the curriculum.\n\nOverall, the both components collaborate to deliver a superior performance.\n\n---\n\n**W2:** Maximum round.\n\n 2-round interaction is enough to finish the task. The 3-round model provides slightly lower performance compared to 2-round model. There are two reasons why 3-round can not improve performance: First, a 3-round approach will reduce the complexity budget assigned to the first and will reduce the accuracy of the feedback module(See line 304-306 for reasons). Second, each round of interaction requires an extra complexity for the feedback message, causing the complexity assigned to drawing decreases.\n\n---\n\n**Limitation:**\n Thanks for your advice! We will explore how to impel the sender and receiver to control complexity in future work.\n\n---\n\nOverall, we hope that our responses can fully address your concerns. We will be grateful for your further feedback."}, {"review_id": "U74HW1E6Rg", "reviewer": "Reviewer_NLdK", "summary": "The work focuses on multi-turn sketch-based emergent communication.\nAuthors propose a novel two-round interactive task, named Interactive Sketch Question Answering (ISQA).\nThey suggest an architecture and an implementation, based mainly on existing components (MCAN, Fast-RCNN) while incorporating several novel ideas such as 1) dynamically restricting the channel capacity by controlling the number of transmitted pixels, and 2) providing feedback from receiver to sender via focus boxes.\nThey suggest a triangular evaluation method that seeks a balance between human interoperability and task accuracy.", "questions": "In section 3.1 what are the dimensions of H_i and A_i? (explained later)\nLine 150 - will be good to stress the fact that b_i is a ratio (explained later)\nLine 244 – will be good to explain what proposals are.\nLines 293-297 the x-axis is not easily defined (line 293) and then referred to as 0.1N, 0.3N, which are hard to find in the graphs. Can’t you use the 0.xN scale? or at least add the values you refer to as labels to the x-axis?\nDatasets are missing the explanation of complexity/difficulty of tasks, namely yes/no, number and other which you refer to in the figure. A baseline random accuracy can also be helpful to add or mention.\nTable-1: is the lower the better? Worth mentioning.", "limitations": "limitation section is provided.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of this paper is by suggesting a two-turn visual communication game that nicely models the need for two parties to communicate, with partial observability, to solve a task.\nIn addition, the paper demonstrates a method to achieve a nice balance between interoperability and pragmatism.\nThe most interesting observation, to my mind, is provided in lines 304-306 where the authors show that when the complexity is too low, the reasoning module cannot infer sufficient useful information in a single round and thus needs to request a more focused information (a clarification question).\nThe way the architecture is composed and implemented for modeling the problem at hand is not trivial and interesting. \n", "weaknesses": "The authors assess human interpretability using the CLIP model. Doing an actual human survey of the results would be more appropriate.\n\nThe interoperability/pragmatism balance is essentially solved by adding a CLIP-based loss that provides additional supervision towards human interpretability, which is not aligned with the intention to model communication emergence.\n\nExperimental datasets are not described in enough detail. For example, it is unclear how the three reported tasks (Yes-No, Number, Other) correspond to the two described datasets.\n\nResults are not totally consistent (for example, in the Yes-No task where the PraGeo is lower than both the geometric and pragmatic models) and more experiments over more datasets seems needed.\n\nNotations and explanations can be further worked out to assist the reader. See some examples in the Questions section.\n\nMaybe worth mentioning references:\nPragmatic inference and visual abstraction enable contextual flexibility during visual communication, by Judith E. Fana, Robert X.D. Hawkins, Mike Wub and Noah D. Goodman,\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable suggestions. Here are our detailed responses.\n\n---\n\n**W1: Human survey**\n\nWe agree with your opinion and added a human survey. The human evaluation results are consistent with the CLIP distance. See general response 3.\n\n---\n\n**W2:  CLIP-based loss provides additional supervision.**\n\nWe think leverage CLIP model is reasonable for four reasons.\n- The primary rationale behind striving for human interpretability is to achieve a consensus with humans on various concepts. This ensures that the emergent communication resonates with human understanding and can be intuitively comprehended.\n- We view the CLIP model as an exceptional tool for learning human consensus and perceptual abilities. It serves as a teacher for our systems to better align with human cognitive results.\n- Previous methods, such as [13], have also leveraged VGG classification models during training. This precedent signifies the utility and relevance of employing such models to drive desired outcomes.\n- While the idea of entirely free emergence is indeed exhilarating, we recognize that there's a gap between this ideal and current research paradigms. Our approach is an endeavor to traverse this gap, balancing the excitement of pure emergence with the practicality of human-aligned understandings.\nWe believe that, through these methods, we maintain the spirit of modeling communication emergence while simultaneously ensuring that the results align with human interpretability.\"\n\n---\n\n**W3 & Q5: Details about experimental datasets.**\n\nYes-or-No, Number and Other are different question types in the dataset(according to the answer). \n  - Yes-or-No means the answer to the question is yes or no. This type of question is easier to solve and mainly focuses on a single object or pattern in the image.\n  - Number means the answer is Arabic numerals and often requires detailed information. \n  - Other includes remaining questions, whose answers include more than 3000 alternative words or phrases. \n  - We will add the related explainations to the updated paper.\n\n---\n\n**W4: Result consistency.**\n\nWe confirm that the results are consistent and reliable. The observation mentioned by the reviewer is commonly known as Simpson's Paradox, where overall statistical results are inconsistent with the local statistical results. Yes-or-No questions often require less information and have a higher possibility to overfit and strong randomness since it has only two options. \n\n---\n\n**W5: Further notations and explanations.**\n\nThanks for your advice! We will correct those issues in the revised version.\n\n---\n\n**W6: References.**\n\nThanks for your advice! We will add those references and do a more comprehensive literature review in the recognitive community.\n\n---\n\n**Q1**: \n\nThe dimension of $H_i$ is $m\\times 4$ and the dimension of $A_i$ is $n\\times 1$. Where $m$ denotes the number of regions and $n$ denotes the number of alternative answers. 4 means $(x, y, height, width)$ to locate a box in the sketch.\n\n---\n\n**Q2**: \n\nThanks for your advice, we will modify our updated paper.\n\n---\n\n**Q3**: \n\nThe \"proposals\" are thus a critical part of the Faster R-CNN architecture, as they define the potential regions where objects might be present. Here's how proposals are generated in more detail:\n  a. Anchors: At each position of the image, the RPN proposes multiple regions of different scales and aspect ratios. These are called anchors. For example, you might use 3 scales and 3 aspect ratios, for a total of 9 anchors at each position.\n  b. Classification and Regression: For each of these anchors, the RPN predicts two things:\n    - Whether it contains an object or not (classification).\n    - Adjustments to the anchor box to better fit the object (regression). These adjustments are typically four real-valued numbers for the coordinates of the top-left and bottom-right corners of the box.\n  c. Proposal Selection: From all the proposed regions, a set number (e.g., 2000) of the best scoring regions are selected as region proposals using a technique called Non-Maximum Suppression (NMS) which helps in eliminating multiple detections for the same object.\n\n---\n\n**Q4**: \n\nSorry for inconvenience, we will modify the x-axis in the updated version.\n\n---\n\n**Q5**: \n\nPlease see W3.\n\n---\n\n**Q6**: \n\nThanks for your advice! The lower is the better for the CLIP distance. Besides we will add human survey. See general response 4.\n\n---\n\nOverall, we hope that our responses can fully address your concerns. We will be grateful for your further feedback."}]

Author rebuttal:
**General Response 1: Uniqueness of the proposed ISQA task for emergent communication:**

We propose a novel task setting, whose goal is to promote multi-round, bilateral, interactive communication between a pair of a sender and a receiver. To achieve this goal, our proposal has three unique characteristics: information disparity,  triangle evaluation, and feedback mechanism.
- Information disparity. Our setting simulates the question answering between two people. Without communication, neither the sender nor the receiver can accomplish this task alone. Specifically, the sender has the source information (image), but does not know the target information (question); and the receiver has the target information (question), but does not know the source information (image). Therefore, such an information disparity creates a necessity to communicate. Note that most previous works [1, 13, 14] do not create such an information disparity.
- Feedback mechanism. We promote bilateral communication by allowing the receiver to provide feedback to the sender. This simulates multi-round conversation between two people. With the feedback mechanism, both sender and receiver are equal rights to send informative message to each other and guide the direction of the communication. Note that most previous works predominantly concentrate on single-round, sender-receiver architecture. Therefore, we extend emergent communication into a multi-round setting, seeking insights into the impact of the feedback mechanism for the emergent communication process.
- Triangle evaluation. We ensure high-quality communication by designing three criterion. Given the criteria of the ability to handle downstream tasks, the communication content between the sender and the receiver has to be informative. Given the criteria of communication complexity, the communication content has to be compact. Given the criteria of human interpretability, the communication content has to be intuitive. These three factors together promote high-quality communication. 

---

**General Response 2: Insights brought by the proposed ISQA task:**

This work provides three insights about how interaction emerges in communication. 
  - First, information disparity is the prerequisite for interaction. Fig. 7 shows the feedback messages for the same image according two different questions. We see that our feedback message transfers question-related information to the sender, enhancing communication efficiency via a gradient-informed region of interest (can be displayed as sketch as shown in Figure 2), which enables a human-like multi-round interaction for the first time in visual emergent communication. This provides an insight that one of the reasons why interaction emerges might be querying task-aligned information when receiver is more acquainted with the object than sender. 
  - Second, the constrain of communication complexity promotes interaction.  Fig.3 shows the ISQA performance comparison between 2round and 1round communication. We see that the downstream performance can be boosted when interaction is introduced to the visual communication pipeline. This provides a insight that the complexity constrains promote the agents to pursue a more efficient communication leads to interaction. 
  - Third, it requires sufficient input information for the feedback to be functional. Fig 3 shows that functional feedback from the receiver relies on adequate communication complexity. This offers the crucial realization that precise feedback hinges on a foundational shared understanding.

---

**General Response 3: Comparison with [14]:**

- Methodologically, the way to design the feedback mechanism is significantly different. The feedback in [14] is a binary signal (yes or no), indicating the receiver's confidence level in rendering a decision. In comparison, the feedback in our work is a region of interests, pointing out which parts of the sketch should get higher attention in the next round.
- Quantatively, the communication quality between [14] and our method can not be directly compared, because the aims and task settings are different and incomparable. [14] focuses on the generation and evolution of pictography and the task it leverages is traditional Lewis' game to create symbols for each class. It is not designed to solve our visual question answering task and cannot execute the visual question answering task. In turns, our work cannot execute Lewis' game as well. So far, there has been no precedents to compare results across different tasks in emergent communication.

---

**General Response 4: Human Experiments to verify CLIP measurement:**

To verify the correlation between CLIP and manual measurement, we conduct a human evaluation, where 12 participants were tasked with assessing the human-interpretability of three different models. Participants were presented with three corresponding images from each model simultaneously and asked to score them based on judgment criteria ranging from 1 to 5 (lower number indicating better interpretability); see Table.1 in rebuttal PDF. We sampled 3,000 sets of images consistent with the settings in Table 1 of the paper. Each set comprised images generated by the Pragmatic, PragGeo, and Geometric models from the same RGB images.

Table.2 in rebuttal PDF compares human interpretability score and CLIP distance. We see that: 1) the results in human experiments and the CLIP distance are consistent to show the order of Geometric, PragGeo Pragmatic in terms of human interpretability; 2) both PragGeo and Geometric models provide an average human interpretability score between partially understand and mainly understand while the sketches provided by Pragmatic models can not be understood; and 3) Comprehensively considering with Fig.3, the PragGeo achieves a more optimal balance among the task performance, drawing complexity and human interpretability.


Meta-review: The paper introduces an innovative approach to multi-round sketch-based emergent communication through the Interactive Sketch Question Answering (ISQA) task. The proposed task explores a novel direction in the field and has garnered interest from the reviewers due to its unique setup and potential contributions.

The concept of multi-round communication, as demonstrated by the ISQA task, presents an exciting avenue for emergent communication research. The reviewers' expertise in the field is evident, and their feedback is based on a solid understanding of the topic. The reviewers' assessments collectively highlight both the strengths and potential limitations of the paper.

Strengths identified by the reviewers include the task's novelty, the integration of sender-understandable feedback (bounding boxes), and the introduction of a triangular evaluation method that takes into account task accuracy, drawing complexity, and human interpretability. The presentation of the paper is generally clear and effective in conveying the problem and proposed solution.

While the paper's contributions are acknowledged, some concerns and suggestions have been raised. Reviewer feedback highlights the need for additional quantitative analysis to establish correlations between automatic and manual measures of communication quality, and the potential to provide more comprehensive comparisons to previous work. Some reviewers also express curiosity about the insights gained from the proposed approach.

Addressing the concerns raised by the reviewers and expanding the analysis could further strengthen the paper's contributions and impact. The novelty of the ISQA task, the balance between interoperability and pragmatism, and the potential to enhance multi-round communication are all promising aspects of this work.

In conclusion, the paper's innovative approach to multi-round emergent communication, coupled with the reviewers' positive perceptions of its potential contributions, suggests that it should be accepted. Revisions guided by the reviewers' feedback will be valuable to ensure that the paper's insights and contributions are well-founded and effectively communicated.
Decision: Accept (poster)