Emergent Graphical Conventions in a Visual Communication Game
Authors: Shuwen Qiu
Sirui Xie
Lifeng Fan
Tao Gao
Jungseock Joo
Song-Chun Zhu
Yixin Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Visual communication
Emergent languages
Representation learning

Abstract: Humans communicate with graphical sketches apart from symbolic languages. Primarily focusing on the latter, recent studies of emergent communication overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity. In this work, we take the very first step to model and simulate this process via two neural agents playing a visual communication game; the sender communicates with the receiver by sketching on a canvas. We devise a novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions. To inspect the emerged conventions, we define three key properties -- iconicity, symbolicity, and semanticity -- and design evaluation methods accordingly. Our experimental results under different controls are consistent with the observation in studies of human graphical conventions. Of note, we find that evolved sketches can preserve the continuum of semantics under proper environmental pressures. More interestingly, co-evolved agents can switch between conventionalized and iconic communication based on their familiarity with referents. We hope the present research can pave the path for studying emergent communication with the modality of sketches.

Emergent Graphical Conventions in a Visual Communication Game
Shuwen Qiu Sirui Xie Lifeng Fan?,
Tao Gao3.1 3,1, Jungseock Joo", Song- Chan Zhu 1,2,1,5 Yixin Zhu
Department of Computer Science, UCLA
Beijing Institute for General Artificial Intelligence (BIGAI) Department of Communication, UCLA Department Statistics, UCLA Institute for Artificial Intelligence, Peking University
https://sites: htt.://ww.............a.a.aaaaa.
Abstract
Humans communicate with graphical sketches apart from symbolic languages (Fay et al., 2014). Primarily focusing on the latter, recent studies of emergent communi- cation (Lazaridou and Baroni, 2020) overlook the sketches; they do not account for the evolution process through which symbolic sign systems emerge in the trade-off between iconicity and symbolicity. In this work, we take the very first step model and simulate this process via two neural agents playing a visual communica- tion game; the sender communicates with the receiver by sketching on canvas. We devise novel reinforcement learning method such that agents are evolved jointly towards successful communication and abstract graphical conventions. spect the emerged conventions, define three fundamental properties-iconicityy symbolicity, and semanticity-and design evaluation methods accordingly. Our experimental results under different controls are consistent with the observation in studics human graphical conventions (Hawkins et al., 2019; Fay et al., 2010). Of note, find that evolved sketches preserve the continuum of semantics (Mikolov et al., 2013) under proper environmental pressures. More interestingly, co-evolved agents switch between conventionalized and iconic communication based on their familiarity with referents. We hope the present research can pave the path for studying emergent communication with the modality of sketches.
Introduction
Communication problem naturally arises when traveling foreign country where you do speak the native language, which necessitates exploring non-linguistic means of communication, such as drawings. Due to iconic nature (i.e., perceptual resemblance to natural association with the referent), drawings serve powerful tool to communicate concepts transcending language barriers (Fay al., 2014). In fact, we humans started to use drawings convey messages dating back to 40,000060,000 years ago (Hoffmann et al., 2018; Hawkins et al., 2019). Some cognitive science studies hypothesize a transition from sketch-based communication before the formation of sign systems and provide evidence that iconic signs gradually become symbolic through repeated communication (Fay et al., 2014). In contrast to icons, symbols are special forms bearing arbitrary relations to the referents. Fig. 1 describes typical scenario of such phenomena: Alice (in green) uses sketch to communicate the concept "rooster" to Bob (in yellow). Initially, they need to ground the sketch the referent. Later, details of the visual concept, such strokes of the head and body, are gradually abstracted away, leaving only the most salient part, the crown. The iconicity in the communicated sketch drops while the symbolicity rises.
indicates equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Round
Boun T
Rond FTT
WWN
Figure I: An example the visual communication game, In an iterative sketch communication game (Hawkins el al., 2019), players first need to ground sketches to referents. The drawer (Alice) gradually simplifies the drawing but keeps the most salient parts of the target concept (rooster crown). This evolution process enables the viewer (Bob) promptly distinguish the target (rooster) from distractors (bird, cup, rabbit, and sheep).
While models of emerging communication protocols has attracted attention (Lazaridou et al., 2017; Cao et al., 201 8; Evtimova et al., 2018; Havrylov and Titov, 2017; Lazaridou et al., 2018; Lazaridou and Baroni, 2020; Mordatch and Abbeel, 2018; Ren et al., 2020; Eecles et al., 2019; Graesser et al., 2019), the initial and primary communication medium is presumed and limited to be symbolic rather than iconic. By simulating multi-agent referential game, prior work seeks for the environmental driving forces behind the emergence of effective communications. games, two agents play similar roles as in the above Alice-Bob example but share primitive set of arbitrary tokens (i.e., the vocabulary). Using these tokens, an agent (the sender) attempts to communicate message another agent (the receiver). A communication convention emerges when two agents successfully communicate by associating visual concepts in the images with tokens in the pre-selected vocabulary. Though this line of work has probed into some linguistic properties of the established communication conventions (Lazaridou et al., 2017; 2018; Ren et al., 2020), some intriguing questions remain open: How do agents make trade-off between iconicity and symbolicity to emerge symbolic sign systems?
a typical setup of referential
In this work, we present the very first step of modeling and simulating the evolution process of graphical conventions (Hawkins et al., 2019), a two-participant communication convention whose medium is drawings in an abstract form. Specifically, we consider our contributions in three folds:
First, we model a multi-agent visual communication game and propose a learning framework, wherein the sender and the receiver evolve jointly. This visual communication game is an alternating sequential Decision-making process, in which the sender generates a sequence of strokes step by step, terminated by the receiver. In contrast to discretized tokens in prior work, strokes can be continuously parametrized (Ha and Eck, 2018; Huang et al., 2019) such that the derivatives of learning objectives can be more effectively back- propagated through communication channels (Foerster et al., 2016). We further derive a novel training surrogate for multi-agent reinforcement learning based on a joint value function and the eligibility trace method (Sutton and Barto, 2018). This differs from the REINFORCE- surrogate in prior works on early-terminable sequential communication (Cao al., 2018; Evtimova et al., 2018) and the TD learning method in the literature of multi-agent turn-taking games, wherein each agent has its own value function and has to model the value of others (Wen et al., 2019). In experiments, approximation and Monte Carlo sampling facilitates the agents' awareness of the correlation between complex and simple sketches, thereby enabling smooth abstraction process.
empirically demonstrate that our integration of function
Second, we define essential properties in studying evolved sketches Specifically, we define iconicity (Fay et al., 2014) as the drawings exhibiting high visual resemblance to the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of general- purpose visual system; we define symbolicity (Fay et al., 2018) as these drawings being consistently separable in the high-level visual embedding, which facilitates new communication participants to casily distinguish between categories without grounding them to referents; and we define semanticity (Harispe et al., 2015) as the topography of the high-level embedding space of the drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space. Of note, this is not the only way to define these cognitive concepts; we intend to align readers on critical concepts in our work.
Third, we present a suite of quantitative and qualitative methods to evaluate the emergent graphical conventions based on the above carefully defined iconicity, symbolicity, and semanticily. This is necessary because a high communication rate does not imply good representations (Bouchacourt and Baroni, 2018). The graphical nature of the communication medium mandates us repurpose

---

representation learning metrics rather than adopt linguistic metrics in emergent symbolic commu- nication. We evaluate the contribution of different environmental drivers, early decision, sender's update, and sequential communication, to the three properties of the emergent conventions. Critically, the empirical results assessed on our metries align well with our prediction based on the findings of human graphical conventions (Fay et al., 2010; Hawkins et al., 2019), justifying our environment, model, and evaluation. One of these setups emerges conventions where the three properties are consistent with our expectation of sign system. Particularly, we find two inspiring phenomena: (i) Evolved sketches from semantically similar classes are perceptually more similar to each other than those falling into different superclasses. (ii) To communicate concepts not included in their established conventions, evolved agents can return to more iconic communications than humans. We hope our work sheds light on emergent communication in the modality of sketches and facilitates the study of cognitive evolutionary theories of pictographic sign systems.
2 Related work
Learning to sketch Ha and Eck (2018) begin the endeavor of teaching modern neural models to sketch stroke by stroke. However, generating meaningful stroke sequences directly from various categories of natural images is still in the early phase (Song et al., 2018; Wang et al., 2021; Zou et al., 2018). To assure the interestingness of the category-leve sketch communication, we design a stage- wise agent that transfers a natural image into a pixel-level sketch (Kampelmuhler and Pinz, 2020) and draws the sketch on the canvas stroke by stroke with a policy (Huang et al., 2019). To pre-train our neural agents to sketch, we select Sketchy (Sangkloy et al., 2016) from many datasets (Yu et al., 2016; Ha and Eck, 2018; Eitz et al., 2012; Sangkloy et al., 2016) for its fine- grained photo-sketch correspondence, rich stroke-level annotations, and well-organized categorization structures.
Communication games While learning to sketch is formulated as single-agent task with explicit supervision, our focus is on how sketches would evolve when utilized as the communication medium between two cooperative agents. Their cooperation always formulated game, recently adopted to study phenomena in human-robot teaming (Yuan et al., 2022) and natural languages, such as symbolic language acquisition (Graesser et al., 2019) and the emergence of compositionality (Ren et al., 2020). Some notable works (Lazaridou et al., 2017: 2018; Evtimova et al., 2018; Havrylov and Titov, 2017) have devised interesting metrics, such as purity (Lazaridou al., 2017) and topographic similarity (Lazaridou et al., 20.8). In comparison, our work is unique due to the distinctive communication medium, continuously parametrized sketches. Although concurrent work (Mihai and Hare, 2021) also enables the agents to sketch in a communication game, it focuses primarily on drawing interpretable sketches without abstracting them into graphical symbols along the communication. We position our work as an alternative to emergent symbolic communication, since the emergent graphical symbols may better represent the continuum of semantics, as encoded in the vector representation of tokens (Mikolov et al., 2013). Methodologically, we devise new evaluation metrics for sketch modality, assessing iconicity, symbolicity, and semanticity in the evolved sketches.
communication
Emergent graphical conventions Evolving from sketches to graphical conventions/symbols is an active field cognitive science under the banner of "emergent sign systems." Fay et al. (2010) show that pair interaction can form their local conventions when they play the Pictionary game. Using natural images instead of texts as the prompt, Hawkins al. (2019) show that visual properties of the images also influence the formed graphical conventions besides partners' shared interaction history; i.e. the evolved sketches highlight visually salient parts. Nevertheless, only a few computational models exist apart from these human behavior studies. Fan et al. (2020) describe model for selecting complex or simple sketches considering the communication context. Bhunia et al. (2020) and Muhammad et al. (2018) consider stroke selection and reordering to simplify the sketch. In contrast to sketch or stroke selection, we model embodied agents who can draw and recognize sketches.
3
The visual communication game
Our visual communication game is formulated tuple
As, G,r,7), where I is the image set to be presented to the sender and the receiver R. These images contain single object in the foreground, and hence the image space I can be partitioned into N classes
3

---

0,80
0s1
0,52
Sender
ANIO
RI
R2
Heceiver
Figure 2: Communication process. In visual communication game, sender S and receiver R only share the observation of the canvas C. The sender converts the natural image Is pixel-level sketch is At each step. the sender first draws five strokes as through the renderer G. which updates the canvas C++1. Next, the receiver uses the updated canvas C++1 query from the context images (/2,... ..., T and the last canvas C.. deciding the action on at this step. The game continues the receiver chooses to wait.
according the object categories. In each game round, the sender is presented with one image Is- and the receiver is presented with M images { .... A }. Prior work shows that category-leve context can force the agent coordinate on more abstract properties and help draw sketches to be more recognizable as the type of object (Lazaridou et al., 2017; Mihai and Hare, 2021). We make the observations of S and R disjoint (i.e., & (7 -.. TM but with target image T7 in the same class as Is. We refer to the M images that the receiver can see as the context. Neither the receiver or the sender would see the image(s) presented information by drawing sketches on the canvas C, observable to both players. As shown in Fig. 2, Co initialized to be blank at the beginning of each round. Only the sender can draw the canvas with actions chosen from As. The action at each time step consists of 5 strokes, which are continuous vectors in R. We constrain each dimension to be in (0, due the limited space of the canvas. The canvas is updated from C, to C+++1 by the renderer G after each step of the sender's 's sketching, The receiver, after observing the updated canvas, would choose among the M images or wait for the next step from the sender; these M + 1 possible actions constitute An- A game round terminates when the receiver gives up waiting and chooses one from the images. After the termination, the sender and the receiver will receive shared reward penalty, depending on if the receiver makes the right choice:
their partner; they can only communicate this
This reward/penalty is temporally decayed with a decay factor 7. That is, if the receiver decides to choose from the images at step I, this cooperating pair will receive either > or -7' Hence, even though the players do not receive explicit penalty for long conversations, there is implicit penalty/reward for delayed positive/negative answers. No reward will be assigned if the receiver chooses wait. The next round starts after the reward/penalty assignment.
4 Agents
The agents involved in the visual communication game are modeled with two decision policies, Ts and TR. for the sender and the receiver, respectively. These policies are stochastic mapping from the agents' observation space to the action space:
I C. P(As), TR I XC - P(AR).
(I)
where ((A) is distribution over the support set A. As shown in Fig. 2, at each time step t E {O....} the sender first emits the stroke parameters for the next five strokes ast ~u Is Ct). These strokes are applied to the canvas by differentiable renderer, C++1 = G(C,,
Next, the updated canvas transmitted to the receiver. The receiver decides whether to terminate the game by making its
prediction (i.e., ant E {1,..., M)) or wait for another round (i.e., an M 1); its decision is sampled from πR. If prediction made, is used to select the image l from (7) (7).... MM and
score this game with r r(ls, R Otherwise, this routine repeats in the next step & +- t 1.

---

4.1 Sender
Prior to playing the visual communication game, the sender should be able (i) extract edges from natural images (Xie and Tu, 2015) and (ii) draw sketches that closely resemble the configurations of salient edges (Li al.. 2019), just as humans do (Sayim and Cavanagh, 2011 1). To endow the sender with these capabilities, we design stage-wise architecture hş = gs fs. Specifically, Is is first converted to target sketch is using visual module fs (Kampelmuhler and Pinz, 2020), capturing the salient edge information the natural image; we directly adopt the pre-trained model from the referred work. Next, Is is concatenated with the current canvas C₁ and fed to the sketching module 9s, whose architecture is built upon Huang et al. (2019). This sketching module outputs five vectors the form (Tn, Yo. 21, 91: 22, 3/2), which parametrizes the curve of one stroke. The policy is parametrized Gaussian distribution during training,
TS = (Np, o²), jr = hs(Ts, CI), o² = c.
(2)
where I is the identity matrix, and c is a constant hyperparameter. During the testing, we set c c= 0.
These stroke parameters are fed into a pre-trained renderer G'' (Huang et al., 2019) to update the canvas, C++1 G(C,, ast). This renderer is fully differentiable, enabling end-to-end model-based training (Hafner et al., 2019) of the sketching module Js. We pre-train 9s on Sketchy (Sangkloy et al., 2016); please refer to the supplementary video for results.
4.2 Receiver
The receiver, similar to the sender, should also carry some rudimentary visual capability to this game. Unlike the low-level vision needed for the sender, the requirement for the receiver high-level visual recognition. Therefore, we adopt pre-trained VGG16 (Simonyan and Zisserman, 2015) as the visual module fR= : + 4006 of the receiver, following a similar practice in recent literature (Lazaridou et al., 2017; Havrylov and Titov, 2017). The output of this visual module is vector, and further transformed by two separate linear layers, IR and Q into visual embeddings, 1 (T) and (T).
That is, h K o fR. h = fR-
When observing both the context [I....,, MM } and the canvas Cr the receiver first embeds each of them with hn. Next, it makes the decision based on the similarity between the current canvas and each option in the context. The decision module is thus realized by Boltzmann distribution based on resemblance:
exp(h2(Ca) (1(")))))
R C.11.
(3)
M+1
exp(h? (IR))
where 1 MM++ C2-1. Of note, although a similar policy was proposed before (Lazaridou et al., 2018; Havrylov and Titov, 2017), our RR is distinct as it is endowed with an external memory of C,-1- Intuitively, if the receiver finds the current canvas C2 closer to the last canvas Ct-1 in the embedding space than all M options in the context, it will choose to emit amu M + and delay the decision to the next step; a prediction can only be made when the receiver finds the current canvas is informative enough. As result, the sender would draw informative strokes as early possible to avoid the implicit penalty in the decayed reward.
4.3 Learning to communicate
Policies of the sender and the receiver are trained jointly to maximize the objective:
Σ E
S R
arg max (T~(TS. .T))
(4)
#S.JR
where 7 {Co, aso. Ci, RRI asi, ...) the simulated episodic trajectory. As well known in rein- forcement learning, the analytical expectation in Eq. (4) is intractable to calculate along the trajectory
T, We devise value functions VXX,) and Vx(X,) for an optimization surrogate:
(Xx) R Cr..as.lls..C- x.) ((ra + (xx
(5)
5

---

where 5((R)) is the Dirac delta function that returns when the action is wair and 0 otherwise. X; Xa)( where X, = [ .... C,-1, The expectation (as(ase Ct, is ap- proximated with the point estimate, as in the reparametrization VAE (Kingma and Welling, 2014). The expectation (x (8,)
[+]] can be calculated analytically because Tuu is categorical distribu-
tion. The connection between Erg and En is one of our contributions. Specifically, C₁ in X, in The (a((a (X,) is generated by the differentiable renderer G with the actions as from the sender policy 0V/0T Ts passs Is. Ct-1). Hence, we can have both 0v/ / Re and 0v based on Eq. (5). This results novel multi-agent variant of the general policy gradient (Sutton et al., 2000; Silver et al., 2014).
(X,,) in Eq. (5) is an eligibility trace approximation (Sutton and Barto, 2018) of the ground-truth value function. Intuitively, value estimate with eligibility trace V1 mixes the bootstrapped Monte at different roll-out lengths k. with h = min(t + k, (hoice) being the maximal timestep. Tchosce is the timestamp when the receiver stops waiting. The articulation of such termination also makes our cligibility trace deviate from the general derivation with infinite horizon. We derive an episodic version as
Carlo estimate V (Xt) Exs.an [E== y "m""rn +
H-1
(1 1) Σ VX(X) + (Xi)
if S T Tchoice
(X,)))
(6)
(X,)
otherwise.
where H = T Tchoice 1 + 1. Please refer to Appendix C for detailed derivation. Finally, us (Xt) is trained by regressing the value returns:
Φ = arg max E BTy
Σ
1 (lvs(X)) (x,111
(7)
The training algorithm is summarized in Alg. 1.
Experiments
Algorithm Training Algorithm
Initialization Initialize neural network purameters 0,
P. @ for TT. wn, and de. respectively. 1 for game round = 1, ..., 1. do
5.1 Settings
2
for time step t = 0, ..., T do
Images We used the Sketchy dataset
age ~ ny (as
(Sangkloy et al., 2016) as the image source.
C++1 = G(Cr,
Due to the limited performance of the sketch-
aR ~ n (axx(a |C2 Ccp 1, I ..., IN
ing module on open-domain image-to-sken sequential generation, we select 40 categories (10 images per category, see Appendix A) with satisfactory sketching behaviors.
R+++1 not wait then
Tenica t
end end
10
Compute (Vx(X.)) via Eq. (6)
Environmental drivers With the visual
Compute (V(X))) via Eq. (5)
communication game and the learning agents hand, we investigate the factors in emer-
Update 9 < 0 V. Σ V(X.))
Update p p V, E. V(Xe)
13
gent graphical conventions with controlled experiments. Tab. lists all designed settings. Specifically, we consider the following:
Update
Φ + -- a. V. (Xi) - VX(X)11
end
Table 1: Game settings and results. The first three columns represent the configurations of the environmental
drivers. Setting names and descriptions specify our purposes for intervention. The last three columns show success rates and conversation length testing games. Games marked with "seen" are validation games with the same image set as training. Games marked with "unscon" are testing games with unseen images.
Game Settings
Communication Accuracy (%) SD (avg. step)
maxime decision stop
early
update
description
setting
instance
class
yes
yes
max
our experimental setting
complete
98.07 0.01(1.03)
70.37 : 0.04(2.36)
39.40 :: 0.05(3.76)
max
control setting for early decision control setting for evolving sender
max-step
86.27 (.15(7.00)
67.93 + 0.02((7.0)
38.40 + 0.04447.00 00)
yes
sender-fixed
99.60
0.01(2.41)
11.80 0.02(3.83)
45.40 + 0.02/4.75)
yes
yes
one
control setting for soquential game
one-step
22.87
0.23(1.60) 1.00)
14:07 0.15(11.0)
9.60 0.09X .00)
buseline for all settings above
retrieve
99.17 1: 0.01(7.00)
76.80 2: 0.02/7.00)
48.00 1: 0.02(7.00)

---

Can receiver make early decisions? The hypothesis is that the receiver's decision before exhausting the allowed communication steps may inform the sender of the marginal benefit of each stroke and incentivize it to prioritize the most informative strokes. The corresponding control setting is max-step, wherein the receiver can only make the choice after the sender finishes its drawing at the maximum step. This factor is on in other settings.
Can the sender change its way of drawing? The hypothesis is that the mutual adaptation of the sender and the receiver may lead to better abstraction in the evolved sketches. Particularly, the sender may develop new ways of drawing in the evolution. The corresponding control setting is sender-fieed, wherein we freeze the parameters of the sender such that the receiver has to adapt to its partner. This factor is on in other settings.
Is the game sequential, and can the receiver observe more complex drawings? ? The hypothesis is that the modeling of a sequential decision-maken may regularize the arbitrariness, which is unique for graphical conventions. The corresponding control setting is one-step: There only exists one step of sketching. thus no sequential decision- making at all. This factor is on in other settings.
game and the evolution from more complex sketches
Training, validation, and generalization We train the sender/receiver with batched forward and back- propaaappag,ion, with batch size of 64 and maximum roll-out step T 7. We update using Adam (Kingma and Ba, 2015) with the learning rate 0.0001 for a total of 30k iterations. We train each model on a single Nvidia RTX A6000; one experiment takes 20 hours. Except for the sender-fixed setting, there is warm-up phase in the first 2000 iterations for the sender where we linearly increase the learning rate to 0.0001, After the warm-up phase, the learning rate of both agents will be decreased
1-0000
where i is the number of training iterations. In all settings, we set M 3, 7 0.85. Between iterations, we randomly sample another 10 batches for validation. We use 30 categories (10 images per category) for training and held out 10 images per category for the unseen-instance test; another 10 categories are for the unseen-class test. Each image is communicated as the target, resulting in 300++100 pairs in the test set. At each iteration, the categories and instances are sampled randomly to constitute the context. Results henceforth are statisties of 5 random seeds.
exponentially by 0.99
1000
5.2 Results
5.2.1
Communication efficacy and sketch abstraction
We record both the success rate and the communication steps over the training iterations; see Fig. 3. The communication
considered successful when the receiver makes the correct prediction at the end of the game. In Fig. 3a, agents in all settings except one-step converge to a success rate above 80%. Among them, the communicating pairs the complete setting and the sender-fixed setting evolve to achieve comparable success rate with the retrieve baseline. Interestingly, these two pairs also emerge a phenomenon resembling the natural observation in human studies, named systematic reduction (Lewis, 1969): The average steps first increase and then gradually decrease as in Fig. 3b. Contrasting complete and render-fieed, we can see: (i) The emergent conventions in the former is much simpler than the latter (less steps in Fig. 3b), which implies the contribution of mutual adaptation in sketch abstraction. (ii) The success rate in Fig. 3a in the former converges a
1.0
1.0
02
0.0
504
50.6
conploupt complosds camplesgy?
204
204
02
02
cander ago
66
Kakkiing Revatiiis
(a) validation accuracy
(b) average communication steps
(c) prediction accuracy
Figure 3: Statistics aggregated over all random seeds. (a) The validation accuracy of different game settings and the ablation baseline. (b) The average communication steps under different settings and ablation basclines. ? 0.85 by default, and 0.95 complete95 and cumulative9S. (c) The prediction accuracy when receivers are presented with sketches drawn by corresponding senders at 1, 3, 5, and 7 steps, respectively. These sketches are collected in standalone roll-out after each iteration, where the early decision is disabled; agents are trained with the complete setting, where the carly decision is enabled.
7

---

bit more slowly, which is reasonable since the senders can explore and change the way of drawing. In comparison, if the receiver cannot make early decisions, it has no intention to relate sketches (i.e., Ct-1 and C₃) at consecutive timesteps. The sender is thus wnaware of each stroke's marginal information gain, which in return makes their learning harder. This might explain the relatively low success rate in the max-step setting. The failure of the one-step pairs reveals the irreplaceable roles of sequential makingg and observing complex sketches in emergent graphical communication.
To further inspect how our proposed modeling and learning on sequential decision-making facilitate the desired evolution in the sketches, we conduct an ablation study by comparing our proposed learning surrogate (Eq. (5)) and vanilla policy gradient baseline, REINFORCE (Williams, 1992)
with Monte Carlo cumulative rewards ET
Σ,
V log ( Ex
Our comparison spans three axes. First, the REINFORCE baseline converges much more slowly than the proposed surrogate; see cumulative vs complete in Fig. 3a. Second, we check the robustness under the variation of decay factor 7: As shown in Fig. 3b, while the proposed method shows stable convergence in the communication steps under 7 = 0.85 and 0.95, the REINFORCE baseline exhibits high-variance behavior under certain setup (7- = 0.95). Third, we check if agents' early terminations are caused by their awareness of the indistinguishable performance in longer and shorter episodes. Given precondition that the longer the episodes are, the earlier the success rate increases, it should be the increase in the average performance of shorter episodes that causes the average timesteps to decrease. Taking 1-step and 3-step communication for example, in the complete setting, we shall the success rate of the 3-step to achieve high earlier, which is then caught up but not exceeded by the 1-step. The not exceeding condition is a crucial cue to validate that the communicating partners were actively pursuing the Pareto front (Kim and De Weck, 2005) / efficiency bound (Zaslavsky et al., 2018) of accuracy and complexity. This is exactly what our proposed method emerges as shown in Fig. 3c. In contrast, in the REINFORCE baseline, under the same decay factor, the performance of 1-step surpasses 3-step communication. It seems if the incapability of learning long episodes caused the agents to avoid them.
Together, all our results success rate and communication steps are consistent with predictions based on our hypotheses, which justifies our environments and models. However, a high success rate does not necessarily imply high convention quality. Another essential property for conventions is stability (Lewis, 1969): There should exist common patterns in the repeated usage of conventions to convey similar concepts. We take the viewpoint of representation learning and concretize the vague stability with three formally defined properties: iconicity, symbolicity, and semanticity. Below, we introduce our experiments to measure these properties systematically.
5.2.2 Iconicity: generalizing to unseen images
We define iconicity as the drawings exhibiting a high visual resemblance with the corresponding images, such that they are proximal to the latter when measured on the high-level embedding of a general-purpose visual system. To quantitatively measure the visual similarities, we need to compute the distance between the sketch and image the visual embedding space (e.g., with cosine similarity). However, since we do not know how the embeddings distribute in their space, this unnormalized measure prone to model-speeific biases. Fortunately, the receiver's 's policy takes the form of a softmax of the cosine similarity between the embedding of the sketch and context set with target image and some randomly sampled distractors, which naturally approximates the normalization we want. Therefore, communication accuracy can serve as visual similarity measure, and its generalizubility to unseen images can pinpoint the emergent preservation of iconicit. Intuitively, in open-domain communication, agents would see novel scenes with known or unknown concepts- unseen instances of seen classes and unseen classes, respectively. They should still be able to communicate with established conventions or with unconventionalized iconic drawings. A successful generalization to unseen classes (i.e., zero-shot generalization) is more difficult than unseen instances; partners may increase the conversation length and communicate with more complex sketches.
Tab. reports the success rates and average timesteps in our generalization tests. The retrieve setting is the baseline since there is no evolution at all and the sketches should resemble the original images the most (i.e., possessing the highest iconicity). Unsurprisingly, its generalization performance upper- bounds all other settings. Among the experimental and controlled settings, the complete, the max-slep, and the sender-fixed agents generalize relatively well in unseen instances (70.37 + 0.04, 67.93 0.02, 71.80 + 0.02) and generalize above chance (39.40 40 + 0.05, 38.40 + 0.04, 45.40 + 0.02 >25.00) in
8

---

unseen classes. Interestingly, complete and sender-fixed communicating partners intelligently turn to longer episodes for better generalization, better than the max-step agents. This finding implies the partners may turn to more iconic communication when there is no established conventions/symin symbols,, just as we humans do. Strikingly, the max-step conventions seem to loose more iconicity, possibly due to confusion on marginal information gains. The one-step drawings seem to lack iconicity.
5.2.3 Symbolicity: separating evolved sketches
Next, we measure symbolicity to evaluate the graphical conventionalization. We define symbolicity as the drawings being consistently separable in the high-level visual embedding, which facilitates new communication partners to easily distinguish between categories without grounding them to referents. Based on this definition, a more symbolic drawing should be more easily separable into their corresponding categories. To measure such separability, we use pre-trained VGG16 as the new learner and finetune the last fully connected layer to classify evolved sketches into the 30 categories. Technically, we first get the 300 final canvases from the communication game, 10 for each category. Among them, we use 70% for training and 30% for testing.
The bar plot in Fig. 4 shows the classification results. Since agents in the one-step setting do not play the game successfully, they may not form a consistent way to communicate, Agents in the complete setting achieve the highest accuracy, higher even compared with the result of the original images. This finding indicates that agents in the complete setting agree on a graphical convention that consistently highlights some features across all training instances in each category, which are also distinguishable between categories. Comparing the max-step with the complete setting, we know that early decision is a crucial factor for more symbolic conventions. Comparison between the sender- fixeer--ieedddee setting and the complete setting suggests that the sender's s evolution also contributes to high symbolicity.
Clasllicaton auosFay
40 60
Table 2: Semantic correlation between the word vector space the communicated target and the feature vector space of the trained VGG.
-00
setting
correlation
complete
0.43 + 0,02
Traiiing settings
max-step
0.41 - 0.13
sender-fixed
0.36 + 0.06
Figure 4: Testing results of classifiers trained with sketches evolved under different settings. img denotes images, and retrieve denotes unevolved sketches.
one-step
0.31 + 0.08
retrieve
0.55 + 0.00
5.2.4 Semanticity: correlating category embedding
The last desired property is that the evolved sketches can preserve the perceptual metric in images (Zhang et al., 2018). We define semanticity as the topography of the high-level visual embedding space of drawings being strongly correlated to that of images, such that semantically similar instances and categories lie close to each other in the embedding space. To examine such topographic correlation, we project category names to the feature space using word2vec (Mikolov et al., 2013) featureA, and project the evolved sketches to the feature space using the trained VGG Sec. 5.2.3 as featureB. We compute the correlation of the distances between all the possible pairs of featureB and the corresponding pairs of featureA as the semanticity measure. The results in Tab. 2 show that semanticity
barer
burge
bararr
apple on
Figure 5: t-SNE of visual embedding of the original images (left), unevolved sketches in the retrieve setting (middle), and evolved sketches in the complete setting (right). These embeddings are from the finetuned VGGNets in Sec. 5.2.3. The evolved sketches have clearer boundaries due to the discrimination game, while maintaining the topography such that similar concepts close to each other.

---

24
8
71
Figure 6: Sketch evolution of rabbit and giraffe. For each example, sketches from the left to the right show the change the final-step canvas from iteration to 30,000. Please refer to Appendix for more results.
can be better retained in the complete setting compared with the retrieve baseline. We further visualize the embeddings of the sketches in 2D space via t-SNE (Van der Maaten and Hinton, 2008). Fig. show the visualization of the original images and the sketches in the retrieve and complete settings; please refer to Appendix B for results of other settings. Images/drawings from the same category are marked in the same color. As shown, boundaries between categories are clearer in the evolved drawings in the complete setting than in retrieve or original images; but semantically similar categories are still close to each other. For example, cow, deer, horse, and camel are proximal to each other, while burger and apple are far from them. These results highlight another uniqueness of visual communication over its symbolic counterpart: The similarity in the visual cues in the conventions may hint the semantic correlation between the referents.
5.2.5 Visualizing sketch evolution
To better understand the nature of the emerged conventions, we inspect the intermediate sketches in the evolution processes. Specifically, visualize the process under the complete setting. Fig. 6 shows three instances in two categories. For each example, drawings from the left to the right show the change of the final-step canvas. Sketches' complexity gradually decreases after an initial increase, echoing the trend of reduction described in Sec. 5.2.1. For rabbits, in the beginning. the strokes may depict instances from different perspectives; through iterations, they converge to highlight the rabbit's 's long ear. As for the giraffe, the agents gradually learn to emphasize the long neck. In the third example, although the giraffe lowers its head, we can still see an exaggerated vertical stroke for the neck, similar to the first example where the giraffe's 's head is raised. These examples show how the sender gradually unifies drawings of the same category: After evolution, the sender is inclined to use the first several strokes to depict the most salient parts of visual concepts.
6 Conclusion
We present the first step of modeling and simulating the evolution of graphical conventions between two agents in a visual communication game, Agents modeled in our framework can successfully communicate visual concepts using sketches as the medium. We measure the emergent graphical conventions over three carefully defined properties, iconicity, symbolicty, and semanticity. The experimental results under different controls suggest that early decision, mutual adaptation, and sequential decision-makin However, two-stage pre-trained senders possess limitations; an ideal sender would not need to convert the images to pixel-level sketches before it starts sketching. The limitations in the pre-trained sketching module also constrain the discriminative need among the selected classes. We hope to investigate these limitations in future research. With the uniqueness of visual conventions demonstrated, we hope our work can invoke the study of emergent communication in the modality of sketches.
can encourage symbolicity while preserving iconicity and semunticity.
Acknowledgments J. Joo was supported in part by the NSF SMA/SBE #1831848, We would like to thank Prof. Marco Baroni (Pompeu Fabra Univ.) and Prof. Yaodong Yang (Peking Univ.) for helpful comments on the draft, Miss Chen Zhen (BIGAI) for making the nice figures, all anonymous reviewers for their constructive comments on prior submissions, and Prof. Ying Nian Wu (UCLA), Prof. Hongjing Lu (UCLA), Prof. Federico Rossano (UCSD), Dr. Zilong Zheng (BIGAI), Dr. Chi Zhang (BIGAI), Baoxiong Jia (UCLA), Dr. Qing Li (BIGAT), and Dr. Bo Pang (UCLA) for discussion on the algorithm and experimental designs.
10

---

References
Bhunia, A. K., Das, A., Muhammad, U. R., Yang. Y., Hospedales, T. M., Xiang, T., Gryaditskaya, Y., and Song, Y.ZZ. (2020). Pixelor: A competitive sketching ai agent. so you think you can sketch? ACM Transactions on Graphics (TOG), 39(6):1-15.
Bouchacourt, and Baroni, M. (2018). How agents see things: On visual representations in an emergent language game. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP). 2
Cao, K. Lazaridou, A., Lanctot, M., Leibo, J. Z., Tuyls, K., and Clark, S. (2018). Emergent communication through negotiation. In International Conference on Learning Representations (ICLR). 2
Eccles, T., Bachrach, Y., Lever, G., Lazaridou, A., and Graepel. T. (2019). Biases for emergent com- munication in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS).
Eitz, M., Hays, J., and Alexa, M. (2012). How do humans sketch objects? ACM Transactions on Graphics (TOG), 31(4):1-10.
Evtimova, K., Drozdov, A., Kiela, D., and Cho, K. (2018). Emergent communication a multi-modal, multi-step referential game. In International Conference on Learning Representations (ICLR). 2,
Fan, J. E., Hawkins, R. D., Wu, M., and Goodman, N. D. (2020). Pragmatic inference and visual abstraction enable contextual flexibility during visual communication. Computational Brain & Behavior, 3(1):86-101.
Fay, N., Ellison, M., and Garrod, S. (2014). Iconicity: From sign system in human communication and language. Pragmatics Cognition, 22(2):244-263, 1,2
Fay, N., Garrod, S., Roberts. L., and Swoboda, N. (2010). The interactive evolution of human communication systems. Cognitive Science, 34(3):351-386, 1,3
Fay, N., Walker, B., Swoboda, N., and Garrod, S. (2018). How create shared symbols. Cognitive Science, 42:241-269. 2
Foerster, J. N., Assael, Y. M., De Freitas, N., and Whiteson, S. (2016). Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS).
Graesser, L., Cho, K., and Kiela, D. (2019). Emergent linguistic phenomena in multi-agent commu- nication games. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP). 2,
Ha, D. and Eck, D. (2018). A neural representation of sketch drawings. In International Conference on Learning Representations (ICLR). 2,
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2019). Dream to control:
Learning behaviors by latent imagination. In International Conference Learning Representations (ICLR). 5
Harispe, S., Ranwez, S., Janaqi, S., and Montmain, J. (2015). Semantic similarity from natural language and ontology analysis. Synthesis Lectures Human Language Technologies. 8(1):1-254.
Havrylov, S. and Titov, (2017). Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Advances in Neural Information Processing Systems (NeurIPS). 3.
Hawkins, R. X., Sano, M., Goodman, N. D., and Fan, J. E. (2019). Disentangling contributions of visual information and interaction history in the formation of graphical conventions. Annual Meeting of the Cognitive Science Society (CogSci). 1, 2,

---

Hoffmann, D. L.. Standish, C. D., García- M.. Pettitt, P. B., Milton, J. A., Zilhão, J., Alcolea-
González, J. J., Cantalejo-Duarte. P., Collado. H., De Balbín, R., et (2018). U-th dating of carbonate crusts reveals neandertal origin of iberian cave art. Science, 359(6378)
Huang, Z., Heng. W., and Zhou, S. (2019). Learning to paint with model-based deep reinforcement learning. In International Conference on Computer Vision (ICCV). 3.
Kampelmuhler, M. and Pinz, A. (2020). Synthesizing human-like sketches from natural images using conditional convolutional decoder. In Proceedings of Winter Conference on Applications of Computer Vision (WACV). 3.5
Kim, Y. and De Weck, O. L. (2005). Adaptive weighted-sum method for bi-objective optimization: Pareto front generation. Structural and Multidisciplinary Optimization, 29(2):149-155) 8
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR). 7
Kingma, D. and Welling, M. (2014). Auto-encoding variational {Bayes). International Conference Learning Representations (ICLR). 6
Lazaridou, and Baroni, M. (2020). Emergent multi-agent communication the deep learning era.
arXiv preprint wrXiv:2006.02413 1,2
Lazaridou, A.,
Hermann, M., Tuyls, K., and Clark, S. (2018). Emergence of linguistic commu- nication from referential games with symbolic and pixel input. In International Conference on Learning Representations (ICLR). 2, 3,5
Lazaridou, A., Peysakhovich, A., and Baroni, M. (2017). Multi-agent cooperation and the emergence of (natural) language. International Conference on Learning Representations (ICLR). 2, 4,
Lewis, D. K. (1969). Convention: A Philosophical Study. John Wiley & Sons. 7,8
Li, M., Lin, Z., Mech, R., Yumer, E., and Ramanan, D. (2019). Photo-sketching: Inferring contour drawings from images. In Proceedings of Winter Conference on Applications of Computer Vision (WACV).
Mihai, D. and Hare, (2021). Learning to draw: Emergent communication through sketching. In Advances in Neural Information Processing Systems (NeurIPS). 3,
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3773 3,
Mordatch, and Abbeel, (2018). Emergence of grounded compositional language multi-agent populations. In AAAI Conference on Artificial Intelligence (AAAI). 2
Muhammad, U. R., Yang, Y., Song, Y.-Z., Xiang, T., and Hospedales, M. (2018). Learning deep sketch abstraction. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Ren, Y. Guo, S., Labeau, M., Cohen, S. B., and Kirby, S. (2020). Compositional languages emerge in
neural iterated learning model. In International Conference on Learning Representations (ICLR). 3
Sangkloy, P., Burnell, N., Ham, C., and Hays, (2016). The sketchy database: learning to retrieve
badly drawn bunnies. ACM Transactions on Graphics (TOG), 35(4):1-12. 5, 6
Sayim, B. and Cavanagh, P. (2011). What line drawings reveal about the visual brain. Frontiers in Human Neuroscience, 5:1 2:118.2 5
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014). Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML). 6
Simonyan, K. and Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations (ICLR). 5
12

---

Song, J., Pang, K., Song, Y.-Z., Xiang, T., and Hospedales, T. M. (2018). Learning to sketch with shorteut cycle consistency. In Conference on Computer Vision and Pattern Recognition (CVPR).
Sutton, R. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press. 2, 6
Sutton, R. S., McAllester, D. A., Singh, P., and Mansour, (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurlPS).
Van der Maaten, I., and Hinton, G. (2008). Visualizing data using t-sne. Journal of Machine Learning Research (JMLR), 9(11). 10
Wang, A., Ren, M.. and Zemel. (2021). Sketchembednet: Learning novel concepts by imitating drawings. In International Conference on Machine Learning (ICML).
Wen, Y. Yang, Y., Luo, R., Wang, J., and Pan, W. (2019). Probabilistic recursive reasoning for multi-agent reinforcement learning. International Conference on Learning Representations (ICLR). 2
Williams, R. (1992). Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8(3-4):229-25 256.. 8
Xie, S. and Tu, Z. (2015). Holistically-nested edge detection. In International Conference on Computer Vision (ICCV).
Yu, Q.. Liu. F., Song, Y.-Z., Xiang, T.,, Hospedales. T. M., and Loy, C.-C. (2016). Sketch me that shoe. In Conference on Computer Vision and Pattern Recognition (CVPR). 3
Yuan, L.,, Gao, X., Zheng, Z., Edmonds, M., Wu, N., Rossano, F., Lu, H., Zhu, Y., and Zhu, S.-C. (2022). situ bidirectional human-robot value alignment. Science Robotics, 7(68).
Zaslavsky, N., Kemp, C., Regier, T., and Tishby, N. (2018). Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences (PNAS), 115431)77937-794.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. (2018). The unreasonable effectiveness of deep features perceptual metric. In Conference Computer Vision and Pattern Recognition (CVPR).
Zou, C., Yu, Q., Du, R., Mo, H., Song, Y.-Z., Xiang, T., Gao, C., Chen, B., and Zhang, H. (2018). Sketchyscene: Richly-annotated scene sketches. In European Conference on Computer Vision (ECCV).
13


Reviews:
[{"review_id": "mMD2LPsCTKy", "reviewer": "Reviewer_b2Pi", "summary": "The authors proposed to address the problem of visual communication with graphical sketches. By proposing a reinforcement learning framework, two agents (a sender and a receiver) play a visual communication game to reach an agreement on what the visual message is. There are three key factors used to provide some insights inside the visual communication. Experiments are conducted on a subset of the sketch-photo paired dataset, i.e., Sketchy, to validate the proposed method. Some interesting results are provided as well.", "questions": "(1) Did the authors try to use alternative sketching methods to measure the impact of sketching quality? One possible candidate could be CLIPasso [A].\n\n[A] CLIPasso: Semantically-Aware Object Sketching\n\n(2) At each time step, five new strokes will be added on the previous canvas $C_{t-1}$. Is there an action to remove existing strokes? I personally think some previously drawn strokes will become redundancy when the sender evolves.\n\n(3) Human study on sketch evolution would be helpful to confirm the quality of the generated sketch by the sender. \n\n(4) Is that the final sketching could be somehow matched to the most informative strokes to represent the object in the original natural image? This could be interesting to know. ", "limitations": "The authors have discussed the limitations in terms of two-stage pretrained senders.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This works proposed to address a significant problem of emergent graphical conventions by designing a visual communication game using sketches, which is very interesting and inspiring.  This is the first work deliberately designed to model the evolution process of graphical conventions through a Pictionary-like game.\n- The technical contributions are concrete. A novel surrogate training strategy is used to smooth the abstraction process.\n- Iconicity, symbolicity and semanticity are proposed to evaluate the emergent graphical conventions, which are of great importance to enable quantitative measurements on visual communication.\n\nWeaknesses\n- The effectiveness of the communication process might be influenced by the quality of the target sketch $\\hat{I_S}$ produced from image $I_S$.\n- The data used for training and testing is very small, i.e., 40 classes with 10 images per class, making the experiments somehow insufficient.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "fL1AYGpyjoz", "reviewer": "Reviewer_kX83", "summary": "This work is an attempt at mimicking communication processes in humans that predate language, namely the process of drawing sketches. Here, a sender tries to draw an image via minimalist sketches step by step in an attempt to communicate with a receiver. The work claims a new value function based MARL training objective and defines three aspects of communication via the sketches: iconicity, symbolicity, semanticity. Results align with the author's expectations of the evolution of emergenet communication.\n", "questions": "How'd you end up with the learning rate schedule and other parameter that you did, they seem relatively non standard (perhaps the result of a grid search?)? - lines 224-236\n", "limitations": "Authors discussed limitations but limited themselves to the accuracy of the vision systems available (the need to convert from real to pixellated images, etc.). What could be better discussed as limitations is the ability to evaluate along the three dimensions proposed in the work without the aid of human annotators.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n\n- The paper is quite refreshing in its motivation: study emergent communication using sketches (as opposed to much of the current literature which uses discrete token based communication).\n\n- This line of work could open up ways to study emergence of pictographic languages, a quite underexplored region of the emergent communication space.\n\n- I quite like the introduced measures of iconicity, symbolicity, semanticity - their underlying definitions are exactly what is needed to measure effectiveness of the communication in such referential games.\n\n\n*Weaknesses:*\n\n- The clarity of the novelty surrounding the training objective via joint value function is unclear with respect to prior works.\n\n- The results surrounding how the sketches evolve into semantically similar classes is interesting and would serve to validate the motivation of this paper, but is rather difficult claim to assess the veracity of. As it appears to be a side effect of the proposed training objective, should the take away be that this particular objective better models emergent communication through sketches in humans? \n\n- A summarized sentence on why the observation spaces between sender and receiver are disjoint (line 120) would be useful instead of just a citation. (The images themselves are distinct but can belong to the same partition of class is an additional missing clarification.)\n\n- Fig 1 choices might not be the best with respect to conveying the goals of the work. Line 170-171 clearly say that the receiver only picks an image when they have info to distinguish from the distractors. The round 1 of Fig 1 already appears to have more than sufficient information to distinguish rooster from the other two, it is only when it is simplified in round 2 that there is ambiguity with the other bird. This calls into question how ''good`` the distrctor images are and whether most of the games can just be solved with maximal reward for both agents if sketching abilities were high within a single step.\n    -  A side effect of this is that it is not clear whether the emergent communication happens because of the communication aspect or if it is simply dependent on the actual drawing abilities of the sender.\n\n- The claims regarding symbolicity and semanticity seem viable to judge through automated methods as in the paper but I am not clear how this could work for iconicity. High visual resemblance to the images is derived from the perspective of an agent - either a human or one of the learning agents based on their training objective. Without such a reference to ground these numbers, once again it is difficult to assess the accuracy of the lines 359-361.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZK63xKCVAMV", "reviewer": "Reviewer_msus", "summary": "In this paper, the authors study the problem of emergent communication with sketching. In each game, the sender describes an image through predicting strokes sequentially, while the receiver picks the target image which belongs to the same visual category as the image seen by the sender from a pool of images. The sender is a pretrained stroke predictor and receiver decides whether to make a decision based on the distance between the embeddings of the current canvas and last canvas and the images in the pool. They are jointly  trained with reinforcement learning optimizing for the discounted guessing reward. Note that the canvas renderer is differentiable with respect to the strokes. According to the results, making early decisions is especially useful for the seen instances, since the receiver could learn to adapt to more abstract protocols for the instances it got trained on; the pretrained sender is already a good enough sender which often better than the sender trained in the game, while the latter learns to use less steps in the game; the sequential game is also very crucial since it makes the sender aware of the information gain of each timestep. The authors also evaluate the iconicity, symbolicity and semanticity of drawings. It is also observed in the case study that the sketch becomes simplified and the most silent part of the concept is always preserved. \n", "questions": "According to Section 5.2.2, \"iconicity [are defined] as the drawings exhibiting high visual resemblance with the corresponding images\", the evaluation I was expecting is on the perceived visual similarities between the drawing and the image. However, the authors merely compared accuracy on the unseen classes. As much as I agree that strong iconicity should implies good generalizability, the other direction does always hold. Can the author explain this mismatch a bit more? I think the authors should directly use generalizability as the property's name or provides human ratings for iconicity. Either way will make the paper much scientifically reliable.", "limitations": "The authors have not discussed any potential negative societal impacts. Since the paper uses human sketches as pre-training data, the data might contains the biases, which might be learned by the model. However, I think this is a minor point.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper studies this very interesting problem of emergent communications of drawing. Language is not the only modality humans have at disposal, and drawing is a more natural way to communicating concrete concepts. Learning to form abstractive conventions will hint us about the way communication protocols are evolved.\n2. The technical method and evaluation is sound as a first step towards this question. \n\nWeaknesses:\n\n1. The sender is pretrained and already performs well according to the comparison between sender-fixed and complete models (accuracy-wise). I think a more interesting setup would be training the sender from scratch with limited prior knowledge about the visual categories it needs to depict. This would give us more information about how well a sender can learn to draw with the communication goals. \n2. The definition of iconicity and the evaluation are misaligned. Please refer to the questions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7MNmZn5jL0k", "reviewer": "Reviewer_6BRM", "summary": "This paper explores the modality of sketches in the context of emergent communication games with visual referents. While other papers have already studied graphical channels as an alternative to discretised tokens explored in prior emergent communication research (e.g. ref 29 in the paper), the unique feature of this work is studying the evolution of these conventions in a multi-step game. The authors define three properties (iconicity, symbolicity and semanticity) to study the evolved sketches with quantitative and qualitative methods. Results show that the agents are able in most cases to learn successful communication channels. ", "questions": "- Can you clarify that M=3 (L231) you mean that there are only three context images? If so,\n    - Why is this set so low? Other works with referential image games have used significantly larger distractor sets and still achieved good to near perfect communication (both with sketching and with token based messages). Further, the unseen images seem to behave significantly worse, whereas in many emergent communication studies performance on unseen test data is often in line with the training data performance - what causes this?\n    - How is it that the communication accuracy is so low in some settings? In the one-step setting it would be significantly worse than random guessing; for many of the other \"unseen class\" settings it is barely better than random\n- What happens when agents from pairs of different communication scenarios are tested (i.e. different image subsets)? Are the conventions emergent from disjoint subsets still holding up?\n- The proposed measure of semanticity is only demonstrated qualitatively; could a quantative measure not be developed? It also would be interesting to understand how this relates to design choices in other works which explictly try to enforce this (e.g. the \"perceptual\" losses in 29 and also in Mihai, D and Hare, J, \"Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?\", SVRHM 2021)\n", "limitations": "As it stands whilst the authors try to draw attention to some of the shortcomings of their work, they do it out of context with respect to the published literature. As highlighted above in the the questions and the weaknesses there are some particular areas where limitations could be better drawn-out and discussed.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- Studying emergent communication through the modality of sketches is an exciting direction, although this paper is not the first to explore it (paper ref 29, but also see Fernando et al, 2020, From language games to drawing games, arXiv preprint arXiv:2010.02820). The study of the *evolution process* of the emergent sketches between two cooperative artificial agents, however, is a fruitful initiative to be explored and the authors compare it to observations from human studies.\n- Using the previous canvas as an external memory and comparing against it, among other distracting referents, to incentivise the sender to draw informative sketches as early on in the communication is a nice idea. \n- This work proposes three evaluation metrics to assess the evolved sketches; these are a valuable contribution for the field going forward.\n\n## Weaknesses\n\n- This work, in its current form, doesn't take into account the current state of emergent graphical communication and advances in differentiable drawing made in the last two years. These could have helped address some of the problems highlighted in this work (e.g. the sender first using a visual module that turns a natural image into a pixel sketch is something overcome by many recent works). \n- Building upon the above, the comparison against other similar models is somewhat lacking; for example the category oriented game setting is seemingly the same as the \"object-oriented\" setting explored by ref 29 (which was published at NeurIPS last year). \n- Due to the choice of sketching module, only 40 categories (with only 10 images per category) of Sketchy dataset are explored in this paper. Other works have considered much more diverse image sets and considerably harder games.\n- Communication performance seems to be relatively low compared to other approaches given the simplicity of games with only two distractors/context images. It's not clear why this is.\n- The new measures proposed are mostly only evaluated qualitatively.\n- Human performance against the agents is not considered as it has been in related works.\n\n\n# Other remarks\n\n- L17 of the abstract \"the unexplored modality of sketches\" isn't really true. Sketching has already been looked at by other papers. It would be much better to refocus the paper on the contribution of this work in studying the evolution of graphical communication and providing new measures. Similar issues appear on L97, L103 and L366. \n- Section 2 \"Learning to sketch\" does not give an accurate picture of the state of the art in this space; there is considerable work in that goes well beyond \"the early phase\". For example:\n    + A. Das et al, 2021. Cloud2curve: Generation and vectorization of parametric sketches. CVPR 2021\n    + T.Z. Li et al. Differentiable Vector Graphics Rasterization for Editing and Learning. SIGGRAPH Asia 2020\n    + D. Mihai and J. Hare, 2021. Differentiable Drawing and Sketching. arXiv:2103.16194\n    + D. Smirnov et al, 2020. Deep parametric shape predictions using distance fields. CVPR 2020\n    + H. Mo et al, 2021. General virtual sketching framework for vector line art. ACM Transaction on Graphics 2021\n- Section 5.2.5 would benefit from expansion and more discussion; there are contributions here.\n- Some of the refs need updating to the published papers rather than arxiv versions (e.g. 23, 29)\n- Please define communication accuracy for the benefit of readers not familiar with emergent communication games\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept