Learning to Draw: Emergent Communication through Sketching
Authors: Daniela Mihai
Jonathon Hare
Venue: NeurIPS 2021 Oral (2021)
Keywords: Emergent Communication
Sketching
Drawing
Perceptual Loss
Interpretability
Visual Communication

Abstract: Evidence that visual communication preceded written language and provided a basis for it goes back to prehistory, in forms such as cave and rock paintings depicting traces of our distant ancestors. Emergent communication research has sought to explore how agents can learn to communicate in order to collaboratively solve tasks. Existing research has focused on language, with a learned communication channel transmitting sequences of discrete tokens between the agents. In this work, we explore a visual communication channel between agents that are allowed to draw with simple strokes. Our agents are parameterised by deep neural networks, and the drawing procedure is differentiable, allowing for end-to-end training. In the framework of a referential communication game, we demonstrate that agents can not only successfully learn to communicate by drawing, but with appropriate inductive biases, can do so in a fashion that humans can interpret. We hope to encourage future research to consider visual communication as a more flexible and directly interpretable alternative of training collaborative agents.

Learning to Draw: Emergent Communication through Sketching
Daniela Mihai
Jonathon Hare
Electronics and Computer Science The University of Southampton Southampton, UK adm1g mamllg55@stoonn.. ac.
Electronics and Computer Science
The University of Southampton Southampton, UK j jsh2@soton.ac acc
Abstract
Evidence that visual communication preceded written language and provided a basis for it goes back to prehistory, in forms such as cave and rock paintings depict- ing traces of our distant ancestors. Emergent communication research has sought to explore how agents can learn to communicate in order to collaboratively solve tasks. Existing research has focused on language, with learned communication channel transmitting sequences of discrete tokens between the agents. In this work, we explore a visual communication channel between agents that are allowed draw with simple strokes. Our agents are parameterised by deep neural networks, and the drawing procedure is differentiable, allowing for end-to-end training. the framework of a referential communication game, we demonstrate that agents can not only successfully learn to communicate by drawing, but with appropriate inductive biases, can do so in a fashion that humans can interpret. We hope encourage future research to consider visual communication as a more flexible and directly interpretable alternative of training collaborative agents.
Introduction
Imagine you and a friend are playing game where you have to get your friend to guess an object in the room by you sketching the object. No other communication is allowed beyond the sketched image. This is an example of referential communication game. To play this game you need have learned how to draw in a way that your friend can understand. This paper explores how artificial agents parameterised by neural networks learn to play similar drawing games. More specifically, we reformulate the traditional referential game such that one agent draws a sketch of given photo and the second agent guesses, based the drawing, the corresponding photo from set of images.
Spurred by innovations in artificial neural networks, deep and reinforcement learning techniques, recent work in multi-agent emergent communication 4 17 form of gameplay between agents to induce human-like communication. Artificial communicating agents can collaborate to solve various tasks: image referential games with realistic visual input [99 28 29),
19
29 37 pursues interactions in the
negotiation (2), navigation of virtual environments 22). reconstruction of missing input 4 [7] and, more recently, drawing games [TT]. The key achieving the shared goal in many of these games collaboration, and implicitly, communication. To date, studies on communication emergence in multi-agent games have focused exploring language- based communication channel, 28 29 (35. However,
with messages represented by discrete tokens or token sequences 66] 17
19
25
these communication protocols can be difficult for human to interpret B 26
33 In this work
propose a direct and potentially self-explainable means of transmitting knowledge: sketching.
Evidence suggests pre- and early-humans were able to communicate by drawing long before develop- ing the various stages of written language 220 38). Drawings such as petrograms and petroglyphs
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

exist from the oldest palaeolithic times and may have been used to record past experiences, events, beliefs or simply the relation with other beings [3] 2T). These pictorial characters which are merely impressions of real objects or beings stand at the basis of all writing [16]. This leads us to question if drawing is a more natural way of starting to study emergent communication and if it could lead to better written communication later on.
This idea has recently gained interest in several domains. In the cognitive science literature, neural models of sketching have been developed to study the factors which enable contextual flexibility in visual communication [8]. Likewise, works such as Fernando et al. [T] attempt to automate the artistic process of drawing by training agents in a reinforcement learning framework, to play variety of drawing games. However, the focus of this paper to open the doorway to exploring different types of communication between artificial agents and humans. The novelty of our work is also evident in the model framework which can be easily extended well beyond aspects previously studied.
Concretely, we propose a visual communication channel in the context of image-based referential games. We leverage recent advances in differentiable sketching that enables to construct an agent that can learn to communicate intent through drawing. Through a range of experiments we show that:
Agents can successfully communicate about real-world images through sketching game. However, training with loss that tries to maximise gameplay alone does not lead to human decipherable sketches, irrespective of any visual system preconditioning:
Introducing a perceptual loss improves human interpretability of the communication protocol, at little to no cost in the gameplay success;
Changes to the game objective, such as playing an object-oriented game, can steer the emer- gent communication protocol towards a more pictographic or symbolic form of expression; Inducing a shape-bies bias into the agents' visual system leads to more explainable drawings; A drawing agent trained with perceptual loss can successfully communicate and play the game with a human.
2 Communication between agents
Communication emerges when two or more participants are involved, share a goal, task or incentive which can be achieved only by transfer of information and is beneficial for all parties involved. Studies on language origins [36] 41 consider cooperation to be key prerequisite to language evolution as implies multiple agents having to self-organise and adapt to the same convention. Studies on the emergence of communication in cooperative multi-agent environments from recent years have focused on (natural) language learning 28 29]
and its inherent properties such as
compositionality and expressivity 17
number of works specifically relate to the overarching ideas of gameplay and learning in this paper. For example, Foerster et al. [12) proposed a framework for differentiable training of communicating agents which was later used by Jorge et al. [23] to solve image search tasks with two interacting agents communicating with atomic symbols. Lazaridou et al. [28] proposed an image-based referential game in which the agents again communicated using atomic symbols, and were trained using policy gradients. Havrylov and Titov [19 and Mordatch and Abbeel (35) both demonstrated that it was possible to use differentiable relaxations to train agents that communicated with sequences of symbols. In the former case, the agents played the referential game that we adopt for our experiments.
One of the long-term goals of this research in language emergence is develop interactive machines that can productively communicate with humans. As such we should ensure that whatever language artificial agents develop, it is one that human agents can understand. In our work, we take inspiration from the process and evolution of writing. Written language has undergone many transitions from early times to reach the forms we now know: from pictures and drawings to word-syllabie, syllabic and, finally, alphabetic systems. In the beginning, our early ancestors did not know how to communi- cate in writing. Instead, they began drawing and painting pictures of their life, representing people and things they knew about [16). Studies on the communication systems developed in primitive societies compare ancient drawings to the very early sketches drawn by children and talk about their tendency of concretely identifying certain things or events in their surrounding world [16] 24). Psychological and behavioural studies have shown that children try to communicate to the world through the images they create even when they cannot associate them with words [9].

---

model for learning to communicate by drawing
We present model consisting of agents, the sender and the receiver in which the sender learns to draw by playing a game with the receiver. The overall architecture the agents in the context of the game they are learning play is shown in Figure [ can be found at htlhttps///. http//ggttttb.t.ee cwm woom/Daaiieall3 1ew.eeeaaaaeeeeannnnggonooo LearningToDraw
Full code for the model and all experiments
Sender
Parnneter
Prinitiv
Victon
VGG16
Differentablee
MLP
Same photo same class
Compotett Seovas
Receiver
Receiver Feature Extractor
Skotch Feature
VGG16 to ReL.
Features
inner producct
Multiclass Hinge
Figure 1: Overview of the agent architecture and game setup. The 'sender' agent is presented with an image and sketches its content through a learnable drawing procedure. The 'receiver' agent is presented with the sketch and collection of photographs, and has to learn to correctly associate the sketch with the corresponding photograph by predicting scores which are compared to a one-hot target. Both agents parameterised by neural networks trained end-to-end using gradient methods.
3.1 The Game Environment
Our experimental setup builds upon the image referential game previously explored in studies of emergent communication [19 28 29 that derives from Lewis's signalling game [30]. We
implemented several variants of Havrylov and Titov [19 's image guessing game. The overall setting of these games is formulated as follows:
1. Two target photographs, P. and P,,, and set of K distractor photographs, (P(x) selected.
( === are
2. There agents: sender and receiver.
3. After being presented the P. target image, the sender has to formulate message conveying information about that image.
4. Given the message and the set of photographs, (p()) yk= U (Pr), consisting of all the distractors and the target P,, the receiver has identify the target correctly.
The specifics of how the photographs are selected (step above) depend on the game variant as described below. Success in these games is measured by the binary ability of the receiver to correctly guess the correct image or not; as such, the measure of communication rate is used to assess averaged performance over many games using independent images those used during training. Unlike Havrylov and Titov [9]'s 's game in which the sender helps the receiver identify the correct image by sending message constructed sequence of tokens drawn from a predefined vocabulary, propose using a directly interpretable means communication: sketching the target photograph.
Original game variant. In Havrylov and Titov 19 's variant of the game there is pool of photos from which the distractors and target P. are drawn randomly without replacement. The target P, set to be equal to P. In our original variant experiments the number of distractors, K, is set to 99.

---

Object-oriented game variants. In addition to the original setup, we explored two slightly different and potentially harder game configurations which were intended to induce the agents to draw sketches that would be more representative to the object class they belong to rather than to the specific instance of the class. These setups use labelled datasets where each image belongs to a class based on its contents. In the first of these variants (we refer to this as 00-game same), the target P, is set to be equal to P.. and the distractors and target are sampled such that their class labels are disjoint (that is every photo provided to the receiver has different class). The second setup (OO-game different) is similar to the first, but the target P. is chosen to be a different photograph with the same class label as target P... The intention behind these games is to explore universally interpretable depiction of the different object classes, which does not focus on individual details but rather conveys the concept. To some extent, this task is an example of multiple instance classification within weakly supervised setting [1, which has been previously explored in the emergent communication literature (28).
3.2 Agents' Architectures
Both agents act on visual inputs. The agents are parameterised by deep neural networks and are trained using standard gradient techniques (Section 3.3)
The agent's early visual system. We choose to model the early visual systems of both agents with the head part of the VGG16 CNN architecture (40] through to the ReI.U activation at the end of the last convolutional layer (commonly referred to as the ReLU5 3 layer) before the final max-pooling and fully connected layers. In all experiments, we utilise pretrained weights and freeze this part of the model during training. We justify this choice on the basis that it provides the agents with an initial grounding in understanding the statistics of the visual world, and ensures that the visual system cannot collapse and remains universal. The weights are the standard torchvision ImageNet weights, except in the cases where we explore the effect of shape bias (see Section 4.5). As these pretrained weights were learned with images that were normalised according to the ImageNet statistics, all inputs to the VGG16 backbone (including sketches) are normalised accordingly. The output feature maps of this convolutional backbone are flattened and are linearly projected to fixed dimensional vector encoding (64-dimensions unless otherwise specified). Because the datasets used in gameplay have different resolutions, the number of weights in the learned projection varies.
Sender Agent. The goal of the sender is to produce sketch from the input photograph. For experiments in Section we restrict the production of sketches to be a drawing composed of 20 black, constant width, straight lines on a white canvas of the same size as the input images. Experiments with fewer lines can be found in Appendix [ of course possible to have much more flexible definition of sketch and incorporate many different modelling assumptions. We choose to leave such exploration for future work and focus on the key question of whether we can actually achieve successful (and potentially interpretable) communication with our simplified but not unrealistic setup.
Given an input image, the agent's early visual system produces a vector encoding which is then processed by three-layer multilayer perceptron (MLP) that learns to decode the primitive parameters used to draw the sketch. This MLP has ReLU activations on the first two layers and tanh activation on the final layer. Unless otherwise specified, the first two layers have 64 and 256 neurons respectively. The output layer produces four values for each line that will be drawn; the values are the start and end coordinates of each line stroke in an image canvas with the origin at the centre and edges at 1.
To produce sketch image from the line parameters output by the MLP, we utilise the differentiable rasterisation approach introduced by Mihai and Hare [34). At a high level, this approach works by computing the distance transform on pixel grid for each primitive being rendered. A relaxed approximation of rasterisation function is applied to the distance transform to compute a raster image of the specific primitive. Finally, differentiable composition function is applied compose the individual rasters into single image. More specifically, the squared Euclidean Distance Transform is computed, D (s, e) over all pixels in the image, for each line segment starting at coordinate s and ending at e. These squared distance transforms are simply images in which the value of each pixel is replaced with the closest squared distance to the line (computed when the pixels are mapped to the same coordinate system as the line the top left of the image is (-1, -1) and bottom-righ -right is

---

(1, Using the subscript to refer to the i-th line in the sketch, each D seg (s,, et) es) is rasterised as sog (8,, Ci))
D2
R² = exp
(1)
σ²
where o² is a hyperparameter that controls how far gradients flow in the image, as well as the visible thickness of the line (o² 5 10 for all experiments this paper). We adopt the soft-or composition function [34] to compose the individual line rasters into a single image, but incorporate an inversion that sketch image, S, with white canvas and black lines is produced,
S s= I (--R.
where n is the number of lines. Finally, because the backbone CNNs work with three- band colour images, we replicate the greyscale sketch image three times across the channel dimension.
Receiver Agent. The receiver agent is given set of photographs and a sketch image, and is responsible for predicting which photograph matches the sketch under the rules of the specific game being played. The receiver's s visual system is coupled with a two-layer MLP with ReLU nonlinearity on the first layer (the latter layer has no activation functionn. Unless otherwise specified, all experiments use 64 neurons in the first layer and 64 in the final layer. The sketch image and each photograph are passed through the visual system and MLP independently to produce a feature vector representation of the respective input. A score vector is produced for the photographs by computing the scalar product of the sketch feature with the feature of each respective photograph. This score vector is un un-normalised but could be viewed as probability distribution by passing through a softmax. The photograph with the highest score is the one predicted.
3.3 Training details
By incorporating loss between the predicted scores of the receiver agent and the known correct target photograph, it is possible to propagate gradients back through both the receiver and sender agents. As such, we can train the agents play the different game settings. For the loss function, follow Havrylov and Titov [19] and choose to use Weston and Watkins 43 's multi-class generalisation of hinge loss (aka multi margin loss),
Igame
( (R, v) max(0, By + x.2)
where æ is the score vector produced by the receiver, and y is the true index of the target, and the subscripts indicate indexing into the vector. The rationale for this choice is that the (soft) margin constraint should help force the distractor photographs' features to be more dissimilar to the sketch feature. Tests using cross-entropy also indicated that it could work well as an alternative, however.
Optimisation of the parameters of both agents is performed using the Adam optimiser with an initial learning rate of 1 x 10-4 for all experiments. For efficiency, we train the model with batches of games where the sender is given multiple images which are converted sketches and passed the receiver which reuses the same set of photographs for each sketch in the batch (with each sketch targeting& different receiver photograph). The order of the targets with respect to the input image's 's sketches is shuffled every batch. Batch size is K + 1, where K is the number of distractors, for all experiments. Unless otherwise stated, training was performed for 250 epochs. A mixture Nvidia GTX1080s, RTX2080s, Quadro RTX8000s, and an CTTX-iitan was used for training the models. Higher resolution images required more memory. Training time varied from around 488 games/second (10 secs/epoch) for games using STL10 to around 175 games/second (around 5 mins/epoch) for Caltech-101 101 experiments with 128px images.
3.4 Making the sender agent's sketches more perceptually relevant
Perception of drawings has a long history of study in neuroscience [see e.g. 39 for overview]. In order to induce the sender to produce sketches that are more interpretable, we explore the idea of using additional loss function between the differences in feature maps of the backbone CNN
5

---

Nommalise Subtract
1.2 Norm Spatial Avg
Weighted Avg
peroeptraal
Figure 2: Computing a 'perceptual' loss with the early visual system. Features are extracted from the sketch S and corresponding photograph P from different layers of the backbone. The features are normalised over channels and subtracted. We take the sum of the squared differences over channels and average spatially. Finally, we compute a weighted average across layers.
from the produced sketch and the input image. Such loss has direct grounding in biology, where has been observed through human brain imaging studies that sketches and photographs of the same scene result in similar activations of neuron populations in V4 of the visual cortex, as well as other areas related to higher-order visual cognition [42]. At the same time, it has also been demonstrated that differences in feature maps from pre-trained CNN architectures can be good proxies for approximating human notions of perceptual similarity between pairs of images 45]
Inspired by Zhang al. 45 we formulate loss based on the normalised differences between feature maps of the backbone network from the application the network to both the input photograph and the corresponding sketch. Unlike Zhang et al.. we choose not to learn weightings for each feature map channel individually, but rather we consider all feature maps produced by a layer of the backbone to be weighted equally. Learning individual channel weighting would be an interesting direction for future research, but is challenging because we would want to avoid the network learning zero weights for each channel, where the perceptual loss is basically ignored.
Figure 2 illustrates our perceptual loss formulation; note also that unlike Zhang et al. [45] the final averaging operation does incorporate (per-layer) weighting, wi, which we explore the effect of in Section 4.2 More formally, denoting the sketch as S and corresponding photo as P, we extract
= 5 feature maps, $(1), p() E Hix XCC for the I-th layer from the backbone VGG16 network and unit normalise each across the channel dimension. The loss is thus defined as,
(S, percceptual (S,P, w) = Σ
Σ h,w
|1 kw
-5()) kww
(4)
H.W
To extract the feature maps we choose max-pooling layers (relul_2, relu2_2, relu3_3 relu4_3 and relu5_3). During training, this perceptual loss is added to the game loss (Lgame). We note that the perceptual loss formulation is basically equivalent to the content loss in neural style transfer [14] Neural style transfer combines this content loss with style loss which encourages the texture statistics of generated raster image to match target image (which could be a sketch). Our model is different because instead of a loss encowraging a sketch-like style we directly impose production of sketches by drawing strokes.
the outputs of the VGG16 layers immediately before the
Experiments
We next present series of experiments where explore if it is possible for the two agents to learn to successfully communicate, and what factors affect human interpretation of the drawings. We report numerical results averaged across 10 seeds for models evaluated test sets isolated from training. Sample sketches from one seed are shown, but an overlay of 10 seeds can be found in Appendix
4.1 Can agents communicate by learning to draw?
We explore the game setups described in Section 3.1 and train our agents to play the games using 96 x 96 photographs from the STL-10 dataset For the original game we use 99 distractors. For the object-oriented games, due to the dataset only having 10 classes, we are limited to distractors.
In Table I we show quantitative and qualitative results of the visual communication game played under the three different configurations. The results demonstrate that it is possible for agents to successfully play this type of image referential game by learning to draw. One can observe that although agents achieve high communication success rate, using only the Igame loss leads to the

---

Table 1: Communication success rate and example sketches produced by the agents in order to achieve the game objective in various setups and with different losses. Sample input images seen by the sender (the left column) are described as the sketches the second and third column. Although successful communication seems to be achieved in all setups, the addition of the perceptual loss significantly improves human interpretability of the drawings. Examples are from STL-10.
game
game + perceptual
Original game
71.8% (+6.1)
69.57% (+2.6)
00-game same
95.46% (+0.6)
96.04% (+0.5)
00- 00-game different
82.72% (=0.8)
81.09% (+0.6)
emergence of communication protocol that is indecipherable human. However, the addition of the perceptual loss, motivated in Section 3.4| significantly improves the interpretability of the communication channel at almost no cost in the actual communication success rate.
One interesting observation is that although the sketches for some of the classes have greatly improved when incorporating the perceptual loss, for photographs of animals birds, the sketches not particularly representative the class instance or distinguishable for the human eyc. In the following sections explore the model to try to better understand what factors affect drawing production.
4.2
What effect does weighting the perceptual loss have the sketches?
Next, we explore the effect of manually weighting the perceptual loss. More precisely, we look at what happens when the perceptual loss is applied to the features maps from just one layer of the backbone network. As previously mentioned in Section 3.2 the feature maps are extracted using a VGGI6 CNN up to ReLU5_3 layer. For example, we can discard all feature maps except those from the first layer by weighting the perceptual loss by [1, 0, O, 0, Oj. The effect of the different weights, which allow only one block of feature maps be used for drawing the sketch, illustrated in Table We apply these constraints in two setups, the original and the 00-game different. In both cases, the drawings are unrecognisable if the perceptual loss takes into account only the first or the second block of feature maps. Blocks 3 through 5 seem to provide increasing structure under both game setups. It is worth noticing that, similar the results shown Section 4.1 the communication success rate the original setup is always lower than that from the 00-game different setup. Overall, the information provided by individual layers in the visual extractor network enough for the agents to develop a visual communication strategy that can be used to play the game. For humans, however, the later layers contribute the most to the emergence of communication protocol that we can understand.
4.3 Does the 00-game influence the sketches to be more recognisable as the type of object?
Comparing the qualitative results of different game formats from Table I we notice that agents develop distinct strategies for representing the target photograph under different conditions. If there is more variability in the sketches that correspond to photographs from the same class in the original game setup, and a bit less in the 00-game same, the sketches become more like symbols representing class when playing 00-game different. In other words, the object-
all the photographs from oriented games influence the sketches to be more recognisable as the type of object, than the specific instance of the class. Further examples are shown in Appendix
Finally, it is worth noting how our results connect to how humans communicate through sketching when constrained under similar settings. The far/close contexts used in [8] are somewhat equivalent
7

---

Table 2: The effect of weighting the perceptual loss such that only the feature maps from one backbone layer are used. The features extracted in the last three layers of the visual system seem to capture information that leads to sketches which resemble to an extent the corresponding photograph.
Loss weights
1,,0,0,00 0,
00,1,0,0,00 0,
[0, 0, 1, 0,
[0, 0, 0, 1,
[0, 0, 0, 0,
Orig. game
68.4% (13.6)
69.6% (12.2)
71.1% (1.24)
76.4% (12.1)
60.5% (14.8)
00-gama diff
81.9% (11.2)
81.5% (10.9)
82.3% (10.9)
82.5% (10.5)
81.4% (10.8)
Table The effect of the model's capacity on its sketches. The wide model's sender encodes the photo into 1024-dimensional vector (baseline 64), and the receiver's MLP linear layers have 1024
neurons each versus 64. Examples from training on Caltech- 101 in the 00-game different setting.
Baseline
Wide
50.46% (+1.5)
64.99% 1+15
to original/objectt settings. As Fan al. g observe when humans play similar drawing game, our agents achieve a higher recognition accuracy in settings that involve targets from different classes and develop different communication behaviours based the context of the receiver.
4.4 How does the model's capacity influence the visual communication channel?
Regarding the model's architecture, we look into how drawings are influenced by the width of the model. In this experiment (results shown in 3) we compare the baseline model architecture detailed in Section 3.2 with a wider variant that has the following changes: the sender encodes the target photograph to a dime-simallo vector (baseline model encodes to (4-diiensional vector), the receiver's 's ML.P capacity also increased from 64 to 1024 in both layers. We present results for the 00-game different setup played with 128 x 128 Caltech- 101 images [TO]. The increased number classes Caltech- 101 may explain the drop in the communication rate this particular game setting, which compared to the same model played under the original game setup (see the MaageNet-pretrainee model in Table is with almost 30% lower. As one might expect, the wider model allows for more details to be captured, and, hence, conveyed in the sketches. Unlike the baseline model which, in this object-oriented setup, develops a communication system that is more representative to the class than the instance (as discussed in Section 4.3), the wider model starts to draw distinctive representations for objects of the same type. More sketches can be found in Appendix G where one observe the difference between all images with chairs, for example.
4.5 How does the texture/shape bias of the visual system alter communication?
Next, show that texture shape bias of the visual system influences visual communication. This experiment was run under the original game setup with 128 x 128 Caltech- 101 images [10]. The results shown in Table Rable&[sugger suggest that inducing a "shape bias" into the model does not significantly improve the agent's 's performance in playing the game, but produces more meaningful drawings. By using the VGG16 weights pretrained on Stylized-Image-Ne also becomes more faithful to the actual shape of the objects. A shape-based sketch is much more interpretable humans, as has been known for a long time that shape is the most important cue for human object recognition (27] Further results from this experiment can be found in Appendix H
(15), the communication protocol

---

Table 4: The effect on the communication protocol of using a VGG16 feature extractor network pretrained on datasets that have texture (ImageNet) or shape (Stylized-ImageNet (15)) bias. Examples are from agents trained using the original game with Caltech- 101 data. The shape-biased sketches are better at capturing the overall object form, particularly for things like faces.
ImageNet weights
Stylized-Imagel weights
78.46% 46% (+2.0)
77.09% (+1.9)
ImtgeNet weights
(98.9%)
Stylized weights
(99.6%)
Figure 3: Sketches from original variant games using the CelebA dataset with perceptual loss and different biases from backbone weights. Both the texture-biasse (ImageNet) and shape-biased settings exhibit near-perfect communication success, but the shape- biased sketches are considerably more interpretable and show visual variations correlated with the photos.
4.6 Do the models learn to pick out salient features?
From the results we have presented so far, it is evident that, particularly with the perceptual loss, the sender agent is able broadly draw pictures of particular classes of object. The high communication rates in the original game setting would also suggest that the drawings can capture something specific about the target images that allow them to be identified amongst the distractors. To further analyse what is being captured by the models we train the agents in the original game setting (using both normal and stylized backbone weights) with images from the CelebA dataset (32), which we take the maximal square centre-crop and resize to 1 12px. As this dataset contains only images of faces, messages between the agents will have to capture much more subtle information to distinguish the target from the distractors. Results are shown in Figure 3 the communication rate is near perfect for both models, but the difference between the texture-blases and chape-biased models is striking. There is subtle variation in the texture biased model's 's sketches which broadly seems to capture head pose, but the overall sketch structure similar. In the shape- biased model head pose evident, but so are other salient features like hairstyle and (see Appendix[[ [
head-wear wear and glasses.
4.7 Do agents learn to draw in fashion that humans can interpret?
In order to assess the interpretability of sketches drawn by artificial agents, we set up a pilot study in which a 'sender' agent, pretrained in five different game configurations on STL10, is paired up with human 'receiver' to play the visual communication game. For this pilot study, we collect results from 6 human participants. Each participant played a total of 150 games, i.e. had to select the target image for each of the 150 sketches drawn by pretrained sender. Depending on the game setting, the list of options differs, but it is composed of distractors and the true target image. The experimental setup is detailed in Appendix K Table Fable53com compares the averaged human gameplay success to that of a trained 'receiver" agent. The results show that the addition of the perceptual loss leads to statistically significant improvement of humans' ability to recognise the identity of sketches. For the original game setting, played in this study with K- = 9 distractors which might be of the same category as

---

Table 5: Human Evaluation results, no learning allowed. Trained agents communicate success- fully between themselves in all settings. Addition of the perceptual loss allows humans to achieve significantly better than random performance (images from STI.-10, original games have distrac- tors/game for these experiments & random chance is 10%). In addition, humans are better at guessing the correct image class when the models are trained with the additional perceptual loss.
Agent
Human
Human class comm. rate
Game
Loss
Lines
comm. rate
comm. rate
original
1 game
20
100%
8.3% (+5.4)
15.0% (+2.5)
original
gamee + perceppual
20
93.3%
38.3% (+2.5)
55.6% (+7.1)
original
[ game
peroeptual
50
100%
37.2% (=5.9)
47.8% (#7.4)
oo diff
(game perceptual
20
83.3%
23.9% (=6.2)
23.9% (+6.2)
00 diff
1 - ggame perosptual
50
90.0%
38.9% (19.9)
38.9% (19.9)
the target, we also assess the ability of participants to recognise the class of the sketch. The human class communication rate shows that humans are better at determining the class of the sketch rather than the specific instance, even in the case of sketches generated with the game loss only, In the appendices, we extend the discussion of these results and look into whether communication with an agent can be improved the human participants are allowed to learn via feedback.
5 Conclusions and Future Work
We have demonstrated that it is possible to develop and study an emergent communication system between agents where the communication channel is visual. Further, we have shown that a simple addition to the loss function (that is motivated by biological observations) can be used to produce messages between the agents that are directly interpretable by humans.
The immediate next steps in this line of work are quite clear. It is evident from our experiments that the incorporation of the perceptual loss dramatically helps produce more interpretable images. One big question to explore in the future is to what extent this is influenced by the original training are these drawings produced
biases of the backbone network the ImageNet training data, or are they in some way more generic than that? We plan to address this by exploring what happens if the weights of the backbone are replaced with ones learned through a self- self-supervised learning approach like Barlow twins [44]. We would also like to explore what happens if the agents' visual systems had independent weights.
result of the original labels of
Going further, as previously mentioned, learning a perceptual loss would be a good direction to explore, but perhaps this should also be coupled with a top-down attention mechanism based on the latent representation of the input. An open question from doing this would be to ask if this allows for richer variation in drawing, and for features be exaggerated in the case of a caricature. Such an extension could also be coupled with a much richer approach to drawing, with variable numbers of strokes, which are not necessarily constrained to being straight lines. Coupling feedback or attention into the drawing mechanism itself could also prove to be a worthy endeavour.
We hope that this work lays the groundwork for more study in this space. Fundamentally our desire is that it provides the foundations for exploring how different types of drawing and communication from primitive drawings through to pictograms, ideograms and ultimately writing emerges between artificial agents under differing environmental and internal constraints and pressures. Unlike other work that 'generates' images, we explicitly focus on learning to capture intent in our drawings. We recognise however that our work may have broader implications beyond just understanding how communication evolves. Could for example in the future we see sketching agent replace a trained illustrator? The creation of messages for communication inherently involves elements of individual creative expression and adaption to the emotive environment of both the sender and receiver of the message. Our current models are clearly incapable of this, but such innovations will happen in the future. When they do we need to be prepared for the surrounding ethical debate and discussions about what constitutes 'art'. This has already been seen in the domain of robot art in which Pix 18 K1 is trailblazer as it not only a robot that paints oil on canvas but can also conceive its own art subject with minimal human intervention.
10

---

Acknowledgments and Disclosure of Funding
D.M. supported by the EPSRC Doctoral Training Partnership (EP/RS13325/1), J.H. received funding from the EPSRC Centre for Spatial Computational Learning (EP/S030069/I). The authors acknowledge the use the IRIDIS High- high-Perffrmance Computing Facility, the ECS Alpha Cluster, and associated support services at the University of Southampton in the completion of this work.
References
[1] Jaume Amores. Multiple instance classification: Review, taxonomy and comparative study. Artificial intelligence, 201:81-105, 2013.
[2] Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In International Conference on Learning Representations, 2018. URL htttps:/ hat.aa/r...tttoourrrro-aaaaaaaa
[3] Rahma Chaabouni, Eugene Kharitonov,, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding emergent communication. CoRR, abs/1905, 12561, 2019. http:://t///xii org/ htg/abs/1990 12561
[4] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. arXiv preprint arXiv:2004.091 09/ /24, 2020.
[5] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[6] Abhishck Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. In Proceedings of the IEEE international conference on computer vision, pages 2951 1951-2960, 2017.
[7] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. International Conference on Machine Learning, pages 1538 1546. PMLR, 2019.
[8] Judith E Fan, Robert Hawkins, Mike Wu, and Noah D Goodman. Pragmatic inference and visual abstraction enable contextual flexibility during visual communication. Computational Brain & Behavior, 301) 2020.
[9] Masoumeh Farokhi and Masoud Hashemi. The analysis of children's drawings: social, emo- tional, physical, and psychological aspects. Procedia-Social and Behavioral Sciences, 30: 2219-2224, 2011.
[10] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[11] Chrisantha Fernando, Daria Zenkova, Stanislav Nikolov, and Simon Osindero. From language games to drawing games. arXiv preprint arXiv:2010.028200 2020.
[12] Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. CoRR, abs/1605.06676, 2016. URI, htt.hhttp:/arxiiv.rrgge abs/ 1605.0600000 006666660666666
[13] Douglas C Fox. Prehistoric rock pictures in europe and africa. The Bulletin of the Museum of Modern Art, 4(5):388, 1937.
[14] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolu- tional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2414-2423, 2016. doi: 10.1109/CVPRR.2066.26.
11

---

[15] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. taagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 181 122331, 2018.
[16]
Ignace Gelb, A study of writing. University of Chicago Press, 1963.
[17] Shangmin Guo. Emergence of numeric concepts in multi-agent autonomous communication. arXiv preprint arXi::I911.000 2019.
[18] Shangmin Guo, Yi Ren, Agnieszka Stowik, and Kory Mathewson. Inductive bias and language expressivity in emergent communication. arXiv preprint arXiv:2012.02885. 2020.
[19] Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 2149-2101 2159. Curran Associates, Inc., 2017.
[20] Christopher S Henshilwood and Benoît Dubreuil. Reading the artefacts: gleaning language skills from the middle stone age in southern africa. The cradle of language, 2:61-99 12.6119,220 2009.
[21] Dirk L Hoffmann, Christopher D Standish, Marcos Gareía-Diez, Paul B Pettitt, James A Milton, João Zilhão, Javier Aloolzz-Gonzátez, Pedro Cantalejo-Duarte, Hipólito Collado, Rodrigo De Balbín, et al. U-th dating of carbonate crusts reveals neandertal origin of iberian cave art. Science, 359(6378)992 915, 2018.
[22] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference Machine Learning, pages 3040-3049. PMLR, 2019.
[23] Emilio Jorge, Mikael Kågebäck, and Emil Gustavsson. Learning to play guess who? and inventing a grounded language as a consequence. CoRR, abs/161 2016. URLhttpp
7/arxiv. wwwggaab///11i. 03218
[24] Rhoda Kellogg. Analyzing children's art. McGraw-Hill Humanities, Social Sciences & World Languages, 1969.
[25] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In International Conference on Machine Learning, pages 5220-5230. PMLR, 2020.
[26] Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Kotturmlb17. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9- 9-1 2017, pages 0662-2967,11 2017.
[27] Barbara Landau, Linda B Smith, and Susan S Jones. The importance of shape in early lexical learning. Cognitive development, 3(3):299-321, 1988.
[28] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations, 2017.
[29] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls. and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Conference on Learning Representations, 2018. URL httptpp://ppeeeeveee..eee.
id=HJGv1Z-AW
[30] David K. Lewis. Convention: A Philosophical Study. Wiley-Blackwell, 1969.
[31] Hod Lipson. Pix18, 2016. http
[32] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
12

---

[33] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. arXiv preprint arXiv:1903.05/000 2019.
[34] Daniela Mihai and Jonathon S. Hare. Differentiable drawing and sketching. CoRR, abs/2103.16194, 2021. URL https://axxiii..../. httpv/..rrggab///1//.// 16194
[35] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. CoRR, abs/1703.04908, 2017. URL http://arxiv.org// arg/////////3 04908
[36] Martin A Nowak and David C Krakauer. The evolution of language. Proceedings of the National Academy of Sciences, 96(14):802) -8033,33333 1999.
[37] Yi Ren, Shangmin Guo, Matthieu Labeau, Shay Cohen, and Simon Kirby. Compositional languages emerge in neural iterated learning model. arXiv preprint arXiv:2002.01365, 2020.
[38] Andrew Robinson. Lost languages. McGraw Hill New York, 2002.
[39] Bilge Sayim and Patrick Cavanagh. What line drawings reveal about the visual brain. Frontiers in Human Neuroscience, 5:118, 2011. ISSN 1662-5161. doi: 10.3389/fnhum.20 201 1.1.1.000 1.00118. URL
https://wwv http://wwwwwonttrrinnnnn owgg atg/articee/10 3ttp////.0.99ff/nmmmmm 2011.00118
[40] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[41] Luc Steels. The synthetic modeling of language origins. Evolution of communication, 1(1): 1-34, 1997.
[42] Dirk B. Walther, Barry Chai, Eamon Caddigan, Diane M. Beck, and Li Fei-Fei. Simple line drawings suffice for functional mri decoding of natural scene categories. Proceedings of the National Academy of Sciences, 108(23):9661 96666 2011. ISSN 0027-8424. doi: 10. 10.1073/pnas. 1015666108
htLLhhtts:/wwe pnas wwtp///ooee//////////oooo 1eatttent//18/22/66666
[43] Jason Weston and Christopher Watkins. Support vector machines for multi-class pattern recognition. In Proc of the 7th European Symposium On Artificial Neural Networks, pages 219-224, 01 1999.
[44] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.
[45] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We believe this be true.
(b) Did you describe the limitations of your work? [Yes] We covered existing limitations and provided ideas overcome them such in Section 3.2 or Section
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see
Section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
13

---

(a) Did include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes) Public github
URL included in Section 3
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Ye]] Please refer to Section 3.3
(c) Did you report error bars (c.g., with respect to the random seed after running ex- periments multiple times)? [No] The results focus on are qualitative (sketch interpretability), and for the quantitative results (game-play success), the agents always converge to the reported scores.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Please Section 3.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes) See Section
(b) Did you mention the license of the assets? [Yes] Where applicable; for example our code (see question 3a) contains the 3-clause BSD license as part of the bundle.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] See answer to question 3a, above, the code included in the supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you's re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "gYFwAmbak4n", "reviewer": "Reviewer_maSs", "summary": "This paper is an exploration into getting agents to communicate through drawing. It's in the multi-agent communication literature and proposes having the agents play the traditional referential game, however the sender agent is going to draw a sketch (with differentiable rasterization) instead of communicating through a symbolic channel. The receiver then has to say which of the target images the sketch aligns with.\n\nThat agents can play this well when gradients connect them is unsurprising as we expect the result to be uninterpretable. The authors go on to show how to yield interpretable images by adding a perceptual loss based on pretrained VGG, either by ImageNet or StyleNet.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality: The paper is written well and gets to the heart of the matter.\n- It sets out to show that a thing can be done and does the thing. Great.\n- Fig 3 is actually what I was looking for for much of the paper and the fact that hte drawings in both the ImageNet and the StyleNet rows are barely differentiable to me suggests that the model is likely using just a couple placements / strokes to define large fetures. For example, in the last two columns I don't think the face parts of the drawings are any different but the hair parts are, and that roughly makes sense. This gives me three ideas that I would like to see explored, only one of which I think would be worth adding to this paper:\n1. (Add to this if you can) Go run some human tests! See if humans can figure out what's going on in all of these experiments. Pair the agent with a human and repeat it a lot of times. Adding those scores in would be fantastic because that's what we're all after (and you too re L67-68).\n2. Try to figure out what the strokes mean. I get that that's not what you're going for in this paper, but I think that that would be really insightful. I think you can do this by running causal interventions on the strokes, perhaps with a language model to learn the distribution first. CLIP would count.\n3. Speaking of CLIP, try using that instead of the VGG. It's open source and been trained largely for this sort of task. It would very likely improve the results a lot.\n\nOriginality: Originality is there afaik.\n- I've been in this subfield for a while and I recall people saying they were going to do this something along these lines for much of the past half decade, and then nothing (afaik) was ever done. So I'm glad to see that this direction is being explored because I think that it's fruitful to consider drawing as the communication protocol; it's much simpler and more interpretable than all of these questions that get tangled up with language.\n- If though a similar paper was surfaced, then my score would likely drop. A lot of the experience of this paper is seeing someone finally do this (and it's done well).\n- With respect to the last few paragraphs, I suggest looking into Pix18 (http://www.pix18.com/). It is a robot that generates art and its works have entered the public domain for sale. It also was rejected from an art competition (after first being accepted) on the basis that what it makes is not art. You should cite it as its a trailblazer in this domain.\n\nClarity: Overall the paper was clear.\n- I do have one request though and that's to say earlier what is the actual task. It wasn't clear until page 5, Sec 3.2.2, that the receiver agent is going to get all of the real images and have to align the sketch with them to predict the maximal scoring image. This was confusing for much of the paper - just put it earlier and you'll help the reader a lot.\n\nSignificance:\n- This isn't a world-bending paper, nor is it going to come across as profound to the general audience. However, it does have the potential to kick this subfield out of a rut and into a more compelling place involving drawing and pictorial approaches.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Somewhat. It's a small paragraph at the end and not its own section. Also, they really should cite Pix18 as it deserves that at the least.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ll7oP6k8mLp", "reviewer": "Reviewer_pCdM", "summary": "This paper is casting the symbolic communication Lewis game into a drawing communication Lewis game.\nTo do so, they use a VGG network followed by a MLP/rastering function for drawing. \nThey show that, with a proper loss, the speaker agent may successfully communicate concepts with basic drawing.\n\nWhile this idea is definitely not novel [1], and it has been in the air again recently [2], the quality of this work is pretty high. It was both an enjoyable and insightful reading. There is a clear effort to mix quantitative and qualitative results (with sketches). Besides, the authors did not stop their research after a few positive results, and have excellent additional experiences in section 4. Nonetheless, the paper still has some minor weaknesses that I am happy to discuss further (cf Main results).\n\n[1] Bartlett, Frederic Charles, and Frederic C. Bartlett. Remembering: A study in experimental and social psychology. Cambridge University Press, 1932.\n[2] Fernando, Chrisantha, et al. \"From Language Games to Drawing Games.\" arXiv preprint arXiv:2010.02820 (2020).  to-cite!", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Key remarks:\n\nA) The perceptual loss is one of the key components of the paper. However, I am wondering what is the impact of $L$ on the training. If $L$ is greater/lower, does it reduce accuracy? Interpretability? More generally, I am missing a comprehensive ablation on the paper core elements num_strokes (20+), $L$, $\\lambda$ where $l = l_{game} + \\lambda l_{perceptual}$ in the appendix would make the current conclusion stronger, and help reproducibilty. Besides, in light on Tab1 Appendix A, the task accuracy seems quite noisy. Std are definitly missing in the paper. How many seeds did you use? Furthermore, having the overlay of num_seeds=10 sketches would be excellent!\n\nB) Regarding the perceptual loss, I would recommend adding a small paragraph to link it to other work that explores low/high feature levels, e.g., [1] or [2] or, more generally, to style transfer [3]. One may argue that the loss is a naive implementation of style transfer.\n\nC) What are the intuitions of using a VGG network instead of ResNet? Using Hinge-loss instead of InfoNCE loss? cf. Tab2 in [4]? Funnily, the paper is using Deep Learning techniques from 2015. Note that it does not undermine the quality of the paper.\n\nD) As one of the major claims of the paper is communication with humans, I would have significantly increased my score to have a human evaluation (i.e. the score of humans playing this game). \n\nE) As mentioned, the paper has many complementary experiments. Yet, I think it is also missing what I call counter-factual experiments, i.e., experiments that try to break the setting to challenge the paper hypothesis further. Those are potential ideas:\n - how important is the Rasteriser (i.e. sketch) -> You can simply extract a cloud of points (which still encode an image)\n - what happens when you inject Out-of-Distribution images?  In Figure 3, put animals instead of human faces? Or similar to [6], put random noise images.\n\nOther remarks:\n - I am not fully convinced by the origins of writing history. As it is quite a detail in the introduction, I would recommend supporting it with more than 1-2 papers if it is important for the authors. \n - Beware of the citations; they are surprisingly centered around one research group/company. Even if it does not break anonymity rules, I would recommend being more inclusive.\n- Object-Oriented game variant. Instance classification within weakly supervised learning [7 or other paper] and/or [5] as the (first?) example in the emergent communication literature.\n- As future work idea: adding extra geometric options while drawing images. (e.g. output [x, y, line/rect/circle, red/blue/green] etc.). If you do so, organize a small expo at Neurips next year :) \n\n\nI made several remarks, and I am happy to increase my score further if they are addressed.\n\n\n[1] Castrejon, Lluis, et al. \"Learning aligned cross-modal representations from weakly aligned data.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n\n[2] De Vries, Harm, et al. \"Modulating early visual processing by language.\" arXiv preprint arXiv:1707.00683 (2017).\n\n[3] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. \"Image style transfer using convolutional neural networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n\n[4] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" International conference on machine learning. PMLR, 2020.\n\n[5] Lazaridou, Angeliki, Alexander Peysakhovich, and Marco Baroni. \"Multi-agent cooperation and the emergence of (natural) language.\" arXiv preprint arXiv:1612.07182 (2016).\n\n[6] Bouchacourt, Diane, and Marco Baroni. \"How agents see things: On visual representations in an emergent language game.\" arXiv preprint arXiv:1808.10696 (2018).\n\n[7] Amores, Jaume. \"Multiple instance classification: Review, taxonomy and comparative study.\" Artificial intelligence 201 (2013): 81-1\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "cf above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "Ds9geWqK5Dt", "reviewer": "Reviewer_9t7y", "summary": "I’m impressed by the authors’ inclusion of a human evaluation. While they paint a somewhat unclear story about human interpretability (hard to tell how good 38% is), including the results in commendable and these kinds of studies are sorely missing from the literature.\n\nI’ve increased my score accordingly. I’m borderline on increasing my score even further: I think I would have liked to see some of the other experiments I suggested in the initial review to really nail down the interpretability idea more completely. But still, this is a very solid paper now—kudos to the authors!\n\n---\n\nThis paper is a study of emergent communication in an image-based referential game where the communication channel between two agents is not discrete sequences of tokens, but instead *sketches* of images; in other words, the agents learn to communicate via drawing. The speaker uses a recently-proposed differentiable drawing procedure, where line segments generated by an agent are combined and converted to a grayscale image via a relaxed rasterization process. The listener then receives the rasterized drawing and needs to identify the speaker's image from a set of distractor images. The entire procedure is end-to-end differentiable.\n\nOptimizing solely communication success unsurprisingly results in non-interpretable sketching. To encourage the communication to be more interpretable, authors propose adding an auxiliary perceptual loss, where the speaker's drawings are optimized not just for listener success, but also trained to be perceptually similar to the target image, by minimizing distance between their representations according to the speaker's trained vision module. This results in sketches that often capture the rough details of the target image, with little-to-no degradation in communication success.\n\nThe authors try several additional experiments, e.g. varying the width of the communication channel, using different pretrained backbones that are more biased towards shape, etc. Many of these results do seem to have some effect, but the comparisons between models are limited to a qualitative comparison between 5 example images, so the conclusions we can gain from these experiments are a bit limited.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\nOverall this is a well-written paper with an interesting and extensible idea at its core; modeling sketch-based communication has several potential downstream applications for multi-agent systems and should be of interest to those interested in simulations of the development of communication systems.\n\n## Weaknesses\n\nMy main concern, which makes me overall on the fence with this paper, is that the claim that the agents learn to draw \"in a fashion humans can interpret\" is a very subtle one and may not be adequately discussed or supported in the paper.\n\nOne consensus of the existing literature is that agents often learn to exploit inscrutable, unintuitive patterns in the either the input or the communication signal; this is doubly true when the communication protocol is continuous (as is the case here). For example, a small shift in a single pixel value in a speaker's drawing could drastically change the listener's prediction.\n\nThe perceptual loss constrains the drawings to \"look like\" the target image, but this doesn't guarantee that the agents interpret the drawings as humans would. For example, the agents could be ignore the \"gestalt\" of the sketch, and instead learn that differences in the mean pixel density of some region of the sketch indicate the mean RGB value of some region of the target image. Such degenerate schemes might be successfully learned even under the constraints of the perceptual loss.\n\nOne suspicion I have is that communication accuracy would remain successful even if the perceptual loss was some meaningless, arbitrary objective: for example, constrain all sketches to look like a single image (of a dog, perhaps). Since the communication is continuous, and agents are able to pick up on small deviations in the input, they may still be able to play this referential game with a wide variety of images.\n\nEven ignoring these questions, it's insufficient to examine solely the target image and the corresponding sketch in isolation, see some broad similarities, and then declare the communication \"human interpretable.\" A more pragmatic claim of \"human interpretability\" would show that a human could successfully play the role of the listener and select the right image based on the speaker's sketches. There is no such evaluation in the paper; at the very least it would be good to see some examples of the full reference games (w/ 99 distractors), so we can see ourselves how \"human interpretable\" the communication is *in the full context given to the listener* (which may contain many perceptually similar distractors, etc). Take Figure 3 for example: even with the stylized image weights, for many of the sketches it's doubtful that humans would be able to correctly identify the target among all images in this figure—even if there are some high-level correspondences between images and sketches. If the agents are able to construct drawings that succeed in reference games where humans would completely fail, is the communication really human-interpretable? \n\nTo be fair, these are tough, somewhat philosophical questions. But I would encourage the authors to add more concrete discussion of what it precisely means to be \"human interpretable\". Finally, to summarize some concrete additions to the paper that would shed light on this ideas:\n\n1. Does it matter what the perceptual loss is? What if we constrain the images under some arbitrary, meaningless objective?\n2. Human evaluation - are sketches \"interpretable\" to a human insofar as a human can play the reference game with the images? Alternatively, consider whether a human could generate sketches for a trained listener?\n3. To what extent does the learned communication protocol generalize to other speakers and listeners? If a speaker can play the game with a new listener, this would support the idea that speaker-listener pairs are not devising their own specific and uninterpretable drawing schemes, but instead converging on some shared communication that indeed focuses on learning a general, human-like interpretation of generated sketches.\n\n## Other questions\n\n- How different is OO-game-same from the original setting where distractors are randomly sampled? It seems like most distractors are probably already in separate categories in the first place?\n\n## Missing citations\n\nSome neural models of sketching have been developed in the cognitive science literature, e.g.:\n\"Pragmatic inference and Visual Abstraction Enable Contextual Flexibility during Visual Communication.\" Judith Fan, Robert X.D. Hawkins, Mike Wu, Noah Goodman. COBB, 2019\n\n## Minor\n\n- L44 \"Maximize gameplay alone\" → \"maximize gameplay success/communication success?\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, there's a good discussion of future downstream impacts of this work in the discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a fun paper! The basic ideas of communication by sketching, emergent communication, and indeed emergent communication via sketches have been floating around recently, but the focus on interpretability of the emergent sketches is a valuable addition. As the reviewers indicate the addition of human evaluations is very nice, and some space should be given in the revision to interpreting these new results.

One comment from me: I would love to see a little discussion of the connections to human communication with sketching, eg "Pragmatic Inference and Visual Abstraction Enable Contextual Flexibility During Visual Communication." J. E. Fan, R. D. Hawkins, M. Wu, & N. D. Goodman. (2019). Computational Brain & Behavior.
Decision: Accept (Oral)