Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper is an exploration into getting agents to communicate through drawing. It's in the multi-agent communication literature and proposes having the agents play the traditional referential game, however the sender agent is going to draw a sketch (with differentiable rasterization) instead of communicating through a symbolic channel. The receiver then has to say which of the target images the sketch aligns with.

That agents can play this well when gradients connect them is unsurprising as we expect the result to be uninterpretable. The authors go on to show how to yield interpretable images by adding a perceptual loss based on pretrained VGG, either by ImageNet or StyleNet.
main_review: Quality: The paper is written well and gets to the heart of the matter.
- It sets out to show that a thing can be done and does the thing. Great.
- Fig 3 is actually what I was looking for for much of the paper and the fact that hte drawings in both the ImageNet and the StyleNet rows are barely differentiable to me suggests that the model is likely using just a couple placements / strokes to define large fetures. For example, in the last two columns I don't think the face parts of the drawings are any different but the hair parts are, and that roughly makes sense. This gives me three ideas that I would like to see explored, only one of which I think would be worth adding to this paper:
1. (Add to this if you can) Go run some human tests! See if humans can figure out what's going on in all of these experiments. Pair the agent with a human and repeat it a lot of times. Adding those scores in would be fantastic because that's what we're all after (and you too re L67-68).
2. Try to figure out what the strokes mean. I get that that's not what you're going for in this paper, but I think that that would be really insightful. I think you can do this by running causal interventions on the strokes, perhaps with a language model to learn the distribution first. CLIP would count.
3. Speaking of CLIP, try using that instead of the VGG. It's open source and been trained largely for this sort of task. It would very likely improve the results a lot.

Originality: Originality is there afaik.
- I've been in this subfield for a while and I recall people saying they were going to do this something along these lines for much of the past half decade, and then nothing (afaik) was ever done. So I'm glad to see that this direction is being explored because I think that it's fruitful to consider drawing as the communication protocol; it's much simpler and more interpretable than all of these questions that get tangled up with language.
- If though a similar paper was surfaced, then my score would likely drop. A lot of the experience of this paper is seeing someone finally do this (and it's done well).
- With respect to the last few paragraphs, I suggest looking into Pix18 (http://www.pix18.com/). It is a robot that generates art and its works have entered the public domain for sale. It also was rejected from an art competition (after first being accepted) on the basis that what it makes is not art. You should cite it as its a trailblazer in this domain.

Clarity: Overall the paper was clear.
- I do have one request though and that's to say earlier what is the actual task. It wasn't clear until page 5, Sec 3.2.2, that the receiver agent is going to get all of the real images and have to align the sketch with them to predict the maximal scoring image. This was confusing for much of the paper - just put it earlier and you'll help the reader a lot.

Significance:
- This isn't a world-bending paper, nor is it going to come across as profound to the general audience. However, it does have the potential to kick this subfield out of a rut and into a more compelling place involving drawing and pictorial approaches.
limitations_and_societal_impact: Somewhat. It's a small paragraph at the end and not its own section. Also, they really should cite Pix18 as it deserves that at the least.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper is casting the symbolic communication Lewis game into a drawing communication Lewis game.
To do so, they use a VGG network followed by a MLP/rastering function for drawing. 
They show that, with a proper loss, the speaker agent may successfully communicate concepts with basic drawing.

While this idea is definitely not novel [1], and it has been in the air again recently [2], the quality of this work is pretty high. It was both an enjoyable and insightful reading. There is a clear effort to mix quantitative and qualitative results (with sketches). Besides, the authors did not stop their research after a few positive results, and have excellent additional experiences in section 4. Nonetheless, the paper still has some minor weaknesses that I am happy to discuss further (cf Main results).

[1] Bartlett, Frederic Charles, and Frederic C. Bartlett. Remembering: A study in experimental and social psychology. Cambridge University Press, 1932.
[2] Fernando, Chrisantha, et al. "From Language Games to Drawing Games." arXiv preprint arXiv:2010.02820 (2020).  to-cite!
main_review: Key remarks:

A) The perceptual loss is one of the key components of the paper. However, I am wondering what is the impact of $L$ on the training. If $L$ is greater/lower, does it reduce accuracy? Interpretability? More generally, I am missing a comprehensive ablation on the paper core elements num_strokes (20+), $L$, $\lambda$ where $l = l_{game} + \lambda l_{perceptual}$ in the appendix would make the current conclusion stronger, and help reproducibilty. Besides, in light on Tab1 Appendix A, the task accuracy seems quite noisy. Std are definitly missing in the paper. How many seeds did you use? Furthermore, having the overlay of num_seeds=10 sketches would be excellent!

B) Regarding the perceptual loss, I would recommend adding a small paragraph to link it to other work that explores low/high feature levels, e.g., [1] or [2] or, more generally, to style transfer [3]. One may argue that the loss is a naive implementation of style transfer.

C) What are the intuitions of using a VGG network instead of ResNet? Using Hinge-loss instead of InfoNCE loss? cf. Tab2 in [4]? Funnily, the paper is using Deep Learning techniques from 2015. Note that it does not undermine the quality of the paper.

D) As one of the major claims of the paper is communication with humans, I would have significantly increased my score to have a human evaluation (i.e. the score of humans playing this game). 

E) As mentioned, the paper has many complementary experiments. Yet, I think it is also missing what I call counter-factual experiments, i.e., experiments that try to break the setting to challenge the paper hypothesis further. Those are potential ideas:
 - how important is the Rasteriser (i.e. sketch) -> You can simply extract a cloud of points (which still encode an image)
 - what happens when you inject Out-of-Distribution images?  In Figure 3, put animals instead of human faces? Or similar to [6], put random noise images.

Other remarks:
 - I am not fully convinced by the origins of writing history. As it is quite a detail in the introduction, I would recommend supporting it with more than 1-2 papers if it is important for the authors. 
 - Beware of the citations; they are surprisingly centered around one research group/company. Even if it does not break anonymity rules, I would recommend being more inclusive.
- Object-Oriented game variant. Instance classification within weakly supervised learning [7 or other paper] and/or [5] as the (first?) example in the emergent communication literature.
- As future work idea: adding extra geometric options while drawing images. (e.g. output [x, y, line/rect/circle, red/blue/green] etc.). If you do so, organize a small expo at Neurips next year :) 


I made several remarks, and I am happy to increase my score further if they are addressed.


[1] Castrejon, Lluis, et al. "Learning aligned cross-modal representations from weakly aligned data." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[2] De Vries, Harm, et al. "Modulating early visual processing by language." arXiv preprint arXiv:1707.00683 (2017).

[3] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. "Image style transfer using convolutional neural networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[4] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[5] Lazaridou, Angeliki, Alexander Peysakhovich, and Marco Baroni. "Multi-agent cooperation and the emergence of (natural) language." arXiv preprint arXiv:1612.07182 (2016).

[6] Bouchacourt, Diane, and Marco Baroni. "How agents see things: On visual representations in an emergent language game." arXiv preprint arXiv:1808.10696 (2018).

[7] Amores, Jaume. "Multiple instance classification: Review, taxonomy and comparative study." Artificial intelligence 201 (2013): 81-1


limitations_and_societal_impact: cf above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: I’m impressed by the authors’ inclusion of a human evaluation. While they paint a somewhat unclear story about human interpretability (hard to tell how good 38% is), including the results in commendable and these kinds of studies are sorely missing from the literature.

I’ve increased my score accordingly. I’m borderline on increasing my score even further: I think I would have liked to see some of the other experiments I suggested in the initial review to really nail down the interpretability idea more completely. But still, this is a very solid paper now—kudos to the authors!

---

This paper is a study of emergent communication in an image-based referential game where the communication channel between two agents is not discrete sequences of tokens, but instead *sketches* of images; in other words, the agents learn to communicate via drawing. The speaker uses a recently-proposed differentiable drawing procedure, where line segments generated by an agent are combined and converted to a grayscale image via a relaxed rasterization process. The listener then receives the rasterized drawing and needs to identify the speaker's image from a set of distractor images. The entire procedure is end-to-end differentiable.

Optimizing solely communication success unsurprisingly results in non-interpretable sketching. To encourage the communication to be more interpretable, authors propose adding an auxiliary perceptual loss, where the speaker's drawings are optimized not just for listener success, but also trained to be perceptually similar to the target image, by minimizing distance between their representations according to the speaker's trained vision module. This results in sketches that often capture the rough details of the target image, with little-to-no degradation in communication success.

The authors try several additional experiments, e.g. varying the width of the communication channel, using different pretrained backbones that are more biased towards shape, etc. Many of these results do seem to have some effect, but the comparisons between models are limited to a qualitative comparison between 5 example images, so the conclusions we can gain from these experiments are a bit limited.
main_review: ## Strengths

Overall this is a well-written paper with an interesting and extensible idea at its core; modeling sketch-based communication has several potential downstream applications for multi-agent systems and should be of interest to those interested in simulations of the development of communication systems.

## Weaknesses

My main concern, which makes me overall on the fence with this paper, is that the claim that the agents learn to draw "in a fashion humans can interpret" is a very subtle one and may not be adequately discussed or supported in the paper.

One consensus of the existing literature is that agents often learn to exploit inscrutable, unintuitive patterns in the either the input or the communication signal; this is doubly true when the communication protocol is continuous (as is the case here). For example, a small shift in a single pixel value in a speaker's drawing could drastically change the listener's prediction.

The perceptual loss constrains the drawings to "look like" the target image, but this doesn't guarantee that the agents interpret the drawings as humans would. For example, the agents could be ignore the "gestalt" of the sketch, and instead learn that differences in the mean pixel density of some region of the sketch indicate the mean RGB value of some region of the target image. Such degenerate schemes might be successfully learned even under the constraints of the perceptual loss.

One suspicion I have is that communication accuracy would remain successful even if the perceptual loss was some meaningless, arbitrary objective: for example, constrain all sketches to look like a single image (of a dog, perhaps). Since the communication is continuous, and agents are able to pick up on small deviations in the input, they may still be able to play this referential game with a wide variety of images.

Even ignoring these questions, it's insufficient to examine solely the target image and the corresponding sketch in isolation, see some broad similarities, and then declare the communication "human interpretable." A more pragmatic claim of "human interpretability" would show that a human could successfully play the role of the listener and select the right image based on the speaker's sketches. There is no such evaluation in the paper; at the very least it would be good to see some examples of the full reference games (w/ 99 distractors), so we can see ourselves how "human interpretable" the communication is *in the full context given to the listener* (which may contain many perceptually similar distractors, etc). Take Figure 3 for example: even with the stylized image weights, for many of the sketches it's doubtful that humans would be able to correctly identify the target among all images in this figure—even if there are some high-level correspondences between images and sketches. If the agents are able to construct drawings that succeed in reference games where humans would completely fail, is the communication really human-interpretable? 

To be fair, these are tough, somewhat philosophical questions. But I would encourage the authors to add more concrete discussion of what it precisely means to be "human interpretable". Finally, to summarize some concrete additions to the paper that would shed light on this ideas:

1. Does it matter what the perceptual loss is? What if we constrain the images under some arbitrary, meaningless objective?
2. Human evaluation - are sketches "interpretable" to a human insofar as a human can play the reference game with the images? Alternatively, consider whether a human could generate sketches for a trained listener?
3. To what extent does the learned communication protocol generalize to other speakers and listeners? If a speaker can play the game with a new listener, this would support the idea that speaker-listener pairs are not devising their own specific and uninterpretable drawing schemes, but instead converging on some shared communication that indeed focuses on learning a general, human-like interpretation of generated sketches.

## Other questions

- How different is OO-game-same from the original setting where distractors are randomly sampled? It seems like most distractors are probably already in separate categories in the first place?

## Missing citations

Some neural models of sketching have been developed in the cognitive science literature, e.g.:
"Pragmatic inference and Visual Abstraction Enable Contextual Flexibility during Visual Communication." Judith Fan, Robert X.D. Hawkins, Mike Wu, Noah Goodman. COBB, 2019

## Minor

- L44 "Maximize gameplay alone" → "maximize gameplay success/communication success?"
limitations_and_societal_impact: Yes, there's a good discussion of future downstream impacts of this work in the discussion.
review_title: null

Meta-review: This is a fun paper! The basic ideas of communication by sketching, emergent communication, and indeed emergent communication via sketches have been floating around recently, but the focus on interpretability of the emergent sketches is a valuable addition. As the reviewers indicate the addition of human evaluations is very nice, and some space should be given in the revision to interpreting these new results.

One comment from me: I would love to see a little discussion of the connections to human communication with sketching, eg "Pragmatic Inference and Visual Abstraction Enable Contextual Flexibility During Visual Communication." J. E. Fan, R. D. Hawkins, M. Wu, & N. D. Goodman. (2019). Computational Brain & Behavior.
Decision: Accept (Oral)