Venue: NeurIPS 2021 Workshop SVRHM

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
# Quality & Clarity: 5

This work investigate several inductive biases in the context of communication games involving visual interpretability. 

In the introduction and background section, the authors describe previous computational approaches clearly, establishing the lineage of the current work under Mihai & Hare 2021. 

However, on line `24` the introduction fails to clearly state the primary contribution of this work, instead opting for deflationary rhetoric of
> In this paper we explore Mihai & Hare (2021b)’s model in more detail.
Without a clear statement of purpose (i.e. "Here, we explore the interaction of more "human-like" perceptual representations in communication with...") the remainder of this paragraph appears to be arbitrary compositions of other works in the field of communications and computer vision. 

In the background, starting on line 40, the authors establish a link between human vision's ability to represent scenes at different levels of representation. However, this link was disorganized. Instead of enumerating cited works at the end of the paragraph, the manuscript's clarity would have been dramatically improved if this sections began with human vision as an inspiration and then referenced the relevant communication research. 

In the model section, lines 57-60 mention that the communication via drawing task that the model is trained to perform "seems impossible for humans" without any citation or further explanation. Humans already enjoy several board games that rely on drawing real-world objects. Without a stronger qualification than intuition this comment is distracting at best or wrong at worst. It is unclear how framing the relevant task as super-human benefits the work. 

# Originality NA

This reviewer is not familiar with the literature on communication and does not have a comment for originality. 

# Significance 5

Understanding and implementing the computational principles that allow humans to effectively communicate under both processing and pragmatic constraints is a central aspect to human intelligence. 

This work aims at establishing a set of working components: 1) a visual system that can effectively represent the world in a way that 2) summarizes key features that 3) can be queried meaningfully. 

From this manuscript, 1 & 3 are given partial service but the degree to which these goals are accomplished are obfuscated by the format of Table 1. 

It is unclear that this work achieves goal 2. In Fig 2, the authors illustrate the qualitative interpretability of model sketches from a variety of images. It is  clear that adding additional inductive biases in the form of perceptual losses leads to differences across models where those that include such losses have recognizable drawings (especially row 3: ViT-B/32 + clipdraw). However, as the authors mention in future directions, it is unclear whether the model is drawing the scene or collapsing to the scene category. The authors only show two scenes that are close in category (horse, horse + jockey) and their resulting sketches do not appear to reflect any interpetable difference; both, sketches appear to be a horse. The additional sketches on page 9 do no alleviate this concern, with objects in several sketches oriented in the reverse direction than those in the input image. Given that this work relies on qualitative comparisons to support the conclusion of the title, it would be prudent to organize such figures by category such that this could be easily illustrated to readers. 


# Pro / Cons

## Pros

- important premise: Human vision can represent key aspects of a scene succinctly 
- exploration of novel model by composing CLIP and prompt engineering
 
## Cons
- paper itself suffers from lack of clarity and structure
- it is unclear that the authors meet their objective of qualitative interpretability based on the Fig 2
- it is unclear why the different approaches in the work lead to differences in performance in terms of interpretability. 
review_title: Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?

final_score: rating: 7: Good paper, accept
review_content: Summary: \
While prior work has shown that it is possible to train a pair of neural networks to come to common ground in a sketching communication game, the sketches the sketcher network makes are meaningful to the receiver network but not to humans. This paper explores how representational biases imposed on such networks can make the sketches more ‘human-like’.
The first contribution is extending prior work by Mihai & Hare by using the recently released CLIP architecture. The sketcher network is forced to make its representations more image like due to the imposition of a ‘perceptual loss’ between the network’s feature representations for the sketch and the target image. They also compute a loss between features from the final layer of the sketcher networks for the target image and transformed versions of the sketch created by the sender. This too increased perceptual similarity between the sketch and target.

The authors also offer an interesting prompt engineering based approach to explore if the models can match text prompts to the actual image shown.
It isn’t surprising that the models can discriminate between the sketch and image modalities due to their very different pixel distributions. However, this result continues to show that the ‘clipdraw’ loss is important for aligning text-based descriptions.

Comments: \
This paper raises some very interesting points and raises important questions about the tradeoffs between perceptual fidelity and communicative efficiency in communication games. However, many of the experiments and architectural elements of the models in this paper refer to the model presented by Mihai & Hare in prior work. Even Figure 1. appears to be a slight modification of a figure from said paper. Without any information about how the models ‘play’ the communication game and adequate motivation as to why this is a sensible way to model visual communication it is difficult to evaluate parts of the paper. I am left with questions such as – “Does the sketcher and receiver model have the same encoder weights?”, “Is the receiver’s guess chosen based solely on the softmaxed dot-product between sketch and image features?”, “How many items of each class are present on each trial? (I think the text implies 10 of each, but it is not explicitly stated)”.
I would recommend shortening the introduction to add some details about Mihai & Hare’s model or to flesh out the Model Setup and Training Details section in the appendix. 
Additionally, some commentary on why the clipdraw loss seems to be better than the perceptual loss would be nice to include.

Finally, it would be interesting to follow up with some human experiments. There are many papers that might prove to be helpful references for doing so. Here are some to consider for future work – \
RD Hawkins, M Sano, ND Goodman, J Fan (under review). Visual resemblance and communicative context constrain the emergence of graphical conventions. - https://cogtoolslab.github.io/pdf/hawkins_arxiv_2021.pdf

RD Hawkins, M Kwon, D Sadigh, ND Goodman (2020). Continual adaptation for efficient machine communication. - https://www.aclweb.org/anthology/2020.conll-1.33/

JE Fan, RD Hawkins, M Wu, ND Goodman (2020). Pragmatic inference and visual abstraction enable contextual flexibility during visual communication. - https://rdcu.be/bQqxr 


Rationale for Score: \
The importance of making the outputs of artificial communicative agents more human-like is a promising research program and the results would be of general interest to the community. The paper could do with a few additional details about the base model and the motivations behind the perceptual loss functions, but these can be addressed fairly easily in the camera-ready version. Overall a nice paper.

review_title: Good paper, some edits needed

final_score: rating: 7: Good paper, accept
review_content: This paper investigates how different loss functions affect the output of networks playing a sketch communication game. The ideas are of interest to the SVRHM community and I believe the paper should be accepted.

My primary comment is that the paper would be significantly strengthened by including human evaluations of the understandability of sketches. The paper makes various assertions about how understandable the sketches are for humans, and while the example sketches are certainly interesting, I would not be surprised if human accuracy in labelling the sketches across a wide set of stimuli is actually not particularly high. Some of the sketches in Figures 2, 3 and 4 are introspectively difficult to interpret without also having seen the target image.

A similar assertion that could be tempered is on lines 58--60. We don't know how hard this game would be for humans. For example, humans can remember specific object images with surprisingly high accuracy (https://www.pnas.org/content/105/38/14325). 

## Minor comments

- Introduction would benefit from an additional opening sentence setting the problem into context. Why study this?
- line 79: it's a bit confusing to formulate the loss with a weight that is then not used (all set to one).
- line 95: the game objective is not explicitly defined. Is it accuracy? likelihood?
- Figure 2: The examples show that it's not simply a perceptual vs game loss: VGG16 with the perceptual loss also fails to produce human-interpretable sketches. This should be more prominently discussed.
- The expected communication rate (line 114) is not really meaningful for sketches, in my opinion. The baseline in a verbal game would be 10%, since there it's possible to only communicate the object label (e.g. "dog") but in human sketches, I find it difficult to imagine communicating the concept of "dog" without also including additional information (e.g. pose and viewpoint), which would increase discriminability of the candidate image set (i.e. raise the baseline).
- Table 1: This table is confusing to interpret and could be better explained in text (lines 118--132). For example, what does it mean that the c(guess)==gt(input) rates are so high (particularly compared to the c(sketch)==c(target) for VGG with perceptual losses)? Does this mean that labels based on sketches are significantly worse than guessing in this setting? If so, this needs further explanation.
review_title: Ideas of interest, some confusing details

final_score: rating: 7: Good paper, accept
review_content: Review of Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent? For SVRHM 2021

In this paper, the authors explore different improvements to the interpretability of drawings produced by agents in a game of pictorial communication. In this game, one agent had to draw an image and share it with a second agent that needed to guess what it was from a limited pool (that also contained the first agent’s source image). Performance with artificial agents can be very good, but the images produced by the drawing agent appear random to humans. In this paper, the authors attempt to make those images more interpretable by using different loss functions designed to include inductive biases that are receptive to humans. They also compare two different network architectures.

I found the paper well written, interesting, and appropriate in scope for SVRHM. I recommend it for the workshop.

Some suggestions to consider:

- My main complaint is that there are many claims throughout the paper about interpretability with no human data to support those claims. Wherever the authors have said that the drawings are more interpretable, we are relying on their intuition. E.g. pg 3 line 105. For what it’s worth, I mostly agree with their characterization, but it is an anecdote/speculation. Consider that ViT-B with the new loss functions are both described as more interpretable -- what if we were to find that these images do not actually help a human identify the sender’s source image? Or that they do so with different levels of efficacy? Would we still say that they are both interpretable? The claims in this paper could be significantly strengthened by validating the method with human data. A simple online task where naive raters attempt to pair the source images with the agent’s drawings in different conditions. This human performance could then be compared to the agent performance, or against chance (performance expected by random guessing). The results of this validation task could be displayed with Figure 2.
- I think VGG16 with l(perceptual) is not very interpretable, and I bet if we conduct the human validation study described above, it would not match human performance. Moreover, VGG16 + l(clipdraw) seems to converge on only two different sketch forms (Figure 4).
- I think it might be helpful to adapt FIgure 1 to illustrate where the new inductive biases occur
- I had a hard time understanding the message of section 4 and the associated results in Table 1. I understand the notion of prompt engineering and the goal of the exercise, but expressing the results in terms of c(I)==etc… without a header or guide to interpreting those results made it difficult. 

review_title: Review of Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent? For SVRHM 2021

Decision: Accept (Poster)