Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors study the problem of emergent communication with sketching. In each game, the sender describes an image through predicting strokes sequentially, while the receiver picks the target image which belongs to the same visual category as the image seen by the sender from a pool of images. The sender is a pretrained stroke predictor and receiver decides whether to make a decision based on the distance between the embeddings of the current canvas and last canvas and the images in the pool. They are jointly  trained with reinforcement learning optimizing for the discounted guessing reward. Note that the canvas renderer is differentiable with respect to the strokes. According to the results, making early decisions is especially useful for the seen instances, since the receiver could learn to adapt to more abstract protocols for the instances it got trained on; the pretrained sender is already a good enough sender which often better than the sender trained in the game, while the latter learns to use less steps in the game; the sequential game is also very crucial since it makes the sender aware of the information gain of each timestep. The authors also evaluate the iconicity, symbolicity and semanticity of drawings. It is also observed in the case study that the sketch becomes simplified and the most silent part of the concept is always preserved. 

strengths_and_weaknesses: Strengths:

1. This paper studies this very interesting problem of emergent communications of drawing. Language is not the only modality humans have at disposal, and drawing is a more natural way to communicating concrete concepts. Learning to form abstractive conventions will hint us about the way communication protocols are evolved.
2. The technical method and evaluation is sound as a first step towards this question. 

Weaknesses:

1. The sender is pretrained and already performs well according to the comparison between sender-fixed and complete models (accuracy-wise). I think a more interesting setup would be training the sender from scratch with limited prior knowledge about the visual categories it needs to depict. This would give us more information about how well a sender can learn to draw with the communication goals. 
2. The definition of iconicity and the evaluation are misaligned. Please refer to the questions. 

questions: According to Section 5.2.2, "iconicity [are defined] as the drawings exhibiting high visual resemblance with the corresponding images", the evaluation I was expecting is on the perceived visual similarities between the drawing and the image. However, the authors merely compared accuracy on the unseen classes. As much as I agree that strong iconicity should implies good generalizability, the other direction does always hold. Can the author explain this mismatch a bit more? I think the authors should directly use generalizability as the property's name or provides human ratings for iconicity. Either way will make the paper much scientifically reliable.
limitations: The authors have not discussed any potential negative societal impacts. Since the paper uses human sketches as pre-training data, the data might contains the biases, which might be learned by the model. However, I think this is a minor point.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work is an attempt at mimicking communication processes in humans that predate language, namely the process of drawing sketches. Here, a sender tries to draw an image via minimalist sketches step by step in an attempt to communicate with a receiver. The work claims a new value function based MARL training objective and defines three aspects of communication via the sketches: iconicity, symbolicity, semanticity. Results align with the author's expectations of the evolution of emergenet communication.

strengths_and_weaknesses: *Strengths:*

- The paper is quite refreshing in its motivation: study emergent communication using sketches (as opposed to much of the current literature which uses discrete token based communication).

- This line of work could open up ways to study emergence of pictographic languages, a quite underexplored region of the emergent communication space.

- I quite like the introduced measures of iconicity, symbolicity, semanticity - their underlying definitions are exactly what is needed to measure effectiveness of the communication in such referential games.


*Weaknesses:*

- The clarity of the novelty surrounding the training objective via joint value function is unclear with respect to prior works.

- The results surrounding how the sketches evolve into semantically similar classes is interesting and would serve to validate the motivation of this paper, but is rather difficult claim to assess the veracity of. As it appears to be a side effect of the proposed training objective, should the take away be that this particular objective better models emergent communication through sketches in humans? 

- A summarized sentence on why the observation spaces between sender and receiver are disjoint (line 120) would be useful instead of just a citation. (The images themselves are distinct but can belong to the same partition of class is an additional missing clarification.)

- Fig 1 choices might not be the best with respect to conveying the goals of the work. Line 170-171 clearly say that the receiver only picks an image when they have info to distinguish from the distractors. The round 1 of Fig 1 already appears to have more than sufficient information to distinguish rooster from the other two, it is only when it is simplified in round 2 that there is ambiguity with the other bird. This calls into question how ''good`` the distrctor images are and whether most of the games can just be solved with maximal reward for both agents if sketching abilities were high within a single step.
    -  A side effect of this is that it is not clear whether the emergent communication happens because of the communication aspect or if it is simply dependent on the actual drawing abilities of the sender.

- The claims regarding symbolicity and semanticity seem viable to judge through automated methods as in the paper but I am not clear how this could work for iconicity. High visual resemblance to the images is derived from the perspective of an agent - either a human or one of the learning agents based on their training objective. Without such a reference to ground these numbers, once again it is difficult to assess the accuracy of the lines 359-361.


questions: How'd you end up with the learning rate schedule and other parameter that you did, they seem relatively non standard (perhaps the result of a grid search?)? - lines 224-236

limitations: Authors discussed limitations but limited themselves to the accuracy of the vision systems available (the need to convert from real to pixellated images, etc.). What could be better discussed as limitations is the ability to evaluate along the three dimensions proposed in the work without the aid of human annotators.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores the modality of sketches in the context of emergent communication games with visual referents. While other papers have already studied graphical channels as an alternative to discretised tokens explored in prior emergent communication research (e.g. ref 29 in the paper), the unique feature of this work is studying the evolution of these conventions in a multi-step game. The authors define three properties (iconicity, symbolicity and semanticity) to study the evolved sketches with quantitative and qualitative methods. Results show that the agents are able in most cases to learn successful communication channels. 
strengths_and_weaknesses: ## Strengths

- Studying emergent communication through the modality of sketches is an exciting direction, although this paper is not the first to explore it (paper ref 29, but also see Fernando et al, 2020, From language games to drawing games, arXiv preprint arXiv:2010.02820). The study of the *evolution process* of the emergent sketches between two cooperative artificial agents, however, is a fruitful initiative to be explored and the authors compare it to observations from human studies.
- Using the previous canvas as an external memory and comparing against it, among other distracting referents, to incentivise the sender to draw informative sketches as early on in the communication is a nice idea. 
- This work proposes three evaluation metrics to assess the evolved sketches; these are a valuable contribution for the field going forward.

## Weaknesses

- This work, in its current form, doesn't take into account the current state of emergent graphical communication and advances in differentiable drawing made in the last two years. These could have helped address some of the problems highlighted in this work (e.g. the sender first using a visual module that turns a natural image into a pixel sketch is something overcome by many recent works). 
- Building upon the above, the comparison against other similar models is somewhat lacking; for example the category oriented game setting is seemingly the same as the "object-oriented" setting explored by ref 29 (which was published at NeurIPS last year). 
- Due to the choice of sketching module, only 40 categories (with only 10 images per category) of Sketchy dataset are explored in this paper. Other works have considered much more diverse image sets and considerably harder games.
- Communication performance seems to be relatively low compared to other approaches given the simplicity of games with only two distractors/context images. It's not clear why this is.
- The new measures proposed are mostly only evaluated qualitatively.
- Human performance against the agents is not considered as it has been in related works.


# Other remarks

- L17 of the abstract "the unexplored modality of sketches" isn't really true. Sketching has already been looked at by other papers. It would be much better to refocus the paper on the contribution of this work in studying the evolution of graphical communication and providing new measures. Similar issues appear on L97, L103 and L366. 
- Section 2 "Learning to sketch" does not give an accurate picture of the state of the art in this space; there is considerable work in that goes well beyond "the early phase". For example:
    + A. Das et al, 2021. Cloud2curve: Generation and vectorization of parametric sketches. CVPR 2021
    + T.Z. Li et al. Differentiable Vector Graphics Rasterization for Editing and Learning. SIGGRAPH Asia 2020
    + D. Mihai and J. Hare, 2021. Differentiable Drawing and Sketching. arXiv:2103.16194
    + D. Smirnov et al, 2020. Deep parametric shape predictions using distance fields. CVPR 2020
    + H. Mo et al, 2021. General virtual sketching framework for vector line art. ACM Transaction on Graphics 2021
- Section 5.2.5 would benefit from expansion and more discussion; there are contributions here.
- Some of the refs need updating to the published papers rather than arxiv versions (e.g. 23, 29)
- Please define communication accuracy for the benefit of readers not familiar with emergent communication games

questions: - Can you clarify that M=3 (L231) you mean that there are only three context images? If so,
    - Why is this set so low? Other works with referential image games have used significantly larger distractor sets and still achieved good to near perfect communication (both with sketching and with token based messages). Further, the unseen images seem to behave significantly worse, whereas in many emergent communication studies performance on unseen test data is often in line with the training data performance - what causes this?
    - How is it that the communication accuracy is so low in some settings? In the one-step setting it would be significantly worse than random guessing; for many of the other "unseen class" settings it is barely better than random
- What happens when agents from pairs of different communication scenarios are tested (i.e. different image subsets)? Are the conventions emergent from disjoint subsets still holding up?
- The proposed measure of semanticity is only demonstrated qualitatively; could a quantative measure not be developed? It also would be interesting to understand how this relates to design choices in other works which explictly try to enforce this (e.g. the "perceptual" losses in 29 and also in Mihai, D and Hare, J, "Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?", SVRHM 2021)

limitations: As it stands whilst the authors try to draw attention to some of the shortcomings of their work, they do it out of context with respect to the published literature. As highlighted above in the the questions and the weaknesses there are some particular areas where limitations could be better drawn-out and discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors proposed to address the problem of visual communication with graphical sketches. By proposing a reinforcement learning framework, two agents (a sender and a receiver) play a visual communication game to reach an agreement on what the visual message is. There are three key factors used to provide some insights inside the visual communication. Experiments are conducted on a subset of the sketch-photo paired dataset, i.e., Sketchy, to validate the proposed method. Some interesting results are provided as well.
strengths_and_weaknesses: Strengths
- This works proposed to address a significant problem of emergent graphical conventions by designing a visual communication game using sketches, which is very interesting and inspiring.  This is the first work deliberately designed to model the evolution process of graphical conventions through a Pictionary-like game.
- The technical contributions are concrete. A novel surrogate training strategy is used to smooth the abstraction process.
- Iconicity, symbolicity and semanticity are proposed to evaluate the emergent graphical conventions, which are of great importance to enable quantitative measurements on visual communication.

Weaknesses
- The effectiveness of the communication process might be influenced by the quality of the target sketch $\hat{I_S}$ produced from image $I_S$.
- The data used for training and testing is very small, i.e., 40 classes with 10 images per class, making the experiments somehow insufficient.
questions: (1) Did the authors try to use alternative sketching methods to measure the impact of sketching quality? One possible candidate could be CLIPasso [A].

[A] CLIPasso: Semantically-Aware Object Sketching

(2) At each time step, five new strokes will be added on the previous canvas $C_{t-1}$. Is there an action to remove existing strokes? I personally think some previously drawn strokes will become redundancy when the sender evolves.

(3) Human study on sketch evolution would be helpful to confirm the quality of the generated sketch by the sender. 

(4) Is that the final sketching could be somehow matched to the most informative strokes to represent the object in the original natural image? This could be interesting to know. 
limitations: The authors have discussed the limitations in terms of two-stage pretrained senders.
review_title: null

Meta-review: metareview: This is a solid work, and all reviewers agree to accept the paper.
confidence: Certain
Decision: Accept