Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The current work proposes to study a signaling game between two agents that uses image patches instead of the entire image. Within this framework, experiments show that the agents select important patches from the image and the emergent language captures semantic information about the patches. Further, the resultant agents are used as self-supervised image classifiers and compared against existing patch-based/self-supervisory methods to show superior performance.
main_review: **Strengths**  
(S1) Leveraging mid-level semantic patches for referential games and using the resultant agent for downstream image classification task (without pretrained features) is a novel direction. 

(S2) The manuscript is well-written and is easy to read and understand. In particular, it states the hypothesis clearly and places it in the context of prior work. 

(S3) The authors do a good job with the experiments supporting the claims in the earlier part of the manuscript. The qualitative examples of heat maps and semantic grouping of emergent language throw more light on the proposed idea, and are useful.

**Weaknesses**  
(W1) The standard deviation for performances is missing for Table 1 and 2. Given that some of these numbers are close, it is not possible to deduce if the improvements are statistically significant. 

**Comments**  
(C1) How generalizable are the features/images if images beyond ImageNet are used, for example MSCOCO or OpenImages?

(C2) L239-240: Is there a way to quantify how much overlap exists between important patches and the dominant objects for images in the dataset? For example, using a subset of images and computing overlap with segmentation masks for either PASCAL or other datasets with segmentation annotations?

(C3) Would a pair of agents participating in a referential game (but without patches) trained from scratch converge? If yes, what would their downstream performance on image classification be?

**Typos**  
L239: shows -> show

limitations_and_societal_impact: Authors discuss the limitations and broader impact of their work in the manuscript.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present PatchGame, "a referential game formulation where given an image, the speaker sends discrete symbolic signals in terms of mid-level patches, and the listener embeds these symbols to match them with another view of the same image in the presence of distractors."
main_review: The novelty of the work is high, and as claimed it is the first work to develop an emergent communication via mid-level patches without using any pre-trained models or external labels. While I enjoyed reading the draft (the presentation is clear), I have a few questions and also comments:

1) From reading the game setting, and its root in emergent communication formation, I have the feeling that the speaker and listener model is essentially trained to encode in the speaker view image into a minimum set of patch-based representations that the listener could re-identify the image similar to the input image from a set of images. Instead of reconstructing the same image back in an encoder-decoder architecture, the game here is trying to encode the input image into a sequence of patches (with length constraint) to help another agent to identify from a set.  Is it essentially a special case of an encoder-decoder system? If not, why?

2) Have you considered and/or observed the speaker and listener learned to bypass the "important" image patch but learned to utilize artifacts (maybe a logo in the input image, and also present in the target image from the set of candidate images) to encode the message? According to the game setting, the optimized message would be the artifact, then the model won't be able to learn to rank "important" image patches right? In other words, since it is an unsupervised game setting, how to prevent the agents from "cheating" to succeed? 

3) It is a comment. Have you considered utilizing this game setting to formulate a novel evaluation mechanism for visual/representation learning, rather than the debatable downstream task classification accuracy-based ones? I feel this might be the more exciting usage of your proposed game. 

Other places need clarification:

Line 332 means we our gain is minimal. -> delete "we"
limitations_and_societal_impact: The broader impact section refers to "A recent study [75] shows that 997 out of 1000 ImageNet categories are not ‘people’ categories."
This sentence reads out of context, could you clarify the purpose of stating this here?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a visual referential game played by a pair of agents. Compared to existing work in this space, the game is a variant of those studied previously as the communication is based directly on encoded patches (rather than a latent vector describing the whole image), and the agent models diverge from prior work in their architectural construction.   

main_review: Originality
-----------

Unfortunately, one of the main claims to originality the paper makes is that prior work in emergence of language using visual referential games all looks at models with pretrained networks for visual feature extraction rather than models in which this inductive bias is removed. There was a paper at the NeurIPS 2019 Emergent Communication workshop that explored exactly this aspect and started to draw out the links between self-supervised learning (with augmentations) and emergent communication: 
    
    Avoiding hashing and encouraging visual semantics in referential emergent language games. Daniela Mihai and Jonathon Hare. 3rd NeurIPS Workshop on Emergent Communication. 2019. https://arxiv.org/abs/1911.05546
    
A quick search on arxiv suggests the same authors might also have followed this up (https://arxiv.org/abs/2101.10253) with a longer paper.

With this in mind, there are still some bits of the proposed model which are potentially interesting - particularly the game setup and the agent architecture. Overall, I think the proposed setup is an interesting variation of what has been done before, although I'd have liked to have seen stronger links made to prior work incorporating visual attention --- for example attending to different parts of an image to generate discrete messages is in essence what the Show, Attend & Tell model (https://arxiv.org/pdf/1502.03044.pdf) does (albeit in a fully supervised setting). I believe the approach to generating variable length message sequences is novel, although I do have questions (below).


Quality
-------

Technically the proposed model and accompanying quantitive evaluation results appear generally sound. As a reader I would have appreciated more to be said about the variance of the model from differing initializations (particularly in the experiments in section 4.6) --- it is clear that the model is very sensitive to learning rate, so one also wonders how much variance from random initializations there is in e.g. Fig 6b and 6c where the model performance under differing vocabulary and message lengths is _very_ close (is it so close that it's within the margin of error?). 

I do wonder if the authors could further justify their choices of metrics for measuring performance of the model (or perhaps clarify their intent in creating the model); more specifically, in the context of emergent communication it seems strange that one would consider a bag-of-words model or a truncated-message model as a good proxy for comparing positive signaling as they do not capture the interesting parts of language that one might wish to observe the emergence of (such as evidence of compositionally, etc). Perhaps the aim was to just explore models with discrete latent spaces (and perhaps an implicit 'importance' ordering on tokens)? If so, such a criticism is obviously mute, but in such a case would it then not make sense to focus more attention on exactly what makes the proposed model perform better than e.g. VQ-VAE or PatchVAE?

Regarding PatchRank, it's not actually clear to me that the proposed approach does really control that message length much. Surely in expectation the length of every message is just ~K/2 as (approximately because of the soft-rank) half of the values in r(x) would be >K/2? Besides the potential for some small information leakage from the relaxation in the soft-rank algorithm I can't see that the model really has much ability to learn appropriate message lengths. Do you have any quantitive evidence (e.g. mean message length and its variance over many games) that the model really is actually learning variable length better (e.g. with higher variability, not necessarily shorter than the maximum) than previous models?


Clarity
-------

Overall I found the text to be easy to follow, however I think many of the points I've raised above hint to ways to improve the content and clarity.

### Minor points
- \theta_{rank} in Eqn. 8 is inconsistent with the rest of the text (and Fig. 2) 


Significance
------------

Commenting on potential significance is always challenging; in this case it's not immediately obvious to me that there is that much new given the points raised above, however this could change if my queries were addressed.  


Rationale for rating
----------------------
Overall my opinion is that there is little novel in the paper in its current form, although there are some potentially interesting aspects which I've tried to highlight in my review. My score reflects this. I've tried to give constructive feedback around some of the more original parts of the paper and sincerely hope these will help the authors make improvements.   

limitations_and_societal_impact: Limitations are covered adequately. I am not convinced that the current Broader Impact section in the paper adds anything relevant to the discussion - my suggestion would be to rework this into something much more tangible about how people might benefit (or not) from potential future machinery that benefits from this work (for example, how an emergent communication system between agents might be used, or how variable-length discrete compressed representations could be utilised).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a referential game in which a speaker “sees” an image and sends a discrete message of variable length to a listener. The message is designed to represent important patches in the input image. In addition to the speaker’s message, the listener is given a batch of images, one of which is an augmented view of the speaker’s input. The listener’s goal is to find this augmented view of the input image among all images in the batch. The speaker and listener are trained with self-supervision using InfoNCE loss. The learned communication protocol is evaluated on the underlying task as well as used as a pre-training step for the downstream task of classification.
main_review: ### Strengths

This work presents a novel referential game inspired by recent developments in self-supervised learning applied to computer vision in which two views of the same input image are used to learn a representation useful for downstream applications such as classification. To the best of my knowledge, the combination of a patch-based architecture of the speaker and a transformer architecture of the listener in addition to a discrete image representation is novel and well motivated. The authors discuss the computational complexity of the proposed model. The paper is written and organized clearly. It explores an interesting research direction at the intersection of self-supervised learning and referential games and favorable classification performance when compared to existing methods such as VQ-VAE-2 and PatchVAE.

### Room for improvement
While the authors do note that resource constraints have prevented them from reporting error bars in their experiments, I believe the work would benefit from including these to support the statistical significance of the results. For example, the plots in Fig 6 are difficult to interpret because there are no error bars present and all curves are very close to one another. Also, Fig 2a does not display the curves past epoch 20. 

### Questions
1. Have you explored what effect the batch size has on classification performance?
2. Just to clarify, is the x-axis of Fig 6b “Length of each patch” the number of symbols used per patch? 
3. What was the motivation behind setting $l $ (the number of symbols per patch) to 1? Is it the computational cost? 
4. What is meant by “meaningful representations” in line 107? Using autoencoders as a pre-training step has been shown useful for downstream tasks such as classification. 

### Nitpicks
- Fig 2’s caption: “ConvNet to compute weights...”
- Line 140: use either “encode” or “embed”, not both
- Line 239: “We also show some of the failure...”
- Line 247: “the performance of the Vision Transformers drops if...”
- Line 302: “an effective batch size of 512”
- Line 332: “means our gain is minimal” 
- Line 339: “from the Internet”

### During discussion
- After reading the authors' response to my comments and questions, I have decided to increase my score.
limitations_and_societal_impact: Yes, the limitations and societal impact (from the perspective of training data) are discussed in the paper. The authors might also consider mentioning any potential negative applications of the proposed model.
review_title: null

Meta-review: The author response addressed most of the concerns of the reviewers and all reviewers recommend to accept the paper after the author response and discussion.

I recommend accept with the expectation that the authors will revise the paper for the camera ready, addressing the reviewer's concerns according to the author response, including, but not limited to the following aspects:
1. Add the mean and standard error to Table 1, 2 and Figure 4, 6
2. Add results on Pascal VOC
3. Add an extra ablation study on augmentations and batch sizes
4. Add ablation study for Mihai et al. and discuss it in the related work
5. Move the discussion on sentence length from the appendix to the main paper
6. Fix typos and make other minor writing improvements
Decision: Accept (Poster)