Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a self-supervised learning objective using a contrastive loss to learn a medium/language for communication between agents to complete a task (in a decentralized training setting). Similar to Lin et al. (2021), the messages here are proposed to be encodings of observations. To learn these encodings/ messages, the contrastive objective uses positive anchors sampled from the current trajectory around a window and negative samples from other messages in the batch. The authors test their method (CACL) on three MARL tasks: Predator-Prey, Find-Goal, and Traffic Junction. Through different analyses for probing representations and testing for zero-shot coordination, the authors present their method's effectiveness.
strength_and_weaknesses: ### Strengths:

- The authors provide a thorough analysis of their proposed method through various embedding probing techniques:
    - qualitative plots for different scenarios,
    - quantitative measures of similarities for messages,
    - transfer of message encodings to auxiliary tasks
- The authors also do a great job of comparing their method to well-known techniques in the literature and show how their approach is better than the others for a wide range of measures
- The section on zero-shot crossplay and the associated analysis was quite interesting to read!
- **************************Clarity:************************** The paper, the methods, the metrics were quite easy to read and understand.

### Weaknesses:

- I think the paper's main weakness lies in justifying the approach's motivation for the relevant problems. I found the introduction and discussion relatively weak in their description of the broader motivations of the work, particularly those relevant to emergent communication: applications/ transfers to the real world and understanding human communication.
- More importantly, similar to Lin et al. (2021), it wasn't clear if this was actually communication; sending the complete representation of the observation without accounting for the other agent as there is no bottleneck. A potential baseline that could test this could use a contrastive loss to learn observation embeddings (w/o multiagent dynamics), freeze that and train agents to communicate by using these. It would help us understand how the setting of sending messages and communicating changes representations.
- **************************Novelty:************************** The approach essentially combines Lin et al. (2021) and with a contrastive info-NCE objective to learn representations.

### Questions:

- In the crossplay setting (Table 4), the interplay agent outperforms the intraplay agent for CACL-AEComm and CACL-PL. Why do the authors think that the interplay agent outperforms agents specifically trained to communicate with each other?
- Is there a systematic split between the training tasks for the agents and the evaluation ones?
clarity,_quality,_novelty_and_reproducibility: Detailed in the main review.
summary_of_the_review: The paper is thorough in comparing with baselines and probing to understand the representations learned by the proposed method. The paper lacks a proper discussion/justification for the setting, the impact of the method used, its implications for different tasks or the emergence of language.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In the paper, the authors propose to communicate between different agents using contrastive learning in a decentralized setting. 

As their main contribution of making agents communicate based on contractive learning for multi-agent reinforcement learning in the decentralized setting, they evaluate the proposed model in different environments and obtain promising results. 
strength_and_weaknesses:  The paper proposes a communication method based on supervised contractive learning with sufficient experiments to evaluate the proposed method.

The weakness of this paper:
1. The problem investigation in Section III is confusing, especially in the usage of symbols with lower and upper case, such as s and S for real states. 
2. Meanwhile, is it right for the transition function P->SxAxS?
3. Some citations are in the wrong formation, such as the caption of Fig. 3 for DBSCAN and t-SNE.
4. Compared with the workshop paper “LEARNING TO GROUND DECENTRALIZED MULTIAGENT COMMUNICATION WITH CONTRASTIVE LEARNING”,  the current version is updated less in the method.
clarity,_quality,_novelty_and_reproducibility: There is still a lot of room for improvement in the clarity of the paper.

It is an exciting job to make agents communicate using contrastive learning in a decentralized setting. While compared with the workshop paper “LEARNING TO GROUND DECENTRALIZED MULTIAGENT COMMUNICATION WITH CONTRASTIVE LEARNING”,  the current version is updated less in the method.

As for Reproducibility,  I believe that it is easy to reproduce the results of the paper reported for the reasons that all technologies adopted in this paper are all used in other related works. 
summary_of_the_review: It is an exciting job to make agents communicate using contrastive learning in a decentralized setting. However, the current version is not suitable for ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: this paper proposes to develop linguistic communication between decentralized agents via multiple view contractive learning. which is an interesting problem. it is suggested to combine the contractive loss with the RL loss with a predefined hyperparameter. 
strength_and_weaknesses: Strength:
The problem of linguistic communication between decentralized agents is important 

Weaknesses:
1.  "Linguistics studies language and its structure, including morphology, syntax, phonetics, and semantics". I do not see any of these in the paper, even though "linguistic communication", linguistics and similar terms are used to describe the advantages of this paper and novelty. There is not language and not linguistics, but the standard multiply view constrictive learning of observations.

2. contrastive (and multi-view) losses can be useful in many scenarios. this paper shows one more such scenario. I am not sure this is enough for a full paper in ICLR. In my opinion the contribution to the knowledge and understanding of linguistic communication for decentralized RL is minor. 

3. The current comparison between different approaches is indirect evidence only, because a) there can be different number of parameters in DNNs, (which is not mentioned), b) with more powerful harsher one can achieve better results (more training..). 

4. there are no conclusions but rather a summary of the paper. 



a few minor comments regarding the phrasing:

1. "..but many works focus on simple, non-MDP environments." what simple non-MDP environments do you mean? 
2. "...requires fewer assumptions about other agents..." which assumptions about other are assumed in centralized and decentralized setting? that will help to understand "..fewer..". It is desired to provide evidence.
clarity,_quality,_novelty_and_reproducibility: the use of the terms linguistics/language/communication is not clear.
there are numerous works on the combination between contrastive loss and RL loss.
the comparison to previous works is implicit evidence only - because the works from 5-6 ago 
utilized simpler / smaller architectures and  less computational resources. 


summary_of_the_review: there is not linguistics/linguistic-based communication. the objective is standard. there is no general  principles/conclusions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes using contrastive learning as a mechanism to encourage agents to effectively communicate with each other in a multi-agent setting. This is achieved by encoding an agents observations into the communication channel, but also maximizing the mutual information between messages within an episode. This simple yet effective idea leads to more effective and symmetric communication as has bene demonstrated quantitatively. The authors also demonstrate a slightly surprising result, i.e. zero-shot communication between separately trained agents.

Three authors reviewed this paper and provided reject ratings. After reading the paper myself, reading the reviews and author rebuttal carefully. I disagree with the reviewers in their recommendation to reject the paper.

Reviewer xPcm has two strong concerns:
(1) Positioning of the work (but has failed to point out specific arguments or lines in the paper that they disagree with) --
In response, the authors have provided a detailed rebuttal regarding the positioning of this work, which I believe addresses this concern.

(2) Novelty of the work --
I agree with the authors take on this. The proposed CACL method isnt just novel by itself, but importantly it connects two different sets of problems that people have been working on -- multi agent communication and contrastive learning for representation learning. I think this connection (which now feels natural after reading this paper) is quite interesting and opens up this area for further research. This connection is novel in my opinion, is simple and has been shown to be effective.

Reviewer dj4f has provided one strong concern that I discuss below. Other concerns are minor formatting concerns and in my opinion, should not be grounds for such a strong negative rating. They have also been addressed in the rebuttal.
(1) Previous workshop paper.
The ICLR guidelines state that: "Submissions that are identical (or substantially similar) to versions that have been previously published, or accepted for publication, or that have been submitted in parallel to this or other conferences, journals or archival workshops, are not allowed and violate our dual submission policy."
The workshop that the reviewer is referring to is non-archival. Hence as per ICLR policy, I am disregarding this workshop submission and making my recommendation purely on the merits of the submitted draft.
I consider this concern resolved.

Reviewer Tpss has a few concerns such as a lack of a conclusion (which is certainly not grounds for rejection). The two strong concerns in my opinion are:

(1) Contrastive learning + X is now well studied --
As the authors have stated and I agree, this does not devalue the contributions of this paper. In particular, the use of contrastive learning to improve multi-agent communication is new and effective and an important contribution to this field.
(2) They are opposed to the use of the term "linguistic communication" --
The authors have noted the context of this use, and have updated the paper to make this clear. This is 

In summary, the strong reject ratings by the three reviewers do not seem justified. Several concerns are minor ones that have been easily fixed in the revised draft and many have been addressed in the rebuttal. These more superficial concerns are not grounds for rejecting the paper. One main concern is about the novelty of the paper, which I have addressed above and where I agree with the authors. And one more strong concern is about an overlapping submission at a workshop which I also consider resolved.

However, we are currently also missing sufficient agreement amongst domain experts that the paper is ready for acceptance. After detailed discussion between PCs, SAC, and AC, Program Chairs indicated that available feedback and consensus isn’t sufficient for overruling strong signals from reviewers (all strong reject). As such, the paper can not be considered for acceptance in the program of ICLR 2023.

We strongly encourage the authors to revise (e.g., with respect to clarity, as two reviewers pointed out, as well as insights into the proposed method) and resubmit their manuscript as it provides promising contributions for the community
Decision: Reject