Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a general framework named truthful self-play (TSP) which is suitable for communicative partially-observable stochastic games and analytically demonstrates convergence to the global optimum.
strength_and_weaknesses: Strength
1. The proposed framework TSP is general and guarantees convergence to the global optimum theoretically and experimentally.

Weaknesses
1. This paper improves the self-play problem by introducing the 'truthful' reward. However, this paper lacks a full explanation of the meaning of 'truthful', in other words, we do not find relevant information about 'truthful' in the method section, and it is too farcical to use this reward as an inverse game mechanism. The contribution of this paper is too few and its innovation is general.
2. The experiment needs improvement. 
   1) In Table 1, the effect of PP-5 and TJ-5 is not significantly improved; compared with IC3Net+SP, the value including standard deviation has some coincidence(need more experiments iteration). In TJ-20, there is no reasonable explanation for the reduction of effect variance. 
   2) In Figure 2, the description of (a) is unclear, and the existence of (c) is unnecessary. 
   3) In Table 2, whether the CommNet+TSP's performance can be further improved should be verified. 
3. Spelling problems and unclear statements.
    1) Introduction para3 pertially -> partially
    2) The input of function $q_{\phi}$ is inconsistent, $\hat{h}_{t i}$ in Section3.2 and $\hat{h}_{t}$ in Section3.3. 
clarity,_quality,_novelty_and_reproducibility: This paper is original and reproducible, but the quality and clarity need to be improved. 
summary_of_the_review: 1. The contribution of this paper is favorable to the community .
2. The experiment needs improvement to be more convincing.
3. The language and symbolic descriptions in this paper need to be clearer to avoid misunderstandings.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Truth Self Play (TSP) is a method for improving SP in partially observable environments which involve communication. The motivation is well grounded adding additional reward term to evaluate the truthfulness of communication sent. This is implemented by the additional reward  p(a_j | z_ij), which is to say, force agent j to listen to the communication sent from agent i.  In a reverse way, as the listener has to care about the message, the speaker (as this is self-play) starts providing useful information.
strength_and_weaknesses: ## Strengths
* Strong theoretical grounding
* Experiments on lots of environments
* This is useful

## Weaknesses
* The math is kinda OOT in my opinion. Section 3 could be way shorter. The use of imaginary numbers is unnecesary - simple use R^2 and then apply a normalization over a single dimension.
* (Figure a) Only showing the Real part of the reward signal?  Can we see how the second reward signal changes over time?
* Experimental results are lacking. I appreciate the variety of environments applied but simply reporting numbers isn’t convincing. I’d like to understand how messages look with some visualisation and how they change over time.
* I’d also like to understand what optimal or desired behaviour looks like in these environments - can you get to welface 0 in PP or TJ?
* Please info the reader more of your baselines (CommNET and IC3Net)
* Why didn’t you run CommNet w TSP
* Why don’t you have MAPPO as a baseline?
* The proper scoring rules seems to be no different agent modelling in other agents. Could you add some of these into the related work section just to help differentiate (or explain in rebuttal if i’m being slow)

clarity,_quality,_novelty_and_reproducibility: Clarity:

Found the paper really unclear!
* Lots of confusing terms - is evolutionary learning any different to automatic curriculum? Just trying to understand the jargon here.
* Unclear what internal-state refers to in the introduction -> is this the internal beliefs of an agent?
* Why is this framed as faithful state representations - the question is about the truthfulness of the communication channel which is explicitly separate to the state space!

Novelty:
I can’t tell if this is Novel cause it just seems like opponent modelling and they haven’t explained why its different.

summary_of_the_review: The paper is to a good standard and with clarity could be a very high impact paper. Theoretical justifications are great, I would like more experimental analysis and clear terminology to understand implications.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper propose a method for multi-agent reinforcement learning in non-cooperative partially observable environments with communication. The proposed method, TSP, adds imaginary rewards using the peer prediction method by evaluating the validity of information exchanged between agents. TSP has guaranteed convergence to the global optimum and has good empirical performance. 

strength_and_weaknesses: Strength
- This work is novel in the sense that it is the ﬁrst attempt to apply mechanism design to multigent evolutionary learning. 
TSP’s convergence is theoretically guaranteed for arbitrary policies and environments.
- The method seems relatively easy to implement and numerical experiments show the effectiveness of TSP. 

Weakness 
- The authors could add a discussion on what causes self-play + curiosity’s bad performance. 
- The author should at least discuss the scalability since the experiments are on simpler and smaller environments. 
- The learning curves show that TSP is not as stable as baselines, how many random seeds are used in the evaluation? And what causes the instability issue? 

clarity,_quality,_novelty_and_reproducibility: Quality
- Novel idea that improves self-play through a simple modification. 
- Empirical results show the effectiveness. Experimental evaluations are extensive. 

Clarify
- Mostly clear. Claims are well supported. Some details missing that could be helpful for reproducibility

summary_of_the_review: This work considers multi-agent reinforcement learning in non-cooperative POMDP. The proposed method, TSP, generalizes self-play to non-cooperative partially observable environments via mechanism design. The method has a convergence guarantee and is shown to outperform state-of-the-art performance for various multi-agent tasks. 
However, there are some weaknesses that I hope the authors could address in the next version.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a framework for evolutionary learning to emerge an unbiased state representation without any supervision. This is relevant to multi-agent reinforcement learning in non-cooperative partially observable environmentsy. The proposed framework is a  modification of self-play that is inspired by mechanism design, so as to elicit truthful signals and make the agents cooperative. Experiments demonstrate that the proposed framework achieves state-of-the-art performance on a variety of tasks.

While there are aspects of this paper that could be improved, such as the legibility of the maths, I and the reviewers agree that it is a quality contribution. One comment mentions that the paper has been submitted and rejected previously, but this should not be held against it.
justification_for_why_not_higher_score: The generality of the work is not strong enough for a spotlight.

justification_for_why_not_lower_score: It's novel and correct.
Decision: Accept: poster