Truthful Self-Play

Abstract: We present a general framework for evolutionary learning to emergent unbiased state representation without any supervision. Evolutionary frameworks such as self-play converge to bad local optima in case of multi-agent reinforcement learning in non-cooperative partially observable environments with communication due to information asymmetry.  Our proposed framework is a simple modification of self-play inspired by mechanism design, also known as {\em reverse game theory}, to elicit truthful signals and make the agents cooperative. The key idea is to add imaginary rewards using the peer prediction method, i.e., a mechanism for evaluating the validity of information exchanged between agents in a decentralized environment. Numerical experiments with predator prey, traffic junction and StarCraft tasks demonstrate that the state-of-the-art performance of our framework.
      

Reviews and author responses:

## Summary

The paper proposes an elegant baseline addition to policy gradient / self play to encourage truthful signaling in Comm-POSGs. It outperforms self-play on various communication domains and uses a number of interesting concepts. Cool paper - I think ICLR will enjoy reading!

## Score

Clear accept - pending a more thorough check of the appendices.

=== Edit ===

Other more confident reviewers have pointed out some concerns. I am still positive about the paper, but due to lack of rebuttal I am going to downgrade from my initial score.

## Positives

The addition is elegant and the imaginary component does not leak into the networks.

Does not add a bias to the PG objective.

Utilizing an imaginary reward is a very interesting approach - maybe there are domains that this is useful?

Clearly good results on the domains tested.

## Concerns

How is beta chosen in the experiments?

Limited alternative baselines. Is there another algorithm that will perform well here?

## Other Things

Page 1: “P*e*rtially observable…”

Page 2: I prefer “identity matrix” over “unit matrix”

Page 3: “...but interacts each other” - ?

Page 4: “weight” is not defined yet (maybe call parameters?) - clarify if theta and phi are scalars or vectors. Wording implies they are scalars.

Page 4: CRA could do with a small diagram showing the interactions


**Summary:**

This work presents an auxiliary loss that promotes truthfulness in non-cooperative multi-agent games with communication channels. The truthfulness is promoted through prediction rewards, which capture how well each agent can model other agents' policies via observing their messages. For this mechanism to work, it assumes that each agent has access to other policies, and such an assumption can be satisfied under the self-play / centralized-learning decentralized-execution / shared-weights setup. Having such access to the policy weights, an agent can observe how consistent was their action with their broadcasted message which is a function over their hidden state. Thus by promoting consistency between their actions and messages, truthfulness is achieved. This paper provides theoretical justification of the proposed mechanism and introduces ideas from mechanism design literature in multi-agent reinforcement learning under a non-cooperative setup (not sharing rewards).

Pros: 
- Introducing truthful mechanism design and providing theoretical guarantees in non-cooperative MARL is an interesting idea. The part that I find significant contribution is more on the theoretical justification than the mechanism itself (see related work section comments).

Cons: 
- Motivation is not that clear to me and the introduction / related work doesn't cover relevant work as it should.
- Notation different from what is common in MARL and makes it difficult to read at parts

**0 - Abstract**

"Evolutionary frameworks such as self-play converge to bad local optima in case of multi-agent reinforcement learning in non-cooperative partially observable environments with communication due to information asymmetry." - I think this needs further discussion on the main text. how information asymmetry causes bad local optima? Also, as I mention later, this work is similar to centralized-learning/decentralized-execution work and as such, it should be presented in the abstract as well.

Also as I mention later, mechanism design in MARL is not new and intrinsic rewards have been applied before to promote cooperation in mixed incentives games (e.g. [1]).

Imaginary rewards are mentioned in the abstract however this is not a common term I believe. The more common term is the intrinsic or auxiliary reward.

**1 - Introduction**

"improved exploration coverage" - the main idea in these papers was not related to exploration coverage but to information passing policies. why exploration is mentioned here?

"improper bias is injected into the state representation" - What is improper bias? I would feel safer with this term if it was referenced or given a concrete example of what that means.

"... also the robots (Jaderberg et al., 2018), " - this citation refers to multiplayer first-person games, not robots.

" ..actions such as concealing information and deceiving other agents at equilibrium.. " - cite [https://arxiv.org/pdf/1803.03453.pdf](https://arxiv.org/pdf/1803.03453.pdf) as well as Singh et al. is citing this for the quote.

**2 - Related Work**

Missing literature on centralized learning / decentralized execution (CLDE) which is related to the current work in the sense that self-play with weight sharing is the same as CLDE [3]

World models seem irrelevant as they are discussed with respect to their exploratory power or as applications of RNNs - why exploration is part of the discussion here? (generally, the application of RNNs is not a concept that worths emphasizing I believe but that's just an opinion - most of the multi-agent communication algorithms are usually using RNNs to equip the agents with some memory and compensate for the potential violation of the markovian assumption of the MDP)

Communication in mixed cooperative-competitive environments but also Sequential Social Dilemmas seems relevant as well and not discussed. For example [1, 2, 3]. In fact, in [1] the authors are using intrinsic rewards which can be thought of as mechanism design promoting cooperation.

Also, the use of intrinsic and auxiliary rewards in multi-agent communication seems relevant to mechanism design but not discussed as well. e.g.[6]

"To the best of our knowledge, introducing mechanism design to MARL is a new direction for
the deep-learning community." - What is the difference between this and intrinsic rewards which are currently under active research? e.g. [1]

**3.1 - COMM-POSG**

Is the state transition probability $\in S \times A \times S$ or $S \times A^n \times S$?

"We denote the social welfare at the BNE is *J , and the global maximum ˆJ. In general, *J != ˆJ holds, which is closely related to the information asymmetry." - is there a reference for this or can you provide an explanation?

**3.3 - Truthfulness**

"eventually it also maximizes *J will be maximized" - can you clarify/fix this sentence?

**4.1 - Imaginary Reward**

"However, the baseline function is a quantity that is determined based on the value function of a single agent, whereas the imaginary reward is different in that (1) it affects the value function of each agent" - This has been addressed before in [4] where a baseline is designed to compute a separate value per agent. 

Although I like the idea of imaginary rewards I don't understand what is the difference between this and auxiliary rewards or intrinsic/extrinsic breakdown which is a common technique in MARL. Why not mix directly on the reals and represent it as a complex number? Eventually, you mix them in Alg 1. (5) so if there isn't any other theoretical justification, why introduce a new notation/term? My point is that it makes reading this harder than it should because it's not common (if you have examples of work using a similar definition please let me know).

**5 - Numerical Experiment**

Can you present the results also in terms of Avg. number of steps (for pred prey) and success % (for junction), similarly to [7]? The way it is I can't verify that the numbers you are reporting are comparable to existing runs of the compared work (if you can reproduce the same number, if not I'd be ok with some justification as to why the numbers differ or why used different reporting metric). Similarly for StarCraft, what is the win ratio and average steps?

**6 - Concluding Remark**

"To the best of our knowledge, introducing mechanism design to MARL is a new direction for the deep-learning community" - As mentioned, intrinsic and auxiliary rewards can be thought of as mechanism design, and as such mechanism design to MARL is not as novel.

[update] reduced score given the lack of 

[1] Jaques, Natasha, et al. "Social influence as intrinsic motivation for multi-agent deep reinforcement learning." International Conference on Machine Learning. PMLR, 2019.

[2] Leibo, Joel Z., et al. "Multi-agent reinforcement learning in sequential social dilemmas." arXiv preprint arXiv:1702.03037 (2017).

[3] Lowe, Ryan, et al. "Multi-agent actor-critic for mixed cooperative-competitive environments." Advances in neural information processing systems. 2017.

[4] Foerster, Jakob, et al. "Counterfactual multi-agent policy gradients." arXiv preprint arXiv:1705.08926 (2017).

[5] Mordatch, Igor, and Pieter Abbeel. "Emergence of grounded compositional language in multi-agent populations." arXiv preprint arXiv:1703.04908 (2017).

[6] Singh, Satinder, et al. "Intrinsically motivated reinforcement learning: An evolutionary perspective." IEEE Transactions on Autonomous Mental Development 2.2 (2010): 70-82.

[7] Singh, Amanpreet, Tushar Jain, and Sainbayar Sukhbaatar. "Learning when to communicate at scale in multiagent cooperative and competitive tasks." arXiv preprint arXiv:1812.09755 (2018).

1. Summary and contributions:
The author claimed that evolutionary learning, such as self-play converge to local optima in multi-agent partially observed non-cooperative settings. They proposed a truthful self-play method and add imaginary rewards into self-play method by using the idea of reverse game theory. They evaluated their method on three different settings.
I have read this paper for several times, however because of the poor writing, it's very hard to follow the key idea of this paper. 
Also, I have some author concerns about the technique and experiment. 
I will list some of the issues as follows.

2. Concerns:
(1) I wonder the author talk about some key self-play methods, such as David's Generalised weakened fictitious play[1], Johannes's Fictitious Self-Play[2]  in partially observed MDPs, i.e., imperfect information game in the game theory community. Typically, these methods have good convergence on partially observed non-cooperative games.
If the authors can compare the Johannes's method on the experiment, it will be better.
(2) The written skill about this paper needs to improve. Some paragraphs need to be reorganized. For example, in section 2, the author talks about the related work on self-play on the second paragraph. It's confusing of say "RNNs such as world models are capable of more comprehensive ranges of exploration in partially observable environments ...". Typically, self-play is a kind of methods in game theory while RNN is a kind of deep learning neural networks. After that, the author talk about evolutionary learning and supervised learning. 
other example, such as "To the best of our knowledge, there are no studies that have introduced truthful mechanisms into the field of MARL, but it may be possible to introduce it by using agents that can learn flexibly, such as neural networks."
(3) The experiment results are not very clear. For example, the x-axis and y-axis in Figure 2 are not defined, so that it's hard to confirm the conclusion.

3. Questions:
(1) What's global maximum $\hat{J}$ ? Giving an example and explaining why $J^{*} \neq \hat{J}$ will be better.
(2) What's post-state? I only find the definition of pre-state in section 3.1.
(3) it's not clear about section 3.2. I's a little difficult to make the connection between section 3.1 and 3.2. Adding some figures will be better.

4. some issues/typos:
(1) page 1, pertially observable stochastic games (POSG): pertially -> partially?
(2) Comm-POSG is used in page 2, however it's defined in page 3.
(3) page 2: "interacts each other" -> "interacts with each other"
(4) remove ", and" in Defintion 3.1 after "an observation probability"
(5) page 3: $r_{ti}$ is reward, $\pi_{i}$ is a stochastic policy.
(6) page 4: all the agent share -> all the agents share
(7) To remain the social welfare of the system is real

[1] Leslie, David S., and Edmund J. Collins. "Generalised weakened fictitious play." Games and Economic Behavior 56, no. 2 (2006): 285-298.
[2] Heinrich, Johannes, Marc Lanctot, and David Silver. "Fictitious self-play in extensive-form games." In International Conference on Machine Learning, pp. 805-813. 2015.


This paper examines the setting of partially observable stochastic games where agents have the possibility of communication.  The approach taken is what would be termed “direct revelation” in mechanism design: agents are supposed to reveal their full internal state / history to each other at each step.  This gives the most complete information possible to enable good decisions.  However, agents may not have an incentive to report truthfully, so an approach based on peer prediction is used to incentivize this.  The results show improvements over prior approaches to communication in three tasks: predator prey, traffic junction, and StarCraft.

I like the motivation behind this paper, and agree that the idea of reasoning about the incentives to communicate and using techniques like peer prediction to encourage this is an interesting and important approach.  However, I have some conceptual issues with exact way the techniques are used and some presentation issues make parts of the approach unclear.

Detailed comments:

1) The paper cites quite a bit of work from the economics, statistics, and game theory communities, some of which is relevant (e.g. scoring rules and peer prediction) but quite a bit of which is not (broad mechanism design papers, VCG).

2) The paper is quite clear about the objective of the system designer, optimizing social welfare per equation (1) but never defines, except perhaps implicitly in Algorithm 1, what the objective of each agent is.  This manifests in a number of ways (see also points 3, 7, and 8), but one immediate issue is that I’m not sure I agree with the claim at the end of 4.1 that J^* != J is related to information asymmetry.  It is possible this is an exacerbating factor, but I would expect problems even in a complete information setting due to agents having different incentives.

3) I don’t see why Proposition 3.1 (and therefore the results that rely on it such as Prop 4.1 and Theorem 4.1) should be true in general.  It makes sense in cooperative settings, such as the predator prey one used in the experiments, where each agent on a team gets the same reward.  But in more general settings, like traffic junction, agents may not have the same incentives.  So even if they all truthfully reveal their information we shouldn’t expect their individual decisions to cause us to reach the global optimum.  There is a large literature on the “Price of Anarchy” which looks at bounding the size of this gap, but apart from very special cases there is almost always a gap.  I tried looking at the proof but couldn’t get any clarity about this.

4) The “Imaginary Rewards” are typically called payments or transfers.  Mechanisms which ensure they add to zero are referred to as “budget balanced” or sometimes “redistribution” mechanisms.  This actually requires quite a bit of care to establish incentive properties because in naïve implementation, which the subtracting the mean one used in this paper appears to be, there can actually be incentive issues introduced by this redistribution process.

5) The notation for a scoring rule F(p_s || s) is a very special case (sometimes called “local”) that really only the log score satisfies.  In general it should be F(p||s) as the score depends on the whole distribution, not just p_s.  See, e.g., the Brier score.

6) I found equation (6) quite hard to parse because the relevant notation is scattered.  If I understand it correctly, the idea is that In particular, z_{tji}  is signal of a single agent j, which we should expect to be it’s pre-state.  Each agent collects the messages of all the other agents and computes its post state (which I’m not sure why is denoted as a sample from q).  \pi takes a post state as an argument, but can equally well be applied to the pre state reported by another agent to predict the agent’s action, which is why we can apply a scoring rule.  But given all this I’m not clear why the score in (6) is computed based on the a_{ti} sampled.  We have both policy distributions so instead could compute the expected score which would be lower variance.

7) Similar to point 3, I don’t see why we should expect Theorem 4.1 to hold in general.  Even if an agent maximizes its imaginary reward by truthfully reporting, it may be worthwhile to lie and accept a lower imaginary reward in exchange for a larger actual reward.  I don’t see anything ruling this out.   This result makes sense in the cooperative setting, but there I don’t see why and agent would have an incentive to lie in the first place so this whole piece seems redundant.

8) In the experiments, there is improvement in both cooperative and non-cooperative settings from SP IC3Net to TSP IC3Net.  I think this needs to be investigated and discussed further.  In particular, as discussed in point 7, in games like predator prey there doesn’t seem to be any incentive to misreport so that whole piece of TSP shouldn’t be doing much.  But clearly something quite different is happening.  Perhaps the TSP version uses the channel in a fundamentally different way?  E.g. is something (and there aren’t enough details for me to understand how this works so I’m not exactly sure what) enforcing a direct revelation structure (even without truthfulness) is a way the SP version does not?  This seems like a separate part of the contribution and important to disentangle from the incentive part.  Perhaps there is a natural way to do an ablation for this?

9) The paper, and particularly the introduction, has a number of grammatical and style issues.  It needs a careful copyediting pass.



Meta-review: This paper exposes a modification of self-play (in a non-cooperative setup) where agents expose their internal states to each other, and adds a "truthfulness" mechanism to ensure the agents do not hide information to each other.

The reviewers generally agree that the ideas presented, in particular the imaginary rewards, is interesting and should be explored further. The experimental results are good.

However, reviewers point out many problems related to clarity, writing and notations, and they note many typos. The motivation of this work also needs to be explained further.
Reviewers also find that the related work section needs to be remade, as it both lacks citations, and cites works that are not really relevant. This work positioning with respect to these previous works also needs to be expanded.
The reviewers also point out that the experimental results should be discussed further.

Generally, the paper cannot be published in its current state, and unfortunately the authors have not submitted an updated version or answers to the reviews. I therefore recommend rejection for this paper.

Decision: withdrawn-rejected-submissions