Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript considers a subclass of stochastic games named Markov potential games. It provides some assumptions that guarantee that a game is a Markov potential game and leads to some nice properties to solve the problem to approximately a Nash equilibrium. It is claimed that the work extends the state of the art by analysing the closed-loop version in a different manner, firstly constraining policies to a parametric family and then deriving conditions for that, instead of the other way around. As someone with no knowledge in the topic, I find the paper interesting to read, but I have not followed any proofs. The experimental setup is quite limited, even though I believe that the intention of the authors is to provide some theoretical ideas rather than applying them. Minor point: there are a few sentences with small errors, this could be improved.
review_title: Interesting work on Markov potential games, from the viewpoint of someone without any prior knowledge on the topic

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper studies multi-agent sequential decision making problems that belong to the class of games called Markov Potential Games (MPG). It considers finding the optimal policy within a parametric space of policies, which can be represented by a function approximator such as a DNN.
A main contribution of this work is that it shows that for MPG, instead of solving a multi-objective optimization problem (Eq. 8), which is difficult, it is sufficient to solve a scalar-valued optimization problem (Eq. 16).  Theorem 1 shows that under certain conditions on the reward function, the game is MPG. It also shows how one might find the potential function J, which is used in the single objective optimization problem.
Finding J can be computationally expensive in general. So the paper provides some properties that lead to finding J easier. For example, obtaining J is easy if we have a cooperative game (Corollary 1) or the reward can be decomposed/decoupled in a certain way (Theorem 2).


Evaluation:

This is a well-written paper that studies an important problem, but I don’t think ICLR is the right venue for it. There is not much about (representation) learning in this work. The use of TRPO as an RL algorithm in the Experiment does not play a critical role in this work either. Aside this general comment, I have several other more specific comments.


- There is a significant literature on the use of RL for multi-agent systems. The paper does not do a good job comparing and positioning with respect to them. For example, refer to the following recent paper and references therein:

Perolat, Strub, et al., “Learning Nash Equilibrium for General-Sum Markov Games from Batch Data,” AISTATS, 2017.


- If I understand correctly, the policies are considered to be functions from the state of the system to a continuous action. So it is a function, and not a probability distribution. This means that the space of considered policies correspond to the space of pure strategies. We know that for some games, the Nash equilibrium is a mixed strategy. Isn’t this a big limitation of this approach?


- I am unclear how this approach can handle stochastic dynamics. For example, the optimization (P1) depends on the realization of (theta_i)_i. But this is not available. The dependence is not only in the objective, but also in the constraints, which makes things more difficult.

I understand that in the experiments the authors used two models (either the average of random realization, or solving a different optimization for each realization), but none of them is an appropriate solution for a stochastic system.


- How large is the MPG class? Is there any structural result that positions them compared to other Markov Games? For example, is the class of zero-sum games an example of MPG?


- There is a comment close to the end of Section 5 that when there is no prior knowledge of the dynamics and the reward, one can use the proposed approach to learn PCL-NE by using any DRL.
This is questionable because if the reward is not known, the conditions of Theorems 1 or 2 cannot be verifies, so it is not possible to use (P1) instead of (G2).


- What comments can you make about the computational complexity? It seems that depending on the dynamics, the optimization problem P1 can be non-convex, hence computationally difficult to solve.


- How is the work related to the following paper?
Macua, Zazo, Zazo, “Learning in Constrained Stochastic Dynamic Potential Games,” ICASSP, 2016

======
I updated the score based on the authors' rebuttal.

review_title: ICLR may not be the right venue; Technical questions: Unclear how to deal with stochastic dynamics, etc.

final_score: rating: 7: Good paper, accept
review_content: While it is not very surprising that in a potential game it is easy to find Nash equilibria (compare to normal form static games, in which local maxima of the potential are pure Nash equilibria), the idea of approaching these stochastic games from this direction is novel and potentially (no pun intended) fruitful. The paper is well written, the motivation is clear, and some of the ideas are non-trivial. However, the connection to learning representations is a little tenuous. 
review_title: review

Meta-review: The paper considers Markov potential games (MPGs),  where the agents share some common resource. They consider MPGs with continuous state-action variables, coupled constraints and nonconvex rewards, which is novel. The reviews are all positive and point out the novel contributions in the paper
Decision: Accept (Poster)