Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This work describes a method for finding mixed-strategy Nash equilibria in (normal form) games with continuous action spaces. Their work builds off (Raghunathan '19), which is a gradient-based method for finding pure-strategy Nash, based on the NI function. (It comes with all the standard caveats of gradient-based methods for finding Nash). The central contribution of this work is to parameterize a mixed strategy via a learned NN mapping from a simple distribution U[0,1]^d to the mixed strategy of interest.

My first impression on reading this work is that this learned parameterization of the mixed strategy is exactly how the GAN generator produces a distribution of samples (images). The generator network takes a noise vector sampled from U[0,1]^d or some other distribution, and samples an "action" (i.e. an image). Indeed, the original GAN paper (Goodfellow, 14) talks about this: "Generative adversarial nets are trained by simultaneously updating the discriminative distribution so that it discriminates between samples from the *data generating distribution*" (Fig. 1). It is not the case, as stated in Section 2, that GANs only produce pure strategies. 

The optimization rule for GNI is different than standard update rules: specifically, there are two step sizes \lambda and \rho, if \lambda -> 0 I think it becomes steepest descent but for \lambda > 0 it is kind of like an optimistic descent method (because each agent takes a gradient step assuming the other agents have already done SD by \lambda). This update rule has some desirable theoretical properties, which I believe are mostly proven by Raghunathan '19. It would be very useful to state more explicitly which of these properties are not known to be true for e.g. independent steepest descent for each agent.

The experimental section is nice in that it shows some toy games where mixed strategy continuous equilibria do better than pure-strategy ones, and shows that MC-GNI can find something better than a pure-strategy equilibrium. But if the claim is that GNI has these good equilibrium-finding properties, and you're already using the GAN decoder strategy, why are there no experiments on GANs? Even Raghunathan has experiments on GANs (albeit ones with pure-strategy (delta-function) equilibria).

Nits:
- MC-GNI is a confusing name because MC usually refers to "Monte Carlo" in this context
- I'm confused at the end of section 3 how "implying (sic) gradient descent on these function parameters" is performed. Are points sampled from U[0,1]^d and used as an estimator for F_i, as in a GAN? I don't see how the non-MC-sampled F_i can be computed. It would be nice to clarify one way or another.
- I don't see \lambda mentioned on the RHS of the last equation in sec. 3.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a method to learn mixed-strategies Nash equilibrium in multi-player games. To do so they describe a gradient-descent method that aims at minimizing a Kikaido-Isoda function (which is zero if an equilibrium is found). The paper offer proofs of the convergence towards a stationary Nash equilibrium in the case of convex cost functions. It also provides an application with a strategy approximation made with a deep neural network. They authors exemplify the strengths of the method on toy problems that are quite standard in the domain. 

I liked the paper very much but I have some concerns. First, I feel that the framework is based on a variational approach which would be well suited for a 0-order optimisation (like a black box or an evolutionary method). I wonder why the authors wanted to use a gradient based approach that adds a second layer of approximation and additional meta parameters to tune. 

Second, I felt the theoretical proofs are not using much more than standard algebra and the convex assumption was a bit unrealistic in most of multi-agent problems. I'd like the authors to comment on this. 

I was also wondering how this work can be related to other papers that try to learn a Nash equilibrium (or an \epsilon-Nash) on the bases of a difference in some norm between the value of the current policy and the value of the NE. For instance "Learning Nash Equilibrium for General-Sum Markov Games from Batch Data" by Perolat et al. This work addresses discrete action spaces but seems similar in spirit to me. Could the authors comment on this ?


review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel algorithm for finding mixed-strategy Nash equilibria in games with continuous action spaces. The paper proves convergence to a stationary Nash equilibrium under the assumption of convex utility functions. In quadratic games, general blotto games and GAMUT games, the proposed algorithm (MC-GNI) significantly outperforms its competitors. This experimental finding is not surprising, as its competitors were designed to compute pure-strategy Nash-equilibria.

This paper investigates an important, relatively unexplored problem. Its proposed algorithm appears theoretically justified and to achieve nontrivial performance in practice. However, I feel that there are some issues with this paper. 

There has been a significant amount of recent work on differentiable games. It would benefit readers for the paper to discuss how this work and those works are related. In particular, the latter are motivated by designing gradient-based algorithms for finding parameters of neural networks corresponding to local equilibria. These neural networks may themselves be playing nondeterministic strategies in some underlying game (e.g., GANs). This closely resembles the paradigm of the algorithm proposed in this paper, which finds deterministic parameters for a network that itself induces a nondeterministic strategy. This leads to a number of questions:
1. In the experiments, the paper’s language suggests that SGA is constrained to pure strategies. Why is it not parameterized by a network producing a stochastic strategy? This seems like the sensible comparison.
2. How are the convergence guarantees discussed in this paper related to the capacity of the network being used to approximate the policy? Are they lost if the network has insufficient capacity?
3. The paper mentions that SGA outperforms other existing algorithms applicable to continuous game settings. What about stable opponent shaping??

The writing in the paper would benefit from revision. While the paper generally gets the point across, much of it feels sloppy. There are also many typos, some of which are listed below.
 
Typos:
- Raghunathan et al. (2019) introduces 
- In precise
- drown
- for $\forall$
- we prepare each distribution πj a corresponding pushforward function
- Therefore, a common algorithm of computing
- On the one hand, global infimum can
- We develop a solution significantly improves the status-quo.
- of Mixed strategy on Continuous game

Other:
- The paper alternates between his and it when referring to players

Overall, I think this work is an outline for a nice paper but would like to see 
1) Clarification regarding its relationship to existing work
2) Experimental baselines in which SGA and stable opponent shaping are parameterized to produce nondeterministic strategies 
3) Cleaner writing
review_title: Official Blind Review #3

Meta-review: The paper presents an algorithm to compute mixed-strategy Nash equilibria for continuous action space games. While the paper has some novelty, reviewers are generally unimpressed with the assumptions made, and the quality of the writing. Reviewers were also not swayed by the responses from the authors. Additionally, it could be argued that the paper is somewhat peripheral to the topic of the conference.¨

On balance, I would recommend reject for now; the paper needs more work.
Decision: Reject