Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is outside of my area of expertise, so I'll just provide a light review:

- the idea of assuming that the opponent will take the worst possible action is reasonable in widely used in classic search, so making value functions follow this intuition seems sensible,
- but somehow I wonder if this is really novel? Isn't there a whole body of literature on fictitious self-play, including need RL variants (e.g. Heinrich&Silver, 2016) that approaches things in a similar way?
- the results on Hex have some signal, but I don’t know how to calibrate them w.r.t. The state of the art on that game? A 40% win rate seems low, what do other published papers based on RL or search achieve?

review_title: n/a

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper makes the simple but important observation that (deep) reinforcement learning in alternating Markov games requires a min-max formulation of the Bellman equation as well as careful attention to the way in which one alternates solving for both players' policies in a policy iteration setting.

While some of the core algorithmic insights regarding Algorithms 3 & 4 in the paper stem from previous work (Condon, 1990; Hoffman & Karp, 1966), I was not actually aware of these previous results until I reviewed this paper.

A nice corollary of Algorithms 3 & 4 is that they make for a straightforward adaptation of policy gradient algorithms since when optimizing one policy, the other is fixed to the greedy policy.

In general, it would be nice to have the algorithms specified as formal algorithms as opposed to text-based outlines.  I found myself reading and re-reading descriptions to make sure I understood what math was being implied by the descriptions.

Section 6

> Hex is simpler than Go in the sense that perfect play can 
> often be achieved whenever virtual connections are found 
> by H-Search

It is not clear here what virtual connections are, what H-Search is, and how these imply perfect play, if perfect play as previously discussed is unknown.

Overall, the results on Hex for AMCPG-A and AMCPG-B vs. standard REINFORCE variants currently used are very encouraging.  That said, empirically it is always a question of whether these results are specific to Hex.  Because this paper is not proposing the best Hex player (i.e., the winning rate against Wolve never exceeds 0.5), I think it is quite reasonable to request the authors to compare AMCPG-A and AMCPG-B to standard REINFORCE variants on other games (they do not need to be as difficult as Hex).

Finally, assuming that the results do generalize to other games, I am left wondering about the significance of the contribution.  On one hand, the authors have introduced me to literature I was not aware of, but on the other hand, their actual novel contribution is a rather straightforward adaptation of ideas in the literature to policy gradients (that could be formalized in a more technically precise way) with an evaluation on a single type of game.  This is a useful contribution no doubt, but I am concerned with whether it meets the significance level that I am used to with accepted ICLR papers in previous years.

review_title: A nice but somewhat minimal paper addressing caveats of existing adversarial RL attempts

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a variation over existing policy gradient methods for two players zero sum games, in which instead of using the outcome of a single policy network rollout as the return, they use the minimum outcome among a few rollouts either from the original position or where the first action from that position is selected uniformly among the top k policy outputs. 

The proposed method supposedly provides slightly stronger targets, due to the extra lookahead / rollouts. Experiments show that this provides faster progress per iteration on the game of Hex against a fixed third party opponent.

There is no comparison against state of the art methods like AlphaGo Zero which uses MCTS root move distribution and MCTS rollouts outcome to train policy and value network, even though the author do cite this work. There is also no comparison with Hexit which also trains policy net on MCTS move distribution, and was also applied to Hex.

The actual proposed method is actually a one liner change, which could be introduced much sooner in the paper to save the reader some time. While the idea is interesting, the paper felt quite verbose on introducing notations and related work, and a bit lacking on actual change that is being proposed and the experiment to back it up.

For example,  was it really necessary to introduce state transition probabilities p(s’, a, s) when all the experiments are done in the deterministic game of Hex ?

Also the experiment seems not fully fair to the reinforce baseline. My understand is that the proposed method is much more costly due to extra rollouts that are needed. It would be interesting to see the same learning curves as in Figure 2, but the x axis would be some computational budget (total number of network forward, or wall clock time). It is conceivable that the vanilla reinforce would do just as well as the proposed method if the plots were aligned this way. It would also be good to know the asymptotic behavior.

So even though the idea is interesting, it seems that much stronger methods AlphaGo Zero / Hexit are now available, and the experimental section is a bit weak. I would recommend to accept for a workshop paper but not sure about the main track.



review_title: A bit verbose on existing methods + notations and low on experiments

Meta-review: The reviewers agree that the paper is below threshold for acceptance in the main track (one with very low confidence), but they favor submitting the paper to the workshop track.

The paper considers policy gradient methods for two-player zero-sum Alternating Markov games.  They propose adversarial policy gradient (fairly obviously), wherein the critic estimates min rather than mean reward.   They also report promising empirical results in the game of Hex, with varying board sizes.  I found the paper to be well-written and easy to read, possibly due to revisions in the rebuttal discussions.

The reviewers consider the contribution to be small, mainly due to the fact that the key algorithmic insights were already published decades ago.  Reintroducing them is a service to the community, but its novelty is limited.  Other critiques mentioned that results in Hex only provide limited understanding of the algorithm's behavior in general Alternating Markov games.  The lack of comparison with modern methods like AlphaGo Zero was also mentioned as a limitation.

Bottom line: The paper provides a small but useful contribution to the community, as described above, and the committee recommends it for workshop.

Decision: Invite to Workshop Track