Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introduces the concepts of counterfactual regret minimization in the field of Deep RL. Specifically, the authors introduce an algorithm called ARM which can deal with partial observability better. The results is interesting and novel. This paper should be accepted.

The presentation of the paper can be improved a bit. Much of the notation introduced in section 3.1 is not used later on. There seems to be a bit of a disconnect before and after section 3.3. The algorithm in deep RL could be explained a bit better.

There are some papers that could be connected. Notably the distributional RL work that was recently published could be very interesting to compare against in partially observed environments.

It could also be interesting if the authors were to run the proposed algorithm on environments where long-term memory is required to achieve the goals.

The argument the authors made against recurrent value functions is that recurrent value could be hard to train. An experiment illustrating this effect could be illuminating.

Can the proposed approach help when we have recurrent value functions? Since recurrence does not guarantee that all information needed is captured.


Finally some miscellaneous points:

One interesting reference: Memory-based control with recurrent neural
networks by Heess et al.

Potential typos: in the 4th bullet point in section 3.1, should it be \rho^{\pi}(h, s')?
review_title: Interesting results on applying counterfactual regret minimization results to Deep RL.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents Advantage-based Regret Minimization, somewhat similar to advantage actor-critic with REINFORCE.
The main focus of the paper seems to be the motivation/justification of this algorithm with connection to the regret minimization literature (and without Markov assumptions).
The claim that ARM is more robust to partially observable domains is supported by experiments where it outperforms DQN.

There are several things to like about this paper:
- The authors do a good job of reviewing/referencing several papers in the field of "regret minimization" that would probably be of interest to the ICLR community + provide non-obvious connections / summaries of these perspectives.
- The issue of partial observability is good to bring up, rather than simply relying on the MDP framework that is often taken as a given in "deep reinforcement learning".
- The experimental results show that ARM outperforms DQN on a suite of deep RL tasks.

However, there are also some negatives:
- Reviewing so much of the CFR-literature in a short paper means that it ends up feeling a little rushed and confused.
- The ultimate algorithm *seems* like it is really quite similar to other policy gradient methods such as A3C, TRPO etc. At a high enough level, these algorithms can be written the same way... there are undoubtedly some key differences in how they behave, but it's not spelled out to the reader and I think the connections can be missed.
- The experiment/motivation I found most compelling was 4.1 (since it clearly matches the issue of partial observability) but we only see results compared to DQN... it feels like you don't put a compelling case for the non-Markovian benefits of ARM vs other policy gradient methods. Yes A3C and TRPO seem like they perform very poorly compared to ARM... but I'm left wondering how/why?

I feel like this paper is in a difficult position of trying to cover a lot of material/experiments in too short a paper.
A lot of the cited literature was also new to me, so it could be that I'm missing something about why this is so interesting.
However, I came away from this paper quite uncertain about the real benefits/differences of ARM versus other similar policy gradient methods... I also didn't feel the experimental evaluations drove a clear message except "ARM did better than all other methods on these experiments"... I'd want to understand how/why and whether we should expect this universally.
The focus on "regret minimization perspectives" didn't really get me too excited...

Overall I would vote against acceptance for this version.

review_title: A regret minimization approach to policy gradient algorithms.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Quality and clarity:

The paper provides a game-theoretic inspired variant of policy-gradient algorithm based on the idea of counter-factual regret minimization. The paper claims that the approach can deal with the partial observable domain better than the standard methods. However the results only show that the algorithm converges, in some cases, faster than the previous work  reaching asymptotically to a same or worse performance. Whereas one would expect that the algorithm achieve a better asymptotic performance in compare to methods which are designed for fully observable domains and thus performs sub-optimally in the POMDPs. 

The paper dives into the literature of counter-factual regret minimization without providing much intuition on why this type of ideas should provide improvement in the case of partial observable domain. To me it is not clear at all why this idea should help in the partial observable domains beside the argument that this method is designed in the game-theoretic settings   which makes no Markov assumption . The way that I interpret this algorithm is that by adding A+ to the return the algorithm  introduces some bias for actions which are likely to be optimal so it is in some sense implements the optimism in the face of uncertainty principle. This may explains why this algorithm converges faster than the baseline as it produces better exploration strategy. To me it is not clear that the boost comes from the fact that the algorithm deals with partial observability more efficiently.


Originality and Significance:

The proposed algorithm seems original. However,  as it is acknowledged by the authors this type of optimistic policy gradient algorithms have been previously used in RL (though maybe not with the game theoretic justification). I believe the algorithm introduced  in this paper, if it is presented well, can be  an interesting addition to the literature of Deep RL, e.g.,  in terms of improving the rate of convergence. However, the current version of paper  does not provide conclusive evidence for that as in most of the domains the algorithm only converge marginally faster than the standard ones. Given the fact that algorithms like dueling DQN and DDPG are   for the best asymptotic results and not  for the best convergence rate, this improvement  can be due to the choice of hyper parameter such as step size or epsilon decay scheduling. More experiments over a range of hyper parameter is needed before one can conclude that this algorithm improves the rate of convergence.
 
review_title: The paper provides a game-theoretic inspired variant of policy-gradient algorithm based on the idea of counter-factual regret minimization. The paper claims that the approach can deal with the partial observable domain better than the standard methods. However the results only show that the algorithm converges, in some cases, faster than the previous work.

Meta-review: The reviewers agree this is a really interesting paper, with an interesting idea (in particular
the use of regret clipping might provide a benefit over typical policy gradient methods). However,
there are two major concerns: 1) clarity / exposition and more importantly 2) lack of a strong
empirical motivation for the new approach (why do standard methods work just as well on these
partially observable domains?).
Decision: Invite to Workshop Track