Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Review: 
This paper proposes a general model-free RL method for no-regret learning based on a repeated reconsideration of past behavior. The ARMAC algorithm using the off-policy policy evaluation algorithm TreeBackup to estimate value function and use regret matching to get the next joint policy.

This paper idea is origin from DCFR, DNCFR, single CFR.  But those are model-based algorithms. ARMAC is model-free algorithms and it can be used in a more border environment.  If the author compares this paper to the DREAM algorithm(Deep Regret minimization with Advantage baselines and Model-free learning) to state ARMAC advantage, I will update my score.
review_title: ARMAC is model-free algorithm and improved from neural based CFR.

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors adopt the idea from gaming theory to reinforcement learning and propose a new algorithm that uses the previous policy to update the current training without using importance sampling. Experiments show that the proposed algorithm cannot only work on the single-player setting but also work on the multi-agent (zero-sum) problems. However, I have the following concerns about the algorithms: 
1) To train the critic, how would the sample complexity be? Like vanilla Actor-Critic algorithms, can it be replaced by doing a one-step TD(0) update on the critic to improve the sample efficiency?
2) Since the original CFR method is solving the multi-agent zero-sum algorithm, it would be interesting why this extension could solve the single-player problem?

Also, it would make the contribution of this work more clear if the author can compare this exploration method with other exploration methods, such as $\epsilon$-greedy or UCB. To me, the algorithm uses sample trajectory $\rho \sim (\mu_i, \pi_{-i}^j)$. If we make all $\mu$ be random policy, is this exploration similar to $\epsilon$-greedy to some extent?

Considering all contributions and concerns mentioned above, I will suggest a borderline accept for this paper. I might change my score after the author's response and discussion.
review_title: Good combination with gaming theory and actor-critic, but still having several concerns

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the problem of counterfactual regret minimization and proposes an algorithm that does not use the importance sampling procedure. The claim is that this helps in reducing the variance usually introduced by the IS procedure. They propose a new algorithm that uses the previously used policies as a buffer and replays those policies to learn a new policy. The algorithm is also claimed to be highly scalable for games with large state-action pairs.


My overall assessment of this paper is that, the problem considered is an important one and a well-implemented/well-written paper would definitely be a good paper that is significantly impactful for this community. However, my personal opinion is that this paper is not yet there. First, the paper is not well-written for someone who is not familiar with the exact prior literature on CFR using deep learning. In particular, the paper is not organized well, some of the important definitions are omitted and in general the key points aren't sufficiently highlighted. Also I find the experiments to be a bit lacking. In particular, one of the key claims of the paper is that it eliminates the variance introduced by the IS procedure, yet there are no experiments to substantiate this (more below). Also the executed experiments are not explained well. My opinion is that the section of "theoretical properties" is totally unhelpful for the main section and can be relegated to the appendix; it does not add value to the understanding of this algorithm. Additionally, I have several comments below that can help with both the writing, the weaker experiments and overall improving this paper to a state that will make it more impactful/easier to understand. My current assessment of this paper is that without a major revision of the writing and execution, this paper is not in a state to be accepted. 


(1) A self-sufficient description of the problem statement. The paper does describe the problem, but it takes a few readings for a new reader not familiar with the exact line of work to get the setup and the contributions of this paper. Moreover, as an arrangement it is spread over introduction and notations. Finally, the paper doesn't explicitly state what the goal of a CFR algorithm is. Is it to minimize the total average counterfactual regret?

(2) Along those lines, I think the paper would be very informative if the algorithm was evaluated and instantiated on a very simple two-player game with  known Nash equilibria (even simpler than Atari and Montezuma's revenge). In particular, there are many such games known in theory (e.g., the prisoner's dillema) and you can pick one of them and instantiate/evaluate this algorithm to help the reader understand the notations precisely. My opinion is that the notations is intertwined with informal descriptions and it is often hard to parse what the authors are meaning to say. Having such a unified simple game will make this process easier on the reader. This also ensures that the implementation and the algorithm is itself correct.

(3) Some definitions in the empirical section are not defined formally. In particular, the quantity NashConv is not defined in this paper and relegated to a related work. Only the informal description is given. This makes the reading really hard. For instance, we know that it should be close to 0, but how close? For instance, how do I interpret a value of 0.5? What is the range of NashConv? These questions could be easily answered if a precise definition of NashConv was included in this paper.

(4) Issues with experiments: First, in the convergence plot for ARMAC in Figure 3, it would be good to plot the line for what the NashConv is (like in Figure 2). Second, the main claim of this paper is that the variance introduced by importance sampling is not present (because this algorithm does not invoke the IS procedure). However, it is surprising that the empirical evaluation section does not include a convincing experiment to drive this point. In particular, it would be good to have a comparison of the variance of the three algorithms and show that the variance introduced by the IS procedure is indeed a meaningful problem for the final outcome (i.e., variance in the convergence value to the NashConv) and that the proposed method indeed reduces it? If the variance introduced by the IS procedure does not lead to a meaningful variance on NashConv, please motivate better why the variance introduced by IS is a problem?

(5) Some suggestions on plots: I find Figure 1 to have particularly difficult to read color schemes. First, the caption states a Pink line and it took me a while to figure which was Pink (I think it is the violet line which is pink). The same problem with Brown. Overall, the 7 colors used are pretty close and I would suggest either trying to use other markers to distinguish (such as a text on the line, different line type etc) or try and use contrasting color schemes. For instance, some of these lines may be indistinguishable for a person with color blindness. Likewise, the titles of the plots are not meaningful/interpretable. In figure 1, both the x and y axis seem like titles that were used internally by the authors. I do not understand what they mean. In the x-axis title, I don't see what that 1 represents and I quite frankly do not understand what the y-axis is trying to say. Likewise, among the three plots, across Figures 1, 2, 3 if the order of the three environments remained consistent, that would help the readability. This is also important, because I still do not understand how to interpret Figure 1. I think 0 implies that the algorithm is good. But if its negative does it mean that the algorithm is better while if its positive it is worse? Please explain Figure 1 better.

(6) Finally there are many typos throughout the paper. Here is a non-exhaustive list.
- page 3: heads on the same neural -> I don't understand what this statement is saying and seems grammatically incorrect as stated.
- Page 3: The first one estimate -> The first one estimates
- In caption of Figure 1, the word modulations is used. This word is not defined anywhere else in the paper. Please either define this or re-use a word that has been introduced.
- Page 6: ARMAC generates experiences using those... -> this sentence is grammatically incorrect. Please fix.
- Page 8: Conclusions. "It is brings back" -> grammar check this sentence
- Page 8: Conclusions: "for convergence one of the classes" -> grammar check





review_title: Requires improved presentation and bit more experimentation

Meta-review: This paper introduces a new algorithm to solve game, more or less similar (in the general idea, yet differences are interesting) than CFR. The concept is to sample from past policies to generate trajectories and update sequentially (via regret matching).

The three reviewers gave rather lukewarm reviews, with possible suggestions of improvements (that were more or less declined by the authors for those proposed by Rev3 and Rev4; the added material focuses more on the clarity of the text than on the content itself).

I have also read the paper, and find it quite difficult to assess. At the end, it is not clear to a reader whether ARMAC is the new state of the art, or just a "variant" of CFR that will be soon forgotten. The performances do not seem astonishing (at least against NSFP) and even though DREAM might not be satisfactory to the authors (EDIT POST DISCUSSION: actually, DREAM is a valid competitor and must be included in the comparative study), it would have been nice to provide some comparison. Maybe the issue is the writing of the paper that could and should be improved so that it is clearer what are the different building blocks of ARMAC (and their respective importance).

If ARMAC is the new state of the art, then I am sure the authors will be able to clearly illustrate it in a forthcoming revision (maybe with more experiments, as suggested by Rev2). Unfortunately, for the moment, I do not think this paper is mature enough for ICLR.
Decision: Reject