Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary:** 
This paper proposes the Learning to Exploit (L2E) framework that can quickly adapt to diverse opponent's unknown strategies. The main contributions of L2E include: 1. learning of the base model based on the optimization similar to MAML (Finn et al., ICML-17) to adapt to a new opponent after a few learning iterations (Section 2.1), 2. the generation of the hard-to-exploit opponent to robustly train the base model (Section 2.2), and 3. the generation of diverse opponent policies using the maximum mean discrepancy (MMD) metric (Section 2.3). Empirical results show that L2E can exploit diverse opponents in the Leduc poker, BigLeduc poker, and Grid Soccer domains. 

**Reasons for Score:**
Overall, I vote for a score of 5. While the opponent strategy generation (OSG) algorithm with the counter adaptability and diverseness is an interesting idea, I am concerned about inappropriate baseline comparisons (please refer to Concerns and Questions below). After seeing the authors' responses to my concerns, I am open to raising my score.

**Pros:**
1. OSG removes the requirement of preparing the population or task distribution in meta-learning, which can be expensive.
2. Section 3.3.1 shows promising results that the proposed MMD regularization term can generate diverse opponents. 

**Concerns and Questions:**
1. OSG, which generates a competitive and diverse opponent population, is a paper's main contribution. However, this paper compares the baselines trained based on the random opponent population, such as the MAML baseline in Section 3.2. Hence, it is unclear how much more effective and diverse opponents that OSG can generate compared to state-of-the-art opponent generation-based algorithms (e.g., population-based RL (Jaderberg et al., Science-19)). 
2. An important claim of the paper is that explicit opponent modeling requires large sample complexity (Section 1). However, an explicit opponent modeling baseline is not compared in the experimental results. Pre-training an explicit opponent modeling method on the same opponent population generated by OSG and then comparing its adaptation against a new opponent will further highlight L2E's fast adaptation performance. 
3. In MAML, a non-meta-learning baseline is compared by pre-training a policy on all of the meta-training tasks and then fine-tuning at a meta-test task (i.e., the "pretrained" baseline in MAML). However, the TRPO baseline in this paper does not perform the pre-training ("The TRPO baseline does not perform pre-training ..." in Section 3.2). Because L2E's base model is pre-trained using opponents generated by OSG, a more fair comparison is to pre-train the TRPO (possibly based on the random opponents) and then fine-tune against a new opponent. 
4. In Figure 2, the trained base policy receives the negative return playing against the oracle opponent (possibly expected as the oracle opponent takes actions based on perfect information). But, why does L2E receive the positive return in Table 1?

**Additional Feedback:**
1. In the appendix, hyper-parameters/training details for L2E and the baselines (e.g., the number of trajectories for each adaptation, details on how the meta-training task distribution is constructed for MAML) are missing.
2. In Tables 1 and 2, adding the variance will be helpful. Specifying the number of random seeds used in the experimental results will also be helpful.
3. Algorithm 1-3 are for learning the base policy (i.e., the meta-training procedure). Adding an algorithm in the Appendix when adapting to a new opponent after the base policy training (i.e., the meta-testing procedure) will be helpful.
4. Adding an explanation about why the particular metric of MMD (instead of other metrics) is chosen to compare the difference between distributions will be helpful.
5. In Section 2.2.2, it is noted that "For trajectories with different length, we clip the long trajectory to the same length as the short one" for the MMD calculation. I wonder whether applying the masking based on the done signal from the environment can be better than applying the clipping.

**Reference:**

Chelsea Finn, Pieter Abbeel, Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML, 2017.

Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, Thore Graepel. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science, 2019.

**After rebuttal:**
The responses address most of my main concerns, and I have increased the rating from 5 to 6. As discussed during the rebuttal, in the future, having additional experiments that compare between OSG and other appropriate population generation baselines would be helpful.
review_title: Interesting idea about opponent population generation but with concern on appropriate baseline comparisons

final_score: rating: 4: Ok but not good enough - rejection
review_content: ## Summary
The paper suggests a novel framework (coined L2E) to learn a policy that is optimized to adapt quickly (and exploit) a wide range of unknown opponents.
To do so it trains a base policy that is optimized so that it can maximize its expected reward against a variety of opponents using only a few updates of its parameters (i.e. a few gradient steps of a straightforward optimization problem).
The various opponents that are used for the training of this base policy are generated in two steps.
First, given a base policy, an "hard-to-exploit" opponent is generated adversarially (in a procedure coined hard-OSG), to minimize the the reward that the base policy would get by adapting to it (using the few updates that are allowed to it in its adaptation step).
Then, given a base policy and a "hard-to-exploit" opponent, more diverse opponents are sequentially generated (in a procedure coined diverse-OSG) by optimizing their expected reward against the base policy while maximizing their "diversity" with the "hard-to-exploit" opponent and the already generated diverse opponents.
More formally, this diversity between two policies is defined (and optimized) as the MMD between the distributions over the trajectories that they generate (when the policies are seen as MDPs "playing" against the given base policy).
The base policy is then trained iteratively (as described above) against the diverse opponents, that are themselves generated (as described above) with the current iterate of the base policy.
After exposing this training procedure, the authors evaluate L2E on 3 toy games, showing that the trained policies are indeed able to benefit from little adaptations to a variety of heuristic opponents and perform better than some baseline methods.
They also empirically confirm that their "diversity-regularized policy optimization" indeed generates diverse policies.
Last, the authors empirically show the effect of their hard-OSG and diverse-OSG modules on the performance of L2E.

## Pros
- This paper is tackling a very relevant, interesting and difficult problem.
- I find the general approach of optimizing a policy to be able to "rapidly" and exploit a broad range of opponents to be very exciting.
- To the best of my (admittedly limited) knowledge, the suggested approach is significantly novel.
- While maybe a bit "roughly used" the diversity-inducing regularization term, using the MMD over the distribution of trajectories induced by the policies is interesting and potentially has a broader applicability than only L2E.

## Cons
- After careful reading, several key points remain unclear to me. Most notably, after training of L2E and when facing opponents with unknown policies, how does the base policy adapts? Is it done using eq. 2? If yes, is the expectation over the trajectories approximated with the actual observations made during the observation? How many observations are being used? If my understanding is correct, clarifying those points would help put in perspective how fast it actually takes for L2E to adapt in practice.
- It looks (to me, because no comment is made about it in the manuscript) like L2E must scale terribly with the size of the action space. First it must be extremely computationally intensive. While this remains feasible for the toy games that were used in the experiments, I am having very high doubts that this would scale well with larger games (even BigLeduc poker is ridiculously tiny compared to actual poker). At least, some comments about the computational aspects of L2E, or empirical evidence that L2E can handle larger games would be nice.
- A point is made, several times in the manuscript, that the base policy becomes "more robust and eliminates its weaknesses by learning to exploit hard opponents". First, it is not really clear what is precisely meant by this. Without further assumptions on the class of games, I do not really see why the base policy would be having a good expected reward before adaptation (either in average over a broad class of opponents or against the optimal opponent), or even less why it would be hard to exploit (especially after adaptation). In fact the empirical results suggest that the base policy is breaking even against a random opponent (before adaptation) at Leduc poker, which seems rather weak to me.
- The last point brings to a more general issue. I understand that the value of the contribution is more empirical than theoretical. Yet, it is absolutely not obvious to me whether L2E is supposed to converge at all (let alone having a clear idea about to what kind of solution it would converge). I am not a specialist of game theory, but I understand this is likely a difficult setting to analyze. At least, empirical evidence on the convergence of L2E would, in my opinion, strengthen greatly the manuscript.
- I find that the writing could also be improved. While the algorithm is admittedly hard to fully describe in a very succinct way, the amount of repetions or redundancies in the first 6 pages suggests that L2E could be more concisely and sharply introduced. The split between the main text and the appendices seems a bit arbitrary to me as I definitely think more content about related work should be exposed in the main document. At the very least, appendices A and B should be referenced in the main text. As it stands, there is literally no indication in the paper that the related work section and the algorithms can be found in the supplementary materials. There are also a lot of imprecisions in the form of somewhat vague claims or missing important details. In addition to the ones already mentioned, I would for instance take the example of section 3.2.2. where it is not clear to me how the first policy of diverse-OSG is generated in the absence of the hard-OSG module. And in that same paragraph the bold statement that hard-OSG helps enhance the stability (what is meant by that exactly?) is not clear at all to me. It is also claimed in 3.2. that positive returns are guaranteed against opponent without a clear style (whatever that precisely means). I see rather mild empirical envidence of this but certainly not guarantees. Another minor point is that I find the Theorem 1 to be a bit weirdly formatted. I imagine the theorem is supposed to be the statement that MMD equals 0 iff the two distributions are equal, but then, the following sentence shuld be more clearly separated (as not part of the theorem) and it should be more clearly stated that the result is not a contribution by adding the reference where this result first appeared (Gretton et al. '07, I assume). If not, the derivation of the gradient computation does not really constitute a "theorem". Last, there are a number of typos, or verbs missing throughout the manuscript, that should be easy to fix (sorry, it's really not convenient for me to list them without line numbers...).
- I'm a bit puzzled by some implementation details of the diverse-OSG. Notably, there seems to be no weighing on the MMD term in eq. 11. That seems pretty arbitrary to me. Could you elaborate on that? More specifically, I would imagine that if the MMD ways too little, the generated policies will be roughly identical while they will be diverse but potentially arbitrarily bad (in terms of expected reward). Also, while I don't have an issue with the somewhat arbitrary choice of an RBF kernel, I am a bit more puzzled by the choice of a width of 1. But I could imagine I'm missing an argument as to why this is a good choice.
- In Table 1, L2E is reported to have a positive average return against the oracle, which is defined as "making decisions based on perfect information". It's not clearly described what those decisions are but unless they are pretty bad, there is no way L2E or any policy can win against it. (And it should be pretty easy to find and implement the optimal strategy for the perfect information game.)

## Reasons for score
While I really want to emphasize that the problem is very interesting and that I like the premise of L2E, I think the paper, in its current form, is missing the target.
The main reasons can already be found in the "cons" that I listed.
To elaborate a bit further, I think that either the selected games for the experiments are too small and toy-like for a purely empirical paper (in contrast with AlphaStar or Liberatus achieving superhuman performance at games like Starcraft 2 or heads-up no limit hold'em, although they definitely tackle a different, and probably simpler problem).
In this current form, I consider the experiments as a crude proof-of-concept, which could be totally fine if there were more theoretical analysis to support the suggested approach.

## Questions during rebuttal period
I think several questions have already been raised in the rest of my review.
Most importantly, I would really love to understand how the base policy is updated in a "real setting", after training (see my cons #1).
review_title: A stimulating idea that misses the mark in its current form.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

The paper motivates the problem setting of quickly adapting to exploit sub-optimal opponents. The goal is to find a base policy which is quick to adapt to a range of sub-optimal opponents. To do this they use L2E which can generate exploiter and diverse opponent agents and updates the base policy based on trajectories from exploiter base policies. The paper focuses on the 2-player zero-sum setting where we have many algorithms which can find NE. I am concerned that this procedure is simply finding a NE and the other algorithms compared (MAML and TRPO) are not. I believe the paper needs to construct an argument that 1) L2E is not finding a NE and that 2) NE is not a good base policy.

=== Rebuttal Edit (Increased score form 4 to 6)

Thanks for the discussions. The primary reason for my score increase is discovering that the power of the framework is finding a representation that is quick at exploiting new opponents. I have been sufficiently convinced that this approach is not simply finding a NE (a sticking point in my review). I think that there are a collection of ideas here that are publishable and are of interest to the community.

The reason for not giving a higher score is that I think the points the paper made could be clearer: specifically I think the phrase "base policy" could be better replaced by "base representation" / "base model". I was stuck on the idea that the base policy had to be a strong one (eg a NE), and close to exploiter policies in *policy space* rather than *parameter space*. Re-reading after the paper update, I am worried that a significant portion of readers may fall into the same trap despite the authors' additional edits. Tightening the story would make this paper more appealing. I also broadly agree with the other reviewers suggestions / concerns.

For future work (also mentioned by another reviewer), I think there is no reason the "base policy" could not also be a strong policy too. I believe with minor adjustments to your framework this could be achieved, and one would have a model that both has low exploitability and is fast to adapt to new opponents - a potentially powerful combination.

## Score

I am recommending reject [UPDATE - see above] - although I will maintain an open mind due to my middling confidence in some of the background literature around the paper.

## Positives

Having a diversity regularized policy optimization procedure with MMD is interesting.

The PG update rules are interesting.

## Concerns

I am unsure about some of the claims wrt implicit/explicit opponent modelling. It seems that we need the exact opponent and base policy at all times in L2E. The literature that is cited in the paper is in a harder setting than this - where the opponent’s policy has to be estimated. Are we actually modelling opponents?

The MMD term seems very expensive and I am not sure it will scale, particularly with population size. It is also unclear to me how much sampling is necessary to estimate MMD. This approach does interest me however, perhaps the authors could comment on the scalability of this metric?

In the experiments section I would like to see more details on the actual training. How many outer and inner iterations were needed to converge? How many diverse and hard opponents were used during training? How many trajectories are sampled? Is learning rate tuning important? I am not sure I could reproduce from the details provided.

The results in Figure 2 do not seem to match up with the numbers in Table 1.

It is not clear whether the results are because the L2E procedure finds base policy close to Nash or if it is finding some other interesting policy. Would it be possible to adapt a Nash policy (with 4 adaptation steps) as an additional baseline for Table 1 and Figure 2 to check this? My concern is that the L2E framework is just finding Nash in an exotic way. From Table 1 it is clear the other baselines are not finding Nash. I appreciate that Nash was included as an opponent. I think testing this is key to back some of the claims made in this paper. If this procedure is indeed finding Nash, then is it doing it more efficiently than PSRO (Lanctot 2017) or self-play?

Zero-sum two-player settings NE is a reasonable thing to optimize for. Things like fictitious self play (FSP) are known to converge to approximate NE, and this algorithm looks like FSP without running the best response calculation to convergence.

Who is the opponent in Figure 4?

## Other Things

Figure 1: Should there be a loop from B’ back to B? Similar for O’ to O?

Page 2: “The key idea underlying L2E to train...” -> “L2E *is* to train”

Page 5: “only consider competitive agents in this work...” -> More specifically only zero-sum agents.

Page 7: “As with the previous experiments, we also use four gradient updates when adapting to a new opponent” -> The previous experiments used three updates?

Page 7: “Positive returns are also guaranteed against opponents without a clear style…” -> Are they really guaranteed? There are a lot of things like step size stochastic rollouts that make this statement tenuous. 

review_title: Is L2E just finding a NE?

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The authors propose an opponent modeling in 1-vs-1 games called the Learning to Exploit (L2E) framework, which exploits opponents by a few interactions with different opponents during training so that it can adapt to new opponents with unknown styles during testing quickly. In particular, the authors propose Opponent Strategy Generation (OSG) that produces effective opponents for training automatically through adversarial training for eliminating its own strategy’s weaknesses and diversity-regularized policy optimization to improve the generalization ability of L2E. Experimental results of two poker games and one grid soccer game indicate that L2E quickly adapts to diverse styles of unknown opponents.

Reasons for score: 
Although motivation, solution, novelty, and overall presentation were almost clear, the implementation details were unclear and there were no shared codes. I think the idea contributed to this community, but mainly for the above reproducibility, it is difficult to provide a higher rating. 

Pros:
1. Significance: the authors propose an opponent modeling called the L2E framework, which can adapt to unknown opponents quickly with a few observation of interactions
2. Methodological novelty: the authors propose OSG that produces effective opponents for training automatically through adversarial training and diversity-regularized policy optimization to improve the generalization ability of L2E. 
3. Experiments: the results of two poker games and one grid soccer game indicate that L2E quickly adapts to diverse styles of unknown opponents.

Cons:
1. The implementation details were unclear (e.g., network and learning hyperparameters and their selections) and there were no shared codes. 
2. There were some unclear points in Method and Experiments (below).

Other comments:

In Eqs. (3) and (4), the theta is updated based on (4) but also updated in computing pi_{theta O_i} described in the right-hand side of eq. (3). The former and the latter seem to update the theta in different manners, but the authors claimed that the theta is updated based on eq. (4). It seems to be confusing for me, but is this correct? 

In Eqs. (7) and (8), notations of the loss and reward should be defined.

The experimental results were about the Leduc poker task, except for Fig. 4 (the Grid soccer task). There were no similar results of the Leduc poker task also in the Appendix. I would like to know the reason.
review_title: Marginally above acceptance

Meta-review: Well, this paper has achieved something remarkable in this review process:  The initial scores came in at fairly low scores (4, 5, 3, 6).  However, as the discussions / rebuttals went back and forth, the reviewers were able understand and see the merits of the proposed methodology.  Namely, the setting of L2E (Learning to Exploit), which makes use of a novel method called Opponent Strategy Generation, to quickly generate very different types of opponents to play against.  One more pertinent component is the use of MMD (maximum mean discrepancy regularization) which can remove the necessity of dealing with task distributions, and does a better job in creating diverse opponents.

Having understood the technical approach, three of the reviewers decided to substantially increase their scores. R4 increased 4->6, R5 increased 5->6, R3 increased 3->4, while R2 held steady with a score of 6. It was also good to see empirical favorable results compared to other baseline methods: L2E had the best return against unclear opponents, such as Rocks opponent and Nash opponent.

Without any reviewer arguing strongly for acceptance, the program committee decided that the paper in its current form does not quite meet the bar, and also that it would benefit from another revision. 
Decision: Reject