L2E: Learning to Exploit Your Opponent
Authors: Zhe Wu
Kai Li
Hang Xu
Meng Zhang
Haobo Fu
Bo An
Junliang Xing
Venue:  (2021)

Abstract: Opponent modeling is essential to exploit sub-optimal opponents in strategic interactions. One key challenge facing opponent modeling is how to fast adapt to opponents with diverse styles of strategies. Most previous works focus on building explicit models to predict the opponents’ styles or strategies directly. However, these methods require a large amount of data to train the model and lack the adaptability to new opponents of unknown styles. In this work, we propose a novel Learning to Exploit (L2E) framework for implicit opponent modeling. L2E acquires the ability to exploit opponents by a few interactions with different opponents during training so that it can adapt to new opponents with unknown styles during testing quickly. We propose a novel Opponent Strategy Generation (OSG) algorithm that produces effective opponents for training automatically. By learning to exploit the challenging opponents generated by OSG through adversarial training, L2E gradually eliminates its own strategy’s weaknesses. Moreover, the generalization ability of L2E is significantly improved by training with diverse opponents, which are produced by OSG through diversity-regularized policy optimization. We evaluate the L2E framework on two poker games and one grid soccer game, which are the commonly used benchmark for opponent modeling. Comprehensive experimental results indicate that L2E quickly adapts to diverse styles of unknown opponents.

Under review as a conference paper at ICLR 2021
L2E: LEARNING TO EXPLOIT YOUR OPPONENT
Anonymous authors
Paper under double-blind review
ABSTRACT
Opponent modeling is essential to exploit sub-optimal opponents in strategic in- teractions. One key challenge facing opponent modeling how fast adapt opponents with diverse styles of strategies. Most previous works focus on build- ing explicit models predict the opponents' styles or strategies directly. How- ever, these methods require a large amount of data to train the model and lack the adaptability to new opponents of unknown styles. In this work, we propose a novel Learning to Exploit (L22) framework for implicit opponent modeling. L2E acquires the ability to exploit opponents by few interactions with different op- ponents during training so that it can adapt new opponents with unknown styles during testing quickly. We propose a novel Opponent Strategy Generation (OSG) algorithm that produces effective opponents for training automatically. By Iearn- ing to exploit the challenging opponents generated by OSG through adversarial training, L2E gradually eliminates its strategy's 's weaknesses. Moreover, the generalization ability of L2E is significantly improved by training with diverse opponents, which are produced by OSG through diversity-regularized policy op- timization. We evaluate the L2E framework on two poker games and one grid soccer game, which are the commonly used benchmark for opponent modeling. Comprehensive experimental results indicate that 1.2E quickly adapts to diverse styles of unknown opponents.
INTRODUCTION
One core research topic in modern artificial intelligence is creating agents that can interact effec- tively with their opponents in different scenarios. To achieve this goal, the agents should have the ability to reason about their opponents' behaviors, goals, and beliefs. Opponent modeling, which constructs the opponents' models to reason about them, has been extensively studied in past decades (Albrecht Stone 2018). In general, an opponent model a function that takes some interaction history as its input and predicts some property of interest of the opponent. Specifically, the interaction history may contain the past actions that the opponent took various situations, and the properties of interest could be the actions that the opponent may take in the future, the style of the opponent (e.g., "defensive", "aggressive"). or its current goals. The resulting opponent model inform the agent's decision-making by incorporating the model's predictions in its planning procedure to optimize its interactions with the opponent. Opponent modeling has already been used in many practical applications, such as dialogue systems (Grosz Sidner 1986), intelligent tutor systems (McCalla et al. 2000), and security systems (Jarvis et al. 2005).
Many opponent modeling algorithms vary greatly in their underlying assumptions and methodology. For example, policy reconstruction based methods (Powers & Shoham, 2005 Banerjee & Sen 2007) explicitly fit an opponent model reflect the opponent's observed behaviors. Type reasoning based methods (Dekel et al., 2004, Nachbar 2005) reuse pre-Icarned models of several known opponents by finding the one which most resembles the behavior of the current opponent. Classification based methods (Huynh et al. (006)) Sukthanka & Sycara (007) build models that predict the play style of the opponent, and employ the counter-strategy, which is effective against that particular style. Some recent works combine opponent modeling with deep learning methods or reinforcement learning methods and propose many related algorithms (He et al., 2016 Foerster et al. 2018, Wen et al, 2018). Although these algorithms have achieved some success, they also have some obvious disad- vantages. First, constructing accurate opponent models requires a lot of data, which is problematic since the agent does not have the time opportunity collect enough data about its opponent in

---

Under review as a conference paper at ICLR 2021
most applications. Second, most of these algorithms perform well only when the opponents during testing are similar to the ones used for training, and it is difficult for them to adapt to opponents with new styles quickly. More related works opponent modeling are in Appendix
To overcome these shortcomings, we propose a novel Learning to Exploit (L2E) framework in this work for implicit opponent modeling, which has two desirable advantages. First, L.2E does not build an explicit model for the opponent, so it does not require a large amount of interactive data and eliminates the modeling errors simultaneously. Second, L2E can quickly adapt new opponents with unknown styles, with only a few interactions with them. The key idea underlying L2E to train a base policy against various styles of opponents by using only a few interactions between them during training. such that it acquires the ability to exploit different opponents quickly. After training, the base policy can quickly adapt to new opponents using only a few interactions during testing. In effect, our 1.2E framework optimizes for base policy that is easy and fast to adapt. It can be seen as a particular case of learning to learn, i.e,, meta-learning (Finn et al.) 2017). The meta-learning algorithm (c.f., Appendix A.2 for details), such MAML (Finn et al. 2017), is initially designed for single-agent environments. It requires manual design of training tasks, and the final performance largely depends the user-specified training task distribution. The L2E framework is designed explicitly for the multi-agent competitive environments, which generates effective training tasks (opponents) automatically (cf-. Appendix A.3 for details). Some recent works have also initially used meta-learning for opponent modeling. Unlike these works, which either use meta-learning to predict the opponent's behaviors (Rabinowitz et al.) (0018) or handle the non-stationarity problem in multi-agent reinforcement learning (Al-Shedivat et al. 2018), we focus on how to improve the agent's s ability to adapt to unknown opponents quickly.
In our L2E framework, the base policy is explicitly trained such that a few interactions with a new opponent will produce an opponent-spece -pecffffc- policy to effectively exploit this opponent, i.e, the base policy has strong adaptability that is broadly adaptive to many opponents. In specific, if a deep neural network models the base policy, then the opponent-specific policy can be obtained by fine-tuning the parameters of the base policy's ss network using the new interactive data with the opponent. A critical step in L2E is how to generate effective opponents to train the base policy. The ideal training opponents should satisfy the following two desiderata. 1) The opponents need to be challenging enough (i.e., hard to exploit). By learning to exploit these challenging opponents, the base policy eliminates its weakness and learns more robust strategy. 2) The opponents need to have enough diversity. The more diverse the opponents during training, the stronger the base policy's s generalization ability is, and the more adaptable the base policy to the new opponents.
To this end, we propose a novel opponent strategy generation (OSG) algorithm, which can produce challenging and diverse opponents automatically. We use the idea of adversarial training to gener- ate challenging opponents. Some previous works have also been proposed to obtain more robust policies through adversarial training and showed that it improves the generalization (Pinto et al. 2017 Pattanaik et al. 2018). From the perspective of the base policy, giving an opponent, the base policy first adjusts itself to obtain an adapted policy, the base policy then optimized to maximize the rewards that the adapted policy gets when facing the opponent. The challenging opponents are then adversarially generated by minimizing the base policy's s adaptability by automatically generat- ing difficult to exploit opponents. These hard-to-exploit opponents are trained such that even if the base policy adapts to them, the adapted base policy cannot take advantage of them. Besides, our OSG algorithm can further produce diverse training opponents with a novel divelsity-regullrriee policy optimization procedure. In specific, we use the Maximum Mean Discrepancy (MMD) met- ric (Gretton et al. (207)) evaluate the differences between policies. The MMD metric is then incorporated as a regularization term into the policy optimization process to obtain a diverse set of opponent policies. By training with these challenging and diverse training opponents, the robustness and generalization ability of our L2E framework can be significantly improved. To summarize, the main contributions of this work are listed bellow in four-fold:
We propose a novel learning to exploit (L22F) framework to exploit sub-optimal opponents without building explicit models for it. L2E can quickly adapt to new opponent with unknown style using only a few interactions.
We propose to an adversarial training procedure to generate challenging opponents tomatically. These hard to exploit opponents help L2E eliminate its weakness and improve its robustness effectively.
2

---

Under review as a conference paper at ICLR 2021
Opponent strategy generation
B
Hand-OSG
obsain adupted policy
Minimize
B
Oyponent policy
Diverse-OSG
aardiimm update
B
O b-
Maximize
obtain aciapted policy
Baso policy
Bo,
gradient update
Base policy training
Figure 1:
The overview of our proposed L2E framework. The entire training process is based on the idea of adversarial learning (Alg. D The base policy training part maximizes the base policy's ss adaptability by continually interacting with opponents of different strengths and styles (Section 2.Tr. The opponent strategy generation part first generates hard-to-explot opponents for the current base policy (Hard-OSC Baard see Section I. then generates diverse opponent policies improve the gen- eralization ability of the base policy (Diverse-OSC see Section (.2.2). The resulting base policy can fast adapt to completely new opponents with a few interactions.
We further propose a reguuaii-ee policy optimization procedure generate di- verse opponents automatically. The generalization ability of L2E is improved significantly by training with these diverse opponents.
We conduct detailed experiments evaluate the L2E framework in three different environ- ments. The experimental results demonstrate that the base policy trained with 1.2E quickly exploits wide range of opponents compared to other algorithms.
2 METHOD
In this paper, propose novel L2E framework to endow the agents to adapt to diverse opponents quickly. As shown Fig. [ L2E mainly consists of two modules, i.e., the base policy training part, and the opponent strategy generation part. In the base policy training part, our goal is find base policy that, given the unknown opponent, can fast adapt to it by using only a few interactions. To this end, the base policy is trained be able adapt to many opponents. In specific, giving an opponent O, the base policy B first adjusts itself to obtain an adapted policy B' by using a little interaction data between O and B, the base policy is then optimized maximize the rewards that B' gets when facing O. In other words, the base policy has learned how to adapt to its opponents and exploit them quickly.
The opponent strategy generation provides the base policy training part with challenging and diverse training opponents automatically. First, our proposed opponent strategy generation (OSG) algorithm can produce difficult to exploit opponents. In specific, the base policy B first adjusts itself to obtain an adapted policy B' by using a little interaction data between and B, the opponent is then optimized to minimize the rewards that B' gets when facing O. The resulting opponent O is hard to exploit since even the base policy adapts to O, the adapted policy B' can not take advantage of O. By training with these hard to exploit opponents, the base policy can eliminate its weakness and improve its robustness effectively. Second, our OSG algorithm can further produce diverse training opponents with novel riversity policy optimization procedure. More specifically,
3

---

Under review as a conference paper at ICLR 2021
first formalize the difference between opponent policies the difference between the distribution of trajectories induced by each policy. The difference between distributions can be evaluated by the Maximum Mean Discrepancy (MMD) metric (Gretton et al.) 2007). Then, MMD is integrated as a regularization term in the policy optimization process to identify various opponent policies. By training with these diverse opponents, the base policy can improve its generalization ability significantly, Next, we introduce these two modules in detail.
2.1 BASE POLICY TRAINING
Our goal is to find a base policy B that can fast adapt to an unknown opponent O by updating the parameters of B using only few interactions between B and O. The key idea is to train the base policy B against many opponents to maximize its payoffs by using only a small amount of interactive data during training, such that it acquires the ability exploit different opponents quickly. In effect, our L2E framework treats each opponent as a training example. After training, the resulting base policy B can quickly adapt to new and unknown opponents using only few interactions. Without loss of generality, the base policy B is modeled by deep neural network in this work, i.e., a parameterized function To with parameters 6. Similarly, the opponent O for training is also deep neural network ns with parameters db We model the base policy as playing against an opponent in a two-player Markov game (Shapley) [153).. This Markov game M (S, (Ap, Ao), (Rp, Ro)) consists of the state space S, the action space AB and Ao, and a state transition function T x AB x Ao + (S) where A (S) is a probability distribution on S. The reward function R; S x Ag x Ao S + R for each player E {B, O} depends on the current state, the next state and both players' actions. Given a training opponent O whose policy is known and fixed, this two-player Markov game M reduces to single-player Markov Decision Process (MDP), ie., Mg = (S. AB, TO R2)) The state and action space of Mg are the same as in M. The transition and reward functions have the opponent policy embedded:
T (s, un) (ss, un, ao), RR un, 5') = Rn(s, an, ao,8'); 8'),
where the opponent's 's action sampled from its policy ao ~ TT(- s). Throughout the paper, M Y represents a single-player MDP, which is reduced from a two-player Markov game (i.e., player X and player Y). this MDP, the player Y is fixed and can be regarded as part of the environment.
Suppose a set of training opponents { given. For each training opponent O₁, an MDP M can be constructed as described above. The base policy B, i.e., Ty is allowed to query limited number of sample trajectories T to adapt to O,, In our method, the adapted parameters go: of the base policy are computed using one or more gradient descent updates with the sample trajectories T. For example, when using one gradient update:
go: = V (Ta),
(1)
cE (TO) = E Mo
0: E, R (s (t) (t--)))]
(2)
MD represents that the trajectory
(8(1), (1)
aB B (22) (1) a s((+1) ..... is sampled
(1)
from the MDP MD. where 11++1) ~ (s(1), (t)
ag and B
To (()))
We use Bd, to denote the updated base policy, i.e., Tgo, Bo, can be seen as an opponent-specific policy, which is updated from the base policy through fast adaptation. Our goal is to find gener- alizable base policy whose opponent-specifie policy 8 can exploit its opponent O, as much as possible. To this end, we optimize the parameters A of the base policy to maximize the rewards that BO: gets when interacting with O: More concretely, the learning to exploit objective function is defined as follows:
= min Σ CO (Too,) = mijn Σ N
Cgo, (T (Te-avedy
It worth noting that the optimization is performed over the base policy's parameters 0, whereas the objective is computed using the adapted based policy's 's parameters 00. The parameters @ of the base policy are updated as follows:
8-0- 3V8
\=== 1=1 Co. (To..)
(4)
4

---

Under review as a conference paper at ICLR 2021
In effect, our L2E framework aims to find base policy that significantly exploit the opponent with only few interactions with (i.e,, with a few gradient steps). The resulting base policy has learned how adapt different opponents and exploit them quickly. An overall description of the base policy training procedure is shown Alg. The algorithm consists of three main steps. First, generating hard to exploit opponents through the Hard-OSG module. Second, generating diverse opponent policies through the Diverse-OSG module. Third, training the base policy with these opponents obtain fast adaptability.
2.2 AUTOMATIC OPPONENT GENERATION
Previously, we assumed that the set of opponents had been given. How to automatically gener- ate effective opponents for training is the key to the success of our L2E framework. The training opponents should be challenging enough (i.e., hard to exploit). By learning to exploit these hard-to- exploit opponents, the base policy can eliminate its weakness and become more robust, Besides, they should be sufficiently diverse. The more diverse they are, the stronger the generalization ability of the resulting base policy. We propose a novel opponent strategy generation (OSG) algorithm to achieve these goals.
2.2.1 HARD-TO-EXPLOT OPPONENTS GENERATION
We use the idea of adversarial learning to generate challenging training opponents for the base policy B. From the perspective of the base policy B, giving an opponent O, B first adjusts itself to obtain an adapted policy, i.e., the opponent-specific policy 8º the base policy is then optimized to maximize the rewards that D gets when interacting with O. Contrary to the base policy's 's goal, we want to find a hard-to-exploit opponent Ô for the current base policy B, such that even if B adapts to Ô, the adapted policy B cannot take advantage of Ô. In other words, the te-pllot- opponent Ô is trained to minimize the rewards that B gets when interacting with Ô The base policy attempts to increase its adaptability by learning to exploit different opponents, while the hard- to-exploit opponent adversarially tries to minimize the base policy's s adaptability, i.e., maximize its counter-adaptabiliyy.
More concretely, the hard-to-exploit opponent Ô is also deep neural network #3 with randomly initialized parameters G. At each training iteration, an MDP MÒ can be constructed. The base policy B first query a limited number of trajectories to adapt to Ô. The parameters @ of the adapted policy BÔ are computed using one gradient descent update,
0ō_ = 0 - V CO
(5)
(T((T)) -E E, R
( .a s(+t)))
The parameters e of Ô is optimized to minimize the rewards that B gets when interacting with Ô. This is equivalent maximize the rewards that Ô gets since we only consider the competitive setting in this work. More concretely, the parameters @ are updated as follows:
Bd (TT)) -= a
C (T
r' Σ,
Rg (s 10
((1)))
(8)
After several rounds of iteration, we can obtain hard-to-exploit opponent σ3 for the current base policy B. An overall description of this procedure is shown in Alg.
2.2.2 DIVERSE OPPONENTS GENERATION
Training an effective base policy requires not only the hard-to-exploit opponents, but also diverse opponents of different styles. The more diverse the opponents used for training, the stronger the generalization ability of the resulting base policy. From human player's perspective, the opponent style is usually defined as different types, such as aggressive, defensive, clusive, etc. The most sig- nificant difference between opponents with different styles lies in the actions taken in the same state.
5

---

Under review as a conference paper at ICLR 2021
Take poker an example; different opponents' styles tend to take different actions when holding the same hand. Based on the above analysis, we formalize the difference between opponent policies the difference between the distribution of trajectories induced by each policy when interacting with the base policy. We argue that differences trajectories better capture the differences between different opponent policies.
Formally, given base policy B. i.e., Tg and an opponent policy O,, i.e., Toi our diversity- regularized policy optimization algorithm is to generate a new opponent Oj, i.e., Thy whose style different from O₁- We first construct two MDPs, i.e., M and M and then sample two sets of trajectories, i.e., T; = {T MŁ.) and T = {T M } from this two MDPs. The stochas- ticity in the MDP and the policy will induce distribution over trajectories. We use the Maximum Mean Discrepancy (MMD) (Gretton et al. 2007) metric (c.f. Appendix[Cite for details) to measure the differences between T; and Ty:
(T,, T3) = E (T, M k 2-, of Mo k(T, r')
+ B & (T,T')
(9)
k is the Gaussian radial basis function kernel defined over pair of trajectories:
k(T,T') exp( (l((() - g(T))11
(10)
2
where I stacks the states and actions of a trajectory into vector. For trajectories with different length, we clip the long trajectory to the same length as the short one. There overall objective function of our proposed
policy optimization algorithm is as follows:
(0) c (の,) = -E E. 7" RB, (s .a,, ((++)) ammd (T,D((T,,T)).
(11)
The first term is to maximize the rewards that O2 gets when interacting with the base policy B. The second term measures the difference between Ô, and the existing opponent O,. By this diversity- regularized policy optimization, the resulting opponent Oj is not only useful in performance but also diverse relative to the existing policy.
We can iteratively apply the above algorithm to find set of N distinct and diverse opponents. In specific, subsequent opponents are learned by encouraging diversity with respect to previously generated opponent S. The distance between opponent Om and opponent set S is defined by the distance between On and O,,, where O,, E S is the most similar policy to Om- Suppose we have obtained set of opponents can be obtained by optimizing:
{0m] m0 M N. The M + 1-th opponent, i.e., Tøxes1
L (6M+1) = -B
Σ R (0(0) Bo ((++)))] -
min MMD* TM41).
M141
O.ES
(12)
By doing so, the resulting M + 1-th opponent remains diverse relative the opponent set S. An overall description of this procedure is shown in Alg.
3 EXPERIMENTS
In this section, we conduct extensive experiments evaluate the proposed L2E framework. We evaluate algorithm performance on the Ledue poker, the BigLeduc poker and Grid Soccer envi- ronment, which are the commonly used benchmark for opponent modeling (Lanctot et al. 2017] Steinberger 2019 He et al. 2016). We first verify the trained base policy using our L2E framework can fast exploit wide range of opponents with only few gradient updates. Then, compare with other baseline methods to show the superiority of our L2E framework. Finally, we conduct series of ablation experiments to demonstrate each part of our L2E framework's effectiveness.
3.1 RAPID ADAPTABILITY
In this section, we verify the trained base policy's 's ability quickly adapt to different opponents in the Ledue poker environment (c.f. Appendix D for details). We provide four opponents with different styles and strengths. 1) The random opponent randomly takes actions whose strategy is

---

Under review as a conference paper at ICLR 2021
relatively weak but hard to exploit since it does not have evident decision decision-making style. 2) The call opponent always takes call actions and has a fixed decision- decision-making style that is easy to exploit. 3) The rocks opponent takes actions based on its hand-strength whose strategy is relatively strong. 4) The oracle opponent a cheating, and the strongest player who can see the other players' hands and make decisions based on this perfect information. As shown in Fig. [ the base policy achieves
Random opponent
Call opponent
Rocks opponent
Oracle opponent
Awerege 0.5
-1.25
0.2
-3.50
Number adaptation steps
The trained base policy using our L2E framework can quickly adapt to different opponents of different styles and strengths in the Leduc poker environment.
Figure 2:
rapid increase in its average returns with few gradient updates against all four opponent strategies. For the call opponent, which has a clear and monotonous style, the base policy can significantly exploit it. Against the random opponent with clear style, the base policy can also exploit quickly. When facing the strong rocks opponent or even the strongest oracle opponent, the base policy can quickly improve its average returns. One significant advantage of the proposed L2E framework is that the same base policy can exploit wide range of opponents with different styles, demonstrating its strong generalization ability.
3.2 COMPARISONS WITH OTHER BASELINE METHODS
Random
Rocks
Oracle
L2E MAML Random TRPO
0.42+0.32
1,340014
0,38
-0.03=0.14
-1.15:00.27
1.27:10.17
-023310.22
-142210.7
-0.77-0.23
-29310.17
-0.02=3.77
-0.0233.31
-0.6823.75
-0.74-4.26
-1,90:24.78
0.07+0.08
-0.22+0.09
-07740.12
-042-0.07
-1.9640.46
TRPO (pretrained)
0.15-40.17
-0.0540.14
00700-0.27
.0.61-033
-1.32-00.27
BOM (Explicit Opponent Modeling)
0:30+0.15
-0.01:0.05
-0.1340.20
-0.3-0.111
-1,8240.28
Table 1: The average return of each method when performing rapid adaptation against different
opponents in the Ledue Poker environment. The adaptation process is restricted to a threc-step gradient update.
As discussed in Section 1 most previous opponent modeling methods require constructing explicit opponent models from a large amount of data before learning to adapt to new opponents. To the best of our knowledge, our L2E framework is the first attempt to use meta-learning to learn to exploit opponents without building explicit opponent models. To demonstrate the effectiveness of the L.E framework, we design several competitive baseline methods. As with the previous experiments, we also use three gradient updates when adapting to a new opponent. 1) MAML. The seminal meta-learning algorithm MAML (Finn et al. 2017) is designed for single-agent environments. We have redesigned and reimplemented the MAML algorithm for the two-player competitive environ- ments. The MAML baseline trains base policy by continually sampling the opponent's ss strategies, either manually specified or randomly generated. 2) TRPO. The TRPO baseline does not perform pre-training and uses the TRPO algorithm (Schulman et al.) 2015) to updated its parameters via three-step gradient updates to adapt to different opponents. 3) Random. The Random baseline is neither pre-trained nor updated online. To evaluate different algorithms more comprehensively. we additionally add new Nash opponent. This opponent's 's policy is a part of an approximate Nash Equilibrium generated iteratively by the CFR (Zinkevich et al. 2008) algorithm. Playing a strategy from Nash Equilibrium in a two-player zero-sum game is guaranteed not to lose in expectation even the opponent is the best response strategy when the value of the game is zero. We show the performance of the various algorithms in Table [ It is clear that L2E maintains the highest profitability against all four types of opponents other than the random type. L2E can exploit the opponent with evident style significantly, such as the the Call opponent. Compared to other baseline methods, L2E achieved the highest average return against opponents with unclear styles, such as the Rocks opponent, the Nash opponent, and the cheating Oracle opponent.
7

---

Under review as a conference paper at ICLR 2021
3.3
ABLATION STUDIES
3.3. 1 EFFECTS OF THE DIVERSIT-REGULARIZED POLICY OPTIMIZATION
Flop strateg es without dventyy
Flop strategpes with diveriyy
80522
8.15
E10
0.33
6.00
800
63G
15 de Geoooieeenteelee
Ar Groarde 19 00000 Koree
10 Gro00d Morti
Op 20000 RPAD 2/ 2r0000 oonirr
Figure 3:
Visualization of the styles of the strategies generated with or without the MMD regular- ization term in the Leduc poker environment.
In this section, we verify whether our proposed diversity-regularried policy optimization algorithm can effectively generate policies with different styles. In Ledue poker, hand-action pairs represent different combinations possibilities, i.e., J, Q. and K. Meanwhile, each player also has three optional actions, i.e., Call (c), Rise (r), and Fold (f). For example, 'Je' means to call when getting the jack. Action probability is the probability that player will take corresponding action with particular hand. Fig. 3 demonstrates that without the MMD regularization term, the two sets of strategies generated both the pre-flop and flop phases have similar styles. By optimizing with the MMD regularization term, the generated strategies are diverse enough which cover a wide range of different states and actions.
hands and actions. the pre-flop phase, each player's hand has three
3.3.2 EFFECTS OF THE HARD-OSG AND THE DIVERSE-OSG
As discussed previously, crucial step in L2E is the automatic generation training opponents. The Hard- OSG and Diverse-OSG modules are used gener- ate opponents that are difficult to exploit and diverse in styles. Fig. shows the impact of each mod- ule on the performance of L2E. 'L,2E w/o counter' is L2E without the Hard-OSG module. Similarly, 'L2E w/o diverse' is L2E without the Diverse-OSG module.
a/s clerreoosuee Li clnvia
Average normalied
'L2E w/o diverse&counter" removes both modules al- together. The results show that both Hard-OS and Diverse-OSG have a crucial influence on L2E's per- formance. It is clear that the Hard-OSG module helps to enhance the stability of the base policy, and the improve the base policy's
Numoee adaptation steps
Figure 4: Each curve shows the aver- age normalized returns of the base policy trained with different variants of L.EE the grid soccer environment.
Diverse-OSG module performance significantly. To further demonstrate the generalization ability of L2E, we conducted a series of additional experiments on the BigLeduc poker and Grid Soccer game environment (c.f. Appendix E and Appendix forr details).
4 CONCLUSION
We propose learning to exploit (L2E) framework to exploit sub-optimal opponents without build- ing explicit opponent models. 1.2E acquires the ability exploit opponents by a few interactions with different opponents during training, so that it adapts to new opponents during testing quickly. We propose a novel opponent strategy generation algorithm that produces effective training oppo- nents for L2E automatically. We first design adversarial training procedure to generate challeng- ing opponents to improve L2E's robustness effectively. We further exploit a diversity- regersityreeullarized policy optimization procedure generate diverse opponents to improve L2E's generalization abil- ity significantly. Detailed experimental results in three challenging environments demonstrate the effectiveness of the proposed 1.2E framework.
8

---

Eddie Dekel, Drew Fudenberg, and David K Levine. Learning to play bayesian games. Games and Under review as a conference paper at ICLR 2021
REFERENCES
Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv: 13/96, 2019.
Alexandros Agapitos, Julian Togelius, Simon M Lucas, Jurgen Schmidhuber, and Andreas Konstan- tinidis. Generating diverse opponents with multiobjective evolution. In IEEE Symposium On Computational Intelligence and Games, pp. 135-142, 2008.
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, llya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In In- ternational Conference on Learning Representations, 2018.
Stefano Albrecht and Peter Stone. Reasoning about hypothetical agent behaviours and their pa- rameters. In International Conference Autonomous Agents and Multiagent Systems, pp. 547- 2017.
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-99 2018.
Dipyaman Banerjee and Sandip Sen. Reaching pareto-optimality in prisoner's dilemma using con- ditional joint action learning. Autonomous Agents and Mulli-Agent Systems, 15(1):91-108, 2007.
George W Brown. Iterative solution of games by fictitious play. Activity analysis of production and allocation, 13(1):374-376, 1951.
Harmen de Weerd, Rineke Verbrugge, and Bart Verheij. Negotiating with other minds: the role of recursive theory of mind negotiation with incomplete information. Autonomous Agents and Multi-Agent Systems, (022)225--287, 2017.
Economic Behavior, 46(2):282-33,, 2004.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. International Conference on Learning Representa- fions, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135, 2017.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In International Conference on Au- tonomous Agents and MultiAgent Systems, pp. 122-130, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, et al. Noisy networks for exploration. In International Conference Learning Representations, 2018.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the two-sample-problen In Advances in neural information processing systems, pp- 513-520, 2007.
Barbara Grosz and Candace L Sidner. Attention, intentions, and the structure of discourse. Compu- tational linguistics, 1986.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning. arXiv preprint PrXiv:1806 104.06000000000000, 2018.
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein-
forcement learning. In International conference on machine learning, pp. 1804-1813, 2016.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leariing in neural networks: survey, arXiv preprint arXiv:2004.05439, 2020.

---

Under review as a conference paper at ICLR 2021
Trung Dong Huynh, Nicholas R Jennings, and Nigel R Shadbolt. An integrated trust and reputation model for open multi-agent systems. Autonomous Agents and Multi-Agent Systems, 13(2):119- 154. 2006.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based train- ing of neural networks. arXiv preprint arXiv: /71 1.09846, 2017.
Max Jaderberg, Wojeiech M Czamecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-base based reinforcement learning, Sci- ence, 364((443)):859 863, 2019.
Peter A Jarvis, Teresa Lunt, and Karen L Myers. Identifying terrorist activity with ai plan recog- nition technology. Al magazine, 26(3):73-73, 2005.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent inforcement learning. Advances in neural information processing systems, pp. 4190-4203, 2017.
Gordon McCalla, Julita Vassileva, Jim Greer, and Susan Bull. Active learner modelling. Interna- lional Conference on Intelligent Tutoring Systems, pp. 53-62, 2000.
Richard Mealing and Jonathan Shapiro. Opponent modeling by pnpectation-mxximinen and sequence prediction in simplified poker. IEEE Transactions on Computational Intelligence and Al in Games, 9(1):11-24, 2015.
Christian Muise, Vaishak Belle, Paolo Felli, Sheila Mcllraith, Tim Miller, Adrian R Pearce, and Liz Sonenberg. Planning over multi-agent epistemic states: A classical planning approach. In AAAI Conference on Artificial Intelligence 2015.
John H Nachbar. Beliefs in repeated games. Econometrica, 73(2):459-43) 2005.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta- eeta-learning algorithms. arXiv preprint arXiv:1803.0299) 2018.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040-2042, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce-
ment learning. In International Conference Machine Learning, pp. 2817-2826. 2017.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018
Rob Powers and Yoav Shoham. Learning against opponents with bounded memory. In International joint conference on Artificial intelligence, pp. 817-822, 2005.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference Machine Learning, pp- 4218-4228 2018.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences. 39(10): 1095-1100, 1953.
Eric Steinberger. Single deep counterfactual regret minimization. arXiv preprint arXiv: 1901.0721, 2019.
10

---

Under review as a conference paper at ICLR 2021
Gita Sukthankar and Katia Sycara. Policy recognition for multi-player tactical scenarios. In Inter- national joint conference on Autonomous agents and multiagent systems, pp. 1-8, 2007.
Gabriel Synnaeve and Pierre Bessiere. A bayesian model for opening prediction in rts games with application to starcraft. In IEEE Conference on Computational Intelligence and Games, pp. 281- 288, 2011.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33 2019.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint
2016.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet):
Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint 1rXiv:1901.01555 2019.
Ben Weber and Michael Mateas. A data mining approach to strategy prediction. In IEEE Sympo- sium on Computational Intelligence and Games, pp. 140-147, 2009.
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representations, 2018.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396-2407, 2018.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In Advances in neural information processing systems, pp. 1729-1733 2008
A RELATED WORK
A. OPPONENT MODELING
Opponent modeling is a long-standing research topic in artificial intelligence, and some the ear- liest works back to the early days of game theory research (Brown) T95T). The main goal of opponent modeling is to interact more effectively with other agents by building models to reason about their intentions, predicting their next moves other properties (Albrecht & Stone 2018). The commonly used opponent modeling methods can be roughly divided into four categories: policy reconstruction, classification, type-bases reasoning and recursive reasoning. Policy reconstruction methods (Mealing & Shapiro 2015) reconstruct the opponents' decision making process by build- ing models which make explicit predictions about their actions. Classification methods (Weber & Mateas 2009 Synnaeve & Bessiere 201 produce models which assign class labels (e.g.. "aggres- sive" or "defensive")
the opponent and employ a precomputed strategy which effective against that particular class of opponent. Type-based reasoning methods (He et al. [0166] Albrecht & Stone 2017) assume that the opponent has one of several known types and update the belief using the new observations obtained during the real-time interactions. Recursive reasoning based methods (Muise) ct al., 2015 de Weerd et al., 2017) model the nested beliefs (e.g., "I believe that you believe that believe..." ") and simulate the reasoning processes of the opponents to predict their actions. Different from these existing methods which usually require a large amount interactive data to generate useful opponent models, L2E framework does not explicitly model the opponent and acquires the ability to exploit different opponents by training with limited interactions with different styles of opponents.
11

---

Under review as a conference paper at ICLR 2021
A.2 META-LEARNING
Meta-leaming is a new trend of research in the machine learning community which tackles the problem of learning to learn (Hospedales et al. 2020). leverages past experiences the training phase to learn how to learn, acquiring the ability to generalize to new environments new tasks. Recent progress in meta-lcarning has achieved impressive results ranging from classification and regression in supervised learning (Finn et al. 2017: Nichol et al., 2018) reinforcement learning (Wang et al. 2016 Xu et al. 2018). Some recent works have also initially explored the application of meta-learning
task adaption in
opponent modeling. For example, the theory of mind network (ToMnet) (Rabinowitz et al.) 2018) uses meta-learning to improve the predictions about the opponents' future behaviors. Another related work (Al-Shedivat al. 2018)) uses meta-learning to handle the non- non- staationtonarty problem in the multi-agent interactions. Different from these methods, we focus on how to improve the agents' ability to quickly adapt to different and unknown opponents.
A.3 STRATEGY GENERATION
The automatic generation of effective opponent strategies for training with is a critical step in our approach, and how to generate diverse strategies has been preliminarily studied in the reinforce- ment learning community. In specific, diverse strategies can be obtained in a variety of ways, in- cluding adding some diversity regularization to the optimization objective (Abdullah et al. 2019).
randomly searching in some diverse parameter space (Plappert et al. 2018 Fortunato et al. 2018
using information baaed strategy proposal (Eysenbach et al. 2018 Gupta et al. (218)) and search-
ing diverse strategies with evolutionary algorithms (Agapitos et al. 2008 Wang et al. 2019 Jader-
berg et al. 2017 2019. More recently, the researchers from Deep.Mind propose a league training
paradigm obtain Grandmaster level StarCraft II AI (i.e., AlphaStar) by training diverse league of clorntinually adapting strategies and counter counter-strategies (Vinyals et al.) 2019). Different from
AlphaStar, our opponent strategy generation algorithm exploits adversarial training and diversity-
regularized policy optimization to produce challenging and diverse opponents respectively.
B ALGORITHM
The base policy training procedure of our L2E framework. Input: Step size hyper parameters o,,3; base policy B with parameters 0: opponent policy with parameters ø.
Algorithm 1:
Output: An adaptive base policy B with parameters randomly initialize 0.6
initialize policy pool M = {0):
for 1 < e I epochs do
Hard-OSG(B)
(ssee Alg. 2
P -DiverseOOGGBB N) (ssee Alg
Update opponent policy pool M = MUP;
Sample batch of opponents O, ~ M:
for Each opponent O, do
Construct single-player MDP MD.
Sample trajectories using B against fixed opponent O
Use Eqn. to update the parameters of B to obtain an adapted policy BO. Resample trajectories r' using B against O,;
Update the parameters 0 of B according to Eqn. 4
C MAXIMUM MEAN DISCREPANCY
We use the Maximum Mean Difference (MMD) Gretton et al. 2007) metric to measure the differ- ences between the distributions of trajectories induced by different opponent strategies.
12

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Hard-OSG, the hard-to-exploit training opponent generation algorithm.
Input: The latest base policy with parameters 0.
Output: A hard-to-exploit opponent Ô for B.
Randomly initialize O's parameters ;; for 1 < i S epochs do
Construct single-player MDP Mº:
Sample small number of trajectories T ~ Mg using B against Ō
Use Eqn. update the parameters of B to obtain adapted policy B²;
Sample trajectories 7' M g using Ô against D
Update the parameters @ of Ô according Eqn. 7
Algorithm 3: Diverse-OSG, the proposed diversity-reegularized policy optimization algorithm to generate diverse training opponents.
Input: The latest base policy B, an exsiting opponent O₁, the total number of opponents that to be generated N.
Output: A set of diverse opponent 5 = {0m L.
Initialize the opponent set S {01}
for i = 2 to N do
Randonly initialize an opponent O,'s parameters di for 1 < S sleps do
Calculate the objective function (Ø:) according to Eqn. 12
S Calucute the gradient L. (())) (cff., Appendix C for details)
Use this gradient to update Φ;
Update the opponent set S, i.e., S = SUO;
Definition Let F be a function space f : - RR. Suppose we have two distributions and q.
X = (f,,..., Fm p. Y := {ui,-.., .... Vn q. The MMD between p and q using test functions
from the function space F is defined as follows:
MMD[F.P.dl = sup (E (Ez~pIf(x)] - -yrrgg/f JEF
(13)
we can pick suitable function space F, we get the following important theorem (Gretton al. 2007
Theorem Let = (ʃ || S 1) be a unit ball in Reproducing Kernel Hilbert Space (RKHS). Then MMD[F, p.q] = 0 if and only if q.
So the MMD distance between two strategies is 0 when the distributions of trajectories induced by them are identical. To obtain set of strategies with diverse styles, should increase the MMD distances between different strategies, xp is a feature space mapping from x to RKHS, we can easily
Algorithm 4:
The testing procedure of our L2E framework.
Input: Step size hyper parameters ot; The trained base policy with parameters 0; an unknown opponent O.
Output: The updated base policy Bº that has been adapted to the opponent O. Construct single-player MDP MO
for 0 < step S steps do
Sample trajectories T from the MDP MD: :
Update the parameters 0 of B according to Eqn. [
13

---

Under review as a conference paper at ICLR 2021
calculate the MMD distance using the kernel method k(x,x')= (f(xx), mx
MMD² (F.p.)
||x (X) - Ey (
(14)
= (Ex~pp(X) By (Y), Ex Exxpf(X) - By xY))
-Ex,x (X, 2I xIx ((X, Y) + gy (X Y')
The expectation terms in Eqn. can be approximated using samples:
MMD x, Y m Eifj k (1i4)
(15)
n(n-1) Dizy Ui. mn
Σn (Ts, Mj)
The gradient of the MMD term with respect to the policy's parameter oj in our L2E framework can be calculated as follows:
V MMD (T,, T2) = V ( Mo. M ))
Er, [k (T, V log(p(r)p(7'))
(16)
22, M8, 5 r' ik (T, Vo, log(p(r))(7')) + Mg 1b (T, Vo, log(p(r))(7)))
where p(r) is the probability of the trajectory. Since T; (T M9.1 J. O, is the known opponent policy that has no dependence on Øj. The gradient with respect to the parameters Øj in first term is The gradient of the second and third terms can be easily calculated as follows:
Va log(p(T)) = =- L= Vay log nay (0r(8)).
(17)
D LEDUC POKER
The Leduc poker generally uses a deck of six cards that includes two suites, each with three ranks (Jack, Queen, and King of Spades, Jack, Queen, and King of Hearts). The game has a total of two rounds. Each player is dealt with private card in the first round, with the opponent's deck information hidden. In the second round, another card is dealt with as a community card, and the information about this card is open to both players. If a player's private card is paired with the community card, that player wins the game; otherwise, the player with the highest private card wins the game. Both players bet one chip into the pot before the cards are dealt. Moreover, betting round follows at the end of each dealing round. The betting wheel alternates between two players, where each player can choose between the following actions: call, check, raise, fold. If player chooses to call, that player will need to increase his bet until both players have the same number of chips. If one player raises, that player must first make up the chip difference and then place an additional bet. Check means that a player does not choose any action on the round, but can only check if both players have the same chips. a player chooses to fold, the hand ends, and the other player wins the game. When all players have equal chips for the round, the game moves on to the next round. The final winner wins all the chips in the game.
Next, we introduce how to define the state vector. Position in a poker game is critical piece of information that determines the order of action. We define the button (the pre-flop first-hand position), the action position (whose turn it is to take action), and the current game round as one dimension of the state, respectively. In Poker, the combination of a player's hole cards and board cards determines the game's 's outcome. We encode the hole cards and the board cards separately. The amount of chips is an essential consideration in a player's decision -aaiiiiinnmminngggggggg process. We encode this information into two dimensions of the state. The number of chips in the pot can reflect the action history of both players. The difference in bets between players in this round affects the choice of action (the game goes to the next round until both players have the same number of chips). In summary, the state vector has seven dimensions, i.e., button, to act, round, hole cards, board cards, chips call, and pot.
14

---

Under review as a conference paper at ICLR 2021
E BIGLEDUC POKER
We use a larger and more challenging BigLeduc poker environment further verify the effectiveness of our L2E framework. The BigLeduc poker has the same rules Leduc but uses a deck of 24 cards with 12 ranks. addition to the larger state space, BigLeduc allows a maximum raises per round. As shown in Fig. L2E still achieves fast adaptation to different opponents. In comparison with other baseline methods, L2E achieves the highest average return Table 2
instead
Random
Call
Raise
Oracle
L2E
0.82++0.28
0.74:10.22
0.68
-1.02
MAML
0.77=20.30
0.1710008
-2.02:1.999
-1.21
Random
-0.00:1..08
-0.00-12.78
-2.8315.25
-1.88:14.28
TRPO TRPO (pretrained)
0.19+0009
0.10:0.116
-2.22.0.71
-1,4240.47
0.36
0.23
-2.03+1.61
-1.69:0.69
EOM
0.56:10.43
0.15
-1.15+00.12
-1.63+00.2
Table 2: The average return of each method when performing rapid adaptation against different opponents in the BigLedue poker environment.
Random opponent
Call opponent
Raise opponent
Oracle opponent
Averete return
3.0
1.0
-0.75
2.0
0.8
-1.00
0.6
0.6
0.5
-1.25
0.4
Number edeettaton steps
Figure 5: The trained base policy using our 1.2E framework can quickly adapt to different opponents of different styles and strengths in the BigLeduc poker environment.
F GRID SOCCER
This game contains a board with a x grid, two players, and their respective target areas. The position of the target area is fixed, and the two players appear randomly in their respective areas the start of the game. One of the two players randomly has the ball. The goal of all players is to move the ball to the other player's target position. When the two players move the player with the ball loses the ball. Players gain one point for moving the ball to the opponent's area. The player can move in all four directions within the grid, and action invalid when moves the boundary.
grid, the
We train the L2E algorithm in this soccer environment in which both players are modeled by neural network. Inputs to the network include information about the position of both players, the position of the ball, and the boundary. We provide two types of opponents test the effectiveness of the resulting base policy. 1) A defensive opponent who adopts a strategy of not leaving the target area and preventing opposing players from attacking. An aggressive opponent who adopts a strategy of continually stealing the ball and approaching the target area with the ball. Facing a defensive opponent won't lose points, but the agent must learn to carry the ball and avoid the opponent moving to the target area score points. Against aggressive opponent, the agent must leam to defend at the target area to avoid losing points. Fig. 6 shows the comparisons between L2E, MAML, and TRPO. L2E adapts quickly to both types of opponents; TRPO works well against defensive opponents but loses many points against aggressive opponents; MAML is unstable due to its reliance on task specification during the training process.
15

---

Under review as a conference paper at ICLR 2021
Aggressive opponent
Defensive opponent
Lzt MANL TRIO No adephatioon
26
Avcroge retur
-0.
-08
Number udaplation steps
Figure 6: The trained base policy using our L2E framework can quickly adapt to opponents with different styles in Grid Soccer environment.
G
CONVERGENCE
Since convergence in game theory is difficult to analyze theoretically, we have designed a series of small-scalc experiments to empirically verify the convergence L2E with the help of Rock-Paper- (RPs)) game. There are several reasons why RPS game is chosen: to analyze due to the small state and action space. 2. RPS game is easy to visualize and analyze due the small state and action space. 3. RPS game is often used in game theory for theoretical analysis. The experiments we designed contains the following parts: 1. Testing the adaptability of Policy Gradient (PG), Self Play (SF), and L2E by visualizing the adaptation process. Analyzing the relationship between L2E strategy and Nash strategy. 3. Analyzing the convergence of L2E.
RPS game is easy
A
c
Figure 7: Policy Gradient Optimize iteratively for the initial opponent (the orange dot), eventu- ally converging the best response of the initial opponent's 's strategy. B. Self Play Each iteration seeks the best response to the previous round of strategy, which does not converge in an intransitive game like RPS. C. L2E's adaptation process when facing policy's 's adaptation process when facing a new opponent (the orange dot).
opponent (the orange dot). D. Nash
As shown in Fig. 7 we can draw the following conclusions: 1. PG eventually converged to the best response, but it took dozens of gradient descent steps our experiments (Each blue dot represents a ten-step gradient descent). SP failed to converge in the RPS game due to the intransitive nature of Peeessors>Paerr>>Pe Paper>Rocck Rock). In contrast, our L2E quickly converged to the best
the RPS game response strategy (Each blue dot represents one-step gradient descent). 2. The strategy visualiza- tion in the Fig. C shows that the base policy of L2E does not converge to the Nash equilibrium strategy after training but converges the vicinity of the Nash equilibrium strategy. 3. If we fix
16

---

Under review as a conference paper at ICLR 2021
the base policy to the Nash strategy by imitation learning and then adapting it, do not get good results either. This further illustrates the difference between the L2E strategy and the Nash equilib- rium strategy. And the Fig. further shows the performance of L2E and Nash strategy in RPS game when facing new opponents.
Rock- Paper-Scisoorrs
1.0
L2E. Rock
L2E, Paper
0.8
Scissors
L2E, Random
L2Einash| Rock
Average return
0.6
L2Efnash] Paper
L2Einashi Scissors
0.4
L2Einashi, Rondom
0.0
0.2
1
Number adaptation steps
Figure 8: The performance of 1.2E and Nash strategy in RPS game when facing new opponents.
Although theoretically difficult analyze the convergence properties of L2E,, from the exper- imental results in Fig. 9 it can be seen that as the training progresses, L2E's adaptability becomes stronger and stronger. After reaching certain number of iterations, the improvement eventually reaches plateau, which provides some empirical evidence for the convergence of 1.2F.
1.0
step0
step1
step2
0.8
step3
Average return
0.6
0.4
0.2
0.0
20
50
Number epochs
Figure 9: The convergence properties of 1.EE.
17

---

Under review as a conference paper at ICLR 2021
H HYPER-PARAMETERS
The hyper-parameters of all experiments are shown in the table below.
Hyper parameter
Value
Training step size hyper parameters (a,
(0.I.0.01)
Testing step size hyper parameter "Y Number of opponents sampled per butch
0.1
40
Number of trajectories to sample for each opponent
20
Number of gradient steps in the training loop
Numher of gradient steps in the testing loop Policy network size(1 Leduc, Bigl edge, Grid Soccer)
164,64,(44445))
Number of training steps required for convergence in Ledue
300
Number of training steps required for convergence in BigLeduc
400
Number of training steps required for convergence GridSaccer Hard-to-exploit opponent training epochs Diverse opponent training epochs
300
20
50
Weight of the MMD term OMMD Bandwidth of RBF kernel
0.8
Minimum trajectory length N to calculate MMD term
20
Number of opponent strategies generated hy OSG per round of iteration
N555
Number of trajectories sampled to compute MMD term
Table 3: Hyper-parameter of 1.2E.
18


Reviews:
[{"review_id": "NDP0xia_rTS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe authors propose an opponent modeling in 1-vs-1 games called the Learning to Exploit (L2E) framework, which exploits opponents by a few interactions with different opponents during training so that it can adapt to new opponents with unknown styles during testing quickly. In particular, the authors propose Opponent Strategy Generation (OSG) that produces effective opponents for training automatically through adversarial training for eliminating its own strategy’s weaknesses and diversity-regularized policy optimization to improve the generalization ability of L2E. Experimental results of two poker games and one grid soccer game indicate that L2E quickly adapts to diverse styles of unknown opponents.\n\nReasons for score: \nAlthough motivation, solution, novelty, and overall presentation were almost clear, the implementation details were unclear and there were no shared codes. I think the idea contributed to this community, but mainly for the above reproducibility, it is difficult to provide a higher rating. \n\nPros:\n1. Significance: the authors propose an opponent modeling called the L2E framework, which can adapt to unknown opponents quickly with a few observation of interactions\n2. Methodological novelty: the authors propose OSG that produces effective opponents for training automatically through adversarial training and diversity-regularized policy optimization to improve the generalization ability of L2E. \n3. Experiments: the results of two poker games and one grid soccer game indicate that L2E quickly adapts to diverse styles of unknown opponents.\n\nCons:\n1. The implementation details were unclear (e.g., network and learning hyperparameters and their selections) and there were no shared codes. \n2. There were some unclear points in Method and Experiments (below).\n\nOther comments:\n\nIn Eqs. (3) and (4), the theta is updated based on (4) but also updated in computing pi_{theta O_i} described in the right-hand side of eq. (3). The former and the latter seem to update the theta in different manners, but the authors claimed that the theta is updated based on eq. (4). It seems to be confusing for me, but is this correct? \n\nIn Eqs. (7) and (8), notations of the loss and reward should be defined.\n\nThe experimental results were about the Leduc poker task, except for Fig. 4 (the Grid soccer task). There were no similar results of the Leduc poker task also in the Appendix. I would like to know the reason.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fbw4L5tUNqZ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** \nThis paper proposes the Learning to Exploit (L2E) framework that can quickly adapt to diverse opponent's unknown strategies. The main contributions of L2E include: 1. learning of the base model based on the optimization similar to MAML (Finn et al., ICML-17) to adapt to a new opponent after a few learning iterations (Section 2.1), 2. the generation of the hard-to-exploit opponent to robustly train the base model (Section 2.2), and 3. the generation of diverse opponent policies using the maximum mean discrepancy (MMD) metric (Section 2.3). Empirical results show that L2E can exploit diverse opponents in the Leduc poker, BigLeduc poker, and Grid Soccer domains. \n\n**Reasons for Score:**\nOverall, I vote for a score of 5. While the opponent strategy generation (OSG) algorithm with the counter adaptability and diverseness is an interesting idea, I am concerned about inappropriate baseline comparisons (please refer to Concerns and Questions below). After seeing the authors' responses to my concerns, I am open to raising my score.\n\n**Pros:**\n1. OSG removes the requirement of preparing the population or task distribution in meta-learning, which can be expensive.\n2. Section 3.3.1 shows promising results that the proposed MMD regularization term can generate diverse opponents. \n\n**Concerns and Questions:**\n1. OSG, which generates a competitive and diverse opponent population, is a paper's main contribution. However, this paper compares the baselines trained based on the random opponent population, such as the MAML baseline in Section 3.2. Hence, it is unclear how much more effective and diverse opponents that OSG can generate compared to state-of-the-art opponent generation-based algorithms (e.g., population-based RL (Jaderberg et al., Science-19)). \n2. An important claim of the paper is that explicit opponent modeling requires large sample complexity (Section 1). However, an explicit opponent modeling baseline is not compared in the experimental results. Pre-training an explicit opponent modeling method on the same opponent population generated by OSG and then comparing its adaptation against a new opponent will further highlight L2E's fast adaptation performance. \n3. In MAML, a non-meta-learning baseline is compared by pre-training a policy on all of the meta-training tasks and then fine-tuning at a meta-test task (i.e., the \"pretrained\" baseline in MAML). However, the TRPO baseline in this paper does not perform the pre-training (\"The TRPO baseline does not perform pre-training ...\" in Section 3.2). Because L2E's base model is pre-trained using opponents generated by OSG, a more fair comparison is to pre-train the TRPO (possibly based on the random opponents) and then fine-tune against a new opponent. \n4. In Figure 2, the trained base policy receives the negative return playing against the oracle opponent (possibly expected as the oracle opponent takes actions based on perfect information). But, why does L2E receive the positive return in Table 1?\n\n**Additional Feedback:**\n1. In the appendix, hyper-parameters/training details for L2E and the baselines (e.g., the number of trajectories for each adaptation, details on how the meta-training task distribution is constructed for MAML) are missing.\n2. In Tables 1 and 2, adding the variance will be helpful. Specifying the number of random seeds used in the experimental results will also be helpful.\n3. Algorithm 1-3 are for learning the base policy (i.e., the meta-training procedure). Adding an algorithm in the Appendix when adapting to a new opponent after the base policy training (i.e., the meta-testing procedure) will be helpful.\n4. Adding an explanation about why the particular metric of MMD (instead of other metrics) is chosen to compare the difference between distributions will be helpful.\n5. In Section 2.2.2, it is noted that \"For trajectories with different length, we clip the long trajectory to the same length as the short one\" for the MMD calculation. I wonder whether applying the masking based on the done signal from the environment can be better than applying the clipping.\n\n**Reference:**\n\nChelsea Finn, Pieter Abbeel, Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. ICML, 2017.\n\nMax Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, Thore Graepel. Human-level performance in 3D multiplayer games with population-based reinforcement learning. Science, 2019.\n\n**After rebuttal:**\nThe responses address most of my main concerns, and I have increased the rating from 5 to 6. As discussed during the rebuttal, in the future, having additional experiments that compare between OSG and other appropriate population generation baselines would be helpful.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7ge66yTQ95F", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper motivates the problem setting of quickly adapting to exploit sub-optimal opponents. The goal is to find a base policy which is quick to adapt to a range of sub-optimal opponents. To do this they use L2E which can generate exploiter and diverse opponent agents and updates the base policy based on trajectories from exploiter base policies. The paper focuses on the 2-player zero-sum setting where we have many algorithms which can find NE. I am concerned that this procedure is simply finding a NE and the other algorithms compared (MAML and TRPO) are not. I believe the paper needs to construct an argument that 1) L2E is not finding a NE and that 2) NE is not a good base policy.\n\n=== Rebuttal Edit (Increased score form 4 to 6)\n\nThanks for the discussions. The primary reason for my score increase is discovering that the power of the framework is finding a representation that is quick at exploiting new opponents. I have been sufficiently convinced that this approach is not simply finding a NE (a sticking point in my review). I think that there are a collection of ideas here that are publishable and are of interest to the community.\n\nThe reason for not giving a higher score is that I think the points the paper made could be clearer: specifically I think the phrase \"base policy\" could be better replaced by \"base representation\" / \"base model\". I was stuck on the idea that the base policy had to be a strong one (eg a NE), and close to exploiter policies in *policy space* rather than *parameter space*. Re-reading after the paper update, I am worried that a significant portion of readers may fall into the same trap despite the authors' additional edits. Tightening the story would make this paper more appealing. I also broadly agree with the other reviewers suggestions / concerns.\n\nFor future work (also mentioned by another reviewer), I think there is no reason the \"base policy\" could not also be a strong policy too. I believe with minor adjustments to your framework this could be achieved, and one would have a model that both has low exploitability and is fast to adapt to new opponents - a potentially powerful combination.\n\n## Score\n\nI am recommending reject [UPDATE - see above] - although I will maintain an open mind due to my middling confidence in some of the background literature around the paper.\n\n## Positives\n\nHaving a diversity regularized policy optimization procedure with MMD is interesting.\n\nThe PG update rules are interesting.\n\n## Concerns\n\nI am unsure about some of the claims wrt implicit/explicit opponent modelling. It seems that we need the exact opponent and base policy at all times in L2E. The literature that is cited in the paper is in a harder setting than this - where the opponent’s policy has to be estimated. Are we actually modelling opponents?\n\nThe MMD term seems very expensive and I am not sure it will scale, particularly with population size. It is also unclear to me how much sampling is necessary to estimate MMD. This approach does interest me however, perhaps the authors could comment on the scalability of this metric?\n\nIn the experiments section I would like to see more details on the actual training. How many outer and inner iterations were needed to converge? How many diverse and hard opponents were used during training? How many trajectories are sampled? Is learning rate tuning important? I am not sure I could reproduce from the details provided.\n\nThe results in Figure 2 do not seem to match up with the numbers in Table 1.\n\nIt is not clear whether the results are because the L2E procedure finds base policy close to Nash or if it is finding some other interesting policy. Would it be possible to adapt a Nash policy (with 4 adaptation steps) as an additional baseline for Table 1 and Figure 2 to check this? My concern is that the L2E framework is just finding Nash in an exotic way. From Table 1 it is clear the other baselines are not finding Nash. I appreciate that Nash was included as an opponent. I think testing this is key to back some of the claims made in this paper. If this procedure is indeed finding Nash, then is it doing it more efficiently than PSRO (Lanctot 2017) or self-play?\n\nZero-sum two-player settings NE is a reasonable thing to optimize for. Things like fictitious self play (FSP) are known to converge to approximate NE, and this algorithm looks like FSP without running the best response calculation to convergence.\n\nWho is the opponent in Figure 4?\n\n## Other Things\n\nFigure 1: Should there be a loop from B’ back to B? Similar for O’ to O?\n\nPage 2: “The key idea underlying L2E to train...” -> “L2E *is* to train”\n\nPage 5: “only consider competitive agents in this work...” -> More specifically only zero-sum agents.\n\nPage 7: “As with the previous experiments, we also use four gradient updates when adapting to a new opponent” -> The previous experiments used three updates?\n\nPage 7: “Positive returns are also guaranteed against opponents without a clear style…” -> Are they really guaranteed? There are a lot of things like step size stochastic rollouts that make this statement tenuous. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5XfTCcoS51", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper suggests a novel framework (coined L2E) to learn a policy that is optimized to adapt quickly (and exploit) a wide range of unknown opponents.\nTo do so it trains a base policy that is optimized so that it can maximize its expected reward against a variety of opponents using only a few updates of its parameters (i.e. a few gradient steps of a straightforward optimization problem).\nThe various opponents that are used for the training of this base policy are generated in two steps.\nFirst, given a base policy, an \"hard-to-exploit\" opponent is generated adversarially (in a procedure coined hard-OSG), to minimize the the reward that the base policy would get by adapting to it (using the few updates that are allowed to it in its adaptation step).\nThen, given a base policy and a \"hard-to-exploit\" opponent, more diverse opponents are sequentially generated (in a procedure coined diverse-OSG) by optimizing their expected reward against the base policy while maximizing their \"diversity\" with the \"hard-to-exploit\" opponent and the already generated diverse opponents.\nMore formally, this diversity between two policies is defined (and optimized) as the MMD between the distributions over the trajectories that they generate (when the policies are seen as MDPs \"playing\" against the given base policy).\nThe base policy is then trained iteratively (as described above) against the diverse opponents, that are themselves generated (as described above) with the current iterate of the base policy.\nAfter exposing this training procedure, the authors evaluate L2E on 3 toy games, showing that the trained policies are indeed able to benefit from little adaptations to a variety of heuristic opponents and perform better than some baseline methods.\nThey also empirically confirm that their \"diversity-regularized policy optimization\" indeed generates diverse policies.\nLast, the authors empirically show the effect of their hard-OSG and diverse-OSG modules on the performance of L2E.\n\n## Pros\n- This paper is tackling a very relevant, interesting and difficult problem.\n- I find the general approach of optimizing a policy to be able to \"rapidly\" and exploit a broad range of opponents to be very exciting.\n- To the best of my (admittedly limited) knowledge, the suggested approach is significantly novel.\n- While maybe a bit \"roughly used\" the diversity-inducing regularization term, using the MMD over the distribution of trajectories induced by the policies is interesting and potentially has a broader applicability than only L2E.\n\n## Cons\n- After careful reading, several key points remain unclear to me. Most notably, after training of L2E and when facing opponents with unknown policies, how does the base policy adapts? Is it done using eq. 2? If yes, is the expectation over the trajectories approximated with the actual observations made during the observation? How many observations are being used? If my understanding is correct, clarifying those points would help put in perspective how fast it actually takes for L2E to adapt in practice.\n- It looks (to me, because no comment is made about it in the manuscript) like L2E must scale terribly with the size of the action space. First it must be extremely computationally intensive. While this remains feasible for the toy games that were used in the experiments, I am having very high doubts that this would scale well with larger games (even BigLeduc poker is ridiculously tiny compared to actual poker). At least, some comments about the computational aspects of L2E, or empirical evidence that L2E can handle larger games would be nice.\n- A point is made, several times in the manuscript, that the base policy becomes \"more robust and eliminates its weaknesses by learning to exploit hard opponents\". First, it is not really clear what is precisely meant by this. Without further assumptions on the class of games, I do not really see why the base policy would be having a good expected reward before adaptation (either in average over a broad class of opponents or against the optimal opponent), or even less why it would be hard to exploit (especially after adaptation). In fact the empirical results suggest that the base policy is breaking even against a random opponent (before adaptation) at Leduc poker, which seems rather weak to me.\n- The last point brings to a more general issue. I understand that the value of the contribution is more empirical than theoretical. Yet, it is absolutely not obvious to me whether L2E is supposed to converge at all (let alone having a clear idea about to what kind of solution it would converge). I am not a specialist of game theory, but I understand this is likely a difficult setting to analyze. At least, empirical evidence on the convergence of L2E would, in my opinion, strengthen greatly the manuscript.\n- I find that the writing could also be improved. While the algorithm is admittedly hard to fully describe in a very succinct way, the amount of repetions or redundancies in the first 6 pages suggests that L2E could be more concisely and sharply introduced. The split between the main text and the appendices seems a bit arbitrary to me as I definitely think more content about related work should be exposed in the main document. At the very least, appendices A and B should be referenced in the main text. As it stands, there is literally no indication in the paper that the related work section and the algorithms can be found in the supplementary materials. There are also a lot of imprecisions in the form of somewhat vague claims or missing important details. In addition to the ones already mentioned, I would for instance take the example of section 3.2.2. where it is not clear to me how the first policy of diverse-OSG is generated in the absence of the hard-OSG module. And in that same paragraph the bold statement that hard-OSG helps enhance the stability (what is meant by that exactly?) is not clear at all to me. It is also claimed in 3.2. that positive returns are guaranteed against opponent without a clear style (whatever that precisely means). I see rather mild empirical envidence of this but certainly not guarantees. Another minor point is that I find the Theorem 1 to be a bit weirdly formatted. I imagine the theorem is supposed to be the statement that MMD equals 0 iff the two distributions are equal, but then, the following sentence shuld be more clearly separated (as not part of the theorem) and it should be more clearly stated that the result is not a contribution by adding the reference where this result first appeared (Gretton et al. '07, I assume). If not, the derivation of the gradient computation does not really constitute a \"theorem\". Last, there are a number of typos, or verbs missing throughout the manuscript, that should be easy to fix (sorry, it's really not convenient for me to list them without line numbers...).\n- I'm a bit puzzled by some implementation details of the diverse-OSG. Notably, there seems to be no weighing on the MMD term in eq. 11. That seems pretty arbitrary to me. Could you elaborate on that? More specifically, I would imagine that if the MMD ways too little, the generated policies will be roughly identical while they will be diverse but potentially arbitrarily bad (in terms of expected reward). Also, while I don't have an issue with the somewhat arbitrary choice of an RBF kernel, I am a bit more puzzled by the choice of a width of 1. But I could imagine I'm missing an argument as to why this is a good choice.\n- In Table 1, L2E is reported to have a positive average return against the oracle, which is defined as \"making decisions based on perfect information\". It's not clearly described what those decisions are but unless they are pretty bad, there is no way L2E or any policy can win against it. (And it should be pretty easy to find and implement the optimal strategy for the perfect information game.)\n\n## Reasons for score\nWhile I really want to emphasize that the problem is very interesting and that I like the premise of L2E, I think the paper, in its current form, is missing the target.\nThe main reasons can already be found in the \"cons\" that I listed.\nTo elaborate a bit further, I think that either the selected games for the experiments are too small and toy-like for a purely empirical paper (in contrast with AlphaStar or Liberatus achieving superhuman performance at games like Starcraft 2 or heads-up no limit hold'em, although they definitely tackle a different, and probably simpler problem).\nIn this current form, I consider the experiments as a crude proof-of-concept, which could be totally fine if there were more theoretical analysis to support the suggested approach.\n\n## Questions during rebuttal period\nI think several questions have already been raised in the rest of my review.\nMost importantly, I would really love to understand how the base policy is updated in a \"real setting\", after training (see my cons #1).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Well, this paper has achieved something remarkable in this review process:  The initial scores came in at fairly low scores (4, 5, 3, 6).  However, as the discussions / rebuttals went back and forth, the reviewers were able understand and see the merits of the proposed methodology.  Namely, the setting of L2E (Learning to Exploit), which makes use of a novel method called Opponent Strategy Generation, to quickly generate very different types of opponents to play against.  One more pertinent component is the use of MMD (maximum mean discrepancy regularization) which can remove the necessity of dealing with task distributions, and does a better job in creating diverse opponents.

Having understood the technical approach, three of the reviewers decided to substantially increase their scores. R4 increased 4->6, R5 increased 5->6, R3 increased 3->4, while R2 held steady with a score of 6. It was also good to see empirical favorable results compared to other baseline methods: L2E had the best return against unclear opponents, such as Rocks opponent and Nash opponent.

Without any reviewer arguing strongly for acceptance, the program committee decided that the paper in its current form does not quite meet the bar, and also that it would benefit from another revision. 
Decision: Reject