SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chengjie Wu $^{1}$ , Pingzhong Tang $^{12}$ , Jun Yang $^{3}$ , Yujing Hu $^{4}$ , Tangjie Lv $^{4}$ , Changjie Fan $^{4}$ , Chongjie Zhang $^{5}$
$^{1}$ Institute for Interdisciplinary Information Sciences, Tsinghua University
$^{2}$ Turingsense
$^{3}$ Department of Automation, Tsinghua University
$^{4}$ Fuxi AI Lab, NetEase
$^{5}$ Department of Computer Science & Engineering, Washington University in St. Louis
wucj19@mails.tsinghua.edu.cn
{kenshin, yangjun603}@tsinghua.edu.cn
{huyujing,hzlvtangjie,fanchangjie}@corp.netease.com
chongjie@wustl.edu
Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progress in offline reinforcement learning, this paper studies offline policy adaptation, which aims to utilize the target agent's behavior data to exploit its weakness or enable effective cooperation. We investigate its distinct challenges of distributional shift and risk-free deviation, and propose a novel learning objective, conservative offline adaptation, that optimizes the worst-case performance against any dataset consistent proxy models. We propose an efficient algorithm called Constrained Self-Play (CSP) that incorporates dataset information into regularized policy learning. We prove that CSP learns a near-optimal risk-free offline adaptation policy upon convergence. Empirical results demonstrate that CSP outperforms non-conservative baselines in various environments, including Maze, predator-prey, MuJoCo, and Google Football.
Reinforcement learning (RL) has shown promise in learning collaborative or adversarial policies in multi-agent games, including cooperative multi-agent reinforcement learning [49, 6, 34, 5, 14, 40], and learning in adversarial environments such as Texas hold'em and MOBA games [3, 45, 46, 50]. Research also recognizes the importance of adapting policy to different players in multi-agent games [10, 26, 51]. For instance, in zero-sum games, although an approximate Nash equilibrium strategy optimizes worst-case performance, it can be overly conservative when competing with opponents of limited rationality and loses opportunities to exploit [27, 17, 24]. This problem is known as opponent exploitation [17, 23, 41, 1, 28]. In cooperative games, the agent may be required to cooperate with other competent players that have diverse behaviors [15, 11], instead of just a specific group of teammates. The problem is investigated in recent literature such as ad-hoc cooperation [11] and zero-shot coordination [15, 26, 51, 47]. In this paper, policy adaptation refers to the general ability to learn to collaborate with or exploit other participants (called target agents) in multi-agent games.
This paper focuses on a specific problem of offline policy adaptation. In this setting, interaction with the target agent is not available in training. Instead, we leverage the target agent's behavior dataset and maintain access to the game environment during training. The problem definition is illustrated
Figure 1: Left. Problem illustration. Mid and right. Winning rates in Google Football 3vs1 (attacker). The x-axis is the training steps. We run evaluations against (mid) proxy model in training and (right) real target agent. Shaded areas represent standard error. The non-conservative method quickly overfits the proxy model. Our method CSP outperforms baseline in testing. Please refer to Appendix G.7 for more explanations.
in Figure 1 (left). This setting is suitable for many practical scenarios where offline information about the target agent is provided. It lies between the fully transparent case, where the target agent's policy is available, and the fully invisible zero-shot adaptation case. For instance, in competitions such as Go, soccer, and MOBA games, playing with the target opponent is not an option until the actual game commences. However, professional human players have the intelligence to analyze their opponents' and teammates' strategic characteristics by studying past records, enhance their skills, and make adaptations accordingly before the competition. Moreover, training AIs with humans in applications that involve human-AI interaction is exorbitant. Additionally, it can be unsafe and poses ethical risks when humans interact with an inadequately trained AI in certain circumstances.
Although offline policy adaptation models a wide range of applications, it has not been adequately investigated both theoretically and empirically in previous literature. For instance, some opponent exploitation research assumes direct interaction in training for explicit opponent model estimation [17, 24] or policy gradient calculation [2, 10, 12]. However, this assumption is strong because it allows queries of the target agent's policy for any input. Other works study exploiting totally unknown opponents [7] or collaborating with completely unseen teammates [26, 51, 47] in a zero-shot manner. These methods do not take advantage of possible prior information about the target agent, and their performances heavily depend on the construction of a diverse and representative policy population constructed for training, which itself is a challenging problem [31, 25].
In this paper, we identify and formalize the distinct challenges in offline policy adaptation, and show that previous opponent exploitation methods, which disregard these challenges, are inadequate for effectively addressing the problem. Firstly, offline policy adaptation inherits the distributional shift [22] challenge from traditional offline RL. A straightforward approach to address offline policy adaptation is by initially creating an estimation of the target agent (e.g., utilizing behavior cloning) from data and subsequently learning an adaptation policy based on the proxy model. We name this method BC-First. However, since the dataset usually will not expose all aspects of the real policy of the target agent due to limited dataset size, the proxy model can differ arbitrarily from the real target on out-of-distribution (OOD) states. The discrepancy between the proxy in training and the real target in testing causes the distributional shift problem for the adaptation policy. As shown in Figure 1, BC-First overfits the dataset, which leads to erroneous generalization on OOD states and results in significant performance degradation in evaluation. Our algorithm outperforms the baseline and improves in both training and testing as the number of training step increases. Secondly, since the adaptation policy has access to the environment in training, unlike standard offline RL, we show that "conservatism" [8, 20, 19] for addressing distributional shift may not be sufficient in this setting. It is possible to benefit from deviation in multi-agent games if the worst-case rewards obtainable outside the dataset exceed those inside the dataset. We coin this challenge as risk-free deviation.
To address these challenges, we propose a novel learning objective, conservative offline adaptation, which optimizes the worst-case performance when maintaining a conservative stance towards the target agent on the policy level. This objective provides a unified approach to addressing the challenges of distributional shift and risk-free deviation. To optimize this objective, we introduce a simple yet efficient algorithm, Constrained Self-Play (CSP), which simultaneously trains an adaptation policy and a proxy model subject to regularizations. We prove that CSP produces a near-optimal risk-free adaptation policy upon convergence, given that the target agent's policy within the support of the dataset can be well approximated. Empirically, we evaluate the effectiveness of our algorithm in four environments: a didactic maze environment, predator-prey in MPE [39], a competitive two-agent
MuJoCo environment [2, 10] requiring continuous control, and Google Football [21]. Our results show that non-conservative methods confront difficulties regarding offline policy adaptation. The experiment results demonstrate that our algorithm significantly outperforms baselines.
RL in multi-agent games. Multi-agent reinforcement learning (MARL) under the centralized training decentralized execution (CTDE) paradigm have shown promising performances in learning coordination behavior for cooperative multi-agent tasks such as SMAC [35] and Google Football [21]. Representative works consist of policy gradient algorithms [6, 48], and value decomposition methods [34, 40]. There also have been extensive research in competitive environments such as Texas hold'em and MOBA games. Some works design theoretically sounded RL algorithms which are guaranteed to converge to approximate NE [3]. Some works propose RL methods based on self-play [45, 46, 50] to train max-min policies which show strong empirical performances.
Offline RL. Offline reinforcement learning [22] learns purely from batched data. Representative works take a conservative view of out-of-distribution state-action pairs to mitigate the distributional shift problem. BCQ [8] uses VAE to propose candidate actions that are within the support of the dataset. IQL [19] uses expectile regression to learn Q function to avoid querying out-of-distribution actions. CQL [20] puts regularization on the learning of Q function to penalize out-of-distribution actions.
Opponent modeling. Opponent modeling, sometimes also called opponent exploitation, is a broad and extensively studied area [1, 28]. The offline adaptation problem studied in this paper is a largely unstudied yet significant and challenging problem within the broader scope of opponent modeling. Some previous works assume access to the target agent's policy in training, and employ deep reinforcement learning to exploit the opponent in continuous control tasks using policy gradients estimated from direct interaction [2, 10, 12]. Some works, including RNR [17] and SES [24] study safe exploitation given an estimated opponent model. They keep the exploitation policy close to NE in order to minimize its own exploitability. Some papers, like GSCU [7], study zero-shot generalization ability to exploit totally unknown opponents. Additionally, this paper is orthogonal to some other opponent modeling subareas, including designing efficient models for predicting opponent behavior [13, 33], and addressing non-stationary opponents [7]. To our best knowledge, there lacks a thorough investigation into the offline adaptation problem.
Please refer to Appendix B for more discussion.
For simplicity, we consider a two-player fully observable game  $G = (N,S,A,P_M,r,p_0,\gamma)$  [37, 38], where  $N$  is a set of two players,  $S$  is a set of states,  $p_0$  is initial state distribution, and  $\gamma$  is the discount factor. We use  $A_i$  to denote the available actions for player  $i$ , and  $A = (A_1,A_2)$ . The transition probability is denoted by  $P_M: S \times A \times S \to \mathbb{R}$ . We use  $r_i: S \times A \to \mathbb{R}$  to denote the reward function for player  $i$ , and  $r = (r_1,r_2)$ . In fully cooperative scenarios,  $r_1(s,\pmb{a}) = r_2(s,\pmb{a}) = r(s,\pmb{a})$ , which is termed multi-agent MDP (MMDP) [43]. In zero-sum games,  $r_1(s,\pmb{a}) + r_2(s,\pmb{a}) = 0$ . Player  $i$ 's policy  $\pi_i(s,\cdot)$  is a distribution over action space  $A_i$  conditioned on state  $s \in S$ . The value function  $V_i^{(\pi_1,\pi_2)}(s)$  is the expected discounted accumulated reward that player  $i$  will achieve under joint policy profile  $(\pi_1,\pi_2)$  at state  $s$ . Similarly, the action value function  $Q_i^{(\pi_1,\pi_2)}(s,\pmb{a})$  calculates the expected return when players play the joint action  $\pmb{a} = (a_1,a_2)$ . Let  $d_{\pi_1,\pi_2}(s)$  be the discounted visitation frequency of state  $s$  under policy profile  $(\pi_1,\pi_2): d_{\pi_1,\pi_2}(s) = \sum_{t=0}^{\infty} \gamma^t P(s_t = s)$ . We use  $J_i^{(\pi_1,\pi_2)}$  to denote the return for player  $i$  under policy profile  $(\pi_1,\pi_2)$ .
Without loss of generality, assume that we are aimed at learning an adaptation policy for player 1 (P1), while player 2 (P2), called the target agent, is playing a unknown fixed strategy  $\pi_B$ . In offline policy adaptation, we do not have access to  $\pi_B$ . Instead, a dataset  $D$  of  $\pi_B$ 's behavior data is given. Formally,  $D = F_p(\{\mathcal{T}_i\})$ , where  $F_{p}$  denotes extra data processing<sup>1</sup>, and  $\mathcal{T}_i = \{s_0, a_0, s_1, \ldots\}$  records states and corresponding actions of an episode collected by  $\pi_B$  playing with rollout policies in
<sup>1</sup>We use  $F_{p}$  to represent possible information loss in data collection, e.g., truncating and randomly masking.
$\Pi_c$ . We do not make assumptions on the data collecting process, therefore,  $\Pi_c$  can contain any polices ranging from a weak random policy to a strong NE policy. Analogous to offline RL and robust RL, the dataset quality varies given different rollout policies, dataset sizes, and processing methods, which results in different problem difficulties. For instance, if the dataset reveals adequate information about the target's behavior under diverse situations, BC-First can be enough. The challenges of offline policy adaptation become prominent if the dataset only contains limited knowledge of  $\pi_B$ .
The purpose of offline adaptation is to learn to adapt to unknown  $\pi_B$  given a fixed batch of data  $D$ . Notably, different from offline RL, the access to the environment itself is still granted in training. If not, such setting can be reduced to standard imitation learning or offline reinforcement learning [44, 29], because the target's behavior can be absorbed into environment dynamics. With the advantage of having access to the environment, this setting encourages conservative exploitation whenever risk-free deviation is possible.
4 Conservative Offline Adaptation
To address the problem of offline adaptation, we first examine the extrapolation error in policy evaluation between joint policies  $(\pi, \pi_B)$  and  $(\pi, \mu)$  in section 4.1. Here,  $\pi$  denotes the learning policy,  $\pi_B$  represents the true target policy, and  $\mu$  denotes any proxy model that estimates  $\pi_B$ . This extrapolation error poses a risk to non-conservative policy adaptation algorithms, rendering them unsafe. However, traditional offline RL methods that aim to minimize the impact of extrapolation error tend to be excessively conservative, which is suboptimal in this context. To allow for beneficial deviations from the dataset, we introduce a novel objective of conservative offline adaptation in section 4.2. This approach maximizes the worst-case performance against any dataset-consistent proxy (defined below). Finally, we propose a practical Constrained Self-Play (CSP) algorithm in section 4.3 and formally prove that it achieves near-optimal conservative offline adaptation. Please note that all proofs pertaining to this section can be found in Appendix A.
4.1 Extrapolation Error in Offline Adaptation
We first analyze the extrapolation error in policy evaluation between the joint policies  $(\pi, \pi_{B})$  and  $(\pi, \mu)$ , which results in the overfitting problem shown in Figure 1. For theoretical analysis, we first introduce the dataset consistent policy in Definition 4.1. We will show that this assumption can be relaxed in algorithm design in a later section. Inspired by offline RL, in Proposition 4.2, we find that the extrapolation error in offline policy adaptation can also be decomposed in a Bellman-like recursion [8, 44]. We focus solely on the value function of our learning agent (Player 1), and omit the subscript in the rewards and value functions for simplicity.
Definition 4.1. Dataset consistent policy. A policy  $\mu$  is said to be consistent with  $\pi_B$  on dataset  $D$  iff.  $\mu (s,a) = \pi_B(s,a),\forall (s,a)\in D.$  We denote the set of all dataset consistent policies as  $\mathcal{C}_D$
Proposition 4.2. The performance gap of evaluating policy profile  $(\pi, \mu)$  and  $(\pi, \pi_B)$  at state  $s$  is  $\varepsilon(s) = V^{\pi, \pi_B}(s) - V^{\pi, \mu}(s)$ , which can be decomposed in a Bellman-like recursion:
$$
\begin{array}{l} \varepsilon (s) = \sum_ {a _ {1}, a _ {2}, s ^ {\prime}} \pi (a _ {1} | s) P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \left(\pi_ {B} \left(a _ {2} \mid s\right) - \mu \left(a _ {2} \mid s\right)\right) \left(r (s, \boldsymbol {a}, s ^ {\prime}) + \gamma V ^ {\pi , \mu} \left(s ^ {\prime}\right)\right) \\ + \sum_ {a _ {1}, a _ {2}, s ^ {\prime}} \pi \left(a _ {1} | s\right) \pi_ {B} \left(a _ {2} | s\right) P _ {M} \left(s ^ {\prime} \mid s, \boldsymbol {a}\right) \gamma \varepsilon \left(s ^ {\prime}\right) \tag {1} \\ \end{array}
$$
Theorem 4.3. We use  $\varepsilon = \sum_{s_0}p_0(s_0)\varepsilon (s_0)$  to denote the overall performance gap between policy profile  $(\pi ,\mu)$  and  $(\pi ,\pi_B)$ . In any 2-player fully observable game, for all reward functions,  $\varepsilon = 0$  if and only if  $\pi_B(a|s) = \mu (a|s),\forall s,s.t.d_{\pi ,\pi_B}(s) > 0$
Theorem 4.3 proves that the extrapolation error vanishes if and only if the estimation  $\mu$  perfectly matches  $\pi_B$  on any state which can be visited by  $(\pi, \pi_B)$ . If extrapolation error exists, improvement of  $\pi$  against proxy  $\mu$  in training will not guarantee monotonic improvement in testing when  $\pi$  is evaluated with  $\pi_B$  (as shown in Figure 1). Because we make no assumptions on  $\pi_B$  outside  $D$ , we cannot bound  $\| \mu(a|s) - \pi_B(a|s) \|$  for  $s \notin D$ . Therefore, it requires that  $d_{\pi, \pi_B}(s) = 0, \forall s \notin D$ .
Corollary 4.4. Given a dataset consistent policy  $\mu \in \mathcal{C}_D$ , which is consistent with  $\pi_B$  on dataset  $D$ , the extrapolation error vanishes if  $\forall s \notin D$ ,  $d_{\pi, \pi_B}(s) = 0$ .
Corollary 4.4 advocates excessive conservatism, similar to standard offline RL, to exclude any state-action pairs outside the dataset. It requires to constrain both  $\pi$  and  $\mu$  such that the trajectories of  $(\pi ,\mu)$  stay within  $D$ . However, this approach comes at the cost of relinquishing opportunities to exploit environmental access and explore the potential benefit of risk-free deviation from the dataset.
4.2 Conservative Offline Adaptation
In order to avoid being overly conservative and allow beneficial deviation from the dataset, we propose a novel learning objective called conservative offline adaptation in Definition 4.5.
Definition 4.5. Conservative offline adaptation (COA). Given an unknown policy  $\pi_B$ , and a dataset  $D$  of its behavior data, conservative offline adaptation optimizes the worst-case performance against any possible dataset-consistent policy:
$$
\max  _ {\pi} \min  _ {\mu} J (\pi , \mu), s. t. \mu \in \mathcal {C} _ {D}. \tag {2}
$$
The adaptation policy  $\pi^{*}$  is an optimal risk-free adaptation policy if it is a solution to Objective (2).
To show that COA enables risk-free deviation, we decompose the return into two parts in Definition 4.6: (1) the return that is solely realized within the dataset, and (2) the return that arises from deviation.
Definition 4.6. Within-dataset return & off-dataset return. The expected return of a joint policy profile  $(\pi_1, \pi_2)$  can be decomposed into the sum of within-dataset return  $J_D(\pi_1, \pi_2)$  and off-dataset return  $J_F(\pi_1, \pi_2)$ :
$$
\begin{array}{l} J (\pi_ {1}, \pi_ {2}) = \mathbb {E} _ {\tau \sim (\pi_ {1}, \pi_ {2})} \sum_ {t} \gamma^ {t} r (s _ {t}, \boldsymbol {a} _ {t}, s _ {t + 1}) \\ = \mathbb {E} _ {\tau \sim \left(\pi_ {1}, \pi_ {2}\right)} \sum_ {t \leq t _ {D} ^ {\tau}} \gamma^ {t} r \left(s _ {t}, \boldsymbol {a} _ {t}, s _ {t + 1}\right) + \mathbb {E} _ {\tau \sim \left(\pi_ {1}, \pi_ {2}\right)} \sum_ {t > t _ {D} ^ {\tau}} \gamma^ {t} r \left(s _ {t}, \boldsymbol {a} _ {t}, s _ {t + 1}\right) \tag {3} \\ = J _ {D} \left(\pi_ {1}, \pi_ {2}\right) + J _ {F} \left(\pi_ {1}, \pi_ {2}\right), \\ \end{array}
$$
where  $t_D^\tau = \arg \max_{t'}\{\forall t\leq t',s_t\in D\}$  is a random variable indicates the last time step in a trajectory  $\tau$  such that the state is still contained in dataset  $D$ .
For any policy  $\pi$ , and any dataset consistent policy  $\mu \in \mathcal{C}_D$ ,  $J_D(\pi, \mu) = J_D(\pi, \pi_B)$  because  $\mu$  and  $\pi_B$  behave exactly the same until the first time an OOD state is reached according to the definition. Thus, the objective 2 is equivalent to  $\max_{\pi} J_D(\pi, \pi_B) + \min_{\mu} J_F(\pi, \mu)$ , s.t.  $\mu \in \mathcal{C}_D$ . The adaptation policy  $\pi$  learns to exploit or cooperate with the target agent's policy exposed by dataset  $D$ . Meanwhile, the max-min optimization of  $\max_{\pi} \min_{\mu} J_F(\pi, \mu)$  encourages  $\pi$  to explore opportunities outside the dataset while maintaining a conservative stance towards the target agent at the policy level. It provides a unified approach to automatically decide whether to deviate depending on the magnitude of within-dataset return and worst-case off-dataset return. In the extreme case, if  $\min_{\mu} J_F(\pi, \mu)$  is much larger than  $J_D(\pi, \pi_B)$ ,  $\pi$  can ignore the dataset completely and try to obtain rewards outside the dataset. Additionally, suppose that  $(\pi^*, \mu^*)$  is the solution to COA, then the extrapolation error  $\varepsilon = J(\pi^*, \pi_B) - J(\pi^*, \mu^*) = J_F(\pi^*, \pi_B) - J_F(\pi^*, \mu^*) \geq 0$ , which suggests that the testing performance will not be worse than training performance.
4.3 Constrained Self-Play
Directly optimizing the objective of COA is hard because in deep reinforcement learning, satisfying the dataset consistency assumption regarding the proxy model  $\mu$  can be difficult, particularly for stochastic policies with limited data. We relax this assumption and propose a simple yet efficient algorithm, Constrained Self-Play (CSP). We prove that CSP can achieve near-optimal risk-free policy adaptation. We start by showing in Theorem 4.7 that the gap in return is bounded by the KL divergence of two policies of the target agent.
Theorem 4.7. In a two-player game, suppose that  $\pi$  is player 1's policy,  $\alpha$  and  $\mu$  are player 2's policies. Assume that  $\max_s D_{\mathrm{KL}}(\alpha (\cdot |s)\| \mu (\cdot |s))\leq \delta$ . Let  $R_M$  be the maximum magnitude of return obtainable for player 1 in this game, and let  $\gamma$  be the discount factor. We use  $J(\pi ,\mu)$  and  $J(\pi ,\alpha)$  to denote the return for player 1 when playing policy profiles  $(\pi ,\mu)$  and  $(\pi ,\mu)$  respectively. Then,
$$
J (\pi , \mu) - J (\pi , \alpha) \geq - R _ {M} \sqrt {2 \delta} \left(1 + \frac {2 \gamma \delta}{(1 - \gamma) ^ {2}}\right). \tag {4}
$$
With this result, we propose to optimize objective 5, which substitutes the hard dataset consistent constraint in objective 2 with KL-divergence.
$$
\max  _ {\pi} \min  _ {\mu} J (\pi , \mu) s. t. \max  _ {s \in D} D _ {\mathrm {K L}} \left(\pi_ {B} (\cdot | s) \| \mu (\cdot | s)\right) \leq \delta \tag {5}
$$
We propose the Constrained Self-Play (CSP) algorithm, which optimizes objective 5 through learning both an adaptation policy and a proxy model with self-play and meanwhile minimizing the KL divergence between  $\mu$  and  $\pi_B$  on the dataset with regularization. The algorithm is shown in Algorithm 1 in Appendix C. CSP utilizes soft BC regularization to minimize the KL-divergence:  $\max_{\pi} \min_{\mu} \{J(\pi, \mu) + C_{\mathrm{BC}} \cdot \mathbb{E}_{(s,a) \sim D}[-\log \mu(a|s)]\}$ . It is because BC is equivalent to KL-divergence minimization [18, 9]. By increasing the coefficient  $C_{\mathrm{BC}}$ , CSP prioritizes KL-divergence minimization. In the max-min optimization, the proxy model is trained adversarially against our adaptation policy by setting the proxy's reward function to be the negative of our agent's reward. We use MAPPO [48] as the basic learning algorithm for players on both sides.