SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 4.3 We use  $\varepsilon = \sum_{s_0}p_0(s_0)\varepsilon (s_0)$  to denote the overall performance gap between policy profile  $(\pi ,\mu)$  and  $(\pi ,\pi_B)$ . In any 2-player fully observable game, for all reward functions,  $\varepsilon = 0$  if and only if  $\pi_B(a|s) = \mu (a|s),\forall s,\mathrm{s.t.}d_{\pi ,\pi_B}(s) > 0$
Proof. Sufficiency. Whenever  $\pi_B(a|s) = \mu(a|s)$  holds, the first term in equation 1 is 0. If  $\forall s$  s.t.  $d_{\pi, \pi_B}(s) > 0$ ,  $\pi_B(a|s) = \mu(a|s)$ , according to Proposition 4.2, by expanding the expression for  $\varepsilon(s)$  recursively, we have for  $\forall s$  s.t.  $d_{\pi, \pi_B}(s) > 0$ ,  $\varepsilon(s) = 0$ . Therefore,  $\varepsilon = \sum_{s_0} p(s_0) |\varepsilon(s_0)| = 0$ .
Necessity. We first show by contradiction that  $\varepsilon = \sum_{s_0}p(s_0)\varepsilon (s_0)$  requires  $\forall s$  s.t.  $p_0(s) > 0$ ,  $\pi_B(a|s) = \mu (a|s)$ . If  $\pi_B(a|s)\neq \mu (a|s)$ , according to Proposition 4.2, we can change the reward function  $r(s,\pmb {a},s^{\prime})$  to change the value of  $\varepsilon (s)$ , because the reward function is arbitrary. So  $\varepsilon = \sum_{s_0}p(s_0)\varepsilon (s_0)$  does not hold anymore, and this is a contradiction. Therefore,  $\varepsilon (s) = \sum_{s_0,s_1}p(s_0)\pi (a_1|s_0)\pi_B(a_2|s_0)P_M(s_1|s,\pmb {a})\varepsilon (s_1) = \mathbb{E}_{s_1\sim (\pi ,\pi_B)}[\varepsilon (s_1)]$ . Using the same arguments, it can be shown that  $\forall s$ , s.t.  $Pr(s_{1} = s) > 0$ ,  $\pi_B(a|s) = \mu (a|s)$ . By expanding recursively, the statement is proved.
Proof of Theorem 4.7
We first prove a Lemma.
Lemma A.1. In a two-player game, suppose that  $\pi$  is player 1's policy,  $\alpha$  and  $\mu$  are player 2's policies. We use  $\pi_{(\pi, \alpha)}^{joint}$  and  $\pi_{(\pi, \mu)}^{joint}$  to denote the joint policy profiles  $(\pi, \alpha)$  and  $(\pi, \mu)$ . Then we have  $D_{\mathrm{KL}}\left(\pi_{(\pi, \alpha)}^{joint}(\cdot | s) \| \pi_{(\pi, \mu)}^{joint}(\cdot | s)\right) = D_{\mathrm{KL}}(\alpha (\cdot | s) \| \mu (\cdot | s))$ , where  $D_{\mathrm{KL}}$  denotes KL divergence.
Proof. According to definition,
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(\pi_ {(\pi , \alpha)} ^ {\mathrm {j o i n t}} (\cdot | s) \| \pi_ {(\pi , \mu)} ^ {\mathrm {j o i n t}} (\cdot | s)\right) = \sum_ {a _ {1}, a _ {2}} \pi (a _ {1} | s) \alpha (a _ {2} | s) \log \frac {\pi (a _ {1} | s) \alpha (a _ {2} | s)}{\pi (a _ {1} | s) \mu (a _ {2} | s)} \\ = \sum_ {a _ {1}} \pi \left(a _ {1} | s\right) \sum_ {a _ {2}} \alpha \left(a _ {2} | s\right) \log \frac {\alpha \left(a _ {2} | s\right)}{\mu \left(a _ {2} | s\right)} \tag {12} \\ = \sum_ {a _ {1}} \pi (a _ {1} | s) D _ {\mathrm {K L}} (\alpha (\cdot | s) \| \mu (\cdot | s)) \\ = D _ {\mathrm {K L}} \left(\alpha (\cdot | s) \| \mu (\cdot | s)\right). \\ \end{array}
$$
Additionally, we use the Theorem 1 in [36] and we provide a restatement. The original paper [36] minimizes accumulated discounted cost. In contrast, an agent maximizes its return in our paper.
Theorem A.2. (Theorem 1 in [36]) Let  $\epsilon = \max_{s}|\mathbb{E}_{a\sim \pi_2(a|s)}[A^{\pi_1}(s,a)]|$ , then
$$
J \left(\pi_ {2}\right) - J \left(\pi_ {1}\right) \geq \mathbb {E} _ {s \sim d _ {\pi_ {1}} (s)} \mathbb {E} _ {a \sim \pi_ {2} (a | s)} \left[ A ^ {\pi_ {1}} (s, a) \right] - \frac {2 \epsilon \gamma}{(1 - \gamma) ^ {2}} \max  _ {s} D _ {\mathrm {K L}} \left(\pi_ {1} (\cdot | s) \| \pi_ {2} (\cdot | s)\right). \tag {13}
$$
Finally, we prove Theorem 4.7.
Theorem 4.7 In a two-player game, suppose that  $\pi$  is player 1's policy,  $\alpha$  and  $\mu$  are player 2's policies. Assume that  $\max_s D_{\mathrm{KL}}(\alpha (\cdot |s)\| \mu (\cdot |s))\leq \delta$ . Let  $R_M$  be the maximum magnitude of return obtainable for player 1 in this game, and let  $\gamma$  be the discount factor. We use  $J(\pi ,\mu)$  and  $J(\pi ,\alpha)$  to denote the return for player 1 when playing policy profiles  $(\pi ,\mu)$  and  $(\pi ,\mu)$  respectively. Then,
$$
J (\pi , \mu) - J (\pi , \alpha) \geq - R _ {M} \sqrt {2 \delta} \left(1 + \frac {2 \gamma \delta}{(1 - \gamma) ^ {2}}\right). \tag {14}
$$
Proof. According to Theorem A.2, we have
$$
\begin{array}{l} J (\pi , \mu) - J (\pi , \alpha) \geq \mathbb {E} _ {s \sim d _ {(\pi , \alpha)}} \mathbb {E} _ {a _ {1} \sim \pi , a _ {2} \sim \mu} [ A ^ {(\pi , \alpha)} (s, \boldsymbol {a}) ] \\ - \frac {2 \epsilon \gamma}{(1 - \gamma) ^ {2}} \max  _ {s} D _ {\mathrm {K L}} \left(\pi_ {(\pi , \alpha)} ^ {\text {j o i n t}} (\cdot | s) \| \pi_ {(\pi , \mu)} ^ {\text {j o i n t}} (\cdot | s)\right), \tag {15} \\ \end{array}
$$
where  $\epsilon = \max_{s}|\mathbb{E}_{a_1\sim \pi ,a_2\sim \mu}[A^{(\pi ,\alpha)}(s,\pmb {a})]|\geq 0$  . Use Lemma A.1, we have
$$
\begin{array}{l} J (\pi , \mu) - J (\pi , \alpha) \geq \mathbb {E} _ {s \sim d _ {(\pi , \alpha)}} \mathbb {E} _ {a _ {1} \sim \pi , a _ {2} \sim \mu} [ A ^ {(\pi , \alpha)} (s, \boldsymbol {a}) ] - \frac {2 \epsilon \gamma}{(1 - \gamma) ^ {2}} \max  _ {s} D _ {\mathrm {K L}} (\alpha (\cdot | s) \| \mu (\cdot | s)) \\ \geq - \epsilon - \frac {2 \epsilon \gamma}{(1 - \gamma) ^ {2}} \max  _ {s} D _ {\mathrm {K L}} (\alpha (\cdot | s) \| \mu (\cdot | s)) \\ = - \epsilon \left(1 + \frac {2 \gamma}{(1 - \gamma) ^ {2}} \max  _ {s} D _ {\mathrm {K L}} (\alpha (\cdot | s) \| \mu (\cdot | s))\right) \\ \geq - \epsilon \left(1 + \frac {2 \gamma \delta}{(1 - \gamma) ^ {2}}\right). \tag {16} \\ \end{array}
$$
Next, we use KL divergence to bound  $\epsilon$ . According to definition,  $|Q^{(\pi ,\alpha)}(s,\pmb {a})|\leq R_M$ . So we have,
$$
\begin{array}{l} \epsilon = \max  _ {s} | \mathbb {E} _ {a _ {1} \sim \pi , a _ {2} \sim \mu} [ A ^ {(\pi , \alpha)} (s, \boldsymbol {a}) ] | \\ = \max  _ {s} \left| \sum_ {a _ {1}} \pi (a _ {1} | s) \sum_ {a _ {2}} \mu (a _ {2} | s) \left(Q ^ {(\pi , \alpha)} (s, \boldsymbol {a}) - V ^ {(\pi , \alpha)} (s)\right) \right| \\ = \max  _ {s} \left| \sum_ {a _ {1}} \pi \left(a _ {1} \mid s\right) \sum_ {a _ {2}} \left(\mu \left(a _ {2} \mid s\right) - \alpha \left(a _ {2} \mid s\right)\right) Q ^ {(\pi , \alpha)} (s, \boldsymbol {a}) \right| \tag {17} \\ \leq \max  _ {s} \sum_ {a _ {1}} \pi (a _ {1} | s) \sum_ {a _ {2}} | \mu (a _ {2} | s) - \alpha (a _ {2} | s) | | Q ^ {(\pi , \alpha)} (s, \boldsymbol {a}) | \\ \leq \max  _ {s} \sum_ {a _ {1}} \pi (a _ {1} | s) \sum_ {a _ {2}} \left| \mu (a _ {2} | s) - \alpha (a _ {2} | s) \right| R _ {M} \\ \end{array}
$$
With Pinsker's inequality [4], the total variation distance is bounded by KL divergence:
$$
\delta (\alpha (\cdot | s), \mu (\cdot | s)) = \frac {1}{2} \| \alpha (\cdot | s) - \mu (\cdot | s) \| _ {1} \leq \sqrt {\frac {1}{2} D _ {\mathrm {K L}} (\alpha (\cdot | s) \| \mu (\cdot | s))}. \tag {18}
$$
$$
\begin{array}{l} \epsilon \leq \max  _ {s} \sum_ {a _ {1}} \pi (a _ {1} | s) \cdot 2 R _ {M} \delta (\alpha (\cdot | s), \mu (\cdot | s)) \\ \leq \sqrt {2} R _ {M} \cdot \max  _ {s} \sqrt {D _ {\mathrm {K L}} (\alpha (\cdot | s) \| \mu (\cdot | s))} \tag {19} \\ \leq R _ {M} \sqrt {2 \delta}. \\ \end{array}
$$
Put inequality 19 back into 16, and we get
$$
J (\pi , \mu) - J (\pi , \alpha) \geq - R _ {M} \sqrt {2 \delta} \left(1 + \frac {2 \gamma \delta}{(1 - \gamma) ^ {2}}\right). \tag {20}
$$
Proof of Theorem 4.8
Theorem 4.8 Let  $\pi^{*}$  be the optimal risk-free offline adaptation policy at the convergence of the optimization of objective 2, and let  $\tilde{\pi}$  be the policy at the convergence of objective 5. Then the worst-case adaptation performance of  $\tilde{\pi}$  is near-optimal:
$$
\min  _ {\mu \in \mathcal {C} _ {D}} J \left(\pi^ {*}, \mu\right) \geq \min  _ {\mu \in \mathcal {C} _ {D}} J (\tilde {\pi}, \mu) \geq \min  _ {\mu \in \mathcal {C} _ {D}} J \left(\pi^ {*}, \mu\right) - R _ {M} \sqrt {2 \delta} \left(1 + \frac {2 \gamma \delta}{(1 - \gamma) ^ {2}}\right). \tag {21}
$$
Proof. According to definition,  $\pi^{*}$  is the solution to the optimization objective 2, so  $\min_{\mu \in \mathcal{C}_D}J(\pi^*,\mu)\geq \min_{\mu \in \mathcal{C}_D}J(\tilde{\pi},\mu)$ .
Suppose that  $(\pi^{*},\mu^{*})$  and  $(\tilde{\pi},\tilde{\alpha})$  are the solutions to objectives 2 and 5 respectively. For  $\forall \mu \in \mathcal{C}_D$ , let  $\mathcal{F}(\mu) = \{\alpha |\forall s\notin D,\alpha (\cdot |s) = \mu (\cdot |s);\max_{s\in D}D_{\mathrm{KL}}(\pi_B(\cdot |s)\| \alpha (\cdot |s))\leq \delta \}$  be the set of corresponding  $\alpha$  policies which are identical to  $\mu$  on OOD states. Observe that  $\max_sD_{\mathrm{KL}}(\pi_B(\cdot |s)\| \alpha (\cdot |s))\leq \delta$  also holds. Therefore, according to Theorem 4.7,  $\forall \pi ,\forall \mu \in$
$\mathcal{C}_D, \forall \alpha \in \mathcal{F}(\mu), J(\pi, \alpha) \geq J(\pi, \mu) - R_M \sqrt{2\delta} \left(1 + \frac{2\gamma\delta}{(1 - \gamma)^2}\right)$ . Taking the minimization over  $\mu$  and  $\alpha$ , we get  $\forall \pi, \min_{\mu} \min_{\alpha \in \mathcal{F}(\mu)} J(\pi, \alpha) \geq \min_{\mu} J(\pi, \mu) - R_M \sqrt{2\delta} \left(1 + \frac{2\gamma\delta}{(1 - \gamma)^2}\right)$ . Observe that the left part is equivalent to  $\min_{\alpha: \max_{s \in D} D_{\mathrm{KL}}(\alpha(\cdot|s) \| \pi_B(\cdot|s)) \leq \delta} J(\pi, \alpha)$ . Taking the maximization over  $\pi$  for both sides, we get  $J(\tilde{\pi}, \tilde{\alpha}) \geq J(\pi^*, \mu^*) - R_M \sqrt{2\delta} \left(1 + \frac{2\gamma\delta}{(1 - \gamma)^2}\right)$ .
Let  $J(\tilde{\pi}, \mu') = \min_{\mu \in \mathcal{C}_D} J(\tilde{\pi}, \mu)$ . Observe that  $\mu'$  also satisfies the KL divergence constraint:  $\max_{s \in D} D_{\mathrm{KL}}(\pi_B(\cdot | s) \| \mu'(\cdot | s)) \leq \delta$ , and  $(\tilde{\pi}, \tilde{\alpha})$  is the optimal solution to objective 5, so  $J(\tilde{\pi}, \mu') \geq J(\tilde{\pi}, \tilde{\alpha})$ . So we get  $\min_{\mu \in \mathcal{C}_D} J(\tilde{\pi}, \mu) \geq J(\tilde{\pi}, \tilde{\alpha}) \geq J(\pi^*, \mu^*) - R_M \sqrt{2\delta} \left(1 + \frac{2\gamma\delta}{(1 - \gamma)^2}\right)$ .
Offline RL. Offline reinforcement learning [22] learns purely from batched data. Representative works take a conservative view of out-of-distribution state-action pairs to mitigate the distributional shift problem. BCQ [8] uses VAE to propose candidate actions that are within the support of dataset. IQL [19] uses expectile regression to learn Q function to avoid querying out-of-distribution actions. CQL [20] puts regularization on the learning of Q function to penalize out-of-distribution actions.
Opponent exploitation. There have been extensive study on various aspects of opponent exploitation [1, 28]. Some employ deep reinforcement learning to exploit the opponent in continuous control tasks using policy gradients estimated from direct interaction [2, 10, 12]. Some works, including RNR [17] and SES [24] study safe exploitation given an estimated opponent model. They keep the exploitation policy close to NE in order to minimize the loss if the opponent changes its policy adversarially. Some work, for example, GSCU [7], studies zero-shot generalization ability to exploit totally unknown opponents. To our best knowledge, there lacks a thorough investigation into the offline adaptation problem.
Robust RL. Formulated as robust-MDP (RMDP), robust RL [32, 30] is dedicated to learn a robust policy against perturbation in environment dynamics, e.g., observation noise, action delay. Formally, the unknown transition model in testing environment lies in an uncertainty set  $\mathcal{P}$  which contains all models within a certain distance from the training model. Robust RL solves a max-min problem, optimizing the worst-case performance against any transition model in  $\mathcal{P}$ . Some works use adversarial training [32, 16]. RORL [42] uses smoothing on the Q function to learn robust policy. The similarity to our work is that, our offline adaptation problem also optimizes the worst-case performance against the target's behavior on out-of-distribution states. However, we do not have any distance constraints on the target's policy on out-of-distribution states. To our best knowledge, our paper is the first to investigate into the offline adaptation problem.
The CSP algorithm is illustrated in Algorithm 1. The proxy model is trained adversarially against our agent, therefore, we set the proxy's reward function to be the negative of our agent's reward. In our experiments on an  $n$ -player environment, we assume that we control  $n_1$  players, while the target controls  $n_2$  players, and  $n_1 + n_2 = n$ . Therefore, we use MAPPO [48] as the basic learning algorithm for players of both sides, since players from the same side are fully cooperative. MAPPO deals with the cooperative multi-agent reinforcement learning problem through learning a centralized value function conditioned on global state, and a decentralized control policy conditioned on local observations. We use self-play with alternative update to learn both adaptation policy  $\pi$ , and target's
proxy model  $\mu$  simultaneously. We use a soft behavior cloning regularization term to minimize the KL-divergence between proxy model  $\mu$  and target policy  $\pi_B$ .
Algorithm 1 Constrained Self-Play (CSP)
Input: dataset  $D$  environment env, learning rate  $\alpha$  , regularization coefficient  $C_\mathrm{BC}$    
Output: adaptation policy  $\pi$  target's policy proxy  $\mu$    
1: Initialize adaptation policy  $\pi$  ,critic  $v^{\pi}$  ; and target's policy proxy  $\mu$  ,critic  $v^{\mu}$    
2: while not converged do   
3: Collect trajectories  $\{\mathcal{T}_i\} \gets$  rollout  $(\pi ,\mu ;env)$    
4: if our turn then   
5: For each trajectory  $\mathcal{T}_i$  , calculate return target  $R_{t}^{\gamma}$  and advantage  $\hat{A}_t$  of our side using GAE with value function  $v^{\pi}$  for each step t   
6:  $v^{\pi}\gets v^{\pi} - \alpha \nabla_{v^{\pi}}\frac{1}{|\cup T_i|}\sum_{(s_t,R_t^\gamma)\sim \cup T_i}(R_t^\gamma -v^{\pi}(s_t))$    
7:  $\pi \gets \pi +\alpha \nabla_{\pi}\frac{1}{|\cup T_i|}\sum_{(o_t,a_t,\pi_{old},\hat{A}_t)\sim \cup T_i}\min \left(\frac{\pi(a_t|o_t)}{\pi_{old}(a_t|o_t)}\hat{A}_t,\right.$    
8: clip  $\left(\frac{\pi(a_t|o_t)}{\pi_{old}(a_t|o_t)},1 - \epsilon ,1 + \epsilon\right)\hat{A}_t\bigg)$    
9: else   
10: For each trajectory  $\mathcal{T}_i$  , calculate return target  $R_{t}^{\gamma}$  and advantage  $\hat{A}_t$  of target agent side using GAE with value function  $v^{\mu}$  for each step t   
11:  $\mathcal{B}\gets$  sample a random batch from D   
12:  $v^{\mu}\gets v^{\mu} - \alpha \nabla_{v^{\mu}}\frac{1}{|\cup T_i|}\sum_{(s_t,R_t^\gamma)\sim \cup T_i}(R_t^\gamma -v^{\mu}(s_t))$    
13:  $\mu \gets \mu +\alpha \nabla_{\mu}\left\{\frac{1}{|\cup T_i|}\sum_{(o_t,a_t,\pi_{old},\hat{A}_t)\sim \cup T_i}\min \left(\frac{\mu(a_t|o_t)}{\mu_{old}(a_t|o_t)}\hat{A}_t,\right.\right.$    
14: clip  $\left(\frac{\mu(a_t|o_t)}{\mu_{old}(a_t|o_t)},1 - \epsilon ,1 + \epsilon\right)\hat{A}_t\bigg) - C_\mathrm{BC}\cdot \frac{1}{|\mathcal{B}|}\sum_{(o,a)\in \mathcal{B}} - \log \mu (a|o)\right\}$    
15: end if   
16: end while
D Experiment on Didactic Maze
We show experiment details of the Maze example in this section. The learning curve in case 1 is shown in Figure 5. In the BC-First algorithm, since the dataset only contains trajectories that  $\mathrm{P}1\rightarrow$  64, the BC model makes a wrong and risky generalization that assumes P1 always goes downwards to 64 no matter how P2 acts. Therefore, the P2 policy learns to exploit this "weakness" by playing  $\mathrm{P}2\to 16$  , and achieves 16 in training phase. However, during testing, P2 obtains -4 reward through trying to exploit an nonexistent weakness of P1.
Our algorithm avoids the trap of imaginary weakness, and learns to safely win the game, while the BC-First method makes risky exploitations which will not work in evaluation finally. Our algorithm keeps conservative for states outside dataset, and admits the possibility that P1 could go leftwards to win the game if P2 does not plays  $\mathrm{P}2 \rightarrow 1$ .
Figure 5: The training curve of amortized average per step reward in maze game case 1 in training (against the proxy). The x-axis represents training steps. Left. Average per step reward for P1. Right. Average per step reward for P2.
The learning curve in case 2 is shown in Figure 6. As can be seen in Figure 6, our algorithm quickly learns to exploit while the max-min strategy produced by Self-Play fails to.
Figure 6: The training curve of amortized average per step reward in maze game case 2 in training (against the proxy). The x-axis represents training steps. Left. Average per step reward for P1. Right. Average per step reward for P2.
We use the same set of hyper-parameters (including the same coefficient  $C_{BC}$  for the BC regularization term) for these two cases. The hyper-parameters are shown in Table 5. The dataset  $D_{1} = \{\text{trajectories}(P1 \to 64, P2 \to 256)\}$  in case 1, and our algorithm learns to play  $P2 \to 1$ ; while in case 2, given dataset  $D_{2} = \{\text{trajectories}(P1 \to 64, P2 \to 256), \text{trajectories}(P1 \to 64, P2 \to 16)\}$ , our algorithm learns to play  $P2 \to 16$ . We use the same set of hyper-parameters for both cases. Therefore, our algorithm produces different policies simply because of different datasets given. It further validates that our algorithm is able to extract useful information from dataset, and only perform risk-free adaptations.
Table 5: Hyper-parameters for maze.
ppo_epoch	1	num_small_batch	1	entropy coef	0.3
use_gae	True	gamma	0.99999	gae_lambda	0.95
critic_lr	7e-4	lr	7e-4	weight Decay	0
adam ECS	1e-5	n_rollback Threads	20	ppo Episode_length	12
data_chunk_length	12	steps	1.8K	max_grad_norm	0.5
bc Regularization coef	10	bc_batch_size	8	network	MLP
E Conservative Offline Adaptation for Cooperative Tasks
We evaluate our method in competitive games to perform opponent exploitation. Nevertheless, our method is also applicable to cooperative environments. The objective of CSP in cooperative games maintains the same min-max structure because it promotes adherence to the policy exposed by dataset for states within the dataset, while aiming to optimize for worst-case performance for states not in the dataset. Therefore, in cooperative games, the policy regularization term incentivizes the adaptation agent to collaborate with the target agent on states within the dataset, and the minimization over opponent proxy  $\mu$  encourages the adaptation agent to be conservative and not to rely on the target agent's OOD policy. Note that in objective 5 and Algorithm 1, no matter whether the game is competitive or cooperative, the minimization over  $\mu$  in training can be achieved by setting the target agent's reward function to be the negative of our adaptation agent's reward function. We then train  $\mu$  to maximize this reward with policy regularization in self-play.
Although our method is applicable to cooperative environments as well, the challenges caused by offline policy adaptation are more significant in competitive games. Recall that conservative offline adaptation optimizes
$$
\max  _ {\pi} \min  _ {\mu} J (\pi , \mu), s. t. \mu \in \mathcal {C} _ {D}. \tag {22}
$$
For cooperative tasks, it requires that the teammate  $\mu$  will not cooperate on states outside dataset in cooperative games. Therefore, for the adaptation policy, staying within dataset could be a trivial and
near-optimal solution for most cooperative problems if we would like to maximize the worst-case performance against any dataset-consistent teammate.
F Potential Negative Social Impacts
Inappropriate use of exploitation algorithms may result in negative social impacts. An example of its negative impact is the exacerbation of inequality in society. Companies who have access to large amounts of data can utilize their customers more effectively. However, our algorithm is general-purpose and depends on pre-collected data. We advocate for strict laws and regulations that protect user privacy data to avoid negative impacts. It is recommended that products utilizing exploitation algorithms are made public and supervised.
G.1 Environments
The experiment environments are illustrated in figure 7.
Figure 7: Illustration of experiment environments used in this paper. From left to right: (1) predator prey in MPE, (2) YouShallNotPassHumans in MuJoCo, (3) Google Football.
Predator-Prey There are one good agent, three adversarial agents and two obstacles in the predator-prey environment. The good agent is faster and receive a negative reward for being hit by adversaries while adversaries are slower and are rewarded for hitting the good agent. All agents are initialized randomly in the environment.
YouShallNotPassHumans The YouShallNotPassHumanoids environment [2, 10] creates a two-player competitive game based on MuJoCo, where one humanoid (the runner) is aimed at passing the other humanoid (the blocker) to reach the red line on the opposite side. The environment itself is challenging because it has high dimensional observation space and requires continuous control. In our experiments, we assume that our agent acts as the runner, while the blocker is the target opponent to exploit. We use four independently pre-trained blocker models as opponent targets. For fairness, we generate these targets with exactly four random seeds without any selection. Since the exploitability of different opponent models can vary significantly, results with respect to different opponents are not directly comparable. So we report results with respect to all targets.
Google Football Google Football [21] is a popular and challenging benchmark for MARL. In our experiments, our adaptation policy controls all players in one team, while the opponent target policy controls the other team. Since all players in the same team are fully cooperative, we use MAPPO [48] to learn decentralized policy. We conduct experiments in 4 scenarios: academy_3_vs_1_withKeeper (3vs1, where our agent acts as either defender or attacker), academy_run_pass_and_shoot_withKeeper (RPS, defender), and academy counterattack_easy (defender). We report the winning rates of adaptation policies for 5 independently pre-trained opponent targets. For attacker, winning rate refers to the percentage of episodes that the attacker scores, while for defender, it refers to the percentage of episodes that the defender prevents the attacker from scoring.
G.2 Hyper-parameters
In our experiments, we use MAPPO [48] as the base RL algorithm to learn policies on both sides (ours and the target's). MAPPO is an extension of single agent PPO to multi-agent reinforcement learning
within the centralized training decentralized evaluation (CTDE) paradigm. It learns a centralized value function conditioning on global state to promote coordination among the agents. We selected MAPPO as the base learning algorithm due to its simplicity and empirical strong performance.
For the BC-First method, we use exactly the same MAPPO (with the same set of hyper-parameters and the same number of training samples) as CSP to train our adaptation policy, while keeping the target's proxy fixed. In order to ensure that the performances of BC-First method are not encumbered by an under-trained proxy model, we use the same network structure for the proxy model as the real target model, and train enough steps to make sure that behavior cloning has converged. For the Google Football environment, we use the same hyper-parameters as reported in the MAPPO paper [48] for all scenarios. The hyper-parameters are listed in Table 6, 7, and 8.
Table 6: Hyper-parameters for predator-prey in MPE.
ppo_epoch	10	num_small_batch	1	entropy coef	0.01
use_gae	True	gamma	0.99	gae_lambda	0.95
critic_lr	7e-4	lr	7e-4	weight Decay	0
adam Alps	1e-5	n Rollout Threads	128	ppo Episode_length	50
data_chunk_length	10	steps	5M	max_grad_norm	0.5
bc Regularization coef	0.003	bc_batch_size	8	network	RNN
Table 7: Hyper-parameters for MuJoCo.
ppo_epoch	4	num_minibatch	1	entropycoef	0.01
use_gae	True	gamma	0.99	gae_lambda	0.95
critic_lr	7e-4	lr	7e-4	weight Decay	0
adam_epochs	1e-5	n Rollout Threads	100	ppo Episode_length	200
data_chunk_length	10	steps	40M	max_grad_norm	0.5
bcRegularExpressioncoef	0.1	bc_batch_size	256	network	MLP