Conservative Offline Policy Adaptation in Multi-Agent Games
Authors: Chengjie Wu
Pingzhong Tang
Jun Yang
Yujing Hu
Tangjie Lv
Changjie Fan
Chongjie Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
opponent exploitation
multi-agent

Abstract: Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progress in offline reinforcement learn- ing, this paper studies offline policy adaptation, which aims to utilize the target agent’s behavior data to exploit its weakness or enable effective cooperation. We investigate its distinct challenges of distributional shift and risk-free deviation, and propose a novel learning objective, conservative offline adaptation, that optimizes the worst-case performance against any dataset consistent proxy models. We pro- pose an efficient algorithm called Constrained Self-Play (CSP) that incorporates dataset information into regularized policy learning. We prove that CSP learns a near-optimal risk-free offline adaptation policy upon convergence. Empirical results demonstrate that CSP outperforms non-conservative baselines in various environments, including Maze, predator-prey, MuJoCo, and Google Football.

Conservative Offline Policy Adaptation in Multi-Agent Games
Chengjie Wu', Pingzhong Tang", Jun Yang", Yujing Hu",
Tangjie Lv Changjie Fan", Chongjie Zhang
Institute for Interdisciplinary Information Sciences, Tsinghua University TTrinngeese
Department of Automation, Tsinghua University
Fuxi Al Lab, NetEase
DDepartment of Computer Science Engineering, Washington University in St. Louis
wucj19@maills
edu. cn
Xkenslii..yenggjuo000Sttttsigggp edu. en
hww.:/iig...lttagggj.e.eaaacaaee aerrr www.aeeeeeee.mm. com
chongjie@vustl.eem
Abstract
Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which be expensive and impractical in real-world scenarios. Inspired by recent progress in offline reinforcement learn- ing, this paper studies offline policy adaptation, which aims to utilize the target agent's behavior data to exploit its weakness or enable effective cooperation. We investigate its distinct challenges of distributional shift and risk-free deviation, and propose a novel learning objective, conservative offline adaptation, that optimizes the worst-case performance against any dataset consistent proxy models. We pro- pose an efficient algorithm called Constrained Self-Play (CSP) that incorporates dataset information into regularized policy learning. We prove that CSP learns a near-optimal risk-free offline adaptation policy upon convergence. Empirical results demonstrate that CSP outperforms non-conservative baselines in various environments, including Maze, predator-pre prey, MuloCo, and Google Football.
1 Introduction
Reinforcement learning (RL) has shown promise in learning collaborative or adversarial policies in multi-agent games, including cooperative multi-agent reinforcement learning 49 and learning in adversarial environments such Texas hold' em and MOBA games 3 45
46 50
Research also recognizes the importance of adapting policy to different players in multi-agent games
[10 26 51]. For instance, in zero-sum games, although an approximate Nash equilibrium strategy optimizes worst-case performance, it can be overly conservative when competing with opponents of limited rationality and loses opportunities to exploit [27]
[7 24 This problem is known opponent exploitation [7 23 41 (88] In cooperative games, the agent may be required to cooperate with other competent players that have diverse behaviors [S III, instead of just specific group of teammates. The problem is investigated in recent literature such as ad-hoc cooperation [[T] and zero-shot coordination [15 26 51 47). In this paper, policy adaptation refers the general ability to learn to collaborate with or exploit other participants (called target agents) in multi-agent games.
This paper focuses on specific problem of offline policy adaptation. In this setting, interaction with the target agent is not available in training. Instead, we leverage the target agent's 's behavior dataset and maintain access the game environment during training. The problem definition is illustrated
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1.6
1.0
darinnet
0.
0.0
0.6
0.6
odtpiaaion poleey
0.4
0.4
30
sip
SAAT
0.
BC Fist
10und
0.2
BC Fist (Ouno)
0.0
0.5
1.3
3.5
2.5
5.0
0.5
2.0
2.5
Inteptation poliey
ensltoment
steps
167
Steps
107
Left. Problem illustration. Mid and right. Winning rates in Google Football 3vsl (attacker). The x-axis is the training steps. We run evaluations against (mid) proxy model in training and (right) real target agent. Shaded areas represent standard error. The con-conservative method quickly overfits the proxy model. Our method CSP outperforms baseline in testing. Please refer to Appendix G.7
Figure 1:
for more explanations.
in Figure (left). This setting is suitable for many practical scenarios where offline information about the target agent is provided. It lies between the fully transparent case, where the target agent's s policy is available, and the fully invisible zero-shot adaptation case. For instance, in competitions such as Go, soccer, and MOBA games, playing with the target opponent is not an option until the actual game commences. However, professional human players have the intelligence to analyze their opponents' and teammates' strategic characteristics by studying past records, enhance their skills, and make adaptations accordingly before the competition. Moreover, training AIs with humans in applications that involve human-Al Al interaction is exorbitant. Additionally, it can be unsafe and poses ethical risks when humans interact with an inadequately trained AI certain circumstances.
Although offline policy adaptation models a wide range of applications, it has not been adequately investigated both theoretically and empirically in previous literature. For instance, some opponent exploitation research assumes direct interaction in training for explicit opponent model estimation
11 [24] or policy gradient calculation [2] IO [1]. However, this assumption is strong because allows queries of the target agent's ' policy for any input. Other works study exploiting totally unknown opponents (7) or collaborating with completely unseen teammates 26 51 47 in zero-shot manner.
These methods do not take advantage of possible prior information about the target agent, and their performances heavily depend on the construction of diverse and representative policy population constructed for training, which itself is a challenging problem B1 25
In this paper, we identify and formalize the distinct challenges in offline policy adaptation, and show that previous opponent exploitation methods, which disregard these challenges, are inadequate for effectively addressing the problem. Firstly, offline policy adaptation inherits the distributional shift (22) challenge from traditional offline RL. A straightforward approach to address offline policy adaptation is by initially creating an estimation of the target agent (e.g., utilizing behavior cloning) from data and subsequently learning an adaptation policy based on the proxy model. We name this method BC-First. However, since the dataset usually will not expose all aspects of the real policy of the target agent due to limited dataset size, the proxy model can differ arbitrarily from the real target on -ii-r---distiitttonnn (OOD) states. The discrepancy between the proxy in training and the real target in testing causes the distributional shift problem for the adaptation policy, As shown in Figure 1 BC-First overfits the dataset, which leads to erroneous generalization on OOD states and results in significant performance degradation in evaluation. Our algorithm outperforms the baseline and improves in both training and testing as the number of training step increases. Secondly, since the adaptation policy has access to the environment in training, unlike standard offline RL, we show that "conservatism" " 8 20, 19 for addressing distributional shift may not be sufficient in this setting. It is possible to benefit from deviation in multi-agent games if the worst-case rewards obtainable outside the dataset exceed those inside the dataset. We coin this challenge as risk-free deviation.
To address these challenges, we propose a novel learning objective, conservative offline adaptation. which optimizes the worst-case performance when maintaining conservative stance towards the target agent on the policy level. This objective provides unified approach to addressing the challenges of distributional shift and risk-free deviation. To optimize this objective, we introduce simple yet efficient algorithm, Constrained Self-Play (CSP), which simultaneously trains an adaptation policy and proxy model subject to regularizations. We prove that CSP produces near-optimal risk-free adaptation policy upon convergence, given that the target agent's policy within the support of the dataset can be well approximated. Empirically, we evaluate the effectiveness of our algorithm in four environments; a didactic maze environment, predator-prey in MPE (3), competitive two-agent
2

---

MuloCo environment 2 [0] requiring continuous control, and Google Football [2 Our results show that con-conservetive methods confront difficulties regarding offline policy adaptation. The experiment results demonstrate that our algorithm significantly outperforms baselines.
2 Related Work
RL in multi-agent games. Multi-agent reinforcement learning (MARL) under the centralized training decentralized execution (CTDE) paradigm have shown promising performances in learning coordination behavior for cooperative multi-agent tasks such as SMAC (35] and Google Football 21. Representative works consist of policy gradient algorithms 6 48 and value decomposition methods [34 40) There also have been extensive research in competitive environments such as Texas hold' em and MOBA games. Some works design theoretically sounded RL algorithms which are guaranteed converge to approximate NE [3. Some works propose RL methods based on self-play to train max-min policies which show strong empirical performances.
45 46
50
Offline RL. Offline reinforcement learning 22 learns purely from batched data. Representative works take a conservative view of out-of-distribution state-action pairs to mitigate the distributional shift problem. BCQ S uses VAE to propose candidate actions that are within the support of the dataset. IQL
[9] uses expectile regression to learn Q function to avoid querying out-of-distribution actions. CQL 20 puts regularization on the learning of function penalize outtof-distribution
actions.
Opponent modeling, Opponent modeling, sometimes also called opponent exploitation, is broad and extensively studied area 28 The offline adaptation problem studied in this paper a largely unstudied yet significant and challenging problem within the broader scope of opponent modeling. Some previous works assume access to the target agent's policy in training, and employ deep reinforcement learning to exploit the opponent in continuous control tasks using policy gradients estimated from direct interaction 12 10 I2]. Some works, including RNR [17] and SES (24) study safe exploitation given an estimated opponent model. They keep the exploitation policy close to NE in order to minimize its own exploitability. Some papers, like GSCU M7.. study zero-shot generalization ability to exploit totally unknown opponents. Additionally, this paper is orthogonal to some other opponent modeling subareas, including designing efficient models for predicting opponent behavior [13 B3, and addressing non-stationary opponents [7]. To our best knowledge, there lacks thorough investigation into the offline adaptation problem.
Please refer to Appendix B for more discussion.
3 Preliminary
For simplicity, consider two-player fully observable game G = (N, S, A, PM, r, 7) where N a set of two players, 5 is a set of states, Po is initial state distribution, and y is the discount factor. We use A, to denote the available actions for player i, and A A2). The transition probability is denoted by Pa : S x A X S - R. We use r; : S x - RR to denote the reward function for player i, and r = (21, In fully cooperative scenarios, r (s, a) r2(s, a) = r(s, which is termed multi-agent MDP (MMDP) 43 In zero-sum games, ri a) + r2(s, = 0. Player i's policy (s(8) -) is distribution over action space A, conditioned state s E S. The value function V. (((.. (s) is the expected discounted accumulated reward that player i will achieve under joint policy profile (IT m22 at state s. Similarly, the action value function Q (TT
37 3
(s, a) calculates the expected return when players play the joint action a = (01, 0z). Let di dr (s) be the discounted
visitation frequency of state s under policy profile (I1, π2): dx (s) 3) Ei 7 P(ss = s). We use J (22 to denote the return for player i under policy profile (TI, TT2).
Without loss of generality, assume that we are aimed learning an adaptation policy for player 1 (PI), while player 2 (P2), called the target agent, is playing a unknown fixed strategy πg- In offline policy adaptation, we do not have access to TB. Instead, a dataset D of TB' behavior data is given. Formally, D = F2 ((T,)), where F3 denotes extra data processing") and T, = {So, ag, .1....} records states and corresponding actions of an episode collected by n playing with rollout policies in We use Fb represent possible information loss in data collection, e.g., truncating and randomly masking.
3

---

IIc We do not make assumptions on the data collecting process, therefore, II, can contain any polices ranging from a weak random policy to a strong NE policy. Analogous to offline RL and robust RL. the dataset quality varies given different rollout policies, dataset sizes, and processing methods, which results in different problem difficulties. For instance, the dataset reveals adequate information about the target's behavior under diverse situations, BC-First can be enough. The challenges of offline policy adaptation becomes prominent the dataset only contains limited knowledge of TT.
The purpose of offline adaptation is to learn to adapt to unknown тB given a fixed batch of data D. Notably, different from offline RL, the access to the environment itself is still granted in training. If not, such setting can be reduced to standard imitation learning or offline reinforcement learning 44 [2)], because the target's behavior can be absorbed into environment dynamics. With the advantage of having access to the environment, this setting encourages conservative exploitation whenever risk-free deviation is possible.
Conservative Offline Adaptation
To address the problem of offline adaptation, we first examine the extrapolation error in policy evaluation between joint policies (π, TB) and (H, ni) in section 4.1 Here, я denotes the learning policy, тB represents the true target policy, and μ denotes any proxy model that estimates πg. This extrapolation error poses a risk to non-conservative policy adaptation algorithms, rendering them unsafe. However, traditional offline RL methods that aim minimize the impact of extrapolation error tend to be excessively conservative, which is suboptimal in this context. To allow for beneficial deviations from the dataset, introduce novel objective of conservative offline adaptation in section 4.2 This approach maximizes the worst-case performance against any dataset-consisten proxy (defined below). Finally, we propose practical Constrained Self-Play (CSP) algorithm in section 4.3 and formally prove that achieves near-optimal conservative offline adaptation. Please note that all proofs pertaining this section can be found in Appendix A
4.1 Extrapolation Error in Offline Adaptation
We first analyze the extrapolation error in policy evaluation between the joint policies (T, RB) and (T,, 14), which results in the overfitting problem shown in Figure I For theoretical analysis, first introduce the dataset consistent policy in Definition 4.] We will show that this assumption can be relaxed in algorithm design in a later section. Inspired by offline RL, in Proposition 4.2 find that the extrapolation error in offline policy adaptation can also be decomposed in a Bellman-like recursion R 44). We focus solely the value function of our learning agent (Player 1), and omit the subscript in the rewards and value functions for simplicity.
Definition 4.1. Dataset consistent policy. A policy 4 said to be consistent with aa on dataset D) iff. (1s, a) (8, a), (ss,a) E D. We denote the set of all dataset consistent policies as Cp-
Proposition 4.2. The performance gap of evaluating policy profile (H, p)) and (T,, TB) at state is E(s) V (s) (z), which can be decomposed in Bellman- recursion:
E(s)= Σ x(agls) (T p(azls)) (r(s,)))))))))))))) a') (V)))(s))
((als)TT(ozls))arr (a))s))))
01.02,5'
Theorem 4.3. We use E = En (o(s0)=(5O) to denote the overall performance gap between policy profile (π, ()) and (I,, TB). In any 2-player fully observable game, for all reward functions, E = 0 if and only (p(ass) p(a)s), Vs. dr (s) > 0.
Theorem 4.3 proves that the extrapolation error vanishes if and only if the estimation μ perfectly matches nB any state which can be visited by (T,, TT)) If extrapolation error exists, improvement of π against proxy y in training will not guarantee monotonic improvement in testing when x is evaluated with #a (as shown in Figure TT. Because make assumptions ay outside D, cannot bound [[p((]))) - T [a]s)] for s & D. Therefore, it requires that dx (s) = 0, Vs & D.
Corollary 4.4. Given a dataset consistent policy f E CD, which is consistent with aB on dataset D, the extrapolation error vanishes if Ys & D, dx x (s) 0.

---

Corollary 4.4 advocates excessive conservatism, similar to standard offline RL. to exclude any state- action pairs outside the dataset. It requires to constrain both π and μ such that the trajectories of (TT, (4) stay within D. However, this approach comes at the cost of relinquishing opportunities to exploit environmental access and explore the potential benefit of risk-free deviation from the dataset.
4.2 Conservative Offine Adaptation
In order to avoid being overly conservative and allow beneficial deviation from the dataset, we propose novel learning objective called conservative offline adaptation in Definition 4.5
Definition 4.5. Conservative offline adaptation (COA). Given an unknown policy B, and a dataset D of its behavior data, conservative offline adaptation optimizes the worst-case performance against any possible dataset-consistent policy:
max min J p); s.t. μ E Cp.
The adaptation policy n² is an optimal risk-free adaptation policy it is a solution to Objective
To show that COA enables risk free deviation, we decompose the return into two parts in Definition 4.6 (I) the return that is solely realized within the dataset, and (2) the return that arises from deviation. Definition 4.6. Within-dataset return & off-dataset return. The expected return of a joint policy profile (x1, x2) can be decomposed into the sum of within-datase return Jp (TT1 X2) and off-dataset return JF(TI, m2):
m2) = (7)(m1,72)
= -Br
Σ
at, 81+1) + B at,
Σ 12t
ar, 84,11)
(3)
15tb
JD( m2) (F( T2),
where 15 = arg max, (A] S 1,, E D) is a random variable indicates the last time step in a trajectory such that the state is still contained in dataset D.
For any policy т, and any dataset consistent policy μ E Cp. (4) = Jn(T, RB) because and тy behave exactly the same until the first time OOD state is reached according to the definition. Thus, the objective 2 is equivalent to max,, JD(n, (T, TB) + ming JF(m, ()), s.t. j E CD. The adaptation policy learns exploit cooperate with the target agent's policy exposed by dataset D. Mcanwhile, the max optimization of max, min,, (T, μi) encourages и to explore opportunities outside the dataset while maintaining conservative stance towards the target agent the policy level. It provides a unified approach to automatically decide whether to deviate depending on the magnitude of within-dataset return and worst-case off- return. In the extreme case, if min, (F(T, (1) is much larger than Jp (T, BB) ). can ignore the dataset completely and try to obtain rewards outside the dataset. Additionally, suppose that (T" ,u") is the solution to COA, then the extrapolation error s J(T", πB) J μx") JF(T", TB) - -J(TT", 2 O, which suggests that the testing performance will not be worse than training performance.
4.3 Constrained Self-Play
Directly optimizing the objective of COA is hard because in deep reinforcement learning, satisfying the dataset consistency assumption regarding the proxy model μ can be difficult, particularly for stochastic policies with limited data. We relax this assumption and propose simple yet efficient algorithm, Constrained Self-Play (CSP). We prove that CSP can achieve near-optimal risk-free policy adaptation. We start by showing in Theorem 4.7 that the gap in return bounded by the KL divergence of two policies of the target agent.
Theorem 4.7. In a two-player game, suppose that æ is player I's policy, a and μ are player 2's policies. Assume that max, DKL. (a(-Is)]lp(((ss)) I 5. Let RM be the maximum magnitude of return obtainable for player in this game, and let 7 be the discount factor. We use [ μ)) and J(T, o)) to denote the return for player I when playing policy profiles (T, 11) and (n, 1)) respectively. Then,
278
[ p) J( 2 -8MV28 1+
(4)
71-
5

---

With this result, we propose to optimize objective 5 which substitutes the hard dataset consistent constraint in objective with KL-divergence.
max max [TT, s.t. max (KL(TB((s)lla(())) 1
(5)
aED
We propose the Constrained Self-Play (CSP) algorithm, which optimizes objective S through learning both an adaptation policy and a proxy model with self-play and meanwhile minimiz- ing the KL divergence between μ and Tg on the dataset with regularization. The algorithm is shown in Algorithm | in Appendix C CSP utilizes soft BC regularization
minimize the KL- divergence: max, min, {J(T. p.) + BC B(s.a)-D|-oog {{a[s)]] }. It is because BC is equivalent to KL--iivegeence minimization 18 91. By increasing the coefficient CBC, CSP prioritizes KL-- divergence minimization. In the max-min optimization, the proxy model is trained adversarially against our adaptation policy by setting the proxy's 's reward function to be the negative of our agent's reward. We use MAPPO [48] the basic learning algorithm for players on both sides.
We prove in Theorem 4.8 that CSP approximates the optimal risk-free offline adaptation policy by some constant upon convergence. The gap depends on the maximum KL-divergence between the learned proxy μ and the target тB on E D. The gap decreases as μ becomes more precise estimation of тB within D. As sanity check, the gap vanishes when perfectly matches TB in D.
Theorem 4.8. Let nt be the optimal risk-free offline adaptation policy at the convergence of the optimization of objective 2 and let be the policy at the convergence of objective Then the worst-case adaptation performance of π is near-optimal:
min J(T", min J(T, ()) 2 min J(TT, - RMS22 (1- peCn
270
(6)
MECn
pECo
71-
Didactic Maze Example
In this section, we adversarial grid grid-world maze game demonstrate the challenges posed by offline policy adaptation and explain the superiority of our algorithm. Figure 2 displays the maze environment, where Player (P1) and Player (P2) start from the labeled positions and can move in four directions within the maze. The grids marked with numbers are terminal states with indicated rewards. There are five corridors (2 for P1 and for P2). each with different lengths and exponential final rewards" For simplicity, we use the notation P2 + 16 to denote P2's policy which going downwards the state with terminal reward 16. The first agent reaching any terminal state wins the reward, and the opponent receives the negative reward as penalty. The game ends immediately. We assume that P1 wins if P1 and P2 arrive at terminal states simultaneously. If no one reaches terminals in 10 steps, they both get penalty of -500. Both agents have full observation of the game states. Assume that P1 the opponent, and we want to find exploiting policies for P2.
We consider four baselines: (1) BC-First as explained in section I
(2) Self-Play (max-min):
approximating the pure max-min policy of P2 using self-play, which does not exploit; (3) Dataset- Only: completely following the dataset; and (4) Oracle: training an exploitation policy directly against the real target øg. Please refer Appendix for detailed learning curves in this environment.
256
258
CSP adaptation
64
suboptimal adaptation
The didactic maze example. The solid line indicates the testing trajectory of adaptation policy produced by CSP, while the dashed line indicates sub-optimal adaptation trajectories: (left) BC-First in maze case and (right) Self-Play in maze case 2.
Figure 2:
Case 1: Inadequacy of Non-Conservative Methods We show in this case that non-conservative algorithms easily overfit to an erroneous proxy model, lured to exploit "fake weaknesses" of the
Rewards are exponentially for efficient exploration of practical deep reinforcement learning algorithms.

---

Table 1: The training and test performances in maze environment (left) case and (right) case
CASE
Train
Test (against TB)
CASE 2
Train
Test (against TB)
BC-First
16
-4
BC-First
16
16
Self-Play
1
Self-Play
1
Dataset-Only
-64
-64
Dataset-Only
16
16
CSP (Ours)
1
CSP (Ours)
16
16
Oracle
Oracle
16 16
opponent. Suppose that PI's real policy TB as follows: (1) if observes P2 + 16, PI - 4; (2) P1 observes P2 - 256, P1 + 64; (3) otherwise PI does not move. This is reasonably strong policy for P1 which tries to exploit P2. Suppose the dataset is collected with Rø and P2's rollout policy is P2 256. Therefore, in the dataset, P1 is only observed going downwards. The proxy model trained by BC thus erroneously assumes that P1 - 64 under any conditions. The results are shown on the left side of Table The BC-First method makes disastrous generalization and responds with a P2 + 16 policy. Trying to exploit the "fake weaknesses" of the opponent results in failure in testing against the real opponent. However, CSP discovers that P2 -> 16 is unsafe and learns to safely win the game by playing P2 + 1. Both CSP and max-min policy achieve the optimal solution. The Dataset-Only method is overly conservative and only gets -64. The trajectories of CSP and BC-First when exploiting the real target are shown on the left side of figure
Case 2: Performing Risk-Free Adaptation In this case, we show that CSP outperforms pure Self-Play (which does not exploit the opponent) by discovering risk-free exploitation opportunities. Suppose that P1 always plays an aggressive policy Te: P1 -> 64. The dataset D is collected by л with P2 playing P2 16 and P2 - 256 randomly. The results are shown the right side of Table I and the trajectories of CSP and Self-Play are shown on the right side of Figure 2 The max-min policy produced by Self-Play cannot exploit P1 reckless policy. The generalization of the BC model indeed correct this specific situation, so BC-First is also optimal. Our algorithm recognizes that P2 16 risk-free exploitation because the dataset has witnessed that P1 still plays P1 64 P2 > 16 and successfully exploits the opponent.
It is noteworthy that different adaptation policies based solely on the different datasets used. It further validates that our algorithm is able to extract useful information from the dataset, and to perform risk-free adaptations.
the same hyperparameters for CSP in both cases and our algorithm learns
Experiment
We conduct extensive experiments on the predator-prey in MPE [39] a competitive two-player MuJoCo environment 2 10 that requires continuous control, and Google Football (1 In all experiments, we compare CSP with (1) BC-First, and (2) Self-Play, which produces an approximate max-min conservative policy of the game. Self-Play can also be viewed as an ablation of CSP which does not utilize an offline dataset to adapt to targets. We also compare with multi-agent imitation learning (MAIL), and previous opponent exploitation method RNR [17], which does not perform conservative offline exploitation. More experiment details are available in Appendix G We focus on performing opponent exploitation in competitive games. However, we also discuss the use of CSP for cooperative tasks Appendix
6.1 Experiments in Predator-Prey and MuJoCo
In the predator-prey environment, there are three adversarial agents, one good agent, and two obstacles. The good agent is penalized for being hit by adversaries, whereas the adversaries are rewarded. In experiments presented in Table 2 our agent controls either the good agent or the adversaries, respectively. We use pre-trained model as target gB- We report the episode rewards evaluation with и. We use three types of datasets (1OR, 100R, 100S) to demonstrate the impact of dataset quality in offline adaptation. The prefix number indicates the number of trajectories in the dataset. Letter "R" indicates that the dataset collected by лв playing with random policy, while "S" indicates that the dataset is collected by Fв playing with another well-trained opponent.
7

---

Table Testing episode rewards of adaptation policy in predator-pree prey of MPE.
Controls Good Agent
Controls Adversaries
Dataset
10R
100R
100S
10R
100R
100S
CSr(Ourr)
-16.7:0..4
-17.5=0.4
-1..1:0.1
34.6+1.6
33.6+:.22
34.7+:4.1
BC-First
-9.88+4.0
-21.8=1.66
-18.5+0.4
21.3+00.8
25.2
17.8
Self-Play
-17.6+1.5
-17.6+1.5
-17.6+1.5
29.742.8
29.7++2.8 29.7+-2.8
MAIL
-59.0+-0.6
-54.2+1.0
-37.9+1.4
4.3++0.3
4.140.2
25.8+00.7
Oracle
-1..0+0.3
-16.0+0.3
-16.0+0.3
34.3+6.2
34.3+662
3..3+662
Table 3; Experiments in MuloCo YouShallNoPassHummn environment. We show the winning rates and episode rewards against 4 independent target opponents..
Method
3
CSP(Ours)
0.72 = 0.06
0.65 + 0.04
0.85 = 0.08
0.88 + 0.02
Winning Rate
BC-First
0.65 0.13
0.46 + 0.14
0.71 + 0.17
0.72
0.07
Self-Play
0.54 + 0.20
0.59 + 0.14
0.52 + 0.02
0.60 + 0.19
CSP(Ours)
878 + 166
676 + 100
1203 + 209
1274 + 73
Episode Reward
BC-First
693 347
179 + 456
848 + 373
858 + 185
Self-Play
419 + 527
500 395
318 - 106
601 + 514
Intuitively, 1OR has the lowest dataset quality, while 100S has the highest. We train the Oracle directly against тh as an upper bound. Self-Play and Oracle do not use datasets. MAIL imitates the dataset collection policy. It fails to learn meaningful behavior collection policy is a random policy. Table 2 shows that CSP outperforms baselines and achieves comparable performances with Oracle, regardless of the dataset quality. CSP effectively learns to exploit the opponent safely, while the non-conservative method BC-First fails to do so. Moreover, BC-First is significantly more sensitive dataset quality. In the controlling adversaries scenario, even with the 100S dataset, BC-First still performs worse than Self-Play, which does not exploit at all. shows that the non-conservative method overfits the proxy model severely.
datasets 10R and 100R where the dataset
The Hanssnoss environment 2 10 is a MuJoCo-based two-player competitive game where the runner's objective is pass the blocker and reach the red line the opposite side. In our experiments, our agent acts as the runner, and the blocker is the target opponent for exploitation. In Table 3 Since the exploitability of different opponent models can vary significantly, results against different opponents are not directly comparable. So we report results for all targets. CSP successfully exploits the opponent, and significantly outperforms the baselines for all targets. Additionally, we include visualization of CSP's learned behavior in Appendix G
four independently pre-trained blocker models opponent targets.
6.2 Experiments in Google Football
Google Football 21 is a popular and challeng-
ing benchmark for MARL. In our experiments, the adaptation policy controls all players in one team, while the opponent target policy controls the 4 scenar- (3vsl, where agent acts as either the defender or the attacker), academy_ B_aaarru_____________ run_pass_anm aid_and__a and_shoot____ fender), and academmy_cuuneeaatten _aasse_aayyeeeee (defender). We report the winning rates of adaptation policies for five independently pre-trained opponent targets.
[SP)Dun)
02
0.0
other team. We conduct experiments ios: academy_3 3_vv______________________ Seth__wtt__eeeee
-32
0.4
-08
(RPS, de-
-20
(defender) (stocker) (drlender) (lelendde)
Figure 3: The average test-train perfor- mance gap in four scenarios of Google Foot- ball. A negative gap indicates the occur- rence of unsafe exploitation.
The experiment results in Table 4 show that our algo- rithm CSP outperforms baselines for almost all targets in all scenarios. Notably, the BC-First algorithm even

---

Table 4: Winning rates of offline adaptation policy scenarios of Google Football: 3vsl (defender),
3vsl (attacker), RPS (defender), and Counterattack Easy (defender). For each scenario, we experiment with independent target opponents.
Scenario
Method
2
4
5
CSP(Ours)
0.9 + 0.06
0.6 + 0.12
0.45 + 0.04
0.32 + 0.11
0.76 + 0.16
3
BC-First
0.64 + 0.04
0.46
0.09
0.2 0.01
0.16 + 0.04
0.56 + 0.09
defender
Self-Play
0.29 + 0.09
0.34 + 0.1
0.26 + 0.16
0.29
0.3 = 0.13
CSP(Ours)
0.81 + 0.14
0.88 + 0.02
0.83 + 0.07
0.84 + 0.05
0.78 + 0.08
3
BC-First
0.83 + 0.03
0.74 + 0.21
0.68 + 0.07
0.79 0.11
0.71 = 0.15
attacker
Self-Play
0.75 + 0.15
0.76 + 0.08
0.73 0.13
0.7
0.74 + 0.08
CSP(Ours)
0.51 + 0.33
0.71 0.07
0.56 0.13
0.38 + 0.07
0.79 + 0.09
RPS
BC-First
0.51 + 0.24
0,41 0.07
0.34 = 0.18
0.38 - 0.04
0.71 + 0.02
defender
Self-Play
0.57 + 0.14
0.36 0.04
0,25 0.04
0.37 0.1
0.76 + 0.03
Counter-
CSP(Ours)
0.93 + 0.02
0.88 + 0.04
0.81 + 0.25
0.81 + 0.02
0.75 + 0.06
attack
BC-First
0.7 + 0.17
0.52 0.07
0.53 0.09
0.69 + 0.14
0.5 + 0.09
defender
Self-Play
0.55 + 0.13
0.41 0.1
0.46 = 0.09
0.36 = 0.08
0.36 = 0.07
performs worse than Self-Play, which does not utilize the opponent's information to exploit. It indi- cates that previous non-conservative algorithms are not guaranteed to use the opponent's information safely in offline adaptation.
In Figure 3
show the gap of performances between testing and training: J(T, TB) J(m, (TTT.,,)) p). A negative gap indicates that although æ learns to exploit j during training, such exploitation is risky and can not be guaranteed in testing. The results demonstrate the significant challenges that offline policy adaptation poses for baseline exploitation algorithms. BC-First overfits the proxy model in performance drop in testing. Conversely, CSP has positive gap on
training and experiences average, indicating its ability to find conservative exploitation opportunities. Because CSP optimizes for the worst-case performance, it tends to produce proxy model that is possibly stronger than the real target. Thus, its evaluation performances are even higher than those in training.
6.3 Comparison with Non-Conservative Opponent Exploitation
We compare CSP with the previous safe exploitation method RNR [7]. which minimizes the distance to NE when exploiting. RNR assumes that the opponent plays NE policy with probability 1 and follows an estimated opponent model with It then learns an exploitation policy. However, RNR does not optimize the conservative offline adaptation objective, and it still fails to handle offline policy adaptation due to the neglect of possible errors in the estimated opponent model. In Figure the results show that CSP outperforms RNR for every dataset and every value of p. RNR(O) is equivalent to Self-Play, and RNR(1) is equivalent to BC-First. Results for these two have already been reported in Table 2 The results further support our claim that previous methods struggle with offline policy adaptation and highlight the efficiency and significance of our algorithm.
scenarlo = Adversery
scenafo Good Agent
40
opleadi roverd
mehhed
1N 2)
RNR 4)
RNRjO 6)
RNR10
(Cuss)
15R
1035
10R
100R
1005
Figure 4: Comparison with RNR [7] in predator-prey, where our agent controls (left) adversaries and (right) good agent respectively. The y-axis represents the episode reward in testing.

---

7 Conclusion
We provide thorough analysis of the unique challenges in offline policy adaptation, which are often neglected by previous literature. We propose the novel learning objective of conservative offline adaptation that optimizes the worst-case performance against any dataset-consis opponent proxy. We propose efficient algorithm that learns near-optimal conservative adaptation policies. Extensive empirical results in Maze, predator-prey, MuJoCo, and Google Football demonstrate that our algorithm significantly outperforms baselines.
One limitation of our work that optimizing worst-case performance in real-world applications may not always be necessary since participants are not fully rational in practice. One future direction to settle this problem is to extend offline adaptation to -ta-stationary opponents, which orthogonal to our current study. Those methods typically focus on adapting policy according to inferred opponent information evaluation, while our algorithm prevents performance degradation in testing against a consistent opponent. One way is to use our algorithm to train population of risk-free exploitation policies against opponents of different types and adapt exploitation policy testing according to the opponent's 's posterior distribution. We discuss potential negative social impacts in Appendix
Acknowledgements and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their insightful discussions and helpful suggestions. This work is supported in part by Science and Technology Innovation 2030- "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).
References
[1] Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018
[2] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent Complexity via Multi-Agent Competition. In International Conference on Learning Representations, 2018.
[3] Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining Deep Reinforcement Learning and Search
Informatonnoonnnoionnn Games. In Advances in Neural Information Processing Systems. volume 33, pages 7057-170699. Curran Associates, Inc., 2020.
[4] Imre Csiszár and János Kömnr. Information Theory- Coding Theorems for Discrete Memoryless Systems, Second Edition. Cambridge University Press, 2011.
[5] Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, lain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling, Bayesian Action Decuder for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1942- 1951. PMLR, June 2019.
[6] Jakob N Foerster. Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[7] Haobo Fu, Yo Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jicchao Xiong, Ying Wen, Kai Li, Junliang Xing. Qiang Fu, and Wei Yang. Greedy when Sure and Conservative when Uncertain about the Opponents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors. Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 6829-6848. PMLR, July 2022.
[8] Scott Fujimoto. David Meger. and Doina Precup. Off-Policy Deep Reinforvement Learning without Exploration. In Proceedings the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR, June 2019.
[9] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. Divergence Minimization Perspective on Imitation Learning Methods. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning. volume 100 of Proceedings of Machine PMLR, November 2020.
Learning Research, pages
10

---

[10] Adam Gleave, Michael Dennis, Cody Wild. Neel Kant, Sergey Levine, and Stuart Russell. Adversarial Poli- cies: Attacking Deep Reinfoncement Learning. In International Conference on Learning Representations. 2020.
[11] Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An. Online Ad Hoc Teamwork under Partial Observability. In International Conference on Learning Representations, 2022.
[12] Wenho Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial Policy Learning in Two-player Competitive Games. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3910-3919. PMLR, July 2021.
[13] He He, Jordan Boyd-Graher, Kevin Kwok, and Hal Daumé, III. Opponent modeling deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1813, New York, New York, USA. 20-22 Jun 2016. PMLR.
[14] Hengyuan Hu and Jakob N. Foerster. Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning. In International Conference Learning Representations, 2020.
[15] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakoh Foerster. "Other-Play" for Zero-Shot Co- ordination. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4399-4410. PMLR, July 2020.
[16] Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust Reinforcement Learning as a Stackelberg Game via Adapively-Reguulrizeed Adversarial Training. In Lud De Raedt, editor, Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3099-3106. International Joint Conferences on Artificial Intelligence Organization, July 2022.
[17] Michael Johanson, Michael Bowling, and Martin Zinkevich. Computing robust counter-strategies. 2007.
[18] Liyiming Kc, Matt Barnes, Wen Sun, Gilwoo Lec, Sanjiban Choudhury, and Siddhartha S. Srinivasa. Imitation Learning as f-Divergence Minimization. CoRR, abs/1905. 12888, 2019. arXiv: 1905.12888
[19] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning. In International Conference Learning Representations, 2022.
[20] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191. Curran Associates, Inc., 2020.
[21] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 4501-4510, 2020.
[22] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Leaming: Tutorial, Review, and Perspectives on Open Problems. CORR, abs/2005.00 00663,, 2020. arXiv: 2005.01643.
[23] Xun Li and Risto Müikkulainen. Dynamic adaptation and opponent exploitation in computer poker. In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence, 2018
[24] Mingyang Liu. Chengjie Wu, Qihan Liu, Yansen Jing, Jun Yang. Pingzhong Tang, and Chongjie Zhang. Safe Opponent-Explie Exploitation Subgame Refinement. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[25] Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, ZHIPENG HU, and Yaodong Yang. Towards Unifying Behavioral and Response Diversity for Open-ende endeeenddd Learning in Zero- sum Games. In Advances in Neural Information Processing Systems, volume 34, pages 941 9522. Curran Associates, Inc., 2021.
[26] Andrei Lupu, Brandon Cui. Hengyuan Hu, and Jakob Foerster. Trajectory Diversity for Zero-Shur Coordination. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7204-7213, PMLR, July 2021.
[27] Peter McCracken and Michael Bowling. Safe strategies for agent modelling in games. In Artificial Multiagent Learning, Papers from the 2004 AAAI Fall Symposium. Arlington, VA, USA, October 22-24, 2004, volume FS-04-02, pages 103-110 AAAI Press, 2004.
11

---

[28] Samer Nashed and Shlomo Zilberstein. A Survey of Opponent Modeling in Adversarial Domains. J. Artif. Int. Res., 73, May 2022. Place: EI Segundo, CA, USA Publisher: Al Access Foundation.
[29] Ling Pan, Longho Huang, Tengyu Ma, and Huazhe Xu. Plan Better Amid Conservatism: Offline Multi- Agent Reinforcement I. Learning with Actor Rectification. In Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17221-17237. PMLR, July 2022.
[30] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust Reinforcement Learning using Offline Data. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[31] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen Roberts. Effective Diversity in Population Based Reinforcement Learning. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18050-18062, Curran Associates, Inc., 2020.
[32] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust Adversarial Reinforcement Learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2817-2826. PMLR, August 2017.
[33] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 4218 PMLR, 10-15 Jul 2018.
[34] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforeement Learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4295-4304, PMLR, July 2018.
[35] Mikayel Samvelyan. Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung. Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs//902..00 1902,002,4443, 2019.
[36] John Schulman, Sergey Levine, Pieter Abheel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.
[37] Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations. Cambridge University Press, 2008.
[38] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[39] J. K Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sulivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall Williams, Yashas Lokesh, Ryan Sullivan, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforvement learning. arXiv preprint arXiv:2009.144: 2020.
[40] Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. (QPLEX): Duplex Dueling Multi-Agent Q-Learming. In International Conference on Learning Representations, 2021.
[41] Zhe Wu, Kai Li, Hang Xu, Meng Zhang, Haobo Fu, Bo An, and Junliang Xing. L2E: Learning to Exploit Your Opponent, 2021.
[42] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[43] Yaodong Yang and Jun Wang. An Overview of Multi-Agent Reinforcement Learning from Game Theoreti- cal Perspective. CoRR, abs/201 1.00883, 2020. arXiv: 201 1.00583.
[44] Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforce- ment Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 10299 10312. Curran Associates, Inc., 2021.
12

---

[45] Deheng Ye. Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu. Hongsheng Yu, Yinyuting Yin. Bei Shi, Liang Wang. Tengfei Shi, Qiang Fu, Wei Yang. Lanxiao Huang, and Wei Towards Playing Full MOBA Games with Deep Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33, pages 621-632. Curran Associates, Inc., 2020.
[46]
Deheng Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang. Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin. Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. Proceedings the AAAI Conference on Artificial Intelligence, (3(44)66672-6677 April 2020.
Scction: AAAI Technical Track: Machine Learning.
[47] Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang. and Yi Wu. Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. In The Eleventh International Conference on Learning Representations, 2023.
[48] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games. CoRR, abs/2103.01955, 2021. arXiv: 2103.01955.
[49] Chongjie Zhang and Victor R. Lesser. Multi-Agent Learning with Policy Prediction. In Proceedings of the Twenty-Fooee Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. AAAI Press, 2010.
[50] Enmin Zhao, Renye Yan, Jinqiu Li, Kai Li, and Junliang Xing. AlphaHoldem: High-Performance Artificial Intelligence for Hcads-Up No-Limit Poker via End-to-Fnd Reinforcement Learning. Proceedings of the AAAI Conference Artificial Intelligence. 36(4):4689-45) 469479999777, June 2022. Section: AAAI Technical Track on Domain(s) Of Application.
[51] Rui Zhao, Jinming Song, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Maximum Entropy Population Based Training for Zero-Shot Human-Al Coordination. CoRR, abs/2112.1170f. 2021. arXiv: 2112.11701.
13

---

A Proof
Proof of Proposition 4.2
Proposition 4.2 The performance gap of evaluating policy profile (T, and (τ, πB) at state is
E(s) V (5) (s), which can be decomposed in Bellman-likk recursion:
E(s) 7 (s) (s)
Σ x Par a) (TB(czls) (r(s, a,s) (V))H(s))
01.02.8'
Proof.
(())
(T.Tn(s) - (8)
Σ t(ails) Σ TB(as (a2 (s)
Σ
(V")((') + + re(s')
41
a2
Σ x(a1ss)
Σ ((azls) s))
Σ (ss( (s))
(8)
82
Σ m (s) Σ [[Tn(a2)s) - (((2ls)) A A+B), B),
41
42
where
A Σ PM(61ss.a)r(s....s)
(9)
and
== (n(0)
Σ PM(6'(s,aa) (V**(s') E('') - m(aals) Σ (a(s')s,a)>Vaut
(V* + e(s')) - ((a2)s) m(ools)7(=(((())
(10)
a) (s) )((l))) +
Putting equation 9 and 10 back, we have
E(8) V (s) (s)
Σ ( (s) Σ
(mD(a2ls)- u(a2ls)) ((nntt))--teb)))RRllllolloo P)
+ +
(I)
Σ x(a]1s) http:////ho//coloooooooooaaaa/aa (V
(1 a))(('))
a1,02,a'
Proof of Theorem 43
Theorem 4.3 We use 8 = En, po(s))e(so) to denote the overall performance gap between policy profile (π, (1) and (H,TB). In any 2-player fully observable game, for all reward functions, E = 0 and only if T (als) ((als), Vs, s.t. an (s) > 0.
14

---

Proof. Sufficiency. Whenever TB(als) = p(aas) holds, the first term in equation | is 0. If ss s.t. dz (s) > O, (a(s)) p(a)s), according to Proposition 4.2) by expanding the expression for (()) recursively, we have for vs s.t. dz (s) 0, e 0. Therefore, E ((8)))
Necessity. We first show by contradiction that ε Eo p(so)=(so) requires s.t. Po(s) > 0, (a(a)s) μ(als)) If (a(s)) + p(a|s), according to Proposition 4.2 we can change the reward function r(s, a, s') to change the value of E(s), because the reward function is arbi- trary. So & E., ((5) does not hold anymore, and this is contradiction. Therefore, E(s) = E s0.01 p(so) IT (41 (T) (22) 90) PM a))(s)) (E))(()))))(())))) Using the same ar- guments, can be shown that Vs, s.t. Pr(si 8) > O. (n(a = p(als). By expanding recursively, statement proved.
Proof of Theorem 4.7
We first prove a Lemma.
Lemma A.1. In a two-player game, suppose that a is player I's policy, a and p are player 2's policies. We use ((r,a) and TI joint
(TT to denote the joint policy profiles (π,, a) and (T, p). Then we have yoin (((aaa) (-1s]] ((rip))) (-1s)
DKL.
joint
DRI. (a(-1s)lla((1s)), where DKL. denotes KL divergence.
Proof. According definition,
T(a s)a(a))s)
DKL J.ooint (-Is) [[s)][[[[a]]]
joint (x, (1s))
Σ T(a2ls)a(a2(s) log a(a1ls))((a))
d ,2
Σ x(alls) Σ ((2))s)) log a(azls)
m(azls)
(12)
62
Σ (a(-Is))lu((ss))
= DEL (a(-Is)l
Additionally, we use the Theorem in 36 and we provide restatement. The original paper 33 minimizes accumulated discounted cost. contrast, an agent maximizes return our paper.
Theorem A.2. (Theorem in (361) Let max, (ala (8. a)]l, then
207
((T2) - ((T)) 2 E Exndyy (s) (awas(als) (s, (1 72
max DKL (2))))) (13)
Finally, we prove Theorem 4.7
Theorem 4.7 In two-player game, suppose that is player I's policy, and μ are player 2's policies. Assume that max, DxL (a(-Is)]lu((Is)) I 5. Let RM be the maximum magnitude of return obtainable for player in this game, and let y be the discount factor. We use J(x (4) and J(T, a) to denote the return for player when playing policy profiles (π,, 4) and (I, p)) respectively. Then,
2
J( J(T,a) - RM
1+ (1+
(14)
(1 (1-7)²
Proof. According Theorem
A.2
have
J( 14) - J(T, a) 2 Bondkr.a .))
E uo ( (T.00) (s, (s)
267
max DKL.
joint
(-|s) n joint
( (-1s))
(15)
(1 - 7)2
15

---

where e max, Ee, ao a)ll 2 0. Use Lemma
A.1
we have
25y
JTT J( a) 2 Bxnd Ba Bai ( a]]
max DKL (a(((s)llu(((s))
(1 (1-
2ey max DEL (a(-1s)llu(-1s))
2 -
(1 7)2
27
=
1 +
max DEL (a((1s))la((sss))
(1 7-
27,
2 -6 1 (1-
(16)
Next, we use KL divergence to bound According to definition, (Q( s Ras. So we have,
Be (,))) (s, (s,a)]
max
x(alls) Σ ((a2)s) (s) ((()a)(s)) -V(())())))))))
max
Σ (ails)
Σ
(u(a2ls) a(azls)) ( (s,
max
(17)
S max T ss)
Σ (u(a2ls) a(azls) Q  (s.a)
$ max
T ss)
Σ [x(aals) a(a2ls) RM
With Pinsker's inequality
4. the total variation distance is bounded by KL divergence:
8(a(-ls).u((1s) [a(((s)) - p(|ss) $ V V DKL (a((1s)llu((Is)).
(18)
Therefore,
S max Σ ((alls) 12)
41
V2RM max VDLL (a((ss)))))
(19)
< RM 26.
Put inequality 19 back into 16 and we get
2 (1 7)2
J(T,P) 28
1+
(20)
Proof of Theorem 4.8
Theorem 4.8 Let n= be the optimal risk-free offline adaptation policy at the convergence of the optimization of objective [ and let मे be the policy at the convergence of objective 5 Then the worst-case adaptation performance of near-optimal:
min [ ((T"; 2 min J( u) 2 min J(T", RMS (1 nECn
(1 2%, 7)
(21)
PECn
uECn
Proof. According to definition, T" is the solution to the optimization objective so mingee, J(x", (4) 2 min p(
Suppose that (T²,,²") and (7, ā) are the solutions to objectives 2 and respectively. For Vy Cp. let F(n) {a|Vs & D)a(s))) = wl-[s); max DIt. ( (=)ss(()ss)) < 8) be the set corresponding a policies which are identical to fr on OOD states. Observe that also holds. Therefore, according to Theorem 4.7 VT, Vu
max, DKL (
16

---

CD. Va E F(u), a(T 2 ((T RM 28 (1
278
Taking the minimization over j
(1-73²/
and a, we get Vn, min, millaF(()) [TT, ar) 2 min, [TT, 1)) - RM 26 1
(1-7)² Observe
2-6
that the left part is equivalent min (ea.a((le)llaa(a [ a). Taking the maximization
over π for both sides, we get J ai 2 J( ,1)) RM v 20 1 (
Let [ 14) minuee, J( p). Observe that p' also satisfies the KL divergence constraint: maX,ED (KL(TT((ss) (-1s) S 6, and (āt, â) is the optimal solution to objective 5 so J(H, 2
J( a). So we get min J a 11) 2 J(t, a) 2 ,(T",H") Ras Rarv25 (1-7)²
B Related Work
RL in multi-agent games. Multi-agent reinforcement learning (MARL) under the centralized training decentralized execution (CTDE) paradigm have shown promising performances in learning coordination behavior for cooperative multi-agent tasks such as SMAC (35) and Google Football 21 Representative works consist of policy gradient algorithms 6 48 and value decomposition methods 34 40 There also have been extensive research in competitive environments such as Texas hold' em and MOBA games. Some works design theoretically sounded RI. algorithms which are guaranteed converge to approximate NE [3]. Some works propose RL methods based on self-play 45 46 50 to train max-min policies which show strong empirical performances.
Offline RL. Offline reinforcement learning (2) learns purely from batched data. Representative works take a conservative view of out-of-distributie state-action pairs to mitigate the distributional shift problem. BCQ 8 uses VAE to propose candidate actions that are within the support of dataset. IQL
19
uses expectile regression to learn Q function to avoid querying -ut-t--distiibution actions.
CQL 20 puts regularization on the learning of function to penalize out-of-distribution actions.
Opponent exploitation. There have been extensive study on various aspects of opponent exploitation 28. Some employ deep reinforcement learning to exploit the opponent in continuous control tasks using policy gradients estimated from direct interaction [0] [2]. Some works, including RNR 17 and SES [24] study safe exploitation given estimated opponent model. They keep the exploitation policy close to NE in order to minimize the loss if the opponent changes its policy adversarially. Some work, for example, GSCU 77). studies zero-shot generalization ability exploit totally unknown opponents. To our best knowledge, there lacks a thorough investigation into the offline adaptation problem.
Robust RL. Formulated as robust-MDP (RMDP), robust RL 32 30 dedicated to learn a robust policy against perturbation in environment dynamics, e.g., observation noise, action delay. Formally, the unknown transition model in testing environment lies in an uncertainty set P which contains all models within a certain distance from the training model. Robust RL solves max-min problem, op- timizing the worst-case performance against any transition model in P. Some works uses adversarial training 32 16]. RORL 42 uses smoothing the Q function to learn robust policy. The similarity to our work is that, our offline adaptation problem also optimizes the worst-case performance against the target's 's behavior
out-of-distribution states. However, we do not have any distance constraints states. To our best knowledge, our paper is the first to investigate into the offline adaptation problem.
on the target's 's policy on
C Algorigthm
The CSP algorithm is illustrated in Algorithm [] The proxy model trained adversarially against our agent, therefore, we set the proxy's 's reward function to be the negative of our agent's 's reward. In our experiments on an n-player environment, we assume that we control nn players, while the target controls ny players, and 71 + n2 n. Therefore, we use MAPPO [8] as the basic learning algorithm for players of both sides, since players from the same side are fully cooperative. MAPPO deals with the cooperative multi-agent reinforcement learning problem through learning a centralized value function conditioned on global state, and a decentralized control policy conditioned on local observations. We use self-play with alternative update learn both adaptation policy and target's
17

---

proxy model simultaneously. We use soft behavior cloning regularization term to minimize the KL-divergence between proxy model j and target policy R--
Algorithm Constrained Self-Play (CSP)
Input: dataset D, environment enw, learning rate a, regularization coefficient CRC Output: adaptation policy и, target's policy proxy μ
1: Initialize adaptation policy critic v"; and target's policy proxy critic v
2: while not converged do
3: Collect trajectories {T:} + rollout(T, p4; env)
4: if' our turn then
5:
For each trajectory Ti calculate return target R, and advantage As of our side using GAE with value function vx for each step
6:
vº t Vd
aV.n.
JUV.] [is,, y-U:: [R] - D" (s,))
π a a [UT:] Lo min x(aslo)) A
7:
Tota( (z))
8:
clip
Kotelas [0:] - 1,, A.)
m((alox)
9:
else
10:
For each trajectory Tr, calculate return target RT and advantage Ag of target agent side using GAE with value function D" for each step t
11:
B s sample random batch from D
ydth t VV pa
E(m. R2] - v)
[UT
13:
n aV |UTS] Fus AOU- min
(oco(a)lo))
ulae los) As,
clip
Hotd(ar (o)) e, 1.11))) Ar)
poa
CBC 1 E(o,)EB log u(alo)
15: end if
end while
D Experiment on Didactic Maze
We show experiment details of the Maze example in this section. The learning curve in case is shown in Figure In the BC-First algorithm, since the dataset only contains trajectories that P1 64, the BC model makes wrong and risky generalization that assumes Pl always goes downwards 64 no matter how P2 acts. Therefore, the P2 policy learns to exploit this "weakness" by playing P2 > 16, and achieves 16 in training phase. However, during testing, P2 obtains -4 reward through trying to exploit an nonexistent weakness of PI.
Our algorithm avoids the trap of imaginary weakness, and learns to safely win the game, while the BC-First method makes risky exploitations which will not work in evaluation finally. Our algorithm keeps conservative for states outside dataset, and admits the possibility that P1 could go leftwards to win the game P2 does not plays P2 1.
algorithm
5.0
5.0
seltplay first csp
2.5
avg stsp reward
avg step rward
0.0
Apa
-2.5
-2.5
5.0
algorilhm selfplay
-5.0
frst
-10.0
50000
100000
150000
50000
100000
150000
step
slep
Figure 5: The training curve of amortized average per step reward maze game case in training (against the proxy). The x-axis represents training steps. Left. Average per step reward for P1. Right. Average per step reward for P2.
18

---

The learning curve in case 2 is shown in Figure 6 As can be seen in Figure 6 our algorithm quickly learns to exploit while the max-min strategy produced by Self-Play fails
10
algorithm
5.0
salfplay first
2.5
5
avg stop reward
0.0
-2.5
-5.0
-7.5
algorthhm
selfplay
-10.0
BC first
-1255
csp
50000
100000
150000
50000
100000
150000
step
The training curve of amortized average per step reward in maze game case in training (against the proxy). The x-axis represents training steps. Left. Average per step reward for P1. Right. Average per step reward for P2.
Figure 6:
We use the same set of hyper-paaee parame-rrseeterss (including the same coefficient Cgc for the BC regular- ization term) for these two cases. The hyper-paramedd
shown in Table S The dataset D1
[trajectories(P1 + 64, P2 + 256) in case 1, and our algorithm learns to play P2 - 1: while in case 2, given dataset D2 { (traeectories(2] P1 + 64, P2 256), trajectories P1 + 64, P2 16)]. our algorithm learns to play P2 > 16. We use the same set of hyper-parameters for both cases. Therefore, our algorithm produces different policies simply because of different datasets given. It further validates that our algorithm is able to extract useful information from dataset, and only perform risk-free adaptations.
Table 5: Hyper-parameten for maze.
ppo_epoch
num_min mini batch
entropy_com coef
0.3
use gae
True
gamma
0.99999
gae gae_lanee lambaa
0.95
critic_Ir
7e-4
Ir
7e-4
weight_decaa
0
adam_epp
le-5
prrollout threads
20
apo_episode_engee
12
data chunk length
12
steps
1.8K
max grad norm
0.5
bc www.rrllaizatton.ccoom _oef
10
be oo_bacch_sip
8
network
MLP
E Conservative Offline Adaptation for Cooperative Tasks
We evaluate our method in competitive games to perform opponent exploitation. Nevertheless, our method is also applicable cooperative environments. The objective of CSP in cooperative games maintains the same min-max structure because it promotes adherence to the policy exposed by dataset for states within the dataset, while aiming to optimize for worst-case performance for states dataset. Therefore, in cooperative games, the policy regularization term incentivizes the adaptation agent to collaborate with the target agent on states within the dataset, and the minimization over opponent proxy μ encourages the adaptation agent to be conservative and not to rely on the target agent's OOD policy. Note that in objective 5 and Algorithm [ no matter whether the game is competitive cooperative, the minimization over f in training can be achieved by setting the target agent's reward function be the negative of our adaptation agent's 's reward function. We then train μ to maximize this reward with policy regularization in self-play.
the
Although our method is applicable cooperative environments well, the challenges caused by offline policy adaptation are more significant in competitive games. Recall that conservative offline adaptation optimizes
max min [[T, ni), s.t. / E Cp-
(22)
For cooperative tasks, requires that the teammate μ will not cooperate on states outside dataset in cooperative games. Therefore, for the adaptation policy, staying within dataset could be trivial and
19

---

near-optimal solution for most cooperative problems if we would like to maximize the worst-case performance against any dataset-consistent teammate.
F Potential Negative Social Impacts
Inappropriate of exploitation algorithms may result in negative social impacts. An example of its negative impact the exacerbation of inequality in society. Companies who have access to large amounts of data can utilize their customers more effectively. However, general-purposs and depends on pre-collected data. We advocate for strict laws and regulations that protect user privacy data avoid negative impacts. It is recommended that products utilizing exploitation algorithms are made public and supervised.
algorithm is
G Experiment Details
G.1 Environments
The experiment environments are illustrated in figure 7
Figure 7: Illustration of experiment environments used in this paper. From left to right: (1) predator prey in MPE, (2) YouShallNooPassHumaa in MuJoCo, (3) Google Football.
Predator-Prey There are one good agent, three adversarial agents and two obstacles in the predator- prey environment. The good agent is faster and receive negative reward for being hit by adversaries while adversaries are slower and are rewarded for hitting the good agent. All agents are initialized randomly in the environment.
The Houshaall environment 10 creates a two- player competitive game based on MuJoCo, where one humanoid (the runner) is aimed at passing the other humanoid (the blocker) to reach the red line on the opposite side. The environment itself is challenging because has high dimensional observation space and requires continuous control. In our experiments, we assume that our agent acts as the runner, while the blocker is the target opponent to exploit. We use four independently pre-trained blocker models as opponent targets. For fairness, we generate these targets with exactly four random seeds without any selection. Since the exploitability of different opponent models can vary significantly, results with respect to different opponents are not directly comparable. So we report results with respect to all targets.
HouShallNooPassHuumas
Google Football Google Football 21 is a popular and challenging benchmark for MARL. In our experiments, our adaptation policy controls all players in one team, while the opponent target policy controls the other team. Since all players in the same team are fully coopera- tive, we use MAPPO [48] to learn decentralized policy. We conduct experiments in 4 scenar- ios: acudemy_3_vv 1_v_yy33___________ with_keeper (3vsl, where our agent acts as either defender or attacker), (RPS, defender), and academy_cone coontertttttktttccccckkkkk _easy (de- fender). We report the winning rates of adaptation policies for 5 independently pre-trained opponent targets. For attacker, winning rate refers to the percentage of episodes that the attacker scores, while for defender, refers to the percentage of episodes that the defender prevents the attacker from scoring.
academy_ree Batps_uuu__ass_____e _ii_and_hhe serpootwwtth_keeee
G.2 parameesss
our experiments, we use MAPPO 4% as the base RI. algorithm to learn policies on both sides (ours and the target's). MAPPO is an extension of single agent PPO to multi-agent reinforcement learning
20

---

within the centralized training decentralized evaluation (CTDE) paradigm. It learns centralized value function conditioning on global state to promote coordination among the agents. We selected MAPPO as the base learning algorithm due to its simplicity and empirical strong performance.
For the BC-First method, we exactly the same MAPPO (with the and the same number of training samples) as CSP to train our adaptation policy, while keeping the target's proxy fixed. In order to ensure that the the performances of BC-First method are not encumbered by an under-trained proxy model, we use the same network structure for the proxy model as the real target model, and train enough steps make sure that behavior cloning has converged. For the Google Football environment, we use the same paper [48] for all scenarios. The hyper- parametrsss are listed in Table and
of hyper-parameters
as reported in the MAPPO
Table 6:
for predator-pre prey in MPE.
ppo_epoch
10
num mini batch
entropy cotoopy_coef
0.01
use gae
True
gamma
0.99
gae lambda
0.95
critic_Ir Ir
7e-4
Ir
7e-4
Bright_decap
0
addam_eps
le-5
n rollout threads
128
ppo ppo_cpisode length
50
10
steps
5M
max max_grad graddnorm _nom
0.5
bc res_ruulaization_cooo coef
0.003
bc s__bachh_ssie
8
network
RNN
Table 7: parpee-prraeters for MuJoCo.
ppo_epoch
num num_mini batch
1
entropy_ _cef
0.01
use_gae
True
gamma
0.99
gac lae_lambda
0.95
critic_Ir
7e-4
7e-4
weight_decay
0
adam eps
le-5
rollout threads
100
ppo leissdd__eength length
200
data chunk length
steps
40M
max grad norm
0.5
bc sog_reullaiiaatin_ncom coef
0.1
be batch_size
256
network
MLP
Table 8: Hyper-parameters for Google Football.
ppo_epoch
15
num_mini batch
2
entropy_coke coef
0.01
use_ gac
True
gamma
0.99
gae_ la__lambd
0.95
critic_Ii
7e-4
7c-4
weight_ _eecay
adam_eps
le-5
rollout threads
50
ppo_episs apo_episode_ler benisodellengthh length
200
data jiaa_chunk_eenttt
10
steps
25M
max_ max__grad_now
0.5
be wt__eeularizaaion_coom
5.0
bc_ bc__atch______ size
256
network
RNN
G.3
Computing Resources
Each seed run on GPU server with one NVIDIA P100 GPU, and Intel(R) Xeon(R) Gold 6145
CPU @ 2.00GHz CPU. Each run can finish within 24 hours.
G.4 Target Models & Dataset Collection
For all the environments, we user different runs of self-play get the group of targets. In order to ensure fair comparison, we use the exactly the same number of random seeds to generate these targets, without any selection. In our experiments, can produce diverse targets. For instance, in Google Football, different targets may have different tendencies to pick which side start attack (left or right). Moreover, can be seen from Table 4 the difficulties to exploit these targets are diverse. For MuJoCo and Google Football, we use dataset consisting 5 trajectories. We collect these trajectories using the target model together with rollout policy. We observe that the 3 kinds of rollout policies: (1) random policy, (2) environment's bot, (3) target itself, do not have significant impacts on the experiment results in the MuJoCo and Google Football environments.
observe that different runs with different seeds
21

---

G.5 Visualization
Blocker (red) datasel tis to block the uppe path
A
Burner louer datser makes doou thoug the upperpoth
Rooker red leared in CSP lears to blok the uppe path and elso tre to book nanner a the over uath
B
Burner cou
(our fernati prdeer poppli lover
Blocker (red) in dataset thes to block the upper path
gereen GoP pidk fee pabrand eeeeiet
Figure 8: The visualization of policy behavior in MuloCo. Top. The trajectory in dataset, where the blocker the real opponent policy Ta and the runner the policy which collects the dataset. Middle. The trajectory during the training of CSP. Both agents are controlled by CSP. Bottom. The evaluation of CSP's runner with the real opponent blocker.
In Figure g we further consolidate our claim by visualizing the policy behavior in MuJoCo. As can be seen in this example, in trajectory contained in dataset, the runner (blue agent) typically makes detour through the upper path to avoid being pushed down by the blocker (red agent). The opponent blocker also has the tendency walk upwards stop the runner. The runner trained by CSP learns to exploit the policy represented by the dataset, and prefers making detour through the lower path, as can be seen from the middle and bottom lines of replays. Although the behavior of runner going downwards is not contained in the dataset, the blocker trained by CSP can still learns to stop such runner, which makes the runner even more robust and stronger than the real target model. Therefore, when confronted with the real opponent blocker, who only knows how to defend the upper path, the runner trained by CSP steadily wins the game.
G.6 Reward.
For predator-prec prey, we use the environment's 's original reward. For MuJoCo, we use dense rewards for locomotion learning as previous works 2 [0 In Google Football, use both the sparse scoring reward as well as the dense checkpoint reward which awards the attacker according to the ball handler's 's distance to the goal. Although the rewards for the attacker are provided by the environment, Google Football does not provide rewards for the defender team in academy scenarios. In our experiments, we just use the negative the attacker's reward train defender's policy to make the game zero-sum.
G.7 Explanation of Training & Testing Performances
The observed lower training performance of CSP than the baseline, as shown in Figure 1 is expected behavior since the opponent model used by CSP during training differs from that of BC-First. In BC- First, the opponent model solely pre-trained on dataset and remains fixed, leading to significant vulnerabilities in aat-of-distribution states which can be easily exploited. Conversely, in CSP, the opponent model is trained an adversarial manner simultaneously with the exploitation policy. Due to its evolutionary nature, it can compensate for its vuinerabilities and becomes significantly more
22

---

challenging to exploit. Consequently, the training performance of BC-First is higher. During testing, both exploitation policies undergo evaluation using the same real target model.
23


Reviews:
[{"review_id": "YNyuq0It7j", "reviewer": "Reviewer_dBj1", "summary": "This paper presents a novel approach to offline policy adaptation in multi-agent games, focusing on leveraging the target agent's behavior dataset and access to the game environment during training. The authors identify the challenges of distributional shift and risk-free deviation in offline policy adaptation and propose a new learning objective called conservative offline adaptation to address these issues. They introduce an efficient algorithm, Constrained Self-Play (CSP), which trains an adaptation policy and a proxy model with regularizations. The paper provides theoretical proof of CSP's effectiveness and demonstrates its superiority over non-conservative methods through empirical evaluations in various environments, including Maze, predator-prey, MuJoCo, and Google Football.\n\n", "questions": "Please refer to Weaknesses Section\n\n", "limitations": "Please refer to Weaknesses Section\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The specific problem of offline policy adaptation is a highly meaningful and novel issue, particularly with a wide range of potential applications in the industrial sector.\n\n2. The article is easy to understand, with a comprehensive background and related work section. \n\n3. Although the ideas presented are simple and heuristic, they are supported by theoretical proofs and significant experimental results.\n\n\n", "weaknesses": "1. The paper does not discuss the additional computational and time costs associated with offline adaptation.\n\n2. Offline adaptation and opponent modeling share some similarities, so it is important to discuss their differences and include comparative experiments to highlight the distinctions between the two approaches, rather than just comparing self-play and behavior cloning methods.\n\n3. The term \"risk-free\" is mentioned several times in the paper. Please provide a more specific description of what \"risk\" refers to in the context of the paper, or provide some examples to clarify its meaning.\n\n4. The paper discusses issues related to human-AI interaction, and it would be beneficial to address the associated ethical concerns in the context of the proposed approach.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for commenting that offline policy adaptation is \"highly meaningful and novel\", and the theoretical proofs and experimental results effectively support our idea.\n\n**Q1. Are there any additional computational and time costs associated with offline adaptation?**\n\n**A1.** The computational and time efficiency of our algorithm CSP is comparable to that of Self-Play. While CSP incorporates an additional policy regularization step, it mainly entails (1) sampling a batch of data from a pre-collected dataset, and (2) minimizing the regularization loss through backpropagation (refer to lines 11 and 13 of Algorithm 1 in Appendix C). The incurred computational and time costs are marginal compared to other RL procedures.\n\nIt's worth mentioning that both CSP and Self-Play use half of the environment steps to train the exploitation policy and the other half to train the opponent model due to alternative update. In contrast, BC-First and other opponent exploitation methods which rely on a fixed pre-trained opponent model can use all steps to train the exploitation policy. In our experiments, we ensured that all methods were given the same overall number of environment steps. CSP outperforms BC-First, even when utilizing only half of the data for training the exploitation policy.\n\nThe table below shows the average training time of different methods in Google Football with 25M overall environment steps.\n\n| Method        | CSP (Ours) | Self-Play | BC-First  |\n| ------------- | ---------- | --------- | --------- |\n| Training Time | 12h 17min  | 12h 10min | 10h 55min |\n\n**Q2. Offline adaptation and opponent modeling share some similarities, so it is important to discuss their differences and include comparative experiments to highlight the distinctions between the two approaches.**\n\n**A2.** We have discussed the differences of our work from previous works in opponent modeling in the Introduction (Lines 39-47) and the Related Work (Lines 91-97). We will refine the discussion in the next revision.\n\nOpponent modeling (sometimes called opponent exploitation) is a broad and extensively studied area [1]. The offline adaptation problem we studied in this work is a largely unstudied yet significant and challenging problem within the broader scope of opponent modeling. Some previous works assume access to the target agent's policy in training [2, 3], while some others assume no prior knowledge [4, 5] and concentrate on zero-shot exploitation or cooperation. In contrast, as the reviewer commented, our work introduces a novel setting of offline policy adaptation, where we leverage the target agent's dataset and access to the environment to train an adaptation policy.\n\nOur paper reveals both theoretically and empirically that methods failing to adopt a conservative perspective on the opponent's out-of-distribution (OOD) policy are inadequate in effectively addressing the problem. Our CSP algorithm promotes worst-case optimization for OOD policy and significantly outperforms the baselines. In addition to Self-Play and BC-First, we also compare CSP with RNR [3], an exploitation algorithm that minimizes the distance to NE during exploitation, in Section 6.3. Notably, CSP outperforms RNR as well.\n\nFinally, it is crucial to note that our paper is orthogonal to other opponent modeling subareas, including designing efficient models for predicting opponent behavior [6, 7], and addressing non-stationary opponents [4], which we have discussed in the Conclusion (Lines 342-349).\n\n**Q3. Provide a more specific description of what \"risk\" refers to in the context of the paper.**\n\n**A3.** We appreciate the reviewer for pointing it out. In the context of this paper, \"risk\" intuitively refers to the likelihood of the adaptation policy experiencing a decline in performance in evaluation. Formally, let $\\pi_B$ denote the unknown real target policy, $\\pi^*$ represent the adaptation policy, and $\\mu$ be any proxy model utilized in training $\\pi^*$ as a substitute for $\\pi_B$. Consider $J(\\pi^*, \\mu)$ as the performance of $\\pi^*$ in training and $J(\\pi^*, \\pi_B)$ as its testing performance against the real target agent. Risk refers to chances that $J(\\pi^*, \\pi_B)< J(\\pi^*, \\mu)$, implying that the performance observed during training may not be guaranteed in evaluation.\n\nFor examples clarifying the concept, please refer to the right part of Figure 1 and the left part of Table 1. Non-conservative adaptation methods may make erroneous generalization. The evalution performance is much lower than that in training.\n\nOn a contrasting note, our paper proves (Lines 190-192) that conservative offline adaptation (COA) is risk-free: $J(\\pi^*, \\pi_B)-J(\\pi^*, \\mu)\\geq0$. Additionally, in section 6.2, we also empirically demonstrate that the average gap for our algorithm CSP is positive, but the baseline method remains highly risky.\n\n**Q4. Explain the associated ethical concerns since the paper discusses issues related to human-AI interaction.**\n\n**A4.** We have discussed the potential negative social impacts in Appendix F. For example, companies with access to extensive data may exploit their customers more effectively. However, our algorithm is general-purpose and depends on pre-collected data. We emphasize the importance of stringent laws and regulations that safeguard user privacy to mitigate any adverse effects.\n\n**References**\n\n[1] Albrecht et al. Autonomous agents modelling other agents: A comprehensive survey and open problems, 2018.\n\n[2] Gleave et al. Adversarial Policies: Attacking Deep Reinforcement Learning, 2020.\n\n[3] Johanson et al. Computing robust counter-strategies, 2007.\n\n[4] Fu, et al. Greedy when Sure and Conservative when Uncertain about the Opponents, 2022.\n\n[5] Lupu et al. Trajectory Diversity for Zero-Shot Coordination, 2021.\n\n[6] He et al. Opponent modeling in deep reinforcement learning, 2016.\n\n[7] Rabinowitz et al, Machine theory of mind, 2018."}, {"review_id": "3dtGFRAAyF", "reviewer": "Reviewer_71Zz", "summary": "In this paper, the authors target the offline policy adaptation problem in multi-agent games where \"offline\" indicates no interactive target agents and an accessible environment. The problem is challenging as it inherits the distributional shift challenge from offline RL, and is likely to lead to risky actions. To tackle these problems, the authors introduced the novel conservative offline adaptation objective to optimize the worst-case performance and proposed the constrained self-play, i.e., CSP algorithm to simultaneously train an adaptation policy and a proxy model subject to regularization.", "questions": "1. If in the cooperative games, the objective for CSP remains the same min-max problem?\n2. What is the connection between Theorem 4.7 with the CPS algorithm?", "limitations": "The authors leave the limitation where not all agents are rational in future work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and easy to follow with good visualizations. \n2. There are plenty of theoretical results supporting the proposed solution. \n3. Extensive experiments are done to validate the proposed solution.", "weaknesses": "1. Novelty. The proposed offline policy adaptation problem seems to be a special case of opponent modeling problem [1,2] where the static opponent's data are available. Therefore, the studied problem is not new. \n\n2. Theoretical. I did not see the connection of Theorem 4.7 with the CPS algorithm. \n\n[1] Nashed, Samer, and Shlomo Zilberstein. \"A survey of opponent modeling in adversarial domains.\" Journal of Artificial Intelligence Research.\n\n[2] Haobo Fu, et al. \"Greedy when Sure and Conservative when Uncertain about the Opponents.\" ICML, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The proposed offline policy adaptation problem seems to be a special case of opponent modeling problem [1,2] where the static opponent's data are available.**\n\n**A1.** We have discussed the differences of our work from previous works in opponent modeling in the Introduction (Lines 39-47) and the Related Work (Lines 91-97). Opponent modeling (sometimes called opponent exploitation in non-cooperative settings) is a broad and extensively studied area [1]. The offline adaptation problem we studied in this work is a largely unstudied yet significant and challenging problem within the broader scope of opponent modeling. Different from previous works that assume access to the opponent's policy in training [2, 3] and from others that assume no prior knowledge [4, 5], we leverage the opponent's dataset and maintain access to the environment to train an exploitation policy.\n\nOur paper presents theoretical and empirical evidence that previous methods, lacking a conservative perspective on the out-of-distribution (OOD) behavior of the opponent, are insufficient in effectively addressing the problem and suffer from considerable performance degradation. In contrast, our algorithm CSP performs worst-case optimization for OOD policy. We prove that its converged solution is close to optimal, with the gap bounded by the approximation accuracy of the opponent's policy on states in the dataset. Experimental results demonstrate that CSP outperforms the baselines.\n\nThe second paper mentioned by the reviewer, GSCU [4], has been discussed in Lines 95-97 of our paper. However, it actually focuses on zero-shot exploitation of completely unknown opponents, which is different from our setting. In the table below, we show the experiment results of GSCU on the predator-prey task where we control the prey agent. We can observe that CSP significantly outperforms the baselines. GSCU fails to effectively exploit the opponent due to its lack of utilization of prior information about the opponent.\n\n| Dataset       | 10R           | 100R          | 100S          |\n| ------------- | ------------- | ------------- | ------------- |\n| **CSP(Ours)** | **-16.7±0.4** | **-17.5±0.4** | **-17.1±0.1** |\n| BC-First      | -29.8±4.0     | -21.8±1.6     | -18.5±0.4     |\n| GSCU          | -27.7±2.8     | -27.7±2.8     | -27.7±2.8     |\n| Self-Play     | -17.6±1.5     | -17.6±1.5     | -17.6±1.5     |\n| Oracle        | -16.0±0.3     | -16.0±0.3     | -16.0±0.3     |\n\n*: We directly use the trained weights of GSCU model provided by the original paper to test in our experiment settings. Since GSCU performs Bayesian updates in testing, we independently evaluate GSCU with three random seeds and report the mean and standard error. GSCU does not depend on the dataset. Exploiting policies control good agents in this task.\n\n**Q2. Explain the connection of Theorem 4.7 with the CSP algorithm.**\n\n**A2.** Theorem 4.7 bridges the gap between Objective 2 (Line 172) and Objective 5 (205), and serves as a necessary prerequisite to establish our main result, Theorem 4.8, which proves the proximity of the converged solution obtained by the CSP to optimality.\n\nWhile Objective 2 (Line 172) produces an optimal risk-free adaptation policy, satisfying its dataset consistency constraint is difficult. To address this issue, we replace the constraint with a KL divergence and propose Objective 5 (Line 205): $\\max_\\pi\\min_\\mu J(\\pi, \\mu)\\ s.t.\\ \\max_{s\\in D}D_{KL}(\\pi_B(\\cdot|s)\\|\\mu(\\cdot|s))\\leq \\delta$. This modified objective becomes feasible for learning in CSP. What remains is to establish a connection between the solutions obtained by optimizing these two objectives. Theorem 4.7 demonstrates that the gap in return can be bounded by the KL divergence of two policies of the target agent. Building upon this result, Theorem 4.8 concludes that the solution obtained by CSP (Objective 5) is also in proximity to the optimal solution (Objective 2), with the gap bounded by the KL divergence between the learned proxy model and the real target policy on the dataset.\n\n**Q3. Why does the objective for CSP remains the same min-max problem in cooperative games?**\n\n**A3.** The objective of CSP in cooperative games maintains the same min-max structure because it promotes adherence to the policy exposed by dataset for states within the dataset, while aiming to optimize for worst-case performance for states not in the dataset (Objective 5, Line 205). Therefore, in cooperative games, the policy regularization term incentivizes the adaptation agent to collaborate with the target agent on states within the dataset, and the minimization over opponent proxy $\\mu$ encourages the adaptation agent to be conservative and not to rely on the target agent's OOD policy.\n\nWe have explained in more details about the implementation of CSP in cooperative games in Appendix E. CSP optimizes the worst-case return with respect to the adaptation agent's utility with policy regularization. No matther whether the game is competitive or cooperative, the minimization over $\\mu$ in training can be achieved by setting the target agent's reward function to be the negative of our adaptation agent's reward function. We then train $\\mu$ to maximize this reward with policy regularization in self-play.\n\n**References**\n\n[1] Albrecht et al. Autonomous agents modelling other agents: A comprehensive survey and open problems, 2018.\n\n[2] Gleave et al. Adversarial Policies: Attacking Deep Reinforcement Learning, 2020.\n\n[3] Johanson et al. Computing robust counter-strategies, 2007.\n\n[4] Fu, et al. Greedy when Sure and Conservative when Uncertain about the Opponents, 2022.\n\n[5] Lupu et al. Trajectory Diversity for Zero-Shot Coordination, 2021."}, {"review_id": "ljD9QZcrhn", "reviewer": "Reviewer_bBXB", "summary": "This paper considers multi-agent games where you have access to the\nenvironment and you have a previously collected dataset from the\nopponent you will be facing.  So it is offline in the sense that you\nonly have data on the opponent, but online in the sense that you can\npractice as much as you want in the environment -- you just need to\nsupply your own opponent policy, possibly informed by the opponent\ndataset you are given.  The paper proposes an RL algorithm (called\nConstrained Self Play) whose loss function incentivizes adherence to\nthe dataset for states in the dataset and worst-case optimization for\nstates not in the dataset.  It provides related theoretical analysis.\nCSP is tested empirically on several multi-agent adversarial games.\n", "questions": "Some smaller comments:\n\nline 60: \"improve in training and testing\".  I'm not sure what the\ntraining part of this refers to.  In figure 1, the training\nperformance of CSP is below the baseline.\n\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea for CSP is interesting and well supported by the arguments in\nthe narrative.  The problem setup is an interesting one to consider as\nmany adversarial applications will match it reasonably well.  The\ngames tested in the empirical study are sufficiently complex to show\nnon-trivial behavior.\n", "weaknesses": "My score is limited by the fact that I did not find the theoretical\nanalysis useful.  My concerns with it are as follows:\n\n- The statements that \"we prove that CSP learns a near-optimal\n  risk-free offline adaptation policy\" are overstatements given the\n  differences between the algorithm and the analysis.  There are a\n  number of factors that mean there *is* risk in applying this method.\n\n- Objective 2 can not be optimized in general.  Objective 5 is designed\n  to address some of the issues by replacing the constraint with a KL\n  divergence.  However, there is no way to guarantee or measure delta.\n  It will be affected by statistical error due to low sample size in\n  the dataset as well as failures in the optimization thus leading to\n  an unknown and arbitrarily large delta.\n\n- Theorem 4.8 assumes the minimizations in Eq 6 can be performed.  The\n  actual CSP algorithm has no guarantees of those minimums being\n  reached.\n\n- Since the theory provides no true guarantees about CSP, we can only\n  hope it provided some insights that led to its development.  I believe\n  the CSP idea is interesting and can be explained without the theory.\n\n- I would rather see the space that was used for theoretical analysis\n  be used to more completely explain the algorithm and the design\n  tradeoffs.  Also, that space could be used to pull in more of the\n  empirical results.  As it is, competitors that are designed just to\n  do imitation or self-play would not be expected to perform well at\n  this task and some of the wins don't look very big.  It would be good\n  to see them illustrated more thoroughly.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the significance of the offline policy adaptation problem and for recognizing that our algorithm CSP is \"well supported\", and our empirical study demonstrate \"non-trivial behavior\". Your positive feedback is greatly appreciated.\n\n**Q1. The statements that \"we prove that CSP learns a near-optimal risk-free offline adaptation policy\" are overstatements due to two concerns: (1) the KL divergence $\\delta$ can be hard to minimize in practice by statistical error due to low sample size in the dataset as well as failures in the optimization, (2) self-play is not guaranteed to converge.**\n\n**A1.** We appreciate the reviewer for bringing up these points. We will refine this statement to be more accurate in the next revision. The revised statement is \"if CSP converges, it learns a near-optimal risk-free offline adaptation policy given that the target agent's policy within the support of the dataset can be well approximated\". We will provide clarification to the reviewer's two concerns as follows.\n\n1. The inclusion of $\\delta$ in our analysis is not overly restrictive. To clarify, $\\delta$ is an upper bound on the KL divergence between the learned proxy model $\\mu$ and real target policy $\\pi_B$ (Line 205) **within the support of the dataset**: $\\max_{s\\in D}D_\\textrm{KL}(\\pi_B(\\cdot|s)\\|\\mu(\\cdot|s))\\leq \\delta$. It quantifies the extent to which an agent model can imitate the real target policy within the dataset, whose success is a necessary condition for any further adaptations. Additionally, as commented by the reviewer, our theoretical analysis through KL divergence $\\delta$ is a critical motivation to the development of CSP. Our theoretical findings in Theorem 4.8 reveals a clear connection between $\\delta$ and adaptation performance, demonstrating that a smaller approximation gap results in better performance.\n\n   Accurately fitting a deterministic policy within the dataset to minimize $\\delta$ is relatively straightforward given sufficient representational capacity and learning steps. However, we acknowledge that controlling $\\delta$ becomes more challenging for highly stochastic policies with limited data.\n\n2. We concur with the reviewer's observation that, although the converged solution of CSP possesses the theoretical properties presented in Theorem 4.8, the convergence of CSP itself is not guaranteed. The development of provable self-play algorithms for zero-sum games [1] remains an active area of research in RL theory. Nevertheless, empirical evidence from RL methods utilizing simple self-play [2, 3, 4] has shown that they can train max-min policies with strong performance in video games. Given that our paper primarily focuses on addressing the unique challenges posed by offline adaptation, we have incorporated a self-play approach in our algorithm for efficiency.\n\nDespite these concerns, the experimental results show CSP's superiority over the baselines. In particular, Figure 3 illustrates that CSP's testing performance, on average, surpasses its training performance (Lines 310-324). These results suggest that the CSP does motivate the adaptation agent to keep conservative with respect to the target's out-of-distributon policy in training, while also enabling it to discover risk-free exploitation opportunities.\n\n**Q2. What does \"improve in training and testing\" in line 60 mean? Why is the training performance of CSP below the baseline?**\n\n**A2.** We mean that both the training and testing performance of CSP improve as the number of training steps increases in Figure 1. Conversely, for BC-First, the improvement over steps in training does not result in better evaluation performance. It shows that the BC-First baseline quickly overfits the proxy model, suffering from distributional shift.\n\n\"The training performance of CSP is below the baseline\" is expected behavior, because the opponent model of CSP in training are not the same as BC-First. In BC-First, the opponent model is pre-trained on dataset only and fixed, which can have significant weaknesses on out-of-distribution states, and can be easy to exploit. In CSP, the opponent model is trained adversarially against the exploitation policy simultaneously. Because it is evolving, it can make up its weaknesses and becomes much harder to exploit. Consequently, the training performance os BC-First is higher. In testing, both exploitation policies are evaluated with the same real target model.\n\n**References**\n\n[1] Yu Bai and Chi Jin. Provable Self-Play Algorithms for Competitive Reinforcement Learning. In *Proceedings of the 37th International Conference on Machine Learning*, 2020.\n\n[2] Ye et al. Towards Playing Full MOBA Games with Deep Reinforcement Learning. In *Advances in Neural Information Processing Systems*, 2020.\n\n[3] Ye et al. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2020.\n\n[4] Zhao et al. AlphaHoldem: High-Performance Artificial Intelligence for Heads-Up No-Limit Poker via End-to-End Reinforcement Learning. In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2022."}, {"review_id": "BLBulgOoZh", "reviewer": "Reviewer_x4rE", "summary": "The paper introduces a new and previously unconsidered problem in multi-agent reinforcement learning: Offline policy adaptation to a target policy. The paper assumes that a dataset of the target policy is available, however the policy itself is not. This means that the training policy can benefit from analysing the dataset in order to exploit potential weaknesses of the opponent which it will encounter after training. In contrast to the classical offline RL problem, the environment is assumed to be available, just the opponent policy is not. The proposed algorithmic solution maximises policy return against a trained target policy that needs to adhere to the true target policy on in-dataset states (via behaviour cloning), but can perform arbitrary actions elsewhere and is otherwise optimised to minimise performance of the adaptation policy.", "questions": "How is the behaviour cloning coefficient chosen? Is it tuned? Is it sensitive? Do you observe issues in the sense that the trained target policy is not able to achieve both objectives at once (cloning on in-data & minimising adaptation return on out-data)?\n\nCould you provide a practical example where the algorithm provides an advantage? I wonder where in reality an automated agent has to compete against another one that is unknown but data is available", "limitations": "One of the most immediate limitations is that opponents are likely not static but will adapt at runtime when faced with an exploiting policy - this is discussed in the conclusion. Potential negative societal impacts are not discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed problem appears to not have been addressed before even though it seems like a very natural situation and immediately .. potential for improvement over prior approaches. The derived solution of exploiting the target policy only in-data and assuming the worst everywhere else appears straightforward, as it is robust towards different true target policies, yet at the same time realises the potential for exploitation on in-dataset states. The derivation and theoretical considerations are clearly formulated and provide a good foundation for the experimental part. I also found the maze example (section 5) extremely helpful for understanding the overall concept in a limited environment. The experimental evaluation considers a variety of different environments and baselines and shows convincing evidence that the proposed algorithm achieves new state of the art performance.", "weaknesses": "In the example environment in section 5, it is made very clear how the adaptation policy can exploit the suboptimal behaviour of the target policy (and which catastrophic mistakes you could make). While the same level of detail is maybe not necessary (or possible) for the other environments, I would have enjoyed a more qualitative comparison: In which way is the target policy here acting sub optimally and in which way is the adaptation policy taking advantage - I realise space is limited but maybe you could showcase 1 or 2 situations in the other environments.\n\nA few details on MAPPO would be good since your whole algorithm builds on it\n\nIt is unclear how the behaviour cloning coefficient is chosen and which value(s) are used\n\nI think there is a small mistake in Theorem 4.7, line 203 (one of the mu should be an alpha)\n\nI am not super sure about significance / practical applicability outside of video games - the examples are very illustrative, but I wonder whether an actual problem can be solved with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for recognizing the value and novelty of our work in addressing offline adaptation. We are also pleased that the reviewer finds our theoretical analyses clear and sound and our empirical results significant.\n\n**Q1. In which way is the target policy acting sub-optimally and in which way is the adaptation policy taking advantage in other environments other than the maze example?**\n\n**A1.** In Appendix G.5, we present a visualization in MuJoCo. In this scenario, the runner can take a detour either through the upper path or the lower path to bypass the blocker. The target policy (blocker, red) exhibits a bias toward blocking the upper path, which is captured by the dataset. The adaptation policy (runner, blue) learned by CSP discovers this weakness and learns to take the lower path to steadily win the game.\n\n**Q2. Provide a few details on MAPPO.**\n\n**A2.** We thank the reviewer for bringing up this request. We will add more details in the next revision. MAPPO [1] is an extension of single-agent PPO to multi-agent RL within the centralized training decentralized evaluation paradigm. It learns a centralized value function conditioning on global state to promote coordination among the agents. See Appendix G.2 for hyperparameters.\n\n**Q3. How is $C_\\textrm{BC}$ chosen? Is it tuned? Is it sensitive? Do you observe issues in the sense that the trained target policy is not able to achieve both objectives at once?**\n\n**A3.** For each environment, we tune $C_\\textrm{BC}$ on one scenario against one opponent. The same hyperparameter is then used for all subsequent experiments in that environment. Please refer to Appendix G for detailed hyperparameters.\n\n1. *Maze.* The table below shows the test rewards obtained by CSP.\n\n   | $C_\\textrm{BC}$ | 0.001 | 0.01 | 0.1  | 1    | 5    | 10   | 100  |\n   | --------------- | ----- | ---- | ---- | ---- | ---- | ---- | ---- |\n   | CASE 1          | 1     | 1    | 1    | 1    | 1    | 1    | 1    |\n   | CASE 2          | 1     | 1    | 1    | 16   | 16   | 16   | 16   |\n\n   In case 1, CSP always finds the optimal solution 1. In case 2, CSP finds the optimal when $C_\\textrm{BC}\\geq 1$, and shows the same behavior as Self-Play otherwise.\n\n2. *Predator-Prey in MPE.* We experiment with $C_\\textrm{BC}\\in$ {0.001, 0.003, 0.01, 0.1, 1, 5, 10} with dataset 10R where the adaptation agent controls adversaries.\n\n   | $C_\\textrm{BC}$  | Test Return  |\n   | ---------------- | ------------ |\n   | 0.001            | 33.8 ± 1.3 |\n   | 0.003 (selected) | 34.6 ± 1.6 |\n   | 0.01             | 33.6 ± 2.7 |\n   | 0.1              | 32.3 ± 0.8 |\n   | 1                | 31.3 ± 1.5 |\n   | 5                | 26.8 ± 2.5 |\n\n   Test return for Self-Play: 29.7 ± 2.8, BC-First: 21.3 ± 0.8.\n\n3. MuJoCo. We experiment with $C_\\textrm{BC}\\in$ {0.01, 0.1, 1, 5, 10} with one opponent.\n\n   | $C_\\textrm{BC}$ | Winning Rate  |\n   | --------------- | ------------- |\n   | 0.01            | 0.68 ± 0.13 |\n   | 0.1 (selected)  | 0.72 ± 0.06 |\n   | 1               | 0.68 ± 0.08 |\n   | 5               | 0.71 ± 0.02 |\n   | 10              | 0.52 ± 0.23 |\n\n   Wining rate for Self-Play: 0.54 ± 0.2, BC-First: 0.65 ± 0.13\n\n4. *Google Football.* We experiment with $C_\\textrm{BC}\\in$ {0.01,0.1,1,5,10} with one opponent in 3vs1 attacker scenario.\n\n   | $C_\\textrm{BC}$ | Winning Rate  |\n   | --------------- | ------------- |\n   | 0.01            | 0.83 ± 0.04 |\n   | 0.1             | 0.78 ± 0.13 |\n   | 1               | 0.80 ± 0.07 |\n   | 5 (selected)    | 0.88 ± 0.02 |\n   | 10              | 0.88 ± 0.12 |\n\n   Winning rate for Self-Play: 0.76 ± 0.08, BC-First: 0.74 ± 0.21.\n\nWe observe that $C_\\textrm{BC}$ needs to be tuned for different environments. In the predator-prey, MuJoCo and Google Football environments, even a small coefficient helps in learning to exploit, resulting in better performance of CSP compared to Self-Play. In the predator-prey and MuJoCo, when $C_\\textrm{BC}$ is too big, the policy fails to remain conservative and loses to Self-Play or BC-First. Overall, CSP is not very sensitive to $C_\\textrm{BC}$, as there exists a wide range of coefficients that allow it to successfully perform conservative adaptation in each environment.\n\n**Q4. Could you provide a practical example outside of video games where the algorithm provides an advantage? I wonder where in reality an automated agent has to compete against another one that is unknown but data is available.**\n\n**A4.** One potential application is in the domain of automated trading agents and online auction for advertisements. In these tasks, multiple automated traders or bidders compete for a variety of resources, aiming to maximize their utility within a limited budget. For example, in advertising, the bidders should select the audience and allocate the budget on each timestep. In such cases, the true policies of other competitors are not accessible. \n\nMoreover, the algorithm can potentially be applied in the field of security. One example is in the prevention for poaching of wild animals, where an automated agent has to decide where to send security guards based on observed trails of poachers. While the dataset of attacker's past behavior is available, the attacker's real policy remains unknown. Therefore, the agent should learn to withstand attacks witnessed in the dataset and also remain cautious when confronted with unfamiliar cases.\n\n**Q5. Are there any potential negative societal impacts?**\n\n**A5.** We have discussed the potential negative impacts in Appendix F. For example, inappropriate use of exploitation algorithms may result in potential exacerbation of inequality in society. However, our algorithm is general-purpose and depends on pre-collected data. We emphasize the importance of stringent laws and regulations that safeguard user privacy to mitigate any adverse effects.\n\n[1] Yu et al. The Surprising Effectiveness of MAPPO in Cooperative Multi-Agent Games, 2021."}]

Author rebuttal:

Meta-review: This paper introduces a new method for adapting to opponents using offline data.
 
During the active discussion phase the authors managed to address the overwhelming majority of open problems from the reviewers resulting in a few of the reviewers raising their scores. 

As a consequence, this paper now is a unanimous accept and should be published at the conference.
Decision: Accept (poster)