Title: Conservative Offline Policy Adaptation in Multi-Agent Games
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

opponent exploitation

multi-agent
Abstract: Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progress in offline reinforcement learn- ing, this paper studies offline policy adaptation, which aims to utilize the target agent’s behavior data to exploit its weakness or enable effective cooperation. We investigate its distinct challenges of distributional shift and risk-free deviation, and propose a novel learning objective, conservative offline adaptation, that optimizes the worst-case performance against any dataset consistent proxy models. We pro- pose an efficient algorithm called Constrained Self-Play (CSP) that incorporates dataset information into regularized policy learning. We prove that CSP learns a near-optimal risk-free offline adaptation policy upon convergence. Empirical results demonstrate that CSP outperforms non-conservative baselines in various environments, including Maze, predator-prey, MuJoCo, and Google Football.

Conservative Offline Policy Adaptation in Multi-Agent Games

Abstract

Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progress in offline reinforcement learning, this paper studies offline policy adaptation, which aims to utilize the target agent's behavior data to exploit its weakness or enable effective cooperation. We investigate its distinct challenges of distributional shift and risk-free deviation, and propose a novel learning objective, conservative offline adaptation, that optimizes the worst-case performance against any dataset consistent proxy models. We propose an efficient algorithm called Constrained Self-Play (CSP) that incorporates dataset information into regularized policy learning. We prove that CSP learns a near-optimal risk-free offline adaptation policy upon convergence. Empirical results demonstrate that CSP outperforms non-conservative baselines in various environments, including Maze, predator-prey, MuJoCo, and Google Football.

1 Introduction

Reinforcement learning (RL) has shown promise in learning collaborative or adversarial policies in multi-agent games, including cooperative multi-agent reinforcement learning [49, 6, 34, 5, 14, 40], and learning in adversarial environments such as Texas hold'em and MOBA games [3, 45, 46, 50]. Research also recognizes the importance of adapting policy to different players in multi-agent games [10, 26, 51]. For instance, in zero-sum games, although an approximate Nash equilibrium strategy optimizes worst-case performance, it can be overly conservative when competing with opponents of limited rationality and loses opportunities to exploit [27, 17, 24]. This problem is known as opponent exploitation [17, 23, 41, 1, 28]. In cooperative games, the agent may be required to cooperate with other competent players that have diverse behaviors [15, 11], instead of just a specific group of teammates. The problem is investigated in recent literature such as ad-hoc cooperation [11] and zero-shot coordination [15, 26, 51, 47]. In this paper, policy adaptation refers to the general ability to learn to collaborate with or exploit other participants (called target agents) in multi-agent games.

This paper focuses on a specific problem of offline policy adaptation . In this setting, interaction with the target agent is not available in training. Instead, we leverage the target agent's behavior dataset and maintain access to the game environment during training. The problem definition is illustrated

dataset

adaptation policy

environment

adaptation policy

environment

target policy

train

test

0.0

0.5

1.0

1.5

2.0

2.5

steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

eval winning rate

alg

BC first

CSP (Ours)

0.0

0.5

1.0

1.5

2.0

2.5

steps

1e7

0.0

0.2

0.4

0.6

0.8

1.0

eval winning rate

alg

BC first

CSP (Ours)

in Figure 1 (left). This setting is suitable for many practical scenarios where offline information about the target agent is provided. It lies between the fully transparent case, where the target agent's policy is available, and the fully invisible zero-shot adaptation case. For instance, in competitions such as Go, soccer, and MOBA games, playing with the target opponent is not an option until the actual game commences. However, professional human players have the intelligence to analyze their opponents' and teammates' strategic characteristics by studying past records, enhance their skills, and make adaptations accordingly before the competition. Moreover, training AIs with humans in applications that involve human-AI interaction is exorbitant. Additionally, it can be unsafe and poses ethical risks when humans interact with an inadequately trained AI in certain circumstances.

Although offline policy adaptation models a wide range of applications, it has not been adequately investigated both theoretically and empirically in previous literature. For instance, some opponent exploitation research assumes direct interaction in training for explicit opponent model estimation [17, 24] or policy gradient calculation [2, 10, 12]. However, this assumption is strong because it allows queries of the target agent's policy for any input. Other works study exploiting totally unknown opponents [7] or collaborating with completely unseen teammates [26, 51, 47] in a zero-shot manner. These methods do not take advantage of possible prior information about the target agent, and their performances heavily depend on the construction of a diverse and representative policy population constructed for training, which itself is a challenging problem [31, 25].

In this paper, we identify and formalize the distinct challenges in offline policy adaptation, and show that previous opponent exploitation methods, which disregard these challenges, are inadequate for effectively addressing the problem. Firstly, offline policy adaptation inherits the distributional shift [22] challenge from traditional offline RL. A straightforward approach to address offline policy adaptation is by initially creating an estimation of the target agent (e.g., utilizing behavior cloning) from data and subsequently learning an adaptation policy based on the proxy model. We name this method BC-First . However, since the dataset usually will not expose all aspects of the real policy of the target agent due to limited dataset size, the proxy model can differ arbitrarily from the real target on out-of-distribution (OOD) states. The discrepancy between the proxy in training and the real target in testing causes the distributional shift problem for the adaptation policy. As shown in Figure 1, BC-First overfits the dataset, which leads to erroneous generalization on OOD states and results in significant performance degradation in evaluation. Our algorithm outperforms the baseline and improves in both training and testing as the number of training step increases. Secondly, since the adaptation policy has access to the environment in training, unlike standard offline RL, we show that 'conservatism' [8, 20, 19] for addressing distributional shift may not be sufficient in this setting. It is possible to benefit from deviation in multi-agent games if the worst-case rewards obtainable outside the dataset exceed those inside the dataset. We coin this challenge as risk-free deviation .

To address these challenges, we propose a novel learning objective, conservative offline adaptation , which optimizes the worst-case performance when maintaining a conservative stance towards the target agent on the policy level. This objective provides a unified approach to addressing the challenges of distributional shift and risk-free deviation. To optimize this objective, we introduce a simple yet efficient algorithm, Constrained Self-Play (CSP) , which simultaneously trains an adaptation policy and a proxy model subject to regularizations. We prove that CSP produces a near-optimal risk-free adaptation policy upon convergence, given that the target agent's policy within the support of the dataset can be well approximated. Empirically, we evaluate the effectiveness of our algorithm in four environments: a didactic maze environment, predator-prey in MPE [39], a competitive two-agent

MuJoCo environment [2, 10] requiring continuous control, and Google Football [21]. Our results show that non-conservative methods confront difficulties regarding offline policy adaptation. The experiment results demonstrate that our algorithm significantly outperforms baselines.

2 Related Work

RL in multi-agent games. Multi-agent reinforcement learning (MARL) under the centralized training decentralized execution (CTDE) paradigm have shown promising performances in learning coordination behavior for cooperative multi-agent tasks such as SMAC [35] and Google Football [21]. Representative works consist of policy gradient algorithms [6, 48], and value decomposition methods [34, 40]. There also have been extensive research in competitive environments such as Texas hold'em and MOBA games. Some works design theoretically sounded RL algorithms which are guaranteed to converge to approximate NE [3]. Some works propose RL methods based on self-play [45, 46, 50] to train max-min policies which show strong empirical performances.

Offline RL. Offline reinforcement learning [22] learns purely from batched data. Representative works take a conservative view of out-of-distribution state-action pairs to mitigate the distributional shift problem. BCQ [8] uses VAE to propose candidate actions that are within the support of the dataset. IQL [19] uses expectile regression to learn Q function to avoid querying out-of-distribution actions. CQL [20] puts regularization on the learning of Q function to penalize out-of-distribution actions.

Opponent modeling. Opponent modeling, sometimes also called opponent exploitation, is a broad and extensively studied area [1, 28]. The offline adaptation problem studied in this paper is a largely unstudied yet significant and challenging problem within the broader scope of opponent modeling. Some previous works assume access to the target agent's policy in training, and employ deep reinforcement learning to exploit the opponent in continuous control tasks using policy gradients estimated from direct interaction [2, 10, 12]. Some works, including RNR [17] and SES [24] study safe exploitation given an estimated opponent model. They keep the exploitation policy close to NE in order to minimize its own exploitability. Some papers, like GSCU [7], study zero-shot generalization ability to exploit totally unknown opponents. Additionally, this paper is orthogonal to some other opponent modeling subareas, including designing efficient models for predicting opponent behavior [13, 33], and addressing non-stationary opponents [7]. To our best knowledge, there lacks a thorough investigation into the offline adaptation problem.

Please refer to Appendix B for more discussion.

3 Preliminary

For simplicity, we consider a two-player fully observable game G = ( N,S,A,P M , r, p 0 , γ ) [37, 38], where N is a set of two players, S is a set of states, p 0 is initial state distribution, and γ is the discount factor. We use A i to denote the available actions for player i , and A = ( A 1 , A 2 ) . The transition probability is denoted by P M : S × A × S → R . We use r i : S × A → R to denote the reward function for player i , and r = ( r 1 , r 2 ) . In fully cooperative scenarios, r 1 ( s, a ) = r 2 ( s, a ) = r ( s, a ) , which is termed multi-agent MDP (MMDP) [43]. In zero-sum games, r 1 ( s, a ) + r 2 ( s, a ) = 0 . Player i 's policy π i ( s, · ) is a distribution over action space A i conditioned on state s ∈ S . The value function V ( π 1 ,π 2 ) i ( s ) is the expected discounted accumulated reward that player i will achieve under joint policy profile ( π 1 , π 2 ) at state s . Similarly, the action value function Q ( π 1 ,π 2 ) i ( s, a ) calculates the expected return when players play the joint action a = ( a 1 , a 2 ) . Let d π 1 ,π 2 ( s ) be the discounted visitation frequency of state s under policy profile ( π 1 , π 2 ) : d π 1 ,π 2 ( s ) = ∑ ∞ t =0 γ t P ( s t = s ) . We use J ( π 1 ,π 2 ) i to denote the return for player i under policy profile ( π 1 , π 2 ) .

Without loss of generality, assume that we are aimed at learning an adaptation policy for player 1 (P1), while player 2 (P2), called the target agent, is playing a unknown fixed strategy π B . In offline policy adaptation, we do not have access to π B . Instead, a dataset D of π B 's behavior data is given. Formally, D = F p ( {T i } ) , where F p denotes extra data processing 1 , and T i = { s 0 , a 0 , s 1 , . . . } records states and corresponding actions of an episode collected by π B playing with rollout policies in

Π c . We do not make assumptions on the data collecting process, therefore, Π c can contain any polices ranging from a weak random policy to a strong NE policy. Analogous to offline RL and robust RL, the dataset quality varies given different rollout policies, dataset sizes, and processing methods, which results in different problem difficulties. For instance, if the dataset reveals adequate information about the target's behavior under diverse situations, BC-First can be enough. The challenges of offline policy adaptation becomes prominent if the dataset only contains limited knowledge of π B .

The purpose of offline adaptation is to learn to adapt to unknown π B given a fixed batch of data D . Notably, different from offline RL, the access to the environment itself is still granted in training. If not, such setting can be reduced to standard imitation learning or offline reinforcement learning [44, 29], because the target's behavior can be absorbed into environment dynamics. With the advantage of having access to the environment, this setting encourages conservative exploitation whenever risk-free deviation is possible.

4 Conservative Offline Adaptation

To address the problem of offline adaptation, we first examine the extrapolation error in policy evaluation between joint policies ( π, π B ) and ( π, µ ) in section 4.1. Here, π denotes the learning policy, π B represents the true target policy, and µ denotes any proxy model that estimates π B . This extrapolation error poses a risk to non-conservative policy adaptation algorithms, rendering them unsafe. However, traditional offline RL methods that aim to minimize the impact of extrapolation error tend to be excessively conservative, which is suboptimal in this context. To allow for beneficial deviations from the dataset, we introduce a novel objective of conservative offline adaptation in section 4.2. This approach maximizes the worst-case performance against any dataset-consistent proxy (defined below). Finally, we propose a practical Constrained Self-Play (CSP) algorithm in section 4.3 and formally prove that it achieves near-optimal conservative offline adaptation. Please note that all proofs pertaining to this section can be found in Appendix A.

4.1 Extrapolation Error in Offline Adaptation

We first analyze the extrapolation error in policy evaluation between the joint policies ( π, π B ) and ( π, µ ) , which results in the overfitting problem shown in Figure 1. For theoretical analysis, we first introduce the dataset consistent policy in Definition 4.1. We will show that this assumption can be relaxed in algorithm design in a later section. Inspired by offline RL, in Proposition 4.2, we find that the extrapolation error in offline policy adaptation can also be decomposed in a Bellman-like recursion [8, 44]. We focus solely on the value function of our learning agent (Player 1), and omit the subscript in the rewards and value functions for simplicity.

Definition 4.1. Dataset consistent policy. A policy µ is said to be consistent with π B on dataset D iff. µ ( s, a ) = π B ( s, a ) , ∀ ( s, a ) ∈ D . We denote the set of all dataset consistent policies as C D .

Proposition 4.2. The performance gap of evaluating policy profile ( π, µ ) and ( π, π B ) at state s is ε ( s ) = V π,π B ( s ) -V π,µ ( s ) , which can be decomposed in a Bellman-like recursion:

Theorem 4.3. We use ε = ∑ s 0 p 0 ( s 0 ) ε ( s 0 ) to denote the overall performance gap between policy profile ( π, µ ) and ( π, π B ) . In any 2-player fully observable game, for all reward functions, ε = 0 if and only if π B ( a | s ) = µ ( a | s ) , ∀ s, s.t. d π,π B ( s ) > 0 .

Theorem 4.3 proves that the extrapolation error vanishes if and only if the estimation µ perfectly matches π B on any state which can be visited by ( π, π B ) . If extrapolation error exists, improvement of π against proxy µ in training will not guarantee monotonic improvement in testing when π is evaluated with π B (as shown in Figure 1). Because we make no assumptions on π B outside D , we cannot bound ∥ µ ( a | s ) -π B ( a | s ) ∥ for s / ∈ D . Therefore, it requires that d π,π B ( s ) = 0 , ∀ s / ∈ D .

Corollary 4.4. Given a dataset consistent policy µ ∈ C D , which is consistent with π B on dataset D , the extrapolation error vanishes if ∀ s / ∈ D , d π,π B ( s ) = 0 .

Corollary 4.4 advocates excessive conservatism, similar to standard offline RL, to exclude any stateaction pairs outside the dataset. It requires to constrain both π and µ such that the trajectories of ( π, µ ) stay within D . However, this approach comes at the cost of relinquishing opportunities to exploit environmental access and explore the potential benefit of risk-free deviation from the dataset.

4.2 Conservative Offline Adaptation

In order to avoid being overly conservative and allow beneficial deviation from the dataset, we propose a novel learning objective called conservative offline adaptation in Definition 4.5.

Definition 4.5. Conservative offline adaptation (COA). Given an unknown policy π B , and a dataset D of its behavior data, conservative offline adaptation optimizes the worst-case performance against any possible dataset-consistent policy:

The adaptation policy π ∗ is an optimal risk-free adaptation policy if it is a solution to Objective (2).

To show that COA enables risk-free deviation, we decompose the return into two parts in Definition 4.6: (1) the return that is solely realized within the dataset, and (2) the return that arises from deviation.

Definition 4.6. Within-dataset return & off-dataset return . The expected return of a joint policy profile ( π 1 , π 2 ) can be decomposed into the sum of within-dataset return J D ( π 1 , π 2 ) and off-dataset return J F ( π 1 , π 2 ) :

where t τ D = arg max t ′ {∀ t ≤ t ′ , s t ∈ D } is a random variable indicates the last time step in a trajectory τ such that the state is still contained in dataset D .

For any policy π , and any dataset consistent policy µ ∈ C D , J D ( π, µ ) = J D ( π, π B ) because µ and π B behave exactly the same until the first time an OOD state is reached according to the definition. Thus, the objective 2 is equivalent to max π J D ( π, π B ) + min µ J F ( π, µ ) , s.t. µ ∈ C D . The adaptation policy π learns to exploit or cooperate with the target agent's policy exposed by dataset D . Meanwhile, the max-min optimization of max π min µ J F ( π, µ ) encourages π to explore opportunities outside the dataset while maintaining a conservative stance towards the target agent at the policy level. It provides a unified approach to automatically decide whether to deviate depending on the magnitude of within-dataset return and worst-case off-dataset return. In the extreme case, if min µ J F ( π, µ ) is much larger than J D ( π, π B ) , π can ignore the dataset completely and try to obtain rewards outside the dataset. Additionally, suppose that ( π ∗ , µ ∗ ) is the solution to COA, then the extrapolation error ε = J ( π ∗ , π B ) -J ( π ∗ , µ ∗ ) = J F ( π ∗ , π B ) -J F ( π ∗ , µ ∗ ) ≥ 0 , which suggests that the testing performance will not be worse than training performance.

4.3 Constrained Self-Play

Directly optimizing the objective of COA is hard because in deep reinforcement learning, satisfying the dataset consistency assumption regarding the proxy model µ can be difficult, particularly for stochastic policies with limited data. We relax this assumption and propose a simple yet efficient algorithm, Constrained Self-Play (CSP). We prove that CSP can achieve near-optimal risk-free policy adaptation. We start by showing in Theorem 4.7 that the gap in return is bounded by the KL divergence of two policies of the target agent.

Theorem 4.7. In a two-player game, suppose that π is player 1's policy, α and µ are player 2's policies. Assume that max s D KL ( α ( ·| s ) ∥ µ ( ·| s )) ≤ δ . Let R M be the maximum magnitude of return obtainable for player 1 in this game, and let γ be the discount factor. We use J ( π, µ ) and J ( π, α ) to denote the return for player 1 when playing policy profiles ( π, µ ) and ( π, µ ) respectively. Then,

With this result, we propose to optimize objective 5, which substitutes the hard dataset consistent constraint in objective 2 with KL-divergence.

We propose the Constrained Self-Play (CSP) algorithm, which optimizes objective 5 through learning both an adaptation policy and a proxy model with self-play and meanwhile minimizing the KL divergence between µ and π B on the dataset with regularization. The algorithm is shown in Algorithm 1 in Appendix C. CSP utilizes soft BC regularization to minimize the KLdivergence: max π min µ { J ( π, µ ) + C BC · E ( s,a ) ∼ D [ -log µ ( a | s )] } . It is because BC is equivalent to KL-divergence minimization [18, 9]. By increasing the coefficient C BC , CSP prioritizes KLdivergence minimization. In the max-min optimization, the proxy model is trained adversarially against our adaptation policy by setting the proxy's reward function to be the negative of our agent's reward. We use MAPPO [48] as the basic learning algorithm for players on both sides.

We prove in Theorem 4.8 that CSP approximates the optimal risk-free offline adaptation policy by some constant upon convergence. The gap depends on the maximum KL-divergence between the learned proxy µ and the target π B on s ∈ D . The gap decreases as µ becomes a more precise estimation of π B within D . As a sanity check, the gap vanishes when µ perfectly matches π B in D .

Theorem 4.8. Let π ∗ be the optimal risk-free offline adaptation policy at the convergence of the optimization of objective 2, and let ˜ π be the policy at the convergence of objective 5. Then the worst-case adaptation performance of ˜ π is near-optimal:

5 Didactic Maze Example

In this section, we use an adversarial grid-world maze game to demonstrate the challenges posed by offline policy adaptation and explain the superiority of our algorithm. Figure 2 displays the maze environment, where Player 1 (P1) and Player 2 (P2) start from the labeled positions and can move in four directions within the maze. The grids marked with numbers are terminal states with indicated rewards. There are five corridors (2 for P1 and 3 for P2), each with different lengths and exponential final rewards 2 . For simplicity, we use the notation P2 → 16 to denote P2's policy which going downwards to the state with terminal reward 16. The first agent reaching any terminal state wins the reward, and the opponent receives the negative reward as a penalty. The game ends immediately. We assume that P1 wins if P1 and P2 arrive at terminal states simultaneously. If no one reaches a terminals in 10 steps, they both get a penalty of -500. Both agents have full observation of the game states. Assume that P1 is the opponent, and we want to find exploiting policies for P2.

We consider four baselines: (1) BC-First as explained in section 1; (2) Self-Play (max-min) : approximating the pure max-min policy of P2 using self-play, which does not exploit; (3) DatasetOnly : completely following the dataset; and (4) Oracle : training an exploitation policy directly against the real target π B . Please refer to Appendix D for detailed learning curves in this environment.

4

1

P2

16

256

P1

64

4

P1

64

1

P2

16

256

CSP adaptation

suboptimal adaptation

Case 1: Inadequacy of Non-Conservative Methods We show in this case that non-conservative algorithms easily overfit to an erroneous proxy model, lured to exploit 'fake weaknesses' of the

opponent. Suppose that P1's real policy π B is as follows: (1) if P1 observes P2 → 16, P1 → 4; (2) if P1 observes P2 → 256, P1 → 64; (3) otherwise P1 does not move. This is a reasonably strong policy for P1 which tries to exploit P2. Suppose the dataset is collected with π B and P2's rollout policy is P2 → 256. Therefore, in the dataset, P1 is only observed going downwards. The proxy model trained by BC thus erroneously assumes that P1 → 64 under any conditions. The results are shown on the left side of Table 1. The BC-First method makes a disastrous generalization and responds with a P2 → 16 policy. Trying to exploit the 'fake weaknesses' of the opponent results in failure in testing against the real opponent. However, CSP discovers that P2 → 16 is unsafe and learns to safely win the game by playing P2 → 1. Both CSP and max-min policy achieve the optimal solution. The Dataset-Only method is overly conservative and only gets -64. The trajectories of CSP and BC-First when exploiting the real target are shown on the left side of figure 2.

Case 2: Performing Risk-Free Adaptation In this case, we show that CSP outperforms pure Self-Play (which does not exploit the opponent) by discovering risk-free exploitation opportunities. Suppose that P1 always plays an aggressive policy π B : P1 → 64. The dataset D is collected by π B with P2 playing P2 → 16 and P2 → 256 randomly. The results are shown on the right side of Table 1, and the trajectories of CSP and Self-Play are shown on the right side of Figure 2. The max-min policy produced by Self-Play cannot exploit P1's reckless policy. The generalization of the BC model is indeed correct in this specific situation, so BC-First is also optimal. Our algorithm recognizes that P2 → 16 is a risk-free exploitation because the dataset has witnessed that P1 still plays P1 → 64 even if P2 → 16 and successfully exploits the opponent.

It is noteworthy that we use the same hyperparameters for CSP in both cases and our algorithm learns different adaptation policies based solely on the different datasets used. It further validates that our algorithm is able to extract useful information from the dataset, and to perform risk-free adaptations.

6 Experiment

We conduct extensive experiments on the predator-prey in MPE [39], a competitive two-player MuJoCo environment [2, 10] that requires continuous control, and Google Football [21]. In all experiments, we compare CSP with (1) BC-First , and (2) Self-Play , which produces an approximate max-min conservative policy of the game. Self-Play can also be viewed as an ablation of CSP which does not utilize an offline dataset to adapt to targets. We also compare with multi-agent imitation learning (MAIL), and previous opponent exploitation method RNR [17], which does not perform conservative offline exploitation. More experiment details are available in Appendix G. We focus on performing opponent exploitation in competitive games. However, we also discuss the use of CSP for cooperative tasks in Appendix E.

6.1 Experiments in Predator-Prey and MuJoCo

In the predator-prey environment, there are three adversarial agents, one good agent, and two obstacles. The good agent is penalized for being hit by adversaries, whereas the adversaries are rewarded. In experiments presented in Table 2, our agent controls either the good agent or the adversaries, respectively. We use a pre-trained model as target π B . We report the episode rewards in evaluation with π B . We use three types of datasets (10R, 100R, 100S) to demonstrate the impact of dataset quality in offline adaptation. The prefix number indicates the number of trajectories in the dataset. Letter 'R' indicates that the dataset is collected by π B playing with a random policy, while 'S' indicates that the dataset is collected by π B playing with another well-trained opponent.

Intuitively, 10R has the lowest dataset quality, while 100S has the highest. We train the Oracle directly against π B as an upper bound. Self-Play and Oracle do not use datasets. MAIL imitates the dataset collection policy. It fails to learn meaningful behavior on datasets 10R and 100R where the dataset collection policy is a random policy. Table 2 shows that CSP outperforms baselines and achieves comparable performances with Oracle, regardless of the dataset quality. CSP effectively learns to exploit the opponent safely, while the non-conservative method BC-First fails to do so. Moreover, BC-First is significantly more sensitive to dataset quality. In the controlling adversaries scenario, even with the 100S dataset, BC-First still performs worse than Self-Play, which does not exploit at all. It shows that the non-conservative method overfits the proxy model severely.

The YouShallNotPassHumanoids environment [2, 10] is a MuJoCo-based two-player competitive game where the runner's objective is to pass the blocker and reach the red line on the opposite side. In our experiments, our agent acts as the runner, and the blocker is the target opponent for exploitation. In Table 3, we use four independently pre-trained blocker models as opponent targets. Since the exploitability of different opponent models can vary significantly, results against different opponents are not directly comparable. So we report results for all targets. CSP successfully exploits the opponent, and significantly outperforms the baselines for all targets. Additionally, we include a visualization of CSP's learned behavior in Appendix G.

6.2 Experiments in Google Football

Google Football [21] is a popular and challenging benchmark for MARL. In our experiments, the adaptation policy controls all players in one team, while the opponent target policy controls the other team. We conduct experiments in 4 scenarios: academy_3_vs_1_with_keeper (3vs1, where our agent acts as either the defender or the attacker), academy_run_pass_and_shoot_with_keeper (RPS, defender), and academy_counterattack_easy (defender). We report the winning rates of adaptation policies for five independently pre-trained opponent targets.

The experiment results in Table 4 show that our algorithm CSP outperforms baselines for almost all targets in all scenarios. Notably, the BC-First algorithm even

3vs1 (attacker)

RPS (defender)

CA (defender)

scenario

0.8

0.6

0.4

0.2

0.0

0.2

0.4

gap

method

CSP(Ours)

BC-First

performs worse than Self-Play, which does not utilize the opponent's information to exploit. It indicates that previous non-conservative algorithms are not guaranteed to use the opponent's information safely in offline adaptation.

In Figure 3, we show the gap of performances between testing and training: J ( π, π B ) -J ( π, µ ) . A negative gap indicates that although π learns to exploit µ during training, such exploitation is risky and can not be guaranteed in testing. The results demonstrate the significant challenges that offline policy adaptation poses for baseline exploitation algorithms. BC-First overfits the proxy model in training and experiences a severe performance drop in testing. Conversely, CSP has a positive gap on average, indicating its ability to find conservative exploitation opportunities. Because CSP optimizes for the worst-case performance, it tends to produce a proxy model that is possibly stronger than the real target. Thus, its evaluation performances are even higher than those in training.

6.3 Comparison with Non-Conservative Opponent Exploitation

We compare CSP with the previous safe exploitation method RNR [17], which minimizes the distance to NE when exploiting. RNR assumes that the opponent plays an NE policy with probability 1 -p and follows an estimated opponent model with p . It then learns an exploitation policy. However, RNR does not optimize the conservative offline adaptation objective, and it still fails to handle offline policy adaptation due to the neglect of possible errors in the estimated opponent model. In Figure 4, the results show that CSP outperforms RNR for every dataset and every value of p . RNR(0) is equivalent to Self-Play, and RNR(1) is equivalent to BC-First. Results for these two have already been reported in Table 2. The results further support our claim that previous methods struggle with offline policy adaptation and highlight the efficiency and significance of our algorithm.

10R

100R

100S

dataset

40

20

0

20

40

episode reward

scenario = Adversary

10R

100R

100S

dataset

scenario = Good Agent

method

RNR(0.2)

RNR(0.6)

RNR(0.8)

BC-First

CSP (Ours)

7 Conclusion

We provide a thorough analysis of the unique challenges in offline policy adaptation, which are often neglected by previous literature. We propose the novel learning objective of conservative offline adaptation that optimizes the worst-case performance against any dataset-consistent opponent proxy. We propose an efficient algorithm that learns near-optimal conservative adaptation policies. Extensive empirical results in Maze, predator-prey, MuJoCo, and Google Football demonstrate that our algorithm significantly outperforms baselines.

One limitation of our work is that optimizing worst-case performance in real-world applications may not always be necessary since participants are not fully rational in practice. One future direction to settle this problem is to extend offline adaptation to non-stationary opponents, which is orthogonal to our current study. Those methods typically focus on adapting policy according to inferred opponent information in evaluation, while our algorithm prevents performance degradation in testing against a consistent opponent. One way is to use our algorithm to train a population of risk-free exploitation policies against opponents of different types and adapt exploitation policy in testing according to the opponent's posterior distribution. We discuss potential negative social impacts in Appendix F.

References

Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence , 258:66-95, 2018.

Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent Complexity via Multi-Agent Competition. In International Conference on Learning Representations , 2018.

Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong. Combining Deep Reinforcement Learning and Search for Imperfect-Information Games. In Advances in Neural Information Processing Systems , volume 33, pages 17057-17069. Curran Associates, Inc., 2020.

Imre Csiszár and János Körner. Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition . Cambridge University Press, 2011.

Jakob Foerster, Francis Song, Edward Hughes, Neil Burch, Iain Dunning, Shimon Whiteson, Matthew Botvinick, and Michael Bowling. Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 1942-1951. PMLR, June 2019.

Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, and Wei Yang. Greedy when Sure and Conservative when Uncertain about the Opponents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 6829-6848. PMLR, July 2022.

Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2052-2062. PMLR, June 2019.

Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A Divergence Minimization Perspective on Imitation Learning Methods. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning , volume 100 of Proceedings of Machine Learning Research , pages 1259-1277. PMLR, November 2020.

Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial Policies: Attacking Deep Reinforcement Learning. In International Conference on Learning Representations , 2020.

Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An. Online Ad Hoc Teamwork under Partial Observability. In International Conference on Learning Representations , 2022.

Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial Policy Learning in Two-player Competitive Games. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 3910-3919. PMLR, July 2021.

He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé, III. Opponent modeling in deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 1804-1813, New York, New York, USA, 20-22 Jun 2016. PMLR.

Hengyuan Hu and Jakob N. Foerster. Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning. In International Conference on Learning Representations , 2020.

Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. 'Other-Play' for Zero-Shot Coordination. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4399-4410. PMLR, July 2020.

Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training. In Lud De Raedt, editor, Proceedings of the ThirtyFirst International Joint Conference on Artificial Intelligence, IJCAI-22 , pages 3099-3106. International Joint Conferences on Artificial Intelligence Organization, July 2022.

Michael Johanson, Michael Bowling, and Martin Zinkevich. Computing robust counter-strategies. 2007.

Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha S. Srinivasa. Imitation Learning as f-Divergence Minimization. CoRR , abs/1905.12888, 2019. arXiv: 1905.12888.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning. In International Conference on Learning Representations , 2022.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Neural Information Processing Systems , volume 33, pages 1179-1191. Curran Associates, Inc., 2020.

Karol Kurach, Anton Raichuk, Piotr Sta´ nczyk, Michał Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4501-4510, 2020.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. CoRR , abs/2005.01643, 2020. arXiv: 2005.01643.

Xun Li and Risto Miikkulainen. Dynamic adaptation and opponent exploitation in computer poker. In Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence , 2018.

Mingyang Liu, Chengjie Wu, Qihan Liu, Yansen Jing, Jun Yang, Pingzhong Tang, and Chongjie Zhang. Safe Opponent-Exploitation Subgame Refinement. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, ZHIPENG HU, and Yaodong Yang. Towards Unifying Behavioral and Response Diversity for Open-ended Learning in Zerosum Games. In Advances in Neural Information Processing Systems , volume 34, pages 941-952. Curran Associates, Inc., 2021.

Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory Diversity for Zero-Shot Coordination. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7204-7213. PMLR, July 2021.

Peter McCracken and Michael Bowling. Safe strategies for agent modelling in games. In Artificial Multiagent Learning, Papers from the 2004 AAAI Fall Symposium. Arlington, VA, USA, October 22-24, 2004 , volume FS-04-02, pages 103-110. AAAI Press, 2004.

Samer Nashed and Shlomo Zilberstein. A Survey of Opponent Modeling in Adversarial Domains. J. Artif. Int. Res. , 73, May 2022. Place: El Segundo, CA, USA Publisher: AI Access Foundation.

Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan Better Amid Conservatism: Offline MultiAgent Reinforcement Learning with Actor Rectification. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 17221-17237. PMLR, July 2022.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust Reinforcement Learning using Offline Data. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective Diversity in Population Based Reinforcement Learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 18050-18062. Curran Associates, Inc., 2020.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust Adversarial Reinforcement Learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2817-2826. PMLR, August 2017.

Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine theory of mind. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4218-4227. PMLR, 10-15 Jul 2018.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4295-4304. PMLR, July 2018.

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR , abs/1902.04043, 2019.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR.

Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations . Cambridge University Press, 2008.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

J. K Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sulivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L Williams, Yashas Lokesh, Ryan Sullivan, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. arXiv preprint arXiv:2009.14471 , 2020.

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. {QPLEX}: Duplex Dueling Multi-Agent Q-Learning. In International Conference on Learning Representations , 2021.

Zhe Wu, Kai Li, Hang Xu, Meng Zhang, Haobo Fu, Bo An, and Junliang Xing. L2E: Learning to Exploit Your Opponent, 2021.

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Yaodong Yang and Jun Wang. An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective. CoRR , abs/2011.00583, 2020. arXiv: 2011.00583.

Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 10299-10312. Curran Associates, Inc., 2021.

Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, Yinyuting Yin, Bei Shi, Liang Wang, Tengfei Shi, Qiang Fu, Wei Yang, Lanxiao Huang, and Wei Liu. Towards Playing Full MOBA Games with Deep Reinforcement Learning. In Advances in Neural Information Processing Systems , volume 33, pages 621-632. Curran Associates, Inc., 2020.

Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering Complex Control in MOBA Games with Deep Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):6672-6679, April 2020. Section: AAAI Technical Track: Machine Learning.

Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. In The Eleventh International Conference on Learning Representations , 2023.

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, and Yi Wu. The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games. CoRR , abs/2103.01955, 2021. arXiv: 2103.01955.

Chongjie Zhang and Victor R. Lesser. Multi-Agent Learning with Policy Prediction. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010 . AAAI Press, 2010.

Enmin Zhao, Renye Yan, Jinqiu Li, Kai Li, and Junliang Xing. AlphaHoldem: High-Performance Artificial Intelligence for Heads-Up No-Limit Poker via End-to-End Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence , 36(4):4689-4697, June 2022. Section: AAAI Technical Track on Domain(s) Of Application.

Rui Zhao, Jinming Song, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination. CoRR , abs/2112.11701, 2021. arXiv: 2112.11701.

A Proof

Proof of Proposition 4.2

Proposition 4.2 The performance gap of evaluating policy profile ( π, µ ) and ( π, π B ) at state s is ε ( s ) = V π,π B ( s ) -V π,µ ( s ) , which can be decomposed in a Bellman-like recursion:

Proof.

where and

Putting equation 9 and 10 back, we have

Proof of Theorem 4.3

Theorem 4.3 We use ε = ∑ s 0 p 0 ( s 0 ) ε ( s 0 ) to denote the overall performance gap between policy profile ( π, µ ) and ( π, π B ) . In any 2-player fully observable game, for all reward functions, ε = 0 if and only if π B ( a | s ) = µ ( a | s ) , ∀ s, s.t. d π,π B ( s ) > 0 .

Proof. Sufficiency. Whenever π B ( a | s ) = µ ( a | s ) holds, the first term in equation 1 is 0. If ∀ s s.t. d π,π B ( s ) > 0 , π B ( a | s ) = µ ( a | s ) , according to Proposition 4.2, by expanding the expression for ε ( s ) recursively, we have for ∀ s s.t. d π,π B ( s ) > 0 , ε ( s ) = 0 . Therefore, ε = ∑ s 0 p ( s 0 ) | ε ( s 0 ) | = 0 .

̸

Necessity. We first show by contradiction that ε = ∑ s 0 p ( s 0 ) ε ( s 0 ) requires ∀ s s.t. p 0 ( s ) > 0 , π B ( a | s ) = µ ( a | s ) . If π B ( a | s ) = µ ( a | s ) , according to Proposition 4.2, we can change the reward function r ( s, a , s ′ ) to change the value of ε ( s ) , because the reward function is arbitrary. So ε = ∑ s 0 p ( s 0 ) ε ( s 0 ) does not hold anymore, and this is a contradiction. Therefore, ε ( s ) = ∑ s 0 ,s 1 p ( s 0 ) π ( a 1 | s 0 ) π B ( a 2 | s 0 ) P M ( s 1 | s, a ) ε ( s 1 ) = E s 1 ∼ ( π,π B ) [ ε ( s 1 )] . Using the same arguments, it can be shown that ∀ s, s.t. Pr ( s 1 = s ) > 0 , π B ( a | s ) = µ ( a | s ) . By expanding recursively, the statement is proved.

Proof of Theorem 4.7

We first prove a Lemma.

Lemma A.1. In a two-player game, suppose that π is player 1's policy, α and µ are player 2's policies. We use π joint ( π,α ) and π joint ( π,µ ) to denote the joint policy profiles ( π, α ) and ( π, µ ) . Then we have D KL ( π joint ( π,α ) ( ·| s ) ∥ π joint ( π,µ ) ( ·| s ) ) = D KL ( α ( ·| s ) ∥ µ ( ·| s )) , where D KL denotes KL divergence.

Proof. According to definition,

Additionally, we use the Theorem 1 in [36] and we provide a restatement. The original paper [36] minimizes accumulated discounted cost. In contrast, an agent maximizes its return in our paper.

Theorem A.2. (Theorem 1 in [36]) Let ϵ = max s | E a ∼ π 2 ( a | s ) [ A π 1 ( s, a )] | , then

Finally, we prove Theorem 4.7.

Theorem 4.7 In a two-player game, suppose that π is player 1's policy, α and µ are player 2's policies. Assume that max s D KL ( α ( ·| s ) ∥ µ ( ·| s )) ≤ δ . Let R M be the maximum magnitude of return obtainable for player 1 in this game, and let γ be the discount factor. We use J ( π, µ ) and J ( π, α ) to denote the return for player 1 when playing policy profiles ( π, µ ) and ( π, µ ) respectively. Then,

Proof. According to Theorem A.2, we have

where ϵ = max s | E a 1 ∼ π,a 2 ∼ µ [ A ( π,α ) ( s, a )] | ≥ 0 . Use Lemma A.1, we have

(16)

Next, we use KL divergence to bound ϵ . According to definition, ∣ ∣ Q ( π,α ) ( s, a ) ∣ ∣ ≤ R M . So we have,

With Pinsker's inequality [4], the total variation distance is bounded by KL divergence:

Therefore,

Put inequality 19 back into 16, and we get

Proof of Theorem 4.8

Theorem 4.8 Let π ∗ be the optimal risk-free offline adaptation policy at the convergence of the optimization of objective 2, and let ˜ π be the policy at the convergence of objective 5. Then the worst-case adaptation performance of ˜ π is near-optimal:

Proof. According to definition, π ∗ is the solution to the optimization objective 2, so min µ ∈C D J ( π ∗ , µ ) ≥ min µ ∈C D J (˜ π, µ ) .

Suppose that ( π ∗ , µ ∗ ) and (˜ π, ˜ α ) are the solutions to objectives 2 and 5 respectively. For ∀ µ ∈ C D , let F ( µ ) = { α |∀ s / ∈ D,α ( ·| s ) = µ ( ·| s ); max s ∈ D D KL ( π B ( ·| s ) ∥ α ( ·| s )) ≤ δ } be the set of corresponding α policies which are identical to µ on OOD states. Observe that max s D KL ( π B ( ·| s ) ∥ α ( ·| s )) ≤ δ also holds. Therefore, according to Theorem 4.7, ∀ π, ∀ µ ∈

C D , ∀ α ∈ F ( µ ) , J ( π, α ) ≥ J ( π, µ ) -R M √ 2 δ ( 1 + 2 γδ (1 -γ ) 2 ) . Taking the minimization over µ and α , we get ∀ π, min µ min α ∈F ( µ ) J ( π, α ) ≥ min µ J ( π, µ ) -R M √ 2 δ ( 1 + 2 γδ (1 -γ ) 2 ) . Observe that the left part is equivalent to min α :max s ∈ D D KL ( α ( ·| s ) ∥ π B ( ·| s )) ≤ δ J ( π, α ) . Taking the maximization over π for both sides, we get J (˜ π, ˜ α ) ≥ J ( π ∗ , µ ∗ ) -R M √ 2 δ ( 1 + 2 γδ (1 -γ ) 2 ) .

Let J (˜ π, µ ′ ) = min µ ∈C D J (˜ π, µ ) . Observe that µ ′ also satisfies the KL divergence constraint: max s ∈ D D KL ( π B ( ·| s ) ∥ µ ′ ( ·| s )) ≤ δ , and (˜ π, ˜ α ) is the optimal solution to objective 5, so J (˜ π, µ ′ ) ≥ J (˜ π, ˜ α ) . So we get min µ ∈C D J (˜ π, µ ) ≥ J (˜ π, ˜ α ) ≥ J ( π ∗ , µ ∗ ) -R M √ 2 δ ( 1 + 2 γδ (1 -γ ) 2 ) .

B Related Work

RL in multi-agent games. Multi-agent reinforcement learning (MARL) under the centralized training decentralized execution (CTDE) paradigm have shown promising performances in learning coordination behavior for cooperative multi-agent tasks such as SMAC [35] and Google Football [21]. Representative works consist of policy gradient algorithms [6, 48], and value decomposition methods [34, 40]. There also have been extensive research in competitive environments such as Texas hold'em and MOBA games. Some works design theoretically sounded RL algorithms which are guaranteed to converge to approximate NE [3]. Some works propose RL methods based on self-play [45, 46, 50] to train max-min policies which show strong empirical performances.

Offline RL. Offline reinforcement learning [22] learns purely from batched data. Representative works take a conservative view of out-of-distribution state-action pairs to mitigate the distributional shift problem. BCQ [8] uses VAE to propose candidate actions that are within the support of dataset. IQL [19] uses expectile regression to learn Q function to avoid querying out-of-distribution actions. CQL [20] puts regularization on the learning of Q function to penalize out-of-distribution actions.

Opponent exploitation. There have been extensive study on various aspects of opponent exploitation [1, 28]. Some employ deep reinforcement learning to exploit the opponent in continuous control tasks using policy gradients estimated from direct interaction [2, 10, 12]. Some works, including RNR [17] and SES [24] study safe exploitation given an estimated opponent model. They keep the exploitation policy close to NE in order to minimize the loss if the opponent changes its policy adversarially. Some work, for example, GSCU [7], studies zero-shot generalization ability to exploit totally unknown opponents. To our best knowledge, there lacks a thorough investigation into the offline adaptation problem.

Robust RL. Formulated as robust-MDP (RMDP), robust RL [32, 30] is dedicated to learn a robust policy against perturbation in environment dynamics, e.g., observation noise, action delay. Formally, the unknown transition model in testing environment lies in an uncertainty set P which contains all models within a certain distance from the training model. Robust RL solves a max-min problem, optimizing the worst-case performance against any transition model in P . Some works uses adversarial training [32, 16]. RORL [42] uses smoothing on the Q function to learn robust policy. The similarity to our work is that, our offline adaptation problem also optimizes the worst-case performance against the target's behavior on out-of-distribution states. However, we do not have any distance constraints on the target's policy on out-of-distribution states. To our best knowledge, our paper is the first to investigate into the offline adaptation problem.

C Algorigthm

The CSP algorithm is illustrated in Algorithm 1. The proxy model is trained adversarially against our agent, therefore, we set the proxy's reward function to be the negative of our agent's reward. In our experiments on an n -player environment, we assume that we control n 1 players, while the target controls n 2 players, and n 1 + n 2 = n . Therefore, we use MAPPO [48] as the basic learning algorithm for players of both sides, since players from the same side are fully cooperative. MAPPO deals with the cooperative multi-agent reinforcement learning problem through learning a centralized value function conditioned on global state, and a decentralized control policy conditioned on local observations. We use self-play with alternative update to learn both adaptation policy π , and target's

proxy model µ simultaneously. We use a soft behavior cloning regularization term to minimize the KL-divergence between proxy model µ and target policy π B .

Algorithm 1 Constrained Self-Play (CSP)

Input: dataset D , environment env , learning rate α , regularization coefficient C BC Output: adaptation policy π , target's policy proxy µ

1: Initialize adaptation policy π , critic v π ; and target's policy proxy µ , critic v µ 2: while not converged do 3: Collect trajectories {T i } ← rollout ( π, µ ; env ) 4: if our turn then 5: For each trajectory T i , calculate return target R γ t and advantage ˆ A t of our side using GAE with value function v π for each step t 6: v π ← v π -α ∇ v π 1 |∪T i | ∑ ( s t ,R γ t ) ∼∪T i ( R γ t -v π ( s t )) 7: π ← π + α ∇ π 1 |∪T i | ∑ ( o t ,a t ,π old , ˆ A t ) ∼∪T i min ( π ( a t | o t ) π old ( a t | o t ) ˆ A t , 8: clip ( π ( a t | o t ) π old ( a t | o t ) , 1 -ϵ, 1 + ϵ ) ˆ A t ) 9: else 10: For each trajectory T i , calculate return target R γ t and advantage ˆ A t of target agent side using GAE with value function v µ for each step t 11: B ← sample a random batch from D 12: v µ ← v µ -α ∇ v µ 1 |∪T i | ∑ ( s t ,R γ t ) ∼∪T i ( R γ t -v µ ( s t )) 13: µ ← µ + α ∇ µ { 1 |∪T i | ∑ ( o t ,a t ,π old , ˆ A t ) ∼∪T i min ( µ ( a t | o t ) µ old ( a t | o t ) ˆ A t , 14: clip ( µ ( a t | o t ) µ old ( a t | o t ) , 1 -ϵ, 1 + ϵ ) ˆ A t ) -C BC · 1 |B| ∑ ( o,a ) ∈B -log µ ( a | o ) } 15: end if 16: end while

D Experiment on Didactic Maze

We show experiment details of the Maze example in this section. The learning curve in case 1 is shown in Figure 5. In the BC-First algorithm, since the dataset only contains trajectories that P1 → 64, the BC model makes a wrong and risky generalization that assumes P1 always goes downwards to 64 no matter how P2 acts. Therefore, the P2 policy learns to exploit this 'weakness' by playing P2 → 16, and achieves 16 in training phase. However, during testing, P2 obtains -4 reward through trying to exploit an nonexistent weakness of P1.

Our algorithm avoids the trap of imaginary weakness, and learns to safely win the game, while the BC-First method makes risky exploitations which will not work in evaluation finally. Our algorithm keeps conservative for states outside dataset, and admits the possibility that P1 could go leftwards to win the game if P2 does not plays P2 → 1.

0

50000

100000

150000

step

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

avg step reward

algorithm

selfplay

BC first

csp

0

50000

100000

150000

step

10.0

7.5

5.0

2.5

0.0

2.5

5.0

avg step reward

algorithm

selfplay

BC first

csp

The learning curve in case 2 is shown in Figure 6. As can be seen in Figure 6, our algorithm quickly learns to exploit while the max-min strategy produced by Self-Play fails to.

0

50000

100000

150000

step

10

5

0

5

10

avg step reward

algorithm

selfplay

BC first

csp

0

50000

100000

150000

step

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

avg step reward

algorithm

selfplay

BC first

csp

We use the same set of hyper-parameters (including the same coefficient C BC for the BC regularization term) for these two cases. The hyper-parameters are shown in Table 5. The dataset D 1 = { trajectories ( P 1 → 64 , P 2 → 256) } in case 1, and our algorithm learns to play P2 → 1; while in case 2, given dataset D 2 = { trajectories ( P 1 → 64 , P 2 → 256) , trajectories ( P 1 → 64 , P 2 → 16) } , our algorithm learns to play P2 → 16. We use the same set of hyper-parameters for both cases. Therefore, our algorithm produces different policies simply because of different datasets given. It further validates that our algorithm is able to extract useful information from dataset, and only perform risk-free adaptations.

E Conservative Offline Adaptation for Cooperative Tasks

We evaluate our method in competitive games to perform opponent exploitation. Nevertheless, our method is also applicable to cooperative environments. The objective of CSP in cooperative games maintains the same min-max structure because it promotes adherence to the policy exposed by dataset for states within the dataset, while aiming to optimize for worst-case performance for states not in the dataset. Therefore, in cooperative games, the policy regularization term incentivizes the adaptation agent to collaborate with the target agent on states within the dataset, and the minimization over opponent proxy µ encourages the adaptation agent to be conservative and not to rely on the target agent's OOD policy. Note that in objective 5 and Algorithm 1, no matter whether the game is competitive or cooperative, the minimization over µ in training can be achieved by setting the target agent's reward function to be the negative of our adaptation agent's reward function. We then train µ to maximize this reward with policy regularization in self-play.

Although our method is applicable to cooperative environments as well, the challenges caused by offline policy adaptation are more significant in competitive games. Recall that conservative offline adaptation optimizes

For cooperative tasks, it requires that the teammate µ will not cooperate on states outside dataset in cooperative games. Therefore, for the adaptation policy, staying within dataset could be a trivial and

near-optimal solution for most cooperative problems if we would like to maximize the worst-case performance against any dataset-consistent teammate.

F Potential Negative Social Impacts

Inappropriate use of exploitation algorithms may result in negative social impacts. An example of its negative impact is the exacerbation of inequality in society. Companies who have access to large amounts of data can utilize their customers more effectively. However, our algorithm is general-purpose and depends on pre-collected data. We advocate for strict laws and regulations that protect user privacy data to avoid negative impacts. It is recommended that products utilizing exploitation algorithms are made public and supervised.

G Experiment Details

G.1 Environments

The experiment environments are illustrated in figure 7.

PUAYER SPRNFDRSDBLE

0.01

YRUNNUSCOR

BALONNIEDS

Predator-Prey There are one good agent, three adversarial agents and two obstacles in the predatorprey environment. The good agent is faster and receive a negative reward for being hit by adversaries while adversaries are slower and are rewarded for hitting the good agent. All agents are initialized randomly in the environment.

YouShallNotPassHumans The YouShallNotPassHumanoids environment [2, 10] creates a twoplayer competitive game based on MuJoCo, where one humanoid (the runner) is aimed at passing the other humanoid (the blocker) to reach the red line on the opposite side. The environment itself is challenging because it has high dimensional observation space and requires continuous control. In our experiments, we assume that our agent acts as the runner, while the blocker is the target opponent to exploit. We use four independently pre-trained blocker models as opponent targets. For fairness, we generate these targets with exactly four random seeds without any selection. Since the exploitability of different opponent models can vary significantly, results with respect to different opponents are not directly comparable. So we report results with respect to all targets.

Google Football Google Football [21] is a popular and challenging benchmark for MARL. In our experiments, our adaptation policy controls all players in one team, while the opponent target policy controls the other team. Since all players in the same team are fully cooperative, we use MAPPO [48] to learn decentralized policy. We conduct experiments in 4 scenarios: academy_3_vs_1_with_keeper (3vs1, where our agent acts as either defender or attacker), academy_run_pass_and_shoot_with_keeper (RPS, defender), and academy_counterattack_easy (defender). We report the winning rates of adaptation policies for 5 independently pre-trained opponent targets. For attacker, winning rate refers to the percentage of episodes that the attacker scores, while for defender, it refers to the percentage of episodes that the defender prevents the attacker from scoring.

G.2 Hyper-parameters

In our experiments, we use MAPPO [48] as the base RL algorithm to learn policies on both sides (ours and the target's). MAPPO is an extension of single agent PPO to multi-agent reinforcement learning

within the centralized training decentralized evaluation (CTDE) paradigm. It learns a centralized value function conditioning on global state to promote coordination among the agents. We selected MAPPO as the base learning algorithm due to its simplicity and empirical strong performance.

For the BC-First method, we use exactly the same MAPPO (with the same set of hyper-parameters and the same number of training samples) as CSP to train our adaptation policy, while keeping the target's proxy fixed. In order to ensure that the the performances of BC-First method are not encumbered by an under-trained proxy model, we use the same network structure for the proxy model as the real target model, and train enough steps to make sure that behavior cloning has converged. For the Google Football environment, we use the same hyper-parameters as reported in the MAPPO paper [48] for all scenarios. The hyper-parameters are listed in Table 6, 7, and 8.

G.3 Computing Resources

Each seed is run on a GPU server with one NVIDIA P100 GPU, and Intel(R) Xeon(R) Gold 6145 CPU @ 2.00GHz CPU. Each run can finish within 24 hours.

G.4 Target Models & Dataset Collection

For all the environments, we user different runs of self-play to get the group of targets. In order to ensure a fair comparison, we use the exactly the same number of random seeds to generate these targets, without any selection . In our experiments, we observe that different runs with different seeds can produce diverse targets. For instance, in Google Football, different targets may have different tendencies to pick which side to start a attack (left or right). Moreover, as can be seen from Table 4, the difficulties to exploit these targets are diverse. For MuJoCo and Google Football, we use dataset consisting of 5 trajectories. We collect these trajectories using the target model together with a rollout policy. We observe that the 3 kinds of rollout policies: (1) random policy, (2) environment's bot, (3) target itself, do not have significant impacts on the experiment results in the MuJoCo and Google Football environments.

G.5 Visualization

Runner (blue) in dataset makes a detour through the upper path

Runner (blue) learned in CSP prefers to pick the lower path

Blocker (red) learned in CSP learns to block the upper path,

and also tries to block the runner in the lower path

Runner (blue) learned in CSP picks the lower path and

steadily wins in evaluation

Blocker (red) in dataset tries to block the upper path

Blocker (red) in dataset tries to block the upper path

T

A

B

C

In Figure 8, we further consolidate our claim by visualizing the policy behavior in MuJoCo. As can be seen in this example, in a trajectory contained in dataset, the runner (blue agent) typically makes a detour through the upper path to avoid being pushed down by the blocker (red agent). The opponent blocker also has the tendency to walk upwards to stop the runner. The runner trained by CSP learns to exploit the policy represented by the dataset, and prefers making a detour through the lower path, as can be seen from the middle and bottom lines of replays. Although the behavior of runner going downwards is not contained in the dataset, the blocker trained by CSP can still learns to stop such runner, which makes the runner even more robust and stronger than the real target model. Therefore, when confronted with the real opponent blocker, who only knows how to defend the upper path, the runner trained by CSP steadily wins the game.

G.6 Reward.

For predator-prey, we use the environment's original reward. For MuJoCo, we use dense rewards for locomotion learning as in previous works [2, 10]. In Google Football, we use both the sparse scoring reward as well as the dense checkpoint reward which awards the attacker according to the ball handler's distance to the goal. Although the rewards for the attacker are provided by the environment, Google Football does not provide rewards for the defender team in academy scenarios. In our experiments, we just use the negative of the attacker's reward to train defender's policy to make the game zero-sum.

G.7 Explanation of Training & Testing Performances

The observed lower training performance of CSP than the baseline, as shown in Figure 1, is expected behavior since the opponent model used by CSP during training differs from that of BC-First. In BCFirst, the opponent model is solely pre-trained on a dataset and remains fixed, leading to significant vulnerabilities in out-of-distribution states which can be easily exploited. Conversely, in CSP, the opponent model is trained in an adversarial manner simultaneously with the exploitation policy. Due to its evolutionary nature, it can compensate for its vulnerabilities and becomes significantly more

challenging to exploit. Consequently, the training performance of BC-First is higher. During testing, both exploitation policies undergo evaluation using the same real target model.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new and previously unconsidered problem in multi-agent reinforcement learning: Offline policy adaptation to a target policy. The paper assumes that a dataset of the target policy is available, however the policy itself is not. This means that the training policy can benefit from analysing the dataset in order to exploit potential weaknesses of the opponent which it will encounter after training. In contrast to the classical offline RL problem, the environment is assumed to be available, just the opponent policy is not. The proposed algorithmic solution maximises policy return against a trained target policy that needs to adhere to the true target policy on in-dataset states (via behaviour cloning), but can perform arbitrary actions elsewhere and is otherwise optimised to minimise performance of the adaptation policy.\n\nSTRENGTHS:\nThe proposed problem appears to not have been addressed before even though it seems like a very natural situation and immediately .. potential for improvement over prior approaches. The derived solution of exploiting the target policy only in-data and assuming the worst everywhere else appears straightforward, as it is robust towards different true target policies, yet at the same time realises the potential for exploitation on in-dataset states. The derivation and theoretical considerations are clearly formulated and provide a good foundation for the experimental part. I also found the maze example (section 5) extremely helpful for understanding the overall concept in a limited environment. The experimental evaluation considers a variety of different environments and baselines and shows convincing evidence that the proposed algorithm achieves new state of the art performance.\n\nWEAKNESSES:\nIn the example environment in section 5, it is made very clear how the adaptation policy can exploit the suboptimal behaviour of the target policy (and which catastrophic mistakes you could make). While the same level of detail is maybe not necessary (or possible) for the other environments, I would have enjoyed a more qualitative comparison: In which way is the target policy here acting sub optimally and in which way is the adaptation policy taking advantage - I realise space is limited but maybe you could showcase 1 or 2 situations in the other environments.\n\nA few details on MAPPO would be good since your whole algorithm builds on it\n\nIt is unclear how the behaviour cloning coefficient is chosen and which value(s) are used\n\nI think there is a small mistake in Theorem 4.7, line 203 (one of the mu should be an alpha)\n\nI am not super sure about significance / practical applicability outside of video games - the examples are very illustrative, but I wonder whether an actual problem can be solved with it.\n\nQUESTIONS:\nHow is the behaviour cloning coefficient chosen? Is it tuned? Is it sensitive? Do you observe issues in the sense that the trained target policy is not able to achieve both objectives at once (cloning on in-data & minimising adaptation return on out-data)?\n\nCould you provide a practical example where the algorithm provides an advantage? I wonder where in reality an automated agent has to compete against another one that is unknown but data is available\n\nLIMITATIONS:\nOne of the most immediate limitations is that opponents are likely not static but will adapt at runtime when faced with an exploiting policy - this is discussed in the conclusion. Potential negative societal impacts are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers multi-agent games where you have access to the\nenvironment and you have a previously collected dataset from the\nopponent you will be facing.  So it is offline in the sense that you\nonly have data on the opponent, but online in the sense that you can\npractice as much as you want in the environment -- you just need to\nsupply your own opponent policy, possibly informed by the opponent\ndataset you are given.  The paper proposes an RL algorithm (called\nConstrained Self Play) whose loss function incentivizes adherence to\nthe dataset for states in the dataset and worst-case optimization for\nstates not in the dataset.  It provides related theoretical analysis.\nCSP is tested empirically on several multi-agent adversarial games.\n\nSTRENGTHS:\nThe idea for CSP is interesting and well supported by the arguments in\nthe narrative.  The problem setup is an interesting one to consider as\nmany adversarial applications will match it reasonably well.  The\ngames tested in the empirical study are sufficiently complex to show\nnon-trivial behavior.\n\nWEAKNESSES:\nMy score is limited by the fact that I did not find the theoretical\nanalysis useful.  My concerns with it are as follows:\n\n- The statements that \"we prove that CSP learns a near-optimal\n  risk-free offline adaptation policy\" are overstatements given the\n  differences between the algorithm and the analysis.  There are a\n  number of factors that mean there *is* risk in applying this method.\n\n- Objective 2 can not be optimized in general.  Objective 5 is designed\n  to address some of the issues by replacing the constraint with a KL\n  divergence.  However, there is no way to guarantee or measure delta.\n  It will be affected by statistical error due to low sample size in\n  the dataset as well as failures in the optimization thus leading to\n  an unknown and arbitrarily large delta.\n\n- Theorem 4.8 assumes the minimizations in Eq 6 can be performed.  The\n  actual CSP algorithm has no guarantees of those minimums being\n  reached.\n\n- Since the theory provides no true guarantees about CSP, we can only\n  hope it provided some insights that led to its development.  I believe\n  the CSP idea is interesting and can be explained without the theory.\n\n- I would rather see the space that was used for theoretical analysis\n  be used to more completely explain the algorithm and the design\n  tradeoffs.  Also, that space could be used to pull in more of the\n  empirical results.  As it is, competitors that are designed just to\n  do imitation or self-play would not be expected to perform well at\n  this task and some of the wins don't look very big.  It would be good\n  to see them illustrated more thoroughly.\n\nQUESTIONS:\nSome smaller comments:\n\nline 60: \"improve in training and testing\".  I'm not sure what the\ntraining part of this refers to.  In figure 1, the training\nperformance of CSP is below the baseline.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
In this paper, the authors target the offline policy adaptation problem in multi-agent games where "offline" indicates no interactive target agents and an accessible environment. The problem is challenging as it inherits the distributional shift challenge from offline RL, and is likely to lead to risky actions. To tackle these problems, the authors introduced the novel conservative offline adaptation objective to optimize the worst-case performance and proposed the constrained self-play, i.e., CSP algorithm to simultaneously train an adaptation policy and a proxy model subject to regularization.

STRENGTHS:
1. This paper is well-written and easy to follow with good visualizations. 
2. There are plenty of theoretical results supporting the proposed solution. 
3. Extensive experiments are done to validate the proposed solution.

WEAKNESSES:
1. Novelty. The proposed offline policy adaptation problem seems to be a special case of opponent modeling problem [1,2] where the static opponent's data are available. Therefore, the studied problem is not new. 

2. Theoretical. I did not see the connection of Theorem 4.7 with the CPS algorithm. 

[1] Nashed, Samer, and Shlomo Zilberstein. "A survey of opponent modeling in adversarial domains." Journal of Artificial Intelligence Research.

[2] Haobo Fu, et al. "Greedy when Sure and Conservative when Uncertain about the Opponents." ICML, 2022.

QUESTIONS:
1. If in the cooperative games, the objective for CSP remains the same min-max problem?
2. What is the connection between Theorem 4.7 with the CPS algorithm?

LIMITATIONS:
The authors leave the limitation where not all agents are rational in future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to offline policy adaptation in multi-agent games, focusing on leveraging the target agent's behavior dataset and access to the game environment during training. The authors identify the challenges of distributional shift and risk-free deviation in offline policy adaptation and propose a new learning objective called conservative offline adaptation to address these issues. They introduce an efficient algorithm, Constrained Self-Play (CSP), which trains an adaptation policy and a proxy model with regularizations. The paper provides theoretical proof of CSP's effectiveness and demonstrates its superiority over non-conservative methods through empirical evaluations in various environments, including Maze, predator-prey, MuJoCo, and Google Football.\n\nSTRENGTHS:\n1. The specific problem of offline policy adaptation is a highly meaningful and novel issue, particularly with a wide range of potential applications in the industrial sector.\n\n2. The article is easy to understand, with a comprehensive background and related work section. \n\n3. Although the ideas presented are simple and heuristic, they are supported by theoretical proofs and significant experimental results.\n\nWEAKNESSES:\n1. The paper does not discuss the additional computational and time costs associated with offline adaptation.\n\n2. Offline adaptation and opponent modeling share some similarities, so it is important to discuss their differences and include comparative experiments to highlight the distinctions between the two approaches, rather than just comparing self-play and behavior cloning methods.\n\n3. The term \"risk-free\" is mentioned several times in the paper. Please provide a more specific description of what \"risk\" refers to in the context of the paper, or provide some examples to clarify its meaning.\n\n4. The paper discusses issues related to human-AI interaction, and it would be beneficial to address the associated ethical concerns in the context of the proposed approach.\n\nQUESTIONS:\nPlease refer to Weaknesses Section\n\nLIMITATIONS:\nPlease refer to Weaknesses Section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}