Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators
Authors: Haoming Wang
Zhaoming Tian
Yunpeng Song
Xiangliang Zhang
Zhongmin Cai
Venue: NeurIPS 2024 poster (2024)
Keywords: human-AI collaboration
Bayesian policy reuse
reinforcement learning
Primary area: human-AI_interaction
TL;DR: An effective human-AI collaboration framework that adaptively selects optimal policies in non-stationary tasks, with guarantees for rapid convergence and superior performance in dynamic environments.

Abstract: In human-AI collaborative tasks, the distribution of human behavior, influenced by mental models, is non-stationary, manifesting in various levels of initiative and different collaborative strategies. A significant challenge in human-AI collaboration is determining how to collaborate effectively with humans exhibiting non-stationary dynamics. Current collaborative agents involve initially running self-play (SP) multiple times to build a policy pool, followed by training the final adaptive policy against this pool. These agents themselves are a single policy network, which is $\textbf{insufficient for handling non-stationary human dynamics}$. We discern that despite the inherent diversity in human behaviors, the $\textbf{underlying meta-tasks within specific collaborative contexts tend to be strikingly similar}$. Accordingly, we propose $\textbf{C}$ollaborative $\textbf{B}$ayesian $\textbf{P}$olicy $\textbf{R}$euse ($\textbf{CBPR}$), a novel Bayesian-based framework that $\textbf{adaptively selects optimal collaborative policies matching the current meta-task from multiple policy networks}$ instead of just selecting actions relying on a single policy network. We provide theoretical guarantees for CBPR's rapid convergence to the optimal policy once human partners alter their policies. This framework shifts from directly modeling human behavior to identifying various meta-tasks that support human decision-making and training meta-task playing (MTP) agents tailored to enhance collaboration. Our method undergoes rigorous testing in a well-recognized collaborative cooking simulator, $\textit{Overcooked}$. Both empirical results and user studies demonstrate CBPR's superior competitiveness compared to existing baselines.

Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators
Haoming Wang"., Zhaoming Tian".,, Yunpeng Song', Xiangliang Zhang", Zhongmin Caiti
MOE KLINNS Lab, Xi'an Jiaotong University, Xi'an, an, Shaanxi, China Department of Computer Science and Engineering. University of Notre Dame, Notre Dame, IN, USA
faanghm,tmm8822eer xjtu. edu. en, yunpengs@ xjttu.duucn, xzhang33@nd edu,
zmcai@sei xjju. edu.cn
Abstract
In human-Al collaborative tasks, the distribution of human behavior, influenced by mental models, non-stationary, manifesting various levels of initiative and different collaborative strategies. A significant challenge in human-Al collaboration is determining how to collaborate effectively with humans exhibiting non-stationary dynamics. Current collaborative agents involve initially running self-play (SP) multiple times to build a policy pool, followed by training the final adaptive policy against this pool. These agents themselves are single policy network, which is insufficient for handling non-stationary human dynamics. We discern that despite the inherent diversity in human behaviors, the underlying meta-tasks within specific collaborative contexts tend to be strikingly similar. Accordingly, propose Collaborative Bayesian Policy Reuse (CBPR)) ), novel Bayesian- based framework that adaptively selects optimal collaborative policies matching the current meta-task from multiple policy networks instead of just selecting actions relying on a single policy network. We provide theoretical guarantees for CBPR's rapid convergence to the optimal policy once human partners alter their policies. This framework shifts from directly modeling human behavior to identifying various meta-tasks that support human decision-making and training mata-task playing (MTP) agents tailored to enhance collaboration. Our method undergoes rigorous testing
vell-recog collaborative cooking simulator, Overcooked. Both empirical results and user studies demonstrate CBPR's 's superior competitiveness compared to existing baselines.
Introduction
An ongoing challenge in artificial intelligence (AI) involves training agents capable of effective collaboration with humans Klien et al. [2004], Bard et al. [2020], Dafoe et al. [2020]. Unlike typical Al-only multi-agent collaboration, human-Al collaborative scenarios such as two-player cooking games, autonomous driving, and managing power grid stability incorporates non-stationary component, humans Jagerman et al. [2019], Chandak et al. [2020], Chandak [2022]. As humans may vary in their level of initiative, alter their collaboration strategies, or sometimes even do not collaborate at all. This variability suggests that for cooperative agents, the probability distribution P(A[sr) (x) of human action A given environmental state 8, changes over time, reflecting different
'Equal contribution. Corresponding author.
We make our code publicly available https://github: com wtt.////lleaaaaaaaonnnnnn////a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Collaboration with Humans through Identification the Meta-Task Being Performed
Collaborate with Humans Using Stationary Policy
Onion
Put onions in pot
Deliver the soup
Serving
Unable to determine the collaborative policy being executed by humans
Put onions a pot & deliver soup
Others
Effectively adapting humans' non-stationary policies
Figure 1: Left: The drawbacks of current collaborative agents, which train a stationary policy manage the non-stationary dynamics of human collaborators but fail to determine the specific collaborative policies executed by humans. Right: Our approach focuses on identifying the meta-tasks underlying human decision-maken making and trains collaborators match these meta-tasks one-to-one manner. This strategy enables effective ad-hoc collaboration with non-stationary humans.
mental states. Such stttsnatinn poses significant challenge in training collaborative agents, as it requires strategies that can adapt to the unpredictable nature of human behavior, which departs from the stable action-outcome associations expected in scenarios dominated by AI.
Recent works mainly develop collaborative agents through two workflows: (1) explicitly model human behavior by using real human trajectories Carroll al. [2019], and then train collaborator by teamiig with human models. (2) train Self-Play (SP) agents to form policy pool (a diverse of AI agents assumed encompass all potential human policies) and then train a collaborator pairing with policies in the policy pool Strouse et al. [2021], J, Yu et al. [2023], Zhao et al. [2023]. However, despite their ability to achieve commendable performance by amassing extensive human data collection or SP agent training, these collaborators share common fundamental flaw: they are essentially policy networks following a stationary distribution, thus making il difficult to cope with non-stationary human dynamics.
In this work, we propose Collaborative Bayesian Policy Reuse (CBPR), which reuses multiple stationary policies tailored to meta-tasks within specific collaborative scenario. CBPR builds upon Bayesian Policy Reuse (BPR) Rosman al. [2016], Chen al. [2022], extending its application to human-AI collaborative tasks with theoretical guarantees. CBPR avoids modeling the non-stations non-stationary dynamies of human collaborators, focusing instead on heuristically modeling available meta tasks within defined collaborative contexts. For example, in the multi-player cooking game Overcooked, meta-tasks include {place onions in pot, deliver soup, place onions pot & deliver soup, others) (Figure l) are available. Noticing that for complex human-Al collaborative task, all of the undefined meta-tasks are categorized as "others," we subsequently train stationary meta-task-playing (MTP) collaborators using reinforcement learning (RI. to precisely match meta-task models on one-to-one basis. During collaboration, CBPR identifies the meta-task being performed by the human partner based on recent actions, subsequently adapting the optimal MTP collaborator for use.
We evaluate CBPR fully-observable two-player nommon-payonn collaborative cooking simulator based on the game Overcooked Carroll et al. [2019], which has recently been proposed coordina- tion challenge for AI Carroll et al. [2019], McKee et [2022], Wang al. [2020], Wu al. [2021]. Knott et al. [2021]. 1]. State-of of-the-art performance of this game was achieved in Carroll et al. [2019], Strouse al. [2021], J, Yu et al. [2023] via training stationary cooperation policy. Both simulated experiments and user studies show that the proposed CBPR agent can collaborate effectively with non-stationary agents and real humans. The novel contributions of this paper can be summarized as follows:

---

1. We introduce human Al collaboration framework, CBPR, which addresses the challenge of modeling non-stationary human dynamics. This framework identifies the meta-tasks performed by human partners and reuses the optimal collaborative policy.
2. Theoretically, based on the Non-Stationary Markov Decision Process (NS-MDP), we provide theorems on Collaboration Convergence and Collaboration Optimality support CBPR's convergence to the optimal collaborative policy over time in Al collaboration.
3. Empirically, we demonstrate CBPR's ' capability to collaborate effectively with non-stationary agents who frequently switch strategies, agents with various collaboration skills, and real humans.
Related Work
2.1 Human-AI Collaboration
Training agents to collaborate with humans has been extensively studied. Recent research can be categorized into two groups based on whether human data is used during training. BCP Carroll et al. [2019] is trained by pairing with supervised human model, while Boltzmann Policy Distribution (BPD) Laidlaw and Dragan [2022] updates its prior based on online human actions. These approaches require human data collection and prone to distributional shifts. In contrast, another category focuses on achieving zero-shot coordination without extensive human data Hu et al. [2020]. These works (e.g., FCP Strouse al. [2021]. Hidden-Utility Utility Self-Play (HSP) Yu al. [2023]. and Maximum Entropy Population-base policy pool-a diverse set of AI agents assumed encompass all potential human policies-and then train a collaborator to pair with policies in this pool. However, these collaborative agents remain single stationary models despite their diverse training partners.
Training (MEP) Zhao et al. [2023]) train Self-Play (SP) agents to form a
Our work represents fundamental departure from previous studies by avoiding the need to model human behavior and instead focusing on constructing meta-tasks that underpin human decision- making. Furthermore, our CBPR framework does not restrict the construction of meta-tasks, which be categorized into two streams: reliant human data (e.g., behavior cloning) and independent of human data (e.g., rule-based methods).
2.2 Policy Reuse
Policy reuse is a kind of transfer learning method that can greatly speed up reinforcement learning for new task by using policies for relevant tasks. Initial methods like PRQL Fernández and Veloso [2013] and OPS-TL Li and Zhang [2018], Li et al. [2018] integrated source policies with limitations in transfer efficiency. Subsequent approaches such as CAPS and CUP Zhang et al. [2022] improved policy selection and introduced more efficient algorithms without the need for extra training components.
Bayesian policy reuse (BPR) Rosman et [2016] represents a specialized stream within policy reuse. Utilizing Bayesian optimization approach, BPR efficiently computes posteriors for novel tasks. Extensions like BPR+ Hernandez-Leal al. [201 6a.b] and Bayes-Pepper Hernandez- Leal and Kaisers [2017] adapt BPR to multiagent scenarios, aligning tasks with opponent strategies and policies with optimal responses to these strategies. However, most BPR methodologies Rosman et al. [2016], Hernandez-Leal et al. [2016a], Hernandez-Leal and Kaisers [2017], Zheng et al. [2018, 202 l], Chen et al. [2022], Xie et al. [2022] primarily address multi-task problems or copy with competitive scenarios. Several studies, such as Zheng et al. [2018, 2021], investigated deep BPR+ in collaborative games.
However, these approaches primarily rely on policy inference to adjust to the changing strategies of opponents (or partners), which may not optimal for human-Al collaboration given the wide spectrum of potential human policies. To our knowledge, our research is pioneering in applying and tailoring Bayesian policy reuse-based algorithms specifically for the human-Al collaboration challenge.
3

---

Stago Ontine Stage
Meta-task& Al Policy
Initialization
trajectory data
Meta-task Library TET
ontonion
Fuke plane
collect
Context
Performance
mate-tan
Model
Game Play
ft
P(ULLT)
mette
Al Policy Library
R
02
0g
Ok-1
a
Context
mete-task
I e
action
arg max
(Cr)(UU r,s)a0
Ad
Σ P(s
T.m.
LA.1A. (A.
11g
policy
deternine human
Figure 2: Overview of the CBPR Framework. This framework is divided into two main phases. Left: Offline Training Phase. This includes (1) constructing meta-task models using collected data and creating a meta-task library; (2) developing cooperative policies for each meta-task to compile an AI policy library: (3) establishing performance model by evaluating each meta-task and Al policy pair. Right: Online Collaboration Phase. During a collaboration round, the process involves (a) gathering a list of historical and current human data; (b) determining the current meta-task undertaken by the human using Bayesian policy inference (refer to Equation 3-4); (c) selecting the most suitable AI policy for cooperation (as per Equation 5); and finally, (d) the AI collaborator executes actions according to the chosen policy.
Collaborative Bayesian Policy Reuse
3.1 Vanilla Bayesian Policy Reuse
Bayesian policy reuse a general framework of transfer learning to cope with unknown tasks or frequently changing opponents. These classes of methods typically involve two phases: offline learning phase and an online reusing phase. The workflow of a typical BPR can be summarized as follows: In the offline phase, it is presupposed that there exists library of tasks T and a corresponding library of learned policies II. Through conducting multiple simulations with varied policies across different tasks, performance model P(U T, IT) derived, where U Ek cumulative utility. This model works as a mapping operator, associating each task and policy with a distribution of predefined utility measure, such as reward.
During the online phase, BPR identifies the current task or opponent policy by maintaining belief model B(-). This model is periodically updated based on observations, defined by the observation model P(a T, x), where a represents any signal aiding cooperation, such as reward or interaction trajectory. Significantly, this update adheres to Bayes' rule as follows:
P r. T. (T)
Bx(T) Er ( ()) Bx-1
With this belief model, the BPR agent can select the optimal response policy by solving the following optimization problem:
π argmax
(())) IT,T) du
where = maxxen ΣTET TE t, represents the average performance of a single policy across all tasks. It's important to note that using Ü as the lower limit of the integral, this optimization problem essentially seeks the policy with the highest likelihood of achieving utility above the average.
3.2 CBPR Framework
Offline stage Initially, we train meta-task processing (MTP) agents æ E TT using the Proximal Policy Optimization (PPO) algorithm by individually pairing them with meta-tasks within specific

---

collaborative context, as exemplified by tasks such as place onions in por, deliver soup, place onions in por & deliver soup, and others in the Overcooked collaboration benchmark. Meta-task models E 'T are constructed through supervised learning, utilizing trajectories from either rule-hased agents enhanced with noise or real humans performing the tasks. In this study, we employ the rule-based agents developed by Yu et al. [2023]. Subsequently, we construct the performance model P(U TT, 11) (i.e., observation model) by fitting a Gaussian distribution over the mean episodic return given a stochastic AI policy π and a noisy rule-based agent
In previous BPR-based algorithms, the belief is designed for measuring the similarity between different tasks or opponents in transfer learning. These algorithms update belief using a observation model P (a 5, т which only considers the game result but overlooks opponent's behavior. This leads to a poor collaborative performance when humans switch policy in a long-episode game. In this study, we used intra-episode belief (T) at timestep /. to measure the similarity between current meta-task T and r' in meta-task model library T. The intra-episode belief was firstly proposed in Chen et al. [2022] and we extend it to the human-AI collaborative scenario.
Online policy reuse At the beginning of online policy reuse, the inter-episode belief (D(T)) is initialized with uniform distribution. For each episode, CBPR maintains a first-in-frst-out (FIFO) human behavior queue Q of length 1, which records the latest human behavior tuples (s;, az). The Al selects initial response MTP agents according to the inter-episode belief 3o (line 5 in Algorithm 1). CBPR collects human state-action pairs and updates the intra-episode belief (T):
P(21 (-) E((T) P E1)
exp( (S log ((a)).))
where P(0 7) =
Then the intra-episode belief and inter-episode
Eder Sip log 7' (a.ls.))
belief are integrated:
G(T)=
Where p E [0, 1] is hyperparameter controlling the weight of the inter-episode and intra-episode beliefs. As the timestep increases in a game with a long episode, the integrated belief (e(T) primarily depends on the intra-episode belief (e()). The Al then uses the integrated belief (T(T) to select a policy cooperate with the human at each timestep.
E arg max TElT
EG(T) [, (T,T)dU du+ TET
(5)
At the end of each episode. CBPR collects the episodic return ug and updates the inter-episode belief' (T). To adapt to non-stationary human dynamics, we store human-Al trajectories in a replay buffer R of the current MTP agent and update its policy. The detailed pseudo-code for the policy reuse of CBPR is presented in Algorithm 1.
3.3 Theory Analysis of CBPR
The selection of cooperative policies (line 11 in the Algorithm 1) is crucial to the performance of CBPR in collaborating with humans. In this section, we propose theorems on the convergence and optimality of CBPR to support our viewpoint: CBPR will converge to the optimal cooperative strategy during the human-Al interaction process. We formulate collaborative process between humans and AI as Non-Stationary MDP (NS-MDP) Chandak et al. [2020]. In this process, the non-stationarity, resulting from the dynamic nature of human policy, can be mitigated by decomposing the entire non-stationary decision process into several stationary ones. Each stationary MDP corresponds to a specific meta-task exccuted by the human. Specifically, for a given NS-MDP ) the transition function integrates human actions part of the environment itself, which can be denoted as P:: : S AAI x Aku. + ASS). Within each stationary MDP M, the human policy Tha,i S - A(AA is assumed to be stationary, although it may exhibit variations across different stationary MDPs. Under this assumption, the CBPR agent could establish a convergent human-AI collaboration:
THEOREM (Collaboration Convergence of CBPR Agent). Ler H; == (S{ (S), Ri be
trajectory collected from a single stationary MDP Mj within the overall NS-MDP (M. under the human meta-task policy Tha.i Denote D == {ii, H.) i E k1, as a random variable representing set of trajectories observed prior to the most recently completed stationary MDP My. Given D, the
5

---

Algorithm Online Policy Reuse of CBPR
Input: Meta-task model library T, meta-task playing (MTP) agent library II, performance model P( T), human behavior queue Q = 0. total timesteps T in one episode
Initialize (T) with uniform distribution
2: for episode k=1,2,3,.... .K do
3: Empty the queue Q
4: Eo(T) < (7)
5: Select initial MTP agent x to cooperate with human using Eq.
6:
while i < T do
7:
Human chooses action a, and Al choose action according to Ta
8:
Append the human behavior tuple (st, a.) to Q
9:
Update belief (T( using Eq. 3
10:
Update integrated belief (e(T) using Eq. 4
11:
Select a optimal MTP agent æ to cooperate with human in next timestep by using Eq. 5
12:
Ee(T) + (e(T)
13:
& t t t++11
14:
end while
15:
(x(T) + (T)
Update belief (x(T) using episodic return un as observation signal following Eq.
17: end for
response policy of CBPR agent could almost sure converge when interacting with human partner, even when the human's policy is non-stationary.
We provide all proofs and detailed explanation in Appendix A. In addition to being able to converge in cooperation with non-stationary humans, the CBPR agent can also establish the optimal collaboration policy:
THEOREM 2 (Collaboration Optimality of CBPR Agent), Denoting CBPR CBPR algorithm, let P PU++ (Tm),, π) dU+] be the expected return of exploiting Al policy т with human meta-task policy T T(m) in MDP Mn. Given a positive integer i and set of trajectories D observed prior to the MDP Mg, it follows that for any subsequent stationary MDP M we have:
p(T,, m) == ElJo U/
Pr Pr(p(CBPR(D),, +6) pl
(6)
when k oo, where गे is the optimal response policy for human meta-task policy at MDP M
4 Experiments
In the context of Overcooked, we use rulc-based policies developed in Yu et al. [2023] for each game layout (see Appendix C.1). These rule-based policies such as place onions pot, deliver soup are used to train corresponding MTP agents in one-to one-to-one manner. In this section, we conduct extensive experiments to answer the following questions:
Q1: When interacting with non-stationary agents who switch their strategies, can CBPR outperform established baselines? Additionally, can CBPR adapt its collaborative strategies to better synchronize with partner behaviors?
Q2: When interacting with non -on-stationary agents of various collaboration skills, can CBPR surpass other basclines?
Q3: Can CBPR exceed the performance of other baselines in collaboration with real humans? Q4: How do hyperparameters and number of predefined meta-tasks influence the collaborative performance (mean reward) CBPR agents?
Overcooked environment Overcooked is popular two-player common caammon-payoff game. It has become a typical environment for studying Auman-Al collaboration Carroll et al. [2019], Knott al. [202] ]], Strouse et al. [2021], McKee et al. [2022], Yu et al. [2023]. In this game, players should place three onions or tomatoes in a pot and deliver as many cooked soups as possible within time limit. Good coordination between two players is crucial for achieving a high score. We employed four layouts in

---

our experiments: Cramped Room, Coordination Ring, Asymmetric Advantage and Soup Coordination (Figure 8 in Appendix) our experiments. Notably, in the Asymm. Adv. and Soup Coord. the players do not interfere with each other, and their movements are unobstructed by their partners.
Baselines We compare CBPR against three well-established baselines: (1) the Behavioral Cloning Play (BCP) Carroll et al. [2019], a human model- method designed for human-Al collaboration; (2) Fictitious Co-Play (FCP) Strouse al. [2021]. a two-stage approach trained with partners of varying skill levels; (3) Self-Play (SP) Silver et al. [2017], a common RL method trained by playing against itself. For fair comparison, we employed PPO Schulman et al. [2017] as the underlying algorithm of CBPR and reimplemented all baselines using identical hyperparameters in our experiments. Further details about environment setting and agents are illustrated in Appendix
4.1 Cooperating with Rule-Based Agents Under Dynamic Policy Switching
To answer question Q1, we conduct a thorough investigation into the collaboration performance of CBPR when paired with non-stationary agents. These agents exhibited changes in their rule- based policies (Appendix Table 3), both inter-episodically and intra-episodically We maintained a consistent random seed for policy switching during the evaluations to ensure fairness when comparing CBPR with baseline methods.
In our experiment, evaluate the collaborative performance of agents at four different policy switching frequencies, as shown in Figure 3. The results show that CBPR consistently outperforms baseline methods most cases. In particular. BCP, which was trained using a stationary human model, exhibited significantly poorer performance compared to CBPR. In addition. FCP and SP agents show greater fluctuations in episodic rewards, primarily due their inability to effectively collaborate with all agents. In some instances, SP agents opted not cooperate, resulting in zero reward.
Moin Waine eahod mwar
Cremped
Coundiawttnn on Ring
A Achentagge
Soup Coontivation
MMMMM
30
1000
250
30
Epicodes
CRIP22
Figure 3: Comparative performance analysis against baselines when collaborators switch their rule- based policies per episode. All agents were evaluated over 50 continuous episodes. The shaded areas denote standard deviation calculated from five random seeds.
Our findings indicate that CBPR is particularly effective at collaborating with partners exhibiting varying degrees of non-stationarity. For detailed overview of the results across the additional three policy switching frequencies (i.e., per 2 episodes, per 200 timesteps, and per 100 timesteps), please refer to the Appendix C.
per.
4.2 Cooperation with Partners of Various Collaboration Skills
The cooperative capacity of non-stationary humans is typically suboptimal. A generalized agent must be capable of collaborating with partners possessing diverse collaboration skills.
During the initial training phase of FCP Strouse et al. [2021]. a policy pool is created by preserving various agent "checkpoints" that represent different levels of expertise. To answer question Q2, we pair CBPR with agents with varying collaboration skills preserved during the first stage of the FCP training. We evaluate collaborative performance over 50 episodes on four layouts. The results show that CBPR consistently achieved higher mean episode rewards than FCP, particularly when collaborating with lower-skilled partners (Figure 4). is noteworthy that BCP performs better in the Asymn. Adv. and Soup Coord. in which players' movements are not hindered by their partners. We replayed the trajectories of BCP in Cramped Rm. and Coord. Ring and observed that BCP occasionally became immobilized and failed to collaborate with partners (Figure 4b).
7

---

150
300
Misee up aphoode wewsed
Mean epesce
200
100
6600
Moan
200
500
100
Mean
LoW
Modium
High
Law
Meduu
High
Low
Meoku
High
LOW
Modium
High
(a) Cramped Rm.
(b) Coord. Ring
(c) Asymm. Adv.
(d) Soup Coord.
Figure 4: Comparative performance analysis against baselines in cooperation with partners of diverse
skill levels (low, medium and high). All agents evaluated over 50 episodes and errors bars denote 95% confidence intervals.
4.3 Cooperation with Real Humans
To address question Q3, we recruited 25 volunteers from a local university, comprising females study involving collaboration
and 20 males, ranging age from 21 to 34 years, to participate with CBPR and baseline agents. These volunteers were randomly assigned to one of four groups, each corresponding to a different game layout. Prior to the experiment, nearly all volunteers were unfamiliar with Overcooked. We provided comprehensive instructions from scratch and allowed them to play at least five practice rounds before beginning the evaluation. Subsequently, participants were instructed to interact with both the CBPR and baseline agents through the human-Al web applications developed by Carroll et al. [2019]. Each volunteer participated in two episodes, during which we recorded the average reward obtained.
According the reward distribution (Figure 5), we ob- serve that CBPR achieves more efficient collaboration than other baselines. most comparisons, CBPR dis- plays significant higher reward according to the one-sided Mann-Whitney U test.
Edtradoo reward
CBPR
500
Case study To further demonstrate how the CBPR is more superior than baseline algorithms when collaborating with real humans, we present a case in Figure 6. In this case, we record five frames from the Overcooked game interface to show that the ability of CBPR to adaptively adjust cooperative policies. Initially, CBPR agent is ready to use a dish to serve the soon-to-be-ready soup. When human partner picks the soup, CBPR will set down the dish and continue to place onions to the pot for new round. Mcanwhile, FCP, after putting down the dish, will appear confused until the human served the soup. BCP, the other hand, will not put down the dish and stubbornly prepare serve the soup, ignoring the fact that the soup had already been served.
200
100
Gasmzed
Goondration
Asyrmetti
Soup
Figure Rewards distribution of agents collaborating with real humans over four layouts. *, p < 0.05; ** < 0.01; *** p < 0.001, and n.s., not significant. (Statistical significance was assessed by sided Mann- Hoin-Whitney U test.)
4.4 Ablation Study
Ablation on the queue size / and inter-episodic belief weight p. In CBPR, the length I of human behavior queue and weight inter-episodic belief mainly influence the collaborative performance. The larger in P(Q r) of Eq. 3 means that CBPR chooses policy considering more past human behaviors. The larger determines that CBPR needs to consider inter-episodic belief more at the beginning of episode. To answer question Q4, we expand the experiments from section 4.2 demonstrate the results in Figure 7 and Appendix D.2. Overall, the results show that 1=20 performs best, and in relative simple layout (i.e., Cramped Rm.), since the belief of cooperative policy converges easily, variation in has little impact on the reward. However, in complex layout (e.g., certain extent.
Soup Coord.) (Figure 16), adjusting p can enhance cooperative performance
Ablation on the number of predefined meta-tasks. The performance of CBPR depends on the design of the meta-tasks. To address the challenge of predefined meta-tasks not covering all

---

Human:
I'm should add new onione pot
but now tive charggu
pollcy and new onions the pot
Pot
CBPR
Serving area
Onion dispenser
Table
FCP
Dish dispenser
Human
BCP
Human with cooked soup
Timesteps
Figure 6: This case study analyzes five discontinuous frames from the Overcooked game interface demonstrate the superiority of the CBPR algorithm. When a human player picks the cooked soup from the pot, the CBPR agent adapts by altering its initial plan deliver the soup: it sets down the dish and places new onions in the pot, thereby showcasing its ability to adjust to human policies. In contrast, the FCP agent displays confusion when the human retrieves the soup and resumes placing onions only after the soup is served. The BCP agent rigidly adheres to its predetermined plan, continuously holding the plate without switching tasks to place onions, ignoring the fact that the soup has already been served.
possible ones in complex task scenarios, we introduce a meta-task category as "other" (Figure 1, bottom-right) which is represented using a random agent in practice. To demonstrate the impact of the number of predefined meta-tasks in the Soup Coord. We pair CBPR of different numbers of predefined meta-tasks with agents employing various skill levels. The results in Table show that without "others" category, the performance deteriorates significantly, while the performances degrade relatively gracefully with less meta-tasks defined and more included in "others" category.
Table 1: Collaboration performance of CBPR with different numbers of meta-tasks and agents employing various skill levels. We report the mean reward over 10 episodes and the values in bracket represent the standard deviation. Here, additionally define four meta-tasks (i.e., place onion & deliver soup, place tomato & deliver soup, pickup tomato & place mix and pickup ingredient place mix), which included in Table 4.
predefined w/ "others"
predefined w/ "otherss"
3 predefined w/ "others"
predefined w/o "others"
High
620.3 (193.3)
600.7 (234.0)
647.7 (159.3)
622.8 (205.8)
Medium
757.8 (100.3)
735.8 (98.7)
717.1 (148.1)
607.3 (278.5)
Low
689.8 (43.9)
680.5 (51.6)
668.9 (49.0)
40.0 (59.1)
4.5 Additional Findings and Analysis
The inherent advantage of SP and FCP agents. Checkpoints, which are essentially SP agents, represent partners with low, medium, and high skill levels at the beginning, middle, and end of FCP training. Therefore, SP and FCP agents have an inherent advantage in the evaluation presented in Figure 4. Despite this, CBPR performs better when dealing with partners of lower skill levels. When collaborating with real humans, FCP and SP no longer hold the same advantages. This leads to almost all FCP and some SP performing well against agents of various skill levels, but falling short when facing human players.
The cooperative advantage of CBPR in non-separated layouts. In separated layouts (i.e., Asymm. Adv. and Soup Coord.), agents can usually complete tasks independently without considering the hindrance of the other partner's 's moves to themselves. However, players' own position (c.g., stand still

---

in front of the serving areas) can obstruct their partners from completing the task in the non-separated layouts. Therefore, non-separated layouts require more cooperation between players compared to separated layouts. As shown in Figure 4, CBPR's better performance in Cramped Rm. and Coord. Ring suggests its advantage in collaborative tasks.
The double-edged sword of simple policy. In Asymm. Adv., SP agent exhibits outstanding performance when it cooperates with the agent
poba
of high skill level (Figure 4c). We replayed the game and found that the SP agent learned
Moar optede ewas
300
Moan usicade swadd
pit
200
200
the simplest and most effective policy (i.e., in the right room, just pick an onion from onion dispenser and then place it in a pot within the
1:00
shortest path). On the contrary, other agents exhibit some superfluous actions due to their own complexity. However, when cooperating with the agent of low skill level, SP performs poorly because the SP agent on the right only learned the simplest policy (putting onions in the pot), and when the agent with low skill level on the left does not deliver the cooked soup, SP will wait in place rather than deliver the cooked soup. In more complex layout Soup Coord., found that the SP agent learned policy of putting only one onion in the pot and starting to cook, leaving its partner confused and uncertain about what went wrong. Therefore, cooperation with SP agents leads to low performance (Figure 4d).
Low
Medium
High
Low
Meduu
High
(a) Ablations of
(b) Ablations of
Figure 7: Episodic reward by using different length I of human behavior queue and weight p of inter- episodic belief in Cramped Rm. layout. All agents evaluated over 50 episodes and error bars de- note 95% confidence intervals.
5 Conclusion and Discussion
Conclusion In this work, we proposed CBPR framework and evaluated in the well-known game Overcooked. CBPR could effectively tackle the challenge of collaborating with humans by utilizing a suite of meta-task aware agents. response to the non-stationary nature of human behavior, CBPR adeptly selects MTP agent based on the most recent human actions and episodic returns. We have theoretically underpinned the collaborative efficacy of the CBPR approach. Empirically, demonstrated that CBPR outperforms baselines when collaborates with simulated humans that change their policies frequently, simulated humans that employ different skill levels and real human players. We remark our primary argument that, given the statiiott inherent in human behaviors, it is more effective to design various agents tailored to corresponding humans in different mental and behavioral states, rather than relying seemingly omnipotent single agent. After all, two heads are better than one.
Limitations and future work In this work, meta-tasks are modeled by manually-designel rule-based policies. In real-world application domains such as assessing power system transient stability in power grid dispatching and autonomous driving, it is time consuming to design various rule-based policies.
CBPPR CBPR offers viable strategy to model meta-tasks, facilitating the training of multiple specialized experts to handle distinct meta-tasks. A notable challenge, however, is the manual summarization of domain experts' meta-tasks. As direction for future research, we are keen to address the task of clustering policies automatically based on human trajectories. While this study Zhang et al. [2023] has made strides in this direction, the clustering approach adopted therein tends to obscure semantic understanding, presenting hurdles for AI in comprehending human behaviors. Splitting human trajectories according to the key state may be a possible solution. Additionally, perceiving the acquisition of a specific class of shaped rewards by agent the execution of meta-task merits future consideration. This approach also does not depend on human data or models and offers enhanced universality and interpretability.
Acknowledgements
We are grateful to Professor Xiaohong Guan for his kind support of this work and anonymous reviewers for their insightful comments. This work was supported by the National Key R&D Program of China (2021 YFB2400800).
10

---

References
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019. 2, 6, 7, 8, 15, 17
Yash Chandak. Reinforcement Learning for Non-stationary problems. PhD thesis, PhD thesis, University of Massachusetts Amherst, 2022.
Yash Chandak, Scott Jordan, Georgios Theocharous, Martha White, and Philip Thomas. Towards safe policy improvement for non-stationary mdps. Advances in Neural Information Processing
Systems, 33:9156-9166. 2020. 5,5
Hao Chen, Quan Liu, Ke Fu, Jian Huang, Chang Wang. and Jianxing Gong. Accurate policy detection and efficient knowledge reuse against multi-strategic opponents. Knowledge-Bacd Based Systems, 242: 108404, 2022. 2, 5,5
Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.00 201.0::0000020.6000 2020. 1
Fernando Fernández and Manuela Veloso. Learning domain structure through probabilistic policy reuse in reinforcement learning. Progress in Artificial Intelligence, 2(1):13-27, 2013.
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai. Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 35:15281-15295, 2022. 16
Pablo Hernandez-Leal and Michael Kaisers. Towards fast detection of opponents in repeated stochastic games. In International Conference on Autonomous Agents and Multiagent Systems, pages 239-257. Springer, 2017.
Pablo Hernandez-Leal Leal, Benjamin Rosman, Matthew E Taylor, L Enrique Sucar, and Enrique Munoz de Cote. bayesian approach for learning and tracking switching, non-stan non-staiionayy opponents. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, pages 1315-1316, 2016a.
Pablo Hernandez- Matthew E Taylor, Benjamin Rosman, L. Enrique Sucar, and Enrique Munoz De Cote. Identifying and tracking switching. non-stationary opponents: A bayesian approach. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, 2016b.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pages 2399-4410. PMLR, 2020. 3
Rolf Jagerman, Ilya Markov, and Maarten de Rijke. When people change their mind: Off-policy evaluation in non-stationary recommendation environments. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 447 455, 2019.
Glen Klien, David D Woods, Jeffrey M Bradshaw, Robert R Hoffman, and Paul J Feltovich. Ten challenges for making automation a" team player" in joint human-agent activity. IEEE Intelligent Systems, 19(6)991995, 2004. 1
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca D Dragan, and Rohin Shah. Evaluating the robustness of collaborative agents. arXiv preprint arXiv:2 2021. 2, 6
Cassidy Laidlaw and Anca Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. arXiv preprint arXiv:2204.10752 2022. 3

---

Siyuan Li and Chongjie Zhang. An optimal online method of selecting source policies for reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 3
Siyuan Li, Fangda Gu, Guangxiang Zhu, and Chongjje Zhang. Context-aware policy reuse. arXiv
preprint arXiv: 03793, 2018. 3, 15
Kevin R McKee, Joel Z Leibo, Charlie Beattic, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1): (6(1):21, 2022. 2, 6
Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy. Bayesian policy reuse. Machine Learning, (04(1):99-11)) 127, 2016. 2,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 7, 16
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017. 7
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021. 2, 3, 6, 17
Rose Wang, Sarah A Wu, James A Evans, Joshua Tenenbaum, David C Parkes, and Max Kleiman- Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. 2020.
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman- Nleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science, 13(2):414 432, 2021. 2
Donghan Xie, Zhi Wang, Chunlin Chen, and Daoyi Dong. Efficient bayesian policy reuse with
scalable observation model in deep reinforcement learning. arXiv preprint vrXiv:220..07729, 2022. 3
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 11-24611-24624 1-2424,, 2022. 16
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. arXiv preprint arXiv:2302.0144 0005, 2023. 2, 3, 5, 6, 16
Jin Zhang, Siyuan Li, and Chongjie Zhang. Cup: Critic-guided policy reuse. arXiv preprint arXiv:2210.08153, 2022. 3
Ziqian Zhang, Lei Yuan, Lihe Li, Ke Xue, Chengxing Jia, Cong Guan, Chao Qian, and Yang Yu. Fast teammate adaptation in the presence of sudden policy change. arXiv preprint 205.01330.. 2023. 10
Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153, 2023. 2,
Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan. A deep bayesian policy reuse approach against non-stationary agents. Advances neural information processing systems, 31, 2018.
Yan Zheng, Jianye Hao, Zongzhang Zhang, Zhaopeng Meng, Tianpei Yang, Yanran Li, and Changjie Fan. Efficient policy detecting and reusing for non-stationarity markov games. Autonomous Agents and Multi-Agent Systems, 35(1):1-29, 2021.
12

---

A Proof of collaboration performance
A.1 Proof of collaboration convergence
THEOREM (Collaboration Convergence of CBPR Agent). Let H; == (S), Thu, ( R)), )=j be trajectory collected from a single stationary MDP M; within the overall NS-MDP (Mi) under the human meta-task policy Tha.j- Denote D == {(i, H:) : E k.} as a random variable representing set of trajectories observed prior the most recently completed stationary MDP My- Given DD the response policy of CBPR agent could almost sure converge when interacting with human partner, even when the human's policy is non-stationary.
To establish the convergence of the posterior distribution, we first note that Doob's 's Martingale Convergence Theorem applies our setting. Specifically, we have the following theorem:
THEOREM (Doob's s Martingale Convergence Theorem). Let Xn be martingale (or sub-marting mub-martingale or super-martingale with respect to the sequence of sigma-algebras Fu, such that El| xx I] oo for all n. If there exists constant C such that F Xx Fn) S C for all n, then there exists random variable X such that X,, converges to X almost surely and L,².
With the aforementioned theorem, we can readily establish the proof of our theorem.
Proof. For non-stationary MDPs, demonstrating convergence involves showing that the algorithm can adapt to changing convergence points and ultimately reach them. Therefore, we will first establish the convergence property of the Bayesian update. Specifically, will be demonstrated that the posterior distribution converges almost surely to the true parameter value. Subsequently, we will prove that, when using Bayesian updates, CBPR algorithms always converge to a fixed response policy, provided that the human policy remains unchanged before reaching the fixed response policy.
To establish the convergence of posterior distribution, we first proof that the Doob's Martingale P(ay)r..a,88 .(T)
Convergence Theorem holds for the Bayesian updating: (x( = Ever P(rral',,T,) (606))("
Consider F& as the sequence of sigma- algebras generated by observations up to time k. A fundamental property of Bayesian updating that the expected value of the posterior distribution conditioned on past data equals the current posterior distribution, expressed as E[3e+1 (T) holds because the posterior distribution (T) encapsulates all relevant information up to time k. Thus, conditioning on F& accounts for all past observations, and in the absence of new data, the expected future posterior must align with the current posterior. This relationship signifies that, given the information available up to time k, the expectation of the next posterior does not deviate from the current posterior, establishing 3x (T) as a martingale with respect to Fx-
sigma-algebras
(a(T). This
Moreover, the bounded nature of (T) within the interval [0, 1] ensures that the Bayesian update satis- fies the conditions of Doob's 's Martingale Convergence Theorem. Since (x( represents a probability, it is inherently bounded, which guarantees that the expected absolute change E|13,+ (x) [FR] remains bounded. Additionally, with E[ (T)] = 1, the integrability condition required for martin- gale convergence is also satisfied. This combination of boundedness and integrability provides the mathematical foundation that guarantees the convergence of the sequence A% (T).
In conclusion, the sequence of Bayesian updates Be(T) adheres to the defining properties of a martingale and satisfies the conditions of Doob's Martingale Convergence Theorem through its inferent property and boundedness. As a result, we can conclude that the belief (A(T)) regarding the human meta- task will converge as k - oc:
Pr (r((e(t) +
Secondly, to prove that the calculated best response policy of AI π² converges to fixed value as k - 20, we consider both the structure of the Bayesian update and the decision- decision-making process in CBPR framework.
Given By (T) converges, we note that the uncertainty about the human behavior policy 7 diminishes with an increasing number of observations. The convergence of (x( to specific distribution implies that the belief about the human's 's policy stabilizes. mathematical terms, as k - oc,
(T) + 3(r) for some fixed distribution ((r).
13

---

Then the stabilized response policy of Al n²* is given by:
π ar argmax
3(T)P (U+ du+
(8)
TET
Here, the decision makinng is a function of both the belief B(T) and the expected utility P (U+ TT) for each Al response policy и. As (x) converges to (T),, the decision-making process becomes increasingly dependent on stable belief about the human's policy. Thus, the variability in the choice of π diminishes, leading to a convergence of n as well.
Formally, the convergence of x²* can be shown by demonstrating that the integral expression defining Λ becomes stable as k + oc. Since ((r) stabilizes, the integral's value, which depends on the belief' about also stabilizes. Consequently, by the linearity of convergence, the policy that maximizes this expression, π will almost sure converge to fixed policy.
Given the convergence property of πT" the almost sure convergence for the response policy of our CBPR agent is established.
A.2 Proof of collaboration optimality
THEOREM (Collaboration Optimality of CBPR Agent). Denoting CBPR CBPR algorithm, let
p(T,m) == :- Elfo graax
P (U+ (mm), x) dU*) be the expected return of exploiting Al policy π with human meta-task policy T T(m) in MDP Mn. Given positive integer k and set of trajectories 'D observed prior the MDP Mg, it follows that for any subsequent stationary MDP MK+6, we have:
Pr(p(CBPR(D), + 2 plat, +6))
when lo + 0c, where oे is the optimal response policy for human meta-task policy at MDP M
Proof. Considering the CBPR algorithm within the framework of MDPs, we define the expected return p(T,, m) as the integral of the probability achieving utility U+ given the Al policy æ and the human meta-task policy T (mm) in MDP Mn
Assuming that the human policy library and policy library encompass all possible human meta- policies and their corresponding best Al response policies. Then, we need to prove that the expected return of exploiting the CBPR algorithm's policy in any subsequent stationary MDP Ma+5 will be greater than or equal to that of the optimal response policy T at Mg. Formally, we can express this and derive follows:
Pr (c(p(CBPR(D)) +6) 2 k P))
UTMA
PPr
Σ B(TPP (U+ T, TCPPR) du+
P T(k: + ((, T(k') au+
TET
=Pr
B()P (U+ T, TCBPR) du
PU T(k + 5), (k)) du+ 20)
TET
=Pr
B(T(k: + 5) (U+ 1 1 8), TCBPR) P (U+ 1 (K + +), av+
U
+
Σ
B(T) (U+ ACBRR) du+ 20)
(10)
TET (F(k88)
Where T(k: + 5) represent the true stationary human meta-task policy at MDP Mr+5, x ((k*) the best response of AI at MDP Mg. TCBPR is the response policy generated by CBPR algorithm.
From theorem 1, we have Pr (B( + 6))) 1, when k + oo.
14

---

Then we have:
VEET- {r(k +8), B(r) - 0.
(11)
And thus the second term:
Σ (U+) #CBPR) dut 110+-1 0,
(12)
TET (ET-(r(b++))
while the first term:
lo mamax
((T(k++)))) (u+ 1 8), TCBPR) - (U+ 1 + (), m?) Jau+
converge to p( + 6) p( k 8). Since oo is the best response policy at MDP M the inequality p( k &+6) ) 2 P( k ++6) 6) would always hold. Consequently, we have Pr(p( k k+ 5) p( k k 5) 0) + 1, when P( k + p)) + when lo oc.
And finally achieve Pr(p(CBPR(D),, (D), k 6)
Note that the above derivation holds when the human meta-task policy library and Al policy library encompass all possible human meta-task policies and their corresponding best response policies. In practice, this assumption is seldom met and is not necessarily required to be satisfied. However, we can still enable to optimality guarantee by augmenting both human and Al policy library with primitive policies 1I, {T1, T2, , where policy m, 1lp takes action d; E A for all states Li et al. [2018].
B Environment settings
The Overcooked environment, as introduced in Carroll et al. [2019], presents cooperative game where two players aim complete many orders as possible within a limited timeframe. In this study, we set the time constraint to 600 timesteps. The players navigate the environment to interact with various objects essential for order completion. version of Overcooked requires additional 'interact' action to initiate cooking in the pot, deviating from the version used Carroll [2019]. This change necessitates an adaptation of the previously collected human data, potentially affecting the performance of the BCP baseline. To align with this modification, we have adapted the latest version of the game to support auto-cooking when three ingredients are in pot.
important aspect to note that the current
The environment's action space comprises the set {up, down, feft, right, stay, interact). The observation space is represented by 96-dimensional vector, capturing each player's s facing direction, absolute position, and relative positions to various game elements such as the partner, the nearest onion, pot, dish, serving area, etc. Our experiments utilize four distinct layouts as depicted in Figure 8. These layouts are chosen to illustrate a range collaborative challenges and rewards associated with different cooking tasks. Detailed specifications of these layouts, including ingredients and reward schemes, can be found our released code repository.
Cramped Room
Coardination Ring
Asymmetric Advanlage
Soup Coordination
Pot
Senving
Dish
Onion deeperser
Tomato
Cooking
Reward
208
70 2131
Figure 8: The four Overcooked experiment layouts used in our study (from left to right): Cramped Room, Coordination Ring, Asymmetric Advantage, and Soup Coordination. The game mechanics involve two players collaborating to prepare and serve dishes, like soups made of onions or tomatoes. Effective teamwork is reflected in the successful delivery of multiple orders. It is noteworthy that the Marshmallow Experiment layout differs from the others in terms of cooking time and reward settings.
15

---

C Implementation details
In our study, we rigorously implemented MTP within the CBPR framework and ensured that all baselines (BCP, FCP, and SP) adhered to unified methodology. This approach utilized the Proximal Policy Optimization (PPO) algorithm, widely acclaimed reinforcement learning technique Schulman al. [2017]. under standardized set of parameters (refer to Table 2). The adoption of PPO was motivated by its balance between sample efficiency and simplicity, making it popular choice in recent multi-agent learning research Yu al. [2022]. To optimize the learning process and mitigate often challenging exploration in the environment, we incorporated tailored reward shaping parameters as delineated in Table 3. This strategy aligns with the established practices in reinforcement learning that emphasize the importance of structured rewards in complex environments Gupta al. [2022]. Additionally, our empirical analyses revealed distinct performance advantage of feature- based observation models the image-based ones, leading their adoption across all agents. The entire training process was facilitated by the computational prowess of an NVIDIA 3080 GPU.
Table 2: PPO hyperparameters for MTP, BCP, FCP and SP agents. Lambda is used in generalized advantage estimation (GAE) to calculate advantage function. Reward shaping parameters in Table gradually anneals to zero over Reward shaping horizons.
Parameter
Value
Learning rate
5e-4
Entropy coefficient
0.01
Epsilon
0.05
Gamma
0.99
Lambda
0.95
Batch size
4096
Clipping
0.05
Hidden dim of actor and critic
128
Reward shaping horizons
0.5 * total timesteps
Table 3: Reward shaping parameters for PPO.
Action
Reward
Place pot
3
Dish pickup
3
Soup pickup
C.1
Collaborative Bayesian Policy Reuse (CBPR)
The CBPR's offline phase is multi-faceted process encompassing meta-task modeling, MTP, and performance modeling.
Initial efforts involved the manual definition of rule-based policies for each layout (Table 4), step inspired by the scripted policies detailed in Yu et al. [2023].
This was followed by the training MTP agents म E 11, which were systematically paired with rule-based agents to facilitate robust policy development. The training phase, as illustrated in Figure 9, was underpinned by a commitment to capturing diverse range strategic interactions. Subsequently, developed meta-task models T. leveraging two-layer feed-forward neural network. This network, initialized orthogonally and optimized at a learning rate 1e-3, was instrumental in deciphering the nuanced mappings from observations actions.
In the final stage, performance models were crafted by pairing each MTP agent æ with rule-based meta-tasks across 50 episodes, adopting Gaussian distribution approach to model episodic rewards.
16

---

Table 4: Predefined rule-based meta-taske
Layouts
Meta-tasks
Cramped Room
Place onion in pot
2. Deliver soup
Place onion and deliver soup
4. Others
Coordination Ring
Place onion in pot
2. Deliver soup
Place onion and deliver soup
4. Others
Asymmetric Advantage
Place onion in pot
2. Deliver soup
Place onion and deliver soup
4. Others
Soup Coordination
1. Place tomato in pot
2. Deliver soup
3. Mixed order
4. Others
Room
Coortination
Asymmetiee
Courdnatio
200
200
0.00
0.25
0.35
5.75
1.30
0.05
0.25
0.55
5.75
1.30
0.00
0.23
0.55
5.75
1.00
0.00
0.22
0.30
2.75
1.00
125
2800,
delver
28000
plice, wwr.tttroooo......
mibal order
Figure 9: Training curves of meta-task playing (MTP) agents over five random seeds. The shaded area denotes the standard deviation. Noticing that the reward should not be directly compared each other because agents vary in the partners they train with.
C.2 Baselines
C.2.1 Behavior Cloning (BC) and Behavioral Cloning Play (BCP) Carroll et al. [2019]
The BC models were trained using human-human trajectory data from Carroll al. [2019]. This process, partitioning 85% of data for training and 15% for validation, aligns with the standard practices in supervised learning. The neural network, characterized by layers with hidden size of 64 and an orthogonal initialization, was optimized for performance with a learning rate of le-4 and an Adam epsilon of 1e-8. Each model underwent a rigorous 120-epoch training regimen across four layouts and five seeds, reflecting a commitment to robustness and generalizability in agent training. The BCP agents, trained in tandem with BC partners, represent a novel amalgamation cloning and playing strategies, with training curves depicted in Figure 10.
C.2.2 Self-Play (SP) and Fictitious Co-Play (FCP) Strouse et al. [2021]
The training of FCP agents, utilizing pool size of 36 in the initial stage, was strategic choice to ensure diverse range of policy interactions. This diversity was further augmented by selecting five seeds from the first stage of FCP training for SP.
The second stage of training. involving a prolonged and intensive regimen over 50,000 episodes (amounting to 3e7 timesteps), was designed to refine and solidify the agents' strategies. Such extensive training critical in environments characterized by high complexity and variability, as allows agents to encounter and adapt wide array of scenarios. This comprehensive approach to training is evident the detailed training curves presented Figures 11 and 12, which provide insights into the progression and refinement of agent strategies over time.
17

---

cramped room
asymmetric advantages
Mean episode reward
600
coordination ring
soup_cos coordination
0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Environment steps
1e6
Figure 10: Training curves of BCP agents over five random seeds. The shaded area denotes the standard deviation. Noticing that the reward should not be directly compared to each other because the difficulty of the task varies with different game layouts.
1000
Mean episode reward
800
pramped_ room
600
asymmetric advantages
coordination _ring
400
soup_ coordination
200
0.0
0.5
1.0
2.0
2.5
Environment steps
167
Figure 11: Training curves of FCP over five random seeds. The shaded area denotes the standard deviation. Noticing that the reward should not be directly compared to each other because the difficulty of the task varies with different game layouts.
Additional results
D.1
Collaborating with rule-based agents with various policy switching frequencies
In this section, delve deeper into the dynamics of collaboration with rule-based agents under different policy switching frequencies. We present a series of additional experiments to complement the findings discussed in Subsection 4.1. These experiments critical in understanding how frequent policy shifts impact the overall performance and coordination in multi-agent environments.
Figure 13 illustrates the comparative performance when rule-based agents switch policies every episodes. Notably, the frequent policy changes introduce unique set of challenges and opportunities for adaptation, as evidenced by the performance fluctuations across 50 continuous episodes. The standard error shaded areas, based on five random seeds, highlight the variability performance under these conditions.
Similarly, Figures 14 and offer insights into the performance impacts when the policy switching occurs every 200 and 100 timesteps, respectively. These results are pivotal in understanding the optimal frequency of policy shifts to achieve efficient collaboration without overwhelming the learning agents with too frequent changes.
18

---

1400
cramped room
1200
asymmetric _atarfiigeddaalaeeessseee
Mean episode reward
coordination_iig
1000
soup_ coattiaattooooon
800
600
400
200
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Environment steps
1e6
Figure 12: Training curves of self-play agents over five random seeds. The shaded area denotes the standard deviation. Noticing that the reward should not be directly compared to each other because the difficulty of the task varies with different game layouts.
Crampod
Ring
Nommetrrshdiinttge
Soup
200
500
CBP9
8DP
PCP
Figure 13: Comparative performance analysis against baselines when rule-based agents swith policies every 2 episodes. All agents were evaluated over 50 continuous episodes. The shaded areas denote standard errors over five random seeds.
Mea mases
Cramped Room
Coordinet on Ring
Arymmetr Achenttages
Soup Coordiaation
300
CBPR
Figure 14: Comparative performance analysis against baselines when rule-based agents swith policies every 200 timesteps. All agents were evaluated over 50 continuous episodes. The shaded areas denote standard errors over five random seeds.
Cramped Room
Coo on on Ring
Asymmetr www.mnettrc..dcccaeeeee Advantages
Soup Coordination
1000
Mos eptoce
820
Map Aance
Es soo
Epoodoo
Figure 15: Comparative performance analysis against baselines when rule-based agents swith policies every 100 timesteps. All agents were evaluated over 50 continuous episodes. The shaded areas denote standard errors over five random seeds.
D.2 Ablation study: collaborating with partners of diverse skill levels
In the following ablation study, we focus on the aspect of collaborating with partners exhibiting diverse skill levels. This study is vital to assess how agents adapt to varying competencies within team setting. The results of this study are shown in Figures and where we examine different weights and behavioral queue lengths.
19

---

Mean episode reward
150
p40.5
8403
300
Mean episode reward
600
200
100
400
50
Mean
100
Low
Medium
High
Low
Medium
High
Low
Medium
High
(a) Coord. Ring
(b) Asymm. Adv.
(c) Soup Coord.
Figure 16: Episodic reward by using different weight rho of inter-episodic belief in other three
layouts. All agents were evaluated over 50 episodes and error bars denote 95% confidence intervals.
190
230
Mean epitode reward
600
103
Mean aptodd
120
400
100
60
Medium
High
LOw
Mecium
High
Medium
High
(a) Coord. Ring
(b) Asymm. Adv.
(c) Soup Coord.
Figure 17: Episodic reward by using different length human behavior queue in other three layouts.
All agents evaluated over 50 episodes and error bars denote 95% confidence intervals.
In Figure 16, explore the episodic rewards obtained by varying the weight p of the inter-episodic belief across three different layouts Coordination Ring, Asymmetric Advantage, and Soup Coordi- nation. Each layout presents unique challenge and thus allows us evaluate the adaptability of the agents to different team dynamics over 50 episodes. The 95% confidence intervals depicted here underscore the consistency of our findings.
Additionally, Figure 17 presents the effects of altering the length I of the human behavior queue. This modification helps us understand how the memory of past interactions influences current decision- making processes in different environmental layouts. The episodic rewards over 50 episodes, along with the bars, provide clear depiction of the performance trends under these varied conditions.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We summarized our contributions at the end of the introduction. Please see
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of the work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

We provided the theoretical result in Section 3.3 and complete proof in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have placed all the code for our algorithms and experiments in an anony- mous repository (https://github.com catttuub.oo//Aaa. /ll// Wangha to facilitate the reproduction our work. In addition, we provide the implementation details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have placed all the code for algorithms and experiments in anony- mous repository (attss//github.com///ke WanghaomingCCBPR) to facilitate the reproduction of our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/ public/guides/CCn otweeeee/DDeeSubiiiiiooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cubbicc/uuides httegguuiiieee/eeuuuiieeeeeeeeee www.SSummisiioPPol...
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: we provide the implementation details (hyperparameters included) in Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
See section 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the type of compute workers in section C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttps:/neuurre nwwrips.....p// http:/uuli/Bthhssaaiittae inneii
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our models do not have this kind risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide MIT License of our released code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We provide the screenshots of the game and details about compensation in user study section.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "uqyPYaTQ2y", "reviewer": "Reviewer_3rF3", "summary": "This paper introduces Collaborative Bayesian Policy Reuse (CBPR), a framework that addresses the challenge of collaborating with non-stationary human behavior by adaptively selecting optimal collaborative policies based on the current meta-task. CBPR identifies meta-tasks underlying human decision-making and trains specialized meta-task playing (MTP) agents to enhance collaboration. Evaluations in the Overcooked simulator demonstrate CBPR's superior performance compared to existing baselines.", "questions": "1. How does the CBPR framework handle environments where human behaviors and tasks are not only non-stationary but also highly unpredictable or undefined?\n\n2. The author used 5 different random seeds, but from the experimentation sections, looks like the variance of the run aren't that much, does this mean that the learning process is more deterministic?\n\n3. How would the performance be like, if the episode has longer horizon?\n\n4. How would the algorithm scales with LLMs or large model?", "limitations": "Yes, the author discussed the limitation and societal impact.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Overall, I found his paper seems to be interesting. It introduces the CBPR framework that addresses collaborating with non-stationary humans by matching meta-tasks rather than directly modeling complex human dynamics. The presentation of the paper could be improved. For instance, I found figure 2 to be confusing, I encourage the author could simply the diagram and improve the presentation style.\n\nThe strengths are:\n\n1. The paper provides strong theoretical proofs and extensive empirical results across various conditions that compellingly validate the effectiveness of CBPR over baseline approaches.\n\n2. The paper presents an approach seem to be different from the main stream: it avoids modeling human behavior but instead \"focusing on constructing meta-tasks that underpin human decision making\"", "weaknesses": "1. The effectiveness of the CBPR framework relies heavily on accurately predefined meta-tasks, which might limit its application in environments where meta-tasks are not clearly defined or are too complex to categorize effectively. \n\n2. While the authors extensively evaluate CBPR in the Overcooked environment across various conditions, the paper lacks experiments in other domains beyond gaming. Overcooked, being a relatively simplistic and controlled environment, may not fully capture the complexities and challenges of human-AI collaboration in real-world applications such as autonomous driving, robotics, or complex decision-making systems. Testing CBPR's performance in more diverse and realistic domains would strengthen the paper's claims and demonstrate its broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the time and effort reviewer 3rF3 has invested in reviewing our paper. For the Weakness1, Weakness2, Q1 and Q3 you mentioned, we have added experiments to support our insight. We apologize for the confusion caused by Figure 2. We have simplified it in the rebuttal PDF to make the framework of CBPR more clear. We provide detailed explanations for the other questions and hope this can address your concerns.\n\n> Weakness1 and Q1: The difficulty of predefined meta-tasks and the unpredictability of human behavior.\n\nFor many tasks, defining meta-tasks based on human knowledge is not so difficult. For instance, in the context of autonomous driving, proactive lane changing and forward driving can serve as meta-tasks. In robotic scenarios, navigation and robotic arm grasping can also be considered as meta-tasks. To address leftover tasks not included in predefined meta tasks, we introduce a meta-task category as ”others” (Fig 1, bottom-Left). \n\nFor the undefined and unpredictable tasks, they can, to some extent, be addressed by the “others” category by learning a general model for them in our current framework. To demonstrate this, we add an ablation study based on section 4.2 to examine the impact of the number of meta-tasks undefined for the Soup Coord. scenario, in which we defined 4 more meta-task (\\textit{place_onion_and_deliver_soup}, \\textit{place_tomato_and_deliver_soup}, \\textit{pickup tomato and place mix}, \\textit{pickup_ingredient_and_place_mix}). The results show that without \"others\" category, the performance deteriorates significantly, while the performances degrade relatively gracefully with less meta-tasks defined and more included in \"others\" category.\n\n| | 7 predefined+\"others\" | 5 predefined+\"others\" | 3 predefined+\"others\" (original paper) | 3 predefined w/o \"others\" |\n| :---- | :----: | :----: | :----: | :----: |\n| High | 620.3(193.3) | 600.7 (234.0) | 647.7(159.3) | 622.8(205.8) |\n| Medium | 757.8(100.3) | 735.8(98.7) | 717.1(148.1) | 607.3(278.5) |\n| Low | 689.8(43.9) | 680.52(51.6) | 668.9(49.0) | 40.0 (59.1) |\n\nIn some extremely complex tasks, it may be difficult to define what a meta-task is, which will be the direction of our future research. We will explore approaches to decompose tasks into finer-grained pieces so that definitive meta-tasks can appear and become definable. We will add these discussions to limitations and future work in our revised manuscript.\n\n\n > Weakness 2: Lacking experiments in other in real-world applications.\n\nFollowing your recommendation, we conducted additional experiments using the CARLA autonomous driving simulator. To acquire driving behavior data, we employed human driving models from the SUMO traffic simulator, selecting random origins and destinations for each vehicle to collect driving data. This data was used to train a Behavioral Cloning (BC) agent (Agent 1) to autonomously control the vehicle. For the Control-Based Priority Routing (CBPR) agent, we designed two meta-tasks: the \"line up\" meta-task, developed by selecting vehicle queuing data and training a second BC agent (Agent 2), and the \"others\" meta-task, utilizing BC Agent 1.\n\nWe evaluated both BC and CBPR agents in a 'Queue for a Left Turn' scenario, employing a rule-based simulated human. This simulated human intervention ensured the vehicle remained in line whenever the AI attempted to change lanes. Initially, as vehicles began to queue, the AI chose to turn right and overtake. At this point, human would redirect the vehicle to maintain its position in the queue through a left turn. After a single instance of human intervention, the CBPR agent switched to the \"line up\" meta-task and continued to wait in the queue, whereas the BC agent repeatedly attempted to overtake by turning right, necessitating continuous human intervention. Detailed results are shown in Figure 4 of the rebuttal material.\n\n> Q2:  The learning process is more deterministic?\n\nIn Appendix D, we provide the training curve of all agents (Fig 9-12) over five random seeds, which indicate the learning process is not that deterministic. The shaded area in Fig 3 is indicated by standard error, which is the standard deviation divided by the square root of the the number of seeds to represent the randomness of multiple experiments. To make the Fig 3 easier to understand, we scaled down the errors in original paper. We have replotted the Fig 3 using the standard deviation as shown in Fig 1 of rebuttal PDF file.\n\n> Q3: CBPR performance in the episode with longer horizon?\n\nThanks for this interesting question. Following your suggestion, we increased the original horizon setting from 600 to 3000, and then had the CBPR agents cooperate with the policy-switching agents, recording the cooperative rewards under five different frequencies of policy switching. The experimental results are shown in Fig 2 of rebuttal PDF, which implies that CBPR remains highly efficient in long-horizon tasks.\n\n> Q4: CBPR scales with LLMs?\n\nThank you for your question, which has inspired us to undertake some work combining large models with CBPR in the future. We consider CBPR scaling with LLMs or large models in following two aspects:\n\\textbf{Meta-task identification} \\ In \\textit{Overcooked} simulator, meta-tasks are easy to defined. However, in environments where meta-tasks are not clearly defined or are too complex to categorize, the context comprehension ability of large models can deconstruct multiple meta-tasks based on behavior data demonstrated by humans. \n\\textbf{Acceleration of belief $\\beta_k(\\tau)$ convergence} \\ In an environment where two meta-tasks do not have clear boundaries. LLMs or large models can accelerate the convergence of $\\beta_k(\\tau)$ by adding an additional factor $\\lambda$ to the numerator of Eq 1 or Eq 3. $\\lambda$ can be determined by self-reflection [1] of LLM. \n\n[1]Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y5N3cBkxXa", "reviewer": "Reviewer_vD8B", "summary": "This work explores how to address the challenges of non-stationary human behavior in human-AI collaboration. The authors propose a Bayesian framework that adaptively selects optimal models during training episodes to capture the underlying consistent human behavior in solving meta-tasks. Theoretical analysis shows that their proposed method can converge to optimal solutions even when the human‘s policy is non-stationary. Both simulations and real human-subject evaluations show  that the proposed method outperforms other baselines in a cooperative game Overcooked.", "questions": "1. In line 113, $\\sigma$ is defined as “any signal aiding cooperation, such as reward or interaction trajectory”. But since reward and trajectory could be derived from the given task $\\tau$ and policy $\\pi$, why is there uncertainty? Is the problem setup a stochastic, or is the policy stochastic? \n2. In Equation (1), the belief is updated in Bayes’ rule, so the task with higher expected utility (if $\\sigma$ represents reward) given current policy will be assigned a higher belief probability. Is this under the assumption that the game is cooperative and both human and AI decision-makers are acting to maximize expected utility? If not, why is such belief updating reasonable? \n3. The intra-episode belief should be more stable and accurate than inner-episode belief, and contain different messages about human behavior. But from ablation studies in Section 4.2, $\\rho$ seems make no difference in mean reward, why is this the case? \n4. In simulation results (Figure 4), SP outperforms CBPR in asymmetric advantages, and FCP outperforms CBPR in soup coordination. Could the author discuss more about this? Similar results appear in the human experiment (Figure 6) as well. Detailed discussions about results should be added to the paper. \n5. In the meta-task of Overcooked, if the player is holding an onion, then their only task is to put it in a pot; if holding a pot, the only task is to deliver it. Why is it necessary to identify the task?", "limitations": "No further limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivation is realistic, and the introduction section is very clear. The proposed solution is solid, with both theoretical analysis and empirical evaluations. Sufficient details are provided for reproducibility. The idea to include both intra-episode belief and inter-episode belief is innovative.", "weaknesses": "The problem formulation and notations are unclear (see more discussion in Questions 1 and 2). Empirical results seem to suggest that the proposed method only makes minor improvements, and ablation studies imply that $\\rho$ has almost no impact (which I believe is one of the innovations of this work). The authors do not thoroughly discuss these results. Instead, they directly conclude that the proposed method is effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really thank you for your valuable comments on improving our work. We sincerely hope the reviewer can raise the assessment score of our paper if the following responses have successfully addressed the concerns.\n\n> Q1: The uncertainty of σ?\n\nThe human-AI collaboration problem we address is inherently non-stationary. We utilized a rule-based agent with\nnoise (i.e. τ ) a stochastic policy (π), thus the reward is uncertain. Signal model P (σ | τ, π), which is also known\nas observation model represents the distribution of observed signal σ when AI performs the meta-task τ with policy\nπ in vanilla BPR. Note that in CBPR, we choose the episodic reward as the signal σ of signal model. During offline\nstage of CBPR, we construct each signal model (i.e. performance model) by fitting a Gaussian distribution given a\nstochastic AI policy π and a noisy rule-based agent τ (see l124-128 of the main text and l533-534 of the Appendix).\nTo further clarify the definition of σ, we will add additional explanation of signal model in the revision.\n\n> Q2: The assumption of Equation 1? Why the belief updating reasonable?\n\nThanks for the question. We would like to clarify that only the AI decision makers are acting to maximize the\nexpected utility, whereas humans may exhibit varying levels of initiative to cooperate in human-AI collaboration.\nThe rationale of the updating the belief in Eq 1 adheres to the vanilla BPR, which reuses prior experience from the\nclass of tasks to cope with unknown task variations. Note that Following works primarily focused on extending BPR\nin competitive tasks (lines 93-95 in original paper). In this work, we firstly extend BPR in human-AI collaboration\nscenario. Human-AI collaboration is essentially an incomplete information game since human players may exhibit\nvarying levels of initiative to cooperate and AI players do not have complete knowledge of payoff functions of human\nplayers. On the contrary, AI players need to adapt to the non-stationarity of human players, and are acting to\nmaximize expected utility. \n\n> Q3: ρ seems make no difference in mean reward, why is this the case?\n\nWe examine the setting of rho in the evaluation of our experiments. According to Eq 4, when ρ=0.1 the weight of the inter-episode belief decays to 0.01 after just 2 time steps, while when ρ=0.9,\nthe inter-episode belief nearly decays to 0.01 within 40 time steps. Therefore, for a game with 600 time steps in\nour evaluation, only the first 40 steps of integrated belief will be influenced by the inter-episode belief, leading to\nthe observation that ρ make no difference in mean reward. In other words, to demonstrate the effectiveness of ρ,\nthe inter-episode belief should decay to below 0.01 after a greater number of time steps in an episode. To further\nevaluate this new setting, we set ρ = 0.99 and ρ = 0.95. Therefore, we have conducted more ablation study of ρ\n(Fig 3 in rebuttal PDF), the result indicates that in simple layout (i.e. Cramped Rm.) since the partner’s policy\nis simple, variations in ρ have little impact on the reward. However, in complex scenarios (i.e., Soup Coord.),\nadjusting rho can enhance cooperative performance to a certain extent. \n\n> Q4: Results of (Figure 4) and Figure 6.\n\nWe provide a more detailed analysis of our results in Fig 4 as follows and include them in our updated paper:\n\n**The inherent advantage of SP and FCP agents.** Partners with low, medium and high skill levels are\nrepresented by checkpoints (essentially are SP agents) at initial, the middle and the end of training of FCP. There-\nfore, SP and FCP agents have an inherent advantage in the evaluation presented in Figure 4. Despite this, CBPR\nperforms better when faced with partners of a lower skill level. When collaborating with real humans (Fig 6), FCP\nand SP no longer have such advantage. This leads to almost all FCP and some SP performing well against agents\nof various skill levels (Fig 4), but falling short when facing human players (Fig 6).\n\n**The cooperative advantage of CBPR in non-separated layouts.** In separated layouts (i.e. Asymm.\nAdv. and Soup Coord.), agents can usually complete tasks independently without considering the hindrance of\nthe other partner’s moves to themselves. However, players’ own position (e.g. stand still in front of the serving\nareas) can obstruct their partners from completing the task in the non-separated layouts. Therefore, non-separated\nlayouts require more cooperation between players compared to separated layouts. As shown in Fig 4, CBPR’s better\nperformance in Cramped Rm. and Coord. Ring suggests its advantage in collaborative tasks.\n\n**The double-edged sword of SP’s simple policy.** In Asymm. Adv., SP agent exhibits outstanding perfor-\nmance when it cooperates with the agent of high skill level (Fig 4c). We replayed the game and found that the SP\nagent learned the simplest and most effective policy (i.e. in the right room, just pick an onion from onion dispenser\nand then place it in a pot within the shortest path) during training. However, other agents exhibit some\nsuperfluous actions due to their own complexity. However, when SP cooperates with the agent of low skill level, it\nperformed poorly because the SP agent on the right only learned the simplest policy (putting onions in the pot), and\nwhen the agent with low skill level on the left does not deliver the cooked soup, SP will wait in place rather than\ndeliver the cooked soup. Thus, cooperating with SP agents results in low performance (Fig 4d).\n\n> Q5: Why is it necessary to identify the task?\n\nWe would like to clarify the process of ”identify” meta-task in the online stage of CBPR. Identifying task is to\ncalculate the probabilities to generate a queue Q of human behaviors given each meta-task τ . During the online\ncollaborating, we are unable to ascertain human cooperative strategies and initiative; however, human behaviors\nreveal the meta-tasks they are currently undertaking. Therefore, it is necessary to identify the task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNxAm40bZL", "reviewer": "Reviewer_QDdz", "summary": "The authors proposes a Bayesian policy reuse-based framework referred to as CBPR, which allows for a collaborative artificial agent to adaptively select optimal collaborative policies from multiple policy networks. They did so by extending intra-episode belief to collaborative scenarios and incorporating this extension to the vanilla BPR. Their novel framework bypasses the need to model human behavior, and is suitable for human-AI collaborative tasks. The authors also provide theoretical proofs that their framework converges in cooperation with non-stationary humans, and that it can also establish the optimal collaboration policy. Finally, the authors evaluated this CBPR framework with Overcooked, a collaborative cooking game. Through experiments with this game, they demonstrated that CBPR outperforms baselines, showing that it is more effective to design multiple agents rather than a single agent.", "questions": "In theorem 1, the statement first mentions a stationary human policy as a given. At the end, the theorem states that convergence is also guaranteed for a non-stationary policy. It's not clear why the theorem specifically starts with non-stationary policy if it could be applied to both cases. Moreover, the proof does not clarify why the convergence guarantee for a stationary policy also applies to a non-stationary policy. Additional clarification to point out this gap would be helpful for readability.", "limitations": "The authors pointed out that their meta-tasks are manually-designed, rule-based policies, meaning that this method would not scale in real-world applications such as power grid dispatching and autonomous driving. While CBPR offers a strategy to model meta-tasks for human-AI collaboration, the challenge of summarizing domain experts' meta-tasks still remains.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The authors' incorporation of intra-episode belief into traditional BPR to develop CBPR was smart and original. According to their experiment with Overcooked, this novel combination of existing concepts made significant advancements in the field of human-AI collaboration. The authors went above and beyond to present the technical soundness of their work by proving their framework's convergence and optimality. Additionally, the paper is well-structured and clearly written, with meticulous attention to the setup of their experiments, enhancing the overall clarity and impact of their findings.", "weaknesses": "I couldn't identify any area of weakness for this manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the time and effort that reviewer QDdz has invested in reviewing our paper, and we appreciate the recognition of the main advantages of our method. Additionally, thank you for your insightful comments regarding Theorem 1 and its proof. \n\n**Clarification on the Relationship Between Stationary and Non-Stationary Policies**\n\nThank you for your thoughtful and perceptive question. Your question highlights an important aspect of our theorem formulation, and we acknowledge the need for clearer explanation in our original manuscript. Our approach is developed on the key insight that, despite the inherent\ndiversity in human behaviors, the underlying meta-tasks within specific collaborative\ncontexts are quite similar. This observation facilitates the transformation of a non-stationary cooperative problem into one involving a series of *stationary* policies addressing distinct meta-tasks.\n\nTheorem 1 outlines this transformation by situating the collaborative process between humans and AI within a Non-Stationary MDP framework. To manage the inherent non-stationarity, a methodical decomposition of the broader non-stationary decision-making process into *a series of temporally contiguous stationary segments, which represent the stationary process of meta-task accomplishing*.\n\nInitially, Theorem 1 defines trajectories collected from interactions with what we referred to as a 'stationary human policy.' To clarify, this term was intended to denote a stationary policy that humans use to accomplish a specific meta-task. It was not meant to imply that our theorem begins by establishing convergence for a stationary human policy and then extends this to a non-stationary human policy. We acknowledge that the term 'stationary human policy' may have led to some confusion. Thank you for your insightful query. In response, we have revised the terminology to 'stationary meta-task performing policy' in our manuscript to improve clarity and readability.\n\n\n**Clarification of Convergence in Stationary and Non-Stationary Contexts**\n\nYour question regarding convergence is astute. We believe our CBPR framework is innovative, offering a straightforward approach to achieving convergence in non-stationary human-AI cooperative settings.\n\nTo ensure convergence while interacting with a non-stationary human, a cooperative policy must: 1) recognize changes in human behavior policies, and 2) adapt a consistent response policy that aligns with these changes.\n\nIn the proof of Theorem 1, we first establish the convergence properties of the Bayesian update mechanism. This ensures that, with the convergence provided by Bayesian updates, the belief on the current meta-task will consistently converge. This convergence is crucial for the CBPR framework to effectively track changes in human behavior. Subsequently, we demonstrate that when utilizing Bayesian updates, CBPR algorithms consistently converge to a specific meta-task cooperative policy, provided that the meta-task being performed by the human remains unchanged until this fixed response policy is achieved.\n\nA key advantage of our CBPR method is the use of Bayesian updates to dynamically update beliefs on human behavior. This approach is effective for both stationary and non-stationary policies. The belief about the meta-task will consistently converge to match the behavior that the human is exhibiting. The primary distinction between stationary and non-stationary human policies is that the meta-task in the former remains constant, while in the latter, it evolves over time.\n\n\nWe hope these clarifications address your concerns and enhance the readability and comprehension of our work. We thank you again for your valuable input, which has significantly contributed to improving our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the time and effort reviewers have invested in reviewing our work. We have provided detailed explanations and clarifications to address your concerns regarding problem definition, experiments and discussions. 

During this stage, we have supplemented the experiments to address reviewers' concerns regarding (1) a clarification of the meta-task. (2) CBPR's application in complex scenarios such as autonomous driving. (3) the performance of CBPR in long-horizon tasks. (4) the impact of inter-episode beliefs on the experimental results. Specific details can be found in the responses below the uploaded PDF file. 

For other questions, we have provided point-by-point responses and have updated our paper accordingly.

We believe that incorporating your feedbacks has greatly strengthened our paper, and we hope you will agree with our improvements. We express our sincere gratitude to all the reviewers for their valuable comments on our manuscript again.

Meta-review: The reviewers agreed that the paper addresses an important problem of effective human-AI collaboration when humans exhibit non-stationary dynamics, the proposed framework is novel, and the results would be of interest to the community. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)