Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators

Authors: haoming  wang
zhaoming  tian
yunpeng  song
xiangliang  zhang
zhongmin  cai
  t
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:30:26

Abstract: In human-AI collaborative tasks, the distribution of human behavior, influenced by mental models, is non-stationary, manifesting in various levels of initiative and different collaborative strategies. A significant challenge in human-AI collaboration is determining how to collaborate effectively with humans exhibiting non-stationary dynamics. Current collaborative agents involve initially running self-play (SP) multiple times to build a policy pool, followed by training the final adaptive policy against this pool. These agents themselves are a single policy network, which is $\textbf{insufficient for handling non-stationary human dynamics}$. We discern that despite the inherent diversity in human behaviors, the $\textbf{underlying meta-tasks within specific collaborative contexts tend to be strikingly similar}$. Accordingly, we propose a $\textbf{C}$ollaborative $\textbf{B}$ayesian $\textbf{P}$olicy $\textbf{R}$euse ($\textbf{CBPR}$), a novel Bayesian-based framework that $\textbf{adaptively selects optimal collaborative policies matching the current meta-task from multiple policy networks}$ instead of just selecting actions relying on a single policy network. We provide theoretical guarantees for CBPR's rapid convergence to the optimal policy once human partners alter their policies. This framework shifts from directly modeling human behavior to identifying various meta-tasks that support human decision-making and training meta-task playing (MTP) agents tailored to enhance collaboration. Our method undergoes rigorous testing in a well-recognized collaborative cooking simulator, $\textit{Overcooked}$. Both empirical results and user studies demonstrate CBPR's superior competitiveness compared to existing baselines.

Main paper: 
 introduction: 
 An ongoing challenge in artificial intelligence (AI) involves training agents capable of effective collaboration with humans Klien et al. [2004] , Bard et al. [2020] , Dafoe et al. [2020] . Unlike typical AI-only multi-agent collaboration, human-AI collaborative scenarios such as two-player cooking games, autonomous driving, and managing power grid stability incorporates a non-stationary component, humans Jagerman et al. [2019] , Chandak et al. [2020] , Chandak [2022] . As humans may vary in their level of initiative, alter their collaboration strategies, or sometimes even do not collaborate at all. This variability suggests that for cooperative agents, the probability distribution P (A|s t ) of a human action A given an environmental state s t changes over time, reflecting different Figure 1 : Left: The drawbacks of current collaborative agents, which train a stationary policy to manage the non-stationary dynamics of human collaborators but fail to determine the specific collaborative policies executed by humans. Right: Our approach focuses on identifying the meta-tasks underlying human decision-making and trains collaborators to match these meta-tasks in a one-to-one manner. This strategy enables effective ad-hoc collaboration with non-stationary humans. mental states. Such non-stationarity poses a significant challenge in training collaborative agents, as it requires strategies that can adapt to the unpredictable nature of human behavior, which departs from the stable action-outcome associations expected in scenarios dominated by AI.Recent works mainly develop collaborative agents through two workflows: (1) explicitly model human behavior by using real human trajectories Carroll et al. [2019] , and then train a collaborator by teaming up with human models. (2) train Self-Play (SP) agents to form a policy pool (a diverse set of AI agents assumed to encompass all potential human policies) and then train a collaborator pairing with policies in the policy pool Strouse et al. [2021] , Yu et al. [2023] , Zhao et al. [2023] . However, despite their ability to achieve commendable performance by amassing extensive human data collection or SP agent training, these collaborators share a common fundamental flaw: they are essentially policy networks following a stationary distribution, thus making it difficult to cope with non-stationary human dynamics.In this work, we propose Collaborative Bayesian Policy Reuse (CBPR), which reuses multiple stationary policies tailored to meta-tasks within a specific collaborative scenario. CBPR builds upon Bayesian Policy Reuse (BPR) Rosman et al. [2016] , Chen et al. [2022] , extending its application to human-AI collaborative tasks with theoretical guarantees. CBPR avoids modeling the non-stationary dynamics of human collaborators, focusing instead on heuristically modeling available meta-tasks within defined collaborative contexts. For example, in the multi-player cooking game Overcooked, meta-tasks include {place onions in pot, deliver soup, place onions in pot & deliver soup, others} (Figure 1 ) are available. Noticing that for a complex human-AI collaborative task, all of the undefined meta-tasks are categorized as "others," we subsequently train stationary meta-task-playing (MTP) collaborators using reinforcement learning (RL) to precisely match meta-task models on a one-to-one basis. During collaboration, CBPR identifies the meta-task being performed by the human partner based on recent actions, subsequently adapting the optimal MTP collaborator for use.We evaluate CBPR in a fully-observable two-player common-payoff collaborative cooking simulator based on the game Overcooked Carroll et al. [2019] , which has recently been proposed as a coordination challenge for AI Carroll et al. [2019] , McKee et al. [2022] , Wang et al. [2020] , Wu et al. [2021] , Knott et al. [2021] . State-of-the-art performance of this game was achieved in Carroll et al. [2019] , Strouse et al. [2021] , Yu et al. [2023] via training stationary cooperation policy. Both simulated experiments and user studies show that the proposed CBPR agent can collaborate effectively with non-stationary agents and real humans. The novel contributions of this paper can be summarized as follows:1. We introduced a human-AI collaboration framework, CBPR, which addresses the challenge of modeling non-stationary human dynamics. This framework identifies the meta-tasks performed by human partners and reuses the optimal collaborative policy.2. Theoretically, based on the Non-Stationary Markov Decision Process (NS-MDP), we provide theorems on Collaboration Convergence and Collaboration Optimality to support CBPR's convergence to the optimal collaborative policy over time in human-AI collaboration.3. Empirically, we demonstrated CBPR's capability to collaborate effectively with nonstationary agents who frequently switch strategies, agents with various collaboration skills, and real humans.2 Related Work 
human-ai collaboration: 
 Training agents to collaborate with humans has been extensively studied. Recent research can be categorized into two groups based on whether human data is used during training. BCP Carroll et al. [2019] is trained by pairing with a supervised human model, while Boltzmann Policy Distribution (BPD) Laidlaw and Dragan [2022] updates its prior based on online human actions. These approaches require human data collection and are prone to distributional shifts. In contrast, another category focuses on achieving zero-shot coordination without extensive human data Hu et al. [2020] . These works (e.g., FCP Strouse et al. [2021] , Hidden-Utility Self-Play (HSP) Yu et al. [2023] , and Maximum Entropy Population-based Training (MEP) Zhao et al. [2023] ) train Self-Play (SP) agents to form a policy pool-a diverse set of AI agents assumed to encompass all potential human policies-and then train a collaborator to pair with policies in this pool. However, these collaborative agents remain single stationary models despite their diverse training partners.Our work represents a fundamental departure from previous studies by avoiding the need to model human behavior and instead focusing on constructing meta-tasks that underpin human decisionmaking. Furthermore, our CBPR framework does not restrict the construction of meta-tasks, which can be categorized into two streams: reliant on human data (e.g., behavior cloning) and independent of human data (e.g., rule-based methods).Policy reuse is a kind of transfer learning method that can greatly speed up reinforcement learning for a new task by using policies for relevant tasks. Initial methods like PRQL Fernández and Veloso [2013] and OPS-TL Li and Zhang [2018] , Li et al. [2018] integrated source policies with limitations in transfer efficiency. Subsequent approaches such as CAPS and CUP Zhang et al. [2022] improved policy selection and introduced more efficient algorithms without the need for extra training components.Bayesian policy reuse (BPR) Rosman et al. [2016] represents a specialized stream within policy reuse. Utilizing a Bayesian optimization approach, BPR efficiently computes posteriors for novel tasks. Extensions like BPR+ Hernandez-Leal et al. [2016a,b] and Bayes-Pepper Hernandez-Leal and Kaisers [2017] adapt BPR to multiagent scenarios, aligning tasks with opponent strategies and policies with optimal responses to these strategies. However, most BPR methodologies Rosman et al. [2016] , Hernandez-Leal et al. [2016a], Hernandez-Leal and Kaisers [2017] , Zheng et al. [2018 Zheng et al. [ , 2021]] , Chen et al. [2022] , Xie et al. [2022] primarily address multi-task problems or copy with competitive scenarios. Several studies, such as Zheng et al. [2018 Zheng et al. [ , 2021]] , investigated deep BPR+ in collaborative games.However, these approaches primarily rely on policy inference to adjust to the changing strategies of opponents (or partners), which may not be optimal for human-AI collaboration given the wide spectrum of potential human policies. To our knowledge, our research is pioneering in applying and tailoring Bayesian policy reuse-based algorithms specifically for the human-AI collaboration challenge.  5); and finally, (d) the AI collaborator executes actions according to the chosen policy.EQUATIONP k k k k k k k                     | | T Offline Stage3 Collaborative Bayesian Policy Reuse 
vanilla bayesian policy reuse: 
 Bayesian policy reuse is a general framework of transfer learning to cope with unknown tasks or frequently changing opponents. These classes of methods typically involve two phases: an offline learning phase and an online reusing phase. The workflow of a typical BPR can be summarized as follows: In the offline phase, it is presupposed that there exists a library of tasks T and a corresponding library of learned policies Π. Through conducting multiple simulations with varied policies across different tasks, a performance model P (U | T , Π) is derived, where U = Σ k i=0 r i is cumulative utility. This model works as a mapping operator, associating each task and policy with a distribution of a predefined utility measure, such as reward.During the online phase, BPR identifies the current task or opponent policy by maintaining a belief model β(•). This model is periodically updated based on observations, as defined by the observation model P (σ | τ, π), where σ represents any signal aiding cooperation, such as reward or interaction trajectory. Significantly, this update adheres to Bayes' rule as follows:EQUATIONWith this belief model, the BPR agent can select the optimal response policy by solving the following optimization problem:EQUATIONwhere Ū = max π∈Π τ ∈T β(τ )E[U | τ, π] represents the average performance of a single policy across all tasks. It's important to note that using Ū as the lower limit of the integral, this optimization problem essentially seeks the policy with the highest likelihood of achieving utility above the average.Offline stage Initially, we train meta-task processing (MTP) agents π ∈ Π using the Proximal Policy Optimization (PPO) algorithm by individually pairing them with meta-tasks within a specific collaborative context, as exemplified by tasks such as place onions in pot, deliver soup, place onions in pot & deliver soup, and others in the Overcooked collaboration benchmark. Meta-task models τ ∈ T are constructed through supervised learning, utilizing trajectories from either rule-based agents enhanced with noise or real humans performing the tasks. In this study, we employ the rule-based agents developed by Yu et al. [2023] . Subsequently, we construct the performance model P (U | T , Π) (i.e., observation model) by fitting a Gaussian distribution over the mean episodic return given a stochastic AI policy π and a noisy rule-based agent τ .In previous BPR-based algorithms, the belief was designed for measuring the similarity between different tasks or opponents in transfer learning. These algorithms update belief using a observation model P (σ | τ, π) which only considers the game result but overlooks opponent's behavior. This leads to a poor collaborative performance when humans switch policy in a long-episode game. In this study, we used intra-episode belief ξ t (τ ) at timestep t to measure the similarity between current meta-task τ and τ ′ in meta-task model library T . The intra-episode belief was firstly proposed in Chen et al. [2022] and we extend it to the human-AI collaborative scenario.Online policy reuse At the beginning of online policy reuse, the inter-episode belief β 0 (τ ) is initialized with a uniform distribution. For each episode, CBPR maintains a first-in-first-out (FIFO) human behavior queue Q of length l, which records the latest human behavior tuples (s i , a i ). The AI selects initial response MTP agents according to the inter-episode belief β 0 (τ ) (line 5 in Algorithm 1). CBPR collects human state-action pairs and updates the intra-episode belief ξ t (τ ):EQUATIONwhereP (Q | τ ) = exp( l i=0 log τ (ai|si)) τ ′ ∈T exp( l i=0 log τ ′ (ai|si)). Then the intra-episode belief and inter-episode belief are integrated:EQUATIONWhere ρ ∈ [0, 1] is a hyperparameter controlling the weight of the inter-episode and intra-episode beliefs. As the timestep t increases in a game with a long episode, the integrated belief ζ t (τ ) primarily depends on the intra-episode belief ξ t (τ ). The AI then uses the integrated belief ζ t (τ ) to select a policy to cooperate with the human at each timestep.π ⋆ t = arg max π∈Π U max Ū τ ∈T ζ t (τ )P U + | τ, π dU + (5)At the end of each episode, CBPR collects the episodic return u k and updates the inter-episode belief β k (τ ). To adapt to non-stationary human dynamics, we store human-AI trajectories in a replay buffer R of the current MTP agent and update its policy. The detailed pseudo-code for the policy reuse of CBPR is presented in Algorithm 1.The selection of cooperative policies (line 11 in the Algorithm 1) is crucial to the performance of CBPR in collaborating with humans. In this section, we propose theorems on the convergence and optimality of CBPR to support our viewpoint: CBPR will converge to the optimal cooperative strategy during the human-AI interaction process. We formulate collaborative process between humans and AI as a Non-Stationary MDP (NS-MDP) Chandak et al. [2020] . In this process, the non-stationarity, resulting from the dynamic nature of human policy, can be mitigated by decomposing the entire non-stationary decision process into several stationary ones. Each stationary MDP corresponds to a specific meta-task executed by the human. Specifically, for a given NS-MDP {M i } ∞ i=1 , the transition function integrates human actions as part of the environment itself, which can be denoted as P i : S ×A AI ×A hu → ∆(S). Within each stationary MDP M i , the human policy π hu,i : S → ∆(A) is assumed to be stationary, although it may exhibit variations across different stationary MDPs. Under this assumption, the CBPR agent could establish a convergent human-AI collaboration: THEOREM 1 (Collaboration Convergence of CBPR Agent). Let H i := {S j i , π hu,i (S j i ), R j } ∞ j=0 be a trajectory collected from a single stationary MDP M i within the overall NS-MDP {M i } ∞ i=1 under the human meta-task policy π hu,i . Denote D := {(i, H i ) : i ∈ [1, k]} as a random variable representing a set of trajectories observed prior to the most recently completed stationary MDP M k . Given D, the Algorithm 1 Online Policy Reuse of CBPR Input: Meta-task model library T , meta-task playing (MTP) agent library Π, performance model P (U |Π, T ), human behavior queue Q = ∅, total timesteps T in one episode 1: Initialize β 0 (τ ) with a uniform distribution 2: for episode k=1,2,3,. . . ,K do 3:Empty the queue Q 4:ξ 0 (τ ) ← β k-1 (τ ) 5:Select initial MTP agent π to cooperate with human using Eq. 5 6:while t < T do 7:Human chooses action a i and AI choose action according to π(a | s) β k (τ ) ← ξ T (τ ) 16:Update belief β k (τ ) using episodic return u k as observation signal following Eq. 1 17: end for response policy of CBPR agent could almost sure converge when interacting with a human partner, even when the human's policy is non-stationary.We provide all proofs and a detailed explanation in Appendix A. In addition to being able to converge in cooperation with non-stationary humans, the CBPR agent can also establish the optimal collaboration policy: THEOREM 2 (Collaboration Optimality of CBPR Agent). Denoting CBPR for CBPR algorithm, let ρ(π, m) := E[U max Ū P (U + | τ (m), π) dU + ]be the expected return of exploiting AI policy π with human meta-task policy τ (m) in MDP M m . Given a positive integer k and a set of trajectories D observed prior to the MDP M k , it follows that for any subsequent stationary MDP M k+δ , we have:Pr ρ CBPR(D), k + δ ≥ ρ(π ⋆ k , k + δ) → 1 (6)when k → ∞, where π ⋆ k is the optimal response policy for human meta-task policy at MDP M k . 
experiments: 
 In the context of Overcooked, we adopt rule-based policies developed in Yu et al. [2023] for each game layout (see Appendix C.1). These rule-based policies such as place onions in pot, deliver soup are used to train corresponding MTP agents in a one-to-one manner. In this section, we conduct extensive experiments to answer the following questions: Q1: When interacting with non-stationary agents who switch their strategies, can CBPR outperform established baselines? Additionally, can CBPR adapt its collaborative strategies to better synchronize with partner behaviors? Q2: When interacting with non-stationary agents of various collaboration skills, can CBPR surpass other baselines? Q3: Can CBPR exceed the performance of other baselines in collaboration with real humans? Q4: How do hyperparameters and number of predefined meta-tasks influence the collaborative performance (mean reward) of CBPR agents?Overcooked environment Overcooked is a popular two-player common-payoff game. It has become a typical environment for studying human-AI collaboration Carroll et al. [2019] , Knott et al. [2021] , Strouse et al. [2021] , McKee et al. [2022] , Yu et al. [2023] . In this game, players should place three onions or tomatoes in a pot and deliver as many cooked soups as possible within a time limit. Good coordination between two players is crucial for achieving a high score. We employed four layouts in our experiments: Cramped Room, Coordination Ring, Asymmetric Advantage and Soup Coordination (Figure 8 in Appendix) in our experiments. Notably, in the Asymm. Adv. and Soup Coord., the players do not interfere with each other, and their movements are unobstructed by their partners.Baselines We compare CBPR against three well-established baselines: (1) the Behavioral Cloning Play (BCP) Carroll et al. [2019] , a human model-based method designed for human-AI collaboration;(2) Fictitious Co-Play (FCP) Strouse et al.[2021], a two-stage approach trained with partners of varying skill levels; (3) Self-Play (SP) Silver et al. [2017] , a common RL method trained by playing against itself. For a fair comparison, we employed PPO Schulman et al. [2017] as the underlying algorithm of CBPR and reimplemented all baselines using identical hyperparameters in our experiments. Further details about environment setting and agents are illustrated in Appendix C.To answer question Q1, we conducted an thorough investigation into the collaboration performance of CBPR when paired with non-stationary agents. These agents exhibited changes in their rulebased policies (Appendix Table 3 ), both inter-episodically and intra-episodically. We maintained a consistent random seed for policy switching during the evaluations to ensure fairness when comparing CBPR with baseline methods.In our experiment, we assessed the collaborative performance of agents across four different policy switching frequencies, as depicted in Figure 3 . The results demonstrate that CBPR consistently outperformed the baseline methods in the majority of cases. Notably, BCP, which was trained using a stationary human model, exhibited significantly poorer performance compared to CBPR. Moreover, FCP and SP agents experienced greater fluctuations in episodic rewards, primarily due to their inability to effectively collaborate with all agents. In some instances, SP agents opted not to cooperate, resulting in zero reward. Our findings indicate that CBPR is particularly effective at collaborating with partners exhibiting varying degrees of non-stationarity. For a detailed overview of the results across the additional three policy switching frequencies (i.e. per 2 episodes, per 200 timesteps, and per 100 timesteps), please refer to the Appendix C.The cooperative capacity of non-stationary humans is typically suboptimal. A generalized agent must be capable of collaborating with partners possessing diverse collaboration skills.During the initial training phase of FCP Strouse et al.[2021], a policy pool is created by preserving various agent "checkpoints" that represent different levels of expertise. To answer question Q2, we paired CBPR with agents at varying collaboration skills preserved during the first stage of FCP training. We assessed the collaborative performance over 50 episodes across four layouts. The results demonstrated that CBPR consistently achieved higher mean episode rewards than FCP, particularly when collaborating with lower-skilled partners (Figure 4 ). It is noteworthy that BCP performed better in the Asymm. Adv. and Soup Coord. in which players' movements are not hindered by their partners. We replayed the trajectories of BCP in Cramped Rm. and Coord. Ring and observed that BCP occasionally became immobilized and failed to collaborate with partners (Figure 4b ). Figure 4 : Comparative performance analysis against baselines in cooperation with partners of diverse skill levels (low, medium and high). All agents were evaluated over 50 episodes and errors bars denote 95% confidence intervals.To address question Q3, we recruited 25 volunteers from a local university, comprising 5 females and 20 males, ranging in age from 21 to 34 years, to participate in a study involving collaboration with CBPR and baseline agents. These volunteers were randomly assigned to one of four groups, each corresponding to a different game layout. Prior to the experiment, nearly all volunteers were unfamiliar with Overcooked. We provided comprehensive instructions from scratch and allowed them to play at least five practice rounds before beginning the evaluation. Subsequently, participants were instructed to interact with both the CBPR and baseline agents through the human-AI web applications developed by Carroll et al. [2019] . Each volunteer participated in two episodes, during which we recorded the average reward obtained. According to the reward distribution (Figure 5 ), we observed that CBPR achieved more efficient collaboration than other baselines. In most comparisons, CBPR displays significant higher reward according to the one-sided Mann-Whitney U test.Case study To further demonstrate how the CBPR is more superior than baseline algorithms when collaborating with real humans, we present a case in Figure 6 . In this case, we record five frames from the Overcooked game interface to show that the ability of CBPR to adaptively adjust cooperative policies. Initially, CBPR was ready to use a dish to serve the soon-to-be-ready soup. When the human partner picked the soup, CBPR would set down the dish and continue to place onions to the pot for a new round. Meanwhile, FCP, after putting down the dish, would appear confused until the human served the soup. BCP, on the other hand, would not put down the dish and stubbornly prepare to serve the soup, ignoring the fact that the soup had already been served.Ablation on the queue size l and inter-episodic belief weight ρ. In CBPR, the length l of human behavior queue and weight ρ of inter-episodic belief mainly influence the collaborative performance.The larger l in P (Q | τ ) of Eq. 3 means that CBPR chooses policy considering more past human behaviors. The larger ρ determines that CBPR need to consider inter-episodic belief more at the beginning of an episode. To answer question Q4, we expanded on the experiments from section 4.2 demonstrate the results in Figure 7 and Appendix D.2. Overall, the results show that l=20 performed best, and in a relative simple layout (i.e. Cramped Rm.), since the belief of cooperative policy converges easily, variations in ρ have little impact on the reward. However, in complex layout (e.g. Soup Coord.) (Figure 16 ), adjusting ρ can enhance cooperative performance to a certain extent.Ablation on the number of predefined meta-tasks. The performance of CBPR depends on the design of the meta-tasks. To address the challenge of predefined meta-tasks not covering allFigure 6 : This case study analyzes five discontinuous frames from the Overcooked game interface to demonstrate the superiority of the CBPR algorithm. When a human player picks the cooked soup from the pot, the CBPR agent adapts by altering its initial plan to deliver the soup: it sets down the dish and places new onions in the pot, thereby showcasing its ability to adjust to human policies. In contrast, the FCP agent displays confusion when the human retrieves the soup and resumes placing onions only after the soup is served. The BCP agent rigidly adheres to its predetermined plan, continuously holding the plate without switching tasks to place onions, ignoring the fact that the soup has already been served.possible ones in complex task scenarios, we introduce a meta-task category as "other" (Figure 1 , bottom-right) which is represented using a random agent in practice. To demonstrate the impact of the number of predefined meta-tasks in the Soup Coord. We paired CBPR of different numbers of predefined meta-tasks with agents employing various skill levels. The results in Table 1 show that without "others" category, the performance deteriorates significantly, while the performances degrade relatively gracefully with less meta-tasks defined and more included in "others" category. The inherent advantage of SP and FCP agents. Partners with low, medium and high skill levels are represented by checkpoints (essentially are SP agents) at initial, the middle and the end of training of FCP. Therefore, SP and FCP agents have an inherent advantage in the evaluation presented in Figure 4 . Despite this, CBPR performs better when faced with partners of a lower skill level. When collaborating with real humans, FCP and SP no longer have such advantage. This leads to almost all FCP and some SP performing well against agents of various skill levels, but falling short when facing human players.The cooperative advantage of CBPR in non-separated layouts. In separated layouts (i.e. Asymm. Adv. and Soup Coord.), agents can usually complete tasks independently without considering the hindrance of the other partner's moves to themselves. However, players' own position (e.g. stand still in front of the serving areas) can obstruct their partners from completing the task in the non-separated layouts. Therefore, non-separated layouts require more cooperation between players compared to separated layouts. As shown in Figure 4 , CBPR's better performance in Cramped Rm. and Coord.Ring suggests its advantage in collaborative tasks. The double-edged sword of SP's simple policy. In Asymm. Adv., SP agent exhibits outstanding performance when it cooperates with the agent of high skill level (Figure 4c ). We replayed the game and found that the SP agent learned the simplest and most effective policy (i.e. in the right room, just pick an onion from onion dispenser and then place it in a pot within the shortest path). On the contrary, other agents exhibit some superfluous actions due to their own complexity. However, when SP cooperates with the agent of low skill level, it performed poorly because the SP agent on the right only learned the simplest policy (putting onions in the pot), and when the agent with low skill level on the left does not deliver the cooked soup, SP will wait in place rather than deliver the cooked soup. In a more complex layout Soup Coord., we found that the SP agent learned a policy of putting only one onion in the pot and starting to cook, leaving its partner confused and uncertain about what went wrong. Therefore, cooperation with SP agents leads to low performance (Figure 4d ). 
conclusion and discussion: 
 Conclusion In this work, we have proposed CBPR framework and evaluated it in the well-known game Overcooked. CBPR could effectively tackle the challenge of collaborating with humans by utilizing a suite of meta-task aware agents. In response to the non-stationary nature of human behavior, CBPR adeptly selects MTP agent based on the most recent human actions and episodic returns. We have theoretically underpinned the collaborative efficacy of the CBPR approach. Empirically, we demonstrated that CBPR outperforms baselines when collaborates with simulated humans that change their policies frequently, simulated humans that employ different skill levels and real human players. We remark our primary argument that, given the non-stationary inherent in human behaviors, it is more effective to design various agents tailored to corresponding humans in different mental and behavioral states, rather than relying on a seemingly omnipotent single agent. After all, two heads are better than one.Limitations and future work In this work, meta-tasks are modeled by manually-designed rule-based policies. In real-world application domains such as assessing power system transient stability in power grid dispatching and autonomous driving, it is time consuming to design various rule-based policies.CBPR offers a viable strategy to model meta-tasks, facilitating the training of multiple specialized experts to handle distinct meta-tasks. A notable challenge, however, is the manual summarization of domain experts' meta-tasks. As a direction for future research, we are keen to address the task of clustering policies automatically based on human trajectories. While this study Zhang et al. [2023] has made strides in this direction, the clustering approach adopted therein tends to obscure semantic understanding, presenting hurdles for AI in comprehending human behaviors. Splitting human trajectories according to the key state may be a possible solution. Additionally, perceiving the acquisition of a specific class of shaped rewards by an agent as the execution of a meta-task merits future consideration. This approach also does not depend on human data or models and offers enhanced universality and interpretability. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: we provide the implementation details (hyperparameters included) in Appendix CGuidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: See section 4.3Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our models do not have this kind of risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We provide MIT License of our released code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: We provide the screenshots of the game and details about compensation in user study section. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of collaboration convergence THEOREM 1 (Collaboration Convergence of CBPR Agent). Let H i := {S j i , π hu,i (S j i ), R j } ∞ j=0 be a trajectory collected from a single stationary MDP M i within the overall NS-MDP {M i } ∞ i=1 under the human meta-task policy π hu,i . Denote D := {(i, H i ) : i ∈ [1, k]} as a random variable representing a set of trajectories observed prior to the most recently completed stationary MDP M k . Given D, the response policy of CBPR agent could almost sure converge when interacting with a human partner, even when the human's policy is non-stationary.To establish the convergence of the posterior distribution, we first note that Doob's Martingale Convergence Theorem applies to our setting. Specifically, we have the following theorem: THEOREM 3 (Doob's Martingale Convergence Theorem). Let X n be a martingale (or sub-martingale or super-martingale) with respect to the sequence of sigma-algebrasF n , such that E[|X n |] < ∞ for all n. If there exists a constant C such that E[|X n+1 -X n ||F n ] ≤ Cfor all n, then there exists a random variable X such that X n converges to X almost surely and in L 1 .With the aforementioned theorem, we can readily establish the proof of our theorem.Proof. For non-stationary MDPs, demonstrating convergence involves showing that the algorithm can adapt to changing convergence points and ultimately reach them. Therefore, we will first establish the convergence property of the Bayesian update. Specifically, it will be demonstrated that the posterior distribution converges almost surely to the true parameter value. Subsequently, we will prove that, when using Bayesian updates, CBPR algorithms always converge to a fixed response policy, provided that the human policy remains unchanged before reaching the fixed response policy.To establish the convergence of posterior distribution, we first proof that the Doob's Martingale Convergence Theorem holds for the Bayesian updating:β k (τ ) = P(σ k |τ,π k )β k-1 (τ ) τ ′ ∈T P(σ k |τ ′ ,π k )β k-1 (τ ′ ) .Consider F k as the sequence of sigma-algebras generated by observations up to time k. A fundamental property of Bayesian updating is that the expected value of the posterior distribution conditioned on past data equals the current posterior distribution, expressed as E[β k+1 (τ )|F k ] = β k (τ ). This holds because the posterior distribution β k (τ ) encapsulates all relevant information up to time k. Thus, conditioning on F k accounts for all past observations, and in the absence of new data, the expected future posterior must align with the current posterior. This relationship signifies that, given the information available up to time k, the expectation of the next posterior does not deviate from the current posterior, establishing β k (τ ) as a martingale with respect to F k .Moreover, the bounded nature of β k (τ ) within the interval [0, 1] ensures that the Bayesian update satisfies the conditions of Doob's Martingale Convergence Theorem. Since β k (τ ) represents a probability, it is inherently bounded, which guarantees that the expected absolute change E[|β k+1 (τ )-β k (τ )||F k ] remains bounded. Additionally, with E[β k (τ )] = 1, the integrability condition required for martingale convergence is also satisfied. This combination of boundedness and integrability provides the mathematical foundation that guarantees the convergence of the sequence β k (τ ).In conclusion, the sequence of Bayesian updates β k (τ ) adheres to the defining properties of a martingale and satisfies the conditions of Doob's Martingale Convergence Theorem through its inferent property and boundedness. As a result, we can conclude that the belief β k (τ ) regarding the human meta-task will converge as k → ∞:Pr β k (τ ) → 1 (7)Secondly, to prove that the calculated best response policy of AI π ⋆ converges to a fixed value as k → ∞, we consider both the structure of the Bayesian update and the decision-making process in CBPR framework.Given β k (τ ) converges, we note that the uncertainty about the human behavior policy τ diminishes with an increasing number of observations. The convergence of β k (τ ) to a specific distribution implies that the belief about the human's policy stabilizes. In mathematical terms, as k → ∞, β k (τ ) → β(τ ) for some fixed distribution β(τ ).Then the stabilized response policy of AI π ⋆⋆ is given by:π ⋆⋆ = argmax π∈Π U max Ū τ ∈T β(τ )P U + |τ, π dU + (8)Here, the decision-making is a function of both the belief β(τ ) and the expected utility P (U + |τ, π) for each AI response policy π. As β k (τ ) converges to β(τ ), the decision-making process becomes increasingly dependent on a stable belief about the human's policy. Thus, the variability in the choice of π ⋆ diminishes, leading to a convergence of π ⋆ as well.Formally, the convergence of π ⋆ can be shown by demonstrating that the integral expression defining π ⋆ becomes stable as k → ∞. Since β(τ ) stabilizes, the integral's value, which depends on the belief about τ , also stabilizes. Consequently, by the linearity of convergence, the policy that maximizes this expression, π ⋆ , will almost sure converge to a fixed policy.Given the convergence property of π ⋆ , the almost sure convergence for the response policy of our CBPR agent is established.A. U max Ū P (U + | τ (m), π) dU + ]be the expected return of exploiting AI policy π with human meta-task policy τ (m) in MDP M m . Given a positive integer k and a set of trajectories D observed prior to the MDP M k , it follows that for any subsequent stationary MDP M k+δ , we have:Pr ρ CBPR(D), k + δ ≥ ρ(π ⋆ k , k + δ) → 1 (9)when k → ∞, where π ⋆ k is the optimal response policy for human meta-task policy at MDP M k .Proof. Considering the CBPR algorithm within the framework of MDPs, we define the expected return ρ(π, m) as the integral of the probability of achieving utility U + given the AI policy π and the human meta-task policy τ (m) in MDP M m .Assuming that the human policy library and AI policy library encompass all possible human meta-task policies and their corresponding best AI response policies. Then, we need to prove that the expected return of exploiting the CBPR algorithm's policy in any subsequent stationary MDP M k+δ will be greater than or equal to that of the optimal response policy π ⋆ k at M k . Formally, we can express this and derive it as follows:EQUATION)Where τ (k + δ) represent the true stationary human meta-task policy at MDP M k+δ , π(k ⋆ ) is the best response of AI at MDP M k , π CBPR is the response policy generated by CBPR algorithm.From theorem 1, we have Prβ k (τ (k + δ)) → 1, when k → ∞.Then we have:EQUATION)And thus the second term:EQUATIONwhile the first term:U max Ū β(τ (k + δ))P U + | τ (k + δ), π CBPR -P U + | τ (k + δ), π ⋆ k dU + converge to ρ(π ⋆ k+δ , k + δ) -ρ(π ⋆ k , k + δ). Since π ⋆ k+δ is the best response policy at MDP M k+δ , the inequality ρ(π ⋆ k+δ , k + δ) ≥ ρ(π ⋆ k , k + δ) would always hold. Consequently, we have Pr ρ(π ⋆ k+δ , k + δ) -ρ(π ⋆ k , k + δ) ≥ 0 → 1, when k → ∞.And we finally we achieve Pr ρ CBPR(D),k + δ ≥ ρ(π ⋆ k , k + ρ) → 1, when k → ∞.Note that the above derivation holds when the human meta-task policy library and AI policy library encompass all possible human meta-task policies and their corresponding best AI response policies. In practice, this assumption is seldom met and is not necessarily required to be satisfied. However, we can still enable to optimality guarantee by augmenting both human and AI policy library with primitive policiesΠ p = {π 1 , π 2 , • • • , π |A| },where policy π i ∈ Π p takes action a i ∈ A for all states Li et al. [2018] .The Overcooked environment, as introduced in Carroll et al. [2019] , presents a cooperative game where two players aim to complete as many orders as possible within a limited timeframe. In this study, we set the time constraint to 600 timesteps. The players navigate the environment to interact with various objects essential for order completion. An important aspect to note is that the current version of Overcooked requires an additional 'interact' action to initiate cooking in the pot, deviating from the version used in Carroll et al. [2019] . This change necessitates an adaptation of the previously collected human data, potentially affecting the performance of the BCP baseline. To align with this modification, we have adapted the latest version of the game to support auto-cooking when three ingredients are in a pot.The environment's action space comprises the set {up, down, lef t, right, stay, interact}. The observation space is represented by a 96-dimensional vector, capturing each player's facing direction, absolute position, and relative positions to various game elements such as the partner, the nearest onion, pot, dish, serving area, etc. Our experiments utilize four distinct layouts as depicted in Figure 8 . These layouts are chosen to illustrate a range of collaborative challenges and rewards associated with different cooking tasks. Detailed specifications of these layouts, including ingredients and reward schemes, can be found in our released code repository. In our study, we rigorously implemented MTP within the CBPR framework and ensured that all baselines (BCP, FCP, and SP) adhered to a unified methodology. This approach utilized the Proximal Policy Optimization (PPO) algorithm, a widely acclaimed reinforcement learning technique Schulman et al. [2017] , under a standardized set of parameters (refer to Table 2 ). The adoption of PPO was motivated by its balance between sample efficiency and simplicity, making it a popular choice in recent multi-agent learning research Yu et al. [2022] . To optimize the learning process and mitigate the often challenging exploration in the environment, we incorporated tailored reward shaping parameters as delineated in Table 3 . This strategy aligns with the established practices in reinforcement learning that emphasize the importance of structured rewards in complex environments Gupta et al. [2022] . Additionally, our empirical analyses revealed a distinct performance advantage of feature-based observation models over the image-based ones, leading to their adoption across all agents. The entire training process was facilitated by the computational prowess of an NVIDIA 3080 GPU. The CBPR's offline phase is a multi-faceted process encompassing meta-task modeling, MTP, and performance modeling.Initial efforts involved the manual definition of rule-based policies for each layout (Table 4 ), a step inspired by the scripted policies detailed in Yu et al. [2023] .This was followed by the training of MTP agents π ∈ Π, which were systematically paired with rule-based agents to facilitate robust policy development. The training phase, as illustrated in Figure 9 , was underpinned by a commitment to capturing a diverse range of strategic interactions. Subsequently, we developed meta-task models τ ∈ T , leveraging a two-layer feed-forward neural network. This network, initialized orthogonally and optimized at a learning rate of 1e-3, was instrumental in deciphering the nuanced mappings from observations to actions.In the final stage, performance models were crafted by pairing each MTP agent π with rule-based meta-tasks across 50 episodes, adopting a Gaussian distribution approach to model episodic rewards. The BC models were trained using human-human trajectory data from Carroll et al. [2019] . This process, partitioning 85% of data for training and 15% for validation, aligns with the standard practices in supervised learning. The neural network, characterized by two layers with a hidden size of 64 and an orthogonal initialization, was optimized for performance with a learning rate of 1e- The training of FCP agents, utilizing a pool size of 36 in the initial stage, was a strategic choice to ensure a diverse range of policy interactions. This diversity was further augmented by selecting five seeds from the first stage of FCP training for SP.The second stage of training, involving a prolonged and intensive regimen over 50,000 episodes (amounting to 3e7 timesteps), was designed to refine and solidify the agents' strategies. Such extensive training is critical in environments characterized by high complexity and variability, as it allows agents to encounter and adapt to a wide array of scenarios. This comprehensive approach to training is evident in the detailed training curves presented in Figures 11 and 12 , which provide insights into the progression and refinement of agent strategies over time. In this section, we delve deeper into the dynamics of collaboration with rule-based agents under different policy switching frequencies. We present a series of additional experiments to complement the findings discussed in Subsection 4.1. These experiments are critical in understanding how frequent policy shifts impact the overall performance and coordination in multi-agent environments.Figure 13 illustrates the comparative performance when rule-based agents switch policies every 2 episodes. Notably, the frequent policy changes introduce a unique set of challenges and opportunities for adaptation, as evidenced by the performance fluctuations across 50 continuous episodes. The standard error shaded areas, based on five random seeds, highlight the variability in performance under these conditions.Similarly, Figures 14 and 15 offer insights into the performance impacts when the policy switching occurs every 200 and 100 timesteps, respectively. These results are pivotal in understanding the optimal frequency of policy shifts to achieve efficient collaboration without overwhelming the learning agents with too frequent changes. In the following ablation study, we focus on the aspect of collaborating with partners exhibiting diverse skill levels. This study is vital to assess how agents adapt to varying competencies within a team setting. The results of this study are shown in Figures 16 and 17 , where we examine different weights and behavioral queue lengths.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have placed all the code for our algorithms and experiments in an anonymous repository (https://github.com/AlexWanghaoming/CBPR) to facilitate the reproduction of our work.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The reviewers agreed that the paper addresses an important problem of effective human-AI collaboration when humans exhibit non-stationary dynamics, the proposed framework is novel, and the results would be of interest to the community. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces Collaborative Bayesian Policy Reuse (CBPR), a framework that addresses the challenge of collaborating with non-stationary human behavior by adaptively selecting optimal collaborative policies based on the current meta-task. CBPR identifies meta-tasks underlying human decision-making and trains specialized meta-task playing (MTP) agents to enhance collaboration. Evaluations in the Overcooked simulator demonstrate CBPR's superior performance compared to existing baselines.
strength_weakness: strength_weakness: 1. The effectiveness of the CBPR framework relies heavily on accurately predefined meta-tasks, which might limit its application in environments where meta-tasks are not clearly defined or are too complex to categorize effectively. 

2. While the authors extensively evaluate CBPR in the Overcooked environment across various conditions, the paper lacks experiments in other domains beyond gaming. Overcooked, being a relatively simplistic and controlled environment, may not fully capture the complexities and challenges of human-AI collaboration in real-world applications such as autonomous driving, robotics, or complex decision-making systems. Testing CBPR's performance in more diverse and realistic domains would strengthen the paper's claims and demonstrate its broader applicability.
questions: questions: 1. How does the CBPR framework handle environments where human behaviors and tasks are not only non-stationary but also highly unpredictable or undefined?

2. The author used 5 different random seeds, but from the experimentation sections, looks like the variance of the run aren't that much, does this mean that the learning process is more deterministic?

3. How would the performance be like, if the episode has longer horizon?

4. How would the algorithm scales with LLMs or large model?
limitations: limitations: Yes, the author discussed the limitation and societal impact.
paper_summary: paper_summary: This work explores how to address the challenges of non-stationary human behavior in human-AI collaboration. The authors propose a Bayesian framework that adaptively selects optimal models during training episodes to capture the underlying consistent human behavior in solving meta-tasks. Theoretical analysis shows that their proposed method can converge to optimal solutions even when the human‘s policy is non-stationary. Both simulations and real human-subject evaluations show  that the proposed method outperforms other baselines in a cooperative game Overcooked.
strength_weakness: strength_weakness: The problem formulation and notations are unclear (see more discussion in Questions 1 and 2). Empirical results seem to suggest that the proposed method only makes minor improvements, and ablation studies imply that $\rho$ has almost no impact (which I believe is one of the innovations of this work). The authors do not thoroughly discuss these results. Instead, they directly conclude that the proposed method is effective.
questions: questions: 1. In line 113, $\sigma$ is defined as “any signal aiding cooperation, such as reward or interaction trajectory”. But since reward and trajectory could be derived from the given task $\tau$ and policy $\pi$, why is there uncertainty? Is the problem setup a stochastic, or is the policy stochastic? 
2. In Equation (1), the belief is updated in Bayes’ rule, so the task with higher expected utility (if $\sigma$ represents reward) given current policy will be assigned a higher belief probability. Is this under the assumption that the game is cooperative and both human and AI decision-makers are acting to maximize expected utility? If not, why is such belief updating reasonable? 
3. The intra-episode belief should be more stable and accurate than inner-episode belief, and contain different messages about human behavior. But from ablation studies in Section 4.2, $\rho$ seems make no difference in mean reward, why is this the case? 
4. In simulation results (Figure 4), SP outperforms CBPR in asymmetric advantages, and FCP outperforms CBPR in soup coordination. Could the author discuss more about this? Similar results appear in the human experiment (Figure 6) as well. Detailed discussions about results should be added to the paper. 
5. In the meta-task of Overcooked, if the player is holding an onion, then their only task is to put it in a pot; if holding a pot, the only task is to deliver it. Why is it necessary to identify the task?
limitations: limitations: No further limitations.
paper_summary: paper_summary: The authors proposes a Bayesian policy reuse-based framework referred to as CBPR, which allows for a collaborative artificial agent to adaptively select optimal collaborative policies from multiple policy networks. They did so by extending intra-episode belief to collaborative scenarios and incorporating this extension to the vanilla BPR. Their novel framework bypasses the need to model human behavior, and is suitable for human-AI collaborative tasks. The authors also provide theoretical proofs that their framework converges in cooperation with non-stationary humans, and that it can also establish the optimal collaboration policy. Finally, the authors evaluated this CBPR framework with Overcooked, a collaborative cooking game. Through experiments with this game, they demonstrated that CBPR outperforms baselines, showing that it is more effective to design multiple agents rather than a single agent.
strength_weakness: strength_weakness: I couldn't identify any area of weakness for this manuscript.
questions: questions: In theorem 1, the statement first mentions a stationary human policy as a given. At the end, the theorem states that convergence is also guaranteed for a non-stationary policy. It's not clear why the theorem specifically starts with non-stationary policy if it could be applied to both cases. Moreover, the proof does not clarify why the convergence guarantee for a stationary policy also applies to a non-stationary policy. Additional clarification to point out this gap would be helpful for readability.
limitations: limitations: The authors pointed out that their meta-tasks are manually-designed, rule-based policies, meaning that this method would not scale in real-world applications such as power grid dispatching and autonomous driving. While CBPR offers a strategy to model meta-tasks for human-AI collaboration, the challenge of summarizing domain experts' meta-tasks still remains.

Comments: title: Clarification of errors in the rebuttal above
Thank you for the responses
comment: Thanks for your reply. We will definitly include above revisions in our updated manuscript.
Thanks the author for the detailed response, I have raised 1 point.
We would like to clarify two errors in the above rebuttal: 
1. Due to the page limit of the rebuttal PDF file, we removed the simplified version of Figure 2, which is inconsistent with the description in above text; 
2. Meta-task category "others" is shown in Fig 1 bottom-Right of original paper, rather than the bottom-Left as stated in above text. 

We sincerely apologize for these errors and hope they have not caused any unnecessary confusion or affected your assessment of our paper.
Thank you so much for your reply. We will certainly include the additional experiments and these discussions in the updated version of our manuscript.
Thank the authors for their detailed responses. The clarifications provided regarding the meta-task (Q1 by reviewer 3rF3) and the experimental results (Q4) have addressed most of my concerns. I would be willing to raise my rating if these discussions are incorporated into the revised manuscript.