Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance

Authors: jinmin  he
kai  li
yifan  zang
haobo  fu
qiang  fu
junliang  xing
jian  cheng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 00:07:10

Abstract: Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is a general framework adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.

Main paper: 
 introduction: 
 Deep reinforcement learning (RL) has undergone remarkable progress over the past decades, showcasing its efficacy across various domains, such as game playing [16, 28] and robotic control [12, 13] . However, most of these deep RL methods primarily focus on learning different tasks in isolation, making it challenging to utilize shared information between tasks to develop a generalized policy. Multi-task reinforcement learning (MTRL) aims to master a set of RL tasks effectively. By leveraging the potential information sharing among different tasks, joint multi-task learning typically exhibits higher sample efficiency than training each task individually [27] .A significant challenge in MTRL lies in determining what information should be shared and how to share it effectively. Recent studies have proposed various approaches to tackle this challenge via network parameter sharing with carefully designed network structures [10, 22, 27] or tailored optimization procedures [4, 14, 30] . We summarize such methods as implicit knowledge sharing in Section 2. Despite these unremitting efforts, another largely overlooked way exists to exploit cross-task similarities to improve the learning efficiency of multiple tasks. Intuitively, humans can effortlessly discern which skills can be shared from other tasks while learning a specific task. For instance, someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to related skills, such as operating controls, maintaining balance, and executing turns. Likewise, a motorcyclist adept in these skills can also quickly learn to ride a bicycle. This ability allows humans to efficiently master multiple tasks by selectively referring to skills previously learned. As shown in Figure 1 , similar full or partial policy sharing is also evident in robotic arm manipulation tasks. These cross-task similarities enable policy guidance, i.e., control policies of tasks already proficient in specific skills can generate valuable training data for unmastered tasks. Compared to the common practice, which blindly generates training trajectories for each task solely with its own control policy, generating training trajectories using a control policy from other tasks that perform better in the current situation can better facilitate the learning procedure. Moreover, this explicit policy sharing approach significantly reduces unnecessary exploration of similar contexts in different tasks.The key challenge encountered in this approach to MTRL is discerning beneficial sharing control policies for each task adaptively. To address this challenge, [32] uses a Q-filter to identify singlestep shareable behaviors without ensuring optimality for long-term policy sharing. In contrast, we propose a simple yet effective framework called Cross-Task Policy Guidance (CTPG) for more robust long-term policy guidance. Initially, we group the control policies of all tasks into a candidate set. Subsequently, for each task, we train a guide policy to identify useful sharing control policies, and then the chosen control policy generates better training trajectories to achieve policy guidance. Furthermore, we design two gating mechanisms to avoid unfavorable policy guidance interfering with learning. The first, policy-filter gate, leverages the value function to refine the candidate set by masking out control policies that are not beneficial for guidance. The second, guide-block gate, withholds extra guidance for the mastered easy tasks, allowing the focus to be on further solidifying the skills already acquired. With the incorporation of the above two gates, CTPG greatly improves the quality of the policy guidance, thereby fostering enhanced exploration and learning efficiency.CTPG is a generalized MTRL framework that can be combined with various existing parameter sharing methods. Among these, we choose several classical approaches and integrate them with CTPG, achieving significant improvement in sample efficiency and final performance on both manipulation [31] and locomotion [11] MTRL benchmarks. Furthermore, we conduct detailed ablation studies to gain insights into how each component of CTPG contributes to its final performance.Algorithm 1 describes a training step for the control policy given a minibatch of data B, which contains a batch of tasks i, states s t , actions a t , rewards r t , and next states s t+1 .In addition, Algorithm 2 describes a training step for the guide policy given a minibatch data B g . The data B g not only contains a batch of tasks i, states s t , actions j t taken by guide policy, and K timesteps ahead states s t+K but also contains rewards {r t ′ } t+K-1 t ′ =t accrued over K timesteps for calculating the guide reward. Moreover, B g includes states {s t ′ } t+K-1 t ′ =t and actions {a t ′ } t+K-1 t ′ =t taken by control policy over the K timesteps for hindsight off-policy correction. 
related work: 
 Multi-task learning is a training paradigm that enhances generalization by leveraging the information inherent in potentially related tasks [3, 19, 34] . Multi-task reinforcement learning extends this concept to reinforcement learning, expecting that information shared across tasks will be uncovered by simultaneously learning multiple RL tasks [25] . In this study, we distinguish between information sharing as implicit knowledge sharing and explicit policy sharing.Implicit Knowledge Sharing. Implicit knowledge sharing primarily focuses on sharing parameters or representations, but it encounters the challenge of negative knowledge transfer due to simultaneous updates within the same network. [14, 30] regard MTRL as a multi-objective optimization problem aimed at managing conflicting gradients resulting from different task losses during training. [22, 27] partition the network into distinct modules and combine these modules to form different sub-policies for different tasks. [5, 21] endeavor to choose or learn better representations as more effective taskconditioned information for policy training. [7, 23] employ distillation and regularization to fuse separate task-specific policies into a unified policy for diverse tasks.Explicit Policy Sharing. Explicit policy sharing is expressed as the direct sharing of behaviors or policies between different tasks. [20] employs a hierarchical policy that decides when to directly use a previously learned policy and when to acquire a new one. Nonetheless, instead of learning multiple tasks simultaneously, it adopts a sequential task-learning approach, necessitating a manually welldefined curriculum of tasks. [32] uses a Q-filter to identify shareable behaviors. During exploration, each control policy proposes a candidate action, and the policy that suggests the maximum Q-value action on the source task is executed for the following timesteps. However, maximizing Q-value in a single timestep does not guarantee the optimality of this policy across continuous timesteps. CTPG is a new explicit policy sharing method that learns a guide policy for long-term policy guidance. 
preliminaries: 
 Multi-Task Reinforcement Learning. We aim to simultaneously learn N tasks, where each task i ∈ T is represented as a Markov decision process (MDP) [1, 18] . Each MDP is defined by the tuple ⟨S, A, P i , R i , γ⟩, where S denotes the state space, A the action space, P i : S × A → S the environment transition function, R i : S × A → R the reward function, and γ ∈ [0, 1) the discount factor. In the scope of this work, different tasks share the same state and action spaces, distinguished by different transition and reward functions. The goal of the MTRL agent is to maximize the average expected return across all tasks, which are uniformly sampled during training.Soft Actor-Critic. In this work, we use the Soft Actor-Critic (SAC) [9] algorithm, an off-policy actor-critic method under the maximum entropy framework. The critic network Q θ (s t , a t ) parameterized by θ, representing a soft Q-function [8] , aims to minimize the soft Bellman residual:EQUATIONEQUATIONwhere D represents the data in the replay buffer, and θ is the target critic network parameter. The actor network π ϕ (a t |s t ) is parameterized by ϕ, and the objective of policy optimization is:EQUATIONwhere α is a learnable temperature parameter to penalize entropy as follows:EQUATIONwhere H is a desired minimum expected entropy. If the optimization leads to an increase in π ϕ (a t |s t ) with a decrease in the entropy, the temperature α will accordingly increase. In the following sections, we use subscripts to signify the networks specific to each task. Specifically, the control policy of task i is represented as π i , and the corresponding Q-value function is denoted as Q i . 
cross-task policy guidance: 
 Explicit policy sharing offers a direct and efficient way to master multiple tasks. If a task is already mastered, its control policy can be fully or partially shared with other tasks to guide tasks requiring similar skills to be quickly learned. Instead of each task generating trajectories constantly by its corresponding control policy, as in most existing MTRL algorithms, we consider using control policies of other tasks to generate training data for the current task when appropriate. To achieve this goal, we propose a novel framework called Cross-Task Policy Guidance (CTPG), which extra learns a guide policy for each task to identify beneficial policies for guidance. We illustrate the trajectory generation process of Task 1 in Figure 2 . For this task, its guide policy Π g 1 selects a policy π ′ from the candidate set of all control policies {π i } N i=1 every fixed K timesteps. It then uses π ′ as the behavior policy to interact with the environment and collect data for the next K timesteps. The CTPG framework alters only the data collection process, guiding the control policy training through better exploration trajectories. In Section 4.1, we introduce the guide policy in detail and propose a hindsight off-policy correction mechanism for its training. In addition, we propose two gating mechanisms to enhance the efficiency of CTPG: the policy-filter gate discussed in Section 4.2 and the guide-block gate detailed in Section 4.3.The control policy π i (a t |s t ) of task i maps the state s t to the environment action a t , and its corresponding Q-value function Q i (s t , a t ) estimates the expected return. The guide policy Π g i (j t |s t ) of task i outputs a task index j t ∈ T, and the corresponding control policy π jt of task j t serves as the behavior policy. The guide Q-value function Q g i (s t , j t ) estimates the expected return of using Π g i to select different control policies for every K timesteps, with its Bellman equation defined as follows:EQUATIONwhere B Π g i is the Bellman operator of the guide policy Π g i , and the corresponding reward function R g i is defined as the expected cumulative discount rewards for behavior policy π jt over K timesteps:EQUATIONFor each task i, during trajectory generation, CTPG first utilizes its guide policy Π g i to sample a behavior policy π jt from the candidate set of all tasks' control policies:EQUATIONand then samples actions using the behavior policy π jt for the next K timesteps:EQUATIONwhere t ′ ∈ {t, t + 1, . . . , t + K -1}. After each timestep, we obtain the reward r t ′ = R i (s t ′ , a t ′ ) and the next state s t ′ +1 . The transition ⟨i, s t ′ , a t ′ , j t , r t ′ , s t ′ +1 ⟩ is stored in the replay buffer.The guide policy is trained to maximize the expected return of the current task by choosing appropriate control policies in certain states. If the control policies of some tasks already proficient in specific skills can be shared with the current task in similar states, the guide policy can quickly learn to use these control policies in those states to generate better training data for the current task. The guide policy and the control policy are trained simultaneously. We can use any off-policy RL algorithms for control policy training and any on/off-policy RL algorithms for guide policy training. In this work, we use SAC [9] for the control policy training and the discrete action space variant of SAC [6] for the guide policy training. Given that the guide policy acts every K timesteps, its training frequency is 1/K that of the control policy. The detailed pseudo-codes for the control policy and guide policy training are provided in Appendix A.1 (Algorithms 1 and 2).Hindsight Off-Policy Correction. During off-policy training, the guide policy faces a nonstationary challenge. Since the control policies are continually updated during the training of the guide policies, the actions chosen by the behavior policies during data collection may no longer align with the improved corresponding control policies, thereby compromising the validity of the training experience. We address this concern by implementing a hindsight off-policy correction mechanism that reassigns the action j t sampled by the past guide policy to a new one j ′ t , whose control policy π j ′ t is more likely to output the historical action sequence {a t ′ } t+K-1t ′ =t. Specifically, we utilize maximum likelihood estimation following:EQUATIONIn this way, we can leverage past experiences effectively to train the guide policy. The workings of the hindsight off-policy correction mechanism in SAC are detailed in Appendix C.In Section 4.1, we set the guide policy's action space as the set of all control policies {π i } N i=1 . However, not all control policies within the action space of Π g i are beneficial for task i in state s t . Some control policies perform even worse than the current task's own control policy π i , rendering them ineffective for guidance. To address this issue, we design a policy-filter gate to refine the action space of the guide policy by adaptively filtering out unfavorable control policies in state s t . The trajectory generation process solely using the current task's control policy π i can be regarded as equipped with a special guide policy Π g i that exclusively selects π i as the behavior policy, i.e., Π g i (i|s t ) = 1 for any s t . The guide Q-value Q g i of Π g i , defined by Equation 5, is:EQUATIONwhere we repeatedly expand Q g i to find that Q g i (s t , i) is equal to V i (s t ), the state value function of task i's control policy. Because the value function can serve as a filter for high-quality training data [17, 29, 33] , it becomes intuitive to judge the quality of the behavior policy π jt following guide policy and the current task's control policy π i by directly comparing Q g i (s t , j t ) and V i (s t ). In our implementation, we estimate V i (s t ) via Monte Carlo sampling of Q i (s t , a t ) with a t ∼ π i (a t |s t ) [15] . Relying on this mechanism, the policy-filter gate serves as a mask vector m(s t ) to indicate whether each control policy is beneficial for guidance in state s t . Specifically, each element of m(s t ) is:EQUATIONwhere j indicates the element index and task index. Then, the behavior policy π jt is sampled by:EQUATION)If none of the control policies are beneficial for guidance, i.e., m(s t ) = 0, it indicates that the current task's control policy π i is the most proficient within the current state, rendering other control policies unnecessary for enhancing trajectory generation.Comparable Guide Q-Value. Typically, RL algorithms estimate Q-values to approximate the expected return of the current state-action pair, allowing for the calculation of the policy-filter gate in Equation 11 through a direct comparison of V i (s t ) and Q g i (s t , j t ). However, in maximum entropy RL algorithms such as SAC, Q-value estimation incorporates the maximum entropy objective, leading to the incomparability of two policies with different entropy objectives. Therefore, we learn another comparable guide Q-value Qg i with discounted entropy of π i following:EQUATIONSince both Qg i (s t , j t ) and V i (s t ) estimate the return with the entropy of the current task's control policy, they can be directly compared to assess whether control policies are beneficial for guidance. A detailed comparability analysis of this comparable guide Q-value in SAC is provided in Appendix B.When simultaneously learning multiple tasks, easy tasks converge faster than difficult ones. The control policies of easy tasks allow for the quick acquisition of some effective skills, which may Initially, the guide-block gate selectively provides guidance on tasks i ∈ T g . Subsequently, the policy-filter gate generates a mask m to sift through the beneficial policies. Finally, the policy chosen by the guide policy or the control policy of the current task itself interacts with the environment over K timesteps to collect training data. be helpful in exploring other tasks. However, these mastered or easy tasks do not need additional guidance from other policies; instead, they focus on further solidifying their already acquired skills. Therefore, not all tasks require assistance from the guide policy.Based on the above analysis, we design another guide-block gate to prevent the guide policy from engaging in tasks that do not necessitate guidance. This mechanism is directly related to SAC's temperature coefficient α i . For difficult tasks i diff , their control policy entropies H (π idiff (•|s t )) tend to be high, and the corresponding temperature parameters α idiff decrease according to Equation 4. Conversely, the temperature parameters α ieasy increase for easy tasks i easy . Therefore, α i is a metric reflecting the relative difficulty and mastery of different tasks. We form the tasks that require guidance into a subset T g following:EQUATIONwhich selects the tasks by comparing the difficulty of the current task versus the average of all tasks. For tasks i / ∈ T g , the guide-block gate restricts them from using the guide policy, so they only use their own control policies to interact with the environment. Consequently, the guide policy can stop training with samples from the tasks i / ∈ T g and focus on learning the guidance for unmastered tasks.The comprehensive CTPG framework, with two special gating mechanisms, is summarized in Figure 3 . The complete pseudo-code for CTPG is described in Appendix A.2 (Algorithm 3). 
experiments: 
 The experiments are designed to answer the following research questions: Q1: Can implicit knowledge sharing approaches be combined with CTPG to further improve performance? Q2: Does the guide policy in CTPG learn useful sharing policies? Q3: How does each component within CTPG contribute to the final performance? Q4: How does CTPG perform without implicit knowledge sharing approaches? Q5: Can CTPG expedite the exploration of new tasks effectively?We conduct experiments on MetaWorld manipulation and HalfCheetah locomotion MTRL benchmarks, selecting two setups for evaluation within each benchmark.MetaWorld Manipulation Benchmark. The MetaWorld benchmark [31] consists of 50 robotics manipulation tasks employing a sawyer arm in the MuJoCo environment [24] . It provides two setups: MetaWorld-MT10, comprising a suite of 10 tasks, and MetaWorld-MT50, comprising a suite of 50 tasks. Following the settings in [9] , the goal position is randomly reset at the start of every episode.We use the mean success rate as our evaluation metric, which is clearly defined in the environment.HalfCheetah Locomotion Benchmark. The HalfCheetah is a 6-DoF walking robot consisting of 9 links and 8 joints connecting them in the MuJoCo environment [24] . The multi-task benchmark HalfCheetah Task Group [11] contains different HalfCheetah robots. HalfCheetah-MT5 includes 5 tasks under various scales of simulated earth-like gravity, ranging from one-half to one-and-a-half of the normal gravity level. HalfCheetah-MT8 includes 8 tasks with various morphology of a specific robot body part. We use the episode return as our evaluation metric.Further information regarding environmental setups is provided in Appendix D.CTPG is a generalized MTRL framework adaptable to various implicit knowledge sharing approaches, wherein both the control policy and the guide policy use a unified network. Specifically, in our implementation, the unified control policy employs the same network structure and update procedure as the implicit knowledge sharing approaches, and the unified guide policy utilizes a straightforward multi-head structure for parameter sharing.To answer Q1, we choose five classical implicit knowledge sharing approaches: (1) MTSAC extends SAC for MTRL by employing one-hot encoding for task representation. (2) MHSAC utilizes a shared network backbone apart from independent heads for each task. (3) PCGrad [30] resolves issues arising from conflicting gradients among tasks through gradient manipulation. ( 4) SM [27] trains a routing network to propose weights for soft module combinations. ( 5) PaCo [22] learns a compositional policy where task-shared parameters combine with task-specific parameters to form task policies. Combined with these implicit knowledge sharing approaches, we compare CTPG against: (i) Base represents the source version of these approaches. (ii) QMP [32] employs a one-step Q-value filter to identify shareable behaviors.We train all combinations with 0.8 million samples per task in the HalfCheetah locomotion benchmark and 1.5 million samples per task in the MetaWorld manipulation benchmark. Each combination is trained 5 times with different seeds. We evaluate the final policy over 100 episodes per task and report the mean performance and standard deviation across different seeds in Table 1 .Based on the experimental findings, it becomes evident that, except for the PaCo algorithm in HalfCheetah-MT8, the combination with CTPG leads to a notable enhancement in performance across all scenarios. QMP does not perform as well as CTPG because QMP only offers single-step behavior guidance, whereas CTPG's guide policy learns long-term policy guidance. Notably, as the number of tasks increases, CTPG exhibits superior performance due to the higher probability of explicit policy sharing between tasks, facilitating more effective policy exploration. Besides the final performance evaluation, comprehensive training curves are presented in Appendix E.1. Behavior Policy: To answer Q2, we visualize the task Pick-Place with guidance on MetaWorld-MT10 to show the specific role of the guide policy. Since the guide policy is only used in trajectory generation during training, we visualize one of the sampled trajectories in Figure 4 . We only present the initial 50 timesteps of this trajectory, in which the task has essentially been completed.To better understand policy sharing between tasks, we first explain the relevant tasks. Pick-Place: Pick and place a puck to a goal. Peg-Insert-Side: Insert a peg sideways. Push: Push the puck to a goal. Button-Press-Topdown: Press a button from the top. Drawer-Close: Push and close a drawer. Reach: Reach a goal position. The visualizations of these tasks are provided in Figure 8 (Appendix D.1).The initial and final 20 timesteps showcase that the guide policy learns useful guidance to successfully complete the source task. In the initial 20 timesteps, Pick-Place and Peg-Insert-Side employ a shared policy directing the robotic arm toward the target object. In the final 20 timesteps, the task is executed using Button-Press-Topdown to raise the gripper and then Drawer-Close to move forward. Interestingly, although the task in the final 20 timesteps intuitively aligns with Reach, the guide policy opts not to use it because the learned Reach's control policy always opens the gripper, causing the puck to fall. In the middle 10 timesteps, the probability of Pick-Place is notably high due to the absence of alternative shared policies at this stage. Specifically, the most similar Peg-Insert-Side grabs the end of the peg for insertion, while Pick-Place requires gripping the puck centrally.To answer Q3, we conduct detailed ablation studies to analyze the impact of each component in CTPG on performance enhancement. CTPG contains three key components: (1) the policy-filter gate, (2) the guide-block gate, and (3) the hindsight off-policy correction mechanism. We study the influence of each component using the MHSAC implicit knowledge sharing approach on MetaWorld-MT10.(1) Figure 5 (a) shows that the policy-filter gate plays a significant role within CTPG, leveraging the value function to constrain the direction of guided exploration. (2) Given the precise definition of binary-valued success signal in the MetaWorld benchmark, we compare the SAC temperature metric, described in Section 4.3, with another success rate metric to determine if tasks need guidance.Specifically, we block guidance for tasks with success rates exceeding 80% during evaluation. Figure 5 (b) illustrates that both metrics showcase competitive performance, exhibiting clear performance gains over no guide-block gate. However, since the success rate is a human-defined metric and difficult to define for some tasks (e.g., HalfCheetah), the SAC temperature metric is more general across most environments. (3) We ablate the hindsight off-policy correction mechanism used in guide policy training in Figure 5 (c), elucidating its improvement in training efficiency and stability. Further ablation studies of SM with CTPG on MetaWorld-MT50 are provided in Appendix E.2. To answer Q4, we train N control and guide policies independently on N tasks without implicit knowledge sharing. We perform experiments on HalfCheetah-MT8 and MetaWorld-MT10, employing the same environment configuration as in Section 5.2. As illustrated in Figure 6 , CTPG significantly improves the performance of Single-Task SAC. Notably, the impact of CTPG on MetaWorld-MT10 is more pronounced. The potential rationale is that the tasks within HalfCheetah-MT8 exhibit minimal variance in difficulty, resulting in a narrow gap in learning progress. Conversely, MetaWorld-MT10 presents a broader disparity in task difficulty, where CTPG facilitates the guidance from simpler to more challenging tasks, thus appearing more efficient. To answer Q5, we split the original task set in half, pre-training expert policies on the one half T e . For the other half, we compare direct learning with CTPG, where the agent leverages guidance from both the control policies being learned and the expert policies. Specifically, we use MT-SAC on HalfCheetah-MT8, where T e includes all tasks that enlarge the size of body parts. On the other hand, we use SM on MetaWorld-MT10, with T e containing tasks indexed from 0 to 4. Figure 7 indicates that, compared to learning the new set of tasks from scratch, CTPG can reasonably utilize experts to explore new tasks rapidly. Notably, in environments where task similarity is high, such as HalfCheetah-MT8, CTPG can quickly transfer the expert's abilities to the control policy being learned with the guide policy. 
conclusion and discussion: 
 This paper proposes the Cross-Task Policy Guidance (CTPG) framework for the MTRL explicit policy sharing. CTPG contains a guide policy and two special gates to identify beneficial sharing policies from the set of all task control policies and choose the most proficient one to generate high-quality trajectories for the current task as policy guidance. In addition, CTPG is a generalized framework adaptable to diverse implicit knowledge sharing approaches. Empirical evidence showcases that these approaches combined with CTPG further improve sample efficiency and final performance.Limitations and Future Works. One limitation of CTPG is its reliance on a predetermined guide step K, necessitating hyperparameter tuning for K across different environments. Moreover, the fixed guide step setting lacks flexibility, as the duration of shared skills execution timesteps varies inconsistently among different tasks. Consequently, exploring methods to automate the selection of the guide step K presents an intriguing avenue for future research. Additionally, irrespective of the human-defined win rate and the unique temperature parameter of SAC, investigating alternative metrics for the guide-block gate emerges as another important direction for future endeavors. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses broader impacts in Appendix F.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The code this paper uses is properly mentioned in Appendix with footnotes.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Without implicit knowledge sharing approaches, each task i has its control policy π i and guide policy Π g i . With implicit knowledge sharing approaches, there is a unified control policy π and guide policy Π g with the input of task representation z i . In both cases, we uniformly represent tasks as subscripts and omit the network parameters for a clear presentation in pseudo-code. Specifically,• The actor network with parameter ϕ for task i's control policy is denoted as π i (a t |s t ).• The critic network with parameter θ for task i's control policy is denoted as Q i (s t , a t ).• The temperature parameter for task i's control policy is denoted as α i .• The guide actor network with parameter ϕ g for task i's guide policy is denoted as Π g i (j t |s t ). • The guide critic network with parameter θ g for task i's guide policy is denoted as Q g i (s t , j t ). • The guide temperature parameter for task i's guide policy is denoted as α g i . • The comparable guide critic network with parameter θg for task i is denoted as Qg i (s t , j t ).Step Input: minibatch B contains ⟨i, s t , a t , r t , s t+1 ⟩ Initialization: actor network π, critic network Q, SAC temperature α 1: Optimize θ with SAC's critic loss in Equation 1JEQUATION2: Optimize ϕ with SAC's actor loss in Equation 3Jπ (ϕ) = E (i,st)∼B [E at∼πi [α i log π i (a t |s t ) -Q i (s t , a t )]]3: Optimize α with SAC's alpha loss in Equation 4J(α) = E i∼B E at∼πi -α i log π i (a t |s t ) -α i HInput: actor network π, SAC temperature α, minibatch B g contains ⟨{s t ′ , a t ′ , r t ′ } t+K-1 t ′ =t , i, j t , s t+K ⟩ Initialization: guide actor network Π g , guide critic network Q g , guide SAC temperature α g 1: Hindsight off-policy correct j t into j ′ t following Equation 9j ′ t = arg max j t+K-1 t ′ =t log π j (a t ′ |s t ′ )2: Calculate the guide reward r g t following Equation 6r g t = t+K-1 t ′ =t γ t ′ -t r t ′3: Optimize θ g with SAC's critic loss for guide policyEQUATION4: Optimize ϕ g with SAC's actor loss for guide policyJ Π g (ϕ g ) = E (i,st)∼B g E jt∼Π g i [α g i log Π g i (j t |s t ) -Q g i (s t , j t )]5: Optimize α g with SAC's alpha loss for guide policyJ(α g ) = E i∼B g E jt∼Π g i -α g i log Π g i (j t |s t ) -α g i Hg A.The full pseudo-code of CTPG is described in Algorithm 3.Input: guide step K Initialization: actor network π, critic network Q, guide actor network Π g , guide critic network Q g , comparable guide critic network Qg , replay buffer D ← ∅ 1: for each epoch do2:Update the task subset T g requiring guidance withT g =    i| log α i ≤ 1 N N j=1 log α j    3: // Inference Stage 4:for each task i do for each timestep t = 0, 1, . . . , T do 6:if t % K = 0 then 7: if i / ∈ T g then ▷ Guide-Block Gate 8:Behavior policy π ′ ← π i 9:else 10:Get comparable guide Q-values { Qg i (s t , j)} N j=1 11:Get control V-value V i (s t ) with Monte Carlo sampling control Q-values 12:Calculate policy-filter gate mask m following ▷ Policy-Filter Gate for each training step t = 0, 1, . . . , T dom j ← 1, Qg i (s t , j) ≥ V i (s t ) 0, Qg i (s t , j) < V i (s t ) 13: Sample π ′ ← π jt with j t ∼ Normalize(Π g i (•|s t ) • m) ▷ GuideSample a minibatch B of all tasks from D 23:Optimize ϕ and θ with B as described in Algorithm 1 24:if t % K = 0 then 25:Sample a minibatch B g of task subset T g from DOptimize ϕ g and θ g with B g as described in Algorithm 2 27:Optimize θg using Bellman Equation in Equation 13 with B g 28:end ifend for 30: end forWe expand the value function of SAC to obtain:EQUATIONBy repeatedly expanding V i , we get the final form, which is actually the optimization objective under the maximum entropy reinforcement learning framework.Then, we consider that the trajectory generation process solely using the current task's control policy π i can be regarded as equipped with a special guide policy Π g i in SAC. The comparable guide Q-value in Equation 13 of this special guide policy Π g i is:Qg i (s t , i) = E a t ′ ∼πi,s t ′ +1 ∼Pi t+K-1 t ′ =t γ t ′ -t (R i (s t ′ , a t ′ ) + α i H(π i (•|s t ′ ))) + γ K E st+K ∼Pi Qg i (s t+K , i) = E a t ′ ∼πi,s t ′ +1 ∼Pi t+K-1 t ′ =t γ t ′ -t (R i (s t ′ , a t ′ ) + α i H(π i (•|s t ′ ))) + γ K Qg i (s t+K , i) = • • • = E a t ′ ∼πi,s t ′ +1 ∼Pi ∞ t ′ =t γ t ′ -t (R i (s t ′ , a t ′ ) + α i H(π i (•|s t ′ ))) = V i (s t ).(16) The ellipsis part is the repeated expansion of Qg i . Therefore, in SAC, the policy-filter gate can refine the action space of the guide policy by directly comparing Qg i (s t , j t ) and V i (s t ).The hindsight off-policy correction mechanism is proposed to mitigate the non-stationarity challenge of the guide policy update process. It reassigns the action j t sampled by the past guide policy to a new one j ′ t , whose control policy π j ′ t is more likely to output the historical action sequence {a t ′ } t+K-1 t ′ =t . During the guide policy update process, we first get the reassigned action j ′ t following the maximum likelihood estimation in Equation 9. For SAC's critic update, the critic loss of the guide policy is,EQUATION)where the hindsight correction explicitly modifies the update procedure by shifting the Q-value estimation from Q g i (s t , j t ) to Q g i (s t , j ′ t ). For SAC's actor update, the actor loss of the guide policy is,EQUATIONwhich is not explicitly different from the original function in SAC. However, it indirectly affects actor learning due to SAC's unique actor update method [9] . Specifically, SAC's actor loss is derived from its optimization objective, and the policy is updated according to,EQUATION)where the partition function Z πold (s t ) normalizes the distribution. In essence, SAC's policy is updated by fitting the distribution of the SoftMax function of Q-value with temperature α. Therefore, modifying the update of Q g i (s t , j t ) to Q g i (s t , j ′ t ) using hindsight correction leads to a different guide policy actor optimization objective, thus affecting the training of the guide actor network. The MetaWorld manipulation benchmark [31] consists of 50 robotics manipulation tasks employing a sawyer arm to acquire diverse manipulation skills. MetaWorld-MT50 encompasses all 50 manipulation tasks, and MetaWorld-MT10 encompasses 10 tasks (a subset of MT50), as illustrated in Figure 8 . In MetaWorld, the source tasks are configured with fixed goals, limiting the policy's ability to generalize to tasks of the same type with varying goals. Following a similar setup as [27] , we extend all the tasks to a random-goal setting, where both items and goals reset randomly at each episode's onset. In addition, unlike [27, 30] , we conduct our experiments on the MetaWorld-V2 benchmark 2 .In addition, we extend the episode length to 200 timesteps, different from the previous setting of 150. We evaluate the rule-based policies provided by the MetaWorld benchmark across 100 sample episodes. Under the initial episode length setting of 150 timesteps, 42 tasks achieve a success rate exceeding 90%. However, among the remaining 8 tasks, the task Disassemble exhibits a mere 50% success rate. In contrast, upon adjusting the episode length setting to 200 timesteps, success rates for all 50 tasks exceed 90%, with the task Disassemble achieving an improved success rate of 91%. The HalfCheetah locomotion benchmark originates from the gym-extensions framework [11] 3 , which focuses on continuous control multi-task reinforcement learning tasks, including several task groups of the standard gym environments [2] . We run our experiments with the HalfCheetah agent, which is a 2-dimensional robot consisting of 9 links and 8 joints connecting them (including two paws). Each episode 1000 timesteps. HalfCheetah-MT5 consists of 5 locomotion tasks. The HalfCheetah agent is tasked with running in environments with various scales of simulated earth-like gravity, ranging from one-half to one-and-a-half of the normal gravity level.The detailed gravity value of each task is illustrated in Table 2 .HalfCheetah-MT8 consists of 8 locomotion tasks, as depicted in Figure 9 . The HalfCheetah agent must run with variations in the morphology of a specific body part, such as the torso, thigh, leg, and foot. The Big body part involves scaling the mass and width of the limb by 1.25, and the Small body part involves scaling by 0.75. Samples/Task (£10 5 )This section complements the result in Section 5.2, which only presents the final performance of the experiment. Here, we show the training curves for different combinations of explicit policy sharing methods and implicit knowledge sharing approaches across four environments in Figure 10 . Each row of the figure represents a distinct implicit knowledge sharing approach, while each column represents a different environment. Within each subfigure, the three curves represent the base one without any explicit policy sharing method and two variations using different explicit policy sharing methods. We evaluate the training policy every 10K samples per task 32 episodes and report the mean episode return or success rate, along with standard deviation, across 5 different seeds. The result shows that beyond the ultimate performance improvement, CTPG also enhances the sample efficiency. In addition to the ablation experiment of MHSAC with CTPG on MetaWorld-MT10 in Section 5.4, we also conduct ablation studies using SM implicit knowledge sharing approach on MetaWorld-MT50.Figure 11 shows similar results to those in Section 5.4. The policy-filter gate is crucial within CTPG.The guide-block gates with two metrics demonstrate competitive performance and show improvement compared to the absence of the guide-block gate. Additionally, the hindsight off-policy correction mechanism significantly enhances training efficiency and stability.Within CTPG, the guide policy Π g i selects a policy π jt from the candidate set of all control policies {π j } N j=1 every fixed K timesteps. The selected policy π jt is then used as the behavior policy to interact with the environment and collect data for the next K timesteps. The guide step K is a predefined hyper-parameter. We conduct ablation experiments on the guide step K in the two setups: HalfCheetah-MT8 and MetaWorld-MT10. In HalfCheetah-MT8, an entire episode contains 1000 timesteps, so we set K ∈ {1, 5, 10, 20, 50}. In MetaWorld-MT10, an entire episode contains 200 timesteps, so we set K ∈ {1, 3, 5, 10, 20}. The result, shown in Figure 12 , indicates that both short and long guide steps lead to decreased performance and increased variance.Overall, CTPG performs well in both environments when K = 10. Furthermore, automating the selection of the guide step K is part of our future work.In Section 4.2, we design a policy-filter gate to refine the action space of the guide policy by adaptively filtering out control policies that are not beneficial for guidance. Specifically, the restriction of the action space is implemented by a mask, which is generated by comparing the control policy's V-value V i (s t ) and the guide policy's Q-value Q g i (s t , j t ). In SAC, the V-value is the expectation of the Q-value with the entropy of policy. It is formulated as:EQUATIONTo avoid introducing an additional network to estimate the V-value V i (s t ), we estimate it via Monte Carlo sampling of the Q-values Q i (s t , a t ) with a t ∼ π i (a t |s t ) [15] . We conduct ablation experiments varying the Monte Carlo sampling times H, as depicted in Figure 13 . It is observed that the performance remains largely unaffected by the number of samples, except in cases where H equals 1, which leads to performance degradation and heightened variance due to excessive randomness.CTPG learns a flexible mixture policy combining different control policies. To verify the advantages of the mixture strategy, we compare CTPG against another baseline that incorporates a guide strategy from [26] , which solely learns from another control policy with the highest performance on the task.Since [26] focuses on continual RL, which requires a predefined task sequence and does not align with the MTRL experimental setting. Therefore, we adapt the core idea, implementing a version that selects the best-performing policy for the current task to guide exploration and generate trajectory data. Specifically, after each H rounds of data collection, we evaluate all policies across all tasks, selecting the top-performing policy for each task to guide data collection in the subsequent rounds. We refer to this baseline as BPT (Best Performance Transfer) and set H to 50 episodes in our implementation. We use MTSAC in HalfCheetah-MT8 and MHSAC in MetaWorld-MT10. The result, shown in Figure 14 , demonstrates that CTPG outperforms BPT. CTPG's guide policy learns a more flexible strategy: it not only can learn to share a single policy within a complete trajectory (like BPT), but also can develop a mixture policy by combining different control policies, making it more transferable between tasks. Notably, BPT performs better in HalfCheetah-MT8 than in MetaWorld-MT10 because the tasks in HalfCheetah-MT8 share more significant similarities, whereas the policy transfer and guidance in MetaWorld-MT10 require combination strategies.CTPG is a general MTRL framework that can be adapted with other RL algorithms, even allowing the control and guide policy to use different RL algorithms. Since SAC is a widely used algorithm in continuous control and serves as the base algorithm for all baselines, we also choose SAC as the base algorithm in this work. In addition, we explore the adaptation of TD3 with the CTPG and also employ different RL algorithms for the control and guide policies. Specifically, the control policy uses TD3, and the guide policy uses DQN. We evaluate MTTD3 (the original TD3 using one-hot encoding for task representation) on HalfCheetah-MT8 and MHTD3 (utilizing a multi-head network for different tasks) on MetaWorld-MT10. The result shown in Figure 15 demonstrates that CTPG can enhance performance and sample efficiency when combined with other backbone RL algorithms. We implement all experiments using the MTRL codebase [21] 4 and access the HalfCheetah locomotion environment to this framework. One significant change we make involves removing the reward normalizer wrapper, which leads to worse results.We use AMD EPYC 7742 64-Core Processor with NVIDIA Geforce RTX 3090 GPU for training. Each method is trained 5 times with different seeds. Using MetaWorld-MT10 as an example, the training time required for a full training run varies from 16 hours (MTSAC) to 52 hours (PcGrad) for the baseline implicit knowledge sharing approaches. For each approach with CTPG, the training times range from 20 hours (MTSAC w/ CTPG) to 56 hours (PcGrad w/ CTPG). The guide policy, using a simple multi-head network architecture, acts every K step (K = 10 in our implementation), so its training frequency is set as 1/K. Consequently, CTPG does not significantly increase the training time. For a detailed hyper-parameter of the guide policy, please refer to Appendix F.3.For a fair comparison, we use the following common training setups on all methods:Disentangled SAC temperature parameters. This setup is standard established in MTRL. Herein, distinct tasks utilize varying SAC temperature parameters, as described in Equations 2 and 3, which are also optimized by independent losses in Equation 4. Therefore, this setup facilitates the individual adaptation of exploration and exploitation balances for each task, effectively accommodating the diverse learning dynamics across different tasks during training.Loss maskout of extreme tasks. This setup, utilized by [21, 22] , involves the selective masking of the potentially destabilized loss J i of task i from the total loss J, aiming to mitigate its adverse effects on other tasks. Specifically, when the task loss J i surpasses a predefined threshold ϵ (set as 3e3, the same as [22] ), that task i is excluded from the total training loss.Multi-task loss rescaling. In recognition of the inherent discrepancy in convergence rates between tasks, where easy tasks usually converge faster, [27] propose an optimization objective weight of task i, formulated as:EQUATIONwhere α i is the SAC temperature parameter of task i, and N is the total number of tasks. Consequently, the total loss is adjusted toJ = E i [w i • J i ],ensuring a balanced training process across different tasks.This section provides the hyper-parameters of each method in our experiment. General hyperparameters shared by all methods are illustrated in Table 3 . Table 4 to Table 8 show the additional hyper-parameters specific to each implicit knowledge sharing approach. In addition, the guide policy can also be trained using implicit knowledge sharing methods. In our implementation, we use the simple multi-head network structure for the guide policy in CTPG, and the relevant additional hyper-parameters are displayed in Table 10 . Justification: The experiments compute resources information is provided in Appendix E.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research conduct in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/Farama-Foundation/Metaworld/tree/v2.0.0https://github.com/Breakend/gym-extensionshttps://github.com/facebookresearch/mtrl 


Decision: The paper introduces Cross-Task Policy Guidance (CTPG), a method for enhancing multi-task reinforcement learning (MTRL) by allowing policies to learn from each other's trajectories. The approach involves a guide policy that selects the most suitable behavior policy from a pool of all tasks' control policies, along with mechanisms to prevent negative transfer and ensure efficient learning. CTPG shows improved performance in benchmarks like MetaWorld and Half-Cheetah.

While the method is innovative, some reviewers noted that the contribution feels incremental, especially in comparison to latest methods such as QMP. Concerns were also raised about the scalability of CTPG to more complex tasks and the focus on converged performance rather than sample efficiency. The computational complexity of the approach was another point of uncertainty.

However, the paper also exhibits strengths. The paper is well-presented, with clear motivation and thorough experiments that demonstrate the effectiveness of CTPG. The innovative approach to leveraging cross-task similarities offers potential for advancing MTRL research. The positive impact and the detailed response to reviewer concerns support its acceptance.

Based on the overall positive reviews, it is recommended to accept the paper, provided that the revisions discussed in the rebuttal and comparison to the most recent multi-task RL algorithms are incorporated in the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a method called CTPG to enable policies trained on different tasks to learn from each others' generated trajectories. CTPG operates by learning a guideline policy that determines which control policy in a given set should best generate trajectories to enable agents to learn a particular task. Then, the authors also designed additional mechanisms that (i) deal with how each policy whose trajectories are sampled may change from their learning process, (ii) prevent negative transfer from using trajectories from irrelevant policies for learning, and (iii) promote the convergence towards good policies by preventing any transfer of experience once a policy is already proficient at its intended task.

Using environments from multi-task reinforcement learning benchmarks, the authors then investigated their method's performance when (i) learning a set of tasks in parallel and learning a new task that is not from the set of previously learned tasks. Furthermore, the authors also conducted further analysis to investigate (i) which component within their learning algorithm is most responsible for the method's performance and (ii) whether the guideline policy learned a sensible experience selection policy.
strength_weakness: strength_weakness: **Minor Weaknesses - Additional Comparisons**

Perhaps another baseline that could be compared is the simple strategy from [1], where one simply learns from another policy with the highest performance at the task. Using this baseline should elucidate the effects of switching between different policies to transfer from as opposed to just transferring from a policy that seems to perform the best.

**Minor Weaknesses - Hindsight Off-Policy Correction**

From Section 4.1 alone, I also do not find how the hindsight off-policy correction mechanism affects the remainder of the learning process. While I did check Algorithm 2 in the appendix, I am uncertain why $j^{'}_{t}$ (the behavioral policy chosen in hindsight) only affects SAC's critic updates and not also the policy updates.

References:

[1] Disentangling Transfer in Continual Reinforcement Learning. Wolczyk et al. NeurIPS 2022.
questions: questions: 1. How would CTPG perform against a simple transfer learning strategy where experience is only transferred from the policy that has the best performance?

2. Does the behavioral policy chosen in hindsight affect the actor-network training? If it does, how would it do so?
limitations: limitations: The authors have sufficiently outlined their method's limitations and interesting directions for future work in the last paragraph of the manuscript.
paper_summary: paper_summary: This paper addresses the problem of multi-task reinforcement learning (MTRL). To this end, the paper proposes a method that selectively shares behaviors from the policies learning to solve other tasks. The experiments conducted in a locomotion domain (multi-task Half-Cheetah) and a robot arm manipulation domain (Meta-World) verify that the proposed method can improve the performance of the learned policies. Ablation studies justify the effectiveness of many proposed components, including the policy-filter gate, the guide-block gate, and the hindsight correct. Yet, I am not convinced that the proposed method could improve the sample efficiency, which is the key goal of MTRL. Also, it seems that the proposed method is incremental and lacks novelty. Therefore, I am leaning toward rejecting this work in its current form.
strength_weakness: strength_weakness: **Main results (Table 1)**
- Table 1 and Section 5.2 present quantitative results of converged performance. However, as far as I am concerned, the comparisons of MTRL should focus on sample efficiency, i.e., how fast each method can learn, instead of converged performance. Therefore, I am not convinced by this evaluation.

**Comparison to k-step QMP**
- The paper states that the proposed method outperforms QMP because "guide policy learns long-term policy guidance." I am not entirely convinced. One can simply use QMP to select from k-step behavior proposals and execute the selected proposal, which can also achieve this long-term policy guidance. Including this variant of QMP would be necessary to show that the performance gain comes from other designs of the proposed method.

**Guide step K**
- The best guide step K in the MT Half-Cheetah and Meta-World are different. It is unclear, given a new MTRL domain, how we should choose K. It seems that we could only tune this hyperparameter via trial and error. 

**Backbone RL algorithms**
- This work adopts SAC as the backbone RL algorithm. Is it possible to use other RL algorithms, such as TD3 or PPO?

**Clarity**
- Section 4 is difficult to follow. Sufficiently describing the intuitions before introducing each component of the proposed method would significantly improve the readability of this section.

**Novelty**
- The proposed method seems incremental given that the QMP paper explores this idea of behavior sharing. Despite the high similarity between this work and the QMP paper, the authors seem to deliberately hide this by avoiding discussing QMP in the introduction.
questions: questions: See above
limitations: limitations: Yes
paper_summary: paper_summary: This paper tackles the problem of multi-task reinforcement learning. Previous works have primarily focused on MTRL through specialized network structures or methods to resolve conflicting gradients. This paper considers the problem from an orthogonal perspective, by actively sharing control policies for each task adaptively, in the hope that sometimes the policy of one task can guide the exploration of other tasks. Specifically, similar to in hierarchical RL, a high-level policy is used to decide which “task policy” to call at each particular state, where “bad” policies are masked out through a learned value function. Evaluated on Metaworld and HalfCheetah MTRL benchmark, the method shows improved performance compared to previous works.
strength_weakness: strength_weakness: - The proposed method contains many floating pieces that seem to add burden both to implementation and hyperparameter tuning. Some of the pieces seem rather like hack than a principled solution. For example:
  - For the hindsight off-policy correction, the action of the guide policy is relabelled to the one that is most likely to generate the sequence of actions. However, it should be likely that none of the control policies can generate the old sequence of actions with high probability, in which case relabeling will not help at all.
  - The action space masking is essentially masking out control policies with low Q values. Shouldn’t this already be reflected in the policy through policy gradient updates? Why do we need additional masking?
  - Using the temperature coefficient of SAC as an indication of the policy performance is very empirical and may not always be true. This also makes the proposed method specific to adaptive temperature SAC.
- It seems that the hindsight off-policy correction would require much more forward passes than a regular SAC. Can the author show the performance of the proposed method with x axis being wall-clock time?
- For the proposed method to be effective, we need the policy of some other tasks to have better performance than the policy that is trained for the current task. One scenario I can think of is when the multiple tasks secretly form a curriculum (e.g. one task is the prerequisite for another task), which might be the case in Metaworld and halfcheetah. But I’m not sure how realistic such an assumption is in the real world.

Minors:
- Line 207: dose → does
- “someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to related skills” → I don’t think that’s actually the case…
questions: questions: See weaknesses
limitations: limitations: The limitations are adequately addressed.
paper_summary: paper_summary: The paper presents Cross-Task Policy Guidance (CTPG), a novel framework designed to improve multi-task reinforcement learning (MTRL) by leveraging cross-task policy similarities. CTPG trains a guide policy for each task to select the most suitable behavior policy from a pool of all tasks' control policies. This method aims to generate better training trajectories and enhance learning efficiency. The authors propose two gating mechanisms: a policy-filter gate to filter out non-beneficial control policies and a guide-block gate to block unnecessary guidance for mastered tasks. Empirical evaluations on manipulation and locomotion benchmarks demonstrate that integrating CTPG with existing parameter sharing approaches significantly enhances performance.
strength_weakness: strength_weakness: ### W1. Scalability Concerns

The scalability of CTPG to more complex and large-scale environments is not thoroughly discussed. While the method shows promising results in the tested benchmarks, a broader analysis of its scalability and practical utility in more complex scenarios may improve the significance of this work.

### W2. Computational Complexity

The computational complexity of training and deploying CTPG, particularly the guide policy and the gating mechanisms, is not explicitly addressed. Understanding the computational requirements and potential limitations in terms of resources and execution time would provide a more comprehensive evaluation of its applicability.
questions: questions: ### Q1. Scalability to Complex Environments

How does the CTPG framework scale to more complex, large-scale environments? Are there any specific challenges or limitations that need to be addressed for practical deployment in such scenarios?

### Q2. Computational Requirements

What are the computational requirements for training and deploying CTPG? How does the method perform in terms of execution time and resource consumption compared to existing MTRL methods?
limitations: limitations: The authors acknowledge the limitations related to the predetermined guide step K and the reliance on specific benchmarks. However, a more detailed discussion on potential negative societal impacts and strategies to mitigate them would be beneficial.

Comments: title: Rebuttal Summary and Appreciation
Official Comment from Reviewer xTD7
Response to authors rebuttal
Re: Rebuttal by Authors
comment: Dear Reviewers, ACs, SACs, and PCs:

As the discussion phase comes to a close, we would like to sincerely express our gratitude for your time and effort throughout the review and rebuttal process. We deeply appreciate the constructive and insightful comments provided by the reviewers after carefully reading our manuscript, which have significantly improved the quality of our work.

Here, we would like to summarize the key points of our rebuttal and discussions:

1. Based on (Reviewer xTD7)'s feedback, we conducted ***additional experiments comparing CTPG with BPT***. These experiments more clearly demonstrate the advantages of CTPG in learning a mixture policy by combining different control policies.
2. In response to (Reviewer 5DoG)’s concerns, we performed ***supplementary experiments using another base RL algorithm TD3***, which further demonstrates that CTPG is a generalized MTRL framework.
3. To address (Reviewers J1dv and k7nL)’s concerns, we provided a complementary ***analysis of the computational complexity*** of CTPG.
4. We are grateful for all the reviewers’ comments on the revisions, which have made our paper more complete, including ***a detailed explanation of the hindsight mechanism's impact on SAC*** (xTD7), ***additions to the experimental hyperparameters setup*** (5DoG), ***a more relevant realistic example with CTPG*** (J1dv), and corrections of some typos.

***During the discussion, all the reviewers provide appreciations and claims for his/her concerns addressed,*** and we were ***especially encouraged*** by (Reviewer xTD7)’s conclusion: ***"Given the authors' responses to the points raised by each reviewer, I am increasing my score because I believe this is a good paper worthy of acceptance."***. We appreciate that we receive ***all positive*** feedbacks from every reviewer. We believe that our work would offer valuable insights to the MTRL community and make a positive contribution to the `NeurIPS 2024` conference.

We would like to thank you once again for your time and dedication throughout the review process. 

Best regards,

Authors
Thank you very much for taking the time to review our response. We are glad that our response has effectively addressed your concerns. Once again, we sincerely appreciate your thorough review and insightful feedback, which have helped us enhance our paper.

Best regards,

All authors
Thank you for considering attempting to address the points I've previously raised.

> Specifically, after each $H$ round of data collection, we evaluate all policies across all tasks and choose the best-performing policy to collect the data for the next $H$ rounds. We refer to this baseline as BPT (Best Performance Transfer)

This is exactly the baseline that I envisioned in my previous feedback. Given the positive results when comparing against this baseline, I no longer view the lack of comparisons against BPT as a minor weakness of this paper.

> Although the actor loss (Line 4 in Algorithm 2) does not explicitly use relabeled $j'_t$, it indirectly affects actor learning due to SAC's unique actor update method [2].

I also agree that this needs to be better highlighted in the next manuscript version. Perhaps it's also important to note that unlike SAC-based actor updates in environments with continuous action spaces (where we rely on a Monte Carlo estimate of the KLD objective since integrating over each possible action is impossible), the KLD objective can be easily evaluated when the input distributions to the KL divergence are categorical.

**Closing remarks**

Given the authors' responses to the points raised by each reviewer, I am increasing my score because I believe this is a good paper worthy of acceptance.
Thank you very much for reviewing our response. We are pleased that our response has effectively addressed your concerns. Once again, we sincerely appreciate your thorough review and insightful feedback.

Best regards,

All authors
I appreciate the authors for the detailed response. Most of my concerns are addressed therefore I'm raising my assessment.
We sincerely thank you for taking the time to read our response. We are pleased that our response has effectively addressed your concerns. Thank you once again for your valuable input.

Best regards,

All authors
Thank you for the rebuttal, including the additional TD3 results and clarifications. I am increasing my score to 5 to reflect what's addressed by the author's rebuttal.
We greatly appreciate your time. If there are any remaining issues or concerns that have not been addressed, please let us know. We would be more than happy to engage in further discussion to resolve them. If all concerns have been addressed, we kindly hope you can reconsider the rating assigned to our submission.

Thank you once again for your valuable input.

Best regards,

All authors
I thank the authors for the clarifications. I will maintain my original score.
Dear Reviewers,

The author responses have been uploaded. Please carefully review these responses to see if your concerns have been adequately addressed and actively participate in discussions.

It is important that **all reviewers should acknowledge having read the author responses by posting a comment**, irrespective of whether there is any change in your rating.

Thank you for your cooperation.

Best regards, \
Area Chair