Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance
Authors: Jinmin He
Kai Li
Yifan Zang
Haobo Fu
QIANG FU
Junliang Xing
Jian Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-task Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is a general framework adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.

Efficient Multi-Tash Reinforcement Learning with Cross-Task Policy Guidance
Jinmin Hel Kai 2 Yifan Zang! Haobo Fu', Qiang Fu', JJuliiag Xing"; Jian Cheng" 1.
'Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences AiRiA *Tiinguua University Tencent Al Lab [he j iimin2021, kai. 11, Faanyiffn2000 , jian. cheng) ac. en, (haobofu, leonfu] @tencent. com, ]xing@tsingghu, edu. cn
Abstract
Multi-task reinforcement learning endeavors to efficiently leverage shared informa- tion across various tasks, facilitating the simultancous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross- Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. addition, propose two gating mechanisms to improve the lcarning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is general frame- work adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.
Introduction
Deep reinforcement learning (RL) has undergone remarkable progress over the past decades, show- casing its efficacy across various domains, such game playing [16 28 and robotic control [3] However, most these deep RL methods primarily focus on learning different tasks isolation, making it challenging to utilize shared information between tasks to develop generalized policy. Multi-task reinforcement learning (MTRL) aims to master a set of RL tasks effectively. By leveraging the potential information sharing among different tasks, joint multi-task learning typically exhibits higher sample efficiency than training each task individually (27).
A significant challenge in MTRL lies in determining what information should be shared and how to share it effectively. Recent studies have proposed various approaches to tackle this challenge
via network parameter sharing with carefully designed network structures [10] 22 27 or tailored optimization procedures 4 30). We summarize such methods as implicit knowledge sharing in Section 2 Despite these unremitting efforts, another largely overlooked way exists to exploit cross-task similarities to improve the learning efficiency of multiple tasks. Intuitively, humans can effortlessly discern which skills can be shared from other tasks while learning specific task. For instance, someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Rosss open
Reach
Prsh
hondle
Reach handle
Push handle
Grah handle
Tiandlaee opeo
(a) Button-Press v.s. Drawer-Close
(b) Door-Open v.s. Drawer-Open
Figure 1: Full or partial policy sharing in the manipulation environment. (a): Task Button-Press and Drawer-Open share almost the same policy, where the robotic arm needs to reach specified position (button or handle) and then push the target object. (b): Task Poor-Open and Drawer-Oper share the policy of grabbing the handle in the first phase, but they required open the target object by different movements (rotation or translation).
related skills, such as operating controls, maintaining balance, and executing turns. Likewise, motorcyclist adept in these skills can also quickly learn to ride bicycle. This ability allows humans to efficiently master multiple tasks by selectively referring to skills previously learned. As shown in Figure [ similar full or partial policy sharing is also evident in robotic arm manipulation tasks. These cross-task similarities enable policy guidance, i.e., control policies tasks already proficient in specific skills can generate valuable training data for unmastered tasks. Compared to the common practice, which blindly generates training trajectories for each task solely with its own control policy, generating training trajectories using a control policy from other tasks that perform better in the current situation can better facilitate the learning procedure. Moreover, this explicit policy sharing approach significantly reduces unnecessary exploration of similar contexts in different tasks.
The key challenge encountered this approach to MTRL discerning beneficial sharing control policies for each task adaptively. To address this challenge, (32) uses Q-filter to identify single- step shareable behaviors without ensuring optimality for long-term policy sharing. contrast, we propose simple yet effective framework called Cross-Task Policy Guidance (CTPG) for more robust long-term policy guidance. Initially, we group the control policies of all tasks into a candidate set. Subsequently, for each task, we train guide policy to identify useful sharing control policies, and then the chosen control policy generates better training trajectories to achieve policy guidance. Furthermore, design two gating mechanisms to avoid unfavorable policy guidance interfering with learning. The first, policy-filted gate, leverages the value function to refine the candidate set by masking out control policies that are not beneficial for guidance. The second, guide- guide-block gate, withholds extra guidance for the mastered easy tasks, allowing the focus to be on further solidifying the skills already acquired. With the incorporation of the above two gates, CTPG greatly improves the quality of the policy guidance, thereby fostering enhanced exploration and learning efficiency.
CTPG is generalized MTRL framework that can be combined with various existing parameter sharing methods. Among these, we choose several classical approaches and integrate them with CTPG, achieving significant improvement in sample efficiency and final performance on both ma- nipulation [3 and locomotion [1T] MTRL benchmarks. Furthermore, we conduct detailed ablation studies to gain insights into how each component of CTPG contributes to its final performance.
2 Related Work
Multi-task learning is a training paradigm that enhances generalization by leveraging the information inherent in potentially related tasks [3] 19 (34] Multi-task reinforcement learning extends this
concept to reinforcement learning, expecting that information shared across tasks will be uncovered by simultaneously learning multiple RL tasks 255. In this study, we distinguish between information sharing as implicit knowledge sharing and explicit policy sharing.
Implicit Knowledge Sharing. Implicit knowledge sharing primarily focuses on sharing parameters or representations, but it encounters the challenge of negative knowledge transfer due to simultaneous updates within the same network. [14 30 regard MTRL as multi-objective optimization problem
aimed at managing conflicting gradients resulting from different task losses during training. [22] 27

---

partition the network into distinct modules and combine these modules to form different sub- policies for different tasks. [3] [ endeavor to choose or learn better representations as more effective task- conditioned information for policy training. 7 2 employ distillation and regularization to fuse separate task-specific policies into a unified policy for diverse tasks.
Explicit Policy Sharing. Explicit policy sharing is expressed as the direct sharing of behaviors or policies between different tasks. (20) employs a hierarchical policy that decides when to directly use previously learned policy and when to acquire new one. Nonetheless, instead of learning multiple tasks simultancously, it adopts a sequential task-learning approach, necessitating a manually well- defined curriculum of tasks. [32 uses Q-filter identify shareable behaviors. During exploration, each control policy proposes a candidate action, and the policy that suggests the maximum Q-value action the source task is executed for the following timesteps. However, maximizing Q-value in single timestep does not guarantee the optimality of this policy across continuous timesteps. CTPG is new explicit policy sharing method that learns guide policy for long-term policy guidance.
3 Preliminaries
Multi-Tast Reinforcement Learning. We aim to simultaneously learn N tasks, where each task i E is represented as Markov decision process (MDP) [ 18 Each MDP is defined by the tuple (S, A, Pr, Ri, 7). where S denotes the state space, A the action space, P: S x A S the environment transition function, R2 S x A - R the reward function, and 7 E [0, 1) the discount factor. In the scope of this work, different tasks share the same state and action spaces, distinguished by different transition and reward functions. The goal of the MTRL agent is to maximize the average expected return across all tasks, which are uniformly sampled during training.
this work, we use the Soft Actor-Critic (SAC) 9 algorithm, an off-policy actor-critic method under the maximum entropy framework. The critic network (s a.) parameter- ized by 0, representing a soft Q-function (8), aims to minimize the soft Bellman residual:
Soft Actor-Critic
1 (Qo(st, as) - (re + YE (Vo(+++)1))
(0) = (ss
(1)
2 E. Eu 1Qa(s, ax) a log To (atls)))
Vi((s))
(2)
where 'D represents the data in the replay buffer, and @ is the target critic network parameter. The actor network To (a (s)) is parameterized by o, and the objective of policy optimization is:
(Ø) = Eg,nD [Ear~z, (ar log (a(a))s)) - Qo(se, at]]]
(3)
where a is learnable temperature parameter to penalize entropy as follows:
(a) B,, a log (a(aass:)= - aH).
where H is a desired minimum expected entropy. the optimization leads to an increase in (at with decrease in the entropy, the temperature will accordingly increase. In the following sections, we use subscripts to signify the networks specific to each task. Specifically, the control policy of task i represented as Ti and the corresponding Q-value function is denoted as Qi.
4 Cross-Task Policy Guidance
Explicit policy sharing offers direct and efficient way to master multiple tasks. If task is already mastered, its control policy can be fully or partially shared with other tasks to guide tasks requiring similar skills to be quickly learned. Instead of each task generating trajectories constantly by its corresponding control policy, as in most existing MTRL algorithms, we consider using control policies of other tasks to generate training data for the current task when appropriate. To achieve this goal, we propose a novel framework called Cross- Policy Guidance (CTPG), which extra learns guide policy for each task to identify beneficial policies for guidance. We illustrate the trajectory generation process of Task 1 in Figure 2 For this task, its guide policy ITς selects a policy T' from the candidate set of all control policies ( { every fixed K timesteps. then uses x' as the behavior policy to interact with the environment and collect data for the next K timesteps.
3

---

The CTPG framework alters only the data col- lection process, guiding the control policy train- ing through better exploration trajectories. In
Canditure (1))
Choose lehavior policy frrm candicbes set (T))..
Guile Policy II?
Inpur
Control Policy
Section 4.1 I we introduce the guide policy in
detail and propose hindsight off-policy correc- tion mechanism for its training. In addition, we propose two gating mechanisms to enhance the efficiency of CTPG: the policy-filter gate dis- cussed in Section 4.2 and the guide-block gate
Behavior Polacy
Truin batch
2. Koll out for timodops
3. Collect
tarsitions
Eavirnnten
Replay Buller
Figure 2: Overview of the CTPG framework.
detailed in Section
4.1 Guide Policy
The control policy (x(()()))) of task maps the state & to the environment action ag and its corre- sponding Q-value function Q5(s,, ae) estimates the expected return. The guide policy I1 (ir(st) of task i outputs task index jt E T, and the corresponding control policy Tj. of task je serves as the behavior policy. The guide Q-value function ()( (t) estimates the expected return of using 11; to select different control policies for every K timesteps, with its Bellman equation defined as follows:
B (st, A R. (8t, J+) + 7 Be-- P [Q (8t+ K, kj)(k))]
(5)
where RTT is the Bellman operator of the guide policy 11,, and the corresponding reward function R. is defined the expected cumulative discount rewards for behavior policy Hj, over K timesteps:
*+K- Σ
RC (Sc. jt) Bag " P
7" ("
(6)
For each task i, during trajectory generation, CTPG first utilizes its guide policy IT to sample behavior policy Ty. from the candidate set of all tasks' control policies:
je 11 (-1ss,,
and then samples actions using the behavior policy TJ. for the next K timesteps:
ar TJe (-Is)),
(8)
where t' E +t,t+1,.... t ++ K - 1}. After each timestep, we obtain the reward ry = (sv. ae) and the next state sy+1- The transition si, Sy, ae, SV-11) is stored in the replay buffer.
The guide policy is trained maximize the expected return of the current task by choosing appropriate control policies in certain states. If the control policies of some tasks already proficient in specific skills can be shared with the current task in similar states, the guide policy can quickly learn to use these control policies in those states to generate better training data for the current task. The guide policy and the control policy are trained simultancously. We can use any off-policy RL algorithms for control policy training and any on/off-policy RL algorithms for guide policy training. In this work, we use SAC (9) for the control policy training and the discrete action space variant of SAC (6 for the guide policy training. Given that the guide policy acts every K timesteps, its training frequency is 1/K K that the control policy. The detailed pscudo-codes for the control policy and guide policy training are provided in Appendix A.I (Algorithms
D and 2).
Hindsight Ofr-Policy Correction. During off-policy training, the guide policy faces a non- stationary challenge. Since the control policies are continually updated during the training of the guide policies, the actions chosen by the behavior policies during data collection may no longer align with the improved corresponding control policies, thereby compromising the validity of the training experience. We address this concern by implementing a hindsight off-policy correction mechanism that reassigns the action sampled by the past guide policy to new one j/. whose control policy Ty
more likely to output the historical action sequence {ay Ve Specifically, we utilize maximum likelihood estimation following:
K+K 1
(+K-1 Σ
arg max
II
(s(ar(s)) = arg max
log 1 (agl)s)))
In this way, we can leverage past experiences effectively to train the guide policy. The workings of the hindsight off off-policy correction mechanism in SAC are detailed in Appendix C

---

4.2
Not All Policies Are Beneficial for Guidance
In Section 4.1 we set the guide policy's action space as the set of all control policies { N However, not all control policies within the action space of πI are beneficial for task i in state st, Some control policies perform even worse than the current task's own control policy wy rendering them ineffective for guidance. To address this issue, we design a policy-filter gate to refine the action space of the guide policy by adaptively filtering out unfavorable control policies in state 8t. The trajectory generation process solely using the current task's control policy can be regarded as equipped with special guide policy I1 that exclusively selects ni the behavior policy, i.e., for any st. The guide Q-value Q7 of LII. defined by Equation is:
11 (ils;)
0 (s+, i) = R2 i) + BESK (+LKK.))))
Σ
y Rss,,ss + + (
(10)
1' -' R3(,,
= -
l'=zt
= VV((s)).
where we repeatedly expand O to find that Qf i) is equal to Vi(ss), the state value function of task i's control policy. Because the value function can serve as a filter for high-quality training data (17 22 333, it becomes intuitive to judge the quality of the behavior policy Tj following guide policy and the current task's control policy n; by directly comparing Q (st. and Ví(s)). In our implementation. we estimate (i(ss) via Monte Carlo sampling of Qi(st, 0t) with ~ (a (t) (I5). Relying on this mechanism, the policy-filte filter gate serves as a mask vector m(st) to indicate whether each control policy is beneficial for guidance in state s,. Specifically, each element of m(ss) is:
my(st) = [L, (s,, 2 Vi(st), (o, (s,,, < Vi(s;),
for E (1,2....,NN) NJ,
(11)
where j indicates the element index and task index. Then, the behavior policy ny is sampled by:
jt Normalize (TT(+)))) m(s:))
(12)
If none of the control policies are beneficial for guidance, i.e,, m(s:) = 0, it indicates that the current task's control policy ,, the most proficient within the current state, rendering other control policies unnecessary for enhancing trajectory generation.
Comparable Guide Q-Value. Typically, RL algorithms estimate Q-values to approximate the expected return of the current state-action pair, allowing for the calculation of the policy-filter gate in Equation II through a direct comparison of (s;)) and Q (e) However, in maximum entropy RL algorithms such as SAC, Q-value estimation incorporates the maximum entropy objective, leading to the incomparability of two policies with different entropy objectives. Therefore, we learn another comparable guide Q-value Q with discounted entropy of n, following:
r+t-1 Σ
(st.8)) = Eap ~ N Pe
n" (Ri(sv, ((
(13)
+ 7 10
(((ck)
Since both (st, jt) and Vs(s:) estimate the return with the entropy of the current task's control policy, they can be directly compared to assess whether control policies are beneficial for guidance. A detailed comparability analysis of this comparable guide Q-value in SAC provided in Appendix B
4.3 Not All Tasks Need Guidance
When simultaneously learning multiple tasks, easy tasks converge faster than difficult ones. The control policies of easy tasks allow for the quick acquisition of some effective skills, which may
5

---

Q
#t
Hi
2
I1
Je
Roll out K timestops
8t
Oi''
(1, Sp' ay. Spis)
Yes
Ot''
(6, Byy de, Te, 4p
Control Policy
Poliey-Filter
No
Eavirocimnn
Replay Buffer
4 T
Guide Policy
Guide-Block
r' (1,24, 1
Figure Illustration of the comprehensive CTPG framework. Initially, the guide-block gate selectively provides guidance on tasks E T9, Subsequently, the policy-filter gate generates a mask m to sift through the beneficial policies. Finally, the policy chosen by the guide policy the control policy of the current task itself interacts with the environment over K timesteps to collect training data.
be helpful in exploring other tasks. However, these mastered easy tasks do not need additional guidance from other policies; instead, they focus on further solidifying their already acquired skills. Therefore, not all tasks require assistance from the guide policy.
Based on the above analysis, we design another guide-block gate to prevent the guide policy from engaging in tasks that do not necessitate guidance. This mechanism is directly related to SAC's temperature coefficient ag, For difficult tasks iaf their control policy entropies H. (Ta (-|s)) tend to be high, and the corresponding temperature parameters d'an decrease according Equation 4 Conversely, the temperature parameters (ayy increase for easy tasks leas. Therefore, a is metric reflecting the relative difficulty and mastery of different tasks. We form the tasks that require guidance into a subset T following:
1 N
T
il log CX; S
log jj
(14)
which selects the tasks by comparing the difficulty of the current task versus the average of all tasks. For tasks i & T", the guide-block gate restricts them from using the guide policy, so they only use their own control policies interact with the environment. Consequently, the guide policy can stop training with samples from the tasks i & T and focus on learning the guidance for unmastered tasks.
The comprehensive CTPG framework, with two special gating mechanisms, is summarized in Figure The complete pseudo-code for CTPG is described in Appendix A.2 (Algorithm 3)
3
Experiments
The experiments are designed answer the following research questions: Q1: Can implicit knowl- edge sharing approaches be combined with CTPG to further improve performance? Q2: Does the guide policy in CTPG learn useful sharing policies? Q3: How does each component within CTPG contribute to the final performance? Q4: How does CTPG perform without implicit knowledge sharing approaches? Q5: Can CTPG expedite the exploration of new tasks effectively?
5.1 Environments
We conduct experiments on MetaWorld manipulation and HalfCheetah locomotion MTRL bench- marks, selecting two setups for evaluation within each benchmark.
Meta World Manipulation Benchmark. The MetaWorld benchmark
31 consists of robotics
manipulation tasks employing sawyer arm in the MuJoCo environment MetaWorld-MT10, comprising a suite of 10 tasks, and Meta MetaWorld-MTSO comprising a suite of 50 tasks. Following the settings in [9), the goal position is randomly reset at the start of every episode. We use the mean success rate as our evaluation metric, which is clearly defined in the environment.
24 It provides two setups:
HalfCheetah Locomotion Benchmark. The HalfCheetah is 6-DoF walking robot consisting of links and joints connecting them in the MuJoCo environment (24). The multi-task benchmark HalfChectah Task Group ||] contains different HalfCheetah robots. HalfCheetah- includes 5

---

Table 1: Quantitative result of five classical implicit knowledge sharing approaches combined with different explicit policy sharing methods. The two HalfCheetah locomotion environments are mea- sured on episode return, and the two Meta manipulation environments are measured on success rate. We highlight the pest-performing explicit policy sharing method in bold and annotate the best combination of two information sharing methods with boxes.
Environment
Method
MTSAC
MHSAC
PCGrad
SM
PaCo
Base
9.16 + 0.42
8.68 + 0.55
9.57 + 0.73
9.57 0.21
7.18 + 0.44
HalfCheetah
QMP
8.81 = 0.22
9.09 = 0.64
9.46 + 0.57
10.09 0.53
7.83 = 0.28
MT5 (x le3)
w/ CTPG
9.59 0.40
9.25 + 0.12
10.27 0.40
0.34 10.47
7.95 = 0.47
Base
9.00 + 0.88
8.90 + 0.60
10.17 + 1.06
10.05 + 0.55
8.44 + 0.56
HalfCheetah
w/ QMP
10.00 = 0.47
9.61 + 0.54
10.65 + 0.43
10.41 = 0.61
9.28 + 0.48
MT8 (x le3)
w/ CTPG
10.17 + 0.31
9.82 + 0.40
11.09 + 0.50
10.81 = 0.51
9.02 + 0.48
Base
62.72 + 6.19
63.51 + 2.97
69.62 + 4.04
74.52 + 2.29
69.77 + 7.28
MetaWorld MT10 (%)
w/ QMP
64.91 + 8.82
65.87 + 3.05
67.53 + 2.93
69.78 + 7.50
69.84 + 3.49
w/ CTPG
75.76 + 3.82
74.94 +: 2.97
73.31 + 3.66
78.97 2.41
70.40 + 3.62
Base
47.51 1.95
$2.04 + 2.78
52.85 4.12
55.04 2.84
59.46 + 5.14
Meta World MT50 (%)
QMP
47.82
1.62
51.79 + 4.83
54.05 1.39
55.91 5.08
53.81 + 2.00
w/ CTPG
55.97 + 2.56
56,91 + 2.57
58.91 + 2.10
66.24 + 3.37
68.10 + 3.44
tasks under various scales of simulated earth-like gravity, ranging from one-half to one-and-a-har a-hl of the normal gravity level. HalfCheetah-MPT includes 8 tasks with various morphology of a specific robot body part. We use the episode return as our evaluation metric.
Further information regarding environmental setups is provided in Appendix D
5.2 Performance Improvement on Implicit Knowledge Sharing Approaches
CTPG is generalized MTRL framework adaptable to various implicit knowledge sharing approaches, wherein both the control policy and the guide policy use a unified network. Specifically, in our implementation, the unified control policy employs the same network structure and update procedure as the implicit knowledge sharing approaches, and the unified guide policy utilizes a straightforward multi-head structure for parameter sharing.
To answer Q1, we choose five classical implicit knowledge sharing approaches: (1) MTSAC extends
SAC for MTRL by employing one-hot encoding for task representation. (2) MHSAC utilizes
shared network backbone apart from independent heads for each task. (3) PCGrad 33 resolves
issues arising from conflicting gradients among tasks through gradient manipulation. (4) SM 27
trains a routing network to propose weights for soft module combinations. (5) PaCo 22 learns a compositional policy where task-shared parameters combine with task-specific parameters to form task policies. Combined with these implicit knowledge sharing approaches. we compare CTPG
against: (i) Base represents the source version of these approaches. (ii) QMP 32] employs one-step Q-value filter to identify shareable behaviors.
We train all combinations with 0.8 million samples per task in the HalfCheetah locomotion benchmark and 1.5 million samples per task in the Meta World manipulation benchmark. Each combination is trained 5 times with different seeds. We evaluate the final policy over 100 episodes per task and report the mean performance and standard deviation across different seeds Table
Based on the experimental findings, it becomes evident that, except for the PaCo algorithm in HalfCheetah-MTS, the combination with CTPG leads notable enhancement in performance across all scenarios. QMP does not perform as well as CTPG because QMP only offers single-step behavior guidance, whereas CTPG's guide policy learns long-term policy guidance. Notably, as the number of tasks increases, CTPG exhibits superior performance due to the higher probability of explicit policy sharing between tasks, facilitating more effective policy exploration. Besides the final performance evaluation, comprehensive training curves presented in Appendix E.
7

---

Guide
Pog Sig--wm---ee Sodee
30.26
Pice Flace
36.5%
04.5%
Pak-Paa
30.5
Pac-
36.1%
Pkk
21.65
Peg frsw Side
30.
Proh
7%
Punre Fros Typleon 30 15
Drowe Clas
24 7%
Pal
4.4%
Iosen Stde
17%
Beae Coos
Barton www.o-rroooppoaa Topdows 71%
Polley-
Deg
Peg
Toydras
Figure 4: We display the state of task Pick-Place at every 10 timesteps, along with the corresponding output probability of the guide policy and the actual sampled behavior policy. Except for employing the Pick-Place task's 's control policy during timesteps 20 to 30, the guide policy selects control policies of other tasks for the remaining timesteps, successfully accomplishing the task.
(a) Ablation on Policy-Filter Gate
(b) Ablation on Guide-Block Gate
(c) Ablation on Hindsight Correct
Success Rate (%)
40
30
Polcy-Pliter Gats
20
w/s Hate Mornt
20
w/jo Hirhight Cornection
Samples/ Task /Task (x10²)
Samples Taauples/Taskk (x10))
Samples Taapple/TTskk /Tusk (xl0²)
Figure 5: Three distinct ablation studies of MHSAC w/ CTPG on MetaWorld-MTIO.
5.3 Guidance Learned by Guide Policy
To answer Q2, we visualize the task Pick-Place with guidance on Meta Mord to show the specific role the guide policy. Since the guide policy only used in trajectory generation during training, we visualize one of the sampled trajectories in Figure g We only present the initial 50 timesteps of this trajectory, which the task has essentially been completed.
To better understand policy sharing between tasks, we first explain the relevant tasks. Pick-Place: Pick and place puck to a goal. Sege Insert a peg sideways. Push: Push the puck goal. Button-Press- Toodown: Press a button from the top. Drawer-Close: Push and close a drawer. Reach: Reach a goal position. The visualizations of these tasks are provided in Figure B (Appendix D.I.
The initial and final 20 timesteps showcase that the guide policy learns useful guidance to successfully complete the source task. In the initial 20 timesteps. Pick-Place and Peg-Insert-Side employ shared policy directing the robotic arm toward the target object. In the final 20 timesteps, the task is executed using Prenon--rre--Todooo
to raise the gripper and then Drawer-Close to move forward. Interestingly, although the task in the final 20 timesteps intuitively aligns with Reach, the guide policy opts not to use it because the learned Reach's 's control policy always opens the gripper, causing the puck to fall. In the middle 10 timesteps, the probability of Pick-Place is notably high due to the absence of alternative shared policies at this stage. Specifically, the most similar Reg-Insert-Side grabs the end of the peg for insertion, while Pick-Place requires gripping the puck centrally.
5.4 Ablation Studies
To answer Q3, we conduct detailed ablation studies to analyze the impact of each component in CTPG on performance enhancement. CTPG contains three key components: (1) the policy-filter gate, (2) the guide-block gate, and (3) the hindsight off-policy correction mechanism. We study the influence of each component using the MHSAC implicit knowledge sharing approach
MetaWorld-MT10.
(1) Figure 5]a) shows that the policy-filter gate plays a significant role within CTPG, leveraging the value function to constrain the direction of guided exploration. (2) Given the precise definition of binary-valued success signal in the Meta World benchmark, we compare the SAC temperature metric, described in Section 4.3 with another success rate metric to determine if tasks need guidance.

---

Specifically, we block guidance for tasks with success rates exceeding 80% during evaluation. Figure 5(b) illustrates that both metrics showcase competitive performance, exhibiting clear performance gains over no guide-block gate. However, since the success rate is human-defined metric and difficult to define for some tasks (e.g., HalfCheetah), the SAC temperature metric is more general across most environments. (3) We ablate the hindsight off-policy correction mechanism used in guide policy training in Figure 3 c). elucidating its improvement in training efficiency and stability. Further are provided in Appendix E.2
ablation studies of SM with CTPG
5.5 CTPG without Implicit Knowledge Sharing
To answer Q4, we train N control and guide policies independently on N tasks without im- plicit knowledge sharing. We perform experi- ments on HaliCheetah-
HallChestah-MTT
(b) Meta Morld-M100
Epicodoo Rerury (x(0)
Nuccew Rate (%5
and MetaWorld- MT10, employing the same environment con- figuration as in Section [.2 As illustrated in Figure 6 CTPG significantly improves the per- formance of Single-Task SAC. Notably, the im- pact of CTPG on Mo0 is more pro- nounced. The potential rationale is that the tasks within HalfCheetah-MTT exhibit minimal vari- ance in difficulty, resulting in a narrow gap in learning progress. Conversely, Meta MetaWorld- World-MT10 10 presents broader disparity in task difficulty, where CTPG facilitates the guidance from simpler to more challenging tasks, thus appearing more efficient.
Yack 540 C1P0
Jaale Zark
Sumples Tusk (x.00)
Sumples TTalk (x11F)
Figure 6: CTPG also improves performance in the absence of implicit knowledge sharing upproaches.
5.6 Exploration of New Tasks with CTPG
To answer Q5, we split the original task set in half, pre-training expert policies on the one half
(a) HalfCheerah-MTT
(b) MetaWorld-MTT0
Epicade Beturn (xto))
T.. For the other half, we compare direct learn- ing with CTPG, where the agent leverages guid- ance from both the control policies being learned and the expert policies. Specifically, SAC on HalfCheetah-MTT
MT- where T includes all tasks that enlarge the size of body parts. On the other hand, we use SM on MTr with T containing tasks indexed from 0 to 4. Figure]Jindicatee that, compared to learning the new set of tasks from scratch, CTPG can reasonably utilize experts to explore new tasks rapidly. Notably, in environments where task similarity is high, such as HalfCheetah-MTTS CTPG can quickly transfer the expert's 's abilities to the control policy being learned with the guide policy.
AERRAC
Samplex/ Taok (x100)
Samphe, Took (x:10P)
Figure 7: CTPG with expert policies can expedite the exploration of new tasks effectively.
Conclusion and Discussion
This paper proposes the Cross-Tass Policy Guidance (CTPG) framework for the MTRL explicit policy sharing. CTPG contains a guide policy and two special gates to identify beneficial sharing policies from the set of all task control policies and choose the most proficient one to generate high-quality trajectories for the current task as policy guidance. In addition, CTPG is generalized framework adaptable to diverse implicit knowledge sharing approaches. Empirical evidence showcases that these approaches combined with CTPG further improve sample efficiency and final performance.
Limitations and Future Works. One limitation of CTPG is its reliance on predetermined guide step K, necessitating hyperparameter tuning for K across different environments. Moreover, the fixed guide step setting lacks flexibility, as the duration of shared skills execution timesteps varies inconsistently among different tasks. Consequently, exploring methods to automate the selection of the guide step K presents an intriguing avenue for future research. Additionally, irrespective of the human-defined win rate and the unique temperature parameter of SAC, investigating alternative metrics for the block gate emerges as another important direction for future endeavors.

---

7 Acknowledgements
This work is supported in part by the National Science and Technology Major Project (022ZD0116401) the Natural Science Foundation of China (Grant Nos. 62076238, 62222606, and 61902402), the Key Research and Development Program of Jiangsu Province (Grant No. BE2023016), and the China Computer Federation (CCF)-Tencen) Open Fund.
References
[1] Richard Bellman. Dynamic programming. Science, 153(3731):34-33 37, 1966.
[2] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. CoRR, abs/1606.01550. 2016.
[3] Rich Caruana. Multitask learning, Machine Learning, 28:41-75, 1997.
[4] Zhao Chen, Vijay Badrinarayanan, Chen-Y Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of International Conference on Machine Learning, pages 794-803, 2018.
[5] Myungsik Cho, Whiyoung Jung, and Youngehul Sung. Multi-task reinforcement learning with task representation method. ICLR Workshop on Generalizable Policy Learning in Physical World, pages 1-11, 2022.
[6] Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR, abs/1910.07207, 2019.
[7] Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine Divide- and-conquer reinforcement learning. In Proceedings of International Conference on Learning Representations, pages 1-10, 2018.
[8] Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of International Conference on Machine Learning, pages 1352-1361, 2017.
[9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of International Conference Machine Learning, pages 1856-1865, 2018.
[10] Jinmin He, Kai Li, Yifan Zang, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Not all tasks are equally difficult: Multi-task deep reinforcement learning with dynamic depth routing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12376-12384, 2024.
[11] Peter Henderson, Wei-Di Chang, Florian Shkurti, Johanna Hansen, David Meger, and Gregory Dudek. Benchmark environments for multitask learning in continuous domains. In ICML Workshop Lifelong Learning: A Reinforcement Learning Approach, pages 1 1-6, 2017.
[12] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):13341373, 2016.
[13] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of International Conference on Learning Representations, pages 1-10, 2016.
[14] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averss gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, pages 18898- 2021.
[15] Samuel Lobel, Sreehari Rammohan, Bowen He, Shangqun Yu, and George Konidaris. Q- functionals for value-based continuous control. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8932-8939, 2023.
10

---

[16] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529553. 2015.
[17] Ashvin Nair, Bob McGrew, Marcin Andrychowiez, Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In Proceedings of International Conference on Robotics and Automation, pages 6292-6299, 2018.
[18] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming.
John Wiley & Sons, 2014.
[19] Sebastian Ruder. An overview of multi-task learning in deep neural networks. CoRR, abs/1706.05098, 2017.
[20] Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acqui- sition in multi-task reinforcement learning. In Proceedings of International Conference on Learning Representations, pages 1-11, 2018.
[21] Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In Proceedings of International Conference on Machine Learning, pages 9767-9779, 2021.
[22] Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. PaCo: Parameter- compositional multi-task reinforcement learning. In Advances in Neural Information Processing Systems, pages
2022.
[23] Yee Teh, Victor Bapst, Wojeiech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pages 4496-4500 2017.
[24] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In Proceedings of IEEE/RS International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[25] Nelson Vithayathil Varghese and Qusay H Mahmoud. survey of multi-task deep reinforcement learning. Electronics, 9()):1333, 2020.
[26] Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciñski, and Piotr Miłoš. Disentan- gling transfer in continual reinforcement learning. Advances in Neural Information Processing Systems, 35:6304-6317, 2022.
[27] Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In Advances in Neural Information Processing Systems, pages 4767-4777, 2020.
[28] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering complex control in MOBA games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6672-6679, 2020.
[29] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. In Advances Neural Information Processing Systems, pages 11501-11516, 2021.
[30] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances Neural Information Processing Systems, pages 5824-58848 2226, 2020.
[31] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- World: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Conference on Robot Learning, pages 1094-1100, 2020.
11

---

[32] Grace Zhang, Ayush Jain, Injune Hwang, Shao-Hua Sun, and Joseph Lim. Efficient multi-task reinforcement learning via selective behavior sharing. CoRR, abs/2302.00671, 2023.
[33] Jin Zhang, Siyuan Li, and Chongjie Zhang. CUP: Critic-guided policy reuse. In Advances in Neural Information Processing Systems, pages 27537-27548, 2022.
[34] Yu Zhang and Qiang Yang. A survey multi-task learning. IEEE Transactions Knowledge and Data Engineering, 34(12):5586-560) 5609, 2021.
12

---

A
Pseudo Code
Without implicit knowledge sharing approaches, each task has its control policy и, and guide policy Th. With implicit knowledge sharing approaches, there a unified control policy и and guide policy 11 with the imput of task representation 2;. both cases, we uniformly represent tasks subscripts and omit the network parameters for a clear presentation in pseudo-code. Specifically,
The actor network with parameter á for task i's control policy is denoted as π. (ax(8s)) The critic network with parameter for task i's control policy is denoted as (i( at).
The temperature parameter for task i's control policy is denoted as
The guide actor network with parameter d for task i's guide policy is denoted as (je(s))
The guide critic network with parameter 0 for task i's guide policy is denoted as Q (tt.
The guide temperature parameter for task i's guide policy denoted a..
The comparable guide critic network with parameter (i for task is denoted as Q (st, Jt).
Algorithm 1 Control Policy's 's Training Step
Input: minibatch B contains (i, St, ag, rts $(41) Initialization: actor network т,, critic network Q. SAC temperature a
1: Optimize 0 with SAC's critic loss in Equation
(Q( = (
2 a)) (rr (V((())))))
2: Optimize o with SAC's actor loss in Equation
Jx (Ø) (i [Ba, [a; log (.(()ss)) (st,, a:)l
Optimize with SAC's alpha loss in Equation
J(a) EiBB
Barwn [-a log (ae(s)))) a:Hl]
Algorithm 2 Guide Policy's Training Step
Input: actor network æ, SAC temperature a, minibatch 59 contains (fse, ar, re Y+ 1, $14K) Initialization: guide actor network II9, guide critic network Q9, guide SAC temperature a
I: Hindsight off-policy correct je into jú following Equation
C-K-1
j arg mnax
log F (a(arlss)
P=t
2: Calculate the guide reward r following Equation 6
t+
-r
3: Optimize 8 with SAC's critic loss for guide policy
Jop (09) Reisee ( 2/ (QI()))))) + [V++,*(((***))))
4: Optimize de with SAC's actor loss for guide policy
Jn (d9) (B [B (a? log (i)()))))) Q (s(.
5: Optimize 9 with SAC's alpha loss for guide policy
J [aa)) Er~B [B [-a] log 11 a.H1
13

---

A.1 Pseudo Code of Policy Training
Algorithm [ describes a training step for the control policy given a minibatch of data B, which contains a batch of tasks i, states St, actions rewards and next states St+1.
In addition, Algorithm 2 describes training step for the guide policy given minibatch data B The data 8 not only contains batch of tasks i, states 8t, actions Je taken by guide policy, and K' timesteps ahead states St+KK but also contains rewards fry hr 1t+K- accrued over timesteps for calculating the guide reward. Moreover, B includes states {sy Jt'=t 1 and actions {ar Jt'=t taken by control policy over the timesteps for hindsight off off-policy correction.
K
A.2 Pseudo Code of Comprehensive CTPG
The full pseudo-code of CTPG is described in Algorithm
Algorithm Cross-Task Policy Guidance
Input: guide step K Initialization: actor network x. critic network Q, guide actor network II9, guide critic network Q?, comparable guide critic network Q", replay buffer D <-
1: for each epoch do
2: Update the task subset T requiring guidance with
N log Orj
T
il log o3
N
3:
// Inference Stage
4:
for each task do for each timestep / 0, 1,... do if % K then if & T then Behavior policy n' n3
5:
6:
7:
>> Guide-Block Gate
8:
9:
else
Get comparable guide Q-values (st,
Get control V-value (s;)) with Monte Carlo sampling control Q-values Calculate policy-filter gate mask m following
12:
Policy-Filter Gate
[1. (( 2 V((ss)
my 0. (s,,)) < Vi(+t)
13:
Sample nt' +- TJ. with j. Normalize (TQ (- m)
D. Guide Policy
end if
15:
end if
16:
Execute action ~ (-(s)) and get reward r2 and next state 8t41 Store the transition into D
17:
18:
end for
19:
end for
20:
// Training Stage
21:
for each training step t 0, 1...., do
22:
Sample a minibatch B all tasks from D Optimize and 0 with B as described in Algorithm if L % K 0 then Sample minibatch 8 of task subset T from D Optimize @ and (5 with 8 as described in Algorithm 2 Optimize 6 using Bellman Equation in Equation 13 with 1
23:
24:
25:
26:
27:
end if
29:
end for
end for
14

---

B Details on Comparable Guide Q-Vaue
We expand the value function of SAC to obtain: QQ; a:) a, log I. (arls)))
(st)
Ear
Ear
Qi((; az) + (-H((T(((sst)
Ea Bar Ri(se, ac) + 7 [V((++1) + o,H((m((ss:))
Earwn
Ri(se, ar) a. (- + - [Vi(st+)]]
(15)
Ea Bay (
Σ , (Ri(se, or) + oz, +m
P'=t
By repeatedly expanding V,, we get the final form, which is actually the optimization objective under the maximum entropy reinforcement learning framework.
Then, consider that the trajectory generation process solely using the current task's control policy T, can be regarded as equipped with a special guide policy 11@ in SAC. The comparable guide Q-value in Equation 13 of this special guide policy II is:
-+K-1
Q (8), B
(Ri(sr, (.H((((()))))))) + KB B. [09(
tIK
Eag
7" (R(r(,()))))))) a. + +
(((+++xx))
2
(-1 (Ri(sr, (aak)aar))+((((())))))
(16)
The ellipsis part is the repeated expansion of Q Therefore, in SAC, the policy-filter gate can refine the action space of the guide policy by directly comparing Q (st.Je) and Vi(sz).
C Details on Hindsight Off-Policy Correction for SAC
The hindsight off off-policy correction mechanism is proposed mitigate the non-stationarity challenge of the guide policy update process. It reassigns the action je sampled by the past guide policy to a new one j/. whose control policy πyद is more likely to output the historical action sequence {ay 1++K- During the guide policy update process, we first get the reassigned action j' following the maximum likelihood estimation in Equation 9 For SAC's critic update, the critic loss of the guide policy is,
JQu (99) (
(ot. - (rg + V (((())))))
(17)
where the hindsight correction explicitly modifies the update procedure by shifting the Q-value estimation from (st. Jc) Q (st, jt. For SAC's actor update, the actor loss of the guide policy is.
JI (d") (i..)))B [ [a log 11, (j. s.)) Q7 (st,
(18)
which is not explicitly different from the original function in SAC. However, indirectly affects actor learning due to SAC's unique actor update method [9]. Specifically, SAC's actor loss is derived from its optimization objective, and the policy is updated according to.
exp ( (mtmm((ts))))
new arg min DEL e'lll
x' (-1(s))
(19)
ZTN (st)
where the partition function 7 (sx) normalizes the distribution. In essence, SAC's policy up- dated by fitting the distribution of the SoftMax function of Q-value with temperature a. Therefore, modifying the update of Q jt) to Q (st: using hindsight correction leads a different guide policy actor optimization objective, thus affecting the training of the guide actor network.
15

---

D Environment Details
D.1 MorlW Manipulation Benchmark
Reach
Push
Pick-Place
Door-Oen
Drawer-Open
Drawer-Close
Button-Press Caeton--Pess-oppppe Pross-Topdown
Peg-Insert-Side
Window-
Window-Close
Figure 8: Visualizations of robotic manipulation tasks
The MetaWorld manipulation benchmark [31 consists of 50 robotics manipulation tasks employing sawyer arm to acquire diverse manipulation skills. Meta Worldd-odddddddd encompasses all manipulation tasks, and MetaWorld-MTII encompasses 10 tasks (a subset of MT50), as illustrated in Figure 8 In MetaWorld, the source tasks are configured with fixed goals, limiting the policy's s ability to generalize to tasks of the same type with varying goals. Following similar setup [27], we extend all the tasks to random-goss setting, where both items and goals reset randomly at each episode's s onset. In addition, unlike [27 (30, conduct our experiments on the MctaWorld-V2 V2 benchmark
In addition, extend the episode length 200 timesteps, different from the previous setting of 150. We evaluate the mlle-basdd policies provided by the Morld benchmark across 100 sample episodes. Under the initial episode length setting of 150 timesteps, 42 tasks achieve success rate exceeding 90%. However, among the remaining 8 tasks, the task Disassemble exhibits mere 50% success rate. contrast, upon adjusting the episode length setting to 200 timesteps, success rates for all 50 tasks exceed 90%, with the task Disassemble achieving an improved success rate of 91%.
D.2 HallCheetah Locomotion Benchmark
BigTorso
BigThigh
Bigl.eg
BigFoot
SmallTono
SmalTThih
SmallLeg
SmallFoot
Figure Visualizations of robotic locomotion tasks HallCChctah-MTT8
16

---

The HalfCheetah locomotion benchmark originates from the gym-extensions framework [ which focuses on continuous control multi-task reinforcement learning tasks, including several task groups of the standard gym environments (2]. We run our experiments with the HalfCheetah agent, which is a 2-dimensional robot consisting of 9 links and 8 joints connecting them (including two paws). Each episode contains 1000 timesteps.
HailCheetah-MTS consists of 5 locomotion tasks, The HalfCheetah agent is tasked with run- ning in environments with various scales of sim- ulated earth-like gravity, ranging from one-half to one-and-a-hai of the normal gravity level. The detailed gravity value of each task is illus- trated in Table
Table 2: The gravity setup of HaliCheetah-MTS.
Task Name
Gravity (m .s s
GravityHalf
4.91
GravityThree ThreeQuarters
7.36
GravityNormai
9.81
Gravity One AndQuarter
12.26
HalfCheetah-MT3 consists of 8 locomotion tasks, as depicted in Figure The HalfCheetah agent must run with variations in the morphol-
GravityOneAndHalf
14.72
ogy of specific body part, such as the torso, thigh, leg, and foot. The Big body part involves scaling the mass and width of the limb by 1.25, and the Small body part involves scaling by 0.75.
Blo
w/ QMP
w/ CTPG
10:
(3) MTSAC
Sccouse Bare (9)
Stacoss Moten (1)
Eataada Racura
8
10
E
Egioook
Samples/Tak (e10)
Samples, (x100)
Sumples Took (x 109)
Sumple (x10P)
(2) MHSAC
Epiaadh Rutumm (x))))
Rpiasd Retum (x:10))
15
Surous Raie (9)
8 16
Succome Httte (1)
Samples, Taak (310))
Samplle, Task (S:07)
Samples Taak (SIP))
Samplee Thak (s
Eptoode Roturn (2.07)
Sterra Eate (4)
5 8 4
Stecems Made (1)
(3) PCOnd
Egioude Rotur
Sumples: (K10)
Taok
Sumppee Teak
Eyiaole Retura (x109)
Eptastle Testrra Exant)
Suacon Hare (F)
Stacos Hote (1)
(4) SM
30
Samplax/lak (s10)
Samples Tbak (xII1)
Samples Taak (xIIP)
Samplen, Dhak (x
Epiook Rotsm (x00))
Rpiaude Rutum Exarg
on 15
Surcau Rate (92)
8 20
Stccore lote (50
Sumples Taak Ex107) (a) HallChoetah-MIT
Samplee,TTee TTok (s:17)
Samples /Takk (S:IP)
Komples Taak (s (d) McraWorld-MTSO
Halfhhetah--MTS
(c) MetWoordd-M100
Figure 10: Training curves of experiment with implicit knowledge sharing approaches. Beyond the
ultimate performance improvement, CTPG also enhances the sample efficiency.
17

---

E Additional Experimental Results
E.1
Training Curves of Experiment with Implicit Knowledge Sharing Approaches
This section complements the result in Section 5.2 which only presents the final performance of the experiment. Here, we show the training curves for different combinations of explicit policy sharing methods and implicit knowledge sharing approaches across four environments in Figure [0] Each row of the figure represents a distinct implicit knowledge sharing approach, while each column represents a different environment. Within each subfigure, the three curves represent the base one without any explicit policy sharing method and two variations using different explicit policy sharing methods. We evaluate the training policy every 10K samples per task with 32 episodes and report the mean episode return or success rate, along with standard deviation, across 5 different seeds. The result shows that beyond the ultimate performance improvement, CTPG also enhances the sample efficiency.
E.2
Additional Results of Ablation Studies
(a) Ablation Folicy-Filter Gate
(b) Ablation Guide-Block Gate
Ablation Hindsight Correct
Success Rate (%)
90-
20-
w/ln Giidd--llok Gate
90
Policy Flax
Suceeso Rule Melrie
Hind-ight Cuerestio
CIPC
12
12
Samples/ Tassk (x10*)
Samples/Taue (x109)
Samples/ Task (x10*)
/Task
Figure 11: Three distinct ablation studies of SM w/ CTPG on MetaWorld-MT50.
In addition to the ablation experiment of MHSAC with CTPG on MetaWorld-MT1O in Section 5.4 we also conduct ablation studies using SM implicit knowledge sharing approach on Meta MetaWorld-MT50. Figure shows similar results to those in Section 5.4 The policy-filter gate is crucial within CTPG. The guide-block gates with two metrics demonstrate competitive performance and show improvement compared the absence of the guide-block gate. Additionally, the hindsight off-policy correction mechanism significantly enhances training efficiency and stability.
E.3 Ablation Study on Guide Step
Within CTPG, the guide policy II selects a policy nj from the candidate set of all control policies { N every fixed K timesteps. The selected policy Tj, then used as the behavior policy to interact with the environment and collect data for the next K timesteps.
(a) HalsCheetah-MTS
(b) Meta
Episode Return (xc11))
10
Success Rate (X)
407
-01
Samples/ Task (x10²)
Taak
Samples Task Task (x100))
Figure 12: MHSAC w/ CTPG with different guide steps K.
The guide step K is a predefined hyper-parameter. step K the two setups: HalfCheetah-MTT and MetaWorld- MT10. In HalfCheetah-M'T an entire episode contains 1000 timesteps, so we set K E (1, 10, 20, 50}. In MetaWorld-MT10, (1,3,5, 5, 10, 20}. The result, shown in Figure 12
parameter. We conduct ablation experiments on the guide
entire
episode contains 200 timesteps, so we set indicates that both short and long guide steps lead decreased performance and increased variance.
18

---

Overall, CTPG performs well in both environments when K = 10. Furthermore, automating the selection of the guide step K part of our future work.
E.4 Ablation Study Monte Carlo Sampling
In Section 4.2
we design a policy-filter gate to refine the action space of the guide policy by adaptively filtering out control policies that not beneficial for guidance. Specifically, the restriction of the action space is implemented by mask, which is generated by comparing the control policy's 's V-value (s,)) and the guide policy's 's Q-value Q7 (.).
MetaWorld-MT10
Success Rate (%)
10
12
Samples/ (x10P)
Figure 13: MHSAC w/ CTPG with various Monte Carlo sampling times H to estimate V-value.
In SAC, the V-value is the expectation of the Q-value with the entropy of policy. It is formulated as:
Vi(++) = Eng [Qi(s,, (x) - og log (a (ax))s)) [s))
(20)
To avoid introducing an additional network to estimate the V-value Vs(s;), we estimate via Monte Carlo sampling of the Q-values Qi(se, at) with a, 7, (a, (s)) [15]. We conduct ablation experi- ments varying the Monte Carlo sampling times H. as depicted in Figure [3] It is observed that the
performance remains largely unaffected by the number of samples, except in cases where H equals 1, which leads to performance degradation and heightened variance due to excessive randomness.
F,5
Additional Comparison with BPT
CTPG learns flexible mixture policy combining different control policies. To verify the advantages of the mixture strategy, we compare CTPG against another baseline that incorporates guide strategy from [26]. which solely learns from another control policy with the highest performance on the task. Since (26 focuses on continual RL, which requires predefined task sequence and does not align with the MTRL experimental setting. Therefore, we adapt the core idea, implementing a version that selects the best-performing policy for the current task to guide exploration and generate trajectory data. Specifically, after each Hf rounds of data collection, the top-performing policy for each task to guide data collection the subsequent rounds. We refer to this baseline as BPT (Best Performance Transfer) and set H to 50 episodes in our implementation. We use MTSAC in HalfCheetah-MM and MHSAC in
evaluate all policies across all tasks, selecting
(a) HalfCheetah-MTS
(b) MetaWorld-MTio
Episode Retuurn (xc1))
Succcess Rate (%)
XITSAC w/ RPT
MBSAC w/ BPT
MITSAD w/ CTPG
SISSAC w/ OTP
Samples/Tee Task (x10P)
Samples/Tae Task (xx1P)
Figure 14: The result of additional comparison with BPT.
The result, shown in Figure 14 demonstrates that CTPG outperforms BPT. CTPG's guide policy learns more flexible strategy: it not only can learn to share single policy within complete
19

---

trajectory (like BPT), but also can develop mixture policy by combining different control policies, making more transferable between tasks. Notably, BPT performs better in MelliCheetah-MTT than in MetaWorld-MTT because the tasks in HalfCheetah-MTTT share more significant similarities, whereas the policy transfer and guidance in MetaWorld-MT10 require combination strategies.
E.6 Adaptability of Other RL Algorithms to CTPG
CTPG a general MTRL framework that can adapted with other RL algorithms, even allowing the control and guide policy to different RL algorithms. Since SAC is widely used algorithm in continuous control and serves as the base algorithm for all baselines, we also choose SAC as the base algorithm in this work. In addition, we explore the adaptation of TD3 with the CTPG and also employ different RL algorithms for the control and guide policies. Specifically, the control policy uses TD3, and the guide policy uses DQN. We evaluate MTTD3 (the original TD3 using one-hot encoding for task representation) on HalfCheetah- MT88 and MHTD3 (utilizing a multi-head network for different tasks) on Meta World-MT10. The result shown in Figure demonstrates that CTPG can enhance performance and sample efficiency when combined with other backbone RL algorithms.
(a) HalfCheetah-MTS
(b) MetaWorld-MTIo
Episode Return (x110))
Rate (SE)
10
60
30
Samples/Tes (x10")
Samples/Tasi (x10F)
Figure 15: The result of CTPG based on TD3 RL algorithm.
F
Implementation Details
We implement all experiments using the MTRL codebase (2] 1 and access the HalfCheetah locomo- tion environment to this framework. One significant change make involves removing the reward normalizer wrapper, which leads to worse results.
F.1 Details on Computational Resources
We use AMD EPYC 7742 64-Core Processor with NVIDIA Geforce RTX 3090 GPU for training. Each method is trained times with different seeds. Using Meta -orld as an example, the training time required for a full training run varies from 16 hours (MTSAC) to 52 hours (PeGrad) for baseline implicit knowledge sharing approaches. For each approach with CTPG, the training times range from 20 hours (MTSAC w/ CTPG) to 56 hours (PeGrad w/ CTPG). The guide policy, using simple multi-head network architecture, acts every K step (K training frequency is set as 1/K. Consequently, CTPG does not significantly increase the training the guide policy, please refer to Appendix F3
in our implementation), so its
time. For a detailed
F.2 Additional MTRL Training Setups
For fair comparison, we use the following common training setups on all methods:
Disentangled SAC temperature parameters. This setup is standard established in MTRL. Herein, distinct tasks utilize varying SAC temperature parameters, as described in Equations 2 and 3 which also optimized by independent losses in Equation 4 Therefore, this setup facilitates the individual adaptation exploration and exploitation balances for each task, effectively accommodating the diverse learning dynamics across different tasks during training.
20

---

Loss maskout of extreme tasks. This setup, utilized by 22], involves the selective masking of the potentially destabilized loss of task from the total loss aiming to mitigate its adverse effects other tasks. Specifically, when the task loss J; surpasses predefined threshold (set as 3e3, the same as [2 that task is excluded from the total training loss.
Multi-task loss rescaling. In recognition of the inherent discrepancy in convergence rates between tasks, where easy tasks usually converge faster, [27] propose an optimization objective weight of task i, formulated as:
exp(-ax)
(21)
2j=1 exp(-aa)
where a; the SAC temperature parameter of task and N is the total number of tasks. Consequently, the total loss is adjusted to J = [,[w; JiJ, ensuring a balanced training process across different tasks.
F.3 Hyper-Parameterd of All Method
This section provides the hyper-parameters each method in our experiment. General hyper- parameters shared by all methods are illustrated in Table Table Table show the additional hyper-parameters specific to each implicit knowledge sharing approach. In addition, the guide policy can also be trained using implicit knowledge sharing methods. In our implementation, simple multi-head network structure for the guide policy in CTPG, and the relevant additional hyper-parameters are displayed in Table 10
use the
Table 3: General
of all methods.
Value
network architecture batch size
feedforward network 128 x number of tasks
non-linearity
ReLU standard Gaussian
policy initialization
# of samples/ / # of train steps per iteration
env step / 1 training step
policy learning rate
le-4
Q function learning rate
le-4
alpha learning rate optimizer
le-4
Adam
discount
.99
episode length
1000 (HalfCheetah) 200 (MetaWorld)
exploration steps
2000
reward scale
0.1
replay buffer size
1e6 (MT5 / MT8 / MT 1e7 (MT50)
Table 4: Additional hyper-parameter of MTSAC.
Hyper-parameter
Value
network hidden layer (HalfCheetah) / 5 (MetaWorld) network hidden size
400
Table 5: Additional hyper-parameters MHSAC.
Value
network architecture network hidden layer
multi-head (1 head / task) (HalfCheetah) 5 (MetaWorld)
network hidden size
400
21

---

Table Additional hyper- hyppe-paramettes of PCGrad.
Hyper-parameter
Value
optimizer network hidden layer
Adam w/ PCGrad
(HalfCheetahh) 5 (MetaWorld)
network hidden size
400
Table Additional hyper parameersss of SM.
Value
state representation
[400] (HalfCheetah) [400, 400] (MetaWorld) [400]
task representation
number of layers
number modules/layer module size routing size
64 (HalfCheetah) 128 (MetaWorld)
128
Table 8: Additional hyper-parameters of Paco.
Hyper-parameter
Value
parameter set number network hidden layer
3
2 (HalfCheetahh) / (MetaWorld)
network hidden size
400
Table 9: Additional hyper-parameters of QMP.
Hyper-parameter
Value
temporally extended length 10
Table 10: Additional hyper- parameters of guide policy in CTPG.
Hyper-parameter
Value
network architecture
multi-head (1 head / task)
network hidden layer
2 (HalfCheetah) 5 (MetaWorld) 400
network hidden size
guide step
10
# of samples of train steps per iteration Monte Carlo sampling time
env step I training step
5
guide policy learning rate guide Q function learning rate
le-4
lc-4
guide alpha learning rate
le-4
G
Broader Impacts
This paper presents work that aims advance the field of Multi-Task Reinforcement Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yss]
Justification: The full code is provided in supplemental material and detailed settings are available in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
The full code is provided supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiipp.ccppulli//uuiii http:/uwdiesseeee CodeeSubeeiieeeee ssionn ceiiionPoicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper speciffy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The statistical significance of the experiments is provided in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The experiments compute resources information is provided in Appendix E.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwr...coooooo.//cccccaaa
Answer: [Yss]
Justification: The research conduct in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses broader impacts Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The code this paper uses is properly mentioned in Appendix with footnotes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "LNTe5P1Hof", "reviewer": "Reviewer_xTD7", "summary": "This paper proposes a method called CTPG to enable policies trained on different tasks to learn from each others' generated trajectories. CTPG operates by learning a guideline policy that determines which control policy in a given set should best generate trajectories to enable agents to learn a particular task. Then, the authors also designed additional mechanisms that (i) deal with how each policy whose trajectories are sampled may change from their learning process, (ii) prevent negative transfer from using trajectories from irrelevant policies for learning, and (iii) promote the convergence towards good policies by preventing any transfer of experience once a policy is already proficient at its intended task.\n\nUsing environments from multi-task reinforcement learning benchmarks, the authors then investigated their method's performance when (i) learning a set of tasks in parallel and learning a new task that is not from the set of previously learned tasks. Furthermore, the authors also conducted further analysis to investigate (i) which component within their learning algorithm is most responsible for the method's performance and (ii) whether the guideline policy learned a sensible experience selection policy.", "questions": "1. How would CTPG perform against a simple transfer learning strategy where experience is only transferred from the policy that has the best performance?\n\n2. Does the behavioral policy chosen in hindsight affect the actor-network training? If it does, how would it do so?", "limitations": "The authors have sufficiently outlined their method's limitations and interesting directions for future work in the last paragraph of the manuscript.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Major Strength - Clarity**\n\nExcept for a few minor clarification details provided below, I find the paper to be well-written. I especially appreciate the clear outlining of (i) the motivation behind the proposed method, (ii) the questions being investigated by the authors, and (iii) the experiment design. I hope that future iterations of the manuscript will keep this same level of clarity.\n\n**Major Strength - Method Soundness**\n\nFrom a high-level perspective, it seems that the learned transfer learning mechanism proposed in this paper seems reasonable. I especially find the authors' design of various mechanisms to address different problems (i.e., policy selection, negative transfer, variable rates of learning between tasks) to be great choices that should improve learning performance in multi-task reinforcement learning scenarios.\n\n**Major Strength - Experiments and Analysis**\n\nI also think that the authors did a great job of thoroughly investigating the efficacy of their method. The five questions in the experiment section provided interesting insights into the method. At the same time, the experiments done to answer each of those questions were well-designed and showed the method's positive performance.\n\n**Minor Strength - Significance**\n\nWhile allowing agents to learn from each others' experience to improve their performance at achieving their respective objectives is not exactly new, to my knowledge, the authors' proposed method for learning in multi-task reinforcement learning (and addressing its various associated issues) is novel. Even if I am wrong on the novelty aspect of this method, I still believe the thorough analysis provided by the authors would contribute to different insights that may be useful for the multi-task RL community.", "weaknesses": "**Minor Weaknesses - Additional Comparisons**\n\nPerhaps another baseline that could be compared is the simple strategy from [1], where one simply learns from another policy with the highest performance at the task. Using this baseline should elucidate the effects of switching between different policies to transfer from as opposed to just transferring from a policy that seems to perform the best.\n\n**Minor Weaknesses - Hindsight Off-Policy Correction**\n\nFrom Section 4.1 alone, I also do not find how the hindsight off-policy correction mechanism affects the remainder of the learning process. While I did check Algorithm 2 in the appendix, I am uncertain why $j^{'}_{t}$ (the behavioral policy chosen in hindsight) only affects SAC's critic updates and not also the policy updates.\n\nReferences:\n\n[1] Disentangling Transfer in Continual Reinforcement Learning. Wolczyk et al. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n> Additional comparison with a simple transfer learning strategy where experience is only transferred from the policy that has the best performance.\n\n[1] focus on continual RL, requiring a predefined task learning sequence, which does not match the experimental setting of MTRL, so we cannot directly compare it with CTPG and other baselines. Therefore, we implement a comparable version based on the core idea of [1] and your description: selecting the policy that performs best in the current task among all task policies to guide exploration in generating trajectory data. Specifically, after each $H$ rounds of data collection, we evaluate all policies across all tasks and choose the best-performing policy for each task to collect the data for the next $H$ rounds. We refer to this baseline as BPT (Best Performance Transfer) and set $H$ to 50 episodes in implementation. We use MTSAC in HalfCheetah-MT8 and MHSAC in MetaWorld-MT10. The result, shown in **Fig.1 of the global Rebuttal Comment PDF**, demonstrates that CTPG outperforms BPT. CTPG's guide policy learns a more flexible strategy: it not only can learn to share a single policy within a complete trajectory (like BPT), but also can develop a mixture policy by combining different control policies, making it more transferable between tasks. Notably, BPT performs better in HalfCheetah-MT8 than in MetaWorld-MT10 because the tasks in HalfCheetah-MT8 share more significant similarities, whereas the policy transfer and guidance in MetaWorld-MT10 require combination strategies.\n\n[1] Disentangling Transfer in Continual Reinforcement Learning. Wolczyk et al. NeurIPS 2022.\n\n> The effect of the hindsight off-policy correction mechanism on SAC's actor network training.\n\nThe hindsight off-policy correction mechanism affects not only the critic update in SAC but also the actor update.\nAlthough the actor loss (Line 4 in Algorithm 2) does not explicitly use relabeled $j'_t$, it indirectly affects actor learning due to SAC's unique actor update method [2]. Specifically, SAC's actor loss is derived from its optimization objective, and the actor policy is updated according to:\n\n$\n\\pi_{new} = \\arg\\min_{\\pi' \\in \\Pi} D_{KL} \\left( \\pi'(\\cdot | s_t) \\left\\| \\frac{\\exp\\left(\\frac{1}{\\alpha} Q^{\\pi_{old}}(s_t, \\cdot) \\right)}{Z^{\\pi_{old}}(s_t)} \\right. \\right),\n$\n\nwhere the partition function $Z^{\\pi_{old}}(s_t)$ normalizes the distribution. In essence, SAC's policy is updated by fitting the distribution of the SoftMax function of Q-value with temperature $\\alpha$. Therefore, modifying the update of $Q(j_t|s)$ to $Q(j'_t|s)$ using hindsight off-policy correction leads to a different actor optimization objective, thus affecting the training of the actor network. We will also elaborate on how the hindsight off-policy correction mechanism affects guide policy training in the revised manuscript.\n\n[2] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Haarnoja, Tuomas, et al. ICML 2018.\n\n**Thank you once again for your valuable feedback. We hope our response has satisfactorily addressed your concerns. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7M0Mgzgz6F", "reviewer": "Reviewer_5DoG", "summary": "This paper addresses the problem of multi-task reinforcement learning (MTRL). To this end, the paper proposes a method that selectively shares behaviors from the policies learning to solve other tasks. The experiments conducted in a locomotion domain (multi-task Half-Cheetah) and a robot arm manipulation domain (Meta-World) verify that the proposed method can improve the performance of the learned policies. Ablation studies justify the effectiveness of many proposed components, including the policy-filter gate, the guide-block gate, and the hindsight correct. Yet, I am not convinced that the proposed method could improve the sample efficiency, which is the key goal of MTRL. Also, it seems that the proposed method is incremental and lacks novelty. Therefore, I am leaning toward rejecting this work in its current form.", "questions": "See above", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Motivation and intuition**\n- The motivation for sharing behaviors for MTRL is convincing.\n\n**Experimental results**\n- The main experimental results show that the proposed method achieves better converged performance compared to the baselines in MT Half-Cheetah and Meta-World.\n\n**Ablation study**\n- The ablation studies justify the effectiveness of the policy filter gate, guide block gate, and correct hindsight.\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed method.", "weaknesses": "**Main results (Table 1)**\n- Table 1 and Section 5.2 present quantitative results of converged performance. However, as far as I am concerned, the comparisons of MTRL should focus on sample efficiency, i.e., how fast each method can learn, instead of converged performance. Therefore, I am not convinced by this evaluation.\n\n**Comparison to k-step QMP**\n- The paper states that the proposed method outperforms QMP because \"guide policy learns long-term policy guidance.\" I am not entirely convinced. One can simply use QMP to select from k-step behavior proposals and execute the selected proposal, which can also achieve this long-term policy guidance. Including this variant of QMP would be necessary to show that the performance gain comes from other designs of the proposed method.\n\n**Guide step K**\n- The best guide step K in the MT Half-Cheetah and Meta-World are different. It is unclear, given a new MTRL domain, how we should choose K. It seems that we could only tune this hyperparameter via trial and error. \n\n**Backbone RL algorithms**\n- This work adopts SAC as the backbone RL algorithm. Is it possible to use other RL algorithms, such as TD3 or PPO?\n\n**Clarity**\n- Section 4 is difficult to follow. Sufficiently describing the intuitions before introducing each component of the proposed method would significantly improve the readability of this section.\n\n**Novelty**\n- The proposed method seems incremental given that the QMP paper explores this idea of behavior sharing. Despite the high similarity between this work and the QMP paper, the authors seem to deliberately hide this by avoiding discussing QMP in the introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n> Sample efficiency comparison.\n\nPlease refer to **Figure 10 (Appendix D.1)**, which presents the full training curves for our main experiment (Table 1). It shows that beyond the ultimate performance improvement, CTPG also enhances the sample efficiency.\n\n> Comparison to K-step QMP.\n\nIn our experiments, we use the hyperparameter settings from the original QMP paper. Thus, the QMP in our experiment is exactly the K-step QMP ($K$ = 10 in implementation) you asked for, which as you mentioned, shows that the performance gain comes from other designs of the proposed method. We are sorry for the ambiguity and we will add this hyperparameter setting to the Appendix in the revised manuscript. Thank you again for pointing this out.\n\n> Hyperparameter guide step $K$.\n\nGuide step $K$ is a necessary hyperparameter introduced by CTPG and may need slight adjustment across testbeds. However, we want to emphasize that the improvement gains of CTPG do not rely on the careful tuning of $K$ at all. As shown in **Fig.2 of the global Rebuttal Comment PDF** (a clear version of Figure 12), the performance improvement is evident even without tuning $K$. If the user pursues the potential best performance, we recommend exploring the value of $K$, and it can be selected easily with the help of hyperparameter search methods, such as Bayesian Optimization, Hyperband, etc. In the Conclusion and Discussion Section, we have already indicated that our future work will focus on dynamically selecting the guide step $K$.\n\nIn addition, the output of CTPG's guide policy can add another action dimension to choose a specific guide step $K$. This approach not only automatically selects the $K$ value in different environments but also dynamically selects it at different time steps. We already have preliminary experimental results demonstrating the effectiveness of this scheme, and it also shows the high scalability of the CTPG framework. However, we believe that the contribution could be a new work, so it is mentioned only in future work.\n\n> Adaptability of other RL algorithms to CTPG.\n\nCTPG is a general MTRL framework that can be adapted with other RL algorithms, *even allowing the control and guide policy to use different RL algorithms*. Since SAC is a widely used algorithm in continuous control and serves as the base algorithm for all baselines, we also choose SAC as the base algorithm in this work. In addition, we explore the adaptation of TD3 with the CTPG and also employ different RL algorithms for the control and guide policies. Specifically, the control policy uses TD3, and the guide policy uses DQN. This approach is compared with the TD3-based MTRL method, and the result is displayed in **Fig.3 of the global Rebuttal Comment PDF**. The result shows that CTPG can enhance performance and sample efficiency when combined with other backbone RL algorithms.\n\n> Section 4 is difficult to follow.\n\nThank you very much for your suggestion. We will include an additional intuition before introducing each module to enhance readability in the revised manuscript.\n\n> Similarity with QMP.\n\nWe highly respect QMP and believe that QMP is an excellent work that greatly inspired our research. We discussed their differences in Related Work Section. Moreover, QMP serves as the most important and the only baseline in our experiments, and we openly compared our method against it. Following your suggestion, we will include a more detailed introduction of QMP and its great contribution in the revised Introduction.\n\nHere, we would like to re-emphasize the difference between CTPG and QMP: QMP uses the maximum Q-value of a single step to select the shared behavior over continuous K steps, which *only* guarantees the optimality of the first step of the shared policy. In contrast, CTPG learns a guide policy with the action space being exactly the task control policies to identify useful sharing policies for guidance by considering the benefits over K steps collectively. Additionally, CTPG proposes two gating mechanisms to avoid negative transfer from unhelpful policies, thereby improving sample efficiency.\n\n**Thank you once again for your valuable feedback. We hope our response has satisfactorily addressed your concerns. If you find that we have addressed your concerns, we kindly hope you reconsider your rating. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QyzzxajN7k", "reviewer": "Reviewer_J1dv", "summary": "This paper tackles the problem of multi-task reinforcement learning. Previous works have primarily focused on MTRL through specialized network structures or methods to resolve conflicting gradients. This paper considers the problem from an orthogonal perspective, by actively sharing control policies for each task adaptively, in the hope that sometimes the policy of one task can guide the exploration of other tasks. Specifically, similar to in hierarchical RL, a high-level policy is used to decide which “task policy” to call at each particular state, where “bad” policies are masked out through a learned value function. Evaluated on Metaworld and HalfCheetah MTRL benchmark, the method shows improved performance compared to previous works.", "questions": "See weaknesses", "limitations": "The limitations are adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of exploring with cross-task policies to facilitate multi-task reinforcement learning is interesting and novel.\n- The method is thoroughly evaluated on a set of test domains, and show superior performance compared to previous methods\n- This paper is well written, and the method is easy to follow.", "weaknesses": "- The proposed method contains many floating pieces that seem to add burden both to implementation and hyperparameter tuning. Some of the pieces seem rather like hack than a principled solution. For example:\n  - For the hindsight off-policy correction, the action of the guide policy is relabelled to the one that is most likely to generate the sequence of actions. However, it should be likely that none of the control policies can generate the old sequence of actions with high probability, in which case relabeling will not help at all.\n  - The action space masking is essentially masking out control policies with low Q values. Shouldn’t this already be reflected in the policy through policy gradient updates? Why do we need additional masking?\n  - Using the temperature coefficient of SAC as an indication of the policy performance is very empirical and may not always be true. This also makes the proposed method specific to adaptive temperature SAC.\n- It seems that the hindsight off-policy correction would require much more forward passes than a regular SAC. Can the author show the performance of the proposed method with x axis being wall-clock time?\n- For the proposed method to be effective, we need the policy of some other tasks to have better performance than the policy that is trained for the current task. One scenario I can think of is when the multiple tasks secretly form a curriculum (e.g. one task is the prerequisite for another task), which might be the case in Metaworld and halfcheetah. But I’m not sure how realistic such an assumption is in the real world.\n\nMinors:\n- Line 207: dose → does\n- “someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to related skills” → I don’t think that’s actually the case…", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts.\n\n> Floating pieces\n\n1. Hindsight off-policy correction: On the one hand, it is deeply integral to our CTPG framework, instead of just a floating piece. It addresses the unavoidable non-stationarity issue in off-policy training. We demonstrate this in Fig 5c and 11c, where it is helpful and its effect is more pronounced with more tasks, reinforcing its necessity within CTPG. On the other hand, it seems that there is a misunderstanding on the reason for proposing this module. The issue you mentioned does indeed exist and is still a matter of interest for the community. In such case, the data is invalid for training because none of the policies can regenerate this old sequence, so the hindsight off-policy correction won't work either. However, our module is not proposed to solve this issue and has its own motivation. It should not be considered a floating piece merely because it does not address a problem outside its intended scope.\n\n2. Action space masking in the policy-filter gate: The integration of this module with CTPG is also essential, which avoids negative policy guidance from unhelpful control policies. Although the guide policy can also be learned by gradient updates only, the policy-filter gate serves to expedite this process by narrowing down the candidate policies, especially when the number of tasks is particularly large, thus further reducing the exploration space and improving sample efficiency. As shown in Fig 5a and 11a, training efficiency will be significantly reduced without this gate. Here we want to emphasize that although the functionality of this gate can be achieved by policy gradient update, our module can significantly improve training efficiency.\n3. Temperature coefficient of SAC in the guide-block gate: We aim to use it to filter tasks that have been mastered or converged on. The choice of SAC and its adaptive temperature mechanism was made due to its robust performance in various RL tasks. We want to emphasize that CTPG is a general MTRL framework that can be adapted with other RL algorithms. We explore the adaptation of TD3 in **Fig.3 of the global Comment PDF**. Using the temperature of SAC as an indicator is indeed an empirical choice. We admit it may not always be true, thus we provide several alternatives not specific to SAC. For example, we try an intuitive metric success rate in Fig 5b and 11b, which shows fair performance with SAC temperature. Besides, policy entropy (in algorithms like PPO) or cumulative rewards (when there is no significant change) can also perform as indicators. The guide-block gate serves as an essential module with various implementations, and we provide several alternatives and welcome further study based on ours.\n\nThe three modules collectively contribute to share effective policies between tasks, thus improving the sample efficiency, which is the key goal of MTRL. In addition, these three modules do not involve any hyperparameter settings, so there is no additional burden for tuning. Hope our response can change your initial perception that they were just floating pieces.\n\n> Forward time of hindsight off-policy correction and performance with wall-clock time\n\nCompared to the gradient update process, hindsight off-policy correction requires only forward computation, making the time consumed negligible. In addition, the calculation of each control policy's probability for generating the historical action sequence can be parallelized. This is achieved by extending the input of the actor from [batch\\_size] to [batch\\_size, task\\_num], where the second dimension distinguishes the different task representations.\n\n**Fig.4 of the global Comment PDF** shows the training curves with wall-clock time on MetaWorld-MT10. Although CTPG takes longer to train, it still outperforms baselines for the same training time.\n\n> Scenarios that CTPG works\n\nCTPG does not require that a task have to be a sub-strategy of another task. If tasks can naturally form a curriculum as you mentioned, CTPG definitely can learn an effective policy. Even if other tasks' control policies do not perform as well on the current task, CTPG's guide policy can still facilitate an effective mixed policy. In specific, the agent can choose different control policies at different stages within a single episode, as shown in Fig 4 in the paper. In other words, the policy guidance uses a learnable mixture policy consisting of all control policies. Furthermore, referring to Reviewer xTD7's suggestion, we set up a new baseline called BPT (Best Performance Transfer), which selects the best-performing policy in the current task among all task policies to guide the exploration. The results shown in **Fig.1 of the global Comment PDF** demonstrate that CTPG also outperforms BPT. This is because CTPG's guide policy learns a more flexible strategy, rather than simply relying on a single best-performing policy for guidance.\n\nHere we extend the earlier example involving bicycle and motorcycle for better understanding. Suppose an agent is learning four tasks: unicycle(U), bicycle(B), motorcycle(M), and automobile(A). They do not form a curriculum and need to be learned simultaneously. (B) and (M) share the skill balancing on two wheels. (U) and (B) share the skill pedal-driven, while (M) and (A) share the skill ignition to move. With CTPG, if the agent has learned the pedaling skill in (B), it can transfer this skill to (U), even if it hasn’t yet mastered balancing. Similarly, if the agent has learned balance in (B) and ignition in (A), it can quickly master (M) by combining these two skills.\n\n> Minor errors\n\nThank you for pointing out the typo \"dose\", we will correct it in the revised manuscript.\n\n**Thank you once again for your valuable feedback. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vfzMqeqP7", "reviewer": "Reviewer_k7nL", "summary": "The paper presents Cross-Task Policy Guidance (CTPG), a novel framework designed to improve multi-task reinforcement learning (MTRL) by leveraging cross-task policy similarities. CTPG trains a guide policy for each task to select the most suitable behavior policy from a pool of all tasks' control policies. This method aims to generate better training trajectories and enhance learning efficiency. The authors propose two gating mechanisms: a policy-filter gate to filter out non-beneficial control policies and a guide-block gate to block unnecessary guidance for mastered tasks. Empirical evaluations on manipulation and locomotion benchmarks demonstrate that integrating CTPG with existing parameter sharing approaches significantly enhances performance.", "questions": "### Q1. Scalability to Complex Environments\n\nHow does the CTPG framework scale to more complex, large-scale environments? Are there any specific challenges or limitations that need to be addressed for practical deployment in such scenarios?\n\n### Q2. Computational Requirements\n\nWhat are the computational requirements for training and deploying CTPG? How does the method perform in terms of execution time and resource consumption compared to existing MTRL methods?", "limitations": "The authors acknowledge the limitations related to the predetermined guide step K and the reliance on specific benchmarks. However, a more detailed discussion on potential negative societal impacts and strategies to mitigate them would be beneficial.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "### S1. Innovative Approach\n\nThe paper introduces a novel method for leveraging cross-task similarities in MTRL. By training a guide policy to select behavior policies from a pool of tasks, the approach provides a direct and efficient way to exploit shared skills, which is an underexplored area in MTRL.\n\n### S2. Empirical Validation\n\nThe proposed framework is validated through extensive experiments on various benchmarks, including MetaWorld and HalfCheetah. The results demonstrate significant improvements in learning efficiency and performance when CTPG is integrated with existing MTRL approaches.\n\n### S3. Clear Presentation\n\nThe paper is well-structured and clearly explains the CTPG framework, including detailed descriptions of the guide policy and gating mechanisms. Figures and tables effectively illustrate the benefits of the proposed method, and the provided pseudocode enhances reproducibility.", "weaknesses": "### W1. Scalability Concerns\n\nThe scalability of CTPG to more complex and large-scale environments is not thoroughly discussed. While the method shows promising results in the tested benchmarks, a broader analysis of its scalability and practical utility in more complex scenarios may improve the significance of this work.\n\n### W2. Computational Complexity\n\nThe computational complexity of training and deploying CTPG, particularly the guide policy and the gating mechanisms, is not explicitly addressed. Understanding the computational requirements and potential limitations in terms of resources and execution time would provide a more comprehensive evaluation of its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive feedback and acknowledgment of our efforts. Following are our responses to all your concerns.\n\n> Scalability to complex environments.\n\nWe have tested CTPG on MetaWorld and HalfCheetah, the commonly used and authoritative benchmarks in the MTRL community, covering two main fields of manipulation and locomotion. In both two environments with diverse task settings, CTPG has shown outstanding performance, which demonstrated that CTPG can effectively learn policy sharing between tasks, thus improving the sample efficiency of MTRL. In addition, CTPG is a general MTRL framework that can be adapted to other MTRL environments and other RL algorithms. We conducted experiments using another base RL algorithm TD3 in **Fig.3 of the global Rebuttal Comment PDF**, and the results demonstrate that CTPG is equally applicable to other algorithms to enhance sample efficiency. CTPG can also be migrated to other MTRL benchmarks without additional configuration and deployment. If you have a recommendation for another benchmark, we'd be happy to try it on there too :)\n\n> Computational requirements and computational complexity.\n\nWe provided detailed information about the computational resources used in our experiments in Appendix E.1. For computational complexity, we show the training curves with the x-axis being wall-clock time on the MetaWorld-MT10 environment in **Fig.4 of the global Rebuttal Comment PDF**. The results indicate that although CTPG takes longer to train, it still outperforms baselines for the same training time. Furthermore, during the evaluation and deployment phase, CTPG's guide policy will no longer be used, so no additional execution time and storage space is required.\n\n> Discussion on potential negative societal impacts.\n\nWe provided Broader Impacts in Appendix F. CTPG is a general MTRL framework and does not introduce additional social impacts that need to be discussed. If you feel there are any necessary social implicates that need to be discussed, welcome to point them out, and we will add discussions about them in the revised manuscript.\n\n**Thank you once again for your valuable feedback. We hope our response has satisfactorily addressed your concerns. If you find that we have addressed your concerns, we kindly hope you reconsider your rating. If you need further elaboration or additional points to include in the response, we welcome further discussion to ensure everything is clear and satisfactory.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To AC and all the reviewers:

We would like to express our sincere gratitude to AC and all the reviewers for their great efforts in evaluating our paper. Your valuable insights and suggestions are greatly appreciated. We have carefully addressed all the questions and concerns raised in the reviews through our rebuttal comments.

The attached PDF to this global rebuttal comment contains:

* Figure 1 is the experimental results with an additional baseline BPT (Best Performance Transfer).

* Figure 2 is an extended version of the ablation study on guide step $K$ (Figure 12 in paper), which includes an additional curve base MHSAC for convenient comparison.

* Figure 3 is the additional experimental results of adapting CTPG to the base RL algorithm TD3.

* Figure 4 is the experiment results on MetaWorld-MT10 with x-axis being wall-clock time.

If there are any remaining queries or uncertainties after reviewing our responses, we welcome further discussions during the upcoming phase. Your continued engagement is highly valued and appreciated. Thank you once again for your time, expertise, and contribution to our paper.

Meta-review: The paper introduces Cross-Task Policy Guidance (CTPG), a method for enhancing multi-task reinforcement learning (MTRL) by allowing policies to learn from each other's trajectories. The approach involves a guide policy that selects the most suitable behavior policy from a pool of all tasks' control policies, along with mechanisms to prevent negative transfer and ensure efficient learning. CTPG shows improved performance in benchmarks like MetaWorld and Half-Cheetah.

While the method is innovative, some reviewers noted that the contribution feels incremental, especially in comparison to latest methods such as QMP. Concerns were also raised about the scalability of CTPG to more complex tasks and the focus on converged performance rather than sample efficiency. The computational complexity of the approach was another point of uncertainty.

However, the paper also exhibits strengths. The paper is well-presented, with clear motivation and thorough experiments that demonstrate the effectiveness of CTPG. The innovative approach to leveraging cross-task similarities offers potential for advancing MTRL research. The positive impact and the detailed response to reviewer concerns support its acceptance.

Based on the overall positive reviews, it is recommended to accept the paper, provided that the revisions discussed in the rebuttal and comparison to the most recent multi-task RL algorithms are incorporated in the final version.
Decision: Accept (poster)