Title: Efficient Multi-task Reinforcement Learning with Cross-Task Policy Guidance
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-task Reinforcement Learning
Abstract: Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is a general framework adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.

Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidance

Abstract

Multi-task reinforcement learning endeavors to efficiently leverage shared information across various tasks, facilitating the simultaneous learning of multiple tasks. Existing approaches primarily focus on parameter sharing with carefully designed network structures or tailored optimization procedures. However, they overlook a direct and complementary way to exploit cross-task similarities: the control policies of tasks already proficient in some skills can provide explicit guidance for unmastered tasks to accelerate skills acquisition. To this end, we present a novel framework called Cross-Task Policy Guidance (CTPG), which trains a guide policy for each task to select the behavior policy interacting with the environment from all tasks' control policies, generating better training trajectories. In addition, we propose two gating mechanisms to improve the learning efficiency of CTPG: one gate filters out control policies that are not beneficial for guidance, while the other gate blocks tasks that do not necessitate guidance. CTPG is a general framework adaptable to existing parameter sharing approaches. Empirical evaluations demonstrate that incorporating CTPG with these approaches significantly enhances performance in manipulation and locomotion benchmarks.

1 Introduction

Deep reinforcement learning (RL) has undergone remarkable progress over the past decades, showcasing its efficacy across various domains, such as game playing [16, 28] and robotic control [12, 13]. However, most of these deep RL methods primarily focus on learning different tasks in isolation, making it challenging to utilize shared information between tasks to develop a generalized policy. Multi-task reinforcement learning (MTRL) aims to master a set of RL tasks effectively. By leveraging the potential information sharing among different tasks, joint multi-task learning typically exhibits higher sample efficiency than training each task individually [27].

A significant challenge in MTRL lies in determining what information should be shared and how to share it effectively. Recent studies have proposed various approaches to tackle this challenge via network parameter sharing with carefully designed network structures [10, 22, 27] or tailored optimization procedures [4, 14, 30]. We summarize such methods as implicit knowledge sharing in Section 2. Despite these unremitting efforts, another largely overlooked way exists to exploit cross-task similarities to improve the learning efficiency of multiple tasks. Intuitively, humans can effortlessly discern which skills can be shared from other tasks while learning a specific task. For instance, someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to

Reach button

Push button

Reach handle

Grab handle

Grab handle

Rotate to open

Translate to open

(a) Button-Press v.s. Drawer-Close

(b) Door-Open v.s. Drawer-Open

related skills, such as operating controls, maintaining balance, and executing turns. Likewise, a motorcyclist adept in these skills can also quickly learn to ride a bicycle. This ability allows humans to efficiently master multiple tasks by selectively referring to skills previously learned. As shown in Figure 1, similar full or partial policy sharing is also evident in robotic arm manipulation tasks. These cross-task similarities enable policy guidance , i.e ., control policies of tasks already proficient in specific skills can generate valuable training data for unmastered tasks. Compared to the common practice, which blindly generates training trajectories for each task solely with its own control policy, generating training trajectories using a control policy from other tasks that perform better in the current situation can better facilitate the learning procedure. Moreover, this explicit policy sharing approach significantly reduces unnecessary exploration of similar contexts in different tasks.

The key challenge encountered in this approach to MTRL is discerning beneficial sharing control policies for each task adaptively. To address this challenge, [32] uses a Q-filter to identify singlestep shareable behaviors without ensuring optimality for long-term policy sharing. In contrast, we propose a simple yet effective framework called Cross-Task Policy Guidance (CTPG) for more robust long-term policy guidance. Initially, we group the control policies of all tasks into a candidate set. Subsequently, for each task, we train a guide policy to identify useful sharing control policies, and then the chosen control policy generates better training trajectories to achieve policy guidance . Furthermore, we design two gating mechanisms to avoid unfavorable policy guidance interfering with learning. The first, policy-filter gate, leverages the value function to refine the candidate set by masking out control policies that are not beneficial for guidance. The second, guide-block gate, withholds extra guidance for the mastered easy tasks, allowing the focus to be on further solidifying the skills already acquired. With the incorporation of the above two gates, CTPG greatly improves the quality of the policy guidance, thereby fostering enhanced exploration and learning efficiency.

CTPG is a generalized MTRL framework that can be combined with various existing parameter sharing methods. Among these, we choose several classical approaches and integrate them with CTPG, achieving significant improvement in sample efficiency and final performance on both manipulation [31] and locomotion [11] MTRL benchmarks. Furthermore, we conduct detailed ablation studies to gain insights into how each component of CTPG contributes to its final performance.

2 Related Work

Multi-task learning is a training paradigm that enhances generalization by leveraging the information inherent in potentially related tasks [3, 19, 34]. Multi-task reinforcement learning extends this concept to reinforcement learning, expecting that information shared across tasks will be uncovered by simultaneously learning multiple RL tasks [25]. In this study, we distinguish between information sharing as implicit knowledge sharing and explicit policy sharing.

Implicit Knowledge Sharing. Implicit knowledge sharing primarily focuses on sharing parameters or representations, but it encounters the challenge of negative knowledge transfer due to simultaneous updates within the same network. [14, 30] regard MTRL as a multi-objective optimization problem aimed at managing conflicting gradients resulting from different task losses during training. [22, 27]

partition the network into distinct modules and combine these modules to form different sub-policies for different tasks. [5, 21] endeavor to choose or learn better representations as more effective taskconditioned information for policy training. [7, 23] employ distillation and regularization to fuse separate task-specific policies into a unified policy for diverse tasks.

Explicit Policy Sharing. Explicit policy sharing is expressed as the direct sharing of behaviors or policies between different tasks. [20] employs a hierarchical policy that decides when to directly use a previously learned policy and when to acquire a new one. Nonetheless, instead of learning multiple tasks simultaneously, it adopts a sequential task-learning approach, necessitating a manually welldefined curriculum of tasks. [32] uses a Q-filter to identify shareable behaviors. During exploration, each control policy proposes a candidate action, and the policy that suggests the maximum Q-value action on the source task is executed for the following timesteps. However, maximizing Q-value in a single timestep does not guarantee the optimality of this policy across continuous timesteps. CTPG is a new explicit policy sharing method that learns a guide policy for long-term policy guidance.

3 Preliminaries

Multi-Task Reinforcement Learning. We aim to simultaneously learn N tasks, where each task i ∈ T is represented as a Markov decision process (MDP) [1, 18]. Each MDP is defined by the tuple ⟨ S, A, P i , R i , γ ⟩ , where S denotes the state space, A the action space, P i : S × A → S the environment transition function, R i : S × A → R the reward function, and γ ∈ [0 , 1) the discount factor. In the scope of this work, different tasks share the same state and action spaces, distinguished by different transition and reward functions. The goal of the MTRL agent is to maximize the average expected return across all tasks, which are uniformly sampled during training.

Soft Actor-Critic. In this work, we use the Soft Actor-Critic (SAC) [9] algorithm, an off-policy actor-critic method under the maximum entropy framework. The critic network Q θ ( s t , a t ) parameterized by θ , representing a soft Q-function [8], aims to minimize the soft Bellman residual:

where D represents the data in the replay buffer, and ¯ θ is the target critic network parameter. The actor network π ϕ ( a t | s t ) is parameterized by ϕ , and the objective of policy optimization is:

where α is a learnable temperature parameter to penalize entropy as follows:

where ¯ H is a desired minimum expected entropy. If the optimization leads to an increase in π ϕ ( a t | s t ) with a decrease in the entropy, the temperature α will accordingly increase. In the following sections, we use subscripts to signify the networks specific to each task. Specifically, the control policy of task i is represented as π i , and the corresponding Q-value function is denoted as Q i .

4 Cross-Task Policy Guidance

Explicit policy sharing offers a direct and efficient way to master multiple tasks. If a task is already mastered, its control policy can be fully or partially shared with other tasks to guide tasks requiring similar skills to be quickly learned. Instead of each task generating trajectories constantly by its corresponding control policy, as in most existing MTRL algorithms, we consider using control policies of other tasks to generate training data for the current task when appropriate. To achieve this goal, we propose a novel framework called Cross-Task Policy Guidance (CTPG), which extra learns a guide policy for each task to identify beneficial policies for guidance. We illustrate the trajectory generation process of Task 1 in Figure 2. For this task, its guide policy Π g 1 selects a policy π ′ from the candidate set of all control policies { π i } N i =1 every fixed K timesteps. It then uses π ′ as the behavior policy to interact with the environment and collect data for the next K timesteps.

The CTPG framework alters only the data collection process, guiding the control policy training through better exploration trajectories. In Section 4.1, we introduce the guide policy in detail and propose a hindsight off-policy correction mechanism for its training. In addition, we propose two gating mechanisms to enhance the efficiency of CTPG: the policy-filter gate discussed in Section 4.2 and the guide-block gate detailed in Section 4.3.

4.1 Guide Policy

The control policy π i ( a t | s t ) of task i maps the state s t to the environment action a t , and its corresponding Q-value function Q i ( s t , a t ) estimates the expected return. The guide policy Π g i ( j t | s t ) of task i outputs a task index j t ∈ T , and the corresponding control policy π j t of task j t serves as the behavior policy. The guide Q-value function Q g i ( s t , j t ) estimates the expected return of using Π g i to select different control policies for every K timesteps, with its Bellman equation defined as follows:

where B Π g i is the Bellman operator of the guide policy Π g i , and the corresponding reward function R g i is defined as the expected cumulative discount rewards for behavior policy π j t over K timesteps:

For each task i , during trajectory generation, CTPG first utilizes its guide policy Π g i to sample a behavior policy π j t from the candidate set of all tasks' control policies:

and then samples actions using the behavior policy π j t for the next K timesteps:

where t ′ ∈ { t, t +1 , . . . , t + K -1 } . After each timestep, we obtain the reward r t ′ = R i ( s t ′ , a t ′ ) and the next state s t ′ +1 . The transition ⟨ i, s t ′ , a t ′ , j t , r t ′ , s t ′ +1 ⟩ is stored in the replay buffer.

The guide policy is trained to maximize the expected return of the current task by choosing appropriate control policies in certain states. If the control policies of some tasks already proficient in specific skills can be shared with the current task in similar states, the guide policy can quickly learn to use these control policies in those states to generate better training data for the current task. The guide policy and the control policy are trained simultaneously. We can use any off-policy RL algorithms for control policy training and any on/off-policy RL algorithms for guide policy training. In this work, we use SAC [9] for the control policy training and the discrete action space variant of SAC [6] for the guide policy training. Given that the guide policy acts every K timesteps, its training frequency is 1 /K that of the control policy. The detailed pseudo-codes for the control policy and guide policy training are provided in Appendix A.1 (Algorithms 1 and 2).

Hindsight Off-Policy Correction. During off-policy training, the guide policy faces a nonstationary challenge. Since the control policies are continually updated during the training of the guide policies, the actions chosen by the behavior policies during data collection may no longer align with the improved corresponding control policies, thereby compromising the validity of the training experience. We address this concern by implementing a hindsight off-policy correction mechanism that reassigns the action j t sampled by the past guide policy to a new one j ′ t , whose control policy π j ′ t is more likely to output the historical action sequence { a t ′ } t + K -1 t ′ = t . Specifically, we utilize maximum likelihood estimation following:

In this way, we can leverage past experiences effectively to train the guide policy. The workings of the hindsight off-policy correction mechanism in SAC are detailed in Appendix C.

1. Choose behavior policy

from candidate set

.

3. Collect

transitions

2. Roll out for

timesteps.

Guide Policy

Behavior Policy

4. Train with mini-batch

Input

Environment

Candidate Set

Task Policy

Control Policy

4.2 Not All Policies Are Beneficial for Guidance

In Section 4.1, we set the guide policy's action space as the set of all control policies { π i } N i =1 . However, not all control policies within the action space of Π g i are beneficial for task i in state s t . Some control policies perform even worse than the current task's own control policy π i , rendering them ineffective for guidance. To address this issue, we design a policy-filter gate to refine the action space of the guide policy by adaptively filtering out unfavorable control policies in state s t . The trajectory generation process solely using the current task's control policy π i can be regarded as equipped with a special guide policy Π ˜ g i that exclusively selects π i as the behavior policy, i.e ., Π ˜ g i ( i | s t ) = 1 for any s t . The guide Q-value Q ˜ g i of Π ˜ g i , defined by Equation 5, is:

where we repeatedly expand Q ˜ g i to find that Q ˜ g i ( s t , i ) is equal to V i ( s t ) , the state value function of task i 's control policy. Because the value function can serve as a filter for high-quality training data [17, 29, 33], it becomes intuitive to judge the quality of the behavior policy π j t following guide policy and the current task's control policy π i by directly comparing Q g i ( s t , j t ) and V i ( s t ) . In our implementation, we estimate V i ( s t ) via Monte Carlo sampling of Q i ( s t , a t ) with a t ∼ π i ( a t | s t ) [15]. Relying on this mechanism, the policy-filter gate serves as a mask vector m ( s t ) to indicate whether each control policy is beneficial for guidance in state s t . Specifically, each element of m ( s t ) is:

where j indicates the element index and task index. Then, the behavior policy π j t is sampled by:

If none of the control policies are beneficial for guidance, i.e ., m ( s t ) = 0 , it indicates that the current task's control policy π i is the most proficient within the current state, rendering other control policies unnecessary for enhancing trajectory generation.

Comparable Guide Q-Value. Typically, RL algorithms estimate Q-values to approximate the expected return of the current state-action pair, allowing for the calculation of the policy-filter gate in Equation 11 through a direct comparison of V i ( s t ) and Q g i ( s t , j t ) . However, in maximum entropy RL algorithms such as SAC, Q-value estimation incorporates the maximum entropy objective, leading to the incomparability of two policies with different entropy objectives. Therefore, we learn another comparable guide Q-value ˆ Q g i with discounted entropy of π i following:

Since both ˆ Q g i ( s t , j t ) and V i ( s t ) estimate the return with the entropy of the current task's control policy, they can be directly compared to assess whether control policies are beneficial for guidance. A detailed comparability analysis of this comparable guide Q-value in SAC is provided in Appendix B.

4.3 Not All Tasks Need Guidance

When simultaneously learning multiple tasks, easy tasks converge faster than difficult ones. The control policies of easy tasks allow for the quick acquisition of some effective skills, which may

(](∨(⊕(∑ ∮(∧(∨(∮(⋃(∧

((∮(⋃(⊗(〈(⋂(](⌋(∑(∮

({(∏(∏(⊕(⊎(∫(∏(∑(∮(∏(∨

((∫(⊗(⊗(](∫(∑(∨(]

(∮(∧

((∫

((∫(⊗(⊕(⊔(⋂(((}(⊕(⊗(∨(∮(⊎

(〈(∑(⊕(⊔(∮(((⌋(⊗(∫(⊔(⊗

(⌈(∫(∏(∨(⊎(∫(⊗(]((∫(⊗(⊕(⊔(⋂

(〈(∑(⊕(⊔(∮(]((∫(⊗(⊕(⊔(⋂

(∮(∮(⊎

be helpful in exploring other tasks. However, these mastered or easy tasks do not need additional guidance from other policies; instead, they focus on further solidifying their already acquired skills. Therefore, not all tasks require assistance from the guide policy.

Based on the above analysis, we design another guide-block gate to prevent the guide policy from engaging in tasks that do not necessitate guidance. This mechanism is directly related to SAC's temperature coefficient α i . For difficult tasks i diff, their control policy entropies H ( π i diff ( ·| s t )) tend to be high, and the corresponding temperature parameters α i diff decrease according to Equation 4. Conversely, the temperature parameters α i easy increase for easy tasks i easy. Therefore, α i is a metric reflecting the relative difficulty and mastery of different tasks. We form the tasks that require guidance into a subset T g following:

which selects the tasks by comparing the difficulty of the current task versus the average of all tasks. For tasks i / ∈ T g , the guide-block gate restricts them from using the guide policy, so they only use their own control policies to interact with the environment. Consequently, the guide policy can stop training with samples from the tasks i / ∈ T g and focus on learning the guidance for unmastered tasks.

The comprehensive CTPG framework, with two special gating mechanisms, is summarized in Figure 3. The complete pseudo-code for CTPG is described in Appendix A.2 (Algorithm 3).

5 Experiments

The experiments are designed to answer the following research questions: Q1: Can implicit knowledge sharing approaches be combined with CTPG to further improve performance? Q2: Does the guide policy in CTPG learn useful sharing policies? Q3: How does each component within CTPG contribute to the final performance? Q4: How does CTPG perform without implicit knowledge sharing approaches? Q5: Can CTPG expedite the exploration of new tasks effectively?

5.1 Environments

We conduct experiments on MetaWorld manipulation and HalfCheetah locomotion MTRL benchmarks, selecting two setups for evaluation within each benchmark.

MetaWorld Manipulation Benchmark. The MetaWorld benchmark [31] consists of 50 robotics manipulation tasks employing a sawyer arm in the MuJoCo environment [24]. It provides two setups: MetaWorld-MT10 , comprising a suite of 10 tasks, and MetaWorld-MT50 , comprising a suite of 50 tasks. Following the settings in [9], the goal position is randomly reset at the start of every episode. We use the mean success rate as our evaluation metric, which is clearly defined in the environment.

HalfCheetah Locomotion Benchmark. The HalfCheetah is a 6-DoF walking robot consisting of 9 links and 8 joints connecting them in the MuJoCo environment [24]. The multi-task benchmark HalfCheetah Task Group [11] contains different HalfCheetah robots. HalfCheetah-MT5 includes 5

tasks under various scales of simulated earth-like gravity, ranging from one-half to one-and-a-half of the normal gravity level. HalfCheetah-MT8 includes 8 tasks with various morphology of a specific robot body part. We use the episode return as our evaluation metric.

Further information regarding environmental setups is provided in Appendix D.

5.2 Performance Improvement on Implicit Knowledge Sharing Approaches

CTPG is a generalized MTRL framework adaptable to various implicit knowledge sharing approaches, wherein both the control policy and the guide policy use a unified network. Specifically, in our implementation, the unified control policy employs the same network structure and update procedure as the implicit knowledge sharing approaches, and the unified guide policy utilizes a straightforward multi-head structure for parameter sharing.

To answer Q1 , we choose five classical implicit knowledge sharing approaches: (1) MTSAC extends SAC for MTRL by employing one-hot encoding for task representation. (2) MHSAC utilizes a shared network backbone apart from independent heads for each task. (3) PCGrad [30] resolves issues arising from conflicting gradients among tasks through gradient manipulation. (4) SM [27] trains a routing network to propose weights for soft module combinations. (5) PaCo [22] learns a compositional policy where task-shared parameters combine with task-specific parameters to form task policies. Combined with these implicit knowledge sharing approaches, we compare CTPG against: (i) Base represents the source version of these approaches. (ii) QMP [32] employs a one-step Q-value filter to identify shareable behaviors.

We train all combinations with 0.8 million samples per task in the HalfCheetah locomotion benchmark and 1.5 million samples per task in the MetaWorld manipulation benchmark. Each combination is trained 5 times with different seeds. We evaluate the final policy over 100 episodes per task and report the mean performance and standard deviation across different seeds in Table 1.

Based on the experimental findings, it becomes evident that, except for the PaCo algorithm in HalfCheetah-MT8, the combination with CTPG leads to a notable enhancement in performance across all scenarios. QMP does not perform as well as CTPG because QMP only offers single-step behavior guidance, whereas CTPG's guide policy learns long-term policy guidance. Notably, as the number of tasks increases, CTPG exhibits superior performance due to the higher probability of explicit policy sharing between tasks, facilitating more effective policy exploration. Besides the final performance evaluation, comprehensive training curves are presented in Appendix E.1.

Timestep:

0

10

20

30

40

50

Pick-Place

Button-Press-Topdown

Drawer-Close

Guide Policy

Output:

Behavior Policy:

7.1%

Button-Press-Topdown

36.1%

Pick-Place

24.2%

Drawer-Close

Drawer-Close

9.7%

30.6%

Pick-Place

30.1%

Button-Press-Topdown

0.7%

94.5%

Pick-Place

1.7%

Push

20.5%

Pick-Place

36.9%

9.6%

Push

30.2%

Push

15.6%

Pick-Place

21.4%

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

80

Success Rate (

%

)

w/o Policy-Filter Gate

CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

80

w/o Guide-Block Gate

w/ Success Rate Metric

CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

80

w/o Hindsight Correction

CTPG

(a) Ablation on Policy-Filter Gate

(b) Ablation on Guide-Block Gate

(c) Ablation on Hindsight Correct

×

×

×

5.3 Guidance Learned by Guide Policy

To answer Q2 , we visualize the task Pick-Place with guidance on MetaWorld-MT10 to show the specific role of the guide policy. Since the guide policy is only used in trajectory generation during training, we visualize one of the sampled trajectories in Figure 4. We only present the initial 50 timesteps of this trajectory, in which the task has essentially been completed.

To better understand policy sharing between tasks, we first explain the relevant tasks. Pick-Place : Pick and place a puck to a goal. Peg-Insert-Side : Insert a peg sideways. Push : Push the puck to a goal. Button-Press-Topdown : Press a button from the top. Drawer-Close : Push and close a drawer. Reach : Reach a goal position. The visualizations of these tasks are provided in Figure 8 (Appendix D.1).

The initial and final 20 timesteps showcase that the guide policy learns useful guidance to successfully complete the source task. In the initial 20 timesteps, Pick-Place and Peg-Insert-Side employ a shared policy directing the robotic arm toward the target object. In the final 20 timesteps, the task is executed using Button-Press-Topdown to raise the gripper and then Drawer-Close to move forward. Interestingly, although the task in the final 20 timesteps intuitively aligns with Reach , the guide policy opts not to use it because the learned Reach 's control policy always opens the gripper, causing the puck to fall. In the middle 10 timesteps, the probability of Pick-Place is notably high due to the absence of alternative shared policies at this stage. Specifically, the most similar Peg-Insert-Side grabs the end of the peg for insertion, while Pick-Place requires gripping the puck centrally.

5.4 Ablation Studies

To answer Q3 , we conduct detailed ablation studies to analyze the impact of each component in CTPG on performance enhancement. CTPG contains three key components: (1) the policy-filter gate, (2) the guide-block gate, and (3) the hindsight off-policy correction mechanism. We study the influence of each component using the MHSAC implicit knowledge sharing approach on MetaWorld-MT10.

(1) Figure 5(a) shows that the policy-filter gate plays a significant role within CTPG, leveraging the value function to constrain the direction of guided exploration. (2) Given the precise definition of binary-valued success signal in the MetaWorld benchmark, we compare the SAC temperature metric, described in Section 4.3, with another success rate metric to determine if tasks need guidance.

Specifically, we block guidance for tasks with success rates exceeding 80% during evaluation. Figure 5(b) illustrates that both metrics showcase competitive performance, exhibiting clear performance gains over no guide-block gate. However, since the success rate is a human-defined metric and difficult to define for some tasks ( e.g ., HalfCheetah), the SAC temperature metric is more general across most environments. (3) We ablate the hindsight off-policy correction mechanism used in guide policy training in Figure 5(c), elucidating its improvement in training efficiency and stability. Further ablation studies of SM with CTPG on MetaWorld-MT50 are provided in Appendix E.2.

5.5 CTPG without Implicit Knowledge Sharing

To answer Q4 , we train N control and guide policies independently on N tasks without implicit knowledge sharing. We perform experiments on HalfCheetah-MT8 and MetaWorldMT10, employing the same environment configuration as in Section 5.2. As illustrated in Figure 6, CTPG significantly improves the performance of Single-Task SAC. Notably, the impact of CTPG on MetaWorld-MT10 is more pronounced. The potential rationale is that the tasks within HalfCheetah-MT8 exhibit minimal variance in difficulty, resulting in a narrow gap in

0

2

4

6

8

Samples/Task (

10 5

)

0

2

4

6

8

×

10 3

)

Single-Task SAC

Single-Task SAC w/ CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

Success Rate (

%

)

Single-Task SAC

Single-Task SAC w/ CTPG

(a) HalfCheetah-MT8

(b) MetaWorld-MT10

×

×

learning progress. Conversely, MetaWorld-MT10 presents a broader disparity in task difficulty, where CTPG facilitates the guidance from simpler to more challenging tasks, thus appearing more efficient.

5.6 Exploration of New Tasks with CTPG

To answer Q5 , we split the original task set in half, pre-training expert policies on the one half T e . For the other half, we compare direct learning with CTPG, where the agent leverages guidance from both the control policies being learned and the expert policies. Specifically, we use MTSAC on HalfCheetah-MT8, where T e includes all tasks that enlarge the size of body parts. On the other hand, we use SM on MetaWorld-MT10, with T e containing tasks indexed from 0 to 4. Figure 7 indicates that, compared to learning the

0

1

2

3

4

5

Samples/Task (

10 5

)

0

2

4

6

8

10

12

×

10 3

)

MTSAC

MTSAC w/ CTPG

0

1

2

3

4

5

Samples/Task (

10 5

)

0

20

40

60

80

100

Success Rate (

%

)

SM

SM w/ CTPG

(a) HalfCheetah-MT8

(b) MetaWorld-MT10

×

×

new set of tasks from scratch, CTPG can reasonably utilize experts to explore new tasks rapidly. Notably, in environments where task similarity is high, such as HalfCheetah-MT8, CTPG can quickly transfer the expert's abilities to the control policy being learned with the guide policy.

6 Conclusion and Discussion

This paper proposes the Cross-Task Policy Guidance (CTPG) framework for the MTRL explicit policy sharing. CTPG contains a guide policy and two special gates to identify beneficial sharing policies from the set of all task control policies and choose the most proficient one to generate high-quality trajectories for the current task as policy guidance. In addition, CTPG is a generalized framework adaptable to diverse implicit knowledge sharing approaches. Empirical evidence showcases that these approaches combined with CTPG further improve sample efficiency and final performance.

Limitations and Future Works. One limitation of CTPG is its reliance on a predetermined guide step K , necessitating hyperparameter tuning for K across different environments. Moreover, the fixed guide step setting lacks flexibility, as the duration of shared skills execution timesteps varies inconsistently among different tasks. Consequently, exploring methods to automate the selection of the guide step K presents an intriguing avenue for future research. Additionally, irrespective of the human-defined win rate and the unique temperature parameter of SAC, investigating alternative metrics for the guide-block gate emerges as another important direction for future endeavors.

References

Richard Bellman. Dynamic programming. Science , 153(3731):34-37, 1966.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. CoRR , abs/1606.01540, 2016.

Rich Caruana. Multitask learning. Machine Learning , 28:41-75, 1997.

Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of International Conference on Machine Learning , pages 794-803, 2018.

Myungsik Cho, Whiyoung Jung, and Youngchul Sung. Multi-task reinforcement learning with task representation method. In ICLR Workshop on Generalizable Policy Learning in Physical World , pages 1-11, 2022.

Petros Christodoulou. Soft actor-critic for discrete action settings. CoRR , abs/1910.07207, 2019.

Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divideand-conquer reinforcement learning. In Proceedings of International Conference on Learning Representations , pages 1-10, 2018.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of International Conference on Machine Learning , pages 1352-1361, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of International Conference on Machine Learning , pages 1856-1865, 2018.

Jinmin He, Kai Li, Yifan Zang, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Not all tasks are equally difficult: Multi-task deep reinforcement learning with dynamic depth routing. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 12376-12384, 2024.

Peter Henderson, Wei-Di Chang, Florian Shkurti, Johanna Hansen, David Meger, and Gregory Dudek. Benchmark environments for multitask learning in continuous domains. In ICML Workshop on Lifelong Learning: A Reinforcement Learning Approach , pages 1-6, 2017.

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, 2016.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceedings of International Conference on Learning Representations , pages 1-10, 2016.

Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems , pages 18878-18890, 2021.

Samuel Lobel, Sreehari Rammohan, Bowen He, Shangqun Yu, and George Konidaris. Qfunctionals for value-based continuous control. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 8932-8939, 2023.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In Proceedings of International Conference on Robotics and Automation , pages 6292-6299, 2018.

Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, 2014.

Sebastian Ruder. An overview of multi-task learning in deep neural networks. CoRR , abs/1706.05098, 2017.

Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In Proceedings of International Conference on Learning Representations , pages 1-11, 2018.

Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with contextbased representations. In Proceedings of International Conference on Machine Learning , pages 9767-9779, 2021.

Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. PaCo: Parametercompositional multi-task reinforcement learning. In Advances in Neural Information Processing Systems , pages 21495-21507, 2022.

Yee Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems , pages 4496-4506, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics , 9(9):1363, 2020.

Maciej Wolczyk, Michał Zaj ˛ ac, Razvan Pascanu, Łukasz Kuci´ nski, and Piotr Miło´ s. Disentangling transfer in continual reinforcement learning. Advances in Neural Information Processing Systems , 35:6304-6317, 2022.

Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In Advances in Neural Information Processing Systems , pages 4767-4777, 2020.

Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, Qiaobo Chen, Yinyuting Yin, Hao Zhang, Tengfei Shi, Liang Wang, Qiang Fu, Wei Yang, and Lanxiao Huang. Mastering complex control in MOBA games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 6672-6679, 2020.

Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. In Advances in Neural Information Processing Systems , pages 11501-11516, 2021.

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems , pages 5824-5836, 2020.

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Conference on Robot Learning , pages 1094-1100, 2020.

Grace Zhang, Ayush Jain, Injune Hwang, Shao-Hua Sun, and Joseph J Lim. Efficient multi-task reinforcement learning via selective behavior sharing. CoRR , abs/2302.00671, 2023.

Jin Zhang, Siyuan Li, and Chongjie Zhang. CUP: Critic-guided policy reuse. In Advances in Neural Information Processing Systems , pages 27537-27548, 2022.

Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering , 34(12):5586-5609, 2021.

A Pseudo Code

Without implicit knowledge sharing approaches, each task i has its control policy π i and guide policy Π g i . With implicit knowledge sharing approaches, there is a unified control policy π and guide policy Π g with the input of task representation z i . In both cases, we uniformly represent tasks as subscripts and omit the network parameters for a clear presentation in pseudo-code. Specifically,

The actor network with parameter ϕ for task i 's control policy is denoted as π i ( a t | s t ) .

The critic network with parameter θ for task i 's control policy is denoted as Q i ( s t , a t ) .

The temperature parameter for task i 's control policy is denoted as α i .

The guide actor network with parameter ϕ g for task i 's guide policy is denoted as Π g i ( j t | s t ) .

The guide temperature parameter for task i 's guide policy is denoted as α g i .

The guide critic network with parameter θ g for task i 's guide policy is denoted as Q g i ( s t , j t ) .

The comparable guide critic network with parameter ˆ θ g for task i is denoted as ˆ Q g i ( s t , j t ) .

Algorithm 1 Control Policy's Training Step

Initialization : actor network π , critic network Q , SAC temperature α

Input : minibatch B contains ⟨ i, s t , a t , r t , s t +1 ⟩

1: Optimize θ with SAC's critic loss in Equation 1

2: Optimize ϕ with SAC's actor loss in Equation 3

3: Optimize α with SAC's alpha loss in Equation 4

Algorithm 2 Guide Policy's Training Step

Input : actor network π , SAC temperature α , minibatch B g contains ⟨{ s t ′ , a t ′ , r t ′ } t + K -1 t ′ = t , i, j t , s t + K ⟩ Initialization : guide actor network Π g , guide critic network Q g , guide SAC temperature α g

1: Hindsight off-policy correct j t into j ′ t following Equation 9

2: Calculate the guide reward r g t following Equation 6

3: Optimize θ g with SAC's critic loss for guide policy

4: Optimize ϕ g with SAC's actor loss for guide policy

5: Optimize α g with SAC's alpha loss for guide policy

A.1 Pseudo Code of Policy Training

Algorithm 1 describes a training step for the control policy given a minibatch of data B , which contains a batch of tasks i , states s t , actions a t , rewards r t , and next states s t +1 .

In addition, Algorithm 2 describes a training step for the guide policy given a minibatch data B g . The data B g not only contains a batch of tasks i , states s t , actions j t taken by guide policy, and K timesteps ahead states s t + K but also contains rewards { r t ′ } t + K -1 t ′ = t accrued over K timesteps for calculating the guide reward. Moreover, B g includes states { s t ′ } t + K -1 t ′ = t and actions { a t ′ } t + K -1 t ′ = t taken by control policy over the K timesteps for hindsight off-policy correction.

A.2 Pseudo Code of Comprehensive CTPG

The full pseudo-code of CTPG is described in Algorithm 3.

B Details on Comparable Guide Q-Value

We expand the value function of SAC to obtain:

By repeatedly expanding V i , we get the final form, which is actually the optimization objective under the maximum entropy reinforcement learning framework.

Then, we consider that the trajectory generation process solely using the current task's control policy π i can be regarded as equipped with a special guide policy Π ˜ g i in SAC. The comparable guide Q-value in Equation 13 of this special guide policy Π ˜ g i is:

The ellipsis part is the repeated expansion of ˆ Q ˜ g i . Therefore, in SAC, the policy-filter gate can refine the action space of the guide policy by directly comparing ˆ Q g i ( s t , j t ) and V i ( s t ) .

C Details on Hindsight Off-Policy Correction for SAC

The hindsight off-policy correction mechanism is proposed to mitigate the non-stationarity challenge of the guide policy update process. It reassigns the action j t sampled by the past guide policy to a new one j ′ t , whose control policy π j ′ t is more likely to output the historical action sequence { a t ′ } t + K -1 t ′ = t . During the guide policy update process, we first get the reassigned action j ′ t following the maximum likelihood estimation in Equation 9. For SAC's critic update, the critic loss of the guide policy is,

where the hindsight correction explicitly modifies the update procedure by shifting the Q-value estimation from Q g i ( s t , j t ) to Q g i ( s t , j ′ t ) . For SAC's actor update, the actor loss of the guide policy is,

which is not explicitly different from the original function in SAC. However, it indirectly affects actor learning due to SAC's unique actor update method [9]. Specifically, SAC's actor loss is derived from its optimization objective, and the policy is updated according to,

where the partition function Z π old ( s t ) normalizes the distribution. In essence, SAC's policy is updated by fitting the distribution of the SoftMax function of Q-value with temperature α . Therefore, modifying the update of Q g i ( s t , j t ) to Q g i ( s t , j ′ t ) using hindsight correction leads to a different guide policy actor optimization objective, thus affecting the training of the guide actor network.

D Environment Details

D.1 MetaWorld Manipulation Benchmark

Reach

Push

Pick-Place

Door-Open

Drawer-Open

Drawer-Close

Button-Press-Topdown

Window-Open

Window-Close

文

The MetaWorld manipulation benchmark [31] consists of 50 robotics manipulation tasks employing a sawyer arm to acquire diverse manipulation skills. MetaWorld-MT50 encompasses all 50 manipulation tasks, and MetaWorld-MT10 encompasses 10 tasks (a subset of MT50), as illustrated in Figure 8. In MetaWorld, the source tasks are configured with fixed goals, limiting the policy's ability to generalize to tasks of the same type with varying goals. Following a similar setup as [27], we extend all the tasks to a random-goal setting, where both items and goals reset randomly at each episode's onset. In addition, unlike [27, 30], we conduct our experiments on the MetaWorld-V2 benchmark 2 .

In addition, we extend the episode length to 200 timesteps, different from the previous setting of 150. We evaluate the rule-based policies provided by the MetaWorld benchmark across 100 sample episodes. Under the initial episode length setting of 150 timesteps, 42 tasks achieve a success rate exceeding 90% . However, among the remaining 8 tasks, the task Disassemble exhibits a mere 50% success rate. In contrast, upon adjusting the episode length setting to 200 timesteps, success rates for all 50 tasks exceed 90% , with the task Disassemble achieving an improved success rate of 91% .

D.2 HalfCheetah Locomotion Benchmark

BigTorso

BigThigh

BigLeg

BigFoot

SmallTorso

SmallThigh

SmallLeg

SmallFoot

The HalfCheetah locomotion benchmark originates from the gym-extensions framework [11] 3 , which focuses on continuous control multi-task reinforcement learning tasks, including several task groups of the standard gym environments [2]. We run our experiments with the HalfCheetah agent, which is a 2-dimensional robot consisting of 9 links and 8 joints connecting them (including two paws). Each episode contains 1000 timesteps.

HalfCheetah-MT5 consists of 5 locomotion tasks. The HalfCheetah agent is tasked with running in environments with various scales of simulated earth-like gravity, ranging from one-half to one-and-a-half of the normal gravity level. The detailed gravity value of each task is illustrated in Table 2.

HalfCheetah-MT8 consists of 8 locomotion tasks, as depicted in Figure 9. The HalfCheetah agent must run with variations in the morphol-

ogy of a specific body part, such as the torso, thigh, leg, and foot. The Big body part involves scaling the mass and width of the limb by 1.25, and the Small body part involves scaling by 0.75.

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

12

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

12

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

12

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

12

×

10 3

)

0

2

4

6

8

Samples/Task (

×

10 5

)

0

2

4

6

8

10

12

×

10 3

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

80

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

80

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

80

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

80

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

80

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

Success Rate (

%

)

0

3

6

9

12

15

Samples/Task (

×

10 5

)

0

20

40

60

Success Rate (

%

)

Base

w/ QMP

w/ CTPG

(1) MTSAC

(2) MHSAC

(3) PCGrad

(4) SM

(5) PaCo

(a) HalfCheetah-MT5

(b) HalfCheetah-MT8

(c) MetaWorld-MT10

(d) MetaWorld-MT50

E Additional Experimental Results

E.1 Training Curves of Experiment with Implicit Knowledge Sharing Approaches

This section complements the result in Section 5.2, which only presents the final performance of the experiment. Here, we show the training curves for different combinations of explicit policy sharing methods and implicit knowledge sharing approaches across four environments in Figure 10. Each row of the figure represents a distinct implicit knowledge sharing approach, while each column represents a different environment. Within each subfigure, the three curves represent the base one without any explicit policy sharing method and two variations using different explicit policy sharing methods. We evaluate the training policy every 10K samples per task with 32 episodes and report the mean episode return or success rate, along with standard deviation, across 5 different seeds. The result shows that beyond the ultimate performance improvement, CTPG also enhances the sample efficiency.

E.2 Additional Results of Ablation Studies

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

Success Rate (

%

)

w/o Policy-Filter Gate

CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

w/o Guide-Block Gate

w/ Success Rate Metric

CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

w/o Hindsight Correction

CTPG

(a) Ablation on Policy-Filter Gate

(b) Ablation on Guide-Block Gate

(c) Ablation on Hindsight Correct

×

×

×

In addition to the ablation experiment of MHSAC with CTPG on MetaWorld-MT10 in Section 5.4, we also conduct ablation studies using SM implicit knowledge sharing approach on MetaWorld-MT50. Figure 11 shows similar results to those in Section 5.4. The policy-filter gate is crucial within CTPG. The guide-block gates with two metrics demonstrate competitive performance and show improvement compared to the absence of the guide-block gate. Additionally, the hindsight off-policy correction mechanism significantly enhances training efficiency and stability.

E.3 Ablation Study on Guide Step

Within CTPG, the guide policy Π g i selects a policy π j t from the candidate set of all control policies { π j } N j =1 every fixed K timesteps. The selected policy π j t is then used as the behavior policy to interact with the environment and collect data for the next K timesteps.

0

2

4

6

8

Samples/Task (

10 5

)

0

2

4

6

8

10

×

10 3

)

K = 1

K = 5

K = 10

K = 20

K = 50

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

80

Success Rate (

%

)

K = 1

K = 3

K = 5

K = 10

K = 20

(a) HalfCheetah-MT8

(b) MetaWorld-MT10

×

×

The guide step K is a predefined hyper-parameter. We conduct ablation experiments on the guide step K in the two setups: HalfCheetah-MT8 and MetaWorld-MT10. In HalfCheetah-MT8, an entire episode contains 1000 timesteps, so we set K ∈ { 1 , 5 , 10 , 20 , 50 } . In MetaWorld-MT10, an entire episode contains 200 timesteps, so we set K ∈ { 1 , 3 , 5 , 10 , 20 } . The result, shown in Figure 12, indicates that both short and long guide steps lead to decreased performance and increased variance.

Overall, CTPG performs well in both environments when K = 10 . Furthermore, automating the selection of the guide step K is part of our future work.

E.4 Ablation Study on Monte Carlo Sampling

In Section 4.2, we design a policy-filter gate to refine the action space of the guide policy by adaptively filtering out control policies that are not beneficial for guidance. Specifically, the restriction of the action space is implemented by a mask, which is generated by comparing the control policy's V-value V i ( s t ) and the guide policy's Q-value Q g i ( s t , j t ) .

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

80

Success Rate (

%

)

H = 1

H = 3

H = 5

H = 10

MetaWorld-MT10

×

In SAC, the V-value is the expectation of the Q-value with the entropy of policy. It is formulated as:

To avoid introducing an additional network to estimate the V-value V i ( s t ) , we estimate it via Monte Carlo sampling of the Q-values Q i ( s t , a t ) with a t ∼ π i ( a t | s t ) [15]. We conduct ablation experiments varying the Monte Carlo sampling times H , as depicted in Figure 13. It is observed that the performance remains largely unaffected by the number of samples, except in cases where H equals 1, which leads to performance degradation and heightened variance due to excessive randomness.

E.5 Additional Comparison with BPT

CTPG learns a flexible mixture policy combining different control policies. To verify the advantages of the mixture strategy, we compare CTPG against another baseline that incorporates a guide strategy from [26], which solely learns from another control policy with the highest performance on the task. Since [26] focuses on continual RL, which requires a predefined task sequence and does not align with the MTRL experimental setting. Therefore, we adapt the core idea, implementing a version that selects the best-performing policy for the current task to guide exploration and generate trajectory data. Specifically, after each H rounds of data collection, we evaluate all policies across all tasks, selecting the top-performing policy for each task to guide data collection in the subsequent rounds. We refer to this baseline as BPT (Best Performance Transfer) and set H to 50 episodes in our implementation. We use MTSAC in HalfCheetah-MT8 and MHSAC in MetaWorld-MT10.

0

2

4

6

8

Samples/Task (

10 5

)

0

2

4

6

8

10

×

10 3

)

MTSAC

MTSAC w/ BPT

MTSAC w/ CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

80

Success Rate (

%

)

MHSAC

MHSAC w/ BPT

MHSAC w/ CTPG

(a) HalfCheetah-MT8

(b) MetaWorld-MT10

×

×

The result, shown in Figure 14, demonstrates that CTPG outperforms BPT. CTPG's guide policy learns a more flexible strategy: it not only can learn to share a single policy within a complete

trajectory (like BPT), but also can develop a mixture policy by combining different control policies, making it more transferable between tasks. Notably, BPT performs better in HalfCheetah-MT8 than in MetaWorld-MT10 because the tasks in HalfCheetah-MT8 share more significant similarities, whereas the policy transfer and guidance in MetaWorld-MT10 require combination strategies.

E.6 Adaptability of Other RL Algorithms to CTPG

CTPG is a general MTRL framework that can be adapted with other RL algorithms, even allowing the control and guide policy to use different RL algorithms. Since SAC is a widely used algorithm in continuous control and serves as the base algorithm for all baselines, we also choose SAC as the base algorithm in this work. In addition, we explore the adaptation of TD3 with the CTPG and also employ different RL algorithms for the control and guide policies. Specifically, the control policy uses TD3, and the guide policy uses DQN. We evaluate MTTD3 (the original TD3 using one-hot encoding for task representation) on HalfCheetah-MT8 and MHTD3 (utilizing a multi-head network for different tasks) on MetaWorld-MT10. The result shown in Figure 15 demonstrates that CTPG can enhance performance and sample efficiency when combined with other backbone RL algorithms.

0

2

4

6

8

Samples/Task (

10 5

)

0

2

4

6

8

×

10 3

)

MTTD3

MTTD3 w/ CTPG

0

3

6

9

12

15

Samples/Task (

10 5

)

0

20

40

60

Success Rate (

%

)

MHTD3

MHTD3 w/ CTPG

(a) HalfCheetah-MT8

(b) MetaWorld-MT10

×

×

F Implementation Details

We implement all experiments using the MTRL codebase [21] 4 and access the HalfCheetah locomotion environment to this framework. One significant change we make involves removing the reward normalizer wrapper, which leads to worse results.

F.1 Details on Computational Resources

We use AMD EPYC 7742 64-Core Processor with NVIDIA Geforce RTX 3090 GPU for training. Each method is trained 5 times with different seeds. Using MetaWorld-MT10 as an example, the training time required for a full training run varies from 16 hours (MTSAC) to 52 hours (PcGrad) for the baseline implicit knowledge sharing approaches. For each approach with CTPG, the training times range from 20 hours (MTSAC w/ CTPG) to 56 hours (PcGrad w/ CTPG). The guide policy, using a simple multi-head network architecture, acts every K step ( K = 10 in our implementation), so its training frequency is set as 1 /K . Consequently, CTPG does not significantly increase the training time. For a detailed hyper-parameter of the guide policy, please refer to Appendix F.3.

F.2 Additional MTRL Training Setups

For a fair comparison, we use the following common training setups on all methods:

Disentangled SAC temperature parameters. This setup is standard established in MTRL. Herein, distinct tasks utilize varying SAC temperature parameters, as described in Equations 2 and 3, which are also optimized by independent losses in Equation 4. Therefore, this setup facilitates the individual adaptation of exploration and exploitation balances for each task, effectively accommodating the diverse learning dynamics across different tasks during training.

Loss maskout of extreme tasks. This setup, utilized by [21, 22], involves the selective masking of the potentially destabilized loss J i of task i from the total loss J , aiming to mitigate its adverse effects on other tasks. Specifically, when the task loss J i surpasses a predefined threshold ϵ (set as 3 e 3 , the same as [22]), that task i is excluded from the total training loss.

Multi-task loss rescaling. In recognition of the inherent discrepancy in convergence rates between tasks, where easy tasks usually converge faster, [27] propose an optimization objective weight of task i , formulated as:

where α i is the SAC temperature parameter of task i , and N is the total number of tasks. Consequently, the total loss is adjusted to J = E i [ w i · J i ] , ensuring a balanced training process across different tasks.

F.3 Hyper-Parameters of All Method

This section provides the hyper-parameters of each method in our experiment. General hyperparameters shared by all methods are illustrated in Table 3. Table 4 to Table 8 show the additional hyper-parameters specific to each implicit knowledge sharing approach. In addition, the guide policy can also be trained using implicit knowledge sharing methods. In our implementation, we use the simple multi-head network structure for the guide policy in CTPG, and the relevant additional hyper-parameters are displayed in Table 10.

G Broader Impacts

This paper presents work that aims to advance the field of Multi-Task Reinforcement Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The full code is provided in supplemental material and detailed settings are available in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The full code is provided in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details are provided in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The statistical significance of the experiments is provided in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experiments compute resources information is provided in Appendix E.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conduct in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses broader impacts in Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code this paper uses is properly mentioned in Appendix with footnotes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Cross-Task Policy Guidance (CTPG), a novel framework designed to improve multi-task reinforcement learning (MTRL) by leveraging cross-task policy similarities. CTPG trains a guide policy for each task to select the most suitable behavior policy from a pool of all tasks' control policies. This method aims to generate better training trajectories and enhance learning efficiency. The authors propose two gating mechanisms: a policy-filter gate to filter out non-beneficial control policies and a guide-block gate to block unnecessary guidance for mastered tasks. Empirical evaluations on manipulation and locomotion benchmarks demonstrate that integrating CTPG with existing parameter sharing approaches significantly enhances performance.\n\nSTRENGTHS:\n### S1. Innovative Approach\n\nThe paper introduces a novel method for leveraging cross-task similarities in MTRL. By training a guide policy to select behavior policies from a pool of tasks, the approach provides a direct and efficient way to exploit shared skills, which is an underexplored area in MTRL.\n\n### S2. Empirical Validation\n\nThe proposed framework is validated through extensive experiments on various benchmarks, including MetaWorld and HalfCheetah. The results demonstrate significant improvements in learning efficiency and performance when CTPG is integrated with existing MTRL approaches.\n\n### S3. Clear Presentation\n\nThe paper is well-structured and clearly explains the CTPG framework, including detailed descriptions of the guide policy and gating mechanisms. Figures and tables effectively illustrate the benefits of the proposed method, and the provided pseudocode enhances reproducibility.\n\nWEAKNESSES:\n### W1. Scalability Concerns\n\nThe scalability of CTPG to more complex and large-scale environments is not thoroughly discussed. While the method shows promising results in the tested benchmarks, a broader analysis of its scalability and practical utility in more complex scenarios may improve the significance of this work.\n\n### W2. Computational Complexity\n\nThe computational complexity of training and deploying CTPG, particularly the guide policy and the gating mechanisms, is not explicitly addressed. Understanding the computational requirements and potential limitations in terms of resources and execution time would provide a more comprehensive evaluation of its applicability.\n\nQUESTIONS:\n### Q1. Scalability to Complex Environments\n\nHow does the CTPG framework scale to more complex, large-scale environments? Are there any specific challenges or limitations that need to be addressed for practical deployment in such scenarios?\n\n### Q2. Computational Requirements\n\nWhat are the computational requirements for training and deploying CTPG? How does the method perform in terms of execution time and resource consumption compared to existing MTRL methods?\n\nLIMITATIONS:\nThe authors acknowledge the limitations related to the predetermined guide step K and the reliance on specific benchmarks. However, a more detailed discussion on potential negative societal impacts and strategies to mitigate them would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of multi-task reinforcement learning. Previous works have primarily focused on MTRL through specialized network structures or methods to resolve conflicting gradients. This paper considers the problem from an orthogonal perspective, by actively sharing control policies for each task adaptively, in the hope that sometimes the policy of one task can guide the exploration of other tasks. Specifically, similar to in hierarchical RL, a high-level policy is used to decide which “task policy” to call at each particular state, where “bad” policies are masked out through a learned value function. Evaluated on Metaworld and HalfCheetah MTRL benchmark, the method shows improved performance compared to previous works.\n\nSTRENGTHS:\n- The idea of exploring with cross-task policies to facilitate multi-task reinforcement learning is interesting and novel.\n- The method is thoroughly evaluated on a set of test domains, and show superior performance compared to previous methods\n- This paper is well written, and the method is easy to follow.\n\nWEAKNESSES:\n- The proposed method contains many floating pieces that seem to add burden both to implementation and hyperparameter tuning. Some of the pieces seem rather like hack than a principled solution. For example:\n  - For the hindsight off-policy correction, the action of the guide policy is relabelled to the one that is most likely to generate the sequence of actions. However, it should be likely that none of the control policies can generate the old sequence of actions with high probability, in which case relabeling will not help at all.\n  - The action space masking is essentially masking out control policies with low Q values. Shouldn’t this already be reflected in the policy through policy gradient updates? Why do we need additional masking?\n  - Using the temperature coefficient of SAC as an indication of the policy performance is very empirical and may not always be true. This also makes the proposed method specific to adaptive temperature SAC.\n- It seems that the hindsight off-policy correction would require much more forward passes than a regular SAC. Can the author show the performance of the proposed method with x axis being wall-clock time?\n- For the proposed method to be effective, we need the policy of some other tasks to have better performance than the policy that is trained for the current task. One scenario I can think of is when the multiple tasks secretly form a curriculum (e.g. one task is the prerequisite for another task), which might be the case in Metaworld and halfcheetah. But I’m not sure how realistic such an assumption is in the real world.\n\nMinors:\n- Line 207: dose → does\n- “someone who can ride a bicycle can quickly learn to ride a motorcycle by referring to related skills” → I don’t think that’s actually the case…\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of multi-task reinforcement learning (MTRL). To this end, the paper proposes a method that selectively shares behaviors from the policies learning to solve other tasks. The experiments conducted in a locomotion domain (multi-task Half-Cheetah) and a robot arm manipulation domain (Meta-World) verify that the proposed method can improve the performance of the learned policies. Ablation studies justify the effectiveness of many proposed components, including the policy-filter gate, the guide-block gate, and the hindsight correct. Yet, I am not convinced that the proposed method could improve the sample efficiency, which is the key goal of MTRL. Also, it seems that the proposed method is incremental and lacks novelty. Therefore, I am leaning toward rejecting this work in its current form.\n\nSTRENGTHS:\n**Motivation and intuition**\n- The motivation for sharing behaviors for MTRL is convincing.\n\n**Experimental results**\n- The main experimental results show that the proposed method achieves better converged performance compared to the baselines in MT Half-Cheetah and Meta-World.\n\n**Ablation study**\n- The ablation studies justify the effectiveness of the policy filter gate, guide block gate, and correct hindsight.\n\n**Reproducibility**\n- The code is provided, which helps understand the details of the proposed method.\n\nWEAKNESSES:\n**Main results (Table 1)**\n- Table 1 and Section 5.2 present quantitative results of converged performance. However, as far as I am concerned, the comparisons of MTRL should focus on sample efficiency, i.e., how fast each method can learn, instead of converged performance. Therefore, I am not convinced by this evaluation.\n\n**Comparison to k-step QMP**\n- The paper states that the proposed method outperforms QMP because \"guide policy learns long-term policy guidance.\" I am not entirely convinced. One can simply use QMP to select from k-step behavior proposals and execute the selected proposal, which can also achieve this long-term policy guidance. Including this variant of QMP would be necessary to show that the performance gain comes from other designs of the proposed method.\n\n**Guide step K**\n- The best guide step K in the MT Half-Cheetah and Meta-World are different. It is unclear, given a new MTRL domain, how we should choose K. It seems that we could only tune this hyperparameter via trial and error. \n\n**Backbone RL algorithms**\n- This work adopts SAC as the backbone RL algorithm. Is it possible to use other RL algorithms, such as TD3 or PPO?\n\n**Clarity**\n- Section 4 is difficult to follow. Sufficiently describing the intuitions before introducing each component of the proposed method would significantly improve the readability of this section.\n\n**Novelty**\n- The proposed method seems incremental given that the QMP paper explores this idea of behavior sharing. Despite the high similarity between this work and the QMP paper, the authors seem to deliberately hide this by avoiding discussing QMP in the introduction.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method called CTPG to enable policies trained on different tasks to learn from each others' generated trajectories. CTPG operates by learning a guideline policy that determines which control policy in a given set should best generate trajectories to enable agents to learn a particular task. Then, the authors also designed additional mechanisms that (i) deal with how each policy whose trajectories are sampled may change from their learning process, (ii) prevent negative transfer from using trajectories from irrelevant policies for learning, and (iii) promote the convergence towards good policies by preventing any transfer of experience once a policy is already proficient at its intended task.\n\nUsing environments from multi-task reinforcement learning benchmarks, the authors then investigated their method's performance when (i) learning a set of tasks in parallel and learning a new task that is not from the set of previously learned tasks. Furthermore, the authors also conducted further analysis to investigate (i) which component within their learning algorithm is most responsible for the method's performance and (ii) whether the guideline policy learned a sensible experience selection policy.\n\nSTRENGTHS:\n**Major Strength - Clarity**\n\nExcept for a few minor clarification details provided below, I find the paper to be well-written. I especially appreciate the clear outlining of (i) the motivation behind the proposed method, (ii) the questions being investigated by the authors, and (iii) the experiment design. I hope that future iterations of the manuscript will keep this same level of clarity.\n\n**Major Strength - Method Soundness**\n\nFrom a high-level perspective, it seems that the learned transfer learning mechanism proposed in this paper seems reasonable. I especially find the authors' design of various mechanisms to address different problems (i.e., policy selection, negative transfer, variable rates of learning between tasks) to be great choices that should improve learning performance in multi-task reinforcement learning scenarios.\n\n**Major Strength - Experiments and Analysis**\n\nI also think that the authors did a great job of thoroughly investigating the efficacy of their method. The five questions in the experiment section provided interesting insights into the method. At the same time, the experiments done to answer each of those questions were well-designed and showed the method's positive performance.\n\n**Minor Strength - Significance**\n\nWhile allowing agents to learn from each others' experience to improve their performance at achieving their respective objectives is not exactly new, to my knowledge, the authors' proposed method for learning in multi-task reinforcement learning (and addressing its various associated issues) is novel. Even if I am wrong on the novelty aspect of this method, I still believe the thorough analysis provided by the authors would contribute to different insights that may be useful for the multi-task RL community.\n\nWEAKNESSES:\n**Minor Weaknesses - Additional Comparisons**\n\nPerhaps another baseline that could be compared is the simple strategy from [1], where one simply learns from another policy with the highest performance at the task. Using this baseline should elucidate the effects of switching between different policies to transfer from as opposed to just transferring from a policy that seems to perform the best.\n\n**Minor Weaknesses - Hindsight Off-Policy Correction**\n\nFrom Section 4.1 alone, I also do not find how the hindsight off-policy correction mechanism affects the remainder of the learning process. While I did check Algorithm 2 in the appendix, I am uncertain why $j^{'}_{t}$ (the behavioral policy chosen in hindsight) only affects SAC's critic updates and not also the policy updates.\n\nReferences:\n\n[1] Disentangling Transfer in Continual Reinforcement Learning. Wolczyk et al. NeurIPS 2022.\n\nQUESTIONS:\n1. How would CTPG perform against a simple transfer learning strategy where experience is only transferred from the policy that has the best performance?\n\n2. Does the behavioral policy chosen in hindsight affect the actor-network training? If it does, how would it do so?\n\nLIMITATIONS:\nThe authors have sufficiently outlined their method's limitations and interesting directions for future work in the last paragraph of the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}