QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing
Authors: Grace Zhang
Ayush Jain
Injune Hwang
Shao-Hua Sun
Joseph J Lim
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-task Reinforcement Learning
Behavior Sharing
Primary area: reinforcement learning
TL;DR: Sharing behaviors between tasks via off-policy data collection accelerates multi-task reinforcement learning algorithms

Abstract: Multi-task reinforcement learning (MTRL) aims to learn several tasks simultaneously for better sample efficiency than learning them separately. Traditional methods achieve this by sharing parameters or relabeling data between tasks.  In this work, we introduce a new framework for sharing behavioral policies across tasks, which can be used in addition to existing MTRL methods. The key idea is to improve each task's off-policy data collection by employing behaviors from other task policies. Selectively sharing helpful behaviors acquired in one task to collect training data for another task can lead to higher-quality trajectories, leading to more sample-efficient MTRL. Thus, we introduce a simple and principled framework called Q-switch mixture of policies (QMP) that selectively shares behavior between different task policies by using the task's Q-function to evaluate and select useful shareable behaviors.  We theoretically analyze how QMP improves the sample efficiency of the underlying RL algorithm.  Our experiments show that QMP's behavioral policy sharing provides complementary gains over many popular MTRL algorithms and outperforms alternative ways to share behaviors in various manipulation, locomotion, and navigation environments. Videos are available at https://qmp-mtrl.github.io/.

Published as a conference paper at ICLR 2025
QMP: Q-SWITCH MIXTURE OF POLICIES FOR MULTI-TASK BEHAVIOR SHARING
Grace Zhang Ayush Jain'
Injune Hwang?
Shao-Hua Sun
Joseph J. Lim?
University of Southern California
KAIST Natioonal Taiwan University
ABSTRACT
Multi-task reinforcement learning (MTRL) aims to learn several tasks simulta- neously for better sample efficiency than learning them separately. Traditional methods achieve this by sharing parameters relaheled data between tasks. In this work, we introduce new framework for sharing behavioral policies across tasks, which can used in addition existing MTRL methods. The key idea is to improve each task's 's off-policy data collection by employing behaviors from other task policies. Selectively sharing helpful behaviors acquired one task collect training data for another task can lead to higher-quality trajectories, leading to more sample-efficient MTRL, Thus, we introduce simple and principled frame- work called Q-switch mixture of policies (QMP) that selectively shares behavior between different task policies by using the task's Q-function to evaluate and select useful shareable behaviors. We theoretically analyze how QMP improves the sample efficiency of the underlying RI. algorithm. Our experiments show that QMP's behavioral policy sharing provides complementary gains over many popular MTRL algorithms and outperforms alternative ways to share behaviors in various manipulation, locomotion, and navigation environments. Videos are available at http://qmp-mmtrl.gihheee
INTRODUCTION
In multi-task reinforcement learning, each task can benefit from the behaviors learned in others. Consider robot learning four tasks simultane- ously: opening and closing drawer and a door on tabletop, as illustrated Figure havior is defined as the policy of how the robot acts in a certain state, with the optimal behavior representing the best response, such as opening its gripper (action) when near the drawer handle (state) in the drawer-open task. As the robot learns, such behaviors are often shareable be- tween tasks. For instance, both drawer-open and drawer-close tasks require behaviors for grasp- ing the handle. Consequently, fines its ability to grasp the drawer handle in one task, it can incorporate these behaviors into the other, reducing the need to explore the entire action space randomly. Following this intuition, such behavior sharing across tasks to accelerate overall learning?
Narawer
Grasping Drawer
Craeping Door
be-
Approaching Tabletop
Hendle
Hondle
trawer Close
Wooor Closs
Selective Sharing for Pollcy Training
We propose sample-efficient MTRL framework that selectively shares behaviors by act- ing with other task policies for data collection. For example, Drawer Open and Drawer Close can share behaviors performed for grasping drawer handle, while Drawer Open. and Door Close share behaviors for approaching the tabletop.
Figure 1:
the robot
can we develop a general framework that leverages
Most multi-task reinforcement learning (MTRL) methods share task information via policy parame- ters (Vithayathil Varghese & Mahmoud, 2020) or data relabeling (Kaelbling, 1993). We propose new framework for MTRL: share behaviors between tasks improve data collection by employing potentially useful policies from other tasks for more informative training data. This approach offers simple, general, and sample-efficient approach that complements existing off-policy MTRL methods.
"Equal contribution. Correspondence to: {gracez, ayushj @usc.edu

---

Published as a conference paper at ICLR 2025
Prior works (Teh et al., 2017; Ghosh et al., 2018) share behaviors between task policies uniformly by regularizing to one shared distilled policy (Rusu et al., 2015). This introduces a bias towards the mean behavior and causes negative interference when tasks might require differing optimal behaviors from the same state. In contrast, reusing other policies for data collection does not introduce any bias.
We propose selective behavioral policy sharing novel and general mechanism to improve sample efficiency in any MTRL architecture. Our key insight is that behaviors being acquired in other tasks can help when appropriately selected and shared, as shown in human learners (Tomov et al., 2021). In the Drawer Open task, while learning to approach the drawer handle, the robot should share behaviors between the Drawer policies, but avoid Door policies which would lead it to the wrong object.
The key question with selective behavioral policy sharing is how to identify helpful behaviors from other policies in principled way. We propose a principled way of selecting shared behaviors: Q-switch Mixture of Policies (QMP). At each state, one policy from a mixture of all policies is selected to collect data. The Q-switch makes this selection based on which policy best optimizes the current task's 's soft Q-value because that is an estimate of the most helpful behavior for the current task. We prove that this selection mechanism preserves the convergence guarantees of the underlying RL algorithm and potentially improves sample efficiency. Crucially, QMP uses other tasks' policies only for data collection, allowing policy training to remain unbiased under any off- off-polic RL algorithm.
Our primary contribution is introducing behavioral policy sharing for MTRL as novel avenue of information sharing between tasks and addressing the problem of principled selective behavior sharing. Our proposed framework, Q-switch Mixture of Policies (QMP), can effectively identify shareable behaviors between tasks and incorporates them to gather more informative training data for off- off-policy RL. We prove that QMP's behavior sharing not only preserves the policy convergence of the underlying RL algorithm, but is at least as sample efficient. We demonstrate that QMP provides complementary gains to other forms of MTRL in range of manipulation, locomotion, and navigation tasks and performs well over diverse task families when compared to other behavior sharing methods.
RELATED WORK
Information Sharing in Multi-Task RL. There are multiple, mostly complementary ways share information in MTRL, including sharing data, sharing parameters or representations, and sharing behaviors. In offline MTRL. prior works selectively share data between tasks (Yu et al., 2021; 2022). Sharing parameters across policies can speed up MTRL through shared representations (Xu et al., 2020; D'Eramo et al., 2020; Yang et al., 2020; Sodhani et al., 2021; Misra et al., 2016; Perez et al., 2018; Devin et al., 2017; Vuorio et al., 2019; Rosenbaum et al., 2019; Yu et al., 2023; Cheng et al., 2023; Hong et al., 2022) and can be easily combined with other types of information sharing. Most similar to our work, Teh et al. (2017) and Ghosh et al. (2018) share behaviors between multiple policies through policy distillation and regularization. Vuong et al. (2019) identify which states between tasks share optimal behavior and regularize to each other there. These works share behaviors through regularization, biasing the policy objective when tasks have differing optimal behaviors. In contrast, our work selectively shares behavioral policies without modifying the training objective.
Multi-Task Learning for Diverse Task Families. Multi-task learning in diverse task families is susceptible to negative transfer between dissimilar tasks, hindering training. Prior works combat this by measuring task relatedness through validation loss on tasks (Liu et al., 2022; Ackermaan al., 2021) or influence of one task to another (Fifty et al., 2021: Standley et al., 2020) find task groupings for training. Other works focus on the challenge of multi-objectio
optimization (Sener & Koltun, 2018; Hessel et al., 2019; Yu et al., 2020; Liu al., 2021: Schaul et al., 2019; Chen et al., 2018; Kurin et al., 2022). Similar to these works, we identify that prior behavior-sharing MTRL approaches are susceptible to negative transfer. However, we avoid the challenge of negative transfer entirely by selectively sharing behaviors only during off-policy data collection.
Exploration in Multi-Task Reinforcement Learning. Our approach of modifying the behavioral policy to leverage shared task structures can be seen as a form of MTRL exploration, which we discuss further in Appendix Section 21c. Bangaru et al. (2016) encourage agents to increase their state coverage by providing an exploration bonus. Zhang & Wang (2021) study sharing information between agents to encourage exploration under tabular MDPs. Kalashnikov et al. (2021b) directly leverage data from policies of other specialized tasks (like grasping a ball) for their general task
2

---

Published as a conference paper at ICLR 2025
variant (like grasping object). In contrast to these approaches, we do not require a pre-defined task similarity measure or exploration bonus; we demonstrate in Section 6 that QMP works across many tasks and domains without these additional measures. Skill learning can be seen as behavior sharing in a single task setting such as learning options for exploration or heirarchical RL (Machado et al., 2017; Jinnai et al., 2019b;a; Hansen et al., 2019; Riemer et al., 2018). We also discuss the difference to single-task exploration in Appendix Section G.3.
Using Q-functions filters. Yu al. (2021) uses Q-functions filter which data should be shared between tasks in a multi-task setting. In the imitation learning setting, Nair et al. (2018) and Sasaki & Yamashina (2020) use Q-functions filter out low-quality demonstrations, so they are not used for training. In both cases, the Q-function is used to evaluate some data that can be used for training. Zhang et al. (2022) reuses pre-trained policies to learn a new task, using Q-function as filter to choose which pre-trained policies to regularize to as guidance. In contrast to prior works, our method uses Q-function to evaluate different task policies to gather training data.
PROBLEM FORMULATION
Multi-task reinforcement learning (MTRL) addresses sequential decision-making tasks, where an agent learns policy to act optimally in an environment (Kaelbling et al., 1996; Wilson et al., 2007). Therefore, in addition to typical multi-task learning techniques, MTRL can also share behaviors, ie., actions, to improve sample efficiency. However, current approaches share behaviors uniformly (Section 2), which assumes that different tasks' behaviors do not conflict. To address this limitation, seek to develop selective behavior- sharing method that can be applied in more general task families for sample-efficient MTRL.
Multi-Task RL with Behavior Sharing. We aim to simultaneously learn a (....... Tv) } of N tasks. Each task T; is a Markov Decision Process (MDP) defined by state space S, action space A. transition probabilities T, reward functions R,, initial state distribution Pi, and discount factor E [0, 1]. While we use S to denote shared state spaces for simplicity, our formulation extends to tasks with different state spaces it complements policy architectures that share state encoders. The agent learns set of N policies {TI..... TN J. where each policy n, (als) represents the behavior on task T,, The objective is to maximize the average expected return over all tasks,
DO 7 (s (s)
N
max {
E BRLWN
N
1=0
Unlike prior works, our tasks can exhibit conflicting optimal behaviors: for any π (als) may differ from (als). Thus, prior methods that bias policy learning objectives like direct policy sharing (Kalashnikov et al., 2021a) behavior regularization (Teh et al., 2017) would be suboptimal.
4 APPROACH
To improve the sample efficiency of multi-task RL, we propose framework that selectively incorpo- rates behaviors from policies of other tasks without introducing bias into the RL objective for the current task. We achieve this by using mixture of all policies as the behavioral policy for the current task, thereby modifying only its off-policy training data. However, naively mixing other policies into the current task's 's behavioral policy does not necessarily improve its sample efficiency. To address this, we derive specific definition of this mixture, named Q-switch Mixture of Policies (QMP), that selects policy based on the current task's Q-function (see Figure 2 and Algorithm 1) and prove that QMP guarantees greater than or equal sample efficiency than using the current task's 's policy alone.
4.1 MULTI-TASK BEHAVIOR SHARING VIA OFF-POLICY DATA COLLECTION
MTRL methods like Teh et al. (2017) use regularization to a common average policy to enforce task policies to share behaviors. However, this introduces bias to each policy's RL objective, leading to suboptimal actions in states where tasks require different actions. To address this, we propose using a mixture of policies for oof-cc data collection as the means of rehavior-sharing any given task, one of the task policies is selected to gather training data as the current behavioral policy. This approach is compatible with any off-policy RI. algorithm (Watkins & Dayan, 1992)
At each state in
3

---

Published as a conference paper at ICLR 2025
Behavior Proposals
Mixture of Policies
Q-switch
Train Q3 & Ri
6, niC (+)
argmaxy (i(s,a)
6. mgC (+)
Best Behavior
Gather Data
Replay
Buffer
d--1Tff (s)
Task
Task 2
Task 3
Figure 2: Our method (QMP) shares behavior between task policies in the data collection phase using mixture of these policies. For example, in Task 1, each task policy proposes action aj- The task-specific Q-switch evaluates each Q1 (s, az) and selects the best scored policy to gather reward-labeled data to train Qi and æ. Thus, Task 1 will be boosted by incorporating high-reward shareable hehaviors into тi and improving Q1 for subsequent Q-switch evaluations.
because the enviromment rewards help determine the best actions from the collected data. However, an effective mixture policy must choose the behavioral policies selective and principled way.
Definition 4.1 (Mixture of Policies). For each task T,, the mixture policy TT (a s) is defined as TTT (a s) = " (s..)......x)) (a s), where fi(s, ,,,.,,,,.,,,................ AN) S x IIN 4 f1...., NJ
} is
mixture- switch function that selects of the policies Mi,..., AN based on the current state
Our intuition of policy mixture shares inspiration with hierarchical RL (Çelik et al., 2021; Daniel al., 2016; End al., 2017: Goyal et al., 2019) where mixture of options is learned according to the downstream task(s). However, a key difference in an MTRL mixture is that each policy is optimized for its own specific task and not designed fit the task where the mixture is employed.
4.2 Q-SWITCH MIXTURE POLICIES (QMP)
We aim to derive principled mixture-switch function fi such that the mixture policy - selectively incorporates behaviors from other policies while being guaranteed to be no worse than the current task's policy T₁. We recall the generalized policy iteration procedure (Sutton & Barto, 2018) underlying single-task SAC (Haamoja et al., 2018): policy evaluation learns Q by minimizing the bellman error the collected data, and policy improvement follows Q by minimizing the KI. divergence between the new policy and the exponential of the current Q-function, Q=
exp Q
(st,
new
arg min DKL e'Ell
n'(- (t)
=="(st)
In practice, the gradient updates in SAC are gradual and do not instantly achieve this optimization in Eq. 1, leaving suboptimality gap to catch up to the Q-function. Thus, mixture policy лī! that selects the best policy from a set of all given policy candidates, including the current policy, ensures that rimix is least good m, for the current state s, while potentially being better optimizer of Eq. due to shareable behaviors from the other task policies:
DEL x'(11 exp( 1 (xp(2 (8.
exp (xT(2
min x'E(M....TN)
< DKL
TT.. st)
(2)
(s)
ZT.
Simplifying the expression on the left results in the following definition (derivation in Appendix B). Definition 4.2 (Q-switch Mixture of Policies: QMP). For task T, and available candidate policies (T J. the QMP T (a 8) selects a policy according to:
arg
max
Ems ( a)] ]
a7l aH(
(3)
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 shows that QMP be plugged into any MTRL framework, mak- ing complementary with various MTRL frameworks like parameter-sharinn and data relabeling (see Section 7.1). In practice, estimate the expectation in Eq. 3 by evaluat- ing the Q-value for the mean action of each task policy's s distribution x' ('((ss ignoring the entropy term. We do not find any empir- ical difference when using sampled esti- mate of the expectation (see Appendix G.2) or including the entropy term, the Q-value is the primary distinguishing factor between policies. In terms of compute, sampling from QMP's nr (als) does require more policy and Q-function evaluations. However, evaluations are parallelized and impact run- time negligibly, as shown in Appendix G.4.
Algorithm 1 Q-switch Mixture of Policies (QMP)
Input:
Task Set {T),... Tx)
Initialize { {Q+) Data buffers [D,
for each epoch do
for = 1 to NN do
while Task T; episode not terminated do
Observe state s
Compute x² in Eq. 3. Take action proposal from m3
Dj Pg+Di (s,a,r,,s))
end while end for
for i= 1 to N do
Update Tú, Qi using D; with SAC
end for end for Output: Trained policies {
While T can mistakenly choose a poor policy due to estimation error in Q this identical to Q-learning or SAC, where the Q-function would be inaccurate at less-seen states. In both Q-learning and QMP. this is corrected with online interactions where the Q-function is trained be more accurate in a subsequent iteration. Furthermore, Trtix actually better maximizes Q": than which is the objective under generalized policy iteration. Note that QMP does not exacerbate the problem of overestimation because the soft policy evaluation step stays the same, i.e., it uses and not Tii.x.
WHY QMP WORKS: THEORY AND DIDACTIC EXAMPLE
5.1
QMP: CONVERGENCE AND IMPROVEMENT GUARANTEES
QMP performs simultaneous MTRL by collecting data using a Q-switch guided mixture of poli- cies n. In Appendix C. we prove that QMP with underlying RL algorithm Soft-Actor Critic (SAC) (Haarnoja et al., 2018) shares the same convergence guarantees in a tabular setting. Par- ticularly, we show that under QMP, policy evaluation converges because QMP only modifies data collection of off-policy RL, policy improvement guarantees are preserved (Theorem 5.1), and policy iteration converges to an optimal policy least as sample-efficientl (Theorem C.2).
The key reason for better policy improvement of QMP over the current task policy n; is the arg max operation in Eq. 3, which ensures that the selected policy na miix E { optimizes the SAC objective at least as well as itself. We formalize this in Theorem 5.1 with proof in Appendix C.I. Due to the suboptimality gap Eq. in SAC, QMP can actually achieve better policy improvement when there are shareable behaviors between policies.
L.
Theorem 5.1 (Mixture Soft Policy Improvement). Consider zoll and its associated Q-function Qi- Apply SAC's policy improvement nith - Ti and then n. - के from Eq. 3. Then, Qx (st, at) 2 ( ax) 2 Q=a" at) for all tasks T; and for all (st, az) E S x with |A| < cc.
While QMP in Def. 4.2 applies any set of candidate policies (#...... TN) }. one expects n. to improve over T, when some π. * Ty proposes an action candidate better than n; for Task T;. This is more likely in MTRL. policies that share structure between tasks than an arbi- trary set of policies. For instance, if T, and T, share subtask that appears early in the episodes for Ty, then Ty would have already learned it before n; and be better policy for certain states of T,, according to Q,.
Q
M
27, ..?
Qi, T
mix
T
Figure 3: QMP generalized policy iteration
QMP making bigger policy improvement steps results in each iteration of generalized policy iteration making more progress towards optimality. This reduces the number of iterations required to converge, improving the sample efficiency of the algorithm illustrated in Fig. 3 and proved in Theorem C.2.
5

---

Published as a conference paper at ICLR 2025
(a) SAC:
(b) QMP:
(c) QMP:
Figure 4: 2D Point Reaching. We visualize the training trajectories of x with different sets of task policies (fixed but stochastic) and color each step with the policy that proposed it. (a) The single-task SAC policy cannot reach the goal. (b) With 3 diverse policies ( + n), QMP often selects other policies, showing the suboptimality gap from Q Eq. (c) When a highly relevant / policy is added, QMP often selects/ / as it is likely to best optimize the learned Q-function.
5.2 ILLUSTRATIVE EXAMPLE: 2D POINT REACHING
2D Point Reaching
We demonstrate when QMP can utilize alternate policy can- didates (πI,..., NN} } to more effectively learn a policy by bridging a policy improvement suboptimality gap as tries to follow Q in Eq. 1. Consider a 2D point-reaching task where the agent must navigate from the bottom-left corner (0, 0) to the goal in the top-right corner (10, 10). The point agent receives dense rewards based on its proximity the goal and takes incremental 2D actions (Az, Ay) E []1,
Average Recurn
SAC: QMP: 1
QMP: T+/
80
15 Environment Timesteps (10k)
05
1.
20
Figure shows that the SAC policy converges to a subop- timal solution. Fig. 4a confirms that the data collected by SAC policy never reaches the goal. This shows that if the suboptimality gap in æ is not successfully bridged, it can make the entire algorithm converge suboptimally.
Figure 5: QMP improves performance using other policies, increasingly so when they are task-relevant (5 seeds).
To illustrate the effect of QMP, we add 3 fixed gaussian policies centered on (1 + / or (f 7), and only let ı be trainable. Fig. 4b, 4c show that x. employs alternate policies at many states in data collection they optimize Eq. 3 better than ı itself. This selectivity enables n to generate more effective goal-reaching trajectories by bridging the suboptimality gap, resulting in better performance in Fig. 5. A policy like that is more relevant to the underlying task leads to a larger gain.
The same principle extends to the simultaneous multi-task RL setting. In MTRL, each task's policy continuously improves and can serve as a valuable candidate in the mixture for other tasks. QMP enables tasks to selectively share their behaviors, allowing each task to benefit from the progress of others. This mutual assistance accelerates learning across all tasks, as the mixture policy Tr for each task T; selects the most promising action proposals from all available policies according to the task-specific Q-function, guaranteed to be at least as good as n. itself. Consequently, MTRL combined with QMP leverages the collective knowledge of all tasks to bridge suboptimality gaps more efficiently, leading to improved sample efficiency and overall performance.
EXPERIMENTS
6.1 ENVIRONMENTS
We evaluate method in 7 multi-task designs in manipulation, navigation, and locomotion environ- ments, shown in Figure 6. These tasks vary in the degree of shared and conflicting behaviors between tasks and the number of tasks in the set. Further details in Appendix Section D.
Multistage Reacher: A 6 DoF Jaco arm learns 5 tasks with ordered subgoals. The first 4 tasks share some subgoals, while the 5th conflicting task requires the agent to stay at its initial position.
Maze Navigation: Building on point mass maze navigation (Fu et al., 2020), we define tasks with various start and goal locations exhibiting coinciding and conflicting segments in the optimal paths.

---

Published as a conference paper at ICLR 2025
fonwer
Cab
balance
(a) Jaco Reacher
(b) Maze Navigation
(c) Meta- Woold
(d) Walker2D
(e) Franka Kitchen
Figure 6: Environments & Tasks: (a) Multistage Jaco Reacher. The agent must reach different subgoals or stay still (Task 4). (b) Maze Navigation. The agent (green circle) must navigate to the goal (red circle). 4 other tasks shown in orange. (c) Meta-World: 10 table-top manipulation tasks. (c) Franka Kitchen: 10 tasks, interacting with one appliance or cabinet.
Meta-
Meta-World World Manipulation: We use three task sets based on the Meta- Meta-World World environment (Yu et al., 2019). Meta- Woold MT10 and Meta-World MTS0 sets of 10 and 50 table-top manipulation tasks involving different objects and behaviors. Meta-Worle CDS is 4-task setup proposed in Yu al. (2021) which places the door and drawer objects next to each other on the same tabletop that all 4 tasks (door open & close, drawer open & close) are solvable in simultaneous multi-task setup.
Walker2D: Walker2D is 9 DoF bipedal walker agent with the multi-task containing locomotion tasks proposed in Lee et al. (2019): walking forward, walking backward, balancing, and crawling. These tasks require different gaits without an obviously identifiable shared behavior the optimal policies but can still benefit from intermediate behaviors like balancing.
Kitchen: We use the challenging manipulation environment proposed by Gupta al. (2019) where 9 DoF Franka robot performs tasks in a kitchen. We create task set out of 10 manipulation tasks: turning on or off different burners and light switches, and opening or closing different cabinets.
6.2 BASELINES
We first select popular and representative MTRL methods that share other forms of information evaluate how behavior-sharing with QMP improves their performance:
No-Sharing consists of N (refers number of tasks) independent RL architectures where each agent is assigned one task and trained to solve it without any information sharing with other agents.
Data-Sharing (UDS) proposed in Yu et al. (2022) shares data between tasks, relabelling with minimum task reward. We modified this offline RL algorithm online.
Parameter-Sharing a multi-heas SAC policy sharing parameters but not behaviors over tasks.
We validate QMP's approach to share behaviors via off-policy data collection with other approaches:
No-Shared-Behaving consists N RL agents where each agent is assigned one task and trained to solve it without any behavior sharing with other agents: no bias and no sharing.
Fully-Shared wew---haaaaa--iiieee Behavvors is single SAC agent that learns one shared policy for all tasks, outputting the same action for a given state regardless of task (full parameter sharing): fully biased sharing.
Divide-and-Conquer RL (DnC) (Ghosh et al. (2018)) uses N policies that share behaviors through policy distillation and regularization to the mean (adapted for MTRL): biased objective for sharing.
DnC (Regularization Only) a no policy distillation variant of DnC we propose as a baseline.
QMP (Ours) learns N policies that share behaviors in off-policy data collection: unbiased sharing.
Our code is available here https://ittub..co/// wttpthhhw..om/////a/qqnppp We used SAC Haarnoja et al. (2018) for all environments and methods. All the non-parameter sharing baselines use the same SAC hyperparameters. Please refer to Appendix for complete details.
RESULTS
Our experiments address: (1) Does QMP provide complementary gains to other forms of MTRL? (2) How does sharing behavioral policies compare with alternate forms of behavior sharing? (3) Can QMP effectively identify shareable behaviors? (4) Ablating key components of QMP.
7

---

Published as a conference paper at ICLR 2025
Multistage Reacher
Maze Navigation
Meta-World CDS
Success Rate (96
Success Rate (%6)
Success Rate (%)
80-
80-
20
40-
42-
40
2.5
5.0
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Kitchen
Meta-Worid MT10
MT50
100
100
50-
Success Rate (%)
Success Rate (9)
80-
Success Rate (%)
60
30-
40
40-
20-
20-
20-
10-
0.00
1.25
2,50
3.75
5.00
10
20
30
0.0
7.5
15:0
22.5
30.0
Environment steps (1M)
Environment steps (1M)
Environment Steps (1M)
No-Sharing QMP
No-Saarnng
Data-Sharing- QMP
Data-Sharing
Parameters
Parameter-SSharing
Figure 7: Behavioral policy sharing is complementary. QMP (solid lines) shows general improve- ment over MTRL frameworks soloecc dashed lines) like No shared architecture (blue), shared parameters (pink), and shared data (green). Methods without parameter-sharinn on MT50 converge very slowly. Success rate means and std (shaded) are over N tasks, 10 episodes per task, and 5 seeds.
7.1 Is BEHAVIOR SHARING COMPLEMENTARY TO OTHER MTRL FRAMEWORKS?
We demonstrate that method is compatible with and provides complementary performance gains with other forms of MTRL that share different kinds of information, including parameter sharing and data sharing. We compare the performance between 3 base MTRL algorithms, No-Sharing, Parameter-Sharing. and Data-Sharing, with the addition of QMP in Figure 7. The No-Sharing baseline provides a baseline comparison of QMP's effectiveness on its own. For the Parameter-Sharing and Data-Sharing baselines we chose the base algorithms for their popularity and simplicity. In each case, we add QMP by simply replacing the data collection policy with T.t... We find that QMP is complementary to all three baseline frameworks, mostly with additive performance gains in sample efficiency and final performance, while not hurting the performance of the base method in all but one case (Data-Sharing in Kitchen). We additionally see that QMP improves PCGrad's (Yu et al., 2020) performance significantly in out of 4 environments tested in Appendix E.4. This shows that QMP is a simple and complementary addition to other forms of MTRL.
QMP significantly improves upon the No-Sharing baseline in all environments except Meta-Worke CDS where it performs comparatively. This demonstrates that sharing behavioral policies is promising avenue for efficient and performant MTRL. In the data-sharing comparison, we see that the addition of QMP improves or performs comparatively to the base algorithm. In Multistage Reacher and Maze Navigation, we see that both Data-Sharing and Data + QMP perform worse than the other MTRL methods, highlighting the fact that sharing data directly between tasks can be ineffective without access to a re-labeled task rewards like in our setting. In environments where data-sharing does well, like Meta-World CDS, we see that adding QMP does improve sample efficiency.
We find that Parameters + QMP generally outperforms Parameter-Sharing, while inheriting its sample efficiency gains. In many cases, the parameter-sharine methods converge sub-optimally, highlighting that shared parameter MTRL has its own challenges. However, in Maze Navigation, we find that sharing Parameters + Behaviors greatly improves the performance over both the Parameter- Sharing baseline and No-Sharing QMP variant of QMP. This demonstrates the additive effect of these two forms of information sharing in MTRL. The agent initially benefits from the sample efficiency gains of the multi-head parameter-sharing architecture, while behavior sharing accelerates learning by selectively using other policies to keep learning even after the parameter-sharin plateaus. demonstrating the compatibility between QMP and parameter sharing key ingredients to sample efficient MTRL. We further highlight that this benefit of QMP increases with the number of tasks increasing from 10 to 50 in World, where we see that QMP is actually more effective
effect
8

---

Published as a conference paper at ICLR 2025
Multistage Reacher
Maze Navigation
Kitchen
100
102
120
Success Rate (96)
Success Rate (9)
60-
Success Rate (%)
50-
40
20-
0.0
12
0.00
1,25
2.50
3:75
5.00
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Meta-Worid CDS
Meta-World MT10
Walker
1.00
100
Success Rate (9))
Success Rate (%)
Average Return
2.083
60-
2,003
40-
1.334
20-
20-
0,667
0.0
25
5.0
7.5
10.0
0.0
32.5
25.0
37.5
50.0
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
QMP[Ours]
No-Shared- Beaaved-
Fully-Shared- Sheye-hrrrd-Beeaiior
DnC
DnC (reg-only)
Figure 8: QMP reliably shares behaviors. In task sets exhibiting conflicting behaviors, QMP
consistently matches or exceeds baselines in rate of convergence and final performance.
when combined with parameter sharing in MT50 than in MT10. QMP scales well with the number of tasks and can actually perform better likely due more shared behaviors in the larger task set.
7.2 BASELINES: COMPARING DIFFERENT APPROACHES TO SHARE BEHAVIORS
To verify QMP's efficacy as behavior-sharing mechanism, we evaluate baselines that share behaviors in different ways on 6 environments in Figure 8. QMP reliably matches or exceeds other methods, especially in tasks that require conflicting behaviors, where alternate approaches are ineffective.
In the task sets with the most directly conflicting behaviors, Multistage Reacher and Maze Navigation, method clearly outperforms other behavior-sharing and data-sharing baselines. In Multistage Reacher, our method reaches > 90% success rate at 0.5 million environment steps, while DnC (reg.), the next best method, takes 3 times the number of steps to fully converge. The rest of the methods fail to attain the maximum success rate. We also see that QMP scales better from 3 to 10 tasks in Maze compared to other behavior sharing methods in Appendix Section E.2.
In the remaining task sets with no directly conflicting behaviors, we see that QMP is competitive with the best-performing baseline for more complex manipulation and locomotion tasks. Particularly, in Walker2D and Meta- World CDS, we see that QMP is the most sample-efficient method and converges to better final performance than any other behavior sharing method. In Kitchen, DnC (regularization only) also performed very well, showing that well-tuned uniform behavior sharing can be very effective in tasks without conflicting behavior. However, QMP also performs competitively and more sample efficiently, showing that QMP is effective under the same assumptions as uniform behavior-shariie methods but can do so adaptively and across more general task families. The Fully-Shared- -ehavvvd- baseline often performs poorly because it totally biases the policies, while the No-Shared S----hrr---BBhiiorrrrr is a surprisingly strong baseline as it introduces no bias.
MT10 and
7.3
CAN QMP EFFECTIVELY IDENTIFY SHAREABLE BEHAVIORS?
Figure 9a shows the average proportion of sharing from other tasks for Multistage Reacher Task over the course of training. We see that QMP learns to generally share less behavior from Policy 4 than from Policies 1-3 (Appendix Figure 20). Conversely, QMP in Task also shares the least total cross-task behavior (Appendix Figure 19). We see this same trend across all 5 Multistage Reacher tasks, showing that the Q-switch successfully identifies confficting behaviors that should not be shared. Further, Figure 9a also shows that total behavior-sharing from other tasks goes down over training. Thus Q-switch learns to prefer its own task-specific policy as becomes more proficient.
We qualitatively analyze how behavior sharing varies within single episode by visualizing QMP rollout during training for the Drawer Open task in Meta- World CDS (Figure 9b). We see that makes reasonable policy choices by (i) switching between all 4 task policies it approaches the

---

Published as a conference paper at ICLR 2025
Brower Open Thel: Grasping Orawer Handle
Policy Percentage Mixture
QMP Behavior Sharing Task 0
Policies 1, Palicy
Drawer
Pulling Handle
12
Pollcy Used
Drawer Open
Environment steps (1M)
Drstoer Open
Doar Down
Drawer Open
Docr Openn
Door Close
(a) Behavior-sharing over training
(b) Behavior-sharing in a single training episode.
Figure 9: (a) Mixture probabilities of other policies for Task in Multistage Reacher with the conflicting task Policy 4 shown in red. (b) Policies chosen by the QMP behavioral policy every 10 timesteps for the Drawer Open task throughout one training episode. The policy approaches and grasps the handle (top row), then pulls the drawer open (bottom row).
grasps the drawer-handle and (iii) using Drawer Open and Door Open policies as it pulls the drawer open (bottom row). In conjunction with the overall results, this supports our claim that QMP can effectively identify shareable behaviors between tasks. For details on this visualization and the full analysis results see Appendix Section F.
drawer (top row), (ii) using drawer-specific policies
Inspired by hierarchical RL (Dabney et al., 2021) and multi-tash exploration (Xu al., 2024), we briefly investigate temporally extended behavior sharing in Appendix E.6. Recently, Xu et al. (2024) showed that if one assumes high overlap between optimal policies of different tasks, other task policies can aid exploration. So, we simply roll out each policy QMP selects for a fixed number of steps. QMP theory no longer holds as it requires selecting a policy at every step. Yet, this naive temporally extended QMP yields improvements in environments like Maze with strong overlap.
7.4 ABLATIONS
Q-switch Ablations
We show the importance of Q-switch in QMP
100
(Def. 4.2) against alternate forms of policy mixtures (Def. 4.1). QMP-Uniform is uniform distribution over policies, f. = U(f1..... [[,.......,.... ,,N) and achieves only 60% success rate (Figure 10), showing the impor- tance of selectivity. QMP-Domain- -nowllee hand-crafted, fixed policy distribution based on an estimate of similarity between tasks. Multistage Reacher measures this similarity by the shared sub- goal sequences between tasks (Appendix D). QMP- Domain performs well initially but plateaus early, showing that which behaviors are shareable depends on the state and current policy. We further ablate the arg max in Q-switch against softmax varation re- sulting in a probabilistic mixture policy in Appendix Section G.I, and evaluating the mean policy actions (Appendix Section G.2) to validate our design.
Success Rate (%)
80
QMP
20
QMP-Uniform
OMP-Domain-Kine KnnwllnKnowledge
0.0
0.5
1.0
1.5
2.0
Environment steps (1M)
Figure 10: QMP outperforms alternate policy mixtures in Multistage Reacher.
8 CONCLUSION
We propose an unbiased approach to sharing behaviors via off-policy data collection in MTRL: Q-switch Mixture of Policies. We demonstrate empirically that QMP effectively improves the rate of convergence and task performance in manipulation, locomotion, and navigation tasks, and is guaranteed to be as good as the underlying RI. algorithm and complementary QMP does not assume that optimal task behaviors always coincide. Thus, its improvement magnitude limited by the degree of shareable behaviors and the suboptimality gap that exists. At the same time, this lets QMP be unbiased and find optimal policies with convergence guarantees while being equally or more cample-efficient Since QMP only shares behaviors via off- off-policy data collection, is not applicable to on on-policy RL base algorithms like PPO (Schulman al., 2017). Promising future directions include temporally-extente behavior sharing and incorporating other forms of prior task information on shareable behaviors, such language embeddings in instructionfollowiinn tasks.
alternate MTRL.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Jesse Zhang for his assistance with writing and discussions. This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant (No.RS- 2019-111 Artificial Intelligence Graduate School Program, KAIST) and National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT). Grace Zhang and Ayush Jain were supported partly as interns Naver AI Lab during the initiation of the project. Shao-Hua Sun was supported by the Yushan Fellow Program by the Ministry of Education, Taiwan.
Brain Pool Research Program),
REFERENCES
Johannes Ackermann, Oliver Richter, and Roger Wattenhofer. Unsupervised task clustering for multi- task reinforcement learning. In Nuria Oliver, Fernando Pérez-Cruz, Stefan Kramer, Jesse Read, and Jose A. Lozano (eds.), Machine Learning and Knowledge Discovery in Databases. Research Track, pp. 222-237, Cham, 2021. Springer International Publishing. ISBN 978-3-030-86486-66
Sai Praveen Bangaru, JS Suhas, and Balaraman Ravindran. Exploration for multi-task reinforcement learning with deep generative models. arXiv preprint arXiv:1611.098%) 2016.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: /www. wandb. com/. Software Cooftarrevaaill available from wandb.com.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1rXiv:1606.015444 2016.
Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning. 2018.
Yuan Cheng, Songtao Feng, Jing Yang, Hong Zhang, and Yingbin Liang, Provable benefit of multitask representation learning in reinforcement learning. In Neural Information Processing Systems, 2023.
Will Dabney, Georg Ostrovski, and Andre Barreto. Temporally-extended e-greedy exploration. In International Conference on Learning Representations, 2021.
Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hicrarchical relative entropy policy search, Journal of Machine Learning Research, 2016.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters, et al. Sharing knowledge in multi-task deep reinforcement learning. In International Conference on Learning Representations, 2020.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In IEEE International Conference on Robotics and Automation, 2017.
Felix End, Riad Akrour, Jan Peters, and Gerhard Neumann. Layered direct policy search for learning hierarchical skills. In IEEE International Conference on Robotics and Automation, 2017.
Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn, Efficiently identifying task groupings for multi-task learning. In Neural Information Processing Systems, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https: // //github com/ rlworkgroup haapkgggoopgggrggg.eee garage, 2019.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. In International Conference Learning Representations, 2018.

---

Published as a conference paper at ICLR 2025
Ruben Glatt, Felipe Leno Da Silva, Reinaldo Augusto da Costa Bianchi, and Anna Helena Reali Costa. Decaf: Deep case- base-based policy inference for knowledge transfer in reinforcement learning. Expert Systems with Applications, 2020.
Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Bengio. Reinforcement learning with competitive ensembles of information-constraiiii primitives. arXiv, abs/1906. 00066.00633 10667, 2019.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning. 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. 2018.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906. 05030, 2019.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. AAAI Conference on Artificial Intelligence, 2019.
Sunghoon Hong, Deunsol Yoon, and Kee-Eung Kim. Structure-aware transformer policy for inhomo- geneous multi-task reinforcement learning. In International Conference Learning Representa- tions, 2022.
Yuu Jinnai, Jee Won Park, David Abel, and George Konidaris. Discovering options for exploration by minimizing cover time. In International Conference on Machine Learning. 2019a.
Yuu Jinnai, Jee Won Park, Marlos C Machado, and George Konidaris. Exploration in reinforcement learning with deep covering options. In International Conference on Learning Representations, 2019b.
Leslie Pack Kaelbling. Learning achieve goals. In International Joint Conference on Artificial Intelligence, 1993.
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 1996.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint arXiv:2104.08211 12, 2021a.
Dmitry Kalashnikov, Jake Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Scaling up multi-task robotic reinforcement learning. In Conference on Robot Learning, 2021b.
Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and M. Pawan Kumar. In defense of the unitary scalarization for deep multi-task learning. arXiv preprint arXiv:2201.04122. 2022.
Youngwoon Lee, Shao- Hua Sun, Sriram Somasundaram, Edward S. Hu, and Joseph J. Lim. Com- posing complex skills by learning transition policies. In International Conference on Learning Representations, 2019.
Bo Liu, Xingchao Liu, Xiaojic Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Neural Information P Processing Systems, 2021.
Shikun Liu, Stephen James, Andrew J Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research, 2022.
12

---

Published as a conference paper at ICLR 2025
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11089 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In IEEE Conference Computer Vision and Pailern Recognition, 2016.
Siddharth Mysore, George Cheng, Yunqi Zhao, Kate Saenko, and Meng Wu. Multi-critic actor learn- ing: Teaching rl policies to act with style. In International Conference on Learning Representations, 2022.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcom- ing exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation, 2018.
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J. Lim. Skill-based meta-reinforcement learning. In International Conference on Learming Representations, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems, 2019.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In AAAI Conference on Artificial Intelligence, 2018.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning, 2021.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 10445-10455, Red Hook, NY, USA, 2018. Curran Associates Inc
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint urXiv:1904.110 2019.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06290 2015.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- tional Conference on Learning Representations, 2020.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu, Ray interference: a source of plateaus in deep reinforcement learning. arXiv preprint /rXiv: 101990...45,,,. 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634 0637777777777, 2017.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Neural Information Processing Systems, 2018.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In International Conference Machine Learning, 202..
Trevor Standley, Amir R. Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning. 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Yee Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nico- las Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Neural Information Processing Systems, 2017.
13

---

Published as a conference paper at ICLR 2025
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In International Conference on Intelligent Robots and Systems, 2012.
Momchil S Tomov, Eric Schulz, and Samuel J Gershman. Multi-task reinforcement learning in humans. Nature Human Behaviour, 2021.
Nelson Vithayathil Varghese and Qusay H Mahmoud. survey of multi-task deep reinforcement learning. Electronics, 2020.
Tung-Long Vuong, Do- Van Nguyen, Tai-Long Nguyen, Cong-Minh Bui, Hai-Dang Kicu, Viet-Cuong Ta, Quoc-Long Tran, and Thanh-Ha Le. Sharing experience multitask reinforcement learning. In International Joint Conference Artificial Intelligence, 2019.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph Lim. Multimodal model-agnostic meta- learning via task- modulation. In Neural Information Processing Systems, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In International Conference on Machine Learning, 2007.
Zhiyuan Xu, Kun Wu, Zhengping Che, Jian Tang, and lieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. In Neural Information Processing Systems, 2020.
Ziping Xu, Zifan Xu, Runxuan Jiang, Peter Stone, and Ambuj Tewari. Sample efficient myopic ex- ploration through multitask reinforcement learning with diverse tasks. International Conference on Learning Representations, 2024.
Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft
modularization. Neural Information Processing Systems, 2020.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. 2019.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Neural Information Processing Systems, 2020.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. In Neural Information Processing Systems, 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, 2022.
Yuanqiang Yu, Tianpei Yang, Yongliang Lv, Yan Zheng, and Jianye Hao. T3s: Improving multi-task reinforcement learning with task-specific feature selector and scheduler. International Joint Conference Neural Networks, 2023.
Chicheng Zhang and Zhi Wang. Provably efficient multi-task reinforcement learning with model transfer. In Neural Information Processing Systems, 2021.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. International Conference on Learning Representations, 2020.
Jin Zhang, Siyuan Li, and Chongjie Zhang. CUP: Critic-guided policy reuse. Neural Information Processing Systems, 2022.
Onur Çelik, Dongzhuoran Zhou, Gen Li, Philipp Becker, and Gerhard Neumann. Specializing versatile skill libraries using local mixture of experts. In Conference on Robot Learning, 2021.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
Table of Contents
A Qualitative Results
16
QMP Derivation
16
C QMP Convergence Guarantees
16
D Environment Details
19
D.1 Multistage Reacher
19
D.2 Maze Navigation....
20
D.3 World Manipulation
20
Walker2D
21
D.5 Kitchen
................................
21
E
Additional Results
21
E.1
Multistage Reacher Per Task Results
21
E,2
QMP Scales with Task Set Size in Maze Navigation
22
E,3
QMP Outperforms Data Sharing with Reward Labeling
22
E.4
PCGrad Results
23
Additional MTRL Comparisons
23
E.6
Temporally-Extente Exttended Behavior Sharing.
24
F
QMP Behavior Sharing Analysis
25
F.1
Qualitative Visualization of Behavior-Shariee
26
G Additional Ablations and Analysis
26
Probabilistic Mixture v/s Arg-Max
26
G.2 Approximation Expected Q Q-value Over Policy Action Distribution
27
G.3 QMP v/s Increasing Single Task Exploration
27
G.4 QMP Runtime
27
H Implementation Details
27
H.1 Hyperparameters.
28
H.2 No-Shared--Bchavirrs
................................
28
28
H.4
DnC
28
H.5 QMP (Ours)
29
H.6 Online UDS
29
15

---

Published as a conference paper at ICLR 2025
A QUALITATIVE RESULTS
The qualitative result videos are provided at https:: qmp- http//ppppmtttrr .ar..uggtthb io/
B QMP DERIVATION
Following Section 4.2, we aim to derive the mixture-switch function fi such that the mixture policy Tith is guaranteed be better than the current task's policy яN We use the generalized policy iteration procedure (Sutton & Barto, 2018) underlying single-task SAC (Haarnoja et al., 2018): policy evaluation learns Q by minimizing the bellman error on the collected data, and policy improvement follows Q by minimizing the KI. divergence between the new policy and the exponential of the current Q-funciio,, QTT, shown in Eq.
In practice, the gradient updates in SAC are gradual and do not instantly achieve this optimization in Eq. 1, leaving suboptimality gap catch up the Q-function. We observe that due to the potential similarity of some tasks in MTRL, this suboptimality gap can be bridged using other policies. Concretely, mixture policy क.. that selects the best policy from a of all given policy candidates, including the current policy, ensures that urnix is an improvement over m, for the current state
Given a set of policies {.. including the current task policy ग. and given state 8, consider the following mixture policy:
expl 11 (TTTT((s)) 7x (s)
mix
arg
min x' (F
DRL.
22(118)
(4)
This mith is a better policy improvement solution to Eq. than Ty. because:
(xp.) Z
exp( ( (
min r'E r'E(,....TN))
DKL
(-)-
< DRL T=(. ss))
z (st)
Now, we can simplify Eq. 4 to obtain Definition 4.2:
exp( ( Qs) a
mix = arg
min
DKL
7'(- <1(1
T'E (N)
2T (s)
eep( 1 (T
arg
min ((((H....NN)
E (1)o()))))
log x (aaas)))))))))))))){( - log
(T)
arg max T'E(NN....NN)
E B (118)
log xs (als)) + QT (s, - loog 2T
1 Q))
arg
max ('E(TT....TN))
B |- log <a(aa))] + (
E Ba [s, aHl in' [)]
arg
max T' T'E(T.....TN)
Thus, the following mixture policy guarantees improvement over π
mix
arg
max
Eawn (Q* + aH (H(( [s)]
E'E (Res...N))
C QMP CONVERGENCE GUARANTEES
We derive the convergence guarantees for mixture soft policy iteration used in the QMP Algorithm 1. We augment the derivation of soft policy iteration in SAC (Haarnoja et al., 2018), which is
16

---

Published as a conference paper at ICLR 2025
base algorithm, with our proposed QMP's mixture policy. Soft policy iteration follows generalized policy iteration (Sutton & Barto, 2018) which refers to the general idea of repeated application of (I) policy evaluation update the critics and (2) policy improvement based on the updated critics, until convergence. Like SAC, we consider the tabular setting and show that QMP's modification to soft policy iteration converges to the optimal policy. Further, QMP can lead to improved policy improvement step when there are shareable behaviors between tasks, consequently improving the sample efficiency. The derivation sketch follows:
1. Soft Policy Evaluation: QMP modifies the off-policy data collection pipeline by replacing the primary task policy и, with the mixture policy =; However, it does not affect the soft Bellman backup operator of SAC, as shown in Haarnoja et al. (2018), and therefore the Q function still converges as SAC.
2. Mixture Soft Policy Improvement: QMP performs policy improvement in two steps: SAC's policy update from nelh and applying the mixture of policies from mix
Soft Policy Improvement: Since QMP does not modify the SAC update proce- dure z7 nT we directly SAC's guarantees of policy improvement following Lemma 2 from Haarnoja et al. (2018).
Mixiure Policy Improvement: We demonstrate QMP's mixture policy Tith guarantees a better policy improvement over the per-task policies т, that compose the mixture. In Theorem C.I, show convergence guarantee by proving that the expected return following .,; mixx is better than following mald
3. Mixture Soft Policy Iteration: In Theorem C.2, we show that the repeated application of the above steps in QMP converges to an optimal policy for each task. Furthermore, the convergence rate faster because of a greedier policy improvement due to Mixture Policy Improvement.
For a given stochastic policy ग and task T, E (.T...TTT TNJ. define V as the expected return of acting T with æ. Given another stochastic policy n', define (s, (')) (s) Q7 (s, a) as the expected return of acting with n' only in and thereafter with
Theorem C.1 (Mixture Soft Policy Improvement). Consider nald and its associated Q-function Qi- Apply SAC's policy improvement nQ Ti and then Te + σ: from Eq. 3. Then, Q= (st, ax) 2 QTi (St, at) 2 Q (s),at) for all tasks T; and for all (St, ar) E x A with |A| < oc.
Proof. From Soft Policy Improvement, Lemma 2 of Haarnoja et al. (2018), we have
Earwr [Q (s:, as) - log n, (arls:) 2 V= (st).
Rewrite the difference as 6(s)).
8(s:) = Earn, (s,, ) loo n. (arls)) - (st) 2 0.
From Eq. 3,
n: mix
arg max E Eg (11)) (Q**(s. a7l (z'(1 [s)]
T'E(T......))
Therefore, we have a positive difference (st).
w(s)) = Engeryi (Q=" (Sr, as) )- - -log Ti (ia (a//st) - Enawn, [Q= (st,
(st, a.) )- - log T: (a. 20.
17

---

Published as a conference paper at ICLR 2025
We use ò to expand the soft Bellman equation to derive the relationship between Q and Q"
Qaa (st, at) = r(St, a²) + Bu up
V (8++1)
rr(s,, ar) + Betting Em
(0xt (8 - log n.. (2 0(8++1)
00
00
EsLtE [r(sc+s; acte) - log mi (ac+4x/5444))
Ea [ō($++k)
k=0
A1
2 (s:, at) A
Likewise, use 8 and w to derive the relationship between Q and Q
QTt" (s,, a,) r(s,, ax) + y F timp
Vm ($1+1)
-r(st, at) + E8 B2 mop
Ea-
(Q
(S1+1, ac+1) - log ग., (a.-1 -6(s(+1)
E 1m (Q=²" ($1+1, a(-1) - log mix (a<+1/$(+1)) - 5($(+1) - w(s++1)
r()),a)) + Eg Bstting
2
Pa ae [r(sc+k, ax+x)
log mix (ar4(((())))))))
k=0
""t"((s,,0) (s.,at)
8
E Ea (8(s++))]
ES [+++++)]
A
= =Q- Ss ax)
Aa
Ω
We assume that the effect of the difference A2 A due to different state coverage is lower compared to the effect of Ω because w is accumulated at every state, i.e., A2 + A + +(22 + 2 A
Since A1, A2 2 0 and 2 0, we have
Q th (St. ar) 2 ( a²) 2 Q" (sc, az)
Theorem C.2 (Mixture Soft Policy Iteration). Repeated application of (i) soft policy evaluation and (ii) mixture soft policy improvement (Theorem C.I) ) to any
II converges policy nद such that i (st, ar) 2 Q (st, a.) for all ni E and (st, ai) E S x A with |A] < co. Furthermore, the sample efficiency and rate of convergence is at least good as SAC in the presence of mixture policy improvement.
optimal
Proof. Let o be the policy at iteration k. By SAC's soft policy iteration, the sequence Q is monotonically increasing, because - only modifies the online data collected and SAC is an off- policy algorithm. Thus, Theorem (Soft Policy Iteration) from Haarnoja al. (2018) Appendix B.3 directly applies here and proves that repeated application of soft policy evaluation and soft policy improvement converges an optimal policy क..
Mixture soft policy improvement (Theorem C.I) shows that milii is greedier policy improvement over ग. with respect to each estimate of Q Thus, the expected returns in the data collected by QMP mack policy, Q is greater than equal to that collected by the individual task policy, Q. Therefore,
18

---

Published as a conference paper at ICLR 2025
every mixture soft policy improvement step constitutes a larger policy improvement step than SAC's soft policy improvement step. This makes the convergence of mixture soft policy iteration (repeated application of soft policy evaluation and Theorem C.I) an improvement over soft policy iteration.
D ENVIRONMENT DETAILS
D.I MULTISTAGE REACHER
We implement multistage reacher tasks the Open AI Gym (Brockman al., 2016) Reacher environment simulated in the MuJoCo physics engine (Todorov et al., 2012) by defining a sequence of 3 subgoals per task, as specified in Table For all tasks, the reacher is initialized at the same start position with small random perturbation sampled uniformly from [-0.01, 0.01] for each coordinate. The observation includes the agent's 's proprioceptive state and how many subgoals have been reached but not subgoal locations, as they must be inferred from the respective task's 's reward function.
We set up the tasks to ensure that we can evaluate behavior sharing when the task rewards are qualitatively different (see Figure 6a):
For every task except Task 3, the reward function is the default gym reward function based
on the distance to the goal. plus an additional bonus for every subgoal completed.
For Task 1, the reward is shifted by at every timestep.
Task 3 receives only sparse reward of 1 for every subgoal reached.
Task has one fixed goal set at its initial position.
Subgoal
Subgoal 2
Subgoal
To
(0.2, 0.3, 0.5) (0.2, 0.3, 0.5)
(0.3, 0, 0.3)
(0.4, -0.3, 0.4)
Ti
(0.3, 0, 0.3)
(0.4, 0.3, 0.2)
T2
(0.3, 0, 0.3)
(0.4, 0.3, 0.2)
(0.4, -0.3, 0.4)
T3
(0.3, 0, 0.3)
(0.4, -0.3, 0.4) initial
(0.2, 0.3, 0.5)
T4 initial
initial
Table 1: Coordinates of subgoal locations for each task in Multistage Reacher, Shared subgoals are highlighted in the same color. For Task 4, the only goal is to stay in the initial position.
QMP-Domain: Section 7.4 ablates the importance of an adaptive and state-dependent Q-switch by distribution over other tasks based on apparent task similarity. replacing it with a domain-depende Specifically, to define the mixture probabilities for QMP-Domain in Multistage Reacher, we use the domain knowledge of the subgoal locations of the tasks determine the mixture probabilities. We use the ratio of shared sub- sub-goal sequences between each pair of tasks (not necessarily the shared subgoals) over the total number of sub-goal sequences, 3, to calculate how much behavior must be shared between two tasks. For that ratio of shared behavior, we distribute the probability mass uniformly between task policies that share that behavior. For Task 4, the conflicting task, we do not do any behavior sharing and only use TT gather data.
Each Task T. consists of sub-goal sequences {So, S1, S2} (c.g. [initial + Subgoal 1]. [Subgoal 1 Subgoal 2]. and [Subgoal 2 + Subgoal 3J). For each sequence E {So. S1, S2). we divide equally the contribution of each task T, 's policy Ty that shares the sequence (i.e. if To and T1 both contain sequence where we use the notation 1(s T₃) as the indicator function for whether Task T; contains sequence 8, then To and क both have contribution for s). Each sequence contributes equally the overall mixture probabilities for Task i (i.e. all policies that shares sequence S; contributes in total 1 to the mixture probability for the behavior policy of Task T,). Thus, the contribution probability of Policy x, Task T, is:
1 1(s ((SET)) T2)
Σ
Pi-+
3 Σx 1(s TT) oE S1,
mix Σ Pi-vi Tyj
n:
19

---

Published as a conference paper at ICLR 2025
Reusing notation for mixture probabilities, we have,
0
mir
To
+
3
T1
2
7
TO
+
R1
3
22
mir
T22
+
33
5
33
mix
+
6 33
4
D.2 MAZE NAVIGATION
The layout and dynamics of the maze follow Fu et al. (2020), but since their original design aims to train single agent reach fixed goal from multiple start locations, modified it have both start and goal locations fixed in each task, as in Nam et al. (2022). The start location is still perturbed with small noise avoid memorizing the task. The observation consists of the agent's current position and velocity. But, it lacks the goal location, which should be inferred from the dense reward based on the distance to the goal. The action space the larget 2D velocity of the point mass agent.
The layout we used is LARGE MAZE which is an 8x 11 maze with paths blocked by walls. The complete set of 10 tasks is visualized in Figure 12, where green and red spots correspond to the start and goal locations, respectively. The environment provides agent dense reward of exp( - dist) where dist is linear distance between the agent's current position and the goal location. It also gives penalty of at each timestep in order prevent the agent from exploiting the reward by staying near the goal. The episode terminates either as soon as the goal reached by having disl < 0.5 or when 600 timesteps have passed.
GP
GPS
GAS
G
G
Figure 12:
Ten tasks defined for the Maze Navigation. The start and goal locations each task are shown in green and red spots, respectively, and an example path is shown in green.
D.3 MORA-WORLD MANIPULATION
For Meta-World World CDS, we reproduce the Meta-world environment proposed by Yu al. (2021) using the Meta-world codebase (Yu et al.. 2019), where the door and drawer are both placed side- by-side on the tabletop for all tasks (see Figure 6c). The observation space consists of the robot's proprioceptive state, the drawer handle state, the door handle state, and the goal location, which varies based on the task. Unlike Yu et al. (2021), we additionally remove the previous state from the observation space so the policies cannot easily infer the current task, making it challenging multi-task environment. The environment also uses the default Meta- World reward functions which composed of two cestance-based rewards: distance between the agent end effector and the object, and distance between the object and goal location. We use this modified environment instead of the Meta-world benchmark because problem formulation of simultaneous multi-task RI. requires
20

---

Published as a conference paper at ICLR 2025
consistent environment across tasks. For Meta- World MT10, we directly use the implementation provided in (Yu et al., 2019) without changes.
In both cases, the observation space consists of the robot's proprioceptive state, locations for objects present in the environment (ic. door and drawer handle for CDS, the single target object location for 1T and the goal location. In Meta-World CDS, due to the shared environment, there no directly conflicting task behaviors, since the policies either go to the door or the drawer, they should ignore the irrelevant behaviors of policies interacting with the other object. In Meta-Worke MT10, each task interacts with a different object but in an overlapping state space so there is mix of shared and conflicting behaviors.
D.4 WALKER2D
Walker2D is DoF bipedal walker agent with the multi-task set of tasks proposed and implemented by Lee et al. (2019): walking forward at target velocity, walking backward at target velocity. balancing under random external forces, and crawling under a ceiling. Each of these tasks involves different gaits or body positions to accomplish successfully without any obviously identifiable shared behavior the optimal policies. Behavior sharing can still be effective during training to aid exploration and share helpful intermediate behaviors, like balancing. However, there is no obviously identifiable conflicting behavior either in this task set. Because each task requires a different gait, is unlikely for states to recur between tasks and even less likely for states that are shared to require conflicting behaviors. For instance, it is common for all policies to struggle and fall at the beginning of training, but all tasks would require similar stabilizing and correcting behavior over these states.
D.5 KITCHEN
We modify the Franka Kitchen environment proposed by Gupta al. (2019) and based the implementation from Fu et al. (2020). Since this environment is typically used for long horizon or offline RL, we chose shorter tasks that are learnable with online RL. Furthermore, we added dense reward based on the Meta-World reward function. We formed our 10 task MTRL set by choosing 10 available tasks in the kitchen environment that interacted with the same objects: turning the top burner or off, turning the bottom burner on or off, turning the light switch on and off, open or closing the sliding cabinet, and opening and closing the hinge cabinet. The observation space consists of the robot's s state, the location of the target object, and the goal location for that object. Similar to the Meta-World CDS environment, these tasks may share behaviors navigating around the kitchen to the target object but have plenty of irrelevant behavior between tasks that interact with different objects and confliciig behaviors when opening or closing the same object.
E ADDITIONAL RESULTS
E MULTISTAGE REACHER PER TASK RESULTS
Additional results and analysis on Multistage Reacher are shown in Figure 13. QMP outperforms all the baselines in this task set, as shown in Figure 8. Task 3 receives only a sparse reward and, thus, can benefit the most from shared exploration. We observe that QMP gains the most performance boost due to selective behavior-shand sharrin in Task The No-Shared-Behaar Behavior baseline is unable to solve Task 3 at all due to its sparse reward nature. The other baselines which share uniformly suffer at Task 3, likely because they also share behaviors from other conflicting tasks, especially Task 4. We explore this further in the following Section F.
For all tasks, QMP outperforms or is comparable to No-Shared Shared-Behavior, which shows that selective behavior-sharing can help accelerate learning when task behaviors are shareable and is robust when tasks conflict. Fully-Shared Shared-Behavior especially underperforms in Tasks 2 and 3, which require conflicting behaviors upon reaching Subgoal 1, as defined in Table 1. In contrast, it excels at the beginning of Task 0 and Task I as their required behaviors are completely shared. However, it suffers after Subgoal 2, as the task objectives diverge.
21

---

Published as a conference paper at ICLR 2025
TaskO
Task1
Task2
Success Rate (96)
Success Rate (%)
Success Rate (%)
82-
20-
22-
0.5
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
Task3
Task4
1.0
Success Rate (%)
60
60-
20
20
0.0
2.
1.5
2.0
5.0
05
3.0
1.5
2.0
Enviranment steps (1M)
Environment steps (1M)
QMP(Ours)
No-Shared-Behavie
Fully-Shared-BB Behavior
Se--Shrhrd-BBeavioor
DnC
DnC (reg)
UDS
Figure 13: Success rates for individual tasks Multistage Reacher. Our method especially helps in learning Task 3, which requires extra exploration because it only receives sparse reward.
Maze Navigation
Tasks
Maze Navigation
Muitistage Reacher
130
1.00
Success Rate (96)
Success Rate (%)
Success Rate (%)
ONP
42-
Shared
AnC
20
DnC
20
12
0.0
0.5
1.0
1.5
2.0
Environment steps (1M)
Enviranment steps (1M)
Environment steps (1M)
(a) Maze Navigation Tasks
(b) Maze Navigation 10 Tasks
(c) Data Sharing Comparison
Figure 14: QMP scales well from (a) 3 tasks to (b) 10 tasks in Maze Navigation, especially in comparison to other behavior sharing methods. (c) Online data sharing is very efficient when given task reward functions (all CDS versions), but suffers without (all UDS versions).
E.2 QMP SCALES WITH TASK SET SIZE IN MAZE NAVIGATION
We look at the behavior sharing methods in the Maze Navigation task for a task set with 3 tasks (Figure 14a) and 10 tasks (Figure 14b) and see that QMP scales well from 3 to 10 tasks, compared to other behavior sharing methods. Similar to Meta- Meta-World, we hypothesize QMP scales better with larger task set size similar tasks due there being more shareable behaviors between tasks. We see that by selectively sharing behaviors, QMP is able to identify and share helpful behaviors in the larger tasks sets whereas other behavior sharing methods struggle.
E.3 QMP OUTPERFORMS DATA SHARING WITH REWARD LABELING
In Figure 14c, we report multiple sharing percentiles for UDS and for CDS (Yu al., 2021) which assumes access to ground truth task reward functions which it uses to re-label the shared data. When the shared data is relabeled with task reward functions, thereby bypassing the conflicting behavior problem, online data sharing approaches can work very well. But when unsupervised, we see that online data sharing actually harm performance in environments with conflicting tasks, with
22

---

Published as a conference paper at ICLR 2025
the more conservative data sharing approach (UDS k=80) out-perforning sharing all data. & is the percentile above with we share a transition between tasks, with higher k representing more conservative data sharing. Full details online UDS and CDS implementation are in Section H.6
E.4 PCGRAD RESULTS
We evaluate whether QMP combined with PCGrad (Yu al., 2020) results in complementary benefits. PCGrad is a popular MTRL algorithm that learns a policy with shared parameters and alleviates negative interference between tasks by modifying the multi-task gradients. In Figure 15 and Table 2, we see that QMP + PCGrad significantly improves PCGrad performance out of 4 environments.
Multistage Reacher
Mare Navigation
Kitchen
NetaWorld
100
PCDre
Succees hare (S)
Success Rate (86)
Success (h6)
(MP 200rd
10 Rate (60
PCCre
Suee
33
12
23
24
175
2.6
764
056
Li
25 20 75 120
Enviorment steps (1M)
Environment stops
Environment steps (2M)
Environment stops
Figure 15: Combining QMP with PCGrad yields complementary improvement in 3 out of the 4 environments we tested on. Dashed lines are PCGrad only and solid lines are QMP + PCGrad.
Approach PCGrad QMP + PCGrad
Reacher
Maze
Kitchen
50
0.78
0.90
0.55
0.35
0.78
1.00
0.60
0.42
Table QMP improves performance of PCGrad across various benchmarks
E.5 ADDITIONAL MTRL COMPARISONS
Multistage Reacher
Multistage
Meta-World CDS
100
Mulb-Ciiicc
102
BEGN GMP
100
Success Rate (%)
QUPP MUti ALMM
Success Rate (96)
Success Rate (9%)
QVP
Mu Cili--CficCAL
QUt Mulli Or Uc ALMH
60
60-
20
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1/0
1's
2.5
0.0
2/4
5.n 7/s 10.0
Environment steps (1M)
Environment steps (1MM
Environment steps (1M)
(a) MCAL Comparison
(b) DECAF Sharing Comparison
(c) DECAF Sharing Comparison
Figure 16: Combining our method with another parameter sharing method, MCAL,, shows comple- mentary benefits in (a). Our method outperforms DECAF in Multistage Reacher (b) and Meta- Meta-World World CDS(c), demonstrating that learning to directly use Q-functions from other tasks is more challenging and sample inefficient than using the current task's Q-function to evaluate other tasks' policies.
Multi-Critic Actor Learning (MCAL) (Mysore al., 2022) is a parameter sharing MTRL method that aims to tackle potential negative interference between tasks by learning separate critics for each task while training single multi-task actor. We add QMP to two variants of MCAL,, Multi-Critic AL-MN which maintains separate networks for each critic and Multi-Critic AL-MH which uses single multi-head network for the critic, in Multistage Reacher in Figure 16a. We see that adding QMP provides around 20% final success rate gain in both variants and is more sample efficient.
We also compare our method with DECAF (Glatt et al., 2020), a MTRL method which shared O-functions between tasks instead of behavioral policies. DECAF learns task specific weights to linearly combine the task fuunctis which used to train the task policy. In contrast, our method uses the task Q-function evaluate different tasks' policies to incorporate into the task's behavioral
23

---

Published as a conference paper at ICLR 2025
Multistage Reacher
Maze Navigation
100
100
QMP (H-25)
Success Rate (%)
Success Rate (%)
(H-1)
80-
80
No. QQM
60-
60
40
40
QMP (H-10)
QMP (H=1)
20
No-QMP
0.0
0.5
1.0
1.5
2.0
12
Environment steps (1M)
Environment steps (1M)
Meta-World CDS
Kitchen
100
100
QMP (H-10)
QMP (H-110)
Success Rate (%)
QMP (H=1)
Success Rate (%)
QMP (H=1)
80
No-QMP
80-
No. NM-QMP
60
60
40
20
0-1
2.5
5.0
0.00
1.25
3.75
5.00
Environment steps (1M)
Environment steps (1M)
Figure 17: In each case above, QMP with H-step rollouts of the behavioral policy (blue) performs no worse than QMP with 1-step rollouts (red), with the H-step rollouts helping significantly in some tasks. Additionally both versions of QMP outperform the No-QMP baseline.
policy. Our method only modifies the data collection process, not the RL objective, and does not have a learned component. In Multistage Reacher (Figure 16b) and Meta- World CDS (Figure 16c), we see that QMP outperforms DECAF by more that 20% final success rate.
E.6 TEMPORALLY-EXTENEE BEHAVIOR SHARING
Motivated by prior work in heirarchical RL (Machado
Policy Sampling Frequency in Maze
et al., 2017; Jinnai al., 2019b;a; Hansen et al., 2019;
Zhang et al., 2020) and skill learning (Pertsch et al., 2021)
100
Success Rate (%)
we explore temporally extended behavior sharing by sim- ply following the actions of the policy тΙ selected by whii for steps before re-evaluating Timer- Furthermore, a re- cent work Xu et al. (2024) provides theoretical results that
(H-1)
shows myopic (c-greedy) policy sharing can be sample efficient in sufficiently diverse multi-task settings, provid- ing theoretical support for temporally extended multi-task behavior sharing in some settings.
40
QMP (H=5)
(H-10)
QMP (H=25)
(H-50)
0
3
12
Environment steps (1M)
We study the effect of sharing temporally extended be- haviors of length H in Maze Navigation in Figure 18, by rolling out the chosen task policy for 1, 5. 10. 25, and 50 timesteps. We see that performance improves when sharing longer behaviors (25 and 50 timesteps) which are more coherent and temporally extended. This is true even though we choose the behavioral policy greedily, only evaluating the current state s every // steps. Importantly, the guarantees from Theorem C. do not extend to H/steep policy roll-outs and increasing H does not help in all enviromments. We compare the performance of No-QMP QMP, and QMP with temporally extended behavior sharing where choose the best performance out of H = 10 and H = 25 in Table 3 and Figure 17. Nevertheless, the impressive results in Maze suggest that multi-task temporally extended behavior sharing is worth exploring in future work.
Figure 18: QMP consistently improves
performance I increases in Maze.
24

---

Published as a conference paper at ICLR 2025
Table 3: Temporally Extended Behavior Sharing
Environment
H-value
No-QMP
QMP
QMP (H>
Reacher
10
100
100 = 0
Maze
25
57.9 + 0.09
7.1
99.9 = 0.0
MT-CDS
10
9.5
93.7
98.8 2.0
MT10
10
79.1
8.00
82. + 4.48
Kitchen
10
65.5 11.0
7.3
84.5
Walker
10
3110
220
3205 218
3310 203
F QMP BEHAVIOR SHARING ANALYSIS
Policy Percentage Mixture
Policy Percentage in Mixture Policy Percentage in Mixture
Policy Percentage Mixture
30-
24
24
t0
12
0.5
1.5
2.0
0.0
05
05
10
1.5
2.0
Environment steps (1
Environment steps (1M)
Environment steps (1M)
Policy Percentage Mixture
30
Policies Policy
12
0.5
03
1.5 2.0
0.0
1.5 Environment steps (1M)
0.5
1.0
2.0
Environment steps (1M)
Figure 19: Mixture probabilities per task of other policies over the course of training for Multistage
Reacher. The conflicting task Policy 4, which requires staying stationary, highlighted in red.
QMP learns to not share from conflicting tasks: We visualize the mixture probabilities task of other policies in Figure 19 for Multi- stage Reacher, highlighting the conflicting Task 4 in red. Throughout training,
Average Mixture Composition
3.05
that QMP learns to share less behavior from Policy 4 than other policies in Tasks 0-3 and shares the least to- tal cross-task behavior Task This supports our claim that the Q-switch can identify con- flicting behaviors that should not be shared. We note that Task has relatively larger amount of sharing than other tasks. Since Task 3 has sparse rewards, benefits the most from exploration via selective behavior-sharing from other tasks.
0.04
Task
0.03
3.02
3.00
3.00
Policy
Figure 20 analyzes the effectiveness of the Q- switch in identifying shareable behaviors by visualizing the average proportion that each task policy is selected for another task over the course of training in the Multistage Reacher en- vironment. This average mixture composition statistic intuitively analyzes whether QMP identifies sharcable behaviors between similar tasks and avoids behavior sharing between conflicting or inrel- evant tasks. As expect, the Q-switch for Task 4 utilizes the least behavior from other policies
Figure 20: Average contribution of each Policy (col j) in each Task i's (row i) data collection on Reacher Multistage (diagonal zeroed for contrast).
25

---

Published as a conference paper at ICLR 2025
Multistage Reacher
Multistage Reacher
Maze Navigation with SAC Exploratio
Success Rate (%)
Success Rate (%)
Success Rate (%)
80-
60-
82-
QMP (i-10)
6.-
QYP
62-
QMP-Dgpectaion (11-10)
QMP 1i-11
40-
OFP
QVP Sotumas T-0.01
QMP-Dppetten (Hi63]
SAC tarpet Ert.
QyP Softmas Tw1
Parameters Behaviors
S4C larpet Ert.
омP Sohtmaa 1-100
Parameters Ope-tapectiioon
SAC Thrpet Ere +]
0.00
5.99
0:38
0.56
6.7
6.0
1.
Environment steps (1M)
Environment steps (1M)
Environment steps (1M)
(a) Probabilistic Mixture Ablation (b) Expected Q-value Approxima-
(c) Single-Task Exploration
tions
Figure 21: (a) Using probabilistic mixtures with QMP by using a softmax over Q values with temperature T, which determines the spread of the distribution. (b)Across different QMP versions, evaluating mean policy actions (solid lines) vs. sampling 10 actions to estimate expected Q-values (dashed lines) result in similar performance. (c) Single-task exploration by varying SAC target entropy. QMP reaches a higher success rate because it shares exploratory behavior across tasks.
(bottom row), and Policy 4 shares the least with other tasks (rightmost column). Since the agent at Task 4 is rewarded to stay at its initial position, this behavior conflicts with all the other geal-neaching tasks, Of the remaining tasks, Task 0 and share the most similar goal sequence, so it is intuitive why they benefit from shared exploration and are often selected by their respective Q-switches. Finally, unlike the other tasks. Task 3 receives only sparse reward and therefore relies heavily on shared exploration. In fact, QMP demonstrates the greatest advantage in this task (Appendix Figure 13).
Behavior-sharing reduces over training: Figure 19 shows that the total amount of behavior-sharing decreases over the course of training all tasks, which demonstrates a naturally arising preference in the Q-switch for the task-specific policy as it becomes more proficient at its own task.
F.1 QUALITATIVE VISUALIZATION BEHAVIOR-SHARMI
We qualitatively analyze behavior sharing by visualizing rollout of QMP during training for the Drawer Open task in Meta-World World Manipulation (Figure 9b). To generate this visualization, we use QMP rollout during training before the policy converges to see how behaviors are shared and aid learning. For clarity, we first subsample the episodes timesteps by 10 and only report timesteps when the activated policy changes to a new one (ie. from timestep 80 to 110, QMP chose the Drawer Open policy). We qualitatively break down the episode into when the agent is approaching the drawer (top row; Steps 1-60), grasping the handle (top row; Steps 61-80), and pulling the drawer open (bottom row). This allows us drawer-specific policies as it grasps the handle, and opening-specific policies as it pulls the drawer open. This suggests that in addition ignoring conflicting behaviors, QMP is able to identify helpful behaviors to share, We note that QMP is not perfect at policy selection throughout the entire rollout, and it is also hard to interpret these shared behaviors exactly because the policies themselves are only partially trained, as this rollout is from the middle of training. However, in conjunction with the overall results and analysis, this supports our claim that QMP can effectively identify shareable behaviors between tasks.
that it switches between all task policies it approaches the drawer, uses
G ADDITIONAL ABLATIONS AND ANALYSIS
G.1 PROBABILISTIC MIXTURE v/s ARG-MAX
A probabilistic mixture of policies is design choice of our approach where arg-max is replaced with softmax. However, in our initial experiments, we found no significant improvement in performance and it came with additional hyperparameter of tuning the temperature coefficient. As see in Figure 21a, QMP actually outperforms a probabilistic mixture over a range of softmax temperatures, justifying the design choice of argmax over softmax due to its reliable performance and simplicity.
26

---

Published as a conference paper at ICLR 2025
G.2 APPROXIMATION EXPECTED Q- Q-VALUE OVER POLICY ACTION DISTRIBUTION
QMP's behavior policy is defined as n!
arg max
(s) a), which picks the task
zy E(FI..,,NT)
policy with the best expected Q value over its action distribution. We approximate the expectation by evaluating the Q-value of only the mean of each policy's 's action distribution which is computationally cheaper 2 2
arg max Qi(*, (s) We compare this to a empirical estimate that ()((F)..,
samples 10 actions from the policy distribution and picks the policy with highest average Q-value in Figure 21b, and find no significant performance difference between the two approximations. This validates that simple approximation works well in practice, which hypothesize is due to the low variance of the task policies.
G.3 QMP v/s INCREASING SINGLE TASK EXPLORATION
Since QMP seeks gather policy, it can be viewed as a form of multi-task exploration. We briefly investigate how single task ex- ploration differs from multi-task exploration by tuning the target entropy in SAC in Figure 21c which influences the policy entropy and therefore exploration. We see that while tuning this exploration parameter affects the sample efficiency by more quickly learning each individual task, QMP achieves higher final success rate by incorporating behaviors form other tasks, and therefore doing multi-task exploration. The benefit exploration or behavior sharing algorithms specialized for multi-task RL precisely this ability transfer and share information between tasks.
informative training data for the task by modifying the behavioral
G.4
QMP RUNTIME
While QMP does require more policy and q-function evaluations to sample from T T in comparison the base RL method, these evaluations be greatly parallelized and do not significantly increase runtime (see Table 4) for average runtimes for our experiments). Each sample from "i requires querying N policy proposals and N Q-values. In QMP + Parameter-Sharing thanks to the multihead architectures of the policy and Q-networks, all N evaluations are done in one single pass. Thus, with two passes through neural networks, we can get N action candidates and their Q-values, Therefore, the increase in time is negligible. Even without parameter- sharing,, Qi az) evaluations can be batched jj and the policy evaluations π (ayale In our implementation, we batch the Q evaluations, but do not parallelize the policy evaluations.
[s)]s)) are all independent, and can be obtained in parallel.
Table 4: Runtime Comparison
No-Sharing
QMP + No-Sharing
Craameter-Shrrnn
QMP Parameter-Sharing
Environment
Reacher Multistage MT50
12.5 hr
14.2 hr
14 hr 7 days, 3hr
16.2 hr days, hr
H IMPLEMENTATION DETAILS
The SAC implementation we used in all our experiments is based on the open-source implementation from Garage (garage contributors, 2019). We used fully connected layers for the policies and Q- functions with the default hyperparameters listed in Table 5. For DnC basclines, reproduced the method in Garage to the best of our ability with minimal modifications.
We used PyTorch (Paszke et al., 2019) for our implementation. We run the experiments primarily on machines with either NVIDIA GeForce RTX 2080 Ti or RTX 3090. Most experiments take around one day less logging and tracking experiments. All the environments were developed using the OpenAl Gym interface (Brockman et al., 2016).
RTX 3090 to run. We use the Weights & Biases tool (Biewald, 2020) for
27

---

Published as a conference paper at ICLR 2025
H.1 HYPERPARAMETERS
Table 5 details the list of important hyperparameters on all the 3 environments. For all environments, we used layer fully connected network with hidden dimension 256 and tanh activation function for the policies and Q functions. We use a target network for the Q function with target update 0.995 and trained with an RL discount of y 0.99.
Table 5: Hyperparameters.
Multistage Reacher
Maze Navigation
Meta-
Meta-World World
Hyperparameter
CDS
Minimum buffer size (per task) Enviromment steps per update (per task)
10000
3000
10000
1000
600
500
# Gradient steps per update (per task)
100
100
50
Batch size
32
256
256
Learning rates for т, Q and a
0.0003
0.0003
0.0015
Meta-World MT10
Walker
Kitchen
Hyperparameter
Minimum buffer size (per task)
500
2500
200
Environment steps per update (per task)
500
1000
200
# Gradient steps per update (per task)
50
1500
50
Batch size
2560
256
1280
Learning rates for ग, and
0.0015
0.0003
0.0003
H.2 NO-SHARED- -EAARED-BEEAVOORS
All N networks have the same architecture with the hyperparameters presented in Table 5.
H.3 FULLY-SHARED- BEAIDD-BEAIOOSSS
Since it is the only model with single policy, we increased the number parameters in the network to match others and tuned the learning rate. The hidden dimension of each layer is 600 in Multistage Reacher, 834 in Maze Navigation, and 512 in Meta- Meta-World Manipulation, and we kept the number of layers at 2. The number of environment steps as well as the number of gradient steps per update were increased by N times so that the total number steps could match those in other models. For the learning rate, we tried different values (0.0003, 0.0005, 0.001, 0.0015) and chose the most performant one. The actual learning rate used for each experiment is 0.0003 in Multistage Reacher and Maze Navigation, and 0.001 in Meta- World Manipulation.
This modification also applics to the Shared Multihead baseline, but with separate tuning for the network size and learning rates. In Multistage Reacher, used layers with hidden dimensions of 512 and 0.001 as the final learning rate. In Maze Navigation, we used 834 for hidden dimensions and 0.0003 for the learning rate.
H.4 DNC
We used the same hy perparraaeters as in Separated, while the policy distillation parameters and the regularization coefficients were manually tuned. Following the settings in the original DnC (Ghosh et al., 2018), we adjusted the period of policy distillation to have 10 distillations over the course of training. The number of distillation epochs was set to 500 to ensure that the distillation is completed. The regularization coefficients were searched among values (0.0001, 0.001, 0.01, 0.1, 1), and we chose the best one. Note that this search was done separately for DnC and DnC with regularization only. For DnC, the coefficients and 0.001 in Meta- World Manipulation. For Dnc with regularization only, the values are: 0.001 in Multistage Reacher, 0.0001 in Maze Navigation, and 0.001 in World Manipulation.
used are: 0.001 in Multistage Reacher and Maze Navigation,
28

---

Published as a conference paper at ICLR 2025
H.5 QMP (OURS)
Our method also uses the default hyperparameters. QMP does not require any task specific hyperpa- rameters. The exception is Meta-World MT10, where found helpful to have more conservative behavior sharing by choosing the task-specific policy 70% of the time. The remaining 30% we use the Q-filter to select policy as usual.
Like in Baseline Multihead (Parameters-Only). the QMP Multihead architecture (Parame- lers+Behaviors) also required a separate tuning. Since QMP Multihead effectively has one network, we increased the network size in accordance with Baseline Multihead and tuned the learning rate in addition to the mixture warmup period. The rest-performing combinations of these parameters we found are O and 0.001 in Multistage Reacher, and 100 and 0.0003 in Maze Navigation, respectively.
H.6 ONLINE UDS
Yu et al. (2022) proposes an offline multi-task RL method (UDS) that shares data between tasks their conservative Q value falls above the 1*th percentile of the task data. Specifically, before training, you would go through all the tasks' data and share some data from Task j to Task the Task Q value of that data is greater than A% of the values of Task i's data. UDS does not require access to task reward functions like other data-sharing approaches. It simply re-labels any shared data with the minimum task reward, making it applicable our problem setting as we also do not assume that reward relabeling is possible.
In order adapt UDS to online RL, instead of doing data sharing once on the given multi task dataset, we apply UDS data sharing before every training iteration to the data in the multi-task replay buffers. Concretely, we implement this on-the-fly for every batch of sampled data by sampling one batch of data from Task i's replay buffer, B,, and one batch of data from the other task's ss replay buffers 95yt- Then following UDS, we would form the effective batch Seet by sharing data from Bzze if it falls above the hh percentile of Q values for Br:
U UDDoniline (s, Ti, s') 1)- Bizi 8: 8 B.
if AT
(x, - Pa (
Note the differences here: (i) the 'data' used for data-sharing is the sampled replay buffer batch instead of the offline dataset, and (ii) we use the standard Q-function to evaluate data instead of the conservative Q-function since we are doing online (not offline) RL. We implement it this way as practical approximation to avoid having to process the entire replay buffer every training iteration.
We the same default hyperparameters as the other baseline methods. Additionally, we need tune the sharing percentile k. For this, we tried Oth percentile (sharing all data) and 80th percentile, and chose the best-performing one.
29


Reviews:
[{"review_id": "4Q8PAkLPAu", "reviewer": "Reviewer_MdZX", "summary": "This paper introduces Q-switch Mixture of Policies (QMP), a novel framework for multi-task reinforcement learning that improves sample efficiency through selective behavior sharing between tasks. The key idea is to enhance each task's off-policy data collection by selectively employing behaviors from other task policies, using the task's Q-function to evaluate and select useful shareable behaviors.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written with clear organization and easy to follow.\n\n- The method is elegantly simple yet theoretically sound, introducing behavior sharing through off-policy data collection rather than policy regularization, which avoids bias in the learning objective while maintaining convergence guarantees.\n\n- The proposed Q-switch mechanism provides a principled approach to selective behavior sharing that is complementary to existing MTRL methods, demonstrating consistent improvements when combined with parameter sharing, data sharing, and gradient-based approaches.", "weaknesses": "1. The Q-switch mechanism requires evaluating all task policies and Q-values at each step. While parallelization helps, this still leads to significant computational costs, especially for large task sets as evidenced by the 7+ days runtime in MT50 experiments.\n2. The method heavily relies on accurate Q-function estimation to select appropriate behaviors. This dependency may lead to suboptimal behavior selection during early training or in tasks with sparse rewards where Q-function learning is unstable.\n\n3. The method requires careful tuning of hyperparameters, such as manually setting 70% task-specific policy usage in Meta-World MT10. This tuning requirement may limit practical applicability.\n\n4. The method's effectiveness heavily depends on the existence of shareable behaviors between tasks. Performance gains might be limited for completely unrelated or highly conflicting task sets.\n\n5. The convergence analysis primarily focuses on tabular MDPs. The theoretical guarantees may not fully extend to continuous state-action spaces with function approximation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUICmqRvEK", "reviewer": "Reviewer_ENcC", "summary": "This paper introduces the Q-switch Mixture of Policies (QMP), a framework for multi-task reinforcement learning (MTRL) aimed at enhancing sample efficiency by sharing behaviors selectively across tasks. The approach enables an agent to leverage useful behaviors from other tasks during off-policy data collection, guided by the Q-function of the current task to identify beneficial actions. Unlike traditional MTRL methods that rely on uniform sharing or regularization, QMP avoids biasing policy objectives by only sharing behaviors during data collection rather than directly influencing policy updates. Experimental results across various environments, including manipulation and locomotion tasks, demonstrate QMP's capability to accelerate training and improve performance when integrated with existing MTRL frameworks.", "questions": "1. Could you provide some information regarding the computational overhead of QMP?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. QMP is designed to complement existing MTRL frameworks, such as parameter sharing and data relabeling. \n2. The authors provide theoretical analysis showing that QMP’s behavior-sharing mechanism preserves the convergence guarantees of the underlying reinforcement learning algorithm.\n3. The paper presents extensive experiments across various multi-task environments (e.g., manipulation, navigation, and locomotion).", "weaknesses": "1. QMP requires evaluating Q-values across multiple task policies at each decision step, which could introduce computational overhead, particularly in settings with a large number of tasks. The paper lacks a detailed comparison of computational costs between QMP and baseline methods, which would be helpful for assessing its scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YDINMUlhCJ", "reviewer": "Reviewer_7Cb7", "summary": "The paper introduces QMP (Q-Switch Mixture of Policies), a novel framework for multi-task reinforcement learning (MTRL) designed to improve sample efficiency by selectively sharing behavioral policies across different tasks. Unlike traditional MTRL methods, which rely on parameter sharing or uniform behavior sharing, QMP identifies and shares beneficial behaviors from other tasks based on a Q-function evaluation. This approach enables each task to benefit from the learning progress in other tasks without introducing bias, as only helpful behaviors are integrated into the data collection phase.", "questions": "Please see previous section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The QMP framework introduces a Q-value-based behavior selection mechanism that enables selective behavior sharing in multi-task reinforcement learning (MTRL), enhancing sample efficiency.\n2. The extensive experimental results across diverse domains—manipulation, navigation, and locomotion—demonstrate QMP’s performance compared to traditional behavior-sharing baselines, showcasing its practical impact in different tasks.", "weaknesses": "1. This paper only proposes a multi-task data sampling strategy implemented within an off-policy framework, which is limited in scope and shows limited improvement in more complex tasks, such as MT10 and MT50. \n2. The other algorithms in paper [1] achieved better results with fewer interactions with the environment during training. I think that the method proposed in this paper makes a limited contribution to the field of multi-task reinforcement learning.\n3. It is unclear how to ensure avoidance of local optima during complex multi-task transfer processes.\n4. There is a lack of comparison with other MTRL baselines.\n\n[1] Sodhani et al., 2021, Multi-Task Reinforcement Learning with Context-based Representations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xX1MZuhT87", "reviewer": "Reviewer_CusU", "summary": "This paper proposes a framework for multi-task reinforcement learning (MTRL) called Q-switch Mixture of Policies (QMP), which enables the selective sharing of behaviors between tasks to improve sample efficiency. QMP enhances off-policy data collection by selecting useful behaviors from other task policies based on the task's Q-function. The authors provide theoretical guarantees that QMP improves sample efficiency while preserving the convergence guarantees of reinforcement learning algorithms. Empirical evaluations demonstrate that QMP achieves complementary gains over existing MTRL algorithms in manipulation, locomotion, and navigation environments.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written with a logical flow. The introduction and problem formulation effectively motivate the need for a new behavior-sharing method, and the diagrams provided help visualize the approach. The results are promising.", "weaknesses": "My impression of this paper is mixed. On the one hand, the authors demonstrate that QMP is effective. However, on the other hand, it's not intuitively clear why QMP works as well as it does. The proposed method is quite simple: it uses the current $Q_i$ function to select the argmax policy $\\pi_j^{mix}$. Then, $\\pi_j^{mix}$ (rather than $\\pi_i$) is used to generate one-step data for training. I have some concerns about the approach of naively taking the maximum over per-task Q-functions—this may not be suitable in general settings, such as more stochastic environments, as it could lead to issues like switching tasks at every step or favoring certain tasks while neglecting others, which is undesirable. If this paper is accepted, it might be perceived as presenting a general framework for MTRL, but I have serious doubts about this claim.\n\nIn the introduction section of the paper, it mentions 'These tasks share many similar behaviors, like approaching the tabletop or grasping the object handle.' Therefore, I think 'behavior' refers to an action sequence or subtask that continues for a period of time. However, Algorithm 1 (one-step action sharing) appears to be misaligned with the paper's title and motivation, which emphasizes behavior (action sequence) sharing. The paper's motivation requires significant revision to address this discrepancy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)