QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing

## Abstract

Abstract Multi-task reinforcement learning (MTRL) aims to learn several tasks simultaneously for better sample efficiency than learning them separately. Traditional methods achieve this by sharing parameters or relabeled data between tasks.
In this work, we introduce a new framework for sharing behavioral policies across tasks, which can be used in addition to existing MTRL methods. The key idea is to improve each task’s off-policy data collection by employing behaviors from other task policies. Selectively sharing helpful behaviors acquired in one task to collect training data for another task can lead to higher-quality trajectories, leading to more sample-efficient MTRL.
Thus, we introduce a simple and principled framework called Q-switch mixture of policies (QMP) that selectively shares behavior between different task policies by using the task’s Q-function to evaluate and select useful shareable behaviors.
We theoretically analyze how QMP improves the sample efficiency of the underlying RL algorithm.
Our experiments show that QMP’s behavioral policy sharing provides complementary gains over many popular MTRL algorithms and outperforms alternative ways to share behaviors in various manipulation, locomotion, and navigation environments.
Videos are available at https://qmp-mtrl.github.io/ .

### 1 Introduction

Figure: Figure 1: We propose a sample-efficient MTRL framework that selectively shares behaviors by acting with other task policies for data collection. For example, Drawer Open and Drawer Close can share behaviors for grasping the drawer handle, while Drawer Open and Door Close share behaviors for approaching the tabletop.
Refer to caption: x1.png

In multi-task reinforcement learning, each task has something to learn from the others.
Consider a task set where a robot is simultaneously learning to open and close a drawer and a door on a tabletop, as illustrated in Figure 1. These tasks share many similar behaviors, like approaching the tabletop or grasping the object handle. While learning to open a drawer, the robot can benefit by exploring behaviors found rewarding in other related tasks (such as grasping the drawer or door handle), then incorporating the helpful behaviors into its own policy, instead of randomly exploring the entire action space.
Can we craft a framework to learn from similar behaviors across tasks to accelerate overall learning?

Most multi-task reinforcement learning (MTRL) methods share task information via policy parameters (Vithayathil Varghese & Mahmoud, ) or data relabeling (Kaelbling, ). We propose a new framework for MTRL: *share behaviors* between tasks to improve data collection by employing potentially useful policies from other tasks for more informative training data. This approach offers a simple, general, and sample-efficient approach that complements existing MTRL methods.

Prior works (Teh et al., ; Ghosh et al., ) share behaviors between task policies uniformly by regularizing to one shared distilled
policy (Rusu et al., ).
This introduces a bias towards the mean behavior and causes negative interference when tasks might require differing optimal behaviors from the same state. In contrast, reusing other policies for data collection does not introduce any bias.

We propose selective behavioral policy sharing as a novel and general mechanism to improve sample efficiency in any MTRL architecture.
Our key insight is that behaviors being acquired in other tasks can help when appropriately selected and shared, as shown in human learners (Tomov et al., ). In the Drawer Open task, while learning to approach the drawer handle, the robot should share behaviors between the Drawer policies, but avoid Door policies which would lead it to the wrong object.

The key question with selective behavioral policy sharing is how to identify helpful behaviors from other policies in a principled way.
We propose a principled way of selecting shared behaviors: a Q-switch Mixture of Policies (QMP). At each state, one policy from a mixture of all policies is selected to collect data. The Q-switch makes this selection based on which policy best optimizes the current task’s soft Q-value because that is an estimate of the most helpful behavior for the current task. We prove that this selection mechanism preserves the convergence guarantees of the underlying RL algorithm and potentially improves sample efficiency.
Crucially, QMP uses other tasks’ policies only for data collection, allowing policy training to remain unbiased under any off-policy RL algorithm.

Our primary contribution is introducing behavioral policy sharing for MTRL as a novel avenue of information sharing between tasks and addressing the problem of principled selective behavior sharing. Our proposed framework, Q-switch Mixture of Policies (QMP), can effectively identify shareable behaviors between tasks and incorporates them to gather more informative training data. We prove that QMP’s behavior sharing not only preserves the policy convergence of the underlying RL algorithm, but is at least as sample efficient. We demonstrate that QMP provides complementary gains to other forms of MTRL in a range of manipulation, locomotion, and navigation tasks and performs well over diverse task families when compared to other behavior sharing methods.

### 2 Related Work

Information Sharing in Multi-Task RL.
There are multiple, mostly complementary ways to share information in MTRL, including sharing *data*, sharing *parameters* or *representations*, and sharing *behaviors*.
In offline MTRL, prior works selectively share *data* between tasks (Yu et al., ; ).
Sharing parameters across policies can speed up MTRL through shared representations (Xu et al., ; D’Eramo et al., ; Yang et al., ; Sodhani et al., ; Misra et al., ; Perez et al., ; Devin et al., ; Vuorio et al., ; Rosenbaum et al., ; Yu et al., ; Cheng et al., ; Hong et al., ) and can be easily combined with other types of information sharing. Most similar to our work, Teh et al. () and Ghosh et al. () share behaviors between multiple policies through policy distillation and regularization.
Vuong et al. () identify which states between tasks share optimal behavior and regularize to each other there.
These works share behaviors through regularization, biasing the policy objective when tasks have differing optimal behaviors. In contrast, our work selectively shares behavioral policies without modifying the training objective.

Multi-Task Learning for Diverse Task Families.
Multi-task learning in diverse task families is susceptible to negative transfer between dissimilar tasks, hindering training. Prior works combat this by measuring task relatedness through validation loss on tasks (Liu et al., ; Ackermann et al., ) or influence of one task to another (Fifty et al., ; Standley et al., ) to find task groupings for training. Other works focus on the challenge of multi-objective optimization (Sener & Koltun, ; Hessel et al., ; Yu et al., ; Liu et al., ; Schaul et al., ; Chen et al., ; Kurin et al., ).
Similar to these works, we identify that prior behavior-sharing MTRL approaches are susceptible to negative transfer. However, we avoid the challenge of negative transfer entirely by selectively sharing behaviors only during off-policy data collection.

Exploration in Multi-Task Reinforcement Learning.
Our approach of modifying the behavioral policy to leverage shared task structures can be seen as a form of MTRL exploration, which we discuss further in Appendix Section 16(c).
Bangaru et al. () encourage agents to
increase their state coverage by providing an exploration bonus.
Zhang & Wang () study sharing information between
agents to encourage exploration under tabular MDPs.
Kalashnikov et al. () directly leverage data from policies of other specialized tasks (like grasping a ball) for their general task variant (like grasping an object).
In contrast to these approaches, we do not require a pre-defined task similarity measure or exploration bonus;
we demonstrate in Section 6 that QMP works across many tasks and domains without these additional measures.
Skill learning can be seen as behavior sharing in a single task setting such as learning options for exploration or heirarchical RL (Machado et al., ; Jinnai et al., ; ; Hansen et al., ; Riemer et al., ). We also discuss the difference to single-task exploration in Appendix Section  G.3.

Using Q-functions as filters.
Yu et al. () uses Q-functions to filter which data should be shared between tasks in a multi-task setting. In the imitation learning setting, Nair et al. () and Sasaki & Yamashina () use Q-functions to filter out low-quality demonstrations, so they are not used for training. In both cases, the Q-function is used to evaluate some data that can be used for training. Zhang et al. () reuses pre-trained policies to learn a new task, using a Q-function as a filter to choose which pre-trained policies to regularize to as guidance. In contrast to prior works, our method uses a Q-function to *evaluate* different task policies to gather training data.

### 3 Problem Formulation

Multi-task reinforcement learning (MTRL) addresses sequential decision-making tasks, where an agent learns a policy to act optimally in an environment (Kaelbling et al., ; Wilson et al., ). Therefore, in addition to typical multi-task learning techniques, MTRL can also share behaviors, i.e., actions, to improve sample efficiency.
However, current approaches share behaviors uniformly (Section 2), which assumes that different tasks’ behaviors do not conflict.
To address this limitation, we seek to develop a selective behavior-sharing method that can be applied in more general task families for sample-efficient MTRL.

Multi-Task RL with Behavior Sharing.
We aim to simultaneously learn a set $\{\mathbb{T}_{1},\dots,\mathbb{T}_{N}\}$ of $N$ tasks. Each task $\mathbb{T}_{i}$ is a Markov Decision Process
(MDP) defined by
state space $\mathcal{S}$, action space $\mathcal{A}$, transition probabilities $\mathcal{T}_{i}$, reward functions $\mathcal{R}_{i}$, initial state distribution $\rho_{i}$, and discount factor $\gamma\in[0,1]$.
While we use $\mathcal{S}$ to denote shared state spaces for simplicity, our formulation extends to tasks with different state spaces as it complements policy architectures that share state encoders.
The agent learns a set of $N$ policies $\{\pi_{1},\dots,\pi_{N}\}$, where each policy $\pi_{i}(a|s)$ represents the behavior on task $\mathbb{T}_{i}$. The objective is to maximize the average expected return over all tasks,

$$ $\{\pi^{*}_{1},\dots,\pi^{*}_{N}\}=\max_{\{\pi_{1},\dots,\pi_{N}\}}\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{a_{t}\sim\pi_{i}}\sum_{t=0}^{\infty}\gamma^{t} \mathcal{R}_{i}(s_{t},a_{t})\right].$ $$

Unlike prior works, our tasks can exhibit conflicting optimal behaviors:
for any $s$, $\pi^{*}_{i}(a|s)$ may differ from $\pi^{*}_{j}(a|s)$. Thus, prior methods that bias policy learning objectives like direct policy sharing (Kalashnikov et al., ) or behavior regularization (Teh et al., ) would be suboptimal.

### 4 Approach

To improve the sample efficiency of multi-task RL, we propose a framework that *selectively* incorporates behaviors from policies of other tasks without introducing bias into the RL objective for the current task.
We achieve this by using a mixture of all policies as the behavioral policy for the current task, thereby modifying only its *off-policy training data*. However, naively mixing other policies into the current task’s behavioral policy does not necessarily improve its sample efficiency. To address this, we derive a specific definition of this mixture, named Q-switch Mixture of Policies (QMP), that selects a policy based on the current task’s Q-function (see Figure 2 and Algorithm 1) and prove that QMP guarantees greater than or equal sample efficiency than using the current task’s policy alone.

Figure: Figure 2: Our method (QMP) shares behavior between task policies in the data collection phase using a mixture of these policies. For example, in Task 1, each task policy proposes an action $a_{j}$. The task-specific Q-switch evaluates each $Q_{1}(s,a_{j})$ and selects the best scored policy to gather reward-labeled data to train $Q_{1}$ and $\pi_{1}$. Thus, Task 1 will be boosted by incorporating high-reward shareable behaviors into $\pi_{1}$ and improving $Q_{1}$ for subsequent Q-switch evaluations.
Refer to caption: x2.png

#### 4.1 Multi-Task Behavior Sharing via Off-Policy Data Collection

MTRL methods like Teh et al. () use *regularization to a common average policy* to enforce task policies to share behaviors. However, this introduces bias to each policy’s RL objective, leading to suboptimal actions in states where tasks require different actions.
To address this, we propose using a *mixture of policies for off-policy data collection* as the means of behavior-sharing. At each state in any given task, one of the task policies is selected to gather training data as the current behavioral policy. This approach is compatible with any off-policy RL algorithm (Watkins & Dayan, ) because the environment rewards help determine the best actions from the collected data. However, an effective mixture policy must choose the behavioral policies in a selective and principled way.

###### Definition 4.1 (Mixture of Policies) .

For each task $\mathbb{T}_{i}$, the mixture policy $\pi_{i}^{\text{mix}}(a\mid s)$ is defined as
$\pi_{i}^{\text{mix}}(a\mid s)=\pi_{f_{i}\big{(}s,\pi_{1},\ldots,\pi_{N}\big{)}
}(a\mid s),$
where $f_{i}(s,\pi_{1},\ldots,\pi_{N}):\mathcal{S}\times\Pi^{N}\to\{1,\ldots,N\}$ is a mixture-switch function that selects one of the policies $\pi_{1},\ldots,\pi_{N}$ based on the current state $s$.

Our intuition of policy mixture shares inspiration with hierarchical RL (Çelik et al., ; Daniel et al., ; End et al., ; Goyal et al., ) where a *mixture* of options is learned according to the downstream task(s). However, a key difference in an MTRL mixture is that each policy is optimized for its own specific task and not designed to fit the task where the mixture is employed.

###### Definition 4.1 (Mixture of Policies) .

#### 4.2 Q-switch Mixture of Policies (QMP)

We aim to derive a principled mixture-switch function $f_{i}$ such that the mixture policy $\pi_{i}^{\text{mix}}$ selectively incorporates behaviors from other policies while being guaranteed to be no worse than the current task’s policy $\pi_{i}$.
We recall the generalized policy iteration procedure (Sutton & Barto, ) underlying single-task SAC (Haarnoja et al., ): policy evaluation learns $Q$ by minimizing the bellman error on the collected data, and policy improvement follows $Q$ by minimizing the KL divergence between the new policy and the exponential of the current $Q$-function, $Q^{\pi^{\text{old}}}$:

$$ $\pi^{\text{new}}=\arg\min_{\pi^{\prime}\in\Pi}\mathrm{D}_{\mathrm{KL}}\left( \pi^{\prime}(\cdot\mid s_{t})\Bigg{\|}\frac{\exp\left(\frac{1}{\alpha}Q^{\pi^{ \text{old}}}(s_{t},\cdot)\right)}{Z^{\pi^{\text{old}}}(s_{t})}\right)$ (1) $$

In practice, the gradient updates in SAC are gradual and do not instantly achieve this optimization in Eq. 1, leaving a suboptimality gap to catch up to the Q-function.
Thus, a mixture policy $\pi_{i}^{\text{mix}}$ that selects the best policy from a set of all given policy candidates, *including the current policy*, ensures that $\pi_{i}^{\text{mix}}$ is at least as good as $\pi_{i}$ for the current state $s$, while potentially being a better optimizer of Eq. 1 due to shareable behaviors from the other task policies:

$$ $\min_{\pi^{\prime}\in\{\pi_{1},...,\pi_{N}\}}\mathrm{D}_{\mathrm{KL}}\left(\pi ^{\prime}(\cdot\mid s)\Bigg{\|}\frac{\exp(\frac{1}{\alpha}Q^{\pi_{i}}(s,\cdot) )}{Z^{\pi_{i}}(s)}\right)\leq\mathrm{D}_{\mathrm{KL}}\left(\pi_{i}(\cdot\mid s _{t})\Bigg{\|}\frac{\exp(\frac{1}{\alpha}Q^{\pi_{i}}(s,\cdot))}{Z^{\pi_{i}}(s) }\right)$ (2) $$

Simplifying the expression on the left results in the following definition (derivation in Appendix B).

###### Definition 4.2 (Q-switch Mixture of Policies: QMP) .

For a task $\mathbb{T}_{i}$ and available candidate policies $\{\pi_{1},...,\pi_{N}\}$, the QMP $\pi_{i}^{\text{mix}}(a\mid s)$ selects a policy according to:

$$ $\pi_{i}^{\text{mix}}=\arg\max_{\pi^{\prime}\in\{\pi_{1},\dots,\pi_{N}\}}\>\> \mathbb{E}_{a\sim\pi^{\prime}(\cdot\mid s)}\left[Q^{\pi_{i}}(s,a)\right]+ \alpha\mathcal{H}\left[\pi^{\prime}(\cdot\mid s)\right]$ (3) $$

Figure: Algorithm 1 Q-switch Mixture of Policies (QMP)

Algorithm 1 shows that QMP can be plugged into any MTRL framework, making it complementary with various MTRL frameworks like parameter-sharing and data relabeling (see Section 7.1).
In practice, we estimate the expectation in Eq. 3 by evaluating the Q-value for the mean action of each task policy’s distribution $\pi^{\prime}(\cdot|s)$ ignoring the entropy term. We do not find any empirical difference when using a sampled estimate of the expectation (see Appendix G.2) or including the entropy term, as the Q-value is the primary distinguishing factor between policies. In terms of compute, sampling from QMP’s $\pi_{i}^{\text{mix}}(a|s)$ does require more policy and Q-function evaluations. However, evaluations are parallelized and impact runtime negligibly, as shown in Appendix G.4.

While $\pi_{i}^{\text{mix}}$ can mistakenly choose a poor policy due to estimation error in $Q^{\pi_{i}}$,
this is identical to Q-learning or SAC, where the Q-function would be inaccurate at less-seen states. In both Q-learning and QMP, this is corrected with online interactions where the Q-function is trained to be more accurate in a subsequent iteration. Furthermore, $\pi_{i}^{\text{mix}}$ actually better maximizes $Q^{\pi_{i}}$ than $\pi_{i}$, which is the objective under generalized policy iteration. Note that QMP does not exacerbate the problem of overestimation because the soft policy evaluation step stays the same, i.e., it uses $\pi_{i}$ and not $\pi_{i}^{\text{mix}}$.

###### Definition 4.2 (Q-switch Mixture of Policies: QMP) .

### 5 Why QMP Works: Theory and Didactic Example

#### 5.1 QMP: Convergence and Improvement Guarantees

QMP performs simultaneous MTRL by collecting data using a Q-switch guided mixture of policies $\pi_{i}^{\text{mix}}$.
In Appendix C, we prove that QMP with underlying RL algorithm Soft-Actor Critic (SAC) (Haarnoja et al., ) shares the same convergence guarantees in a tabular setting. Particularly, we show that under QMP, policy evaluation converges because QMP only modifies data collection of off-policy RL, policy improvement guarantees are preserved (Theorem 5.1), and policy iteration converges to an optimal policy at least as sample-efficiently (Theorem C.2).

The key reason for *better policy improvement* of QMP over the current task policy $\pi_{i}$ is the $\arg\max$ operation in Eq. 3, which ensures that the selected policy $\pi_{i}^{\text{mix}}\in\{\pi_{j}\}_{j=1}^{N}$ optimizes the SAC objective at least as well as $\pi_{i}$ itself. We formalize this in Theorem 5.1 with proof in Appendix C.1. Due to the suboptimality gap in Eq. 1 in SAC, QMP can actually achieve better policy improvement when there are shareable behaviors between policies.

###### Theorem 5.1 (Mixture Soft Policy Improvement) .

Consider $\pi_{i}^{\text{old}}$ and its associated Q-function $Q_{i}$. Apply SAC’s policy improvement $\pi_{i}^{\text{old}}\to\pi_{i}$ and then $\pi_{i}\to\pi_{i}^{\text{mix}}$ from Eq. 3.
Then, $Q^{\pi_{i}^{\text{mix}}}(\mathbf{s}_{t},\mathbf{a}_{t})\geq Q^{\pi_{i}}(
\mathbf{s}_{t},\mathbf{a}_{t})\geq Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t},
\mathbf{a}_{t})$ for all tasks $\mathbb{T}_{i}$ and for all $(s_{t},a_{t})\in\mathcal{S}\times\mathcal{A}$ with $\left|\mathcal{A}\right|<\infty$.

Figure: Figure 3: QMP generalized policy iteration
Refer to caption: x3.png

While QMP in Def. 4.2 applies to any set of candidate policies $\{\pi_{1},...,\pi_{N}\}$, one expects $\pi_{i}^{\text{mix}}$ to improve over $\pi_{i}$ when some $\pi_{j}\neq\pi_{i}$ proposes an action candidate better than $\pi_{i}$ for Task $\mathbb{T}_{i}$. This is more likely in MTRL policies that *share structure between tasks* than an arbitrary set of policies. For instance, if $\mathbb{T}_{i}$ and $\mathbb{T}_{j}$ share a subtask that appears early in the episodes for $\mathbb{T}_{j}$, then $\pi_{j}$ would have already learned it before $\pi_{i}$ and be a better policy for certain states of $\mathbb{T}_{i}$, according to $Q_{i}$.

QMP making bigger policy improvement steps results in each iteration of generalized policy iteration making more progress towards optimality. This reduces the number of iterations required to converge, improving the sample efficiency of the algorithm as illustrated in Fig. 3 and proved in Theorem C.2.

###### Theorem 5.1 (Mixture Soft Policy Improvement) .

#### 5.2 Illustrative Example: 2D Point Reaching

Figure: (a) SAC: $\pi$
Refer to caption: x4.png

Figure: Figure 5: QMP improves performance using other policies, increasingly so when they are task-relevant (5 seeds).
Refer to caption: x7.png

We demonstrate when QMP can *utilize alternate policy candidates* $\{\pi_{1},\dots,\pi_{N}\}$ to more effectively learn a policy by bridging a *policy improvement suboptimality gap* as $\pi$ tries to follow $Q$ in Eq. 1. Consider a 2D point-reaching task where the agent must navigate from the bottom-left corner $(0,0)$ to the goal in the top-right corner $(10,10)$. The point agent receives dense rewards based on its proximity to the goal and takes incremental 2D actions $(\Delta x,\Delta y)\in[-1,1]^{2}$.

Figure 5 shows that the SAC policy $\pi$ converges to a suboptimal solution. Fig. 4(a) confirms that the data collected by SAC policy never reaches the goal. This shows that if the suboptimality gap in $\pi$ is not successfully bridged, it can make the entire algorithm converge suboptimally.

To illustrate the effect of QMP, we add 3 fixed gaussian policies centered on ($\color[rgb]{0.1328125,0.546875,0.1328125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.1328125,0.546875,0.1328125}{\uparrow}\;\color[rgb]{1,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{1,0,0}{\rightarrow}\;\color[rgb]{.5,0,.5}
\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}{\swarrow}$) or ($\color[rgb]{0.1328125,0.546875,0.1328125}\definecolor[named]{pgfstrokecolor}{
rgb}{0.1328125,0.546875,0.1328125}{\uparrow}\;\color[rgb]{1,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{1,0,0}{\rightarrow}\;\color[rgb]{.5,0,.5}
\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}{\nearrow}$), and only let $\pi$ be trainable. Fig. 4(b), 4(c) show that $\pi_{i}^{\text{mix}}$ employs alternate policies at many states in data collection as they optimize Eq. 3 better than $\pi$ itself. This *selectivity* enables $\pi_{i}^{\text{mix}}$ to generate more effective goal-reaching trajectories by bridging the suboptimality gap, resulting in better performance in Fig. 5. A policy like $\color[rgb]{.5,0,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,0,.5}{\nearrow}$ that is more relevant to the underlying task leads to a larger gain.

The same principle extends to the simultaneous multi-task RL setting. In MTRL, each task’s policy continuously improves and can serve as a valuable candidate in the mixture for other tasks. QMP enables tasks to selectively share their behaviors, allowing each task to benefit from the progress of others. This mutual assistance accelerates learning across all tasks, as the mixture policy $\pi_{i}^{\text{mix}}$ for each task $\mathbb{T}_{i}$ selects the most promising action proposals from all available policies according to the task-specific Q-function, guaranteed to be at least as good as $\pi_{i}$ itself. Consequently, MTRL combined with QMP leverages the collective knowledge of all tasks to bridge suboptimality gaps more efficiently, leading to improved sample efficiency and overall performance.

### 6 Experiments

#### 6.1 Environments

We evaluate our method in 7 multi-task designs in manipulation, navigation, and locomotion environments, shown in Figure 6. These tasks vary in the degree of shared and conflicting behaviors between tasks and the number of tasks in the set.
Further details in Appendix Section D.

Multistage Reacher: A 6 DoF Jaco arm learns 5 tasks with ordered subgoals. The first 4 tasks share some subgoals, while the 5th *conflicting* task requires the agent to stay at its initial position.

Maze Navigation: Building on point mass maze navigation (Fu et al., ), we define 10 tasks with various start and goal locations exhibiting coinciding and conflicting segments in the optimal paths.

Meta-World Manipulation:
We use three task sets based on the Meta-World environment (Yu et al., ). Meta-World MT10 and Meta-World MT50 are sets of 10 and 50 table-top manipulation tasks involving different objects and behaviors. Meta-World CDS is a 4-task setup proposed in  Yu et al. () which places the door and drawer objects next to each other on the same tabletop so that all 4 tasks (door open & close, drawer open & close) are solvable in a simultaneous multi-task setup.

Walker2D:
Walker2D is a 9 DoF bipedal walker agent with the multi-task set containing 4 locomotion tasks proposed in  Lee et al. (): walking forward, walking backward, balancing, and crawling. These tasks require different gaits without an obviously identifiable shared behavior in the optimal policies but can still benefit from intermediate behaviors like balancing.

Kitchen:
We use the challenging manipulation environment proposed by Gupta et al. () where a 9 DoF Franka robot performs tasks in a kitchen. We create a task set out of 10 manipulation tasks: turning on or off different burners and light switches, and opening or closing different cabinets.

Figure: (a) Jaco Reacher
Refer to caption: extracted/5906707/figures/envs/env_fig.001.png

#### 6.2 Baselines

We first select popular and representative MTRL methods that share other forms of information to evaluate how behavior-sharing with QMP improves their performance:

- •
No-Sharing consists of $N$ (refers to number of tasks) independent RL architectures where each agent is assigned one task and trained to solve it without any information sharing with other agents.
- •
Data-Sharing (UDS) proposed in Yu et al. () shares data between tasks, relabelling with minimum task reward. We modified this offline RL algorithm to online.
- •
Parameter-Sharing a multi-head SAC policy sharing parameters but not behaviors over tasks.

We validate QMP’s approach to share behaviors via *off-policy data collection* with other approaches:

- •
No-Shared-Behaviors consists of $N$ RL agents where each agent is assigned one task and trained to solve it without any behavior sharing with other agents: no bias and no sharing.
- •
Fully-Shared-Behaviors is a single SAC agent that learns one shared policy for all tasks, outputting the same action for a given state regardless of task (full parameter sharing): fully biased sharing.
- •
Divide-and-Conquer RL (DnC) (Ghosh et al. ()) uses $N$ policies that share behaviors through policy distillation and regularization to the mean (adapted for MTRL): biased objective for sharing.
- •
DnC (Regularization Only) is a no policy distillation variant of DnC we propose as a baseline.
- •
QMP (Ours) learns $N$ policies that share behaviors in off-policy data collection: unbiased sharing.

We used SAC Haarnoja et al. () for all environments and methods. All the non-parameter sharing baselines use the same SAC hyperparameters. Please refer to Appendix H for complete details.

Figure: Figure 7: Behavioral policy sharing is complementary. QMP (solid lines) shows general improvement over MTRL frameworks (same-colored dashed lines) like No shared architecture (blue), shared parameters (pink), and shared data (green). Methods without parameter-sharing on MT50 converge very slowly. Success rate means and std (shaded) are over $N$ tasks, 10 episodes per task, and 5 seeds.
Refer to caption: x8.png

### 7 Results

Our experiments address: (1) Does QMP provide complementary gains to other forms of MTRL? (2) How does sharing behavioral policies compare with alternate forms of behavior sharing? (3) Can QMP effectively identify shareable behaviors? (4) Ablating key components of QMP.

#### 7.1 Is Behavior Sharing Complementary to other MTRL frameworks?

We demonstrate that our method is compatible with and provides complementary performance gains with other forms of MTRL that share different kinds of information, including parameter sharing and data sharing.
We compare the performance between 3 base MTRL algorithms, No-Sharing, Parameter-Sharing, and Data-Sharing, with the addition of QMP in Figure 7. The No-Sharing baseline provides a baseline comparison of QMP’s effectiveness on its own. For the Parameter-Sharing and Data-Sharing baselines we chose the base algorithms for their popularity and simplicity. In each case, we add QMP by simply replacing the data collection policy with $\pi_{i}^{\text{mix}}$.
We find that QMP is complementary to all three baseline frameworks, mostly with additive performance gains in sample efficiency and final performance, while not hurting the performance of the base method in all but one case (Data-Sharing in Kitchen).
We additionally see that QMP improves PCGrad’s (Yu et al., ) performance significantly in 3 out of 4 environments tested in Appendix E.3. This shows that QMP is a simple and complementary addition to other forms of MTRL.

QMP significantly improves upon the No-Sharing baseline in all environments except Meta-World CDS where it performs comparatively. This demonstrates that sharing behavioral policies is a promising avenue for efficient and performant MTRL. In the data-sharing comparison, we see that the addition of QMP improves or performs comparatively to the base algorithm. In Multistage Reacher and Maze Navigation, we see that both Data-Sharing and Data + QMP perform worse than the other MTRL methods, highlighting the fact that sharing data directly between tasks can be ineffective without access to a re-labeled task rewards like in our setting.
In environments where data-sharing does well, like Meta-World CDS, we see that adding QMP does improve sample efficiency.

We find that Parameters + QMP generally outperforms Parameter-Sharing, while inheriting its sample efficiency gains. In many cases, the parameter-sharing methods converge sub-optimally, highlighting that shared parameter MTRL has its own challenges. However, in Maze Navigation, we find that sharing Parameters + Behaviors greatly improves the performance over both the Parameter-Sharing baseline *and* No-Sharing + QMP variant of QMP.
This demonstrates the additive effect of these two forms of information sharing in MTRL. The agent initially benefits from the sample efficiency gains of the multi-head parameter-sharing architecture, while behavior sharing accelerates learning by selectively using other policies to keep learning even after the parameter-sharing effect plateaus.
demonstrating the compatibility between QMP and parameter sharing as key ingredients to sample efficient MTRL.
We further highlight that this benefit of QMP increases with the number of tasks increasing from 10 to 50 in Meta-World, where we see that QMP is actually more effective when combined with parameter sharing in MT50 than in MT10. QMP scales well with the number of tasks and can actually perform better likely due to *more shared behaviors* in the larger task set.

#### 7.2 Baselines: Comparing Different Approaches to Share Behaviors

To verify QMP’s efficacy as a behavior-sharing mechanism, we evaluate baselines that share behaviors in different ways on 6 environments in Figure 8. QMP reliably matches or exceeds other methods, especially in tasks that require conflicting behaviors, where alternate approaches are ineffective.

In the task sets with the most directly conflicting behaviors, Multistage Reacher and Maze Navigation, our method clearly outperforms other behavior-sharing and data-sharing baselines. In Multistage Reacher, our method reaches $>$ 90% success rate at 0.5 million environment steps, while DnC (reg.), the next best method, takes 3 times the number of steps to fully converge. The rest of the methods fail to attain the maximum success rate. The UDS baseline performs particularly poorly, illustrating that data sharing can be ineffective without ground truth rewards.
We also see that QMP scales better from 3 to 10 tasks in Maze compared to other behavior sharing methods in Appendix Section  E.4.

In the remaining task sets with no directly conflicting behaviors, we see that QMP is competitive with the best-performing baseline for more complex manipulation and locomotion tasks.
Particularly, in Walker2D and Meta-World CDS, we see that QMP is the most sample-efficient method and converges to a better final performance than any other behavior sharing method.
In Meta-World MT10 and Kitchen, DnC (regularization only) also performed very well, showing that well-tuned uniform behavior sharing can be very effective in tasks without conflicting behavior. However, QMP also performs competitively and more sample efficiently, showing that QMP is effective under the same assumptions as uniform behavior-sharing methods but can do so *adaptively* and across more *general task families*. The Fully-Shared-Behaviors baseline often performs poorly because it totally biases the policies, while the No-Shared-Behavior is a surprisingly strong baseline as it introduces no bias.

Figure: Figure 8: QMP reliably shares behaviors. In task sets exhibiting conflicting behaviors, QMP consistently matches or exceeds baselines in rate of convergence and final performance.
Refer to caption: x15.png

#### 7.3 Can QMP effectively identify shareable behaviors?

Figure 9(a) shows the average proportion of sharing from other tasks for Multistage Reacher Task 0 over the course of training. We see that QMP learns to generally share less behavior from Policy 4 than from Policies 1-3 (Appendix Figure 15). Conversely, QMP in Task 4 also shares the least total cross-task behavior (Appendix Figure 14). We see this same trend across all 5 Multistage Reacher tasks, showing that the Q-switch successfully identifies conflicting behaviors that should not be shared. Further, Figure 9(a) also shows that total behavior-sharing from other tasks goes down over training. Thus, Q-switch learns to prefer its own task-specific policy as it becomes more proficient.

Figure: (a) Behavior-sharing over training
Refer to caption: x22.png

We qualitatively analyze how behavior sharing varies within a single episode by visualizing a QMP rollout during training for the Drawer Open task in Meta-World CDS (Figure 9(b)). We see that it makes reasonable policy choices by (i) switching between all 4 task policies as it approaches the drawer (top row), (ii) using drawer-specific policies as it grasps the drawer-handle, and (iii) using Drawer Open and Door Open policies as it pulls the drawer open (bottom row). In conjunction with the overall results, this supports our claim that QMP can effectively identify shareable behaviors between tasks. For details on this visualization and the full analysis results see Appendix Section F.

Inspired by hierarchical RL (Dabney et al., ) and multi-task exploration (Xu et al., ), we briefly investigate *temporally extended behavior sharing* in Appendix E.6.Recently, Xu et al. () showed that if one assumes a high overlap between optimal policies of different tasks, other task policies can aid exploration. So, we simply roll out each policy QMP selects for a fixed number of steps. QMP theory no longer holds as it requires selecting a policy at every step. Yet, this naive temporally extended QMP yields improvements in some environments like Maze with strong overlap.

Figure: Figure 10: QMP outperforms alternate policy mixtures in Multistage Reacher.
Refer to caption: x24.png

#### 7.4 Ablations

We show the importance of Q-switch in QMP (Def. 4.2) against alternate forms of policy mixtures (Def. 4.1). QMP-Uniform is a uniform distribution over policies, $f_{i}=\mathbb{U}(\{1,\dots,N\})$ and achieves only 60% success rate (Figure  10), showing the importance of selectivity.
QMP-Domain-Knowledge is a hand-crafted, fixed policy distribution based on an estimate of similarity between tasks. Multistage Reacher measures this similarity by the shared sub-goal sequences between tasks (Appendix D). QMP-Domain performs well initially but plateaus early, showing that which behaviors are shareable depends on the state and current policy.
We further ablate the $\arg\max$ in Q-switch against a softmax varation resulting in a *probabilistic mixture policy* in Appendix Section G.1, and evaluating on the *mean policy actions* (Appendix Section G.2) to validate our design.

### 8 Conclusion

We propose an unbiased approach to sharing behaviors in MTRL: Q-switch Mixture of Policies. QMP shares policies between tasks for off-policy data collection.
We demonstrate empirically that QMP effectively improves the rate of convergence and task performance in manipulation, locomotion, and navigation tasks, and is guaranteed to be as good as the underlying RL algorithm and complementary to alternate MTRL. QMP does not assume that optimal task behaviors always coincide. Thus, its improvement magnitude is limited by the degree of shareable behaviors and the suboptimality gap that exists. At the same time, this lets the QMP algorithm be unbiased and find optimal policies with convergence guarantees while being equally or more sample-efficient.
Promising future directions include temporally-extended behavior sharing and incorporating other forms of prior task information on shareable behaviors, such as language embeddings in instruction-following tasks.

### Acknowledgements

We thank Jesse Zhang for his assistance with writing and discussions. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant (No.RS-2019-II190075, Artificial Intelligence Graduate School Program, KAIST) and National Research Foundation of Korea (NRF) grant (NRF-2021H1D3A2A03103683, Brain Pool Research Program), funded by the Korea government (MSIT). Grace Zhang and Ayush Jain were supported partly as interns at Naver AI Lab during the initiation of the project.
Shao-Hua Sun was supported by the Yushan Fellow Program by the Ministry of Education, Taiwan.

### Appendix

## 

### Appendix A Qualitative Results

The qualitative result videos are provided at [https://qmp-mtrl.github.io/](https://qmp-mtrl.github.io/)

### Appendix B QMP Derivation

Following Section 4.2, we aim to derive the mixture-switch function $f_{i}$ such that the mixture policy $\pi_{i}^{\text{mix}}$ is guaranteed to be better than the current task’s policy $\pi_{i}$.
We use the generalized policy iteration procedure (Sutton & Barto, ) underlying single-task SAC (Haarnoja et al., ): policy evaluation learns $Q$ by minimizing the bellman error on the collected data, and policy improvement follows $Q$ by minimizing the KL divergence between the new policy and the exponential of the current $Q$-function, $Q^{\pi^{\text{old}}}$, shown in Eq. 1.

In practice, the gradient updates in SAC are gradual and do not instantly achieve this optimization in Eq. 1, leaving a suboptimality gap to catch up to the Q-function. We observe that due to the potential similarity of some tasks in MTRL, this suboptimality gap can be bridged using other policies. Concretely, a mixture policy $\pi_{i}^{\text{mix}}$ that selects the best policy from a set of all given policy candidates, *including the current policy*, ensures that $\pi_{i}^{\text{mix}}$ is an improvement over $\pi_{i}$ for the current state $s$:

Given a set of policies $\{\pi_{1}\dots\pi_{N}\}$ including the current task policy $\pi_{i}$ and a given state $s$, consider the following mixture policy:

$$ $\pi_{i}^{\text{mix}}=\arg\min_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathrm {D}_{\mathrm{KL}}\left(\pi^{\prime}(\cdot\mid s)\Bigg{\|}\frac{\exp(\frac{1}{ \alpha}Q^{\pi_{i}}(s,\cdot))}{Z^{\pi_{i}}(s)}\right)$ (4) $$

This $\pi_{i}^{\text{mix}}$ is a better policy improvement solution to Eq. 1 than $\pi_{i}$, because:

$$ $\min_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathrm{D}_{\mathrm{KL}}\left( \pi^{\prime}(\cdot\mid s)\Bigg{\|}\frac{\exp(\frac{1}{\alpha}Q^{\pi_{i}}(s, \cdot))}{Z^{\pi_{i}}(s)}\right)\leq\mathrm{D}_{\mathrm{KL}}\left(\pi_{i}(\cdot \mid\mathbf{s}_{t})\Bigg{\|}\frac{\exp(\frac{1}{\alpha}Q^{\pi_{i}}(\mathbf{s}_ {t},\cdot))}{Z^{\pi_{i}}(\mathbf{s}_{t})}\right)$ $$

Now, we can simplify Eq. 4 to obtain Definition 4.2:

$$ $\displaystyle\pi_{i}^{\text{mix}}$ $\displaystyle=\arg\min_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathrm{D}_{ \mathrm{KL}}\left(\pi^{\prime}(\cdot\mid s)\Bigg{\|}\frac{\exp(\frac{1}{\alpha }Q^{\pi_{i}}(s,\cdot))}{Z^{\pi_{i}}(s)}\right)$ $\displaystyle=\arg\min_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathbb{E}_{a \sim\pi^{\prime}(\cdot\mid s)}\left[\log\pi^{\prime}(a|s)-\log\left\{\frac{ \exp(\frac{1}{\alpha}Q^{\pi_{i}}(s,a))}{Z^{\pi_{i}}(s)}\right\}\right]$ $\displaystyle=\arg\max_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathbb{E}_{a \sim\pi^{\prime}(\cdot\mid s)}\left[-\log\pi^{\prime}(a|s)+\frac{1}{\alpha}Q^{ \pi_{i}}(s,a)-\log{Z^{\pi_{i}}(s)}\right]$ $\displaystyle=\arg\max_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathbb{E}_{a \sim\pi^{\prime}(\cdot\mid s)}\left[-\log\pi^{\prime}(a|s)\right]+\mathbb{E}_{ a\sim\pi^{\prime}(\cdot\mid s)}\left[\frac{1}{\alpha}Q^{\pi_{i}}(s,a)\right]$ $\displaystyle=\arg\max_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathbb{E}_{a \sim\pi^{\prime}(\cdot\mid s)}\left[Q^{\pi_{i}}(s,a)\right]+\alpha\mathcal{H} \left[\pi^{\prime}(\cdot\mid s)\right]$ $$

Thus, the following mixture policy guarantees improvement over $\pi_{i}$

$$ $\pi_{i}^{\text{mix}}=\arg\max_{\pi^{\prime}\in\{\pi_{i},\dots\pi_{N}\}}\mathbb {E}_{a\sim\pi^{\prime}(\cdot\mid s)}\left[Q^{\pi_{i}}(s,a)\right]+\alpha \mathcal{H}\left[\pi^{\prime}(\cdot\mid s)\right]\\$ $$

### Appendix C QMP Convergence Guarantees

We derive the convergence guarantees for mixture soft policy iteration used in the QMP Algorithm 1. We augment the derivation of soft policy iteration in SAC (Haarnoja et al., ), which is our base algorithm, with our proposed QMP’s mixture policy. Soft policy iteration follows generalized policy iteration (Sutton & Barto, ) which refers to the general idea of repeated application of (1) policy evaluation to update the critics and (2) policy improvement based on the updated critics, until convergence. Like SAC, we consider the tabular setting and show that QMP’s modification to soft policy iteration converges to the optimal policy. Further, QMP can lead to an improved policy improvement step when there are shareable behaviors between tasks, consequently improving the sample efficiency. The derivation sketch follows:

- 1.
Soft Policy Evaluation: QMP modifies the off-policy data collection pipeline by replacing the primary task policy ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}}}$ with the mixture policy ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix}}}}$. However, it does not affect the soft Bellman backup operator of SAC, as shown in  Haarnoja et al. (), and therefore the $Q$ function still converges as in SAC.
- 2.
Mixture Soft Policy Improvement: QMP performs policy improvement in two steps: SAC’s policy update from ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{old}}}}~
{}\to$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}}}$ and applying the mixture of policies from ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}}}$ $\to$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix}}}}$.
•
Soft Policy Improvement: Since QMP does not modify the SAC update procedure ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{old}}}}~
{}\to$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}}}$ , we directly use SAC’s guarantees of policy improvement following Lemma 2 from  Haarnoja et al. ().
•
Mixture Policy Improvement: We demonstrate QMP’s mixture policy ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix}}}}$ guarantees a better policy improvement over the per-task policies ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}}}$ that compose the mixture. In Theorem C.1, we show convergence guarantee by proving that the expected return following ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix}}}}$  is better than following ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{old}}}}$.
- 3.
Mixture Soft Policy Iteration: In Theorem C.2, we show that the repeated application of the above steps in QMP converges to an optimal policy for each task. Furthermore, the convergence rate is faster because of a greedier policy improvement due to Mixture Policy Improvement.

For a given stochastic policy $\pi$ and task $\mathbb{T}_{i}\in\{\mathbb{T}_{1}\dots\mathbb{T}_{N}\}$, define $V_{i}^{\pi}$ as the expected return of acting with $\pi$. Given another stochastic policy $\pi^{\prime}$, define $Q_{i}^{\pi}(s,\pi^{\prime}(s))=\mathbb{E}_{a\sim\pi^{\prime}(s)}Q_{i}^{\pi}(s,a)$ as the expected return of acting with $\pi^{\prime}$ only in $\mathbf{s}$ and thereafter with $\pi$.

###### Theorem C.1 (Mixture Soft Policy Improvement) .

Consider $\pi_{i}^{\text{old}}$ and its associated Q-function $Q_{i}$. Apply SAC’s policy improvement $\pi_{i}^{\text{old}}\to\pi_{i}$ and then $\pi_{i}\to\pi_{i}^{\text{mix}}$ from Eq. 3.
Then, $Q^{\pi_{i}^{\text{mix}}}(\mathbf{s}_{t},\mathbf{a}_{t})\geq Q^{\pi_{i}}(
\mathbf{s}_{t},\mathbf{a}_{t})\geq Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t},
\mathbf{a}_{t})$ for all tasks $\mathbb{T}_{i}$ and for all $(\mathbf{s}_{t},\mathbf{a}_{t})\in\mathcal{S}\times\mathcal{A}$ with $\left|\mathcal{A}\right|<\infty$.

###### Proof.

From Soft Policy Improvement, Lemma 2 of Haarnoja et al. (), we have

$$ $\mathbb{E}_{\mathbf{a}_{t}\sim\pi_{i}}\left[Q^{\pi_{i}^{\text{old}}}(\mathbf{s }_{t},\mathbf{a}_{t})-\log\pi_{i}(\mathbf{a}_{t}|\mathbf{s}_{t})\right]\geq V^ {\pi_{i}^{\text{old}}}(\mathbf{s}_{t}).$ $$

Rewrite the difference as $\delta(\mathbf{s}_{t})$,

$$ $\delta(\mathbf{s}_{t})=\mathbb{E}_{\mathbf{a}_{t}\sim\pi_{i}}\left[Q^{\pi_{i}^ {\text{old}}}(\mathbf{s}_{t},\mathbf{a}_{t})-\log\pi_{i}(\mathbf{a}_{t}| \mathbf{s}_{t})\right]-V^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t})\geq 0.$ $$

From Eq. 3,

$$ $\pi_{i}^{\text{mix}}=\arg\max_{\pi^{\prime}\in\{\pi_{1},\dots,\pi_{N}\}}\>\> \mathbb{E}_{a\sim\pi^{\prime}(\cdot\mid s)}\left[Q^{\pi_{i}}(s,a)\right]+ \alpha\mathcal{H}\left[\pi^{\prime}(\cdot\mid s)\right].$ $$

Therefore, we have a positive difference $\omega(\mathbf{s}_{t})$,

$$ $\omega(\mathbf{s}_{t})=\mathbb{E}_{\mathbf{a}_{t}\sim\pi_{i}^{\text{mix}}} \left[Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t},\mathbf{a}_{t})-\log\pi_{i}^{ \text{mix}}(\mathbf{a}_{t}|\mathbf{s}_{t})\right]-\mathbb{E}_{\mathbf{a}_{t} \sim\pi_{i}}\left[Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t},\mathbf{a}_{t})-\log \pi_{i}(\mathbf{a}_{t}|\mathbf{s}_{t})\right]\geq 0.$ $$

We use $\delta$ to expand the soft Bellman equation to derive the relationship between $Q^{\pi_{i}^{\text{old}}}$ and $Q^{\pi_{i}}$,

$$ $\displaystyle Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t},\mathbf{a}_{t})$ $\displaystyle=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\,\mathbb{E}_{\mathbf{s}_ {t+1}\sim p}\left[V^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t+1})\right]$ $\displaystyle=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\,\mathbb{E}_{\mathbf{s}_ {t+1}\sim p}\left[\mathbb{E}_{\mathbf{a}_{t+1}\sim\pi_{i}}\left(Q^{\pi_{i}^{ \text{old}}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})-\log\pi_{i}(\mathbf{a}_{t+1}| \mathbf{s}_{t+1})\right)-\delta(\mathbf{s}_{t+1})\right]$ $\displaystyle\vdots$ $\displaystyle=\underbrace{\sum_{k=0}^{\infty}\gamma^{k}\,\mathbb{E}_{\mathbf{s }_{t+k}\sim p,\,\mathbf{a}_{t+k}\sim\pi_{i}}\left[r(\mathbf{s}_{t+k},\mathbf{a }_{t+k})-\log\pi_{i}(\mathbf{a}_{t+k}|\mathbf{s}_{t+k})\right]}_{Q^{\pi_{i}}( \mathbf{s}_{t},\mathbf{a}_{t})}-\underbrace{\sum_{k=1}^{\infty}\gamma^{k}\, \mathbb{E}_{\mathbf{s}_{t+k}\sim p}\left[\delta(\mathbf{s}_{t+k})\right]}_{ \Delta_{1}}$ $\displaystyle=Q^{\pi_{i}}(\mathbf{s}_{t},\mathbf{a}_{t})-\Delta_{1}$ $$

Likewise, we use $\delta$ and $\omega$ to derive the relationship between $Q^{\pi_{i}^{\text{old}}}$ and $Q^{\pi_{i}^{\text{mix}}}$,

$$ $\displaystyle Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t},\mathbf{a}_{t})$ $\displaystyle=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\,\mathbb{E}_{\mathbf{s}_ {t+1}\sim p}\left[V^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t+1})\right]$ $\displaystyle=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\,\mathbb{E}_{\mathbf{s}_ {t+1}\sim p}\left[\mathbb{E}_{\mathbf{a}_{t+1}\sim\pi_{i}}\left(Q^{\pi_{i}^{ \text{old}}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})-\log\pi_{i}(\mathbf{a}_{t+1}| \mathbf{s}_{t+1})\right)-\delta(\mathbf{s}_{t+1})\right]$ $\displaystyle=r(\mathbf{s}_{t},\mathbf{a}_{t})+\gamma\,\mathbb{E}_{\mathbf{s}_ {t+1}\sim p}\left[\mathbb{E}_{\mathbf{a}_{t+1}\sim\pi_{i}^{\text{mix}}}\left(Q ^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t+1},\mathbf{a}_{t+1})-\log\pi_{i}^{\text{ mix}}(\mathbf{a}_{t+1}|\mathbf{s}_{t+1})\right)-\delta(\mathbf{s}_{t+1})- \omega(\mathbf{s}_{t+1})\right]$ $\displaystyle\vdots$ $\displaystyle=\underbrace{\sum_{k=0}^{\infty}\gamma^{k}\,\mathbb{E}_{\mathbf{s }_{t+k}\sim p,\,\mathbf{a}_{t+k}\sim\pi_{i}^{\text{mix}}}\left[r(\mathbf{s}_{t +k},\mathbf{a}_{t+k})-\log\pi_{i}^{\text{mix}}(\mathbf{a}_{t+k}|\mathbf{s}_{t+ k})\right]}_{Q^{\pi_{i}^{\text{mix}}}(\mathbf{s}_{t},\mathbf{a}_{t})}$ $\displaystyle\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;-\underbrace{\sum _{k=1}^{\infty}\gamma^{k}\,\mathbb{E}_{\mathbf{s}_{t+k}\sim p}\left[\delta( \mathbf{s}_{t+k})\right]}_{\Delta_{2}}-\underbrace{\sum_{k=1}^{\infty}\gamma^{ k}\,\mathbb{E}_{\mathbf{s}_{t+k}\sim p}\left[\omega(\mathbf{s}_{t+k})\right]}_ {\Omega}$ $\displaystyle=Q^{\pi_{i}^{\text{mix}}}(\mathbf{s}_{t},\mathbf{a}_{t})-\Delta_{ 2}-\Omega,$ $$

We assume that the effect of the difference $\Delta_{2}-\Delta_{1}$ due to different state coverage is lower compared to the effect of $\Omega$ because $\omega$ is accumulated at every state, i.e., $\Delta_{2}+\Omega=\Delta_{1}+(\Delta_{2}-\Delta_{1})+\Omega\geq\Delta_{1}$

Since $\Delta_{1},\Delta_{2}\geq 0$ and $\Omega\geq 0$, we have

$$ $Q^{\pi_{i}^{\text{mix}}}(\mathbf{s}_{t},\mathbf{a}_{t})\geq Q^{\pi_{i}}( \mathbf{s}_{t},\mathbf{a}_{t})\geq Q^{\pi_{i}^{\text{old}}}(\mathbf{s}_{t}, \mathbf{a}_{t})$ $$

∎

###### Theorem C.2 (Mixture Soft Policy Iteration) .

Repeated application of (i) soft policy evaluation and (ii) mixture soft policy improvement (Theorem C.1) to any $\pi_{i}\in\Pi$ converges to an optimal policy $\pi_{i}^{*}$ such that $Q_{i}^{\pi_{i}^{*}}(\mathbf{s}_{t},\mathbf{a}_{t})\geq Q_{i}^{\pi_{i}}(\mathbf
{s}_{t},\mathbf{a}_{t})$ for all $\pi_{i}\in\Pi$ and $(\mathbf{s}_{t},\mathbf{a}_{t})\in\mathcal{S}\times\mathcal{A}$ with $\left|\mathcal{A}\right|<\infty$.
Furthermore, the sample efficiency and rate of convergence is at least as good as SAC in the presence of mixture policy improvement.

###### Proof.

Let ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{k}}}$ be the policy at iteration $k$. By SAC’s soft policy iteration, the sequence $Q_{i}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{k}}}}$ is monotonically increasing, because ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix}}}}$ only modifies the online data collected and SAC is an off-policy algorithm. Thus, Theorem 1 (Soft Policy Iteration) from Haarnoja et al. () Appendix B.3 directly applies here and proves that repeated application of soft policy evaluation and soft policy improvement converges to an optimal policy $\pi_{i}^{*}$.

Mixture soft policy improvement (Theorem C.1) shows that  ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix}}}}$ is a greedier policy improvement over ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}}}$with respect to each estimate of $Q_{i}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{k}}}}$. Thus, the expected returns in the data collected by QMP policy, $Q_{i}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{\text{mix; k}}
}}}$, is greater than or equal to that collected by the individual task policy, $Q_{i}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\pi_{i}^{k}}}}$. Therefore, every mixture soft policy improvement step constitutes a *larger policy improvement step* than SAC’s soft policy improvement step. This makes the convergence of mixture soft policy iteration (repeated application of soft policy evaluation and Theorem C.1) an improvement over soft policy iteration.

∎

###### Theorem C.1 (Mixture Soft Policy Improvement) .

###### Proof.

###### Theorem C.2 (Mixture Soft Policy Iteration) .

###### Proof.

### Appendix D Environment Details

#### D.1 Multistage Reacher

We implement our multistage reacher tasks on top of the Open AI Gym (Brockman et al., ) Reacher environment simulated in the MuJoCo physics engine (Todorov et al., ) by defining a sequence of 3 subgoals per task which are specified in Table 1. For all tasks, the reacher is initialized at the same start position with a small random perturbation sampled uniformly from $[-0.01,0.01]$ for each coordinate. The observation includes the agent’s proprioceptive state and how many sub-goals have been reached but not subgoal locations, as they must be inferred from the respective task’s reward function.

We set up the tasks to ensure that we can evaluate behavior sharing when the task rewards are qualitatively different (see Figure 6(a)):

- •
For every task except Task 3, the reward function is the default gym reward function based on the distance to the goal, plus an additional bonus for every subgoal completed.
- •
For Task 1, the reward is shifted by -2 at every timestep.
- •
Task 3 receives only a sparse reward of 1 for every subgoal reached.
- •
Task 4 has one fixed goal set at its initial position.

**Table 1: Coordinates of subgoal locations for each task in Multistage Reacher. Shared subgoals are highlighted in the same color. For Task 4, the only goal is to stay in the initial position.**
|  | Subgoal 1 | Subgoal 2 | Subgoal 3 |
| --- | --- | --- | --- |
| $T_{0}$ | (0.2, 0.3, 0.5) | (0.3, 0, 0.3) | (0.4, -0.3, 0.4) |
| $T_{1}$ | (0.2, 0.3, 0.5) | (0.3, 0, 0.3) | (0.4, 0.3, 0.2) |
| $T_{2}$ | (0.3, 0, 0.3) | (0.4, 0.3, 0.2) | (0.4, -0.3, 0.4) |
| $T_{3}$ | (0.3, 0, 0.3) | (0.4, -0.3, 0.4) | (0.2, 0.3, 0.5) |
| $T_{4}$ | initial | initial | initial |

QMP-Domain: Section 7.4 ablates the importance of an adaptive and state-dependent Q-switch by replacing it with a domain-dependent distribution over other tasks based on apparent task similarity. Specifically, to define the mixture probabilities for QMP-Domain in Multistage Reacher, we use the domain knowledge of the subgoal locations of the tasks to determine the mixture probabilities. We use the ratio of *shared sub-goal sequences* between each pair of tasks (not necessarily the shared subgoals) over the total number of sub-goal sequences, 3, to calculate how much behavior must be shared between two tasks.
For that ratio of shared behavior, we distribute the probability mass uniformly between all task policies that share that behavior. For Task 4, the conflicting task, we do not do any behavior sharing and only use $\pi_{4}$ to gather data.

Each Task $\mathbb{T}_{i}$ consists of 3 sub-goal sequences $\{S_{0},S_{1},S_{2}\}$ (e.g. [initial $\to$ Subgoal 1], [Subgoal 1 $\to$ Subgoal 2], and [Subgoal 2 $\to$ Subgoal 3]).
For each sequence $s\in\{S_{0},S_{1},S_{2}\}$, we divide equally the contribution of each task $\mathbb{T}_{j}$’s policy $\pi_{j}$ that shares the sequence $s$ (i.e. if $\mathbb{T}_{0}$ and $\mathbb{T}_{1}$ both contain sequence $s$, where we use the notation $\mathbbm{1}(s\in\mathbb{T}_{i})$ as the indicator function for whether Task $\mathbb{T}_{i}$ contains sequence $s$, then $\pi_{0}$ and $\pi_{1}$ both have $\frac{1}{2}$ contribution for $s$).
Each sequence contributes equally to the overall mixture probabilities for Task $i$ (i.e. all policies that shares sequence $S_{i}$ contributes in total $\frac{1}{3}$ to the mixture probability for the behavior policy of Task $\mathbb{T}_{i}$).
Thus, the contribution probability of Policy $\pi_{j}$ to Task $\mathbb{T}_{i}$ is:

$$ $\displaystyle p_{j\to i}$ $\displaystyle=\sum_{s\in\{S_{0},S_{1},S_{2}\}}\frac{1}{3}\cdot\frac{\mathbbm{1 }(s\in\mathbb{T}_{j})}{\sum_{k}\mathbbm{1}(s\in\mathbb{T}_{k})}$ $\displaystyle\pi_{i}^{\text{mix}}$ $\displaystyle=\sum_{j}p_{j\to i}\;\pi_{j}$ $$

Reusing notation for mixture probabilities, we have,

$$ $\displaystyle\pi_{0}^{mix}$ $\displaystyle=\frac{2}{3}\pi_{0}+\frac{1}{3}\pi_{1}$ $\displaystyle\pi_{1}^{mix}$ $\displaystyle=\frac{1}{3}\pi_{0}+\frac{2}{3}\pi_{1}$ $\displaystyle\pi_{2}^{mix}$ $\displaystyle=\frac{5}{6}\pi_{2}+\frac{1}{6}\pi_{3}$ $\displaystyle\pi_{3}^{mix}$ $\displaystyle=\frac{1}{6}\pi_{2}+\frac{5}{6}\pi_{3}$ $\displaystyle\pi_{4}^{mix}$ $\displaystyle=\pi_{4}$ $$

#### D.2 Maze Navigation

The layout and dynamics of the maze follow Fu et al. (), but since their original design aims to train a single agent to reach a fixed goal from multiple start locations, we modified it to have both start and goal locations fixed in each task, as in Nam et al. (). The start location is still perturbed with a small noise to avoid memorizing the task. The observation consists of the agent’s current position and velocity. But, it lacks the goal location, which should be inferred from the dense reward based on the distance to the goal. The action space is the target 2D velocity of the point mass agent.

The layout we used is LARGE_MAZE which is an 8$\times$11 maze with paths blocked by walls. The complete set of 10 tasks is visualized in Figure 12, where green and red spots correspond to the start and goal locations, respectively. The environment provides an agent a dense reward of $\exp(-dist)$ where $dist$ is a linear distance between the agent’s current position and the goal location. It also gives a penalty of 1 at each timestep in order to prevent the agent from exploiting the reward by staying near the goal. The episode terminates either as soon as the goal is reached by having $dist<0.5$ or when 600 timesteps have passed.

Figure: Figure 12: Ten tasks defined for the Maze Navigation. The start and goal locations in each task are shown in green and red spots, respectively, and an example path is shown in green.
Refer to caption: x25.png

#### D.3 Meta-World Manipulation

For Meta-World CDS, we reproduce the Meta-world environment proposed by  Yu et al. () using the Meta-world codebase  (Yu et al., ), where the door and drawer are both placed side-by-side on the tabletop for all tasks (see Figure 6(c)). The observation space consists of the robot’s proprioceptive state, the drawer handle state, the door handle state, and the goal location, which varies based on the task. Unlike  Yu et al. (), we additionally remove the previous state from the observation space so the policies cannot easily infer the current task, making it a challenging multi-task environment. The environment also uses the default Meta-World reward functions which is composed of two distance-based rewards: distance between the agent end effector and the object, and distance between the object and its goal location. We use this modified environment instead of the Meta-world benchmark because our problem formulation of simultaneous multi-task RL requires a consistent environment across tasks. For Meta-World MT10, we directly use the implementation provided in  (Yu et al., ) without changes.

In both cases, the observation space consists of the robot’s proprioceptive state, locations for objects present in the environment (ie. door and drawer handle for CDS, the single target object location for MT10) and the goal location. In Meta-World CDS, due to the shared environment, there are no directly conflicting task behaviors, since the policies either go to the door or the drawer, they should ignore the irrelevant behaviors of policies interacting with the other object. In Meta-World MT10, each task interacts with a different object but in an overlapping state space so there is a mix of shared and conflicting behaviors.

#### D.4 Walker2D

Walker2D is a 9 DoF bipedal walker agent with the multi-task set of 4 tasks proposed and implemented by  Lee et al. (): walking forward at a target velocity, walking backward at a target velocity, balancing under random external forces, and crawling under a ceiling. Each of these tasks involves different gaits or body positions to accomplish successfully without any obviously identifiable shared behavior in the optimal policies.
Behavior sharing can still be effective during training to aid exploration and share helpful intermediate behaviors, like balancing. However, there is no obviously identifiable conflicting behavior either in this task set. Because each task requires a different gait, it is unlikely for states to recur between tasks and even less likely for states that are shared to require conflicting behaviors.
For instance, it is common for all policies to struggle and fall at the beginning of training, but all tasks would require similar stabilizing and correcting behavior over these states.

#### D.5 Kitchen

We modify the Franka Kitchen environment proposed by  Gupta et al. () and based on the implementation from Fu et al. (). Since this environment is typically used for long horizon or offline RL, we chose shorter tasks that are learnable with online RL. Furthermore, we added a dense reward based on the Meta-World reward function. We formed our 10 task MTRL set by choosing 10 available tasks in the kitchen environment that interacted with the same objects: turning the top burner on or off, turning the bottom burner on or off, turning the light switch on and off, open or closing the sliding cabinet, and opening and closing the hinge cabinet. The observation space consists of the robot’s state, the location of the target object, and the goal location for that object. Similar to the Meta-World CDS environment, these tasks may share behaviors navigating around the kitchen to the target object but have plenty of irrelevant behavior between tasks that interact with different objects and conflicting behaviors when opening or closing the same object.

Figure: Figure 8: Success rates for individual tasks in Multistage Reacher. Our method especially helps in learning Task 3, which requires extra exploration because it only receives a sparse reward.
Refer to caption: x26.png

Figure: Figure 9: Combining QMP with PCGrad yields complementary improvement in 3 out of the 4 environments we tested on. Dashed lines are PCGrad only and solid lines are QMP + PCGrad.
Refer to caption: x32.png

### Appendix E Additional Results

#### E.1 Multistage Reacher Per Task Results

Additional results and analysis on Multistage Reacher are shown in Figure 8.
QMP outperforms all the baselines in this task set, as shown in Figure 8. Task 3 receives only a sparse reward and, thus, can benefit the most from shared exploration. We observe that QMP gains the most performance boost due to selective behavior-sharing in Task 3. The No-Shared-Behavior baseline is unable to solve Task 3 at all due to its sparse reward nature. The other baselines which share uniformly suffer at Task 3, likely because they also share behaviors from other conflicting tasks, especially Task 4. We explore this further in the following Section F.

For all tasks, QMP outperforms or is comparable to No-Shared-Behavior, which shows that selective behavior-sharing can help accelerate learning when task behaviors are shareable and is robust when tasks conflict. Fully-Shared-Behavior especially underperforms in Tasks 2 and 3, which require conflicting behaviors upon reaching Subgoal 1, as defined in Table 1. In contrast, it excels at the beginning of Task 0 and Task 1 as their required behaviors are completely shared. However, it suffers after Subgoal 2, as the task objectives diverge.

Figure: (a) Maze Navigation 3 Tasks
Refer to caption: x36.png

Figure: (a) MCAL Comparison
Refer to caption: x39.png

#### E.2 Data Sharing Results

In Figure  10(c), we report multiple sharing percentiles for UDS and for CDS  (Yu et al., ) which assumes access to ground truth task reward functions which it uses to re-label the shared data. When the shared data is relabeled with task reward functions, thereby bypassing the conflicting behavior problem, online data sharing approaches can work very well. But when unsupervised, we see that online data sharing can actually harm performance in environments with conflicting tasks, with the more conservative data sharing approach (UDS k=80) out-performing sharing all data. $k$ is the percentile above with we share a transition between tasks, with higher $k$ representing more conservative data sharing. Full details on our online UDS and CDS implementation are in Section H.6

#### E.3 PCGrad Results

We evaluate whether QMP combined with PCGrad (Yu et al., ) results in complementary benefits. PCGrad is a popular MTRL algorithm that learns a policy with shared parameters and alleviates negative interference between tasks by modifying the multi-task gradients. We see in Figure 9 that QMP + PCGrad significantly improves PCGrad performance in 3 out of 4 environments.

#### E.4 QMP Scales with Task Set Size in Maze Navigation

We look at the behavior sharing methods in the Maze Navigation task for a task set with 3 tasks (Figure  10(a)) and 10 tasks (Figure  10(b)) and see that QMP scales well from 3 to 10 tasks, even compared to other behavior sharing methods. Similar to Meta-World, we hypothesize QMP scales better with a larger task set size of similar tasks due to there being more shareable behaviors between tasks. We see that by selectively sharing behaviors, QMP is able to identify and share helpful behaviors in the larger tasks sets whereas other behavior sharing methods struggle.

#### E.5 Additional Comparisons

Multi-Critic Actor Learning (MCAL) (Mysore et al., ) is a parameter sharing MTRL method that aims to tackle potential negative interference between tasks by learning separate critics for each task while training a single multi-task actor. We add QMP to two variants of MCAL, Multi-Critic AL-MN which maintains separate networks for each critic and Multi-Critic AL-MH which uses a single multi-head network for the critic, in Multistage Reacher in Figure 11(a). We see that adding QMP provides around a 20% final success rate gain in both variants and is more sample efficient.

We also compare our method with DECAF (Glatt et al., ), a MTRL method which shared Q-functions between tasks instead of behavioral policies. DECAF learns task specific weights to linearly combine the task Q-functions which is used to train the task policy. In contrast, our method uses the task Q-function to evaluate different tasks’ policies to incorporate into the task’s behavioral policy. Our method only modifies the data collection process, not the RL objective, and does not have a learned component. In Multistage Reacher (Figure 11(b)) and Meta-World CDS (Figure 11(c)), we see that QMP outperforms DECAF by more that 20% final success rate.

Figure: Figure 12: In each case above, QMP with H-step rollouts of the behavioral policy (blue) performs no worse than QMP with 1-step rollouts (red), with the H-step rollouts helping significantly in some tasks. Additionally both versions of QMP outperform the No-QMP baseline.
Refer to caption: x42.png

#### E.6 Temporally-Extended Behavior Sharing

Figure: Figure 13: QMP consistently improves performance as $H$ increases in Maze.
Refer to caption: x46.png

Motivated by prior work in heirarchical RL (Machado et al., ; Jinnai et al., ; ; Hansen et al., ; Zhang et al., ) and skill learning (Pertsch et al., ) , we explore temporally extended behavior sharing by simply following the actions of the policy $\pi_{j}$ selected by $\pi_{mix}^{i}$ for $H$ steps before re-evaluating $\pi_{mix}^{i}$. Furthermore, a recent work  Xu et al. () provides theoretical results that shows myopic ($\epsilon$-greedy) policy sharing can be sample efficient in sufficiently diverse multi-task settings, providing theoretical support for temporally extended multi-task behavior sharing in some settings.

We study the effect of sharing temporally extended behaviors of length $H$ in Maze Navigation in Figure 13, by rolling out the chosen task policy for 1, 5, 10, 25, and 50 timesteps. We see that performance improves when sharing longer behaviors (25 and 50 timesteps) which are more coherent and temporally extended. This is true even though we choose the behavioral policy greedily, only evaluating the current state $s$ every $H$ steps. Importantly, the guarantees from Theorem C.1 do not extend to $H$-step policy roll-outs and increasing $H$ does not help in all environments. We compare the performance of No-QMP, QMP, and QMP with temporally extended behavior sharing where we choose the best performance out of $H=10$ and $H=25$ in Table 2 and Figure 12. Nevertheless, the impressive results in Maze suggest that multi-task temporally extended behavior sharing is worth exploring in future work.

**Table 2: Temporally Extended Behavior Sharing**
| Environment | H-value | No-QMP | QMP | QMP (H¿1) |
| --- | --- | --- | --- | --- |
| Reacher | 10 | 80 ± 0 | 100 ± 0 | 100 ± 0 |
| Maze | 25 | 57.9 ± 0.09 | 72.9 ± 0.1 | 99.9 ± 0.0 |
| MT-CDS | 10 | 97.5 ± 4.5 | 93.7 ± 8.5 | 98.8 ± 2.0 |
| MT10 | 10 | 79.1 ± 5.97 | 89.0 ± 0.01 | 82. ± 4.48 |
| Kitchen | 10 | 65.5 ± 11.0 | 77.3 ± 5.3 | 84.5 ± 8.7 |
| Walker | 10 | 3110 ± 220 | 3205 ± 218 | 3310 ± 203 |

### Appendix F QMP Behavior Sharing Analysis

Figure: Figure 14: Mixture probabilities per task of other policies over the course of training for Multistage Reacher. The conflicting task Policy 4, which requires staying stationary, is highlighted in red.
Refer to caption: x47.png

Figure: Figure 15: Average contribution of each Policy $j$ (col $j$) in each Task $i$’s (row $i$) data collection on Reacher Multistage (diagonal zeroed for contrast).
Refer to caption: x53.png

QMP learns to not share from conflicting tasks: We visualize the mixture probabilities per task of other policies in Figure  14 for Multistage Reacher, highlighting the conflicting Task 4 in red. Throughout training, we see that QMP learns to share less behavior from Policy 4 than other policies in Tasks 0-3 and shares the least total cross-task behavior in Task 4. This supports our claim that the Q-switch can identify conflicting behaviors that should not be shared. We note that Task 3 has a relatively larger amount of sharing than other tasks. Since Task 3 has sparse rewards, it benefits the most from exploration via selective behavior-sharing from other tasks.

Figure 15 analyzes the effectiveness of the Q-switch in identifying shareable behaviors by visualizing the average proportion that each task policy is selected for another task over the course of training in the Multistage Reacher environment. This average mixture composition statistic intuitively analyzes whether QMP identifies shareable behaviors between similar tasks and avoids behavior sharing between conflicting or irrelevant tasks. As we expect, the Q-switch for Task 4 utilizes the least behavior from other policies (bottom row), and Policy 4 shares the least with other tasks (rightmost column). Since the agent at Task 4 is rewarded to stay at its initial position, this behavior conflicts with all the other goal-reaching tasks. Of the remaining tasks, Task 0 and 1 share the most similar goal sequence, so it is intuitive why they benefit from shared exploration and are often selected by their respective Q-switches. Finally, unlike the other tasks, Task 3 receives only a sparse reward and therefore relies heavily on shared exploration. In fact, QMP demonstrates the greatest advantage in this task (Appendix Figure 8).

Behavior-sharing reduces over training: Figure 14 shows that the total amount of behavior-sharing decreases over the course of training in all tasks, which demonstrates a naturally arising preference in the Q-switch for the task-specific policy as it becomes more proficient at its own task.

#### F.1 Qualitative Visualization of Behavior-Sharing

We qualitatively analyze behavior sharing by visualizing a rollout of QMP during training for the Drawer Open task in Meta-World Manipulation (Figure  9(b)). To generate this visualization, we use a QMP rollout during training before the policy converges to see how behaviors are shared and aid learning. For clarity, we first subsample the episodes timesteps by 10 and only report timesteps when the activated policy changes to a new one (ie. from timestep 80 to 110, QMP chose the Drawer Open policy). We qualitatively break down the episode into when the agent is approaching the drawer (top row; Steps 1-60), grasping the handle (top row; Steps 61-80), and pulling the drawer open (bottom row). This allows us to see that it switches between all task policies as it approaches the drawer, uses drawer-specific policies as it grasps the handle, and opening-specific policies as it pulls the drawer open. This suggests that in addition to ignoring conflicting behaviors, QMP is able to identify helpful behaviors to share. We note that QMP is not perfect at policy selection throughout the entire rollout, and it is also hard to interpret these shared behaviors exactly because the policies themselves are only partially trained, as this rollout is from the middle of training. However, in conjunction with the overall results and analysis, this supports our claim that QMP can effectively identify shareable behaviors between tasks.

### Appendix G Additional Ablations and Analysis

#### G.1 Probabilistic Mixture v/s Arg-Max

A probabilistic mixture of policies is a design choice of our approach where arg-max is replaced with softmax. However, in our initial experiments, we found no significant improvement in performance and it came with an additional hyperparameter of tuning the temperature coefficient. As we see in Figure 16(a), QMP actually outperforms a probabilistic mixture over a range of softmax temperatures, justifying the design choice of argmax over softmax due to its reliable performance and simplicity.

Figure: (a) Probabilistic Mixture Ablation
Refer to caption: x54.png

#### G.2 Approximation Expected Q-value Over Policy Action Distribution

QMP’s behavior policy is defined as $\pi_{i}^{\text{mix}}=\underset{\pi_{j}\in\{\pi_{1},\dots,\pi_{N}\}}{\arg\max}
\;\mathbb{E}_{a\sim\pi_{j}(s)}Q_{i}(s,a)$, which picks the task policy with the best expected Q value over its action distribution. We approximate the expectation by evaluating the Q-value of only the mean of each policy’s action distribution which is computationally cheaper $\pi_{i}^{\text{mix}}\approx\underset{\pi_{j}\in\{\pi_{1},\dots,\pi_{N}\}}{\arg
\max}\;Q_{i}(s,\mathbb{E}_{a\sim\pi_{j}(s)}[a])$. We compare this to a empirical estimate that samples 10 actions from the policy distribution and picks the policy with highest average Q-value in Figure 16(b), and find no significant performance difference between the two approximations. This validates that our simple approximation works well in practice, which we hypothesize is due to the low variance of the task policies.

#### G.3 QMP v/s Increasing Single Task Exploration

Since QMP seeks to gather more informative training data for the task by modifying the behavioral policy, it can be viewed as a form of multi-task exploration. We briefly investigate how single task exploration differs from multi-task exploration by tuning the target entropy in SAC in Figure 16(c) which influences the policy entropy and therefore exploration. We see that while tuning this exploration parameter affects the sample efficiency by more quickly learning each individual task, QMP achieves a higher final success rate by incorporating behaviors form other tasks, and therefore doing multi-task exploration. The benefit of exploration or behavior sharing algorithms specialized for multi-task RL is precisely this ability to transfer and share information between tasks.

#### G.4 QMP Runtime

While QMP does require more policy and q-function evaluations to sample from $\pi_{mix}^{i}$ in comparison to the base RL method, these evaluations can be greatly parallelized and do not significantly increase runtime (see Figure 3) for average runtimes for our experiments). Each sample from $\pi_{mix}^{i}$ requires querying $N$ policy proposals and $N$ Q-values. In QMP + Parameter-Sharing, thanks to the multihead architectures of the policy and Q-networks, all N evaluations are done in one single pass. Thus, with two passes through neural networks, we can get N action candidates and their N Q-values. Therefore, the increase in time is negligible. Even without parameter-sharing, $Q_{i}(s,a_{j})$ evaluations can be batched $\forall j$ and the policy evaluations $\pi_{j}(a_{j}|s)$ are all independent, and can be obtained in parallel. In our implementation, we batch the Q evaluations, but do not parallelize the policy evaluations.

**Table 3: Runtime Comparison**
| Environment | No-Sharing | QMP + | Parameter-Sharing | QMP + |
| --- | --- | --- | --- | --- |
|  | No-Sharing |  | Parameter-Sharing |  |
| Reacher Multistage | 12.5 hr | 14.2 hr | 14 hr | 16.2 hr |
| MT50 | – | – | 7 days, 3hr | 7 days, 6 hr |

### Appendix H Implementation Details

The SAC implementation we used in all our experiments is based on the open-source implementation from Garage (garage contributors, ). We used fully connected layers for the policies and Q-functions with the default hyperparameters listed in Table  4. For DnC baselines, we reproduced the method in Garage to the best of our ability with minimal modifications.

We used PyTorch (Paszke et al., ) for our implementation. We run the experiments primarily on machines with either NVIDIA GeForce RTX 2080 Ti or RTX 3090. Most experiments take around one day or less on an RTX 3090 to run. We use the Weights & Biases tool (Biewald, ) for logging and tracking experiments. All the environments were developed using the OpenAI Gym interface (Brockman et al., ).

#### H.1 Hyperparameters

Table 4 details the list of important hyperparameters on all the 3 environments. For all environments, we used a 2 layer fully connected network with hidden dimension 256 and a tanh activation function for the policies and Q functions. We use a target network for the Q function with target update $\tau=0.995$ and trained with an RL discount of $\gamma=0.99$.

**Table 4: Hyperparameters.**
| Hyperparameter | Multistage | Maze | Meta-World |
| --- | --- | --- | --- |
| Reacher | Navigation | CDS |  |
| Minimum buffer size (per task) | 10000 | 3000 | 10000 |
| # Environment steps per update (per task) | 1000 | 600 | 500 |
| # Gradient steps per update (per task) | 100 | 100 | 50 |
| Batch size | 32 | 256 | 256 |
| Learning rates for $\pi$, $Q$ and $\alpha$ | 0.0003 | 0.0003 | 0.0015 |

#### H.2 No-Shared-Behaviors

All $N$ networks have the same architecture with the hyperparameters presented in Table  4.

#### H.3 Fully-Shared-Behaviors

Since it is the only model with a single policy, we increased the number of parameters in the network to match others and tuned the learning rate. The hidden dimension of each layer is 600 in Multistage Reacher, 834 in Maze Navigation, and 512 in Meta-World Manipulation, and we kept the number of layers at 2. The number of environment steps as well as the number of gradient steps per update were increased by $N$ times so that the total number of steps could match those in other models. For the learning rate, we tried 4 different values (0.0003, 0.0005, 0.001, 0.0015) and chose the most performant one. The actual learning rate used for each experiment is 0.0003 in Multistage Reacher and Maze Navigation, and 0.001 in Meta-World Manipulation.

This modification also applies to the Shared Multihead baseline, but with separate tuning for the network size and learning rates. In Multistage Reacher, we used layers with hidden dimensions of 512 and 0.001 as the final learning rate. In Maze Navigation, we used 834 for hidden dimensions and 0.0003 for the learning rate.

#### H.4 DnC

We used the same hyperparameters as in Separated, while the policy distillation parameters and the regularization coefficients were manually tuned. Following the settings in the original DnC (Ghosh et al., ), we adjusted the period of policy distillation to have 10 distillations over the course of training. The number of distillation epochs was set to 500 to ensure that the distillation is completed. The regularization coefficients were searched among 5 values (0.0001, 0.001, 0.01, 0.1, 1), and we chose the best one. Note that this search was done separately for DnC and DnC with regularization only. For DnC, the coefficients we used are: 0.001 in Multistage Reacher and Maze Navigation, and 0.001 in Meta-World Manipulation. For Dnc with regularization only, the values are: 0.001 in Multistage Reacher, 0.0001 in Maze Navigation, and 0.001 in Meta-World Manipulation.

#### H.5 QMP (Ours)

Our method also uses the default hyperparameters. QMP does not require any task specific hyperparameters. The exception is Meta-World MT10, where we found it helpful to have more conservative behavior sharing by choosing the task-specific policy 70% of the time. The remaining 30% we use the Q-filter to select a policy as usual.

Like in Baseline Multihead (Parameters-Only), the QMP Multihead architecture (Parameters+Behaviors) also required a separate tuning. Since QMP Multihead effectively has one network, we increased the network size in accordance with Baseline Multihead and tuned the learning rate in addition to the mixture warmup period. The best-performing combinations of these parameters we found are 0 and 0.001 in Multistage Reacher, and 100 and 0.0003 in Maze Navigation, respectively.

#### H.6 Online UDS

Yu et al. () proposes an offline multi-task RL method (UDS) that shares data between tasks if their conservative Q value falls above the $k^{th}$ percentile of the task data. Specifically, before training, you would go through all the tasks’ data and share some data from Task $j$ to Task $i$ if the Task $i$ Q value of that data is greater than $k\%$ of the Q values of Task $i$’s data. UDS does not require access to task reward functions like other data-sharing approaches. It simply re-labels any shared data with the minimum task reward, making it applicable to our problem setting as we also do not assume that reward relabeling is possible.

In order to adapt UDS to online RL, instead of doing data sharing once on the given multi-task dataset, we apply UDS data sharing before every training iteration to the data in the multi-task replay buffers. Concretely, we implement this on-the-fly for every batch of sampled data by sampling one batch of data from Task $i$’s replay buffer, $\beta_{i}$, and one batch of data from the other task’s replay buffers $\beta_{j\neq i}$. Then following UDS, we would form the effective batch $\beta_{i}^{\text{eff}}$ by sharing data from $\beta_{j\neq i}$ if it falls above the $k^{th}$ percentile of Q values for $\beta_{i}$:

$$ $\displaystyle UDS_{\text{online}}:(s,a,r_{i},s^{\prime})\sim\beta_{j\neq i}\in \beta_{i}^{\text{eff}}$ $\displaystyle\text{ if }\Delta^{\pi}(s,a):=\hat{Q}^{\pi}(s,a,i)-P_{k^{\text{th }}}[\hat{Q}^{\pi}(s^{\prime},a^{\prime},i):s^{\prime},a^{\prime}\sim\beta_{i}]\geq 0$ $$

Note the differences here: (i) the ‘data’ used for data-sharing is the sampled replay buffer batch instead of the offline dataset, and (ii) we use the standard Q-function to evaluate data instead of the conservative Q-function since we are doing online (not offline) RL. We implement it this way as a practical approximation to avoid having to process the entire replay buffer every training iteration.

We use the same default hyperparameters as the other baseline methods. Additionally, we need to tune the sharing percentile $k$. For this, we tried $0^{\text{th}}$ percentile (sharing all data) and $80^{\text{th}}$ percentile, and chose the best-performing one.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: See above.
soundness: 2
strengths: - The paper is well-written with clear organization and easy to follow.

- The method is elegantly simple yet theoretically sound, introducing behavior sharing through off-policy data collection rather than policy regularization, which avoids bias in the learning objective while maintaining convergence guarantees.

- The proposed Q-switch mechanism provides a principled approach to selective behavior sharing that is complementary to existing MTRL methods, demonstrating consistent improvements when combined with parameter sharing, data sharing, and gradient-based approaches.
summary: This paper introduces Q-switch Mixture of Policies (QMP), a novel framework for multi-task reinforcement learning that improves sample efficiency through selective behavior sharing between tasks. The key idea is to enhance each task's off-policy data collection by selectively employing behaviors from other task policies, using the task's Q-function to evaluate and select useful shareable behaviors.
weaknesses: 1. The Q-switch mechanism requires evaluating all task policies and Q-values at each step. While parallelization helps, this still leads to significant computational costs, especially for large task sets as evidenced by the 7+ days runtime in MT50 experiments.
2. The method heavily relies on accurate Q-function estimation to select appropriate behaviors. This dependency may lead to suboptimal behavior selection during early training or in tasks with sparse rewards where Q-function learning is unstable.

3. The method requires careful tuning of hyperparameters, such as manually setting 70% task-specific policy usage in Meta-World MT10. This tuning requirement may limit practical applicability.

4. The method's effectiveness heavily depends on the existence of shareable behaviors between tasks. Performance gains might be limited for completely unrelated or highly conflicting task sets.

5. The convergence analysis primarily focuses on tabular MDPs. The theoretical guarantees may not fully extend to continuous state-action spaces with function approximation.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Could you provide some information regarding the computational overhead of QMP?
soundness: 3
strengths: 1. QMP is designed to complement existing MTRL frameworks, such as parameter sharing and data relabeling. 
2. The authors provide theoretical analysis showing that QMP’s behavior-sharing mechanism preserves the convergence guarantees of the underlying reinforcement learning algorithm.
3. The paper presents extensive experiments across various multi-task environments (e.g., manipulation, navigation, and locomotion).
summary: This paper introduces the Q-switch Mixture of Policies (QMP), a framework for multi-task reinforcement learning (MTRL) aimed at enhancing sample efficiency by sharing behaviors selectively across tasks. The approach enables an agent to leverage useful behaviors from other tasks during off-policy data collection, guided by the Q-function of the current task to identify beneficial actions. Unlike traditional MTRL methods that rely on uniform sharing or regularization, QMP avoids biasing policy objectives by only sharing behaviors during data collection rather than directly influencing policy updates. Experimental results across various environments, including manipulation and locomotion tasks, demonstrate QMP's capability to accelerate training and improve performance when integrated with existing MTRL frameworks.
weaknesses: 1. QMP requires evaluating Q-values across multiple task policies at each decision step, which could introduce computational overhead, particularly in settings with a large number of tasks. The paper lacks a detailed comparison of computational costs between QMP and baseline methods, which would be helpful for assessing its scalability.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see previous section.
soundness: 2
strengths: 1. The QMP framework introduces a Q-value-based behavior selection mechanism that enables selective behavior sharing in multi-task reinforcement learning (MTRL), enhancing sample efficiency.
2. The extensive experimental results across diverse domains—manipulation, navigation, and locomotion—demonstrate QMP’s performance compared to traditional behavior-sharing baselines, showcasing its practical impact in different tasks.
summary: The paper introduces QMP (Q-Switch Mixture of Policies), a novel framework for multi-task reinforcement learning (MTRL) designed to improve sample efficiency by selectively sharing behavioral policies across different tasks. Unlike traditional MTRL methods, which rely on parameter sharing or uniform behavior sharing, QMP identifies and shares beneficial behaviors from other tasks based on a Q-function evaluation. This approach enables each task to benefit from the learning progress in other tasks without introducing bias, as only helpful behaviors are integrated into the data collection phase.
weaknesses: 1. This paper only proposes a multi-task data sampling strategy implemented within an off-policy framework, which is limited in scope and shows limited improvement in more complex tasks, such as MT10 and MT50. 
2. The other algorithms in paper [1] achieved better results with fewer interactions with the environment during training. I think that the method proposed in this paper makes a limited contribution to the field of multi-task reinforcement learning.
3. It is unclear how to ensure avoidance of local optima during complex multi-task transfer processes.
4. There is a lack of comparison with other MTRL baselines.

[1] Sodhani et al., 2021, Multi-Task Reinforcement Learning with Context-based Representations

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 2
strengths: The paper is clearly written with a logical flow. The introduction and problem formulation effectively motivate the need for a new behavior-sharing method, and the diagrams provided help visualize the approach. The results are promising.
summary: This paper proposes a framework for multi-task reinforcement learning (MTRL) called Q-switch Mixture of Policies (QMP), which enables the selective sharing of behaviors between tasks to improve sample efficiency. QMP enhances off-policy data collection by selecting useful behaviors from other task policies based on the task's Q-function. The authors provide theoretical guarantees that QMP improves sample efficiency while preserving the convergence guarantees of reinforcement learning algorithms. Empirical evaluations demonstrate that QMP achieves complementary gains over existing MTRL algorithms in manipulation, locomotion, and navigation environments.
weaknesses: My impression of this paper is mixed. On the one hand, the authors demonstrate that QMP is effective. However, on the other hand, it's not intuitively clear why QMP works as well as it does. The proposed method is quite simple: it uses the current $Q_i$ function to select the argmax policy $\pi_j^{mix}$. Then, $\pi_j^{mix}$ (rather than $\pi_i$) is used to generate one-step data for training. I have some concerns about the approach of naively taking the maximum over per-task Q-functions—this may not be suitable in general settings, such as more stochastic environments, as it could lead to issues like switching tasks at every step or favoring certain tasks while neglecting others, which is undesirable. If this paper is accepted, it might be perceived as presenting a general framework for MTRL, but I have serious doubts about this claim.

In the introduction section of the paper, it mentions 'These tasks share many similar behaviors, like approaching the tabletop or grasping the object handle.' Therefore, I think 'behavior' refers to an action sequence or subtask that continues for a period of time. However, Algorithm 1 (one-step action sharing) appears to be misaligned with the paper's title and motivation, which emphasizes behavior (action sequence) sharing. The paper's motivation requires significant revision to address this discrepancy.

Meta-review: This paper introduces QMP (Q-switch Mixture of Policies), a novel framework for sharing behaviors across tasks in multi-task reinforcement learning through off-policy data collection. The reviewers appreciated the paper's clear writing and theoretical soundness, particularly in demonstrating how QMP complements existing MTRL methods. Through extensive revisions, the authors effectively addressed initial concerns about computational overhead (showing only 1.75% overhead for 22.8% performance gain), clarified the stability of QMP's maximization objective, and better positioned the work within the off-policy MTRL context. While some reviewers initially expressed concerns about QMP's generality beyond grid worlds and its reliance on Q-function accuracy, the authors provided convincing theoretical and empirical evidence for QMP's robustness. Moving forward, exploring QMP's behavior sharing benefits with more complex task relationships would be valuable. 

Given the paper's solid theoretical foundation, thorough empirical validation across diverse environments, and demonstrated complementary gains to existing MTRL approaches, I recommend accepting this work.

Decision: Accept (Poster)