QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing

Authors: Grace Zhang, Ayush Jain, Injune Hwang, Shao-Hua Sun, Joseph J Lim
Primary area: reinforcement learning
Keywords: Multi-task Reinforcement Learning, Behavior Sharing

Abstract:
Multi-task reinforcement learning (MTRL) aims to learn several tasks simultaneously for better sample efficiency than learning them separately. Traditional methods achieve this by sharing parameters or relabeling data between tasks.  In this work, we introduce a new framework for sharing behavioral policies across tasks, which can be used in addition to existing MTRL methods. The key idea is to improve each task's off-policy data collection by employing behaviors from other task policies. Selectively sharing helpful behaviors acquired in one task to collect training data for another task can lead to higher-quality trajectories, leading to more sample-efficient MTRL. Thus, we introduce a simple and principled framework called Q-switch mixture of policies (QMP) that selectively shares behavior between different task policies by using the task's Q-function to evaluate and select useful shareable behaviors.  We theoretically analyze how QMP improves the sample efficiency of the underlying RL algorithm.  Our experiments show that QMP's behavioral policy sharing provides complementary gains over many popular MTRL algorithms and outperforms alternative ways to share behaviors in various manipulation, locomotion, and navigation environments. Videos are available at https://qmp-mtrl.github.io/.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729604693149

thread_type:
official_review

<thread_object>
creator: reviewer_cusu

<content>
summary:
This paper proposes a framework for multi-task reinforcement learning (MTRL) called Q-switch Mixture of Policies (QMP), which enables the selective sharing of behaviors between tasks to improve sample efficiency. QMP enhances off-policy data collection by selecting useful behaviors from other task policies based on the task's Q-function. The authors provide theoretical guarantees that QMP improves sample efficiency while preserving the convergence guarantees of reinforcement learning algorithms. Empirical evaluations demonstrate that QMP achieves complementary gains over existing MTRL algorithms in manipulation, locomotion, and navigation environments.

strengths:
The paper is clearly written with a logical flow. The introduction and problem formulation effectively motivate the need for a new behavior-sharing method, and the diagrams provided help visualize the approach. The results are promising.

weaknesses:
My impression of this paper is mixed. On the one hand, the authors demonstrate that QMP is effective. However, on the other hand, it's not intuitively clear why QMP works as well as it does. The proposed method is quite simple: it uses the current $Q_i$ function to select the argmax policy $\pi_j^{mix}$. Then, $\pi_j^{mix}$ (rather than $\pi_i$) is used to generate one-step data for training. I have some concerns about the approach of naively taking the maximum over per-task Q-functions—this may not be suitable in general settings, such as more stochastic environments, as it could lead to issues like switching tasks at every step or favoring certain tasks while neglecting others, which is undesirable. If this paper is accepted, it might be perceived as presenting a general framework for MTRL, but I have serious doubts about this claim.

In the introduction section of the paper, it mentions 'These tasks share many similar behaviors, like approaching the tabletop or grasping the object handle.' Therefore, I think 'behavior' refers to an action sequence or subtask that continues for a period of time. However, Algorithm 1 (one-step action sharing) appears to be misaligned with the paper's title and motivation, which emphasizes behavior (action sequence) sharing. The paper's motivation requires significant revision to address this discrepancy.

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730393715129

thread_type:
official_review

<thread_object>
creator: reviewer_7cb7

<content>
summary:
The paper introduces QMP (Q-Switch Mixture of Policies), a novel framework for multi-task reinforcement learning (MTRL) designed to improve sample efficiency by selectively sharing behavioral policies across different tasks. Unlike traditional MTRL methods, which rely on parameter sharing or uniform behavior sharing, QMP identifies and shares beneficial behaviors from other tasks based on a Q-function evaluation. This approach enables each task to benefit from the learning progress in other tasks without introducing bias, as only helpful behaviors are integrated into the data collection phase.

strengths:
1. The QMP framework introduces a Q-value-based behavior selection mechanism that enables selective behavior sharing in multi-task reinforcement learning (MTRL), enhancing sample efficiency.
2. The extensive experimental results across diverse domains—manipulation, navigation, and locomotion—demonstrate QMP’s performance compared to traditional behavior-sharing baselines, showcasing its practical impact in different tasks.

weaknesses:
1. This paper only proposes a multi-task data sampling strategy implemented within an off-policy framework, which is limited in scope and shows limited improvement in more complex tasks, such as MT10 and MT50. 
2. The other algorithms in paper [1] achieved better results with fewer interactions with the environment during training. I think that the method proposed in this paper makes a limited contribution to the field of multi-task reinforcement learning.
3. It is unclear how to ensure avoidance of local optima during complex multi-task transfer processes.
4. There is a lack of comparison with other MTRL baselines.

[1] Sodhani et al., 2021, Multi-Task Reinforcement Learning with Context-based Representations

questions:
Please see previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646901236

thread_type:
official_review

<thread_object>
creator: reviewer_encc

<content>
summary:
This paper introduces the Q-switch Mixture of Policies (QMP), a framework for multi-task reinforcement learning (MTRL) aimed at enhancing sample efficiency by sharing behaviors selectively across tasks. The approach enables an agent to leverage useful behaviors from other tasks during off-policy data collection, guided by the Q-function of the current task to identify beneficial actions. Unlike traditional MTRL methods that rely on uniform sharing or regularization, QMP avoids biasing policy objectives by only sharing behaviors during data collection rather than directly influencing policy updates. Experimental results across various environments, including manipulation and locomotion tasks, demonstrate QMP's capability to accelerate training and improve performance when integrated with existing MTRL frameworks.

strengths:
1. QMP is designed to complement existing MTRL frameworks, such as parameter sharing and data relabeling. 
2. The authors provide theoretical analysis showing that QMP’s behavior-sharing mechanism preserves the convergence guarantees of the underlying reinforcement learning algorithm.
3. The paper presents extensive experiments across various multi-task environments (e.g., manipulation, navigation, and locomotion).

weaknesses:
1. QMP requires evaluating Q-values across multiple task policies at each decision step, which could introduce computational overhead, particularly in settings with a large number of tasks. The paper lacks a detailed comparison of computational costs between QMP and baseline methods, which would be helpful for assessing its scalability.

questions:
1. Could you provide some information regarding the computational overhead of QMP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653870728

thread_type:
official_review

<thread_object>
creator: reviewer_mdzx

<content>
summary:
This paper introduces Q-switch Mixture of Policies (QMP), a novel framework for multi-task reinforcement learning that improves sample efficiency through selective behavior sharing between tasks. The key idea is to enhance each task's off-policy data collection by selectively employing behaviors from other task policies, using the task's Q-function to evaluate and select useful shareable behaviors.

strengths:
- The paper is well-written with clear organization and easy to follow.

- The method is elegantly simple yet theoretically sound, introducing behavior sharing through off-policy data collection rather than policy regularization, which avoids bias in the learning objective while maintaining convergence guarantees.

- The proposed Q-switch mechanism provides a principled approach to selective behavior sharing that is complementary to existing MTRL methods, demonstrating consistent improvements when combined with parameter sharing, data sharing, and gradient-based approaches.

weaknesses:
1. The Q-switch mechanism requires evaluating all task policies and Q-values at each step. While parallelization helps, this still leads to significant computational costs, especially for large task sets as evidenced by the 7+ days runtime in MT50 experiments.
2. The method heavily relies on accurate Q-function estimation to select appropriate behaviors. This dependency may lead to suboptimal behavior selection during early training or in tasks with sparse rewards where Q-function learning is unstable.

3. The method requires careful tuning of hyperparameters, such as manually setting 70% task-specific policy usage in Meta-World MT10. This tuning requirement may limit practical applicability.

4. The method's effectiveness heavily depends on the existence of shareable behaviors between tasks. Performance gains might be limited for completely unrelated or highly conflicting task sets.

5. The convergence analysis primarily focuses on tabular MDPs. The theoretical guarantees may not fully extend to continuous state-action spaces with function approximation.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732493132696

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal. Kindly requesting discussion or response.

comment:
We would appreciate the opportunity to discuss any remaining concerns with the reviewers. Following the valuable feedback from the reviewers, we have made several revisions and clarifications to address their concerns and improve the clarity and impact of our work.

We are pleased that the reviewers appreciated the *clarity* of our writing, the *novelty* and *simplicity* of our approach, its *theoretical* soundness, and the *extensive experiments* across diverse domains, which show promising results demonstrating how QMP is *complementary* to a variety of MTRL methods.

----

## Summary of revisions and clarifications

----

### 1. Negligible compute overhead of QMP: +1.75% time for +22.8% performance in MT50 (Reviewer MdZX, ENcC)

We clarify that QMP scales really well because its operations are parallelizable. In 50-task Metworld, QMP adds only a **1.75% time overhead** on the baseline due to a single extra Q-function pass (Table 4), yielding a significant performance improvement of +22.8% for a minimal increase in computation time.

----

### 2. QMP’s maximization objective is stable and consistent with GPI and SAC  (Reviewer MdZX, CusU)
- We clarify that QMP’s $\arg \max$ step is not unnatural and is consistent with the *fundamental objective underlying SAC*, as maximization is inherent in SAC’s formulation (see Eq. 1). While **SAC maximizes via gradient ascent**, QMP enhances (not replace) this by an **explicit maximization** over available policies. Consequently, Theorem D.2 theoretically proves that QMP converges at least as fast as SAC.
- Importantly, QMP **does not switch tasks** in between — the MDP (transition dynamics and reward function) remains fixed in one episode. QMP only switches the best policy to collect data for the current task at each decision point.
- By optimizing the generalized policy iteration (GPI) objective more effectively than SAC, QMP actually **improves robustness** to *Q-function estimation* (by quickly exploring and correcting), *stochastic environments*, *sparse reward* (Task 3 in Figure 13), and *conflicting tasks* (task 4 in Figure 13), because QMP accelerates the policy improvement step inherent to GPI.

----

### 3. Writing Revision: Emphasize *off-policy* MTRL as QMP’s scope and explicitly define *behavior* in motivation (Reviewer 7Cb7, CusU)

- **Scope**: In introduction and limitation, we explicitly note that QMP is a simple, off-policy, and hyperparameter-free plug-in that complements existing off-policy MTRL by modifying the data collection, and thus, not applicable to on-policy RL algorithms like PPO.

- **Behavior definition**: We revised the motivating paragraph in the introduction to explicitly define 'behavior' as the policy of how an agent acts in response to a particular state, aligning with established terminology in reinforcement and imitation learning, and eliminating any doubts that behavior-sharing might mean “multi-step action sharing” or “temporally-extended behavior sharing” (which is a separate heuristic extension in Appendix F.6)

----

### 4. Comparison to MTRL baselines: QMP complements various MTRL frameworks (Reviewer 7Cb7)
- We show QMP’s behavior-sharing **improves the performance when combined** with 4 frameworks of MTRL, including the best-performing methods from Sodhani et al. [1]:
1. No-Sharing: Base RL methods like SAC [3]
2. Data-Sharing: CDS [4], UDS[5]
3. Parameter-Sharing: Multi-headed SAC [6], DECAF [7]
4. Reducing gradient interference to **avoid local optima**: PCGrad [2]

For instance, on the best approach PCGrad, QMP still improves further!
| Approach | Reacher | Maze | Kitchen | MT 50 |  
|:------|:---------:|:----:|:------:|:------:|  
| PCGrad (dotted-orange) | **0.78** | 0.90 | 0.55 | 0.35 |  
| QMP \+ PCGrad (orange) | **0.78** | **1.00** | **0.60** | **0.42** |

---

- We also show that QMP’s off-policy behavior-sharing is better than **baseline** direct behavior sharing methods that bias each policy: Divide-and-Conquer [8] and full-sharing.

- Even as a standalone improvement, QMP’s gains are competitive to other ways to share information like via parameters or data. Crucially, QMP’s behavior-sharing gains are positive, orthogonal and complementary, so it is an easy-to-add and robust plug-in for off-policy MTRL.
| MTRL approach v/s Zero-MTRL | Reacher | Maze | MT-CDS| Kitchen | MT10 |  
|:----|:-----------:|:-----:|:-----:|:--------:|:------:|  
| Parameter-sharing (dotted-pink vs. dotted-blue) | \-35% | **\+50%** | **\+5%** | \+20% | \-5% |  
| Unsupervised data-sharing (dotted-green v/s dotted-blue) | \-50% | \-5% | \-10% | \+25% | **\+15%** |  
| QMP: Behavior-sharing (Solid blue v/s dotted-blue) | **\+25%** | \+15% | 0% | **\+50%** | **\+15%** |

[References in response to Reviewer 7Cb7]

----

We thank the reviewers for your valuable time and kindly request you to reconsider your scores if we have satisfactorily addressed your concerns. We are happy to incorporate any further feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces QMP (Q-switch Mixture of Policies), a novel framework for sharing behaviors across tasks in multi-task reinforcement learning through off-policy data collection. The reviewers appreciated the paper's clear writing and theoretical soundness, particularly in demonstrating how QMP complements existing MTRL methods. Through extensive revisions, the authors effectively addressed initial concerns about computational overhead (showing only 1.75% overhead for 22.8% performance gain), clarified the stability of QMP's maximization objective, and better positioned the work within the off-policy MTRL context. While some reviewers initially expressed concerns about QMP's generality beyond grid worlds and its reliance on Q-function accuracy, the authors provided convincing theoretical and empirical evidence for QMP's robustness. Moving forward, exploring QMP's behavior sharing benefits with more complex task relationships would be valuable. 

Given the paper's solid theoretical foundation, thorough empirical validation across diverse environments, and demonstrated complementary gains to existing MTRL approaches, I recommend accepting this work.