MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning

Authors: Suning Huang, Zheyu Aqa Zhang, Tianhai Liang, Yihan Xu, Zhehao Kou, Chenhao Lu, Guowei Xu, Zhengrong Xue, Huazhe Xu
Primary area: reinforcement learning
Keywords: Visual Reinforcement Learning, Robotics, Mixture-of-Experts

Abstract:
Visual deep reinforcement learning (RL) enables robots to acquire skills from visual input for unstructured tasks. However, current algorithms suffer from low sample efficiency, limiting their practical applicability. In this work, we present MENTOR, a method that improves both the architecture and optimization of RL agents. Specifically, MENTOR replaces the standard multi-layer perceptron (MLP) with a mixture-of-experts (MoE) backbone, enhancing the agent's ability to handle complex tasks by leveraging modular expert learning to avoid gradient conflicts. Furthermore, MENTOR introduces a task-oriented perturbation mechanism, which heuristically samples perturbation candidates containing task-relevant information, leading to more targeted and effective optimization. MENTOR outperforms state-of-the-art methods across three simulation domains---DeepMind Control Suite, Meta-World, and Adroit. Additionally, MENTOR achieves an average of 83% success rate on three challenging real-world robotic manipulation tasks including peg insertion, cable routing, and tabletop golf, which significantly surpasses the success rate of 32% from the current strongest model-free visual RL algorithm. These results underscore the importance of sample efficiency in advancing visual RL for real-world robotics. Experimental videos are available at https://mentor-vrl.github.io/.

Decision: reject
Reviewer scores: 5, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730325370938

thread_type:
official_review

<thread_object>
creator: reviewer_tuwr

<content>
summary:
This paper addresses the challenge of reinforcement learning with visual observations, where learning an efficient policy from high-dimensional image data is difficult. The authors propose a novel approach by incorporating a mixture-of-experts (MoE) architecture in the policy and applying task-oriented perturbation to optimize learning efficiency. The method, called MENTOR, is tested on several reinforcement learning benchmarks, including DeepMind Control Suite, Meta-World, and Adroit, as well as real-world experiments. MENTOR demonstrates superior performance compared to prior state-of-the-art (SOTA) methods.

strengths:
* The paper is clearly written and easy to follow.  
* The proposed approach—integrating a mixture-of-experts in the policy architecture and applying task-oriented perturbation—is well-motivated and empirically supported, as demonstrated in Figures 3 and 4\.  
* MENTOR shows significant empirical improvements over baseline methods in both simulated environments and real-world experiments.

weaknesses:
* The paper lacks a discussion of its limitations and possible future directions for addressing them.  
* Several clarifications could improve the writing and presentation of the work.  
* A more detailed analysis of hyperparameter sensitivity would be beneficial. It would be helpful to understand how MENTOR's performance is affected by hyperparameters such as the number of experts, the number of top-k experts, the perturbation rate, and the size of the set S\_{top}​.

questions:
1. **Ablation study**: If the method only used the MoE component and random perturbation (similar to DrM), what would the performance be? It would be valuable to analyze whether the mixture-of-experts or task-oriented perturbation contributes more to the success of MENTOR.  
2. **Task-oriented perturbation and self-imitation learning**: The task-oriented perturbation shares similar intuition with self-imitation learning (https://arxiv.org/abs/1806.05635), where agents benefit from their own past high-rewarding network weight or trajectories. Citing relevant work on self-imitation learning would strengthen the paper. Additionally, a discussion comparing the advantages and disadvantages of task-oriented perturbation versus self-imitation learning would enhance the contribution.  
3. **Expert output architecture (Line 199\)**: The paper mentions that expert i produces output a\_i​, but it is unclear how this output is derived from the latent vector z. Could you provide more details about the architecture of the feedforward network FFN\_i​ and its role in generating the expert output?  
4. **Clarification on MW (Line 215\)**: The paper refers to the "Assembly task from MW," but MW is not defined in the text. Does MW refer to Meta-World? A clear definition would improve readability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730434566167

thread_type:
official_review

<thread_object>
creator: reviewer_1tob

<content>
summary:
This paper proposes a sample-efficient visual reinforcement learning approach called MENTOR, which utilizes a mixture-of-experts network instead of the traditional MLP network to mitigate gradient conflicts, along with a task-oriented perturbation method to enhance exploration. Evaluation results in multiple simulation environments show that MENTOR is sample efficient. Further, MENTOR can be successfully used for real-world reinforcement learning, which facilitates the application of reinforcement learning to real-world scenarios.

strengths:
1) Attempts to alleviate the burden of shared parameters by introducing MoE architectures into reinforcement learning

2) A simple and effective perturbation method is proposed that can better guide the policy learning

3) The proposed method achieves an improvement in sample efficiency compared to DrM

4) Validates the effectiveness of the method on real-world robotics tasks, providing a valuable reference for the community

weaknesses:
1) Lack of persuasion and ablation in the use of MoE. MoE has been widely used in the field of multi-task learning, and it can effectively alleviate the conflict problem due to multi-objective optimization. However, policy optimization in a single robot manipulation task often has only one optimization objective, which does not fit the context of multi-task learning. Although it is claimed in the paper that the architecture advantage can be propagated to a single task to alleviate the burden of shared parameters, there is no further analysis and ablation experiments on this.

2) Lack of correlation between the two main improvements. MENTOR makes improvements in both architecture and optimization, yet there seems to be no necessary connection between the two. This makes the improvements in the paper appear as if they are just a combination of two tricks. In fact, optimization is often related to architecture, and it remains uncertain whether the use of MoE will introduce new challenges for policy optimization.

3) Lack of ablation of the two improvements. The paper only provides performance curves for MENTOR in simulation tasks, lacking ablation studies on architecture and optimization, which makes the reasons for the final performance improvement unclear. Although incremental comparisons are made in real-robot experiments, comparisons in simulation tasks will be more convincing and fairer.

questions:
1) Whether it can be shown that the multi-stage property of the task in single-task learning leads to the gradient conflict problem or the existence of a shared parameter burden in policy optimization?

2) Is MoE more prone to dormancy than MLP or can it mitigate dormancy to some extent?

3) In Figure 6, why MENTOR performs worse on hammer than on hammer (sparse)?

4) Does the performance improvement in Fig. 6 arise mainly from the task-oriented perturbations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656670827

thread_type:
official_review

<thread_object>
creator: reviewer_8sj6

<content>
summary:
The paper presents MENTOR, an innovative approach to enhance sample efficiency in visual deep reinforcement learning (RL) for robotics. By replacing the standard multi-layer perceptron with a mixture-of-experts (MoE) architecture and introducing a task-oriented perturbation mechanism, MENTOR improves the agent's performance in complex tasks and facilitates more effective optimization. The method demonstrates superior results across three simulation domains and achieves an impressive 83% success rate on challenging real-world robotic tasks, significantly outperforming the current best model-free visual RL algorithm, which only achieves 32%.

strengths:
1. MENTOR introduces a mixture-of-experts (MoE) architecture that enhances learning efficiency by dynamically allocating gradients to modular experts, effectively mitigating gradient conflicts in complex scenarios.
2. The evaluation extends beyond simulations to real-world robotic manipulation tasks, demonstrating MENTOR’s practical value and sample efficiency, which are crucial for advancing reinforcement learning applications in robotics.

weaknesses:
While MENTOR demonstrates impressive performance in both simulation and real-world tasks, the paper could benefit from a more detailed analysis of the limitations of the proposed approach, particularly in terms of scalability and generalization across diverse robotic platforms and environments. This would provide a clearer understanding of the framework's applicability in broader contexts.

questions:
1. Are the experimental results in the real-world obtained through sim2real transfer of models trained in simulation, or are they trained from scratch entirely in a real environment?  
2. Why are external disturbance experiments not conducted in the simulation environment?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672407924

thread_type:
official_review

<thread_object>
creator: reviewer_784n

<content>
summary:
The authors introduce MENTOR, a visual deep RL algorithm designed to improve sample efficiency in robotic tasks. MENTOR enhances RL agents by replacing traditional MLPs with a MoE architecture. Additionally, the authors introduce a task-oriented perturbation mechanism that heuristically samples task-relevant perturbations. Their experiments show MENTOR can get good performance over the diverse tasks.

strengths:
1. The paper is well-structured and easy to understand, with a clear presentation of the proposed method.
2. The authors conduct extensive experiments in both simulated and real environments, effectively demonstrating the method’s efficacy.

weaknesses:
1. The proposed MoE architecture is not evaluated over multi-task environments, especially ones that need different strategies for the different tasks in the environments.
2. The benefit of the MoE and the task-oriented exploration strategies are coupled. The authors need to decouple this two components and show the effectiveness of the MoE.
3. The authors need to compare with other techniques that can handle the multi-modality like transformers, diffusion-based policy.

questions:
The authors need to address my concerns in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731997741937

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their insightful comments and helpful suggestions. We are pleased that the reviewers find the proposed method **well-motivated and effective** (Reviewers 8sj6, 1toB, TuwR), and supported by **extensive experimental results** validating its efficacy in both simulation and **real-world RL** environments (Reviewers 784n, 8sj6, 1toB, TuwR).

We provide additional clarifications, explanations and discussion in the per-reviewer responses as well as our **rebuttal website** [here](https://sites.google.com/view/iclr2025mentor).

The main concerns from the reviewers focus on the lack of ablation studies to separately demonstrate the effectiveness of both the architectural (MoE) and optimization (Task-oriented Perturbation) improvements of MENTOR. To address reviewers' concern, we have conducted additional ablation studies on **five diverse tasks**: Hopper Hop, Disassemble, Coffee-Push (Sparse), Soccer (Sparse), and Hammer (Sparse). These studies aim to decouple the effects of the MoE architecture and the Task-oriented Perturbation (TP) mechanism proposed in our paper.

For the experiments, we evaluate **four ablated versions of MENTOR** using the same four random seeds as in the original experiments, as shown in rebuttal website Section 1:

* **MENTOR**: Full model with both MoE and Task-oriented Perturbation.
* **MENTOR_w/o_TP**: Task-oriented Perturbation is replaced with random perturbation.
* **MENTOR_w/o_MoE**: The policy backbone uses an MLP architecture instead of MoE.
* **MENTOR_w/o_TP_MoE**: Neither MoE nor Task-oriented Perturbation is used.

The results, summarized below, demonstrate the individual contributions of each component:
* **MENTOR_w/o_MoE** consistently outperforms **MENTOR_w/o_TP_MoE** and **MENTOR_w/o_TP** outperforms **MENTOR_w/o_TP_MoE** in 4 out of 5 tasks, indicating that both the MoE architecture and Task-oriented Perturbation independently contribute to improved policy learning.
* However, the overall sample efficiency and performance of **MENTOR_w/o_TP** and **MENTOR_w/o_MoE** remain lower than the full **MENTOR** model. This underscores the complementary nature of these two components in enhancing the overall learning efficiency and robustness of MENTOR.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces MENTOR, a method aimed at addressing sample efficiency and gradient conflict in Visual RL. It achieves this by employing a mixture-of-experts (MoE) network and a task-oriented perturbation mechanism. The paper demonstrates the effectiveness of this approach through experiments conducted in both simulated and real-world robotic manipulation tasks.

Strengths:

- The use of MoE to address gradient conflicts and alleviate the burden of shared parameters is well-motivated and innovative in the context of RL. (8sj6, 1toB, TuwR)
- The paper is well-written, clear, and easy to follow. (784n, TuwR)

Weaknesses:
- The contributions of individual components—MoE and task-oriented perturbation—are not well-isolated, leading to ambiguity in attributing performance gains. (784n, 1toB, TuwR)
- The hyperparameters are not sufficiently explored, and their impact on the experimental findings is unclear. (784n, TuwR)

While MENTOR introduces a compelling and innovative approach, the paper’s weaknesses in experimental rigor are notable. The lack of comprehensive ablation studies limits the clarity of the contributions. Given these concerns, I lean toward rejection. However, the foundational ideas hold significant promise, and with the outlined improvements, this work could make a substantial contribution in the future.