MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Authors: Suning Huang
Zheyu Aqa Zhang
Tianhai Liang
Yihan Xu
Zhehao Kou
Chenhao Lu
Guowei Xu
Zhengrong Xue
Huazhe Xu
Venue: ICML 2025 poster (2025)
Keywords: Visual Reinforcement Learning
Robotics
Mixture-of-Experts
Primary area: reinforcement_learning
TL;DR: MENTOR is a highly efficient visual RL algorithm that excels in both simulation and real-world complex robotic learning tasks.

Abstract: Visual deep reinforcement learning (RL) enables robots to acquire skills from visual input for unstructured tasks. However, current algorithms suffer from low sample efficiency, limiting their practical applicability. In this work, we present MENTOR, a method that improves both the *architecture* and *optimization* of RL agents. Specifically, MENTOR replaces the standard multi-layer perceptron (MLP) with a mixture-of-experts (MoE) backbone and introduces a task-oriented perturbation mechanism. MENTOR outperforms state-of-the-art methods across three simulation benchmarks and achieves an average of 83\% success rate on three challenging real-world robotic manipulation tasks, significantly surpassing the 32% success rate of the strongest existing model-free visual RL algorithm. These results underscore the importance of sample efficiency in advancing visual RL for real-world robotics. Experimental videos are available at https://suninghuang19.github.io/mentor_page/.

MENTOR: cexture-of-Experre Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Suning Huang" 12
Zheyu Zhang" * 13 Tianhai Liang'
Yihan Xu' Zhehao Kou" Huazhe Xu"
Chenhao Lu'
Guowei Xu
Zhengrong Xue
Abstract
1. Introduction
Visual deep reinforcement learning (RL) enables robots to acquire skills from visual input for un- structured tasks. However, current algorithms suf- fer from low sample efficiency, limiting their prac- tical applicability. In this work, we present MEN- TOR, a method that improves both the architec- ture and optimization of RL agents. Specifically, MENTOR replaces the standard multi-layer per- ceptron (MLP) with -ffturr-of-eepperss (MoE) backbone and introduces task-oriented pertur- bation mechanism. MENTOR outperforms state- of-the-art methods across three simulation bench- marks and achieves an average of 83% success rate on three challenging real-world robotic ma- nipulation tasks, significantly surpassing the 32% success rate of the strongest existing model-free visual RL algorithm. These results underscore the importance of sample efficiency in advancing visual RL for real-world robotics. Experimental vidcos are available at mentor-vrl.
Visual deep reinforcement learning (RL) focuses agents that perceive their environment through high-dimensional image data, closely aligning with robot control scenarios where vision is the primary modality. Despite substantial progress in this field (Kostrikov et al., 2020; Yarats et al., 2021; Schwarzer et al., 2020; Stooke et al., 2021; Laskin et al., 2020b), these methods still suffer from low sample efficiency. As result, most visual RL pipelines have to be first trained in the simulator and then deployed to the real world, inevitably leading to the problem of sim-to-real gap (Zhao et al., 2020; Salvato et al., 2021).
To bypass this difficulty, one approach is to train visual RL agents from scratch on physical robots, which known real-world RL (Dulac-Arnodd al., 2019; Luo al., 2024; Zhu et al., 2020). Given the numerous challenges of real-world RL, we argue that the fundamental solution lies not in task-specific tweaks, but in developing substan- tially more sample-efficient RL algorithms. In this paper, we introduce MENTOR: Mixture-of Mixture-of-Experts Network with Task Iris-Oriented perturbation for visual Reinforcement learn- ing, which significantly boosts the sample efficiency of visual RL through improvements in both agent network architecture and optimization.
In terms of architecture, visual RL agents typically use convolutional neural networks (CNNs) for feature extrac- tion from high-dimensional images, followed by multi-layer perceptrons (MLPs) for action output (Yarats et al., 2021; Zheng et al., 2023; Cetin et al., 2022: Xu et al., 2023). However, the learning efficiency of standard MLPs is hin- dered by intrinsic gradient conflicts in challenging robotic tasks (Yu et al., 2020a; Liu et al., 2023; Zhou et al., 2022; Liu et al., 2021), where the gradient directions for optimiz- ing neural parameters across different stages of the task trajectory or between tasks may conflict. this work, we propose alleviate gradient conflicts by integrating mixture- of-experts (MoE) architectures (Jacobs et al., 1991; Shazeer et al., 2017; Masoudnia & Ebrahimpour, 2014) as the back- bone to the visual RI. framework. Intuitively, MoF archi- tectures can alleviate gradient conflicts due to their ability to dynamically allocate gradients to specialized experts for
Peg
Routing
Tabletop
Figure 1. MENTOR validation on real-world tasks. We design three challenging robotic learning tasks where agents are trained from scratch in the real world. MENTOR demonstrates the most efficient and robust policies compared baseline methods.
'Equal contribution 'Tsinghua University "Stanford Uni- versity UUniversity of Illinois Urbana- Chhaana-haapaiinn... Correspon- dence Suning Huang <suning @edaanford.cdu.. Zheyu Zhang <zheyuz5 cetnus@@lliiiisc... ilinnosseeuu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MENTOR: Nixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
each input through the sparse routing mechanism (Akbari
2. Preliminary
et al., 2023; Yang et al., 2024).
Mixture-of-Experts (MoE). Mixture-of-experts (MoE), introduced by Jacobs al. (1991) and Jordan & Jacobs (1994), is a framework where specialized model compo- nents, called experts, handle different tasks or aspects of task. A sparse MoE layer comprises multiple experts and router. The router predicts probability distribution over the experts, activating only the top-k for each input (Shazeer et al., 2017). With N experts, each being a feed-forward network (FFN), the MoE output is:
In terms of optimization, visual RL agents often struggle with local minima due to the unstructured nature of robotic tasks. Recent works have shown that periodically perturb- ing the agent's weights with random noise can help escape local minima (Nikishin al., 2022: Sokar et al., 2023; Xu al., 2023: Ji et al., 2024). However, the choice of pertur- bation candidates (i.e., the network weights used to perturb the current agent's s weights) has not been thoroughly ex- plored. Building on this idea, we propose a task-oriented perturbation mechanism. Instead of sampling from a fixed distribution, we maintain a heuristically shifted distribution based on the top-performing agents from the RI. history. The intuition is that the distribution gradually formed by the weights of previous top- uop-erforming agents may accumu- late task-relevant information, leading to more promising optimization directions than purely random noise.
w(i; x) = softmax (topk (h())))
(1)
FM) = Σ w(1; x() FFN,
where w(t;x) is the gating function determining the weight of the i-th expert for input x. h(x) provides logits for expert selection, and topk (h(x) selects the top k.
Empirically, we find MENTOR outperforms current state-of- the-art methods (Xu et al., 2023; Yarats et al., 2021; Cetin et al., 2022: Zheng et al., 2023) across all tested scenar- ios in DeepMind Control Suite (Tassa al., 2018), Meta- World (Yu et al., 2020b), and Adroit (Rajeswaran et al., 2017). Furthermore, we present three challenging real- world robotic manipulation tasks, shown in Figure 1: Peg invertion-nsertiee three kinds of pegs into the correspond- ing sockets; Cable Routing- - maneuvering one end of a rope to make it fit into two non- non-parallel slots; and Tabletop golf ball into the target hole while avoiding getting stuck into the trap. In these experiments, MEN- TOR demonstrates significantly higher learning efficiency, achieving an average success rate of 83%, compared to 32% for the state-of-the-ae counterpart (Xu et al., 2023) within the same training time. This confirms the effectiveness of our approach and underscores the importance of improving sample efficiency for making RL algorithms more practical in robotics applications.
Visual Reinforcement Learning. We employ visual rein- forcement learning (RL) to train policies for robotic systems, modeled as a Partially Observable Markov Decision Pro- cess (POMDP) defined by the tuple (S, O, A, P. 7)). Here, S is the true state space. O represents visual observations, A is the robot's action space, P S A + S defines the transition dynamics, r(s,a) S x A R specifies the reward, and E (0, 1] is the discount factor. The goal is to learn an optimal policy Te(a, | o:) that maximizes the expected cumulative reward E, 1E ('r(s,,)
Dormant- Ratio- Perturbation in RL. The concept of dormant neurons, introduced by Sokar et al. (2023), refers to neurons that have become nearly inactive. It formally defined follows:
Definition 2.1. Consider a fully connected layer with N neurons. Let (i)ear((x)) denote the output of neuron in layer for an input distribution x: E I. The score of neuron is given by:
Our key contributions are threefold. First, we introduce the MoE architecture to replace the MLP as the agent backbone in model-free visual RL, improving the agent's learning ability to handle complex robotic environments and reduc- ing gradient conflicts. Second, we propose task-oriented perturbation mechanism which samples candidates from heuristically updated distribution, making network perturba- tion a more efficient and targeted optimization process com- pared to the random parameter exploration used in previous RL perturbation methods. Third, we achieve state-of-the-ar performance in both simulated environments and three chal- lenging real-world tasks, highlighting the sample efficiency and practical value of MENTOR.
Emez linear
(3)
Eker Erez linear(
This neuron is considered 7-dormant if its score sf T.. Definition 2.2. In layer 1, the total number of d-dormnnt neurons is denoted by DJ. The r-dormant ratio of a neural network 0 is defined as:
Eieg D4, Σieo N
Br
(4)
As shown by Xu et al. (2023); et al. (2024), the dormant ra- tio is a key metric in neural network behavior and enhances
2

---

MENTOR: Mixture--o-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Actions
Collect Top-agent
Stop
W2
W4
QOP
Rewards
FFN1
FFN2
FFN
FFN4
FFNN
Task- Oriented Perturbation
top top N N B
Top k
Router
Policy Update
Reinforcement Learning
CNN Encoder
Agent
Visual Observations
Figure Overview. MENTOR uses an MoE backbone with CNN encoder to process visual inputs. router selects and weights the relevant experts based the inputs generate the final actions. In addition to regular reinforcement learning updates, periodic task-oriented perturbations are applied during training by sampling from top-performing agents to adjust the current agent's 's weights.
RL efficiency via parameter perturbation. This periodically resets network weights by interpolating between current parameters and random initialization (Ash Adams, 2020; D'Oro et al., 2022):
When using shared parameters (e.g., MLP), meaning all parameters must be simultaneously active to function, the optimization process using gradient descent may compro- mise individual loss optimization. This issue, known as conflicting gradients (Yu et al., 2020a; Liu et al., 2021), hin- ders the agent's 's ability optimize its behavior when facing complex scenarios effectively.
00c10 + (1 aaΦ, initializer. (5)
Here, a is the perturbation factor, 0k-1 and 02 are the weights pre- and post-reset, and denotes randomly ini- tialized weights (e.g.. Gaussian noise). The value of a dynamically adjusts as = clip(1 143, Ommin Omax), where is the perturbation rate and 3 the dormant ratio.
We propose replacing the MLP with an MoE back- bone. The MoE, composed modular experts MMOE = (0,,02,.. ... 0v) which allows the agent to activate differ- ent experts via a dynamic routing mechanism flexibly. This enables gradients dynamically route from different tasks to specific experts, reducing gradient conflicts. Each expert is updated using gradients from related tasks, addressing conflicts effectively. As shown in Figure 2, the MoE agent uses a CNN encoder to map visual inputs to latent space Z. The router computes probability h(i 2) over experts for latent E Z. The top-k experts are selected, and their outputs a; are combined using softmax weights wy (Equa- tions I and 2). This structure routes inputs to specialized experts, improving multi-task performance.
3. Method
In this section, we introduce MENTOR, which includes two key enhancements to the architecture and optimization of agents, aimed at improving sample efficiency and overall performance in visual RL tasks. The first enhancement ad- dresses the issue of low sample efficiency caused by gradient conflicts in challenging scenarios, achieved by adopting an MoE structure in place of the traditional MLP as the agent backbone, as detailed in Section 3.1. The second enhance- ment introduces a toie-trie perturbation mechanism that optimizes the agent's training through targeted pertur- bations, effectively balancing exploration and exploitation, as outlined in Section 3.2. The framework of our method is illustrated in Figure 2.
To illustrate the important role of dynamic modular expert learning for RL agents, we conduct a multi-task experi- ment (MT5) Meta-World, training an agent (#Experts 16, k: 4) for five opposing tasks: Open (Door-Open, Drawer-Open, Window-Open) and Close (Drawer-Close, Window-Closed Figure 3a shows Open and Close tasks share some experts but also utilize dedicated ones. To quan- titatively demonstrate how much the MoE alleviates the gra- dient conflict issue, we evaluate the cosine similarities (Yu et al., 2020a) for both MLP and MoE agents Figure 3b. The MLP's gradients show significant conflicts between opposing tasks, while the MoE model demonstrates higher gradient compatibility. As result, there is performance gap, with the MI. achieving 100% success in Close tasks
3.1. Mixture-of-Expertt as the Policy Backbone
In challenging robotic tasks, RL. agents often assigned K 2 2 different tasks or subgoals, each with a loss L.).). The goal is to optimize shared weights f E R by minimizing the average loss:
A 1 K (o
M L
8" = arg min
K 1=1
3

---

MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
but only 82% in Open tasks, whereas the MoE achieves 100% success in both task types.
past hirh-performing agents, to provide more task-oriented candidates that better facilitate optimization.
This structural advantage can also be propagated to challeng- ing single tasks, as the dynamic routing mechanism auto- matically activates different experts to adjust the agent's 's behavior throughout the task, alleviating the burden on shared parameters. We illustrate this through training a same-structure MoE agent on single, highly challenging Assembly task from Meta-Wort
We define T Poriented as a distribution from which high- performing agent weights can be sampled. This distribution is obtained by maintaining a fixed-size set Sup = {(0, R)), where (0, R) represents an agent with weights 0 and episode reward R. The distribution approximated as Q Poriented = N(Hg top .ogg) where Pe and og are the mean and stan- dard deviation of weights in Sug- As shown in Figure 2, Swo is updated during training: at episode I, if an agent with weights O2 achieves reward Rg exceeding the lowest in Stop (0,, R,) replaces the lowest-reward tuple. This ensures o Porseneed reflects current high-performing agents, improving perturbation candidates & for subsequent iterations. The pseudocode is in Algorithm 1.
(MW). Figure 4 shows the engagement of the k- = 4 most active experts during task execution, with Expert 15 serving as the shared module throughout the entire policy execution. The other experts vary and automatically divide the task into four distinct stages: Expert 9 handles gripper control for grasping and releasing, Expert 13 manages arm movement while maneu- vering the ring: and Expert 14 oversees the assembly process as the ring approaches its fitting location. More detailed results about how MoE alleviates gradient conflicts in the single task are shown in Appendix F.
For illustration, we conduct experiments on the Hopper Hop task from the DeepMind Control Suite (DMC), compar- ing task-oriented perturbation approach to leading model- free visual RL baselines (DrM (Xu et al., 2023) and DrQ- v2 (Yarats et al., 2021)). Our approach solely replaces DrM's perturbation mechanism with task-oriented perturba- tions. Both our method and DrM outperform DrQ-v2 due dormant-ratio-ba based perturbation, but our method achieves faster skill acquisition and maintains a lower, smoother dormant ratio throughout training (Figure 5a and 5b). By directly testing perturbation candidates as agents in the task (Figure 5c), we observe that candidates sampled from T Porsented steadily improve throughout training, sometimes even surpassing the performance of the agent they perturb, This demonstrates that ooriemed progressively captures the optimal weight distribution, rather than simply interpolating from past agents, leading to more targeted optimization. In contrast, perturbation candidates from DrM (initialized with Gaussian noise) consistently yield zero reward, indicating the lack of task-relevant information.
Algorithm 1
Perturbation Mechanism
Initialize the set Stop = 0, perturb interval Tp for each episode 1 = 1, 2.... do
Execute policy TT, and obtain episode reward R2 and dormant
ratio 33
if [Smppl < N then Add (0., to Sup
else
if R. > min(Ri R4 | (0,, R.) E } then
Replace (0z, R2) with (0,, Re), where j argmin
end if end
if (Number of steps since last perturb) 2 Tp then
Compute mean (4g and standard deviation un from Suo Sample perturbation weight 6 ~ Poriented -N po )
Calculate perturb factor as in Sokar al. (2023): clip(1 - n&, aminn amax
Update agent weights: 0g = al;- + (1 - aae
end if end for
4. EExpermmenss
In this section, we present a comprehensive empirical eval- uation of MENTOR. Section 4.1 showcases its effective- ness on three simulation benchmarks: DeepMind Control Suite (DMC) (Tassa al., 2018), Meta-World (MW) (Yu et al., 2020b), and Adroit (Rajeswaran al., 2017), which feature rich visuals and complex dynamics. MENTOR con- sistently outperforms leading visual RI. algorithms. How- ever, a critical limitation in visual RL research is the over- reliance on simulated environments, raising concerns about practical applicability. To bridge this gap, Section 4.2 val- idates MENTOR on three challenging real-world robotic tasks, highlighting the importance of real-world testing.
3.2. Task-oriented Perturbation Mechanism
Neural network perturbation is employed to enhance the exploration capabilities in RL. Two key factors influence the effectiveness of this process 0₂ = a0x-1++ (al1-a)). )6, 6 ~ 4. a is the perturbation factor controlling the mix between cur- rent agent and perturbation candidate weights. & represents the perturbation candidate sampled from a distribution + which typically is a fixed Gaussian noise N(,, o). Previous works (Xu et al., 2023; Ji et al., 2024) have investigated the use of the dormant ratio to determine resulting in im- proved exploration efficiency (see Section 2). However, the selection of perturbation candidates has not been thoroughly examined. In this work, we propose sampling & from a heuristically updated distribution & Prieeted, generated from
4

---

MENTOR: Eexture-of-Expertt Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Drawer
Window
Drawer
Window
1.0
Open Tasks
Open
Open
Close
Close
Close Tasks
0.15
MoE
Expert Usage Intensity
0.8
Drawer
MLP
Open
Window
0.05
Open
0.4
0.00
Drawer
Close
-0.05
0.2
Window
13
10 14 15
12 11
Close
Expert ID (Sorted by expert usage intensity for open tasks)
Figure 3. MoE in multi-task scenarios. Left: Expert usage intensity distribution of the MoF agent in opposing tasks. Right: Gradient conflict among opposing tasks for both MLP and MoE agents. The MLP agent frequently encounters gradient conflicts (indicated by negative cosine similarity) when learning multiple skills, while the MoE agent avoids these conflicts (indicated by positive values). We also provide a comparison of gradient conflicts for MLP and MoE agents in single-task settings, as detailed Appendix F.
Usage Intenslty (Top km
Expert
- Expert 13
-O
Expert 14
Expert 15
0.75
0.5
0.25
0
Frame
13
Grasp
Move
Assemble
Release
Figure MoE multi-stage scenarios. We present expert usage intensity during the Assembly task Meta- World. While Expert 15 remains highly active throughout the entire process, other experts are activated with varying intensity over time, automatically dividing the task into four distinct stages.
4.1. Simulation Experiments
on long-horizon continuous locomotion and manipulation challenges. The MW environment provides suite of robotic tasks including Assembly, Disassemble, Pick Place, Cof- fee Push (Sparse), Soccer (Sparse), and Hammer (Sparse), which test the agent's manipulation abilities and require se- quential reasoning. The Adroit environment includes com- plex robotic manipulation tasks such as Door and Hammer, which involve controlling dexterous hands to interact with articulated objects. Notably, DMC tasks are evaluated using episode reward, while tasks in MW and Adroit are assessed based success rate. We conducted experiments with four random seeds on each task, with detailed hyperparameters and training settings provided in Appendix B.
Baselines: We compare MENTOR against four leading model-free visual RL methods: DrM (Xu et al., 2023), ALIX (Cetin et al., 2022), TACO (Zheng al., 2023), and DrQ-v2 (Yarats et al., 2021). DrM, ALIX, and TACO all use DrQ-v2 as their backbone. DrM periodically perturbs the agent's 's weights with random noise based on the proportion of dormant neurons in the neural network; ALIX adds reg- ularization to the encoder gradients to mitigate overfitting; and TACO employs contrastive learning to improve latent state and action representations.
Experimental Settings: We evaluate MENTOR di- verse set of tasks across three simulation environments with complex dynamics and even sparse reward. The DMC in- cludes challenging tasks like Dog Stand, Dog Walk, Manip- ulator Bring Ball, and Acrobot Swingup (Sparse), focusing
Results: Figure 6 presents performance comparisons be- tween MENTOR and the baselines. In the DMC tasks, Dog Stand and Dog Walk feature high action dimensionality with 38-dimensional action space representing joint controls for
5

---

MENTOR: cesture-offExpertt Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Episode Reward obtained by
2001
175
Aclor Dormant Ralio
0.8
1/5
Episode Reword
150-
150
125
0.6
125
100
100
0.4
75-
50-
0.2
50-
0.0
0.0
0.2
1.0
0.8
1.0
0.0
0.2
0.4
1.0
Number Frames
Number Frames
Number of Frames
(a)
(b)
(c)
Task-Oriented Perturbation (Ours)
Vanilla Perturbation (DrM)
No Perturbation (DrQ-v2)
Figure 5. Validation of task-oriented perturbation on Hopper Hop (a MENTOR, DrM, and DrQ-v2 agent trained on the Hopper Hop task during the first 1M frames). Our method consistently achieves higher episode rewards with a consistently lower dormant ratio. (c) shows the episode reward obtained by perturbation candidate & sampled from Poiessed steadily increases and occasionally surpasses that of the corresponding RL. agent (replotted as the light red line), whereas in DrM, the reward remains at zero due to the use of randomly generated perturbation parameters.
the dog model. These tasks also have complex kinematics involving intricate joint coordination, muscle dynamics, and collision handling, making them challenging to optimize. Our method outperforms the top baseline, achieving approx- imately 17% and 10% higher episode rewards, respectively. In the MW tasks, the Hammer (Sparse) task stands out. It requires a robotic arm to hammer a nail into a wall, with highly sparse rewards: success yields significantly larger rewards than merely touching or missing the nail. In fact, the reward for failure is only the-thous of the success reward, making the task extremely sparse. However, our task-oriented perturbation effectively captures these sparse rewards, reducing the required training frames by 70% com- pared to the best baseline. In the Adroit tasks, our method achieves nearly 100% success with significantly less train- ing time, while the most competitive counterpart (DrM) requires more frames, and other baselines fail to match per- formance even after 6 million frames. A key highlight is the Door task, which involves multiple stages of dexter- ous hand manipulation grasping, turning, and opening the door. Leveraging the MoF. architecture, our method reduces training time to achieve over 80% success by approximately 23% compared the best baseline. In summary, MENTOR demonstrates superior efficiency and performance compared to the strongest existing model-free visual RL baselines across all 12 tasks. For the robustness against disturbances of agent trained by MENTOR, please see Appendix I.
policies trained from scratch in the real world.
Experimental Settings: We use a Franka Panda arm for execution and RealSense D435 cameras for RGB observa- tions, capturing both global and local views. Rewards are based on the absolute distance between current and desired states. To prevent trajectory overfitting, the end-effector's ss initial position is randomly sampled within a predefined region at the start of each episode. Tasks are detailed be- low, with illustrations in Figure 7. Additional details are in Appendix D.
Peg Insertion: requiring the agent to insert pegs of three shapes (Star, Trian- gle, Arrow) into corresponding sockets. Simulating contact- rich interactions in these multi-task settings is highly chal- lenging, making
This task mimics assembly-line scenarios
evaluation essential.
Cable Routing: Manipulating deformable cables presents significant challenges due to the complexities of modeling and simulating their physical dynamics, making this task ideal for direct, model-free visual RL training in real-world environments. The robot must guide a deformable cable sequentially into two parallel slots. Since both slots cannot be filled simultaneously, the agent must perform the task sequentially, requiring long-horizon, multi-stage planning to successfully accomplish the task.
Tabletop Golf: In this task, the robot uses golf club to strike a ball on a grass-like surface, aiming to land it in a target hole. An automated reset system retrieves the ball when it reaches the hole, enters a mock water hazard, or rolls out of bounds, and randomly repositions it. The agent must learn to approach the ball, control the club's 's striking force and direction guide the ball toward the hole while avoiding obstacles through real-world interaction.
We conduct a detailed ablation study to demonstrate the significance contribution of MoE and Task- oriented Perturbation separately in Appendix C.
Ablation Study:
4.2. Real-World Experiments
Our real-world RL experiments evaluate MENTOR on three key challenges: multi-task learning, multi-stage deformable object manipulation, and dynamic skill acquisition with
Results: Our policies exhibit strong performance during
6

---

MENTOR: Mixtere-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
DMC Dog Stand
DMC Dog Walk
DMC Manipulator Bring Ball
DMC Acrobot Swingup (Sparse)
1000
400
Episode Reward
800
800
680
600
400
200
400
200
200
10m
25m 30m
5m
15m 20m 30m
10m
20m 25m
1m
3m
Adrolt Door
Adrolt Hammer
Assembly
MW Disassemble
100%
100%
100%
100%
60%
60%
60%
60%
40%
40%
40%
40%
20%
20%
20%
1.2m 175m
0.4m
1.5m
MW Pick Place
MW Coffee Push (Sparse)
MW Soccer (Sparse)
MW Hammer (Sparse)
1000%
100%
100%
80%
80%
80%
80%
60%
60%
60%
60%
40%
40%
20%
20%
20%
20%
0%
0%
0%
0%
0.4m O. 0.0m 2m 1.6m
2m
0.4m 0.8m 1.2m 1.6m
2m
0.4m 0.0m 1.2m 1.6m
2m
0.4m 0.0m 1.2m 1.6m 2m Number Frames
Number of Frames
Number of Frames
Number of Frames
MENTOR (Ours)
DrM
DrQ-v2
ALIX
TACO
Figure 6. Performance comparisons simulations. This figure compares the performance of method DrM, DrQ-v2, ALIX, and TACO across 12 tasks with four random seeds in three different benchmarks (DMC, MW, and Adroit). The shaded region indicates standard deviation DMC and the range success rates in MW and Adroit.
evaluation, as shown in Figure 7. In Peg Insertion, the agent randomly picks a peg and inserts it from varying initial positions, learning to align the peg with the hole and adjust the angle for accurate insertion. During one execution, as the peg nears the hole, we manually disturb by altering the robot arm's pose significantly. Despite this interference, the agent successfully completes the task relying solely visual observations. In Cable Routing, the agent learns to prioritize routing it into the farther slot first, then into the closer one. This second step requires careful handling to avoid dislodging the cable from the first slot. During execution, if the cable is randomly removed from the slot, the agent visually detect this issue and re-route it back into position. In Tabletop Golf, the agent must master two key skills: striking the ball with the correct direction and force, and repositioning the club to follow the ball after the strike. Due to "water hazard", the ball cannot be struck directly toward the target hole from its starting position. The agent learns angle its shots to bypass hazards and guide the ball into the hole, even the ball's rolling the grass-like surface naturally introduces significant variability.
from scratch results in an average performance improve- ment of 9%. However, no significant performance gain is observed in simulation benchmarks with this substitution. This discrepancy may arise from the gap between simulation and real-world environments, where real scenes offer richer textures more aligned with the pretraining domain.
Furthermore, the results confirm the effectiveness of our technical contributions. When the MoE structure is removed from the agent (i.e., replaced with an MLP, as in MENTOR w/o MoE), overall performance drops by nearly 30%. Ad- ditionally, further switching the task-oriented perturbation mechanism basic random perturbation (as in DrM) leads to additional performance decline of approximately 30%.
5. Related work
Visual Reinforcement Learning. Visual reinforcement learning (RL), which operates on pixel observations rather than ground-truth state vectors, faces significant challenges in decision-making due to the high-dimensional nature of visual inputs and the difficulty in extracting meaningful fea- tures for policy optimization (Ma et al., 2022; Choi et al., 2023; Ma et al., 2022). Despite these challenges, there has been considerable progress this area. Methods such Hafner et al. (2019; 2020: 2023); Hansen al. (2022) improve visual RL by building world models. Other ap- proaches (Yarats et al., 2021; Kostrikov et al., 2020; Laskin
Ablation Study:
We conduct a detailed ablation study to demonstrate the effectiveness of MENTOR in improving sample efficiency and performance, as shown in Table I.
The first two rows reveal that utilizing the pretrained vi- sual encoder (Lin al., 2024) instead of CNN trained

---

MENTOR: Sopuur--o-Exxperre Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Start Execution
Disturbance
Success!
Figure 7. Real- world experiments (up to down rows: Peg Insertion, Cable Routing, and Tabletop Golf). This set of images illustrates the exccution of the learned visual policy trained using MENTOR. The agent consistently and accurately accomplishes tasks in the presence of human disturbances.
Table Comparison success ratios between MENTOR and ablations with equal training times. Peg Insertion and Cable Routing are trained for 3 hours, and Tabletop Golf for 2 hours. During evaluation, each subtask in Peg Insertion is rolled out 10 times, while Cable Routing Tabletop Golf are rolled out 20 times.
Peg Insertion (Subtasks)
Method
Cable Routing
Tabletop Golf
Star
Triangle
Arrow
MENTOR w/ pretrained encoder
1.0
1.0
1.0
0.9
0.8
MENTOR
1.0
1.0
1.0
0.8
0.7
MENTOR w/o MoE
1.0
0.7
0.6
0.45
0.55
DrM
0.2
0.1
0.2
al.. 2020a), use data augmentation enhance learning robustness from pixel inputs. Contrastive learning, as in Laskin et al. (2020b); Zheng et al. (2023), aids in learning more informative state and action representations. Addi- tionally, Cetin et al. (2022) applies regularization to prevent catastrophic self-overfitting, while DrM (Xu et al., 2023) enhances exploration by periodically perturbing the agent's parameters. Despite recent progress, these methods still suffer from low sample efficiency in complex robotic tasks. In this paper, we propose enhancing the agent's learning capability by replacing the standard MI.P backbone with MoE architecture. This dynamic expert learning mechanism helps mitigate gradient conflicts in complex scenarios.
et (2023) identified correlation where improved learn- ing capability is often accompanied by decline in the dormant neural ratio in agent networks. Building on this insight, Xu et al. (2023): Ji et al. (2024) introduced param- eter perturbation mechanisms that softly blend randomly initialized perturbation candidates with the current ones, aiming to reduce the agent's dormant ratio and encourage exploration. However, previous works have fully ex- plored the choice of perturbation candidates. In this work, uncover the potential of targeted perturbation for more efficient policy optimization by introducing a simple yet effective task-oriented perturbation mechanism. This mech- anism samples perturbation candidates from a time-varian distribution formed by the top-performing agents collected throughout RL history.
Neural Network Perturbation in RL. Perturbation the- ory has been explored in machine learning to escape local minima during gradient descent (Jin et al., 2017; Neclakan- tan et al., 2015). In deep RL, agents often overfit and lose expressiveness during training (Song et al., 2019; Zhang al., 2018; Schilling, 2021). To address this issue, Sokar
6. Conclusion
In this paper, we present MENTOR, a state-of-the-art model- free visual RL framework that achieves superior perfor- mance challenging robotic control tasks. MENTOR en-
8

---

MENTOR: Mixture-oo-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
hances learning efficiency through two key improvements in both agent network architecture and optimization. MEN- TOR consistently outperforms the strongest baselines across 12 tasks in three simulation benchmark environments. Fur- thermore, we extend our evaluation beyond simulations, demonstrating the effectiveness of MENTOR in real-world settings on three challenging robotic manipulation tasks. We believe MENTOR is a capable visual RL algorithm with the potential to push the boundaries of RL application in real-world robotic tasks. While MENTOR has proven effec- tive, it has been evaluated on single tasks with single robot embodiment. Future work could scale our method to handle hundreds of tasks or diverse robot embodiments, enabling broader real-world applications.
lenges of real-world reinforcement learning. arXiv preprint arXiv:1904. 2019.
Fedus, W. Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Hafner, D., Lillicrap. T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. arXiv preprint MtXiv:1912.0166. 2019.
Hafner, D., Lillicrap. T., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
Hafner, D., Pasukonis. J., Ba, J., and Lillicrap. T. Mastering diverse domains through world models. arXiv preprint arXiv:2301 1044444 2023.
Impact Statement
This paper presents work whose goal advance the field of Model-free Visual Reinforcement Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hansen, N., Wang, X., and Su, H. Temporal difference
learning for model predictive control. arXiv preprint arXiv:223.009555 2022.
Jacobs, R. A., Jordan, M. I., Nowlan, J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3(1):79-77, 1991.
References
Akbari, H., Kondratyuk, D., Cui, Y., Hornung, R., Wang, H., and Adam, H. Alternating gradient descent and mixture-of-experts for integrated multimodal perception. Advances in Neural Information Processing Systems. 36: 79142-79154, 2023.
Ji, T., Liang, Y., Zeng, Y., Luo, Y., Xu, G., Guo, J., Zheng, R., Huang, F., Sun, F., and Xu, H. Ace: Off-policy actor- critic with causality-aware entropy regularization. arXiv preprint arXiv:2402.14528, 2024.
Ash, J. and Adams, R. P. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894, 2020.
Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., and Jor- dan, M. I. How to escape saddle points efficiently. In International conference on machine learning, pp. 1724- 1732. PMLR, 2017.
Cetin, E., Ball, P. J., Roberts, S., and Celiktutan, O. Stabi- lizing off-policy deep reinforcement learning from pixels. arXiv preprint 2rXiv:2207.00986 2022.
Jordan, M. I. and Jacobs, R. A. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2): 181-214, 1994.
Chen, Z., Shen, Y., Ding, M., Chen, 7., Zhao, H., Learned- Miller, E. G., and Gan, C. Mod-squad: Designing mixtures of experts as modular multi-task learners. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11828-11831 11837, 2023.
Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 2020.
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020a.
Choi, H., Lee, H., Jeong, S., and Min, D. Environment ag- nostic representation for visual reinforcement learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 263 (63-277,222 2023.
Laskin, M., Srinivas, A., and Abbeel, P. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020b.
D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Lepikhin, D.,, Lec, H., Xu, Y., Chen, D.,, Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Dulac-Arnold, G., Mankowitz, D., and Hester, T. Chal-
16668,

---

MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Lin, X., So, J., Mahalingam, S., Liu, F., and Abbeel, P. Spawnnet: Learning generalizable visuomotor skills from pre-trained network. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4781-4787. IEEF, 2024.
September 14-17, 2021, Proceedings, Part IV 30, pp. 638-649. Springer, 2021.
Schwarzer, M., Anand, A., Goel, R., Hjelm, D., Courville, A., and Bachman, P. Data-efficient reinforcement learn- ing with self-predictiv representations. arXiv preprint arXiv:2007.0592 2020.
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878- 18890, 2021.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-exper layer. 70177010000 1.658,,,, 2017.
Liu, S., Chen, Z., Liu, Y., Wang, Y., Yang, D., Zhao, Z., Zhou, Z., Yi, X., Li, W., Zhang, W., et al. Improving gen- eralization in visual reinforcement learning via conflict- aware gradient agreement augmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23436-23446, 2023.
arXiv preprint arXiv: 17XIv
Shen, Y., Zhang. Z., Cao, T., Tan, S., Chen, Z., and Gan, C. Moduleformer: Learning modular large language models from uncurated data. arXiv preprint atXiv:2306.046400 2023.
Sokar, G., Agarwal, R., Castro. P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforcement learning. arXiv preprint arXiv:2302. 2023.
Luo, J., Hu, Z., Xu, C., Tan, Y. L., Berg, J., Sharma, A., Schaal, S., Finn, C., Gupta, A., and Levine, Serl: A software suite for sample-efficient robotic reinforcement learning. arXiv preprint arXiv:2401.16011 2024.
Song, X., Jiang. Y., Tu, S., Du, Y., and Neyshabur, B. Ob- servational overfitting in reinforcement learning. arXiv preprint arxiv: 19ti::19922.29.. 2019.
Ma, G., Wang, Z., Yuan, Z., Wang, X., Yuan, B., and Tao, D. comprehensive survey of data augmentation in visual re- inforcement learning. arXiv preprint arXiv:2210.04561, 2022.
Stooke, A., Lee, K., Abbeel, P., and Laskin, M. Decoupling representation learning from reinforcement learning. In International conference on machine learning, pp. 9870- 9879. PMLR, 2021.
Masoudnia, S. and Ebrahimpour, R. Mixture of experts: literature survey. Artificial Intelligence Review, 42: 275-293, 2014.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. Deepmind control suite. arXiv preprint arXiv: 1801: 88010066000 2018
Neelakantan, A., Vilnis, L., Le, Q. V., Sutskever, I., Kaiser, L., Kurach, K., and Martens, J. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.0660 066007, 2015.
Xu, G., Zheng, R., Liang, Y.. Wang, X., Yuan, Z., Ji, T., Luo, Y., Liu, X., Yuan, J., Hua, P., et al. Drm: Master- ing visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:23 196688 2023.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforce- ment learning. In International conference machine learning, pp.
PMLR, 2022.
Yang, L., Sheng, D., Cai, C., Yang, F,, Li, S., Zhang, D., and Li, X. Solving token gradient conflict in mixture-of- experts for large vasion-language model. arXiv preprint MtXi:22406.19905 2024.
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schul- man, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 2017.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented re- inforcement learning. arXiv preprint arXiv:2107.0966 2021.
Salvato, E., Fenu, G., Medvet, E., and Pellegrino, F. A. Crossing the reality gap: A survey on sim-to-real trans- ferability of robot controllers in reinforcement learning. IEEE Access, 9:153171-153187, 2021.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836 2020a.
Schilling, M. Avoid overfitting in deep reinforcement learning: Increasing robustness through decentralized control. In Artificial Neural Networks and Machine Mouning-ICANNI 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia,
Yu, T. Quillen, D.,, He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world A benchmark and eval- uation for multi-task and meta reinforcement learning.
10

---

MENTOR: Nixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
In Conference robot learning, pp. 1094-1100. PMLR, 2020b.
Zhang, C., Vinyals, O., Munos, R., and Bengio, A study on overfitting in deep reinforcement learning. arXiv preprint arXiv: 804, 06893, 2018.
Zhao, W., Queralta, J. P., and Westerlund, T. Sim-to-real transfer in deep reinforcement learning for robotics: a sur- vey. In 2020 IEEE symposium series on computational intelligence (SSCI), pp. 737-744. IEEE, 2020.
Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., Daumé III, H., and Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning, arXiv preprint arXiv:2306. 13229, 2023.
Zhou, S.. Zhang, W., Jiang, J., Zhong, W., Gu, J., and Zhu, W. On the convergence of stochastic multi- objective gradient manipulation and beyond. Advances in Neural Information Processing Systems, 35:38103- 38115, 2022.
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A. Kumar, V., and Levine, S. The ingredients of real- world robotic reinforcement learning. arXiv preprint 1rXiv:2004.12570, 2020.

---

MENTOR: Nixture-of-Expert Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Appendix
Algorithm 2 Mixture-of-Experts the Policy Backbone
A. Algorithm Details
Require: Batch of visual inputs {x6 b=
Ensure: Final actions {as 1 Load balancing loss LLB
We illustrate the overview framework of MENTOR in Sec- tion 3, where we employ two enhancements terms of agent structure and optimization: substituting the MLP backbone with MoE to alleviate gradient conflicts when learning complex tasks, and implementing task-oriented perturbation mechanism to update the agent's weights in a more targeted direction by sampling from a distribution formed by the rop-performing agents in training history. The detailed implementation of task-orienter perturbation is shown in Algorithm 1, and the implementation of using MoE as the policy backbone is described as follows:
Initialize experts (FFN1, FFN2,.......... .FFNN]
FFNN,...,,,NNN
[ Encoder (xb (6)) 6=1)
[h, b= f h((z))=
for b to B do
& +- topk(hs,
Vi E & wa(i) < softmax (ho M..
for each E & do
fo.; t (z,) end for G6 <- Action Projectr (Eres, wo(() fb.e)
end for
I/ Compute load balancing loss
Algorithm 2 illustrates how MENTOR employs the MoE architecture as the backbone of its policy network. In ad- dition to the regular training process, using MoE as the policy agent requires adding an additional loss to prevent MoE degradation during fhaninggwhers fixed subset of experts is consistently activated. The MoE layer computes the output action while simultancously calculating aux- iliary loss for load balancing (Lepikhin et al., 2020; Fedus et al., 2022). Specifically, we extract the distribution over experts produced by the router for each input. By averaging these distributions over a large batch, we obtain an overall expert distribution, which we aim to keep uniform across all experts. To achieve this, we introduce auxiliary loss term-the negative entropy of the overall expert distribu- tion (Chen et al., 2023; Shen et al., 2023). This loss reaches its minimum value of log( .N),, where Ne is the number of experts in the MoE, when all experts are equally utilized, thus preventing degradation. This auxiliary loss is added to the actor loss and used to update the actor during the RL process.
{pr,)== + softmax( (( (b=))))
p(1) + 18 Eb=1Pro Vi
LLB t - (p) = 1 log(((i)
B. Simulation Experimental Settings
The hyperparameters employed in our experiments are de- tailed in Table 3. In alignment with previous work, predominantly followed the hyperparameters utilized in DrM (Xu et al., 2023).
For the hyperparameters used in MoE module, Figure shows the ablation study on the number of experts and top chosen experts. The results indicate the optimal setting for the Hammer task is MoB has around experts, performance remains consistent across 4, 8, and 32 experts long as top k ==4 4. That is, given proper top k, the performance is not sensitive to the number of experts.
C. Ablation Study on Key Contributions
Ablation: {Num_ of Experts) [ipeerss[[Top____ [Top_k) }}
We conducted additional ablation studies on four diverse tasks: Hopper Hop, Disassemble, Coffee-Push (Sparse), and Hammer (Sparse). These studies aim to decouple the effects of the MoE architecture and the Task-oriented Perturbation (TP) mechanism proposed in our paper.
100%
80%
Success Rate
80%
For the experiments, we evaluate four ablated versions of MENTOR using the same four random seeds as in the origi- nal experiments, as shown in Figure 9:
40%
20%
0%
MENTOR: Full model with both MoE and Task- oriented Perturbation.
0.4m 0.8m
1.6m Number of Frames
1 2m
2m
MENTOR w/o.TP.. Trsk-eriented Perturbation is re- placed with random perturbation.
Figure 8. Ablation study the hyperparameter of MoE. We evaluated MENTORon MW Hammer (Sparse) wit different MoE settings.
MENTOR COROOR.wooMMMM MoE: The policy backbone uses an MLP architecture instead of MoE.
12

---

MENTOR: Mixture--o-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Hopper Hop
Disassemble
Coffee Push (Sparse)
Hammer (Sparse)
250
100%5
100%
10059
Episods Rewan
200
80%6
80%
80%
Success Rate
Success Rale
150
6.0%
Surcess
100
50
20%
20%
Number of Frames
Number of Frames
Number of Frames
Number of Framnss
MENTOR
w/o_TP
MENTOR w/o_Tr TP___P___
Figure Ablation study on key contributions. This figure shows the experiment result four ablated versions of MENTOR using the same four random seeds as in the original experiments, illustrating our Mixture-of-Frye of-Experts (MoE) and Task-oriented Perturbation (TP) are both significant to improving performance.
Table Sample efficiency comparison across tasks.
Sample Efficiency (Training Time)
Hopper Hop
Disassemble
Coffee Push
Hammer
MENTOR (Ours)
0.6167
0.7056
0.8066
0.7167
MENTOR w//.////............. TP
1
0.8505
0.9481
0.875
MENTOR_W/o.Ma
0.85
1
MENTOR w/o TP MoE: Neither MoE
Task-
better).
oriented Perturbation is used.
Table 2 shows the normalized sample efficiency of each setting. MENTOR (Ours) achieves an average of 22.6 % and 26.1 % less training time over the 4 tasks compared with MENTOR SOTTTOR_/o_TTTT w/o.TP and MENTOR w//.///ooooo. MoE.
The results, summarized below, demonstrate the individual contributions each component:
The overall sample efficiency and performance MEN- TOR_wi [iR_w/o_TT TP and MENTOR_W/o Moo remain lower than the full MENTOR model. This underscores the comple- mentary nature of these two components in enhancing the overall learning efficiency and robustness of MENTOR.
D. Real-World Experimental Settings
The training and testing videos are available mentor-vrl. The hyperparameters for the real-world experiments are the same as those used in the simulator, as shown in Table 3. We use 16 experts, with the top experts activated.
Considering the performance, MENTOR MoE and MENTOR w/o. TP also consistently outperform MEN- TOR_w/o. TP MoEE, indicating that both the MoE archi- tecture and Task-oriented Perturbation independently con- tribute to improved policy learning.
D.1. Observation Space
The observation space for all real-world tasks is constructed from information only provided by several cameras. Each camera delivers three 84x84x3 images (3-channel RGB, with a resolution 84x84), which capture frames from the beginning. midpoint, and end of the previous action.
Considering efficiency, the following part shows quantita- tive result based on standard training time.
Standard Training Time: Let TMENTOR, TMENTOR__WA SOANTTR__Wo_MWR wOE, and T MENTOR.. TP denote the time required for the three different methods reach the same performance (the final performance of the worst method). The standard training time stundand is defined as the training time for the worst method to achieve this performance:
For the Peg Insertion and Tabletop Golf tasks, the observa- tion space is provided by two cameras: a wrist camera and side camera. As shown in Figure 10, these two cameras in Tabletop Golf offer different perspectives. The wrist cam- era is attached the robot arm's wrist, capturing close-up images of the end-efffcctor, while the side camera provides more global view. As previously mentioned, each camera provides three images, resulting total of six 3-channel 84x84 images.
Tanndard = max(TMENNOR T T T WHNTT
We define normalized sample efficiency as Tiantan
T.
(lower is
13

---

MENTOR: Nixture-of-Expert. Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Table 3. Hyper-parameters used our experiments.
Parameter
Setting
Architecture
Features dimension
100 (Dog)
50 (Others)
Hidden dimension
1024
Number of MoE experts
4 or 32
Activated MoE experts (top-k)
MoE experts hidden dimension
256
Optimization
Optimizer Learning rate
Adam
10 (DMC)
10 (MW & Adroit)
Learning rate of policy network
0.5 Ir or Ir
Agent update frequency Soft update rate
2
0.01
MoE load balancing loss weight
0.002
Perturb
Minimum perturb factor amin
0.2
Maximum perturb factor max
0.6 (Dog, Coffee Push & Soccer)
0.9 (Others)
Perturb rate Oraate
2
Perturb frames Task-oriented perturb buffer size
200000
10
Replay Buffer
Replay buffer capacity
106
Action repeat
2
Seed frames
4000
n-step returns
Mini-batch size
256
Discount ?
0.99
Exploration
Exploration steps
2000
Linear exploration stddev. clip
0.3
Linear exploration stddev. schedule
linear( 1.0, 0.1, 2000000) (DMC)
linear( (1.0, 0.1, 3000000) (MW Adroit)
Awaken exploration temperature T
0.1
Target exploitation parameter À
0.6
Exploitation temperature T"
0.02
Exploitation expectile
0.9
In the Cable Routing task, the observation space is con- structed using three cameras: a side camera for an overview, and two dedicated cameras for each slot to capture detailed views of the spatial relationship between the slots and the cable. This setup results in total of nine 3-channel 84x84 images.
restriction in DOF is specific to our setting: in our case, we train model-free visual reinforcement learning algorithms directly the real-world environment from scratch, without any initial demonstrations and prior knowledge toward the tasks. As a result, the exploration process highly random, and limiting the degrees of freedom is erucial for safeguard- ing both the robotic For instance, in the Peg Insertion task, the use of rigid 3D- printed materials means allowing the end-effector to attempt insertion at arbitrary angles could easily cause damage. Sim- ilarly, in the Cable Routing task, an unrestricted end-effector might collide with the slot, posing risk to the equipment.
and the experimental equipment.
D.2. Action Space
The policy outputs an end-effector delta pose from the cur- rent pose tracked by the low-level controller equipped in robot arm. Typically, the end-effector of robotic arm has six degrees of freedom (DOF); however, in tasks, the action space is constrained to be fewer. The reason for this
Peg Insertion: The end-effector in this task has four degrees
14

---

MENTOR: Ceature-of-Expeert Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Wrist Camera View
Side Camera View
Figure 10. Agent's visual observation example in tabletop golf. MENTOR only uses visual data as policy input. At every step. we capture and stack the frames at the beginning, midpoint, and end the actuation process. The images captured from cameras are resized resolution of 84x84 before heing input to the agent.
of freedom: J, V, 2, and r. Here, and y represent the planar coordinates, represents the height, and denotes the rotation around the z-axis. The y. and dimensions are normalized based on the environment's size, ranging from -1 to 1, while is normalized over feasible rotation range of 0.6T.
Tabletop Golf: The end-effector in this task has three de- grees of freedom: 2, y. and r. Here, and y represent the planar coordinates, and denotes the angle around the normal vector to the zy plane. The and dimensions are normalized based on the environment's s size, ranging from to while r is normalized over a feasible rotation range of 0.5x.
The action space a 4-dimensional continuous space (Ar, Ay, Az, Ar), where each action updates the end- effector's state as:
The action space has four dimensions: three spatial dimen- sions (Ar, Ay, Ar) and strike dimension, where the val- ues range from -1 to 1. The end-effector's state is updated as:
Ar U
Ay 8
Az
Ar 8
(I, (2,,,,,r)
10
Ar
Ay
Ar
v,) - x
Cable Routing: In this task, the end-effector is constrained to two degrees of freedom: I and z. The r-axis controls movement almost perpendicular to the cable, while the z- normalized
10
10
8
and strike > O, the end-effector performs a swing with strength proportional to the value of strike.
axis controls the height. Both dimensions based the environment's size, with values ranging from restrict the action space two di-
D.3. Reward Design
-1 to 1. Although mensions, this task remains extremely challenging for the RI. agent to master, as it requires inserting cable in both slots sequentially, making the most time-consuming task among the three, as shown in Figure 14. The difficulty stems largely from the structure and parallel configuration of the two slots: the agent cannot route the cable into both slots simultaneously and must insert one first. However, as shown in Figure 11b, without hook-like structure to secure the cable in the slot, the cable casily slips out when the agent attempts to route it into the second slot. This task therefore requires highly precise movements, forcing the agent learn the complex dynamics of soft cables.
In this section, we describe the reward functions for the three real-world robotic tasks used in our work: Peg Inser- tion, Cable Routing, and Tabletop Golf. The basic principle behind these functions is measure the distance between the current state and the target state. These reward functions are designed to provide continuous feedback-thuugh they can be extremely sparse, as seen in Cable Routing- on the task's progress, enabling the agent to learn efficient strategies to achieve the goal. Notably, we trained two vi- sual classifiers for the Cable Routing task to determine the relationship between the cables and the slots for reward cal- culation. Other positional information is obtained through feedback from the robot arm image processing algorithms. The lower and upper bounds of each dimension in the pose are normalized to -1 and 1, respectively. The coefficients used in the reward functions are listed in Table 4.
The action space is 2-dimensional continuous space (Az, Az), where each action updates the end-effector's po- sition as:
Ar
Az
(x,2) + x
5
Peg Insertion: The reward is computed as the negative
15

---

MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
absolute difference between the current robot arm pose and the target insertion pose, which varies for each peg-
Partially above the slot: Tstetz
Above the slot and at the edge: Tslot =
Above the slot and close to the middle: Tstot, 5
Partially inside the slot: Yslme 10
Rpog = 2((v2- 11g - -xel) C1 (2- (2-1A=1) C2 =2
1
Fully inside the slot: Tilnz
15
Indicator function that activates only if the cable is inserted correctly in the first slot, allowing the agent to receive rewards for the second slot.
[[ 2 2):
+ [ 10.- -b1)) C3 C4
Where:
The reward consists of two components: the negative absolute distance between the robot arm and the ball, and the negative absolute distance between the ball and the target hole. This encourages the agent to learn how to move the robot arm toward the ball and control the striking force and direction guide the ball toward the hole while avoiding obstacles. Additional rewards include: Rgolt+ = C (if the ball reaches the hole) and Rgalt = C7 (if the ball goes out of bounds). In this experiment, we deploy two cameras at the middle of two adjacent sides of the golf court. The pixel locations of the ball in both cameras used to roughly estimate its location to calculate the reward function. Despite using an approximate estimation for the reward, MENTOR still quickly learns to follow the ball and strike it with the appropriate angle and force, demonstrating the effectiveness of our proposed method.
Tabletop Golf:
X, and x.: Represent the goal position and the current position of the robot's end-effector in the x-y plane.
||, - xell: Euclidean distance between the goal and current positions of the end-effector.
Ae: The height difference between the current and target positions.
0g and 0g: Current and goal angles of the end-effector, respectively.
Cable Routing: To provide continuous reward feedback, we trained a simple CNN classifier to detect whether the cable correctly positioned in the slot, awarding full reward when the cable is in the slot and zero when is far outside. The CNN classifier was trained by labeling images to classify the spatial relationship between the cable and the slot into several categories, with different rewards assigned based the classification. However, when the cable remains in particular category without progressing to different stages, the agent receives constant rewards, making difficult for the agent to learn more refined cable manipulation skills.
Rgolt = (2 - ||Petub Pball ( Cs + (2 P/alll
Phole Il) Cy
T(strike) + (2 - |0bett Ocument) 1) C10
-
max (0, Pban[y] - Pelub [v] 0.05) C11
Table 4. Coefficients used in the reward functions three real-world robotic tasks.
Reable ssso,, + [[Fsttoe 2 2) (Filaty + Cs)
Symbol
Value
Ci
16
Where:
C2
C3
Tstoq Reward for the first slot, determined by the po- sition of the cable relative to the slot. The possible rewards are:
C₁
17
Cs
3
Ce
20
C7
5
Outside the slot: rslot, 3
Cs
4
On the side of the slot: Tslot, = -1
C.
8
Above the slot: Fslot,
Cio
Inside the slot: Talot,
Cn
10
rsley Reward for the second slot, with more detailed classifications:
Where:
Outside the slot: Nstols 3
Pelub and Pean: Positions of the robot's golf club and the ball, respectively.
On the side of the slot: Tstata = -1
16

---

MENTOR: Mixtere-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
(a)
(b)
(c)
Figure 11. Blueprints of the self-designed mechanisms for the three real-world robotic manipulation tasks (from left to right: Peg Insertion, Cable Routing, and Tabletop Golf).
Table 5. Comparison of time efficiency the simulation task (FPS).
Task Name
MENTOR
DrM
DrQ-v2
ALIX
TACO
Hopper
37
55
78
49
23
Phote: Position of the target hole.
tabletop golf device has two layers: the top golf court sur- face and a lower inclined floor. When the ball is hit into the hole or out of bounds, it rolls down to the corner the lower layer, where light sensor triggers a motor to return the ball the court. The variability in the ball's initial velocity during reset introduces randomness its starting position.
PPPutu Poaul|: Distance between the club and the ball.
||alll Phote ||:
Distance between the ball and the hole.
Obea and : Best calculated angle and current an- gle of the robot's arm for optimal striking.
T(strike):
Indicator function that penalizes unnecessary strikes.
E. Time Efficiency of MENTOR
[ba and Palob [y]: The y-axis is the long side of the golf course. The ball should be hit from the positive to the negative y-axis, so the club should always be the positive y-side of the ball.
We run all simulation and real-world experiments on an Nvidia RTX 3090 GPU and assess the speed of the algo- rithms compared to baselines. Frames per second (FPS) is used as the evaluation metric for time efficiency.
For simulation, we use the Hopper Hop task to compare time efficiency. shown in Table 5. While MENTOR demon- strates significant sample efficiency, its time efficiency is relatively lower. This primarily due to the implementation of plain MoE version in this work, where input feature vectors passed to all experts, and only the top-k outputs are weighted and combined generate the final output. In most tasks, the active expert ratio (i.e., top-k/total number of experts) is equal to or below 25%. More efficient implemen- tations of MoE could significantly improve time efficiency, which we leave for future exploration.
D.4. Auto-Resar Mechanisms
One major challenge in real-world RL is the burden of frequent manual resets during training. To address this, we designed auto-reset mechanisms to make the training process more feasible and efficient.
In the Peg Insertion task, the robot arm is set to frequently switch among different pegs help the agent acquire multi- tasking skills. To facilitate this, we design shelf to hold spare pegs while the robot arm is handling one. With the fixed position of the shelf, we pre-programmed a peg- switching routine, eliminating the need for manual peg re- placement. After switching, the robot arm automatically moves the peg to the workspace and randomizes its initial position for training.
We also evaluate time efficiency on three real-world tasks, as shown in Table In real-world applications, the primary bottlenecks in improving time efficiency data collec- tion efficiency and reset speed. Additionally, the sample efficiency of the RL algorithm plays crucial role. If the algorithm has low sample efficiency, it may take many poor actions over a long training period, leading to frequent auto- resets and ultimately lowering the overall FPS.
In the Cable Routing task, manual resets are unnecessary, as the robot arm can auto-reset the cable by simply moving back to its initial position with added randomness.
In the Tabletop Golf task, we design an auto-collection mechanism reset the task. As shown in Figure 11c the
As result, MENTOR and DrM achieve similar levels of
17

---

MENTOR: Mixture-of-Experss Network with Task-Oriented Perturbation for Visual Reinforcement Learning
MTS
MTT
draver-open
xindow-open
4
MT4
faucet-close
Faucet-open
plate-slide
plate-slide-back
Harder 150
MTS
door-open
drawer-close
MTS
MT7
draver-open
xindow-open
xindox-close
burton-press-topple
door-open
draver-close
draver-open
peg-insert-sie
xindow-opan
Runder
Number 6Thomms
xindow-close
MLP
MoE (Duns)
Figure 12. Multi-task performance of MoE Meta-World Simulator: Evaluation accuracy during training for 3, 4, 5, and 7-task combinations, highlighting MoE's advantage over MLP in multitask settings.
Multi-Stage Gradient Cosine Similarity coue
Table 6. Comparison of time efficiency in real-world
tasks (FPS).
Task Name
MENTOR
DrM
Peg Insertion
0.46
0.40
Cable Routing
0.67
0.62
Tabletop Golf
0.52
0.47
efficiency. However, due its superior learning capability, MENTOR quickly acquires skills and transitions out of the initial frequent-reset phase faster than DrM, leading to slightly better overall time efficiency during training.
MME
HLP
Figure 13. Cosine similarity of multistage single task. This figure shows the cosine similarities of gradients on the correspond- ing four stages (Grasp, Move, Assemble, and Release) both MLP and MoE agents.
We further extend the training process of the DrM baseline reach the same performance level MENTOR three real-world experiments with the training time comparison shown in Figure 14, which demonstrates an average 37% im- provement in time efficiency for our method. These findings underscore the importance of each component achieving superior results.
MENTOR
Training Efficiency (hours)
F. Mixture-of-Experts Alleviation Gradient Conflicts in Single Task
In Meta- World, manipulation tasks are associated with com- pound reward functions that typically include components such reaching, grasping. and placing. Conflicts between these objectives arise, creating burden for shared pa- rameters.
Insertion
Cable Rout Real-Worth Tacks
Tabletop Gaf
Figure 14. Time efficiency comparison. This figure compares the training time required for DrM to reach the performance level MENTOR, shown in Table 1.
To validate this, we analyze the gradient cosine similarities for the Assembly task. The Assembly task, as shown in Figure can naturally be divided into four stages: Grasp, Move, Assemble, and Release.
illustrate how Mixture-or of offEExerts alleviates gradient con- flicts in a single task, we evaluate the cosine similarities of
18

---

MENTOR: Nixture-of-Expertt Network with Task-Oriented Perturbation for Visual Reinforcement Learning
1.0
Expert Selection
Start Execution
Disturbance
Success!
Taak
Figure 16. Random disturbances in simulation. This figure shows the execution of the learned agent using MENTOR. The agent consistently accomplishes Assembly task even with the dis- turbance.
0.4
0.2
2
6
10 12
14
Expert ID
1. Random Disturbances in Simulation
To demonstrate the generalization capabilities of the agents trained by MENTOR, we have introduced random distur- bances in the real-world experiments presented in Sec- tion 4.2. Additionally, we make evaluation Meta-World Assembly task with random disturbance. In detail, the train- ing phase remains unchanged, but during evaluation, we introduce random disturbance: after the robot grasps the ring and moves toward the fitting area, the fitting pillar ran- domly changes its location (Disturbance). This forces the robot agent to adjust its trajectory to the new target posi- tion. Figure 16 shows the agent consistently accomplishes Assembly task even with the disturbance, showing the poli- cies learned by MENTOR exhibit strong robustness against disturbances.
Figure 15. Expert utilization on Peg Insertion task. This figure shows the usage intensity of the 16 experts in MENTOR during the Peg Insertion task for three different plug shapes.
gradients on the corresponding four stages for both MLP and MoE agents, as shown in Figure 13. The result show that the MLP agent experiences gradient conflicts between grasping and the other stages. This can oceur because the procedure of reaching grasp objects could increase the distance between the robot and the target pillar, leading to competing optimization signals. In contrast, the MoE agent mitigates these conflicts, achieving consistently positive gra- dient cosine similarities across all stage pairs. This validates ability of the MoE architecture to alleviate the burden of shared parameters and facilitate more efficient optimization, in single-task scenarios.
G. MENTOR in Simulator Multi-Tasking Process
To further demonstrate the multi-task capabilities of MoF, we conducted additional multitask learning experiments on Meta-World. In these experiments, used four task combinations, consisting of 3, 4. 5, and tasks, respectively. Figure 12 shows the evaluation accuracy during training and the detailed composition of the multi-tasks. In this experiment, neither MENTOR nor DrM used perturbation, with the only difference being the use of MoE versus MLP, indicating the effectiveness of MoE in multi multi-tasking process.
H. MENTOR in Real-Worke Kulti-Tasking Process
Figure 15 shows the utilization of experts in the Peg Inser- tion task for various plug shapes. Each shape is handled by some specialized experts, which aids in multi-task leamn- ing. This specialization helps mitigate gradient conflict by directing gradients from different tasks to specific experts, improving learning efficiency, as discussed in the main text.
19


Reviews:
[{"review_id": "Ji3xns86qm", "reviewer": "Reviewer_nuJ2", "summary": "This paper introduces a MOE backbone approach to tackle multi-task visual RL, addressing the problem of conflicting gradients when training on completely opposite tasks (close vs open door). Experimental results show interestingly how the learned model switches between experts smoothly to tackle different sub-tasks of a problem, which further show evidence of some ability to learn across multiple tasks in a single model (with multiple experts). Neural network perturbation is also used to improve exploration of the RL process which mixes the weights with the current weights and some perturbation candidate weights. Sim and real world experiments are performed, with the real world training facilitated by some state estimation model to generate rewards.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Regarding the weakness of real-world RL:\n\nWe agree that real-world RL still faces several limitations. However, we have seen progress in this area. Recent work, such as Serl [1], has released a software suite that facilitates the rapid deployment of real-world learning paradigms. Moreover, recent advances in foundation models [2][3][4] can help simplify the design of perception systems for computing object-relevant rewards. In addition, we would like to emphasize that the main scope of our work is to propose a sample-efficient RL algorithm, which is not limited to real-world RL scenarios.\n\n> Q2: Figure 5 is missing error bars?\n\nThanks for your comment, we will add it to the final camera-ready version.\n\n> Q3: In the real world tasks are there any multi-task setups?\n\nWe trained three separate policies for three real-world tasks, each featuring specific challenges (details are provided in Section 4.2). In particular, the peg insertion task is designed to evaluate multi-task learning capability, as the policy must learn to insert different pegs into corresponding target holes using a single visual policy. We visualized the expert usage heatmap of MENTOR for this task in the original submission **Appendix G**, which shows that the MoE agent tends to utilize different experts for different plug shapes. The ablation study results shown in the original submission **Table 1** also demonstrate the effectiveness of the MoE structure compared to a standard MLP.\n\n\n> Q4: If the experiments in Figure 6 really leverage MOE? Can you show more multi-task experiments?\n\nGenerally speaking, MENTOR is designed to improve the sample efficiency of visual RL, rather than focusing purely on multi-task scenarios. We found that MoE can promote learning efficiency by automatically leveraging distinct experts to decompose a hard task into several sub-tasks and learn them separately. Therefore, the experiments in Figure 6 are trained with a single policy per task. We really appreciate your valuable advice, so we conducted several multi-task learning experiments (detailed multi-task setup information is on [rebuttal Section A](https://mentor-vrl.github.io/)) to demonstrate MoE’s contribution in multi-task setups.\n\n> Q5: Explanation of Task-oriented Perturbation Mechanism:\n\n$\\Phi_{\\text{oriented}}$ is an approximate distribution over a set of high-performing agent weights $S_{\\text{top}}$, which is constructed as training processes. Specifically, we maintain a fixed-size set $S_{\\text{top}} = \\{(\\theta, R)\\}$, which stores the weights $\\theta$ and corresponding episode rewards $R$ of the top-performing agents seen so far. At each episode $t$, if the current agent’s reward $R_t$ exceeds the lowest reward in $S_{\\text{top}}$, we replace the corresponding tuple with $(\\theta_t, R_t)$. For task-oriented perturbation, we approximate $\\Phi_{\\text{oriented}}$ as a Gaussian $\\mathcal{N}(\\mu_\\theta^{\\text{top}}, \\sigma_\\theta^{\\text{top}})$, where the mean and standard deviation are computed over the network weights in $S_{\\text{top}}$. This dynamic update ensures that $\\Phi_{\\text{oriented}}$ remains focused on the most promising regions of the parameter space and generates more effective perturbation candidates $\\phi$ compared with perturbation using random weights.\nA detailed description could be found in the original paper **Section 3.2**.\n\n> Q6: “Why specifically experts 9, 13, 14, and 15 are selected for visualization…”:\n\nThe MoE agent used in this task (Meta-World Assembly) has 16 experts, with the top_k parameter set to 4. Therefore, in Figure 4, we mainly visualize the 4 most active experts during task execution, which happen to be experts 9, 13, 14, and 15. This selection may vary depending on the environment's random seed and the random initialization of expert weights. After training, the agent primarily uses these 4 experts, while the others exhibit low utilization during execution, so we did not include them in the visualization.\n\n-----\n[1] Luo, J.,et.al. Serl: A software suite for sample-efficient robotic reinforcement learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 16961-16969). IEEE.\n\n[2] Liu, S.,et.al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision (pp. 38-55). Cham: Springer Nature Switzerland.\n\n[3] Wen, B.,et.al. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17868-17879).\n\n[4] Wen, B.,et.al. FoundationStereo: Zero-Shot Stereo Matching. arXiv preprint arXiv:2501.09898.", "claims_and_evidence": "Clear", "methods_and_evaluation_criteria": "They make sense and appropriate evaluations are made. Real experiments well appreciated.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "They are sound.", "relation_to_broader_scientific_literature": "They are related to past work on model-free visual RL, working towards more sample-efficient methods which can possibly enable faster real-world RL. Also related to some neural network research on network perturbation for avoiding local minima.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Great experimental results in simulation and real\n- Clear ablation studies on the importance of MOE modelling to address the conflicting gradients problem for better multi-task RL\n- Overall a well written paper\n\nWeaknesses:\n- Not a weakness of this specific paper but with real world RL there are way too many limitations to be practical sometimes that one should be aware of (real world RL in this case requires a dense/sparse reward function which requires a separate perception system usually to compute, and a auto-reset mechanism to be designed).\n-", "other_comments_or_suggestions": "- Figure 5 is missing error bars?", "questions_for_authors": "- In the real world tasks are there any multi-task setups? Is one policy trained to do peg insertion, cable routing and the other tasks? I am wondering where MOE will be helpful in the real world experiments as they otherwise look like single-task.\n- It seems MOE modeling is a big contribution of this paper, however it is unclear to me if the experiments in figure 6 really leverage MOE? Is it a single policy trained per task or one policy trained on all of those tasks? They seem quite a bit different from the motivating experiments with e.g. open/close door tasks in meta world. My best understanding is that some of the harder tasks have multiple sub-tasks which MOE is helping to learn on.\n- The network perturbation process requires already trained expert policies / weights, how are those obtained? Specifically what is $\\Phi_{\\text{oriented}}$? Further reading suggests this is obtained online as training progresses but im not sure if I understood it correctly.\n- Why specifically experts 9, 13, 14, and 15 are selected for visualization? Are they truly interpretable or is this possibly just some magical number/selection. What do the other experts look like?\n\nHappy to raise score if above questions can be clarified.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfrUpyrK80", "reviewer": "Reviewer_xNhQ", "summary": "This paper proposes MENTOR (Mixture-of-Experts Network with Task-Oriented Perturbation) to improve sample efficiency and performance in visual reinforcement learning (RL). MENTOR replaces the policy’s backbone with a mixture-of-experts (MoE) architecture. This MoE design aims to mitigate gradient conflicts in challenging multi-stage or multi-task settings by allocating different parts of the network (“experts”) to specialized subtasks or states. Additionally, the authors introduce a task-oriented perturbation mechanism that periodically “resets” or “perturbs” the policy parameters in a guided fashion—specifically by sampling from a distribution formed by top-performing past agents instead of adding purely random noise. This approach is shown to offer more directed exploration than baseline methods. Empirically, the paper demonstrates strong gains on three simulation benchmarks (DMC, Meta-World, Adroit) and three real-world robotic tasks (peg insertion, cable routing, and tabletop golf), claiming both superior sample efficiency and higher final performance than prior state-of-the-art methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Regarding the use of more recent benchmarks such as LIBERO that emphasize large-scale multi-task and lifelong learning:\n\nWe appreciate the reviewer’s suggestion of more recent benchmarks, like life-long learning tasks. We consider this a valuable avenue for future work. However, we note that this is beyond the current scope of our work, which focuses on evaluating our approach under widely used and established benchmarks and does not consider learning throughout lifetime. Recent works such as Streaming RL [5] and Ace [6] also primarily evaluate on the same or similar settings. Incorporating benchmarks like LIBERO remains an exciting extension for future research direction.\n\n> Q2: Regarding the specificity of the MoE architecture to visual RL:\n\nWe agree with the reviewer that the MoE architecture is versatile and not tied to visual RL. In this work, we focus on visual RL because of its growing attention in research ([6][7][8]) and importance in real-world applications, where agents often rely on visual input. However, we believe that MENTOR is also applicable to state-based RL tasks. To support this, we conducted a fully state-based RL training experiment using Humanoid-Gym [9], where an agent must control a humanoid to accomplish a locomotion task. The comparison results, shown on the [rebuttal Section C](https://mentor-vrl.github.io/), indicate the effectiveness of our method.\n\n> Q3: Regarding the computational overhead of MoE layers:\n\nWe thank the reviewer for pointing this out. We have discussed this in the original submission **Appendix E**: Time Efficiency of MENTOR, while MoE generally improves sample efficiency, it does introduce higher latency compared to standard MLPs due to its routing and expert selection. We acknowledge this tradeoff, but we want to highlight that the latency mostly comes from hardware optimization, not more computation overhead. Recent advances such as MoE-Infinity [10] offer promising directions to significantly reduce this overhead, potentially narrowing the gap to a negligible level.\n\n----\n[5] Elsayed, M.,et.al. Streaming Deep Reinforcement Learning Finally Works. arXiv preprint arXiv:2410.14606.\n\n[6] Ji, T., Liang,et.al. Ace: Off-policy actor-critic with causality-aware entropy regularization. arXiv preprint arXiv:2402.14528.\n\n[7] Hafner, D.,et.al. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.\n\n[8] Laskin, M.,et.al. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning (pp. 5639-5650). PMLR.\n\n[9] Gu, X.,et.al. Humanoid-gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer. arXiv preprint arXiv:2404.05695.\n\n[10] Fu, Y.,et.al. MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems. arXiv preprint arXiv:2412.07067.", "claims_and_evidence": "The main claims in this paper are (1) adopting a mixture-of-experts backbone for the policy helps resolve gradient conflicts that arise in complex RL tasks, and (2) task-oriented perturbation—sampling perturbation candidates from a distribution of well-performing agents—improves exploration and stability over purely random perturbation. Both are well supported by clear and convincing evidence, including didactic examples (e.g., Figure 3, 4, 5), as well as comprehensive experiments in both simulation and real world tasks.", "methods_and_evaluation_criteria": "Yes, the benchmarks such as MetaWorld, DMC, are standard in the visual RL literature. Additionally, real-world results are demonstrated, which are quite impressive.", "theoretical_claims": "This paper does not provide any proofs or theoretical claims.", "experimental_designs_or_analyses": "Yes. The simulated tasks (DMC, Meta-World, Adroit) are each tested under consistent hyperparameters, showing that MENTOR consistently outperforms baselines. The paper uses multiple seeds (usually four) to report average performance. For real-world tasks, the authors design three testbeds that stress multi-task learning (peg insertion with multiple shapes), sequential multi-stage manipulation (cable routing), and dynamic hitting (tabletop golf). The hardware aspects are well thought out (auto-reset mechanisms, camera angles, etc.) so that training remains feasible without an excessive manual setup.", "relation_to_broader_scientific_literature": "The authors build directly on the use of data augmentation in visual RL (e.g., DrQ-v2) and on dormant neuron perturbation in DrM. Their main architectural novelty is bringing the mixture-of-experts concept—long used in large-scale language modeling or multi-task learning—to the standard visual RL pipeline.\nThe results also connect to prior work on multi-task learning where gradient conflicts arise (like conflict-averse gradient descent, gradient surgery, etc.). MENTOR is conceptually aligned with that tradition, providing an RL-specific approach using MoE. Overall, the paper extends known methods (DrQ-v2, DrM, MoE, etc.) in a novel combination well-suited for visual RL.", "essential_references_not_discussed": "All essential related works are discussed to the best of my knowledge.", "other_strengths_and_weaknesses": "Weaknesses:\n1. I feel that some more recent benchmarks that emphasize large-scale multi-task, lifelong learning, such as LIBERO, may be better suited to demonstrate the superiority of the proposed approach. The tested benchmarks are a bit saturated. \n2. The use of MoE architecture may not be specific to visual RL; it would be good to show why visual RL particularly benefits from this approach. Would state-based RL also benefit?\n3. The additional computational burden of the MoE layers is not discussed in details; while MoE can provide sample efficiency, but how does it compare to MLP in terms of wall-clock time as well as compute?", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "My suggestions and questions are listed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfGdHEeqLD", "reviewer": "Reviewer_KF8H", "summary": "This paper aims to improve the performance of reinforcement learning agents in robotic tasks. Addressing the issues of gradient conflicts in standard MLPs for robotic tasks and the tendency of visual RL agents to get stuck in local minima, two key improvements are proposed. First, the Mixture-of-Experts (MoE) architecture is used to replace MLP as the policy backbone, reducing gradient conflicts through a dynamic routing mechanism. Second, a task-oriented perturbation mechanism is designed to sample perturbation candidates from a heuristically updated distribution based on past high-performing agents. Experimental results show that the method based on MoE and the new perturbation mechanism outperforms baseline models on three simulation benchmarks and three challenging real-world robotic manipulation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: “The impact of internal parameter changes on the overall performance may not have been further analyzed in depth… Additionally, there were no relevant ablation experiments regarding the perturbation factor.”:\n\nWe appreciate the reviewer’s attention to this point. Due to time limitations, we only report the ablation study for the number of experts and top_k in the Hammer (Sparse) task, as shown in the [rebuttal Section B](https://mentor-vrl.github.io/). The results indicate that in the Hammer (Sparse) task, the optimal choice for the number of experts is 8 and for top_k is 4. When top_k is 4, there are no significant performance differences when the number of experts is set to 4, 8, or 32, which suggests that 4 experts are enough to learn the skill in this task. The ablation on top_k further validates our hypothesis, as reducing top_k (to 2) results in a worse learning curve. If the number of experts is set to 1 and top_k is also 1, the MoE will downgrade to a standard MLP, resulting in the worst performance among all configurations.\nRegarding the perturbation hyperparameters, we follow DrM [11] by adopting the same perturbation factor and α values from the open-source github repo.\n\n> Q2: “Whether the MoE architecture and the task-oriented perturbation mechanism increase the computational complexity of the algorithm.”:\n\nWe agree that computational efficiency is an important concern. Importantly, both MoE and the perturbation mechanism do not increase the theoretical computational complexity of the algorithm. Empirically, as noted in the original submission **Appendix E**: Time Efficiency of MENTOR, while MoE may introduce higher latency compared to standard MLPs, this is primarily due to hardware-level optimizations (e.g., memory access patterns), rather than additional computation. Recent developments such as MoE-Infinity [12], which demonstrate the potential to reduce MoE latency to negligible levels.\n\n> Q3: “Insufficient in-depth theoretical analysis and mathematical proof”:\n\nWe thank the reviewer for the insightful comment. We acknowledge the importance of a thorough theoretical analysis of how the MoE architecture and task-oriented perturbation help reduce gradient conflicts; however, this is beyond the scope of the present work. Concurrent works also investigate similar patterns through an empirical manner, for example, STGC [13] also applied dense experiments to show the use of MoE to mitigate gradient conflicts for NLP tasks.\n\n----\n[11] Xu, G.,et.al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668.\n\n[12] Fu, Y.,et.al. MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems. arXiv preprint arXiv:2412.07067.\n\n[13] Yang, L.,et.al. Solving token gradient conflict in mixture-of-experts for large vision-language model. arXiv preprint arXiv:2406.19905.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes.\n\n- Extensive experiments were conducted across multiple different tasks and environments, including three simulation benchmarks and three challenging real-world robotic manipulation tasks. Especially in the case of real-world robotics, it encompasses rich aspects such as multi-task learning, multi-stage deformable object manipulation, and dynamic skill acquisition.\n\n- Although ablation experiments were conducted to verify the effectiveness of each component, for some complex components (such as the dynamic routing mechanism in the MoE architecture), the impact of internal parameter changes on the overall performance may not have been further analyzed in depth. For example, the performance differences of different values of (k) (the number of selected experts) in different tasks were not discussed in detail. Additionally, there were no relevant ablation experiments regarding the perturbation factor. What are the bases for determining $\\alpha_{min}$ and $\\alpha_{max}$?\n\n- Moreover, I am also curious about whether the MoE architecture and the task-oriented perturbation mechanism increase the computational complexity of the algorithm.", "relation_to_broader_scientific_literature": "- Previous works (Yu et al., 2020a; Liu et al., 2021) have identified the problem of gradient conflicts in robotic tasks when using shared-parameter architectures like MLPs. In complex robotic scenarios where an agent is assigned multiple tasks or sub-goals, the gradients for optimizing neural parameters across different task stages or between tasks can conflict, hindering the agent's learning ability.\n\n- Previous works (Xu et al., 2023; Ji et al., 2024) have explored using the dormant ratio to determine the perturbation factor $\\alpha$, which improved exploration efficiency. However, these works mainly focused on the perturbation factor and did not thoroughly examine the selection of perturbation candidates.", "essential_references_not_discussed": "Both papers, MENTOR and [1], revolve around the related theme of the multi-expert mixture architecture in reinforcement learning.\n\n[1] Ren, Jie, et al. \"Probabilistic mixture-of-experts for efficient deep reinforcement learning.\" arXiv preprint arXiv:2104.09122 (2021).", "other_strengths_and_weaknesses": "### Strengths\n- **Architectural innovation**: Introduces the MoE architecture into model-free visual RL, replacing MLP as the agent backbone. This design solves the gradient conflict problem in robotic tasks via dynamic routing, offering new ways to boost the agent's learning ability in complex environments.\n- **Perturbation mechanism innovation**: Proposes a task-oriented perturbation mechanism. Sampling from a heuristically updated distribution (based on past high-performing agents) instead of a fixed one makes perturbation more task-relevant and enhances the optimization and exploration in RL.\n\n### Weakness\n- Despite demonstrating MENTOR's effectiveness through experiments, there is insufficient in-depth theoretical analysis and mathematical proof on how the MoE architecture and task-oriented perturbation mechanism reduce gradient conflicts and improve optimization efficiency across different tasks. Relying only on experimental results may not help readers fully grasp the underlying principles.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqZOeDTT4o", "reviewer": "Reviewer_akYq", "summary": "This paper proposes to improve the sampling efficiency in RL learning by mixture-of-experts (MoE) network design and dormant-based parameter perturbation. The MoE design aims to decrease gradient conflicts and perturbation from top-k agent's parameter helps to accelerate the learning. Experiments in simulator and real world proves the efficiency and effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Evaluation on Multi-task Learning:\n\nWe appreciate the reviewer’s suggestion regarding multi-task evaluation. We have conducted additional experiments where our method is trained jointly on Meta-World tasks using a single policy. The performance is compared against MLP baseline under the same multi-task setup. Experimental details and results are provided on the [rebuttal Section A](https://mentor-vrl.github.io/).\n\n> Q2: Cut-off Frames of different tasks:\n\nThank you for your comment. Our frame settings follow prior works such as DrQv2 [14], TACO [15], and DrM [16], and are chosen to reflect task difficulty while maintaining consistency in comparison. We agree that cut-off frames should account for whether baselines have converged. As noted, our main comparisons are against DrM [16], which consistently converges by the reported cut-off in most tasks.\n\n> Q3: “While the design of two main contributions makes sense separately, I feel that they are contradicting each other.”:\n\nThanks for pointing out your confusion. We will explain why these two main contributions do not contradict each other.\n\n**For MoE:** we would like to firstly clarify the different concepts of *agent* and *expert*. In our context, *agent* refers to the policy agent, whose backbone is a MoE, and MoE consists of several experts. So the assumption of MoE in your original comment — “different agents will master different skills to avoid gradient conflicts” — should be rephrased as “different experts will master different skills to avoid gradient conflicts”.\n\n**For Task-oriented Perturbation Mechanism:** During RL training, we maintain a fixed-size set $S_{\\text{top}} = \\{(\\theta, R)\\}$, which stores the weights $\\theta$ and corresponding episode rewards $R$ of the top-performing agents seen so far. At each episode $t$, if the current agent’s reward $R_t$ exceeds the lowest reward in $S_{\\text{top}}$, we replace the corresponding tuple with $(\\theta_t, R_t)$. \n\nSuppose now we have $N$ history top-performing agent weights (each has $M$ experts) in $S_{\\text{top}}$ and we want to execute task-oriented perturbation, the goal distribution to perturbing $\\mathrm{expert}_{i}$ is $\\mathcal{N}(\\mu_i, \\sigma_i)$, where $\\mu_i$ and $\\sigma_i$ are the mean and std of\n\n{exper$\\text t_i^{\\theta_k}$ $\\mid \\theta_k \\in S_{\\text{top}}$},\n\nin these $|S_{\\mathrm{top}}|$ agents, and has nothing to do with other experts. So the perturbation process will not make experts similar but will further diversify them from each other.\n\nWe hope this is sufficient to clarify that these two designs are not contradictory. More details could be found in the original submission **Section 3.2** and feel free to ask if you have more questions!\n\n> Q4: Generalization and Robustness of MoE Design:\n\nWe understand the reviewer’s concern regarding the robustness of the MoE design to hyperparameter choices. To this end, we conducted experiments on the Hammer (Sparse) task, as shown in the [rebuttal Section B](https://mentor-vrl.github.io/) (due to the time limitations, we only report the ablation study on this task), varying the number of experts and top_k. Results show that while the optimal setting is MoE has 8 experts, performance remains consistent across 4, 8, and 32 experts as long as top_k = 4. This suggests the model is not overly sensitive to the number of experts. \n\n----\n[14] Yarats, D.,et.al. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645.\n\n[15] Zheng, R., Wang, X., Sun, Y., Ma, S., Zhao, J., Xu, H., ... & Huang, F. (2023). TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning. Advances in Neural Information Processing Systems, 36, 48203-48225.\n\n[16] Xu, G.,et.al. Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:2310.19668.", "claims_and_evidence": "The main claim of this paper is that the introduced method improves the sampling efficiency in RL, which can be validated in two aspects. Firstly, the expert usage intensity distribution shows that different experts are in charge of different skills, which alleviates gradient conflicts. Besides, the parameter perturbation leads to faster learning in experiments and ablations. Therefore, the main claim is supported.", "methods_and_evaluation_criteria": "Yes. The MoE design and perturbation aims to solve the sampling inefficiency and sparsity in reward. The evaluation uses episode rewards or successful rate as metric.", "theoretical_claims": "Do not apply.", "experimental_designs_or_analyses": "While the overall evaluation is valid with diverse tasks and environments, I have two concerns.\n\nFirstly, it seems that multi-tasking is the advantage of the proposed method. However, most of the evaluation is on single task. Why not evaluating on multi-task learning (e.g., leaning on all DMC tasks with one model). Besides, the cut-off frames of different task varies a lot (e.g., 2m and 30m). While the complexity of different tasks varies, I believe it's better to show the results after enough frames, especially when considering that the baselines are not converging in some tasks as shown in the figures.", "relation_to_broader_scientific_literature": "It's related to RL in robot manipulation and locomotion. Specifically, robot RL when reward is sparse and exploration space is huge.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### Strengths\n\n1. The motivation of improving sampling efficiency in robot RL makes sense.\n\n2. The design of MoE to distribute learning burden and decrease gradient conflicts makes sense. The parameter perturbation from top agents empirically accelerate the learning process.\n\n2. The writing is clear and easy to follow.\n\n### Weaknesses\n\n1. While the design of two main contributions makes sense separately, I feel that they are contradicting each other. The assumption of MoE is that different agents will master different skills to avoid gradient conflicts, thus improving data efficiency. However, the parameter perturbation uses the mean value of parameters of top k agents. This design implies two messages. First, each agent can independently perform the task, which is not what the usage intensity distribution in single task shows. Furthermore, It means that the agents are somehow similar to each other, which contradict to the benefit of using MoE. I think the two designs are based on contradicting assumptions.\n\n2. I am also concerned about the generalization of MoE model design. The number of experts could be highly dependent on the number and complexity of tasks, which limits the application of this approach on more diverse tasks.", "other_comments_or_suggestions": "1. Evaluation on more tasks and combination of tasks (as said in previous section) could further prove the effectiveness of the method.\n\n2. Please explain more on the two designs as in Weaknesses.", "questions_for_authors": "Please see the Weaknesses and Evaluation designs. I am willing to raise my rating if the concerns are solved.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles multi-task reinforcement learning by utilizing a Mixture-of-Experts (MoE) backbone and introducing a task-oriented perturbation mechanism.

Strengths:

* Intuitive and reasonable idea to tackle hard problem
* Strong experimental results in both simulation and real-world settings
* Clear and well-structured writing

Weaknesses:
* Latency introduced by the MoE architecture
* Lack of theoretical analysis

Overall, I believe this paper makes a meaningful contribution and will be valuable to the community
Decision: Accept (poster)