Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
Authors: Ahmed Hendawy
Jan Peters
Carlo D'Eramo
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Multi-Task Learning
Mixture of Experts
Primary area: reinforcement learning
TL;DR: A novel method for Multi-Task Reinforcement Learning that encourages diversity across the shared representations extracted by a mixture of experts.

Abstract: Multi-Task Reinforcement Learning (MTRL) tackles the long-standing problem of endowing agents with skills that generalize across a variety of problems. To this end, sharing representations plays a fundamental role in capturing both unique and common characteristics of the tasks. Tasks may exhibit similarities in terms of skills, objects, or physical properties while leveraging their representations eases the achievement of a universal policy. Nevertheless, the pursuit of learning a shared set of diverse representations is still an open challenge. In this paper, we introduce a novel approach for representation learning in MTRL that encapsulates common structures among the tasks using orthogonal representations to promote diversity. Our method, named Mixture Of Orthogonal Experts (MOORE), leverages a Gram-Schmidt process to shape a shared subspace of representations generated by a mixture of experts. When task-specific information is provided, MOORE generates relevant representations from this shared subspace. We assess the effectiveness of our approach on two MTRL benchmarks, namely MiniGrid and MetaWorld, showing that MOORE surpasses related baselines and establishes a new state-of-the-art result on MetaWorld.

Published as a conference paper at ICLR 2024
MULTI-TASK REINFORCEMENT LEARNING WITH MIXTURE OF ORTHOGONAL EXPERTS
Ahmed Hendawyl.23 Jan Peters Carlo D'Eramo1.25
1 DDpaartment of Computer Science, TU Darmstadt, Germany
Hessian Center for Artificial Intelligence (Hessian.ai), Germany
*Center for Cognitive Science, TU Darmstadt, Germany
*German Research Center for AI (DFKI), Systems AI for Robot Learning
'Center for Artificial Intelligence and Data Science, University of Würzburg, Germany
ABSTRACT
Multi-Task Reinforcement Learning (MTRL) tackles the long-standing problem of endowing agents with skills that generalize across variety of problems. To this end, sharing representations plays a fundamental role in capturing both unique and common characteristics of the tasks. Tasks may exhibit similarities in terms of skills, objects, or physical properties while leveraging their representations eases the achievement of a universal policy. Nevertheless, the pursuit of learn- ing shared set of diverse representations is still open challenge. In this paper, we introduce a novel approach for representation learning in MTRL that encapsu- lates common structures among the tasks using orthogonal representations pro- mote diversity. Our method, named Mixture Of Orthogonal Experts (MOORE), leverages Cram-Schmidt process to shape shared subspace of representations generated by a mixture of experts. When task-specific information is provided, MOORE generates relevant representations from this shared subspace. We assess the effectiveness of our approach on two MTRL benchmarks, namely MiniGrid and Meta World, showing that MOORE surpasses related basclines and establishes a new state-of oftthe-art result on Meta Horld.
INTRODUCTION
Reinforcement Learning (RL) has shown outstanding achievements in wide array of decision- making problems, including Atari games (Mnih et al., 2013; Hessel et al., 2018a) 8a), board games (Sil- ver et al., 2016; 2017), high-dimensional continuous control (Schulman et al., 2015; 2017; Haarnoja et al., 2018), and robot manipulation (Yu et al., 2019). Despite the success of RL, generalizing the learned policy to a broader of related tasks remains an open challenge. Multi-Task Reinforce- ment Learning (MTRL) is introduced to scale up the RL framework, holding the promise of enabling learning& a universal policy capable of addressing multiple tasks concurrently. To this end, sharing knowledge is vital in MTRL (Teh et al., 2017; D' E'Eamo et al., 2020; Sodhani et al., 2021; Sun et al., 2022). However, deciding upon the kind of knowledge to share and the set tasks to share that knowledge is crucial for designing an efficient MTRL algorithm. Human beings exhibit remarkable adaptability across a multitude of tasks by mastering some essential skills as well as having the intuition of physical laws. Similarly, MTRL can benefit from sharing representations that capture unique and diverse properties across multiple tasks, easing the learning of an effective policy.
Recently, sharing compositional knowledge (Devin et al., 2017: Calandriello al., 2014; Sodhani et al., 2021; Sun et al., 2022) has shown potential as an effective form of knowledge transfer in MTRL. For example, Devin al. (2017) investigate knowledge transfer challenges between distinct robots and tasks by sharing a modular policy structure. This approach leverages task-specific and robot-specific modules, enabling effective transfer of knowledge. Nevertheless, this approach re- quires manual intervention to determine the allocation of responsibilities for each module, given some prior knowledge. contrast, we aim for an end-to-end approach that implicitly learns
Ahhmed Hendawy (ahmed. hww.hhy@tuddaaa rmstadt. de) is the corresponding author. 'The code is available at https: htt.thtttttttcccg/gggo/gg con/ httm:///meMaaallllaaaaa///oo/ooo /MOORK..

---

Published as a conference paper at ICLR 2024
and shares the prominent components of the tasks for acquiring universal policy. Furthermore, CARE (Sodhami et al., 2021) adopt a different strategy by focusing on learning representations of different skills and objects encountered by the tasks by utilizing context information. However, there is no inherent guarantee of achieving diversity among the learned representations. In this work, our goal is to ensure the diversity of the learned representations maximize the representation capacity and avoid collapsing to similar representations.
Consequently, we propose a novel approach for representation learning in MTRL to share a set of representations that capture unique and common properties shared by all the tasks. To ensure the richness and diversity of these shared representations, our approach solves a constrained opti- mization problem that orthogonalizes the representations generated by mixture of experts via the application of the Gram-Schmidt process, thus favoring dissimilarity between the representations. Hence, we name our approach, Mixture Of ORthogonal Experts (MOORE). Notably, the orthogo- nal representations act as bases that span a subspace of representations leveraged by all tasks where task-relevant properties can be interpolated. More formally, we show that these orthogonal represen- tations are a set of orthogonal vectors belonging to a particular Riemannian manifold where the inner product is defined, known as Stiefel manifold (James, 1977). Interestingly, the Stiefel manifold has recently drawn substantial attention within the field of machine learning (Ozay & Okatani, 2016; Huang et al., 2018a; Li et al., 2019; Chaudhry et al., 2020). For example, several works focus on enhancing the generalization and stability of neural networks by solving an optimization problem to learn parameters in the Stiefel manifold. Another line of work aims to reduce the redundancy of the learned features by forcing the weights to inhabit the Stiefel manifold. Additionally, Chaudhry et al. (2020) propose a continual learning method that forces each task to learn in a different subspace, thus reducing task interference through orthogonalizing the weights.
In this paper, our objective is to ensure diversity among the shared representations across tasks by imposing a constraint that forces these representations to exist within the Stiefel manifold. Thus, we aim to leverage the extracted representations, in combination with deep RL algorithms, to en- hance the generalization capabilities of MTRL policies. In the following, mathematical formulation of the MTRL problem, inspired by Sodhani et al. (2021), where latent representations belong the Stiefel manifold. Then, we devise our MOORE approach for ob- taining orthogonal task representations through the application of a Gram-Schmidt process on the latent features extracted from mixture of experts. We empirically validate MOORE on two widely used and challenging MTRL problems, namely MiniGrid (Chevalier-Boisvent et al., 2023) and Meta- World (Yu et al., 2019), comparing to recent baselines for MTRL. Remarkably, MOORE establishes a new state-of sfute-of-the-ar performance on the MetaWorld MT10 and MT50 collections of tasks.
provide a rigorous
To recap, the contribution of this work is twofold: (i) We propose a mathematical formulation, named Stiefel Contextual Markov Decision Process (SC-MDP). that defines the MTRL problem where the state encoded in the Stiefel manifold through mapping function. (ii) We devise a novel representation learning method for Multi- Reinforcement Learning that leverages a mod- ular structure of the shared representations to capture common components across multiple tasks. Our approach, named MOORE, learns mixture of orthogonal experts by encouraging diversity through the orthogonality of their corresponding representations. Our approach outperforms related baselines and achieves state-of-the-an results on the MetaWorld benchmark.
PRELIMINARIES
A Markov Decision Process (MDP) (Bellman, 1957; Puterman, 1995) is a tuple M =< r.
S S,A, Prr, P,7 >, where S is the state space, A is the action space, P S A - S is the transition
distribution where P(s [s, a) is the probability of reaching when being in state and performing action a, r S x A + R is the reward function, p is the initial state distribution, and 7 E (0, 1] is the discount factor. A policy
maps each state to a probability distribution over the action space A. The goal of RL is to learn a policy that maximizes the expected cumulative discounted return ((x) = R. 2 7 7'r(s,, a.)]. We parameterize the policy no (dress) and optimize the parameters 0 to maximize (TT) = J(0).
2.1 MULTI-TASK REINFORCEMENT LEARNING
In MTRL, the agent interacts with different tasks J E T, where each task T is a different MDP M =< S A PT, rt, p x The goal of MTRL is to learn single policy n that maximizes
2

---

Published as a conference paper at ICLR 2024
the expected accumulated discounted return averaged across all tasks J(0) = ΣΣ, (f). Tasks can differ in one or more components of the MDP. A class of problems in MTRL assumes only a change in the reward function r". This can be exemplified by navigation task where the agent learns to reach multiple goal positions or a robotic manipulation task where the object's position changes. In this class, the goal position is usually augmented to the state representation. Besides the reward function, a bigger set of problems deals with changes in other components. In this category, tasks access a subset of the state space S while the true state space S is unknown. For example, learning a universal policy that performs multiple manipulation tasks interacting with different objects (Yu et al., 2019). Task information should be provided either in the form of task ID (e.g.. one-hot vector) or metadata, e.g., task description (Sodhani et al., 2021).
Following Sodhani et al. (2021), we define the MTRL problem as a Block Contextual Markov Deci- sion Process (BC-MDP). It is defined by 5-tuple < C, S, A, 7, 7 > where is the context space, is the true state space, A is the action space, while M is a mapping function that provides the task-specific MDP components given the context c E C, M (c) = {r", S ; As of now, we refer the task and its components by the context parameter denoted c.
RELATED WORKS
Sharing knowledge among tasks is key benefit of MTRL over single-task learning, as broadly ana- lyzed by several works that propose disparate ways to leverage the relations between tasks D'Eramo et al., 2020; Sodhani et al., 2021: Sun et al., 2022; Calandriello et al., 2014; Devin et al., 2017; Yang et al., 2020). Among many. D'Eramo et al. (2020) establish theoretical benefit of MTRL over single-task learning as the number of tasks increases, and Teh et al. (2017) learn individual policies while sharing prior among tasks. However, naive sharing may exhibit negative transfer since not all knowledge should be shared by all tasks. An interesting line of work investigates the task in- terference issue in MTRL from the gradient perspective. For example, Yu et al. (2020) propose gradient projection method where each task's ' gradient is projected to an orthogonal direction of the others. Nevertheless, these approaches are sensitive to the high variance of the gradients. Another approach, known as PopArt (Hessel et al., 2018b), examines task interference focusing on the insta- bility caused by different reward magnitudes, addressing this issue by normalizing technique on the output of the value function.
Recently, sharing knowledge in a modular form has been advocated for reducing task interference. Yang et al. (2020) share base model among tasks while having a routing network that generates task-specific models. Moreover, Devin et al. (2017) divide the responsibilities of the policy by sharing two policies, allocating one to different robots and the other to different tasks. Additionally, Sun al. (2022) propose a parameter composition technique where a subspace of policy is shared by a group of related tasks. Moreover, CARE Sodhani et al. (2021) highlight the importance of using metadata for learning a mixture of state encoders shared among tasks, based on the claim that the learned encoders produce diverse and interpretable representations through attention mechanism. Despite the potential of this work, the method is highly dependent on the context information as shown in this recent work (Cheng et al., 2023). However, we argue that all of these approaches lack the guarantee of learning diverse representations.
In this work, we promote diversity across a mixture of experts by enforcing orthogonality among their representations. The mixture-of-experts has been well-studied in the RL literature (Akrour al., 2021; Ren al., 2021). Moreover, some works dedicate attention maximizing the diversity of the learned skills in RL (Eysenbach et al., 2018). Previous works leverage orthogonality for dis- parate purposes (Mackey et al., 2018). For example, Bansal et al. (2018) promote orthogonality on the weights by a regularized loss to stabilize training in deep convolutional neural networks. Simi- larly, Huang et al. (2018a) employ orthogonality among the weights for stabilizing the distribution of activation in neural networks. In the context of MTRL, Paredes et al. (2012) enforce the represen- tation obtained from a set of similar tasks to be orthogonal to the one obtained from selected tasks known to be unrelated. Recently, Chaudhry et al. (2020) alleviate catastrophic forgetting in con- tinual learning by organizing task representations in orthogonal subspaces. Finally, Mashhadi et al. (2021) favor diversity in an ensemble of learners via Gram-Schmiae process. As opposed to it, our primary focus lies in the acquisition of a set of orthogonal representations that span a subspace shared by group of tasks where task-relevant representations be interpolated.
3

---

Published as a conference paper at ICLR 2024
Tisk Encoday
Gram- Schmidk
V
Mixture Experts
Sticfel Manifold
Oulput Head
Figure 1: MOORE illustrative diagram. A state is encoded as a set of representations using mixture of experts. The Gram-Schmidt process orthogonalizes the representations encourage diversity. Then, the output head processes the representations V, by interpolating the task-specific representations Ve using the task tasc-ipecific weights w/s from which we compute the output using the output function fo- In our approach, we employ this architecture for both the actor and the critic.
4 SHARING ORTHOGONAL REPRESENTATIONS
We aim to obtain set of rich and diverse representations that can be leveraged to find a universal policy that accomplishes multiple tasks. this end, we propose enforce the orthogonality of the representations extracted by mixture of experts.
the following, we first provide mathematical formulation from which we derive our approach. In particular, we highlight the connection between method and the Stiefel manifold theory (Huang et al., 2018b; Chaudhry et al., 2020; Li et al., 2020), together with the description of the role played process. Then, proceed devise our novel method for Multi-Task Rein-
by the forcement Learning on orthogonal representation obtained from a mixture of experts.
4.1 ORTHOGONALITY IN CONTEXTUAL MARKOV DECISION PROCESSES
We study the optimization of policy म. given a set of fe-orthonormal representations in Rd for the state We define the orthonormal representations of state as matrix V, = [e].... Unl E daxk where vi E I2d < k. can be shown that the orthonormal representations V, belong topological space known the Stiefel manifold, smooth and differentiable manifold largely used in machine learning (Huang et al., 2018b; Chaudhry al., 2020; Li et al., 2020).
Definition 4.1 (Stiefel Manifold) Stiefel manifold V& (R is defined as the set of all orthonormal k-vectors in the Euclidean space Rd,, where
V4 (R(R) (V. E Bdxk VT IR, Vs E S).
Under this lens, our goal can be interpreted as finding set of orthogonal representations belonging to the Stiefel manifold that capture the common characteristics in the true state space S. Thus, we propose novel MDP formulation for MTRL, which call Stiefel Contextual Markov Deci- sion Process (SC-MDP), that inspired by the BC-MDP introduced in Sodhani et al. (2021). An SC-MDP includes function that maps the state to k-orthonormal representations V, E VA (R.).
Definition 4.2 (Stiefel Contextual Markov Decision Process) A Stiefel Contextual Markov Decision Process (SC-MDP) is defined as tuple < C, AM , > where C is the context space, S is the true state space, A is the action space. M is function that maps context C to MDP parameters and observation space M (c) {r Se Sp.p),) ) is a function that maps every state s E S to a k-orthonormal representations V, V4 (Bd), V p(s).
We define our MTRL policy as <als, c) fo(((s) w.), where w. IB* is the task-specific weight that combines the k-orthogonal representations into a task-relevant one and fe Rd + RIA is an output function with learnable parameters 0 that generates actions from task-specific representations. To leverage diverse set of representations across tasks, the mapping function y
4

---

Published as a conference paper at ICLR 2024
plays a fundamental role. Hence, approximate yp by mixture of experts h [ ..... houl with learnable parameters o [ ox] that generate k-representations U, E _d for state s, while ensuring that the generated representations are orthogonal to enforce diversity. Conveniently, this objective finds rigorous formulation as constrained optimization problem where impose a hard constraint to enforce orthogonality:
max
J(O)
e= (=.0)
(I)
(+) (o) Ix As E
S.t.
where I& R is the identity matrix. Instead of solving the constrained optimization problem in Eq. 1, we ensure the diversity across experts through the application of the Gram-Schmidt (GS)
process to orthogonalize the k-representations U..
Definition 4.3 (Gram-Schmiee Process) Gram-Schmidt process is a method for orthogonalizing set of linearly independent U {u1, ... up U4 Rd, Vi < k}. It maps the vectors to set of k-orthonormal vectors V = {UI, .... vs E IRd, vi S k} defined as
(wi, ux) (us, 04) Us.
2k uk
where the representation of the i-th expert u, is projected in the orthogonal direction to the sub- space spanned by the representations of all 1 experts. Therefore, we apply the GS process to map the generated representations by the mixture of experts U, ho(s) to set of orthonormal representations V2 = GU,( satisfying the hard constraint in Eq.
4.2 MULTI-TASK REINFORCEMENT LEARNING WITH ORTHOGONAL. REPRESENTATIONS
Following our policy π (a|s, c). each task can interpolate its relevant representation from the sub- space spanned by the k-orthonormal representations Vg. We train a task encoder to produce the task-specific weights w. R given task information (e.g. task ID). The orthonormal representa- tions are combined using the task-specific weight to produce relevant representations Ue E Rd to the task as Ue V, w.. The interpolated representation v. captures the relevant components of the task that can be utilized by the RL algorithm and fed to an output function fa. The output function can be learned for each task separately (multi-head) or shared by tasks (single-head) to compute the action components given the representations Vg. Similarly, the same policy (actor) structure (Alg, 1) can be used for the critic (Alg. 2). In conclusion, this approach results in a Mixture Of ORthogonal Experts, thus, we call it MOORE, whose extracted representation is used to learn universal policy for MTRL. A visual demonstration of our approach is shown in Fig.
We adopt two different RL algorithms, namely Proximal Policy Optimization (PPO) and Soft Actor- Critic (SAC), with the purpose demonstrating that our approach agnostic to the used RL algo- rithms. PPO (Schulman et al., 2017) is policy gradient algorithm that has the merit of obtaining satisfactory performance in a wide range of problems while being easy to implement. It is a first- order method that enhances the policy update given the current data by limiting the deviation of the new policy from the current one. Moreover, we integrate our approach to SAC, high-performing off-policy RL algorithm that leverages entropy maximization to enhance exploration.
5 EXPERIMENTAL RESULTS
In this section, we evaluate MOORE against related baselines on two challenging MTRL bench- marks, namely MiniGrid Shhevalier-ooivvee et al., 2023), set of visual goal-oriented tasks, and MetaWorld (Yu et al., 2019), collection of robotic manipulation tasks. The objective is to assess the adaptability of our approach in handling different types state observations and tackling a variable number of tasks. Moreover, the flexibility of MOORE is evinced by using it for on-policy (PPO for MiniGrid) and off- off-policy RL (SAC for MetaWorld) algorithms. Additionally, we conduct ablation studies that support the effectiveness of MOORE in various aspects. We assess the following points: the benefit of using Schmid to impose diversity across experts, the quality of the learned representations, as well the transfer capabilities, and the interpretability of the diverse experts.
5

---

Published as a conference paper at ICLR 2024
MT3
MTS
MT7
Average Return
Average Return
Average Return
0.6
0.6
0.4
0.2
0.2
0.0
0
100
100
100
WEpochs
WEpochs
WEpochs
MTPPO
MOE
MOORE (ours)
(a) Multi Head Architecture
MT3
MTS
MT7
0.6
0.6
Average Return
Average Return
Average Retur
0.6
0.4
0.4
0.2
0.0
0
100
100
25
100
#Epochs
#Epochs
#Epochs
PCGrad
MOORE (ours)
(b) Single-Head Architecture
Figure 2: Average return on the three MTRL scenarios of MiniGrid. We utilize both multi-head and single-head architectures for our approach MOORE as well as the related baselines. For MOORE, MOE and PCGrad, the number of experts & is 2, 3, and 4 for MT3, MT5, and MT7, respectively. The black dashed line represents the final single-task performance of PPO averaged across all tasks. For the evaluation metric, compute the accumulated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
5.1 MINIGRID
We consider different tasks in MiniGrid (Chevalier-Boisverree al., 2023), a suite of 2D goal-oriented environments that requires solving different mazes while interacting with objects like doors, keys, or boxes of several colors, shapes, and roles. MiniGrid offers visual representation of the state, which we adopt for multi-task setting. We consider the multi-task setting from Jin et al. (2023) that includes three multi-task scenarios. The first scenario, MT3, involves the three tasks: LavaGap, RedBlueDoors, and Memory; the second scenario, MT5, includes the five tasks: DoorKey, Lava- Gap, Memory, SimpleCrossing, and MultiRoom. Finally, MT7 comprises the seven tasks: DoorKey, DistShift, RooBl LavaGap, Memory, SimpleCrossing, and MultiRoom. In Sec. A.1, provide descriptions and more details for the tasks.
We compare MOORE against four baselines. The first one is PPO, considered reference for comparing to single-task performance. The second baseline is Multi-Task PPO (MTPPO), an adaptation of PPO (Schulman al., 2017) for MTRL. Then, we consider MOE, which employs a mixture of experts to generate representations without enforcing diversity across experts. Ad- ditionally, we have PCGrad (Yu et al., 2020), which is an MTRL approach that tackles the task interference issue by manipulating the gradients. We integrate PCGrad on top of the MOE base- line for a fair comparison. As for the MTRL architecture, we utilize multi-head and single-head architectures for all methods, showing their average return across all tasks in Fig. 2(a), and Fig. 2(b) respectively. MOORE outperforms the aforementioned baselines in almost all the MTRL scenar- ios. Notably, our method exhibits faster convergence than the baselines. It is interesting to observe that MOORE outperforms the single-task performance with a significant margin in comparison to the other baselines (Fig.2(a)), which is usually considered mance in previous works. This highlights the quality the learned representations and the role of MOORE representation learning process in MTRL.
upper-bor bound of the MTRL perfor-

---

Published as a conference paper at ICLR 2024
MT3 MTS
MTS - MT7
1.00
0.8
Average Return
0.75
0.6
0.50
Average
0.4
0.25
0.2
0.00
60
#Epochs
#Epochs
MOE
Transfer-MOE
MOORE (ours)
Transfer-MOORE (ours)
Figure 3: Evaluating MOORE against MOE on the transfer setting. The study is conducted on the two transfer learning scenarios in MiniGrid, employing multi-head architecture. The number of experts is and 3 for MT3 + MTS and MTS + MT7, respectively. For the evaluation metric, compute the accumulated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
5.1.1 ABLATION STUDIES
0.7
Transfer Learning. We examine the advantage of transferring the trained experts
Average Retur
MOORE (ours)
of base tasks to novel tasks in order to assess the quality and generalization of these learned ex- perts in comparison to the MOE baseline. We refer the transfer variant of our approach as Transfer-MOORE while Transfer-MOE for the baseline. Moreover, we include the perfor- mance of MOORE and MOE as a MTRL refer- ence for learning the novel tasks from scratch, completely isolated from the base tasks. In Fig. 3, we show the empirical results on two transfer learning scenarios where we transfer set of experts learned on MT3 to MT5 (MT3 - MT5) and on MT5 to MT7 (MT5 - MT7). MT3 is a subset of MTS, while MT5 is subset of MT7. First, then we transfer the learned experts (frozen) to the novel tasks (the difference between the two sets). As illustrated in Fig. Transfer-MOORE outperforms Transfer-MOL in the context of transfer learning. Moreover, the study demonstrates the ability of our approach as an effective MTRL algorithm that provides competitive results against the transfer variant (Transfer- MOORE). contrast, MOE struggles to beat the transfer variant as in the MT3 - MT5 scenario. Consequently, this study advocates the diversification of the shared representations in transfer learn- ing and MTRL. We highlight more details in B.2.
0.6
0.5
0.4
3
4
5
6 7
10
Number Experts
Figure 4: Ablation study on the effect of changing the number of experts. We compare the perfor- mance of MOE and MOORE (ours) on MiniCrid MT77 using single-head architecture. We report the mean of the evaluation metric across 30 seeds. For the evaluation metric, compute the accu- mulated return averaged across all tasks.
train the base tasks, and
in the two scenarios, showing the quality of the learned representations
Number of Experts. Additionally, we focus the impact of changing the number of experts the performance of our approach, as well as on MOE. Fig. 4, we consider different numbers of experts the MT7 scenario. We observe the effect of utilizing more experts in MOORE algorithm compared to MOE. The study shows that MOORE exhibits a noticeable advantage, on average, for increasing number of experts. On the contrary, slower enhancement of the performance encountered by MOE. It is also worth noting that the performance of MOORE with & = 4 slightly outperforms MOE with & 10 while being comparable MOE with i: (MOE best setting). This supports our claim about efficiently utilizing expert capacity through enforcing diversity.
5.2 METAWORLD
Finally, evaluate approach another challenging MTRL setting with large number of manipulation tasks. We benchmark against MetaWorld (Yu et al., 2019), a widely adopted robotic manipulation benchmark for Multi-Task and Meta Reinforcement Leamning We consider the MT10

---

Published as a conference paper at ICLR 2024
Total Env Steps
1M
2M
3M
5M
10M
15M
20M
SAC (Yu ct al., 2019)
10.0+8.2
17.7+2.1
18.7+1.1
20.0-2.0
48.0=
57.7+3.1
61.9+3.3
MTSAC (Yu et al., 2019)
34.9
49.349.0
57.14988
602--96
61.626.7
65.61100.4
62.948.0
SAC + FiLM (Perez ct al., 2017)
32.7+6.5
46.919.4
52.916.4
572-422
59.7.4.6
61.745.4
58.3+4.3
PCGrad (Yu et al., 2020)
32.2.1688
46.619.3
54.0184
60.2-9.7
62.6111.0
62.61
61.7:1009
Soft-Module (Yang et al., 2020)
24.2.14.8
41.012.9
47.415.3
514-68
53.6.49
56.614888
63.014.2
CARE (Sodhani et al., 2021)
26.0+9.1
52.6+9.3
63.8+7.9
66.5-8.3
69.8-5.1
72.2+7.1
76.0+6.9
PaCo (Sun et al., 2022)
30.5+9.5
4.28+
65.7444.5
64.7
71.0+5.5
81.0+5.9
85,4+4.5
MOORE (ours)
37.219,9
63.0172
68.616.9
77.3_9.6
82.7773
88.215.6
88.715.6
Table 1: Results MetaWorld MT (Yu et al., 2019) with random goals (MT10-rand) The results of the baselines are from Sun et al. (2022). MOORE uses k = 4 experts. For all methods, we report the mean and standard deviation of the evaluation metric across 10 different runs. The evaluation metric is the average success rate across all tasks. We highlight with bold text the best result.
and MT50 settings, where a single robot has to perform 10 and 50 tasks, respectively. For the baselines, et al., 2018) is the off policy RL algorithm that is trained on each task separately, thus being a SAC (MTSAC) is the adaptation of
compare our approach against the following algorithms. First, SAC (Haarnoja
reference for the single-task setting. Second, Multi- SAC to the MTRL setting, where we employ a single-head architecture with a one-hot vector concatenated with the state. Then, SAC+FiLM is a task-conditional policy that employs the FiLM module (Perez et al., 2017). Furthermore, PCGrad (Yu et al., 2020) is an MTRL approach that tackles the task interference issue by manipulating the gradients. Soft-Module (Yang et al., 2020) utilizes a routing network that proposes weights for soft combining of activations for each task. CARE (Sodhani et al., 2021) is an attention-based approach that learns a mixture of experts for encoding the state while utilizing context information. Finally, PaCo (Sun et al., 2022) is the state-of-the-art method for MetaWorld that learns a compositional policy where task-specific weights are utilized for interpolating task-specific policies. Our approach uses a similar framework as in the MiniGrid experiment and employs a multi-head architecture.
Following Sun et al. (2022), we benchmark against variants of the MT10 and MT50 sce- narios, MT10-rand and MT50-rand, where each task is trained with random goal positions. The goal position is concatenated with the state rep- resentation. As performance metric, we com- pute the success rate averaged across all tasks. We run our approach for different runs and report their mean and standard deviations of the metric, similar in Sun et al. (2022). As stated in Tab. 1, MOORE outperforms all the base- lines regarding sample efficiency and asymp- totic performance. Moreover, in Tab. 2, our approach shows significant final performance, indicating the scalability of MOORE to a large number of tasks. It is important to mention that all baselines use tricks to enhance the stability of the learning process. For instance, PaCo avoids task and gradient explosion by proposing two empirical tricks, named loss maskout and w-reset, where pruming every task loss that reaches above a certain threshold, besides resetting the task-specific weight for that task. Also, as in Sun et al. (2022), the other baselines resort to more expensive tricks, such as termi- nating and re-launching the training session when a loss explosion is encountered. On the contrary, our approach does not need such tricks to improve the stability of the learning process, which can indicate the stability of the chosen architecture and the importance of learning distinct experts.
Algorithms
Success Rate (20M)
MTSAC (Yu el al., 2019)
4..
SAC + FiLM (Perez et al., 2017)
36.5+112.0
CARE (Sodhani et al., 2021)
50.8:11.0
PaCo (Sun et al., 2022)
57.311..
MOORE (ours)
72.91:3.3
Table 2: Results on MetaWorld MT50 (Yu et al.,
2019) with random goals (MT50-rand). The re- sults of the baselines are from Sun al. (2022). MOORE uses k 6 experts.
5.2.1 ABLATION STUDIES
Diversity. Similarly, we want to evince the advantage of favoring diversity across experts. We evaluate MOORE against MOE, a baseline that uses the same architecture of MOORE but without the Gram-Schmidt process. We evaluate MOORE against MOE on the two MTRL scenarios of MerlW MT10-rand and MT50-rand. In Fig. 5(a). MOORE exhibits superior cumple-efficiency compared to MOE. Moreover, MOORE significantly outperforms the baseline also in MT500rrad
8

---

Published as a conference paper at ICLR 2024
0.8
Success Rate
Success Rate
0.6
0.50
0.4
0.25
0.2
MDE
0.00
MODRE (ours)
0.0
MOORE (ours)
0
5
10
15
20
0
5
10
15
20
#Epochs
#Epochs
(a) Success rate MTIO-rand.
(b) Success rate MT50-rand.
Figure 5: (a) Success rate on MetaWorld MT10-rand comparing MOORE, against MOE, using experts. (b) Success rate MetaWorld MT50-rand comparing MOORE, against MOE, given 6 experts. We show the average success rate across all tasks and the 95% confidence interval across 10 and 5 different runs for MT10-rand and MT50-rand, respectively.
(Fig. 5(b)), evincing the scalability of our approach to large-scale MTRL problems. This study illustrates the importance of enforcing diversity across experts in MTRL algorithms. Interpretability. Additionally, we verify the interpretability of the learned representations. Fig. shows application of PCA the learned task-specific weights we that interpolate the represen- tations of the experts. On the one hand, the pick-place task is close to the peg-insert-side since both tasks require picking up an object. On the other hand, the weights of door-open and window-open tasks are similar as they share the open skill. Therefore, enforcing diversity across experts dis- tributes the responsibilities across them in capturing common components across tasks (e.g., objects or skills). This confirms that the learned experts have some roles that can be interpretable.
6 CONCLUSION AND DISCUSSION
We proposed novel MTRL approach for di- versifying mixture of shared experts across tasks. Mathematically, we formulate our ob- jective a constrained optimization problem where hard constraint is explicitly imposed to ensure orthogonality between the represen- tations. As result, the orthogonal representa- tions live on a smooth and diffcrentiable man- ifold called the Stiefel manifold. We formu-
ynncipall Lomponent
pap-nuent-side
window-dise
0.0
Aucton Pi 255 Top dondown
-0.2
0.4
late our MTRL as a novel contextual MDP while mapping each state the Stiefel mani- fold using a mapping function, which we learn through mixture of experts while enforcing orthogonality across their representations with the SraamSShhmin process, hence satisfying the hard constraint. Our approach demonstrates su- perior performance against related baselines on two challenging MTRL. baselines.
0.6
-1.0
-0.5
0.0
0.5
1.0
1.5
Principal Component
Figure 6: Principle Component Analysis (PCA) on the task-specific weights learned by MOORE on MetaWorld MT10-rand for run with 100% success rate across all tasks.
Taking advantage all the experts during inference, our approach has the limitation potentially suffering from high time complexity compared to sparse selection of few experts. This leads to trade-off between the representation capacity and time complexity. which could be investigated in transfer learning study,
the future by selection of a few orthogonal experts. In addition are interested in investigating extensions of our approach into a continual learning setting.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We want to thank Aliaa Khalifa for her support in writing the paper and Firas Al-Hafez for his feedback on the method. This work was funded by the German Federal Ministry of Education and Research (BMBF) (Project: 011S22078). This work was also funded by Hessian.ai through the project "The Third Wave of Artificial Intelligence 3AT by the Ministry for Science and Arts of the state of Hessen. Calculations for this research were conducted on the Lichtenberg high- performance computer of the TU Darmstadt and the Intelligent Autonomous Systems (IAS) cluster at TU Darmstadt.
REFERENCES
Riad Akrour, Davide Tateo, and Jan Peters. Continuous action reinforcement learning from a mixture of interpretable experts. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(10): 6795-6806 2021.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regu- larizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
Richard Bellman. Dynamic Programming. Princeton University Press, Princeton, NJ, USA, edi- tion, 1957.
Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli. Sparse multi-task reinforcement learning. In Advances in Neural Information Processing Systems, 2014.
Arslan Chaudhry, Nacemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-
rank orthogonal subspaces. Advances in Neural Information Processing Systems, 33:9900-9911, 2020.
Guangran Cheng, Lu Dong, Wenzhe Cai, and Changyin Sun. Multi-task reinforcement learning with attention- mixture of experts. IEEE Robotics and Automation Letters, 86):33122-819,, 2023. doi: 10.1 R01.01....202..32.2.2 2A..33.22222233274555555
Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modu- lar & customizable reinforcement learning environments for goal-oriented tasks. arXiv preprint arXiv:2306. 13831, 2023.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowl- edge in multi-task deep reinforcement learning. In International Conference on Learning Repre- sentations, 2020.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research, 22(131): 1-5. 2021. URL http: //mtt:::/mmm..... org/ papers/ 18 00mm.mmm....
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In International Conference on Robotics and Automation, 2017.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint urXiv:1802.000 2018.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, 2018a.
10

---

Published as a conference paper at ICLR 2024
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. CoRR, abs/ 809.04474, 2018b.
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight multiple dependent stiefel manifolds in deep neural
normalization: Solution to optimization networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018a.
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018b.
I. M. James. The Topology of Stiefel Manifolds. London Mathematical Society Lecture Note Series. Cambridge University Press, 1977. doi: 10.1017/C180978551 16007533
Yonggang Jin, Chenxu Wang, Liuyu Xiang, Yaodong Yang, Jie Fu, and Zhaofeng He. Deep inforcement learning with multitask episodic memory based on conditiond hypernetwork. arXiv preprini arXiv:2306./006 2023.
Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01113, 2020.
Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 43(4):1352-136; 2019.
Lester Mackey, Vasilis Syrgkanis, and Ilias Zadik. Orthogonal machine learning: Power and limita- tions. In International Conference on Machine Learning, pp. 3375-3383, PMLR, 2018.
Peyman Sheikholharam Mashhadi, Slawomir Nowaczyk, and Sepideh Pashami. Parallel orthogonal deep neural network. Neural Networks, 140:167-183, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13Xiv:1312.56000 2013.
Mete Ozay and Takayuki Okatani. Optimization submanifolds of convolution kernels in cnns. arXiv preprint arXiv: 2016.
Bernardino Romera Paredes, Andreas Argyriou, Nadia Berthouze, and Massimiliano Pontil. Ex- ploiting unrelated tasks in multi-task learning. In Artificial intelligence and statistics, pp. 951- 959. PMLR, 2012.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with general conditioning layer. CoRR,
Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. Journal of the Operational Research Society, 1995.
Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture--f-expprts for efficient deep reinforcement learning. arXiv preprint arXiv:2104.01122, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region
policy optimization. In International conference on machine learning, pp. 1889-1897, PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):4844485 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint drXiv:1712.0188 2017.
11

---

Published as a conference paper at ICLR 2024
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. International Conference on Machine Learning. 2021.
Lingfeng Sun, Haichao Zhang, Wei Xu, and Masayoshi Tomizuka. Paco: Parameter-comppsitional
multi-task reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net. forummid--xutPPPI
https://openreve//
Yee Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nico- las Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Ruihan Yang, Huazhe Xu, WU, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In Advances in Neural Information Processing Systems, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. 2019.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, 2020.
12

---

Published as a conference paper at ICLR 2024
(a) DoorKey
(b) DistShift
(c) RedBlueDoors
(d) LavaGap
(e) Memory
(I) SimpleCrossing
MultiRoom
Figure 7: MiniGrid (Chevalier-Boisvertt et al., 2023) Tasks, where the red triangle represents the agent, and the green square refers to the goal.
A ADDITIONAL DETAILS ON THE EXPERIMENTS
In this section, we elaborate on the implementation details of our approach, MOORE, for bench- marking against MiniGrid Schevalier-Booiveed et al., 2023) and MetaWorte (Yu et al., 2019). Be- sides, we provide additional ablation studies that demonstrate various aspects of our approach. In this work, we used Mushroom-RL (D" Eramo al., 2021) the RL library.
A.1 MINIGRID
A.1.1 ENVIRONMENT DETAILS
MiniGrid Beevaliee--oisverr et al., 2023) collection of 2D goal-oriented environments where the agent learns how solve different mazes while interacting with various objects in terms of shape, color, and role. The library MiniGrid provides multiple choice for state representation. For our MTRL setting, we adopt the visual representation of the state where 3-dimensional input of shape 7x7x3 is provided. As mentioned in Sec. 5.1, our MTRL setting consists of three scenarios that include seven tasks in total that distributed differently. A render example of each task is demonstrated in Fig. 7. Additionally, the description of each task is provided in Tab. 3.
Task
Description
DoorKey
Use the key to open the door and then get to the goal.
DistShift RedBlueDoors
Get the green goal square.
Open the red door and then the blue door
LavaGap
Avoid the lava and get the green goal square. Go to the matching object at the end of the hallway Find the opening and get to the green goal square Traverse the rooms get the goal.
Memory
SimpleCrossing
MultiRoom
Table 3: MiniGrid (Chevalier-Boisvert et al., 2023) task descriptions.
13

---

Published as a conference paper at ICLR 2024
A.1.2 IMPLEMENTATION DETAILS
RL algorithm. We use PPO (Schulman et al., 2017), which is considered a state-of-the on-policy RL algorithm on many benchmarks. Moreover, has been used in the official paper the MiniGrid benchmark (Chevalier-Boiseen et al., 2023). We adapt PPO the MTRL setting by computing the loss functions of both the actor and critic averaged on transitions sampled from all tasks. We refer this adapted algorithm MTPPO. Tab, 4, we highlight the important hyperparameters needed to reproduce the results on MiniGrid.
Hyperparameter
Value
General Hyperparameters
Discount factor ?
0.99
Number of environments
[3,5,7]
Steps per environment
1 step per environment
Number of epochs
100
Steps per epoch
2000
Train frequency
2000
Number of episodes for evaluation
16
PPO Hyperparameters
Lambda coefficient in GAE formula
0.95
Entropy term coefficient
0.01
Clipping Epsilon
0.2
Number of epochs for Policy
8
Batch Size for Policy
256
Number of epochs for Critic
1
Batch Size for Critic Critic Loss
2000
Mean Squared Error
Optimizer
Adam
Learning rate for Policy
0.001
Learning rate for Critic
0.001
Table 4: MiniGrid Sntevalier-Boiveer al., 2023) hyperparameters.
Architecture. The network architecture consists of two main parts, a representation block, and an output head. The representation block is agnostic to the context The role of the representation block is to encode the state s. On the other hand, the output head includes an output function for generating the network output. In general, we use similar network architecture for the actor and the critic.
For single-expert approaches (PPO and MTPPO), the representation block consists of a single Con- volutional Neural Network (CNN) to encode the visual representation of the state to a latent space. For multiple-experts approaches (MOORE, MOE, and PCGrad), k-CNNs are used represent the mixture of experts responsible for encoding the state as k-representations in the representation block.
For MTRL approaches, the output function can utilize single-head fo or a multi-heat [fe...., fê architecture. For the single-head architecture, we condition the network on the context by concatenating the context e (one-hot vector) to the output of the representation block. other hand, for the multi-head architecture, we select task-specific oulput function fe, given the context c.
For multiple-experts approaches, in addition the output function, the output head includes task- encoder. Given a context c, the task-encoder generates a task-specific weight w. responsible for combining the output of the representation block V, produce the task-specific representation Ve-
In Tab. 5, we illustrate the byperparameters of both the representation block and the output head. It is worth noting that MOORE, MOE, and PCGrad linearly combine the generated representations from different experts before applying the last activation function of the representation block v, = Tanh(V; W.). Moreover, the whole architecture is trained end-to-end, including the task-encoder.
14

---

Published as a conference paper at ICLR 2024
Hyperparameter
Value
Representation Block
Number of Experts (k)
(MT3: k = MT5: k MT7: 4}
Number of convolution layers
3
Channels per layer
[16, 32, 64]
Kernel size
[22,2,, (2,2), (2,2)
Activation functions
[ReLU, ReLU, Tanh]
Output Function
Number of linear layers
2 (x number of tasks [7
Number of output units
[128. |A| for actor and for critic]
Activation functions
[Tanh, Linear]
Task Encoder
Number of linear layers
Number of output units
Number of Experts (k)
Use bias
False
Activation function
Linear
Table 5: Actor and Critic Architecture for PPO
Algorithm MOORE for Actor
Algorithm 2 MOORE for Critic
Require: Mixture of experts h3, state context task-specific weights we output function fo.
Require: Mixture of experts hg, state-action (s. a). con-
text task-specific weights w.. output function fa. U. hs(
1: U.
(s)
2: GS(U.)
Apply Eq.
2: Va, GS(U,,a)
D Apply Eq.
3: 6. Va We
3: Ve = Ve,a We
4: a fe(ve)
4: g fo(ve)
5: Return: a
5: Return: q
A.2 METAWORLD
A.2.1 ENVIRONMENT DETAILS
MetaWorld (Yu al., 2019) is suite of many robotic manipulation tasks. All tasks require dealing with one or two objects. Moreover, they are similar in terms of the state space's s dimensionality, yet the state components' semantics differ. The state space consists of the following: the 3D position of the end effector, a normalized measure of how much the gripper is open, the 3D position of the first object, the quaternion of the first object (4D), as well the 3D position and quaternion of the second object (zeroed out, if not needed). Two consecutive data frames are stacked together, in addition to the 3D goal position, forming a 39-dimensional state space. On the other hand, the action space is the same, representing the 3D change of the end effector in addition to the normalized torque applied by the gripper. We benchmark our approach against the MT10 and MT50 scenarios. Following Sun al. (2022), we randomize the goal or object positions across all tasks and refer to them as MT10-rand and MT50-rand.
A.2.2 IMPLEMENTATION DETAILS
RL algorithm. this benchmark, we use SAC (Haarnoja al., 2018), a ofate-of-the-art off- off-policy algorithm that enhances the exploration of the agent by maximizing the entropy. Similar to Yu et al. (2019); Sun et al. (2022), we adapt SAC by computing the actor and the critic losses averaged on transitions sampled from all tasks. We have replay buffer for each task from which sample transitions equally. In addition, we disentangle the temperature parameter of SAC by learning sepa- rate temperature parameters for each task. We refer this adapted algorithm MTSAC. In Tab. 6, we list the hyperparameters required for reproducing our results on MetaWorld
Architecture. Similar MiniGrid, block and an output head, We made a couple of changes for Meta World. For instance, the actor and the critic slightly differ since the action is concatenated with the state for computing the Q values
network architecture that consists of representation
15

---

Published as a conference paper at ICLR 2024
in the critic. As result, the representation block is responsible for encoding the state-action space. Another difference is that we use a Dense Neural Network (DNN) to represent the representation block. Consequently, Finally, we adopted multi-head architecture for the output function where we use the context e to select the corresponding task-specific output function fo.-
use k-DNNs to represent the mixture of experts for MOORE and MOE.
worth mentioning that the results of the baselines Tab. 1 and Tab. are borrowed from Sun al. (2022). The implementation details of the baselines be found in Yu al. (2019); Sun al. (2022). We demonstrate the MOORE algorithm for the actor and the critic in Alg. and Alg. 2, respectively. Similarly, MOE follows the same procedure but without the Sehm-- process in line 2.
Hyperparameter
Value
General Hyperparameters
Horizon
150
Discount factor
0.99
Number environments
10
Steps per environment
I step per environment
Number epochs
20
Steps per epoch
100000
Train frequency
Number of episodes for evaluation
10
SAC Hyperparameters
Batch Size
128
Critic Loss
Mean Squared Error
Disentangled temperature Alpha a Optimizer
True
Adam
Learning rate for Policy Learning rate for Critic Learning rate for Alpha Policy minimum standard Policy maximum standard Soft target interpolation Exploration steps
3 x 10-4
3 x 10-4
1 x 10
e 10
e²
5 x 10-3
1500
Replay buffer steps
1 10"
Table 6: MetaWorld (Yu et al., 2019) Hyperparameters.
Hyperparameter
Value
Representation Block
Number of Experts (k)
MTT0: 4, MT50: 6)
Number of Linear layers
Number of output units
[400, 400, 400] [ReLU, ReLU, Linear]
Activation functions
Output Block
Number of linear layers
number of tasks TT)
[[A] for actor and 1 for critic] Linear Number of linear layers
Number of output units
Activation functions
Task Encoder
Number of output units
Number of Experts (l:)
Use bias
False
Activation function
Linear
Table 7: Actor and Critic Architecture for SAC
16

---

Published as a conference paper at ICLR 2024
LavaGap
RedBlueDoors
Memory
0.6
0.8
Average Return
0.75
Average Return
Average Return
0.5
0.6
0.50
0.4
0.4
0.25
0.2
0.2
0.00
0.0
#Epochs
#Epochs
#Epochs
PPO MТPPО
PCGrad
MOE
MOORE (ours)
Figure 8: Individual task average return the MT3 scenario of MiniGrid. We utilize the multi- head architecture for our approach MOORE as well as the related baselines. For MOORE, MOE, and PCGrad, the number of experts is The black dashed line represents the final single-task performance of PPO averaged across all tasks. For the evaluation metric, we compute the accumu- lated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
DoorKey
LavaGap
RedBlueDDors
0.8
Average Returm
Average Return
0.6
0.6
0.4
Average
0.44
0.4
0.2
0.0
0.0
0.0
25
50
75
100
0
25
50
75
100
25
50 #Epochs
75
100
Epochs
#Epochs
DistShift
Memory
1.0
0.6
Average Return
Average Return
0.5
0.6
0.4
0.2
0.2
50
75
100
25
100
#Epochs
#Epochs
PPO
MТPPО
PCGrad
MOE
MOORE (ours)
Figure 9: Individual task average return on the MT5 scenario of MiniGrid. We utilize the multi- head architecture for our approach MOORE as well as the related baselines. For MOORE, MOE, and PCGrad, the number of experts is The black dashed line represents the final single-task performance of PPO averaged across all tasks. For the evaluation metric, we compute the accumu- lated return averaged across all tasks. We report mean and the 95% confidence interval across 30 different runs.
B ADDITIONAL EMPIRICAL RESULTS
B. MINIGRID
In Sec. 5.1, we present the performance averaged across all the tasks. Here, we want to show the individual task performance of all three scenarios of MiniGrid.
17

---

Published as a conference paper at ICLR 2024
Doorkey
DistShift
RedBlueDoors
1.00
1.00
Average Return
0.75
Average Return
0.75
Average Return
0.50
0.50
0.25
0.25
0.2
0.00
0.00
75
100
50
100
25
50 #Epochs
# Epochs
# #Epochs
LavaGap
Memory
SimpleCressing
Average Return
0.75
Average Return
Average Return
0.4
0.25
0.2
0.00
50
75
100
25
50
75
100
25
50 #Epochs
75
100
Epochs
#Epoccss
MultiRoom
0.6
0.4
0.2
0.0
25
50
75
100
#Epochs
PPO MTPPO
PCGrad
MOE
MOORE (ours)
Figure 10: Individual task average return on the MT7 scenario of MiniGrid. We utilize the multi- head architecture for our approach MOORE as well as the related baselines. For MOORE, MOE, and PCGrad, the number of experts performance of PPO averaged across all tasks. For the evaluation metric, we compute the accumu- lated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
The black dashed line represents the final single-task
B.2 TRANSFER LEARNING WITH MOORE
Furthermore, discuss the experimental details of the Transfer Learning ablation study in Fig. 3. In this study, we assess the transfer capability of our approach in utilizing the diverse representations learned on of base tasks for set of novel but related tasks. We evaluate our approach, MOORE, against the MOE baseline on MiniGrid. We refer to the transfer learning adaptation of our approach as Transfer-MOORE and Transfer-MOE for the MOE baseline.
We conducted two experiments based on the sets of tasks defined on MiniGrid (MT3, MT5, and MT7). In Fig. 3, we show the empirical results on transfer learning scenarios where we transfer a set of experts learned on MT3 to MT5 (MT3 - MT5) and on MT5 to MT7 (MT5 - MT7). worth noting that MT3
subset MT5, and MT5 is a subset of MT7. The base tasks are the MT3 and MT5 for MT3 -> MT5 and MT5 - MT7, respectively, while the novel tasks are the difference between the corresponding sets. For instance, in the - scenario, the base tasks LavaGap, RedBlueDoors, and Memory (common for MT3 and MT5), while having DoorKey, and MultiRoom as novel tasks (only in MT5).
For MOORE,, we train the base tasks; then, we use the learned mixture of experts in a frozen state to learn the novel ones. On the contrary. MOORE is only trained on novel tasks from scratch. This also holds for MOE and Transfer-MOE. In this study, employ multi-head architecture for the actor and critic. Hence, each task has a decoupled output head from other tasks, easing the transfer learning experiment. However, they all share the representation stage (mixture
18

---

Published as a conference paper at ICLR 2024
LavaGap
RedBlueDoors
Memory
Average Retum
0.75
Avera ge Return
Average Return
0.50
0.4
0.25
0.2
0.2
0.00
100
#Epochs
#Epochs
#Epochs
PPO
MТPPО
PCGrad
MOE
MOORE (ours)
Figure 11: Individual task average return the MT3 scenario of MiniGrid. We utilize the single- head architecture for our approach MOORE as well as the related baselines. For MOORE, MOE, and PCGrad, the number of experts is The black dashed line represents the final single-task performance of PPO averaged across all tasks. For the evaluation metric, we compute the accumu- lated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
DoorKey
LavaGap
RedBlueDDors
0.8
0.6
Average Return
Average Return
0.6
0.6
0.4
0.4
0..4
0.2
0.2
0.0
0.0
0.0
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Epochs
DistShift
#Epochs
Memory
#Epochs
1.0
0.6
Average Reture
Average Return
0.5
0.6
0.4
0.3
0.2
0.2
50
75
100
25
100
#Epochs
#Epochs
PPO
MТPPО
PCGrad
MOE
MOORE (ours)
Figure 12: Individual task average return on the MT5 scenario of MiniGrid. We utilize the single- head architecture for our approach MOORE as well as the related baselines. For MOORE, MOE, and PCGrad, the number of experts is The black dashed line represents the final single-task performance of PPO averaged across all tasks. For the evaluation metric, we compute the accumu- lated return averaged across tasks. We report the mean and the 95% confidence interval across 30 different runs.
of experts). We add randomly initialized output heads to learn the novel tasks while keeping the mixture of experts frozen. For MT3 - MT5, the number of experts l: is 2. On the other hand, for MT5 + MT7, we use 3 experts.
B.3 COSINE SIMILARITY
We investigate the ability of MOORE to diversify the shared representations, compared to relaxing the hard constraint in Eq. 1. Therefore, replace the hard constraint with regularization term
19

---

Published as a conference paper at ICLR 2024
Doorkey
DistShift
RedBlueDoors
1.00
1.00
Average Return
Average Return
Average Return
0.6
0.75
0.75
0.50
0.50
0.25
0.2
0.00
0.00
75
100
50
100
25
50 #Epochs
# Epoochs
* #Epochs
LavaGap
Memory
SimpleCressing
0.6
Average Return
0.75
Average Return
Average Return
0.50
0.4
0.25
0.2
0.00
50
75
100
25
50
75
100
25
50 #Epochs
75
100
Epochs
Epochs
MultiRoom
Return
0.6
0.4
0.2
0.0
25
50
75
100
#Epochs
PPO MTPPO
PCGrad
MOE
MOORE (ours)
Figure 13: Individual task average return on the MT7 scenario of MiniGrid. We utilize the single- head architecture for our approach MOORE as well as the related baselines. For MOORE, MOE, and PCGrad, the number of experts performance of PPO averaged across all tasks. We show the accumulated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
The black dashed line represents the final single-task
equivalent to cosine similarity loss computed over the set of representations:
Leg FLES (ho(s)" (8) Tx]-
The regularization loss is optimized jointly with the primary objective, where we weigh the contri- bution of this regularization loss by 1. We benchmark MOORE against the Cosine-Similarity on the three scenarios of MiniGrid. As shown in Fig. 14. MOORE outperforms the baseline across all settings, highlighting the advantage of using Gram-Schmidt in diversifying the experts over regularization-basse -aaedd techniques. In addition, our approach is hyperparameter-free contrary to the regularization-- based techniques that require delicate hyperparameter tuning to not interfere with the main loss function, which is usually the case.
B.4 INFLUENCE OF THE SINGLE-HEAT ARCHITECTURE ON MOORE
In this section, we discuss the reason hehind the degradation in the performance of MOORE when employing a single-head architecture, especially on MT7 (Fig. 2(b)). We argue that the reason is the task interference caused by the single-head architecture since all tasks share the same output function fe. MOORE is highly affected by the later output stage, causing a drop the performance relative to the experiments done with the multi-head architecture. It is worth noting that as the number of tasks increases, the possibility of having task interference increases. This why issue is prominent in the MT7 scenario.
20

---

Published as a conference paper at ICLR 2024
MT3
MTS
MT7
0.6
0.6
0.6
Average Return
Average Return
Average Return
0.4
0.4
0.4
0.2
0.2
100
100
100
WEpochs
#Epochs
#Epochs
Cosine-Similarity
MODRE (ours)
Figure 14: Evaluating the diversity capabilities of our approach, MOORE, against using Cosine- Similarity. The study is conducted the three MTRL scenarios of MiniGrid employing a single- head architecture. The number of experts l is 2, 3, and 4 for MT3, MTS, and MT7, respectively. For the evaluation metric, compute the accumulated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
0.8
PPO
Average Return
0.6
MTPPO
PCGrad
0.4
MOE
MOORE (Single-Head)
0.2
MOORE+
MOORE (Multi-Head)
0.0
20
40
60
80
100
# #Epochs
Figure 15: Ablation study the effect of combining MOORE with PCGrad to reduce the task interference issue in the output stage. All methods employ a single-head architecture except for MOORE Multi-HHad), The study is conducted the MT7 scenario MiniGrid. The number of experts 4. For the evaluation metric, we compute the accumulated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
We have two reasons to support our claim:
When using a multi-head architecture, MOORE outperforms all the baselines on all of the MiniGrid scenarios. Employing the multi-head architecture decouples the output functions for all tasks, completely removing the task interference in the output stage.
In Fig. 15, we conduct an ablation study highlighting the effect of combining PCGrad (ex- plicit MTRL method tackle task interference) and our approach. Since MOORE or- thogonal to PCGrad, we can integrate them easily. This study shows that MOORE+PCGrad outperforms MOORE, PCGrad, MOE, and MTPPO. However, MOORE with multi-head architccture still outperforms MOORE+PCGrad. showing that PCGrad only partially reduce the interference the output stage, while MOORE with multi-heas architecture removes the interference completely.
C COMPUTATION AND MEMORY REQUIREMENSS
The difference between MOORE and MOE is in the Gram-Schmidt stage, where we orthogonalize the k representations. The time complexity of the Cramm-Shmmitt process is T = O x d) (Golub & Van Loan, 2013; Mashhadi et al., 2021), where d is the representation dimension and & is the number of experts. Our approach MOORE and the baseline MOE belong to the family of soft mixtures of experts since they compute all representations from all the experts during inference.
21

---

Published as a conference paper at ICLR 2024
Average Return
0.6
0.4
MOORE-u1 MOORE-u2
0.2
MOORE-U3
0
20
40
60
80
100
# Epochs
b
Figure 16: Ablation study on the effect of the initial expert selected for the -chn-St process. In this study, we employ multi-head architecture. The number of experts & is 3. ul, u2, and u3 are the representations of the three experts before applying the Gram-Schmin process. For the evaluation metric, we compute the accumulated return averaged across all tasks. We report the mean and the 95% confidence interval across 30 different runs.
On the other hand, one can only select top-k experts based on some weights computed using a gating network as in the direction sparse mixtures of experts. The trade-off between the representation capacity and time complexity is well-known. As a future work, we can investigate the adaptation of MOORE to pick only a few orthogonal experts, hence lowering the time complexity. MOORE similar to the MOE baseline regarding the memory required for storing all the experts. It is worth noting that we use fewer experts than PaCo (Sun et al., 2022) in MeraWorld, hence lower memory requirements.
D THE GRAM-SCHMIDT PROCESS AND THE INITIAL EXPERT
In MOORE, consider the first expert's representation the initial vector for the process. In a normal setting, we can expect the process to yield a different set orthonormal vectors depending the initial selected vector. It does not matter in case since the representations actually generated from a mixture of experts which are being learned. We conduct an ablation study on the MT5 scenario of MiniGrid, where we utilize 3 experts. We provide variations of MOORE process. For instance, MOORE-ul selects
based on the initial vector selected for the the representation of the first expert ul the initial vector of the Tram--Schmid process (adopted). On the other hand, MOORE-u2 and MOORE-už choose the representation of the second u2 and third u3 expert, respectively, the initial vector for the Gram-Schmin process. As expected, Fig. 16 shows that the performance is almost identical for different selected initial vectors.
22


Reviews:
[{"review_id": "EuWG5533s7", "reviewer": "Reviewer_gg3H", "summary": "The authors introduce a novel representation learning method for multi-task reinforcement learning (MTRL) that promotes orthogonal representations to enhance diversity. Initially, they present a Stiefel Contextual Markov Decision Process (SC-MDP) to interpret the orthogonal representation spaces. Following this, the authors propose a constrained optimization problem that ensures orthogonality in the representations and employ the Gram-Schmidt process to address the constraints. Empirical results demonstrate that their proposed method, MOORE, consistently surpasses previous MTRL algorithms across two MTRL benchmarks.", "questions": "1. The Gram-Schmidt process can yield different vectors based on which vector is chosen as the first. How do the authors determine the initial vector for the Gram-Schmidt process?\n\n2. Can you provide the results for the MT50 benchmark?\n\n3. How many representations $k$ are used in each experiment?\n\n4. Could you elucidate the distinction between MOORE and Transfer-MOORE?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes an interesting approach to promote the diversity of representation with Gram-Schmidt process and shows empirical improvements on two MTRL benchmarks.", "weaknesses": "The paper doesn't introduce a method that is fundamentally distinct from the approach in prior work[1], which utilizes a representative parameter set {$\\phi_1,...,  \\phi_k$} and task-specific weight $w_c$ derived from the task-id. The proposed structure for the policy appears to be merely different combinations of the existing learnable parameters discussed in paper[1]. While Paco[1] employs a linear combination of the representative parameters with the task-specific weight as the policy parameters, this paper adopts the same strategy to compute an additional input for policy inputs combined with the state. Moreover, although the authors argue that orthogonality results in more diverse representations, there's an absence of theoretical backing or ablation studies to corroborate this claim. Given that the vector $k$-representations $U_s$ is contingent upon the state $s$, the authors cannot guarantee representational diversity at the task level, even if state-level orthogonality is ensured. Additionally, the paper does not address issues related to complexity and memory requirements. Storing a set of parameters for representation implies that the proposed \"MOORE\" method could demand significant memory and computational resources. Finally, the authors should benchmark their method against the MT50 benchmark from Meta-World, as all their experiments are conducted on a limited task set size.\n\n[1] Sun, Lingfeng, et al. \"PaCo: Parameter-Compositional Multi-Task Reinforcement Learning.\" Advances in Neural Information Processing Systems 35 (2022): 21495-21507.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jU9vFYccDG", "reviewer": "Reviewer_pAYF", "summary": "The paper aims to learn shared representations for multi-task reinforcement learning. The authors propose that these representations should be as diverse as possible. They define a contextual MDP called the Stiefel Contextual Markov Decision Process (SC-MDP) where the states are projected into the Stiefel manifold. The property of a Stiefel manifold is all elements can be represented as k-orthonormal representations i.e. representations that comprise k orthonormal vectors, each of dimension d. They impose a hard constraint on the optimization using the Gram-Schmidt process. They achieve competitive results on minigrid and metaworld.", "questions": "(1) What do you mean by frames in Definition 4.1?\n\n(2) How do you parameterized the critic? You have only mentioned the parameterization of actor.\n\n(3) Explain MT-PPO - How is PPO adapted for multi-task RL?\n\n(4) What do you mean by single-head and multi-head architectures? Do you refer to single-head and multi-head attention?\n\n(5) For transfering experts, you write \"we transfer experts learnt on MT3 to MT5 (MT3 $\\rightarrow$ MT5) and on MT5 to MT7 (MT5 $\\rightarrow$ MT7) \". But in the next sentance you write \"First, we train on base tasks (intersection of two scenarios), and then we transfer the learnt experts on novel tasks (the difference between the two scenarios)\". Both statements sort of contradict each other.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) The use of the Gram-Schmidt process to enforce a hard constraint of orthonormality on the optimization is interesting. \n\n(2) If the vectors are orthonormal, the state will be represented using k diverse vectors, providing a greater span of the state space. This bears fruit as empirically, they perform better when the vectors are orthonormal i.e. when the hard constraint is applied.\n\n(3) They introduce the SC-MDP formulation where each state can be mapped to an element in the Stiefel manifold.", "weaknesses": "(1) Please provide an algorithm box. I am not sure why these different vectors are called experts. Is it for every step of the convex optimization, you perform the Gram-Schmidt orthonormalization like projected gradient descent? Or do you perform the normalization towards the end? \n\n(2) Say you obtain $U^* = [u_1, u_2, ..., u_k]$ by minimizing $J(\\Theta)$. Are these k vectors linearly independent? Gram-Schmidt process works on linearly independent vectors. Moreover, the vectors that you get after the process, $V^*= [v_1, v_2, ..., v_k]$ span the same space as $U^*$ but the $w_c$ for both will be different. I think these confusions can be cleared if you provide an algorithm box.\n\n(3) You mention that MOORE even outperforms the single task performance is a few cases. But in those cases, most baselines also outperform the single task performance.\n\n(4) Please address the questions in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3GmPEpynYJ", "reviewer": "Reviewer_stvx", "summary": "The authors propose MOORE, a method for learning a common state representation using a mixture of orthogonal experts to be combined with task-specific weights for Multi-Task Reinforcement Learning. Building upon a Block Contextual MDP, a Stiefel-Contextual MDP is defined that formally allows for a compositional policy, taking advantage from a latent state representation retrieved from a mixture of experts. In various empirical studies, this representation is shown to improve the performance of PPO in various discrete tasks, and the performance of SAC in various continuous control tasks. Furthermore, ensuring orthogonality of experts via the Gram-Schmidt Process yields even improved results.", "questions": "From my understanding, orthogonality is induced via the Gram-Schmidt Process. Could you elaborate on the necessity of the additional hard constraint introduced in Eq. 1?\n\nRegarding the experimental results, why is the performacne of PPO only reported as a horizontal line?\n\nHow many experts are used for the Meta World evaluations? How does the number of experts impact the scalabiltiy of the proposed approach? Further elaborations on the computational overhead introduces, as well as possible limitations might be more insightful, than the presented remarks regarding interpretability. \n\nMinor Comments:\n- p. 2: To improve readability, the contributions could be displayed as a list. \n- p. 2: $\\pi$ is defined in a deterministic but used in a stochastic manner.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tackles a relevant problem and provides a well-motivated solution approach. Overall, the paper is well structured and the presented considerations are theoretically sound. A wide variety of tasks, domains,  action-, and state-spaces, are used for evaluation, using a sufficient amout of baselines and sensible ablations.", "weaknesses": "Overall, the clarity of the paper should be improved. Concretely: \n\n- The introduction does not state whether the proposed method is used for learning in a multi-task scenario or to curate diverse tasks for learning. Moreover, providing a motivation why a common state representation across tasks is needed could be useful. Also, the empirical studies could be included with the contributions listed in the introduciton.\n- In Section 4, in addition to the visualization of the mixture of experts in Fig. 1, a visualizing the parts of the compositional policy $\\pi(a|s)=(V_sw_c)^T\\theta$ an their connection would be helpful. To further elaborate on the intertwinement of the components and the proposped process, showcasing an exemplary algorithm might be useful (Section 4.2).\n- Regarding the experiments: I assume MOORE to be used in conjuction with PPO for the MiniGrid results (5.1) and with SAC for the MetaWorld results (5.2). Yet this should be clearly stated in the paper. Furthermore, if not introduced in related work, used baselines should be described (e.g., MTPPO, MTSAC, PCGrad, ...). To improve the overall readability, the Figure placement could be improved.  \n- Limitations should be stated \n\nRelated work should be extended with regards to work on mixtures of experts for reinforcmeent learning. \nAlso, the baselines used later on should be briefly introduced and compared. Further related work regarding diversity-based RL (e.g., B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, ‘Diversity is All You Need: Learning Skills without a Reward Function’, in International Conference on Learning Representations, 2019.) could also be helpful. \n\nFinally, to foster reproducibility, the authors should consider open-sourcing their implementations upon publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WR296u03mr", "reviewer": "Reviewer_78VR", "summary": "The paper proposes a novel approach for representation learning in Multi-Task Reinforcement Learning (MTRL) known as MOORE (Mixture Of Orthogonal Experts). The core idea is to share representations across tasks in an orthogonal manner to promote diversity and prevent representations from collapsing. MOORE utilizes the Gram-Schmidt process to shape a shared subspace of representations derived from a mixture of experts. This approach ensures diversity by pushing these representations to exist within the Stiefel manifold, a Riemannian manifold known for its orthogonality properties. The authors validate their method on two MTRL benchmarks: MiniGrid and MetaWorld, and report state-of-the-art performance on the latter.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of orthogonalizing shared representations is novel and aligns with the goal of maximizing representation capacity in MTRL.\n\nLeveraging the properties of the Stiefel manifold for enforcing orthogonality is innovative and could have implications beyond MTRL.\n\nThe authors have demonstrated strong empirical results, notably achieving state-of-the-art performance on the MetaWorld MT10-rand tasks.", "weaknesses": "The author used MoE and Transfer-MoE as its baseline. As far as I know, there are other MoE baselines, like PMOE [1], why the author didn't compare them in experiments.\n\nIn environments like DoorKey, as the author showed in Fig. 13, MOORE is worse than some other baselines, why and how this happened needs more discussion.\n\n[1] Ren, Jie, et al. \"Probabilistic mixture-of-experts for efficient deep reinforcement learning.\" arXiv preprint arXiv:2104.09122 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVyAb2wi78", "reviewer": "Reviewer_duFe", "summary": "This paper studies learning a good representation for multi-task setting. Their definition of a good representation acroos tasks crucially relies on the idea of capturing unique and common properties shared by all the tasks. To do this they solve a constrained optimization problem that orthogonalizes the representations generated by a mixture of experts by applying the Gram-Schmidt orthogonalization technique. This results in ensuring independence between the representations. They draw a nice connection between their orthogonal representations to the Stiefel manifold, a particular type of Riemannian manifold where the inner product is defined. In fact they define a Steifel Contextual MDP (SC-MDP) in definition 4.2 that maps any arbitrary state $s$ to a vector representation in Steifel Manifold (consequently these vector representations are orthonormal). They learn this mapping function $\\phi$ (called experts) from the data and while doing so concurrently enforce a constraint so that the learned representations are orthonormal (see eq(1)). Finally, to enforce diversity across experts, they apply the Gram-Schmidt process to orthogonalize the k-representations. They generate the final RL policy by combining these representations with a linear parametrization. Finally, they conduct several empirical evaluation of their algorithm in several standard RL benchmarks.", "questions": "1) There is no discussion in the paper regarding the computational cost of the constrained optimization in (1). How hard is it to solve and enforce this constraint? \n2) I understand that your approach enhances the diversity of feature representation which in turn leads to a good exploration of the state space. How do you ensure the balance between exploration and exploitation? Do you rely on the PPO and SAC to do a standard exploration/exploitation with the diverse learnt features?\n3) The discussion/implementation details between single and multi-head version of MOORE is missing. This seems important because in MT7 (figure 2) MOORE is outperformed by PC-grad and MTPPO? Why?\n4) It was vaguely referred to in section 5.1 (below figure 4) that MOORE has a faster convergence rate than other baselines. Can you elaborate on this? Do you believe that diverse representations (the key thrust of this paper) lead to a faster convergence?\n5) The ablation study is somewhat confusing. Observe that the ablation study is done on single-head architecture and you already show in MT7 (figure 1) that MOE is outperformed by MOORE. However, PC-grad and MTPPO outperform MOORE in figure 1 (single head). So it is not clear whether even PC-grad and MTPPO are learning a diverse set of representations. Can you elaborate on this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) Learning a diverse, meaningful, and common representation across several tasks is an important area of research.\n2) Their approach is somewhat novel where they enforce as they draw this nice connection between the Stiefel manifold and orthonormal representations (in the Stiefel manifold) leads to diverse and meaningful representations across tasks.\n3) Their approach is simple and leads to good empirical performance in some benchmarks.\n4) The Mixture of Experts (MOE) is their main competitor which employs a mixture of experts without enforcing diversity. Principally they show that as you increase the number of experts the average return of MOE is less than MOORE. They also show how task-specific weights align with representations.", "weaknesses": "1) While they draw some connection to the Stiefel manifold, they do not analyze theoretically any properties of their representations that might give some meaningful insight as to why these orthonormal representations are useful.\n2) The paper is not clear about the computation cost of the constrained optimization problem in eq(1).\n3) The paper lacks some more experimental discussion to prove conclusively that the orthonormal representations are helping over their closest competitor MOE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)