PaCo: Parameter-Compositional Multi-task Reinforcement Learning
Authors: Lingfeng Sun
Haichao Zhang
Wei Xu
Masayoshi Tomizuka
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-task reinforcement learning

Abstract: The purpose of multi-task reinforcement learning (MTRL) is to train a single policy that can be applied to a set of different tasks. Sharing parameters allows us to take advantage of the similarities among tasks. However, the gaps between contents and difficulties of different tasks bring us challenges on both which tasks should share the parameters and what parameters should be shared, as well as the optimization challenges due to parameter sharing. 
In this work, we introduce a parameter-compositional approach (PaCo) as an attempt to address these challenges. In this framework, a policy subspace represented by a set of parameters is learned. Policies for all the single tasks lie in this subspace and can be composed by interpolating with the learned set. It allows not only flexible parameter sharing, but also a natural way to improve training.
We demonstrate the state-of-the-art performance on Meta-World benchmarks, verifying the effectiveness of the proposed approach. 

PaCo: Parameter-Commpostioaa Multi-Task
Reinforcement Learning
Lingfeng Sun I+t Haichao Zhang Wei Xu² Masayoshi Tomizuka'
University of California Berkeley
Horizon Robotics
iingfongguuebarkeeee.... (haichao zhang, wei raixxuhhooizzn..
tomizuka@berkollyyccoo
Abstract
The purpose of multi-task reinforcement learning (MTRL) is to train single policy that can be applied to set of different tasks. Sharing parameters allows us take advantage of the similarities among tasks. However, the gaps between contents and difficulties of different tasks bring us challenges the parameters and what parameters should be shared, as well as the optimization challenges due to parameter sharing. In this work, we introduce parameter- compositional approach (PaCo) as an attempt to address these challenges. In this framework, a policy subspace represented by set of parameters is learned. Policies for all the single tasks lie in this subspace and can be composed by interpolating with the learned set. It allows not only flexible parameter sharing but also natural way to improve training. We demonstrate the state-of-the-arr performance on Meta-World benchmarks, verifying the effectiveness of the proposed approach.
both which tasks should share
1 Introduction
Deep reinforcement learning (RL) has made massive progress solving complex tasks in different domains. Despite the success of RL in various robotic tasks, most of the improvements restricted to single tasks locomotion or manipulation. Although many similar tasks with different target and interacting objects are accomplished by the same robot, they are usually defined as individual tasks and solved separately. On the other hand, as intelligent agents, humans usually spend less time skills using existing ones. This motivates
learning similar tasks and can acquire about the advantages of training set of tasks with certain similarities together efficiently. Multi-task reinforcement learning (MTRL) aims train effective policy that can be applied to the same robot to solve different tasks. Compared to training each task separately, multi-task policy should be efficient in the number of parameters and training samples and benefit from the sharing process.
think
The key challenge in multi-task RL methods is determining what should be shared among tasks and how share. It is reasonable to assume the existence of similarities among all the tasks picked (usually on the same robot) since training completely different tasks together is meaningless. However, the gaps between different tasks be significant even within the set. For tasks using the same skill but with different goals, it's natural to share all the parameters and add the goal into state representation turn the policy into a coal-conditioned policy. For tasks with different skills, sharing policy parameters can be efficient for related tasks but may bring additional difficulties for uncorrelated skills (e.g., push and peg-insert-side in Meta-World (36)), due to additional challenges in learning brought by the conflictions between tasks.
Recent works multi-tase RL proposed different methods on this problem, which can be roughly divided into three categories. Some focus on modeling thare-structuse for sub aol-policies of different
* Equal contribution.
Work done while interning Horizon Robotics.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

button press
door open
drawer close
drawer open
peg side
window open
window close
Figure 1: Example tasks from Meta-World
33
while some focus more on algorithms and aim to handle conflicting gradients from different tasks losses during training (35. In addition, many works attempt to select learn better representations as better task-condition for the policies [24). In this paper, we focus on the share- structure design for multiple tasks. We propose parameter-compee compoiiiiooiiio MTRL method that learns a task- aask-agnostic parameter set forming a subspace in the policy parameter space for all tasks. We infer the task-specific policy in this subspace using compositional vector for each task. Instead of interpolating different policies' output in the action space, we directly compose the policies in the parameter space. In this way, two different tasks can have identical or independent policies. With different subspace dimensions (i.e., size of parameter set) and additional constraints, this compositional formulation can unify many previous works on sharing structures MTRL. Moreover, separating the task-specific and the task-agnostic parameter set brings advantages in dealing with instability RI training of certain tasks, which helps improve the multi-task training stability.
tasks 3 33
The key contributions of our work are summarized as below. i) We present a general Parameter Compositional (PaCo) MTRL training framework that can learn representative parameter sets used to compose policies for different tasks. ii) We introduce a scheme to stabilize MTRL training by leveraging PaCo's s decompositional structure. iii) We validate the state-of -ae--offtheeteeee performance of PaCo on Meta- World benchmark compared with number of existing methods.
Preliminaries
2.1 Markov Decision Process (MDP)
discrete-time Markov decision process defined by a tuple A, (,, where S is the state space; A is the action space: P is the transition process between states; r S A- is the reward function; μ E P(S) is distribution of the initial state, and 7 [0, 1] is the discount factor. At each time step the learning agent generates the action with policy xa ss)) as the decision. The goal is to learn a policy to maximize the accumulated discounted return.
2.2 Soft Actor-Critic
In the scope of this work, will use Soft Actor-Critic (SAC) 99 to train the universal policy for the multi-task RL problem. SAC is an off-policy actor-critic method that uses the maximum entropy framework. The parameters in SAC framework include the policy network <(a, (st) used in evaluation, the critic network Q(s,, a.) as a soft Q-function. A temperature parameter a is used to maintain the entropy level of policy. In multi-task learning, the one-hot id of task skill and the goal is appended to the state space. Different from single-task SAC, multiple tasks may have different learning dynamics. Therefore, we follow previous works (24) to assign a separate temperature ar for each task with different skills. The policy and critic function optimization procedure remains the same the single-task setting.
3 Revisiting and Analyzing Multi-Task Reinforcement Learning
3.1 Multi-Task Reinforcement Learning Setting
Each single task can be defined by unique MDP, and changes in state space, action space, transition, reward function can result in completely different tasks. MTRL, instead of solving single MDP, we solve a bunch of MDPs from a task family using a universal policy to(als). The first assumption for MTRL to have universal shared state space S and each task has disjoint state space S CS. where E T is any task from the full task distribution. In this way, the policy would be able to

---

recognize which task it is currently solving, Adding the one-hot encoding for task id is common implementation of getting disjoint state space during experiments.
In general MTRL setting, we don't It have strict restrictions on the which tasks are involved, but we assume that tasks in the full task distribution share some similarities. In real applications, depending on how a task is defined, we can divide it into Multi-Goal MTRL and Multi-Skill MTRL. For the former one, the task set is defined by various "goals" in the same environment. The reward function r" is different for each goal, but the state and transition remains the same. Typical examples of this Multi-gon goal settings are locomotion tasks like Cheetah- Velocity/Direction and all kinds of goal-conditioned manipulation tasks [18). For the later one, besides changes in goals in the same environment, the task set also involves different environments that share similar dynamics (transition functions). This happens more in manipulation tasks where different environments train different skills of a robot, and one natural example the Meta-World [36] benchmark which includes multiple goal-conditioned manipulation tasks using the same robot arm. In this setting, the state space of different tasks changes across different skills since the robot manipulating different objects (c.f. Figure D In both Multi-goal and Multi-skill setting, we have to form the set of MDPs into universal Multi-task MDP and find a universal policy that works for all tasks. For multi-goal tasks, we need to append "goal" information into state; for multi-skill tasks, we need to append "goal" (usually position) as well as "skill" (usually one-hot encoding). After getting state S the corresponding transition and reward pr, y" can be defined accordingly.
3.2 Challenges in Multi-Task Reinforcement Learning
Parameter-Sharing. Multi-task learning aims to learn a single model that can be applied to a set of different tasks. Sharing parameters allows us to take advantage of the similarities among tasks. However, the gaps between contents and difficulties of different tasks bring us the challenges on both which tasks should share the parameters and what parameters should be shared. Failure in the design may result in low success rate on certain tasks that could have been solved if trained separately. This a challenge in designing an effective structure to solve the MTRL task.
Multi-Task Training Stability. Although we have assumed some similarity in the task sets used for multi-task learning, conflicts between different skills may affect the whole training process (35). Also, failure like loss explosion in some tasks can severely affect the training of other tasks due to parameter sharing 224 In multi-task training with large task numbers, the uncertainty of single task training is enlarged. This is a challenge in designing an algorithm to avoid negative influence brought by parameter-sharing among multiple tasks.
4 Parameter-Compositonn Multi-Task RL
Motivated by the challenges in training universal policies for multiple tasks discussed Section 3 we will present a Aurameter-Composition approach to MTRL. The proposed approach is conceptually simple, yet offers opportunities in addressing the MTRL challenge as detailed in the sequel.
4.1 Formulation
In this section, we describe how we formulate the parameter-comppoitionn framework for MTRL... Given a task T, where T denotes the set of tasks with [T|=T, we use 0, E R to denote the vector of all the trainable parameters of the model (i.e., policy and critic networks) for task r. We employ the following decomposition for the task parameter vector 0,:
0, dwr,
(1)
where 4- - -.. Φx] E Raxk denotes a matrix formed by a set of K parameter vectors { ) (referred as parameter set, which is also overloaded for referring 4). each of which has the same dimensionality 0r, ie.., & E R." w, E E is compositional vector, which is implemented as a trainable embedding vector for the task index r. We refer a model with parameters in the form of Eqn. as a parrameter-compostioon
model.
By Cheetah/Ant- Difreitt/Direction, we refer to the tasks that have the same dynamics as the standard locomotion tasks but with goal of running at specific velocity in specific direction.
3

---

compositional vectors
task parameter vectors
parameter
press
8: 2
0x 4w2
W
0, @w.
door open
0, 4WY
T
1 02 0xl
insert side
Figure 2: Cesammter-Composttionnnn method (PaCo) for multi-task reinforcement learning. In this framework, the network parameter vector 0, for task instantiated a compositional form based on the shared base parameter set & and the task-specific compositional vector w,. Then the networks are used in the standard way for generating actions or computing the loss [9]. During training, will be impacted by all the task losses, while w, is impacted by the corresponding task loss only.
In the presence single task, the decomposition in Eqn. ( brings no additional benefits, as is essentially equivalent to the standard way of parameterizing the model. However, when faced with multiple tasks, in the MTRL setting considered in this work, the decomposition in Eqn. offers opportunities for tackling the challenges posed by the MTRL setting. More concretely, since Eqn. decomposes the parameters to two parts: i) task-agnostic and ii) task-aware W,, we can share the task-agnostic d across all the tasks, while still ensure task awareness via W,, leading to:
01 0,, Orl = + W1, Wr, W,.'..W. Wrl
e Φw.
For MTRL, let (6) denotes the summation of both actor and critic losses implemented in the same way as in SAC [9] for task т, the multi-task loss is defined as the summation of individual loss J, across tasks:
JØ 4 Σ, (+)
where @ denotes the collection of all the trainable parameters of both actor and critic networks. Together with Bqn.22), it can be observed that the multi-task loss Je contributes to the learning of the model parameters in two ways:
aJe/0 Σ, 0.,/00: all the T tasks contribute the learning of the shared parameter set 4:
@Je/Ow=>> aJ//Ow,: as for the training of the the task specific compositional vectors, each task loss J, will impact only its own task specific compositional vector W,-
The PaCo framework is illustrated in Figure 2
Additional implementation details about PaCo are provided Appendix AA. The proposed approach has several attractive properties towards addressing the MTRL challenges discussed earlier:
the compositional form parameters as in Eqn. offers flexible parameter sharing between tasks, by learning the appropriate compositional vectors for each task over the shared parameter set;
because of the clear separation between task-specific and task-agnostic parameters, it also offers natural solution for improving the stability of MTRL training, as detailed in the sequel.
In addition, the separation between task-specific and lusk-agnosti information has other benefits that beyond the scope the current work. For example, the task-agnostic parameter set & could be reused as pre-trained policy basis in some transfer scenarios (initial attempts in Appendix A.3
4.2 Stable Multi- Task Reinforcement Learning
One inherent challenge in MTRL is the interference during training among tasks due to parameter sharing. One consequence of this is that the failure of training on one task may adversely impact the training of other tasks (28 35). For example, it has been empirically observed that some task
losses may explode during training Meta-World [24] which will contribute significant portion in

---

Algorithm 1 Carameter-Commosstiiona MTRL (PaCo
Input: param-sel set size K, loss threshold c, learning rate À while termination condition is not satisfied do
0, = 2W,
b compose task parameter vector D loss (actor+oritic as in SAC) across tasks (Reset Step 1: loss maskout) Jn + 0 if Jn
J, (0)
Je Σx J
calculate multi-task loss
T t T AV
parameter set update
for each task do
W -W,,,- -, -V w, Jr (Wr)
B> composition parameter update
end for
(Reset Step 2: w-reset) Wyy f Equ. end while
if > c
updating the shared parameters because of their dominance. As a consequence, this will significantly impact the training of the other tasks through the shared parameters. To mitigate this issue, [24 adopted an empirical trick by terminating the training once this issue is spotted and the whole training is discarded (referred to stop-relaunch) Here we show that in PaCo, there is natural way to mitigate this issue without resorting to an ad-hoc training pipeline (24] or more expensive schemes (35].
The straightforward idea is to mask out J, of task v) with exploding loss from the total loss avoid its adverse impacts others. More specifically, once a task loss J, surpasses some threshold it will be excluded from the training loss. We will refer to this step as loss maskout.
Because of the compositional nature of the PaCo model, we can introduce further improvement beyond loss maskout. This can be achieved by re-initializing W,, without impacting the parameters of other tasks and then following the normal training. One way to re-initialize w, is:
W
ΣB,w,, JEV
3 3= ... ~ 1P
(3)
where v2 واززز < <)). and / is uniformly sampled from a unit |V| - 1-simplex A V--1 We refer to this step as w-reset. During training, w-reset offers an opportunity with exploding loss by resetting its w-parameter, which will generate new task parameter vector when composed with & and Wyy At the same time, this reset has no influence on other non-expln explloidd tasks, which provides opportunities to further improves over loss maskout. The overall stabilization scheme including both loss maskout and w-reset termed as Reset. The complete procedure of PaCo is presented in Algorithm !
continue learning for the task
the scheme of w-reset is unique feature of PaCo, due to its clear separation between aas-agnostic and task-specific parameters, Previous methods such as Soft Modularization (33 and CARE (24) cannot employ this due the lack of clear decomposition between the two parts. Instead, the stop-relaunch trick is applied to all the baselines following [24]. which is related to but more expensive than loss maskout as it discards the whole training. Results show that the proposed Reset scheme can improve training with better performance (c.f. Table
It is worthwhile point out that the ability
4.3 Unified Perspective on Some Existing Methods
Apart from the interesting compositional form and the features of PaCo, it also provides unified perspective on viewing some existing methods. Using this formulation, we are able to re-derive some existing methods with specific instantiations of & and
Single-Task Model: if set + [<b Φ ...] and w, one-hot task-id vector, this essentially instantiates a single-task model, i.e. each task has its dedicated parameters.
Multi-Task Model: if we set [Ø] e R wi=Wa=.. then all the tasks share the same parameter vector 0 Φ By taking state and the task-id id as input, we have the multi-task model.
htt.:///ttttttrtt//////////eee pyeL322
To highlight the core algorithm, we have omitted the steps that are identical to standard SAC [9. including environmental unroll, temperature tuning and target critic update.
5

---

Multi-Head Multi-Taan Task Model: by setting + as follows:
&' ØK
22
2/5
where +b, is the sub-parameeer-veetoo of the output layer for task т. Setting w, as one-hot task-id vector, we recover the multi-head model for MTRL, where all the tasks share the same trunk network parameterized by ø' with independent head sे for each task т.
Soft-Modularization
[ divides each layer into several groups "modules" ((/s)) }) and then
combines their outputs with "soft weights" =ss, 7) from another "routing" network. To obtain these soft weights, the routing network takes both the task id and state as input. Mathematicallly, can be represented as
Wsst
Sot
Jo, =
Woa for
T0g)z=((0
where & in specially structured form, with the combination done each level with "per-level" soft combination vector 2(s, r) conditioned on current state and task-id T.
The dependency of the combination vector z(s, r) on state makes it diffuse task-relevant and task-agnostic information together; therefore, all the parameters are entangled with state information and are less flexible in some use cases. For example, w-reset-like operation is inapplicable to Soft-Modularization
33 because of the mixed role of z on state and task
Related Work
Multi-Task Learning. Multi-Task learning is one of the classical paradigm for learning in the presence of multiple potentially related tasks [4]. It holds the promise that the joint learning of multiple tasks with proper way of information sharing can make the learning effective. It has been
extensively investigated from different perspectives 12
14
21
22 38 27 [ 15) and been applied in
many different fields, including computer vision [37 1 16 26 natural language processing [3] 25
and robotics 2].
Multi-Task Reinforcement Learning. The idea of multi-task learning has also been explored in MTRL, with a similar objective of improving the performance single-task RL by exploiting the similarities between different tasks. Many different approaches have been proposed in the
32 20). One of the most straight- approach to MTRL is to formulate the multi- task model as task conditiioa one (36). commonly used in coal-conditional RL 18 and visual- Ssual--anguage grounding 17). Although simple and has shown some success in certain cases, one inherent limitation is that it is more vulnerable to the negative interferences among tasks, because of the complete sharing of network parameters. [ proposes an approach by assuming the functional approximator for each task is linear combination of set of shared feature vectors, and then exploited the similarities among different tasks by employing a structured sparse penalty over the combination matrix.
literature [I 8 36 17
33 24
g utilizes mix-and-matee design of the model to facilitate transferring between tasks and robots. 77 leverages the shared knowledge between multiple tasks by using shared network followed by multiple task speciipec heads. (33 further extends these approaches by softly sharing features (activations) from base network among tasks, by generating the combination weight with an additional modularization network taking both state and task-id input. Since the base and modularization networks take state and task information as input, there is no clear separation between task-agnostc and task specifce parts. This limits its potential on tasks such as continual learning of a novel task. Differently, PaCo explores a compositional structure in the parameter space [19 30) instead in feature/activation space [3] 33), and does so in a way such that the task-agnostic and task-specifir parts are decomposed. This not only enables learning of multiple tasks, but also leads to natural schemes for stabilizing and improving MTRL training (c.f. 6.2).
Resolving Conflicts in Multi- Task Learning. Because of parameter sharing for multiple tasks (thus multiple task losses), the shared parameters are impacted by the gradients from all the task losses. Whenever the gradients is not consistent with each other, there will be conflicts in updating the shared parameter. This issue of conflicting gradients general problem that is present in general multi-task learning (35] 29], and could lead to degraded performance and unstable training

---

if not properly handled 124 sharing, but instead distilling each task policy into a centralized policy. [24 alleviates the negative effects of interference by deciding which information should be shared across tasks, using a context- based attention over a mixture of state encoders. This demonstrates the benefits of a aask-grouping mechanism but requires additional context information. There are also approaches on mitigating the interferences by balancing of the multiple tasks from the perspective of loss [10] or gradient [6]. [35] proposes to address the conflicts by gradient projection, which could be less reliable in the case where gradients are noisy, as is the case in RL. Differently, PaCo enjoys improved stability in training by using schemes leveraging its decomposed structure of task task-agnostic and task task-specific parameters.
12 bypassed the conflicting gradient issue by discarding parameter
Experiments
We now empirically test the performance of our Parameter- Parameter-Coompostiooee Compositional Multi-Task RI. framework on the Meta-Worke benchmark (36). Meta-Worke benchmark is a robotic environment consisting number of distinct manipulation tasks, with some examples tasks shown in Figure Each task itself is a goal-conditioned environment, and the state space of all the tasks has the same dimension. The action space of different task is exactly the same, but certain dimensions in the state space represent different semantic meaning in different tasks (e.g. goal position or object position).
6.1 Benchmark Results on Meta-World
Benchmarks. Meta-World 36 has been used in benchmarking many recent MTRL algorithms B3 24). In the original Meta-World Multi-task benchmark [36), each manipulation task is configured with fixed goal, therefore the learned policies are not goal-conditioned as it cannot generalize to a task of the same type by with different goals. This setting is easier, but more restrictive and less realistic in robotic learning [33] Following (33, we extend all the tasks random-gou -oollommmolllao setting and refer to the 10 task Meta-World with random goals as MT100rand.
Baselines. We compare against (i) Multi-task SAC: extended SAC 99 for MTRL with one-hot task encoding; (ii) Multi-Head SAC: SAC with shared a network apart from the output heads, which are independent for each task; (iii) SAC+FiLM: the task-conditional policy is implemented with the FiLM module 17 on top of SAC; (iv) PCGrad [35 a representative method for handling conflicting gradients during multi-task learning via gradient projection during optimization; (v) Soft- Module [33]: which learns a routing network that guides the soft combination of modules (activations) for each task; (vi) CARE [24]: recent method that achieves the ofate-of-the--e art performance Meta- World benchmark by leveraging additional task-relevant metadata for state representation [
Training Settings. The convergence performance is related to both the number of parallel environ- ments for training and the number of tasks in the environments. There is also a balance between the number of training iterations and the number of roll-out samples. For training on MT 10-radd we follow the settings introduced in (24) and use i) 10 parallel environments, ii) 20 million environment steps for the 10 tasks together (2 million per task), üii) repeated training with 10 different random seeds for each method. The implementation of PaCo and the training scripts are available. More
resources are available on the project page.
Success Rate (%)
Evaluation Metries and Results. The evaluation
Methods
(mean + std)
metric for the learned universal policy for all tasks based on the success rate of the policy for all the tasks. For World benchmarks, we evaluate each skill with 10 episodes of different sampled goals using the final policy. The success rate is then averaged across all the skills. The randomness in the MTRL training is unpredictable, some methods may converge to higher success rate right after 20M steps. and some may drop if the training continues.
Multi-Task SAC [3
62.9 1 8.0
Multi-Heea SAC 33
62.0 1 8.2
SAC + FiLM 17
58.3 4.3
PCGrad 35
61.7 1 10.9
Soft-Module 33
63.0 1 4.2
CARE 24
76.0 1 6.9
PaCo (Ours)
85.4 + 4.5
Table 1: Results on Meta-World [36] MT10 with random goals (MT10- (oTTO.)
Note that the experiments reported in the works mentioned above are implemented and evaluated on Meta- Worl-Word--Vi and/or with the fixed-goal setting. We adapt these methods and experiment on Meta- World-V2 V2.
htttps:/eeites google. con/ http://tteeccccaaggggppppp/a//op
7

---

Instead of picking the maximum evaluation success rate across training, we use the policy at 20M total environment steps (2M per task) for fair evaluation. We report the mean performance together with standard derivations of the models trained with 10 different seeds, as summarized in Table [ An improved success rate on MT110-rad benchmark can be observed compared to baseline methods. It is interesting to note that the previous state-of-the-art method CARE (24] leverages additional metadata such as language-basee resorting to such meta information, highlighting the benefits brought by algorithmic innovation itself. Note that all the baseline results are obtained with the empirical trick of terminating and re-launching training in the presence of exploding loss 24]. More results on MT10 and MT50 provided in Appendix A.1.2
task-description for helping with learning. PaCo outperforms CARE without
Single-Tash Performance. The tasks in Meta- World are designed such that they can be trained with standard single-task RI. algorithms (e.g. SAC). Given enough environmental steps (varies for different tasks), they can converge to close to 1.0 success rate. The average success rate at convergence for the 10 tasks in MT10 is around 95.0% (each SAC policy trained with 5 Million environmental steps per task). In some previous works (24), performance under this setting is regarded as "upper bound". As a reference, a 2M steps per task setting for Single-Task SAC training results in a 61.9% average success rate. Single-Tas Task SAC cannot reach its convergence performance for all tasks if limited to 2M per task for training.
6.2 Stable MTRL Training
In single task RI., training failure gradient explosion may occur due many random factors including initialization, bad exploration etc.. This is even worse for MTRL, especially on those models with shared parameters, since the loss and gradient explosion other tasks through the shared parameters, In PaCo, we use the loss maskout and w-reset schemes introduced in Section 4.2 to avoid the influence of extreme losses of certain tasks. We perform an ablation study on them to analyze their roles in stabilizing the MTRL training. We compare PaCo with two variations: i) PaCo-Maskout: the PaCo variant with only loss maskout, and ii) PaCo- Vanilla: PaCo without w-reset and loss maskout.
certain tasks would influence
Stabilization Scheme
Variations
Success Rate (%)
loss maskout w-reset
PaCo PaCo-Maskout Vaanlla
V
85.4 = 4.5
76.0 + 8.8
X 71.6 + 25.7
Table 2: Stable MTRL training results on MTI0-rand tasks.
In order show the stability precisely, we don't 't early-stop the training process even if the loss of some task has already exploded. The final success rates are shown in [ (trainin curves provided in (opendixAA.1.I] It observed that during training (c.f. training curves in Appendix [A..II] resulting from exploding loss in some cases. PaCo-Maskout variant can potentially mitigate the adverse effects of the exploding loss to some degree, but could also compromise the performance because of the reduced opportunity of learning on the masked-out tasks. The complete PaCo improves both the average success rate and the stability of MTRL training compared with both variants, demonstrating the effectiveness of the PaCo design. More details on stability comparison in the training process can be found Appendix A.I.I
has larger variation and less stable
6.3 Ablation Study
Parameter Set Size. One goal of MTRL is find good trade-off between model size and performance. The
in PaCo for controlling this off is the size of parameter set K. For a group of similar skills, we may be able reach high success rate with a small parameter set. For skills with high variance, a larger parameter set might be required due to the increased diversity.
Param Set Size
Success Rate (%)
Compositional Variations
Success Rate (%)
PaCo (K=3)
74.4 5.1
Output-only
64.0 + 5.5
PaCo (K=5)
85.4 = 4.5
Actor-only
73.3 + 5.8
PaCo (K=8)
81.0 - 8.4
AC-shared (PaCo)
85.4 + 4.5
Table 3: Impacts of (a) parameter set size and (b) compositional structure variations.
We investigated the performance of PaCo under different values for the parameter set size K. More concretely, for the benchmark MT10-rand which contains 10 different manipulation skills, we set
8

---

Sampled [-0.06, 0.99]
Sampled 10.28, 0.96]
Peg-ianert-siiee [0.59, 0.81]
Reach [-0.68, 0.74]
move towards peg
pick up peg
move towardd box
peg
peg
pick up PeR
Push
(0)
[-0.40, -0.92]
(b)
towards DOX
reach insertin OR
Figure 3: (a) Compositional vectors in the space of a unit circle: learned reach, push, peg-inset-side policies (denoted with A) and sampled policies on the unit circle. (b) Visualization (using the peg reg-insert-side task) of example policies: the learned peg-inset-side policy and two sampled ones.
3=3,5, 5, 8, and the results are summarized in Table (a). It observed that with a small value for K, PaCo can already achieve performance competitive to many baseline methods (c.f. Table I. Naively increasing K to a large number could compromise the performance, partially due to decreased sample efficiency because of the additional complexities from paremmtteetri..ioonn. However, in a special case of K=10, by initializing w as one-hot vectors, PaCo can achieve 88.9 1.9. We leave further improvements based on better initialization in the general case future work. Empirically we find setting K=5 lead to good trade-off for PaCo. Overall, the increasing in parameter set size K grows with the task number n but essentially a lower speed.
Compositional Structure Variations. PaCo employs the same compositional structure to all (actor and critic) networks (AC-shared). There several other possible variations this design choice. One variant is to use the compositional structure only for the parameters of the actor-network (Actor- only). Also, we can choose apply the structure only to the output layer (Outpul-only), which is architecturally similar to Multi-Head SAC. The results in Table (bb) show that when applied to output layer only, the performance is comparable to Multi-Head SAC (c.f. Table DD. although with less parameters (K 5) than Multi- Head SAC (K 1 10). This shows that the compositional structure is also effective in the output layer up to the performance limit imposed by the network architecture. Using the compositional structure for the full actor network (Actor-only) improves over Output-ont Applying the compositional structure to the full networks of both actor and critic can further unleash its potential and gives the best performance.
6.4 Qualitative Results on Parameter Set and Compositional Vectors
Visualization of Samples from Policy Subspace. We conduct experiments to inspect the policy subspace spanned by the learned parameter set 4 via visualization. For ease of visualization, we train PaCo on a representative set of tasks (reach, push and peg-insens peg-insert-side with random goals), using a parameter set size of K =2, i.e., = Φ2] and W = [W1, W2. W3] E 2x The three tasks cover simple, medium, and hard tasks, using the number of environmental interactions required solve the task as the criteria for the level of difficulty. To better visualize the subspace, we further add normalize activation to the compositional vectors, i.e., all the policies lie on the unit circle in the W space (the space of all possible w vectors, c.f. Figure 3 (a)). Although not the main focus of this experiment, as side note, for this group of tasks, it typically difficult to learn using one model like MT-SAC or Multi-Head SAC, with a low success rate for the most difficult task peg-insert-ssie). In contrast, PaCo reaches a success rate close to 100% across all three tasks.
In Figure 3 (a), we show the position of {w,} for three task-specific policies on the unit circle. It can be observed that: i) We are not learning all the tasks with a single column of Φ. Instead, all columns of & are involved in solving the three tasks with different compositional weights. ii) We are able to find policies for three completely different manipulation skills on this ID low-dimensional manifold in the policy parameter space. This aligns well with the purpose of multi-task learning in sharing parameters. iii) The learned two column of parameter set (Φ,, Φ2) can be viewed as "basis" policies. Although individually, they may not necessarily correspond to fully capable skill for particular task, collectively, they can be used to instantiate various other policies through parameter space composition, including the policies for solving the three example tasks (reuch, push and peg- insert-side). Since ( <b,) are not random vectors but have already been trained on multiple tasks, the policies instantiated with a random w could lead to policies with different types of behaviors,

---

possibly with some interactions with objects in the environment. As a test, we sampled some points on the unit circle (labeled as Sampled), The resulting policies are visualized in Figure (b)) showing various behaviors with some potential interactions with objects in the peg-insert-side environment.
(
Visualization of Compositional Vectors. We visualize the learned compositional vectors for MTI0- rand in 2D space (referred as w-space) via Principal Component Analysis (PCA) in Figure There are several interesting observations: i) Some skills could lie in different part of the w-space, e.g. reach v.s. others; ii) Some skills are close in the w-space, e.g. window-open v.s door-open, window-open v.s. window- close in Figure 4 iii) Another interesting observation is that some skills that are not literally related are also appear
Principal Compositiona Vector
reach
1.0
0.5
pick-place
to be close in the w-space learned by PaCo, e.g. peg- insert v.s. wwwpo//oopee/indowwooooeeeeeeeee caassedddorr-pendrraweer open, Although literally distinct, peg- insert and the other skills mentioned above are related from the perspective of behavior, Le., first interacting with object (e.g.,
0.0
button drawer-close
top-down
door open
-0.5
push
peg-insert
drawer-opan
window-open
kindow-close
-2
0
peglwindowldoor), and then taking a trajectory of mo- tions to accomplish the subsequent operation, (e.g. in- serl/window/door). Therefore, these literally unrelated skills appear be semantically related at the behavior level. This is something that could be useful but not able to be leveraged by CARE [24]. implying one possible reason why PaCo more effective. The learned compositional vectors as well more visualization results can be found in Appendix A.1.4
Principal Compositional Vector
Figure 4: 2D PCA projection of the com- positional vectors.
7 Conclusion, Limitation and Future Work
Based on a revisit to MTRL, and its challenges, we present PaCo, simple parameter compositional approach as way to mitigates some of these challenges. The proposed approach has the benefits of clear separation between task-agnostic and task ssee-fpe components, which is not only flexible in learning but also useful for stabilizing and improving MTRL. Without resorting more complicated design [33] or additional metadata (24], PaCo has demonstrated clear improvement over current state-of-the-art methods standard benchmarks.
One limitation of the proposed approach that simple linear compositional form is used globally, which may limit its representation power in some cases. How to extend the current method beyond simple global linear compositional form while retaining its key advantages is one interesting direction to explore in the future.
There several other interesting directions that are worth of exploration well (with some initial attempts in Appendix [A.3). For example, incorporating a higher-level mechanism for adjusting the task distributions based on task difficulties or progresses [34] to further improve the training of the more difficult tasks. The separation between task-agnostic and task-relevant parameters in PaCo also suggests a possibility for extending transfer learning.
Acknowledgement
We would like to thank group members at Horizon Robotics and anonymous reviewers for discussions and feedback on the project and paper. We would also like to thank the Horizon Al platform team for infrastructure support.
10

---

References
[1] Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning. 2017.
[2] H. Arora, R. Kumar, J. Krone, and C. Li. Multi-task learning for continuous control. CoRR. abs/1 has/1802.01034, 2018.
[3] D. Calandriello, A. Lazaric, and M. Restelli. Sparse multi-tass reinforcement learning. In Advances in Neural Information Processing Systems, 2014.
[4] R. Caruana. Multitask learning. Mach. Learn., 28(1): Jul 1997.
[5] J. Chen, J. Zhou, and J. Ye. Integrating low-rank and group-sparse structures for robust multi-task learning. In International Conference on Knowledge Discovery and Data Mining, 2011.
[6] Z. Chen, Badrinarayanan, C.-Y. Lee, and A. Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, 2018.
[7] C. D'Eramo, D. Tateo, A. Bonarini, M. Restelli, and Peters. Sharing knowledge multi-task deep reinforcement learning. In International Conference on Learning Representations, 2020.
[8] C. Devin, Gupta, T. Darrell, Abbeel, and Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In International Conference on Robotics and Automation, 2017.
[9] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
[10] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. CoRR, abs/1809.04474, 2018.
[11] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski. C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning scale. CoRR. abs/2104.08212,, 2021.
[12] Z. Kang, K. Grauman, and F. Sha. Learning with whom to share in multi-task feature learning. In International Conference on Machine Learning, 2011. 2011.
[13] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[14] A. Kumar and H. Daumé. Learning task grouping and overlap in multi-task learning. In International Conference on International Conference on Machine Learning, 2012.
[15) X. Lin, H.-L. Zhen, Z. Li, Q. Zhang, and S. Kwong. Pareto multi-task learning. In Thirty-third Conference on Neural Information Processing Systems (NeurIPS), pages 12037-12047, 2019.
[16] S. Liu, E. Johns, and A. J. Davison. End-to-end multi-task learning with attention. In IEEE Conference Computer Vision and Pattern Recognition, 2019.
[17] E. Perez, Strub, de Vries, Dumoulin, and A. Courville. FiLM: Visual reasoning with
a general conditioning layer. CoRR, abs/1709.07871 2017.
[18] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, V. Kumar, and W. Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/1802.00964, 2018.
[19] M. Plappert, R. Houthooft, Dhariwal, S. Sidor, R. Y. Chen, X. Chen, Asfour, Abbeel, and M. Andrychowicz. Parameter space noise for exploration. CoRR, abs/ 776.01999 2017.
11

---

[20] E. Sarafian, S. Keynan, and S. Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. Proceedings of the 38th International Conference on Machine Learning. 2021.
[21] O. Sener and V. Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, 2018.
[22] V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
[23] S. Sodhani and A. Zhang. MTRL- multi task rl algorithms. Github, 2021.
[24] S. Sodhani, A. Zhang, and Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning. 2021.
[25] S. Subramanian, A. Trischler, Y. Bengio, and C. J. Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference on Learning Representations, 2018.
[26] G. Sun, T. Probst, D. P. Paudel, N. Popović, M. Kanakis, J. Patel, D. Dai, and L. Van Gool. Task switching network for multi-task learning. In IEEE International Conference on Computer Vision, 2021.
[27] Y. Tay, Z. Zhao, D. Bahri, D. Metzler, and D.-C. Juan. Hypergrid transformers: Towards single model for multiple tasks. In International Conference on Learning Representations, 2021.
(28) Y. Teh, V. Bapst, W. M. Czarnekki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. Advances Neural Information Processing Systems, 2017.
[29] Z. Wang. Y. Tsvetkov, O.
Firat, and Y. Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Conference on Learning Representattons, 2021.
[30] H. Xu, C. Zhang, J. Wang. D. Ouyang, Y. Zheng, and Shao. Exploring parameter space with structured noise for meta-reinforcemmnt learning. In International Joint Conference on Artificial Intelligence, 2020.
[31] Y. Xu, X. Liu, Shen, Liu, and J. Gao. Multi-task learning with sample re-whigtting for machine reading comprehension. In Conference of the North American Chapter of the Association for Computational Linguisties, 2019.
[32] Z. Xu, K. Wu, Z. Che, Tang, and Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. In Advances in Neural Information Processing Systems, 2020.
[33] R. Yang, H. Xu, Y. WU, and X. Wang. Multi-task reinforcement learning with soft modulariza- tion. In Advances in Neural Information Processing Systems, 2020.
[34] H. Yao, Y. Wang, Y. Wei, P. Zhao, M. Mahdavi, D. Lian, and C. Finn. Meta-learning with an adaptive task scheduler. In Advances in Neural Information Processing Systems, 2021.
[35] Yu, S. Kumar, Gupta, S. Levine, K. Hausman, and Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, 2020.
[36] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2019.
[37] H. Zhang, N. M. Nasrabadi, Y. Zhang, and T. S. Huang. Multi-observation visual recognition via joint dynamic sparse representation. In IEEE International Conference on Computer Vision, 2011.
[38] Y. Zhang and Q Yang. A survey on multi-task learning. CoRR, abs/1 707.70 08100,1111. 14, 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] An anonymized placeholder URL is included in paper. Will replace it with de-anonymized URL and release code at a future time point.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] And will also release the training scripts along with the code.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provided details on computational resources in the supplementary file (Appendix Section 2.3), due to limited space in the main paper.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "fFuc1H3Uj-", "reviewer": "Reviewer_U1FU", "summary": "Authors propose a simple, yet effective way to learn parameters in multi-task RL setting, where parameters for a task are a task-specific linear combinations of task-agnostic parameters. Parameters of linear combinations as well as task-agnostic parameters are both learned end to end. This simplicity allows for resets of linear combination parameters for tasks with exploding losses, which should lead to more stable learning. To this end, authors propose to use a random convex-combination of non-exploding linear combination parameters as a stabilization scheme. ", "questions": "- In lines 65-66 you mention that you use separate temperature for each task with different skills. Does that mean that you use 10 different values of alpha in MT10-rand? \n  - If yes: Why is this necessary? Is the performance sensitive to the choice of these parameters and would it degrade with single value?\n  - If no: Can you please explain what you meant by this?\n- I am wondering about the relation to soft-modularization. As far as I can tell it seems to be more general because it allows for dependency on s. Your method should then be a specific case of this ($z$ does not depend on s and are shared for all $m$). Why should it then be less flexible (line 189.)? \n- I am wondering about the performance of single-task model where you use separate network for each task (line 175). Have you tried to run such baseline?\n- How does the value of parameter epsilon affect performance? Have you tried runs with different values? If yes what was the performance?\n- I am surprised that in table 3 K=8 performs worse than K=5 since the model with K=8 should be able to represent anything that K=5 can. In the footnote you also mention that K=10 performs better than K=5. Do you have performance metrics for other K? Why does high K lead to worse results(you mention additional complexities from over-parametrization, could you elaborate on that)? \n- Are the networks used for other baselines the same? Do all methods have same amount of hyper-parameters?\n", "limitations": "I do not foresee any negative social impact. Limitations of the work seem to be addressed as far as I can tell.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think that the paper is well written and presents a reasonably novel and interesting idea with surprising results. I do have some questions about experiments and baselines and I am a bit surprised that the algorithm performs worse with more parameters. Therefore I am not certain if more experimental evaluation is not needed. I also think that the contribution and the paper could be strengthened by some theoretical results/motivation or intuition about why the particular proposed reset scheme increases the performance by so much (10-15%). I do not yet have a very strong opinion about whether the paper should be accepted although I am leaning more towards acceptance. I am willing to change my score based on the discussion with authors and fellow reviewers.\n\n**Strengths**:\n- Paper is in general well written and clear, the setting and method are well explained. I liked section 3 in particular\n- In my view the idea seems novel enough even though part of it is closely related to previous approaches (specific case of Soft-Modularization?). However, I am not very familiar with current Multi-task RL approaches so I am not 100% certain about this.\n- The results indicate that proposed re-initialization is quite effective on a common Meta-world benchmark (MT10)\n\n**Weaknesses**:\n- Only one domain is considered which might make it hard to judge the significance and scalability of the results especially since one of the ablations is a bit counter-intuitive (see questions). Appendix seems to contain experiments from a larger MT50 domain. However, these are not part of the main text.\n- Experiment are mostly sound and include ablations. However, I do have several questions about them and would like to (among others) see learning curves from the experiments or intermediate/max values of the performance metrics instead of just final values\n\n**Comments/Suggestions/Typos (do not affect the score):**\n- line 80: use former one instead of previous one?\n- line 84: I am not sure if about \"share similar dynamics (actions space)\" in this case since I presume action space is shared in general. Maybe transition function could fit better in this context?\n- line 156 $\\eta$ is used again but denotes something different than in background section as far as I can tell\n- line 159: \"compositional natural\" nature?\n- line 331 \"tasks covers\" cover\n\n_______________________________________________________________\nUpdate: I believe that the concerns about the domain and missing intermediate scores were sufficiently addressed by the authors. I have therefore increased my score. However, after observing the results, I also agree with reviewer CCrp that seeing the results beyond 2 million would be helpful. This is because while some method (SAC+ Film, PCGrad) have already converged, others (Care, Paco, single-task SAC) seem to still be improving so it would be nice to see how training progresses further and if/how long Paco keeps an advantage. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HDKtUvjbfq", "reviewer": "Reviewer_DW33", "summary": "The paper proposes PaCo, a framework for multi-task learning in RL. It uses a set of parameters that are composed in a unique way for each task. A compositional vector w is learned for each task, which weights the items in the parameter set. This results in a weighted policy for each task that relies on shared underlying parameters. This can help with learning shared structure across tasks and improve sample efficiency.", "questions": "* Are all experiments using 10 seeds? I see that table 1 is using 10 seeds, but I'm not sure about tables 2 and 3. For example, the std of PaCo-vanilla in table 2 seems very high and I'd expect it to be lower with 10 seeds.\n\n* Page 7, footnote 3: If these baselines were implemented for a different setting, are there any hyperparameters that need to be re-tuned for MetaWorldV2? It would be helpful to know which hyperparameters from appendix table 4 were tuned for PaCo specifically and what values for each were included in the search.\n\n* I'd like to see a baseline where each task has its own policy (no parameter sharing) and another baseline where the size of the parameter set (K) is the same as the number of tasks. In the K=10 case for MT10, for example, does each task essentially claim its own item in the parameter set and there is limited parameter sharing?\n\n* I'm curious about the importance of assigning/tuning separate temperatures for each task (L89). Is this a crucial aspect of PaCo's success over baselines?\n\nMinor (no need to comment on the below):\n* Appendix section 3.1: I don't think this section is that important, but if it's going to stay in the paper, please add a figure with evidence of this experiment (e.g. reward curves for PaCo and the modified ratio PaCo).\n\n* L313: \"Overall, the increasing in parameter set size K grows with the task number n but essentially at a lower speed.\" I don't see enough evidence for this claim given that the parameter set size ablation is only shown for MT10 (10 tasks). I would probably remove the claim unless the parameter set size ablation can be repeated on MT20 or MT50 (to show how it scales).\n\nTypos:\n* L136: details about PaCo is --> details about PaCo are\n* L159: compositional natural --> compositional nature\n* L294: some task already explodes --> some task has already exploded", "limitations": "I appreciate the limitation discussion of the linear compositional form.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work is original and clear with strong results on a multitask benchmark. If my concerns are addressed (e.g. number of seeds, hyperparameters, and a couple of unsupported claims), I support the acceptance of this paper.\n \nStrengths\n* Parameter sharing is an interesting area for improving sample efficiency in MTRL, and I have not seen published work that composes policies in this way at the parameter level.\n* The interpolated policy visualization in Figure 3 is unique and interesting.\n\nWeaknesses\n* One of the potential benefits of this framework is the ability to add new tasks after training the parameter set, which is mentioned on L144 and described in appendix section 3.2 (which says \"in experiments, we do find successful extensions from existing skill set to a new skill when the skills are similar\"). This claim is unfortunately unsupported. Please either provide evidence of such experiments or do not include the claim.\n\n-------\nUpdate: I thank the authors for their responses. The concerns (especially about number of seeds and hyperparameters) were sufficiently addressed; thus I am increasing my score from borderline accept to weak accept. I do recommend expanding the discussion/analysis of new results, i.e. \"continual learning without forgetting\" (what to compare these numbers to) and single-task SAC vs K=10 vs K=10 with one-hot initialization (what relative performance numbers mean for these three versions).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C8VEfbxcWir", "reviewer": "Reviewer_CCrp", "summary": "The paper proposes an approach for composing parameters for solving multi-task RL problems and obtains promising results on MetaWorld benchmark", "questions": "1. The paper mentions a \"stop-relaunch\" trick ion [23] but I could not find a reference to that trick in [23[.\n\n2. In line 154, it says \" in PaCo, there is a natural way to mitigate this issue without resorting to ad-hoc [23] or more expensive schemes [34].\" Isnt the loss maskout another adhoc trick. Infact this trick also introduces a new hyper-parameter.\n\n3. I am not sure I understand why w-reset is used. I understand that we can do it but isnt PaCO already ignoring the loss when it is greater than $\\eps$. I didnt understand the need for both loss masking and w-reset\n\n4. It would be helpful if the authors can comment on the \"novelty\" of the work. Is the main novelty that they are doing composition in the parameter space, instead of the function space? I am not considering this as a weakness - I just want to make sure I am not missing anything here.\n\n5. Could the authors add details about the computation overhead of the different approaches?\n\n6. The analysis related to Figure 4 is not very convincing. eg arguing that \"peg-insert v.s. window-open/window-close/door open/drawer-open are related from the perspective of behavior, i.e., first interacting with an object (e.g., peg/window/door), and then taking a trajectory of motions to accomplish the subsequent operation, (e.g. insert/window/door)\" does not sound convincing as the same argument applied to \"pick-place\" while it is quite far in the latent space. ", "limitations": "Covered in section 7", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. It is a relatively straightforward approach that works well in practice. \n2. The paper is mostly well written (though see next section)\n3. The baselines are both recent and relevant.\n\n## Weakness\n\n1. The paper should add the \"one-model-per-task\" baseline to their results, to get a sense of performance gap.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept