Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning
Authors: Minjong Yoo
Sangwoo Cho
Honguk Woo
Venue: NeurIPS 2022 Accept (2022)
Keywords: mutli-task reinforcement learning
offline reinforcement learning
task inference
skill embedding

Abstract: Reinforcement learning (RL) with diverse offline datasets can have the advantage of leveraging the relation of multiple tasks and the common skills learned across those tasks, hence allowing us to deal with real-world complex problems efficiently in a data-driven way.  In offline RL where only offline data is used and online interaction with the environment is restricted, it is yet difficult to achieve the optimal policy for multiple tasks, especially when the data quality varies for the tasks. In this paper, we present a skill-based multi-task RL technique on heterogeneous datasets that are generated by behavior policies of different quality. To learn the shareable knowledge across those datasets effectively, we employ a task decomposition method for which common skills are jointly learned and used as guidance to reformulate a task in shared and achievable subtasks. In this joint learning, we use Wasserstein Auto-Encoder (WAE) to represent both skills and tasks on the same latent space and use the quality-weighted loss as a regularization term to induce tasks to be decomposed into subtasks that are more consistent with high-quality skills than others. To improve the performance of offline RL agents learned on the latent space, we also augment datasets with imaginary trajectories relevant to high-quality skills for each task. Through experiments, we show that our multi-task offline RL approach is robust to different-quality datasets and it outperforms other state-of-the-art algorithms for several robotic manipulation tasks and drone navigation tasks.

Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning
Minjong Yoo, Sangwoo Cho, Honguk Woo" Department of Computer Science and Engineering Sungkyunkwan University {mjyoo2, jsw7460, hwoo)@skku.eedu edu
Abstract
Reinforcement learning (RL) with diverse offline datasets can have the advantage of leveraging the relation of multiple tasks and the common skills learned across those tasks, hence allowing us to deal with real-world complex problems efficiently in a data-driven way. In offline RL where only offline data is used and online interaction with the environment is restricted, it is yet difficult to achieve the optimal policy for multiple tasks, especially when the data quality varies for the tasks. In this paper, we present a skill-based multi-task RL technique on heterogeneous datasets that are generated by behavior policies of different quality. To learn the shareable knowledge across those datasets effectively, we employ a task decomposition method for which common skills are jointly learned and used guidance reformulate task in shared and achievable subtasks. In this joint learning, we use Wasserstein auto-encoder (WAE) to represent both skills and tasks on the same latent space and use the tasks to be decomposed into subtasks that are more consistent with high-quality skills than others. To improve the performance of offline RL agents learned on the latent space, we also augment datasets with imaginary trajectories relevant to high-quality skills for each task. Through experiments, we show that our multi- task offline RL approach is robust to the mixed configurations of different-quality datasets and it outperforms other state-of-the th-tt---rttttttto- algorithms for several robotic manipulation tasks and drone navigation tasks.
loss as regularization term to induce
1 Introduction
In the reinforcement learning (RL) field, the offline RL research has recently gained much attention, as numerous works showed that is effective for various problems of sequential decision making to adopt
data-driven learning mechanism using previously collected data of experiences and trajectories [1]
2. In the meanwhile, multi-task RI. is considered promising to enhance the generality of RT.
policies and improve the learning efficiency [4 5 6 7
TO]. Recently, a data sharing method
for multi-task learning was introduced to address the issue of limited data for real-world control
applications [II. Yet, multi-task RL has not been fully investigated offline settings.
In the offline RL context, we present a novel multi-task model by which a single policy for multiple tasks can be data-efficiently achieved and its learning procedure is robust heterogeneous datasets of different quality. In offline RL where interaction with the environment is not allowed and arbitrary or low- performance behavior policies might be involved in data collection. is important maintain the robustness in learning on different-quality data. To this end, we devise joint learning mechanism of skill (short-term action sequences from the datasets) and task representation, which enables the task decomposition into achievable subtasks via quality-aware aware skill regularization. We also employ
Honguk Woo is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

data augmentation based on high-quality skills, thus creating plausible trajectories and alleviating the limited quality and scale issues of offline datasets. Through experiments, we demonstrate that our model achieves robust performance for multi-task robotic manipulation and drone navigation, without requiring additional interaction with the environment.
The main contributions of this paper are summarized as follows.
We present novel multi-task offline RL model that enables the task decomposition into achievable subtasks through the quality-aware joint learning on skills and tasks. The model ensures the robustness of learned policies upon the mixed configurations of different-quality datasets.
We devise the data augmentation scheme specific multi-task RL on limited offline datasets. aiming at creating imaginary trajectories that are likely to be generated by expert policies.
We evaluate our model under multi-task robot and drone scenarios and demonstrate its benefit particularly upon heterogeneous data conditions.
2
Overall Approach
In this section, we describe the problem of making optimal decisions upon a multi-task Markov decision process (MDP) and briefly present our data-driven approach to the problem.
2.1 Preliminary
Reinforcement learning (RL) offers an active learning framework based on MDPs for tackling sequential decision problems. In conventional RL formulation, a learning environment is represented MDP M with AS, P, R, 7) where S is state space, AA is an action space, P(* 88 ae) is
a transition probability for states St, St+1 E S and an action 0₂ E A. R(st, Qc) is a reward function, and y E [o. 1] is a discount factor. The objective of an RL agent is find an optimal policy "² (als) by which the cumulative discounted reward ((T) = E, وΣح= y're] [12] is maximized. In general, the policy æ is learned via continual interaction with learning environment defined in an MDP.
Offline RL Offline RL aims at maximizing the cumulative discounted reward J(x) as explained above within the same MDP formulation; however, unlike conventional RL, it is assumed to use only static datasets of previously collected trajectories D = {(.,, Cr, Tt, 1t)+ for training. Likewise, it rarely considers direct interaction with the environment. Offline RL algorithms can increase the usability of previously collected data in the domain of making sequential decisions where temporal credit assignment with long time horizons important.
Multi-task RL Multi-task RL considers more than a single task when achieving the optimal policy rr It is normally formulated as family of MDPs (T: A, R,, 7)); where each individual task T is associated with its respective MDP and it is sampled according to a task distribution p(T).
Hidden Parameter MDP To represent the implicit temporal dynamics properties in multi-task environment, which are relevant to the Markovian properties of each task, we introduce a hidden latent variable et [3). That is, we have R(st, Vt, ax) == Rx,,,, 0t) and P(s Vt+1 [0e, 8t, U2.) == Po, (S141 (a,, 8t) for St, 8t+1 E and a, E A, where the actual state space extended to S x and V is the collection of latent variables vt. Then, we obtain the respective partially observable MDP (POMDP) that is formulated as a tuple (S x V, A, Q2 Pv, Ry, O, 7) where Q = S and O(st, v.) + s; denote the observation space and the observation function, respectively.
2.2 Overall Approach For Multi-task Offline RL
In principle, offline RL algorithms train a policy π by optimizing the objective argmax, (T)
a c(T,, TD) which tends to mitigate the extrapolation problem [1]. J/D is the average return of the learned policy ı in the empirical MDP M derived from static datasets D, which are generated by a behavior policy Tp. Some cost function c(-) is used to regulate the distance of the policies л and TD and is hyperparameter. With this regularization by the behavior policy, offline RL algorithms are often vulnerable to low-quality datasets. Overfitting problems can occur such that the maximum average return max, JD(TT) of M is much lower than that of its respective true MDP M, when low-performance or arbitrary policy is used for data generation.
ii

---

(a) Skill regularized task decomposition
(b) Data augmentation by imaginary dcmonstrations
Task I High quiliy sab-rnjectory
Task emboddings
Latent space Z
Imaginary demonstrations
Offine RL training
E
G0
Multi-task offline datasets with imaginary demo.
Skill embeddings
Task N Low quality sib-mjjectry
ZN
D
Sabtask embedding
bx
50
Policy m(a(5,7)
Task embeddinng
Skill embeddings
( High quality case
(
Low quality case
Figure I: Our proposed multi-task offline RL model consisting of (a) task decomposition and (b) data augmentation. In (a), sub-trajectories from static datasets are converted into skill embeddings and task embeddings on the same latent space, which together enable the decomposition of tasks into achievable subtasks. The bluc-colored dots denote task embeddings that model the environment, and the green-colored dots denote skill embeddings. In the green-colored dotted circle, a sub-trajectory T1 of task is embedded as 2 and then located as z1 closer to its corresponding high-quality skill bj (the action sequence of the sub- sub-trajectory Ti with large returns), while in the red-colored dotted circle, another sub-tr trajectryy IN of task N is embedded as n and located as 2'y further from its corresponding low-quality skill by (the action sequence of the sub-trajectory TN with small returns). In (b), for training offline RL agents, imaginary trajectories similar to expert demonstrations are sampled from the latent space and added to the datasets.
In multi-task offline RL, we reformulate family of MDPs {Ti): as a hidden parameter MDP in that multiple MDPs are combined into a single POMDP based on hidden parameters that specify temporal Markovian properties of the environment. While the overfitting issue of offline RI can be alleviated by exploring the relation of multiple tasks and inducing the shareable knowledge from their datasets multi-task setting, is not guaranteed that inferring the hidden parameters fully enables the well- well-structured representation of related tasks. It is because the behavior policy heterogeneity and state-action pair disparity of tasks can prevent the sub-trajectories of common-knowd knowledge tasks from being closely mapped on the latent space [14] 15
To address the issue of different- quality and heterogeneous datasets in the context of multi-tasi offline RL, we take novel task embedding approach that combines (a) regl-regulrri task decomposition and (b) data augmentation by imaginary demonstrations, as illustrated Figure I (a) The former enables the decomposition and reformulation of an individual task in the latent space of achievable subtasks by jointly learning the common skills and adapting the subtasks in more achievable repre- sentation via quality-aware skill regularization (in Section 3). This task decomposition establishes subtask embeddings on the same latent space in which agents can be efficiently trained. (b) The latter improves the performance of offline RL agents trained on the latent space by augmenting datasets with imaginary demonstrations (in Section 4. For each task, its well-matched skills are consequently used to generate additional trajectories similar to expert data without online interaction. This can improve the adaptability of offline RL algorithms for such tasks that have only low-quality data or do not exist in the static datasets. Then, using the subtask decomposition and augmented datasets with imaginary demonstrations, a multi-task policy is learned via offline RI. algorithms.
3 Task Decomposition with Quality-aware Skill Regularization
In this section, we describe the task decomposition model with quality-aware skill regularization on multi-task offline datasets. The model includes skill embedding and task embedding networks that are jointly learned with a regularization term based on the behavior quality, as shown in Figure 2 Specifically. we define short-term action sequences as skills and embed them in a latent space. By jointly learning skill embeddings and task embeddings, we induce tasks to be transformed into subtasks that are achievable by and aligned with skills. In doing so, we incorporate the behavior quality of each sub-trajectory into the joint learning procedure as part of skill regularization, thus mitigating the adverse effect by low-quality data.

---

Restrict 2 to prior distribution
Quality-aware skill regularization
LpB
Subtask prior distribution
Task encoder Q
Task
Ras
Sabtnak umboddings
decoder pa
4
Quality aware skil regularization
LsR
reconstruction
Transitions
LTE
Skill
Skill encoder 9
Skill decoder P
Skill
91
Skill enbesldings
Low qualityi task embedkings
Action saquence
Ise
Hiihh-uulit task embodddigss
reconstruction
Figure 2: Task decomposition procedure with quality-aware aware skill regularization. In the right side of the figure, the red arrow denotes LPR in (2) that makes low-quality sub-trajectories stretch within the prior distribution of tasks (in gray), and the blue arrow denotes Lsn in (4) that makes high-quality sub-trajectories shrink around the distribution of skills (in blue).
3.1 Learning Skill Embeddings
To represent the agent's 's behavior to a latent space 2, we use an auto-encoder mechanism. An encoder de takes as input sequence of state-action pairs d, (s, ()) for time interval f--n, n. + n - 1]. mapping to a latent vector b, E Z, while a decoder P& reconstructs the input action sequence n--n=t-n-1 from the pair of b, and $t-n:t-n
We term the latent vector b, skill embeddings, considering that action sequences on short-term horizons capture agent's 's behaviors for Wasserstein auto-encoder (WAE) [16) with the maximum mean discrepancy (MMD)-based penalty and prior distribution b, Then, the loss function defined as
specific task. For maintaining the learning stability skill embeddings by 2,
LoE(Ø)
Σ ||a,+s
)) + X LPR((b)) ao ((
where (6:)m1 ~ Pn sampled by a prior of skill embeddings, À > 0 a prior distribution- based regulation hyperparameter, and {8t. Ue,, di ym D. Here, LPR is used to restrict skill embeddings,
LPR(Eb) (b)) m(m - 1) izj
Σ
k(bs, by) m(m 1)
Σ k(b,,b,) ifj
m Σ ((b.,b)) (2)
where m is the size of {b}, {b} and k 2 x
x Z IR is the positive-definite reproducing kernel.
3.2 Skill-regularized Task Decomposition
To decompose an individual task into set of shared and achievable subtasks, we use skill embeddings as guidance for the analysis of relation between tasks. We first view each task as composition of subtasks which can be modeled a hidden parameter MDP. For task embeddings, we then use the WAE-based model architecture similar to skill embeddings previously described. For sub- trajectories T. = (St-nut: Ot-n- 1, (nt-nn of n-length transitions each, we have an encoder
qo Tr E Z to yield task embeddings and a decoder Pe (St, de, #t) - (8141,7e) Tr) to express the transition probability and reward function Rt. Then, for (8t, O1 Tt, Jmm D, the loss function for task embeddings is defined as
LTE(O)= m
(94.4i+1,7t44 pe (s ar,ti (e4x)))
(3)
We also add skill regularization term. Since in offline RI. datasets, trajectories are not necessarily generated by experts or optimal policies for all tasks, we incorporate the quality of trajectories in
iv

---

Algorithm 1 Skill regularized task decomposition
Offline datasct D. subtask embedding parameter 8, skill embedding parameter +
Regulation hyperparameter A, batch size m. learning rate
loop
Sumple (de,, 8t,,,,,,110 -td at; Mig- tn F4, mast; )im D
{bi,, (by.:11100 bem ) = gol , dig. dim, Hb, (2t E1g, .............................. se (T(( Tlg. Thm
(bo, bi........ bu} ~ PB= N(., 1). {Eo, F1.... 3n} Pz A(
(o) = 1 -i=1 Σ -=n-nlloctt>
Po(s,, b12 (20))++++00 LpR((D,, Jim (b:) using
m
(T)(0) = 1
m -i=1
using B
(8) 1
m -i=1 R()t, [11) (o((),) la using
LSRTD(0) = LTæ(0) + LpR([=; 12 (&.)=,)) (S)(0) using
ó VLSE, 0 VLSTTD
end loop return 8,
the regularization term. Once sub-trajectories are transformed skill embeddings, their quality is estimated by the episodic returns R(ss a). The quality-aware skill regularization loss is defined as
LsR(
R(s ar,) (Ilgo(TT))) - go (dr,
m
The overall loss is defined as
- + LsR(O) + ALPR((E) Pz, go (o((Te)
(5)
where Pz is a prior of task embeddings. This allows the encoder ge to generate the embeddings at subtask-leve (or subtask embeddings) upon a sequence of sub-trajectories which are agnostic to tasks in multi-task settings. In particular, each task is represented within close proximity to some high-quality skills learned from the trajectories with large episodic returns. By increasing the use of high-quality skills task- aggossigttc this task decomposition reduces the adverse effect of low-quality data and induces the task decomposition into more achievable subtasks. When training offline RL agents, use the output of go part of the state input to RL algorithms. Algorithm iimplements the learning procedure explained in (I-55..
Here we provide the analysis of skill regularization effects in 44 Let q and p be skill encoder and decoder obtained by minimizing the loss in ( and consider p as part of the environment similarly in [1 [8] Then, the decoder follows the MDP MM (S, A = Z, PP, R2, 7) in which a high-level (skill) action 2; E Z is converted to such a low-level (primitive) action at ~ p(-1se, St.22 2t) that directly interacts with the environment. Furthermore, assume that sub-trajectories T in (3) and a sequence of state-action pairs d in ① are restricted to the current state. Then, we obtain such high-level policies go and q which
trained for the MDP M As the output of qa is contained in the input states of M", our objective is to maximize the performance gap between qe and q; ie., maximize n(() Jp(o - [q) where J, is the average return in MP. Following (19), obtain that ()) = Eg zz- R2 - (x)] where dg. is a state visit distribution induced by qa. R2.- is an episodic return induced by g. and V4 is a value function of q. However, in offline RL, it difficult to approximate qo precisely, so we rather want to use the distribution of q for the state visitation distribution of qe without much propagation error. To do this, we optimize ni()) Es [Rg, [s)] under the restriction that g and go remain in close proximity (20).
maximize, n(0) := E.nd, . R - V*)) subject to K L(go ((g((\1)))l(((1))) <
(6)
Under the KL-divergence constraint in (6). we obtain that the state distribution of d, corresponds to that of the dataset D, implying d, D.
Using the classical Lagrangian argument, we can simplify the optimization problem in 66 into a nonconstraint optimization problem.
LsR(0) Eong [R [s)] + B(E (L(((((s)))(((1s)))

---

(a) Skill decoder training
(b) Data augmentation
Imaginary demo.
Datasets
St-1
embbeddings
Submack enioeddings
RL-1
Skill decoder P
Skill
decoder Psp
Pe (p)
de
Po bt)
optimize
Task decoder pe
St-1. Po(s, Rg.22)
Figure 3: Data augmentation by imaginary demonstrations
where 33 is Lagrangian multiplier. By differentiating the right-hand side of 7 with respect to qo and following the optimal policy derivation procedure in 22). we obtain the closed form solution that satisfies the return- -weighee condition below.
qo (-1s) oc exp( (R2. - (+)3)))((1ss)
When omitting the baseline term (s) and up to the constant, we also obtain that the weighted skill regularized loss 4 renders subtask embeddings to matched with high-quality skills for a given task, thereby facilitating the task decomposition into shareable and achievable subtasks.
4 Data Augmentation by Imaginary Demonstrations
In offline RL, since a given static dataset might not fully represent its respective true MDP and further exploration is not allowed, it is common for RL agents to experience sub-optimal performance. Generative models and noise are used to generate additional trajectories and enable local exploration for offline RL algorithms without interaction. In this section, we introduce a data augmentation method specific to the aforementioned task decomposition with quality-aware skill regularization, so that we can tackle the overfitting and limited performance issue. While existing works aim reducing the adverse effect of unseen states by exploiting state augmentation methods (23) 24 225. we focus on augmenting such trajectories (imaginary demonstrations) that are likely to be generated by a high-qualit skill-based learned policy.
Specifically, we integrate the task decoder po and skill decoder ps into generative model, as illustrated in Figure 3 Note that both po and pa are established through the skill-regularized task decomposition. Then, obtain
âe, ($141, fi) = Po 21). Po(Se, ae, 21) where 2, ge(TT)
(9)
where go is the task embedding model and ~ D. Note that in this generative model, po performs same role of the world model in conventional model-based RL approaches.
The skill-regularized task decomposition enables to be an optimal policy in the MDP Mr as explained in (8) Accordingly, it turns out that the augmentation procedure in (9) yields a plausible trajectory similar to expert demonstrations, given that the high-quality skill corresponding to the trajectory is incorporated into Pg. In consequence, the procedure in (9) can alleviate the negative effect of low-quality datasets. Figure X depicts the effect of our data augmentation by imaginary demonstrations, where (a) and (b) show the difference of low- low-quality and high-quality datasets in terms of the state-action pair distribution. Our imaginary demonstrations in (c) generated from (a) the source dataset share similarity with (b) the expert dataset, while (d) the dataset generated by some conventional augmentation using Gaussian noise does not. The table shows the average reward of datasets caleulated by reward relabeling, indicating that our bbill- augmentation by imaginary demonstrations produces higher quality data compared to the Gaussian noise-based augmentation.
vi

---

(a) Source dataset
(b) Expert dataset
(c) Imaginary demo.
(d) By Gaussian noise
Datasets
(a) Source Datasel
(h) Expert dataset dataset
(c) Imaginary demo.
(d) By Gaussian noise
Average reward
0.3947
0.2519
Figure 4: Examples of state-action pair distribution. (c) The imaginary demonstrations generated from (a) the source dataset look more similar (b) the expert dataset than (a) the source dataset, while (d) the augmented dataset by Gaussian noise does not. The table lists the average rewards calculated by reward relabeling on the datasets in (a)-(d), respectively, illustrating the quality gain of (c) compared to (d) in terms of average rewards.
Experiments
In this section, we evaluate the performance of our model under various configurations of multi-task offline datasets.
Experiment settings For evaluation, we use several robotic manipulation tasks and drone navigation tasks using the Meta-world environment (26) and the Airsim drone simulator [27]. The detailed
settings including hyperparameters and environment conditions can be found in Appendix.
Comparison methods We implement several multi-tase RL methods for comparison including:
TD3+BC [2) is state-of- the-art offline RL algorithm. which incorporates a behavior cloning regularization term into the update steps of TD3, inducing the learned policy towards the actions found in the dataset. For multi-task learning, TD3+BC is extended to include one-hot encoded task representation as part of the state. We use this baseline to compare our approach with conventional multi-tass task RL algorithms without considering subtask decomposition.
PCGrad is gradient surgery- based multi-task RL algorithm, which uses projection function remove the directional conflicts between gradients. This is implemented based on the hypothesis such that performance degradation can exacerbated by gradient conflicts when training uncorrelated tasks.
Soft modularization (SoftMod) (8] is modular deep neural network architecture tailored for multi-task RL. To mitigate the negative impact of learning different tasks on a single policy, it leverages the softly weighted routing path set of modules that are specifically trained on multiple tasks. It also employs a loss- bass-bblancing strategy to rapidly adapt to the different learning progress of the tasks.
Offline datasets We create and use three types of datasets according to specific behavior policies at different quality levels. Medium-Replay (MR) denotes the datasets sampled by learning process from the initial to partially-trained medium policies, Replay (RP) denotes the datasets sampled during a whole learning process, and Medium -xpeer (ME) denotes the datasets sampled by a learning process from the medium to expert policies. Note that each dataset in MR, RP, and ME for task contains episodic trajectories of 150, 100, and 50, respectively, unless otherwise stated.
5.1 Meta-world Tests
Here, evaluate our model using the MT10 benchmark (i.e., 10 different control tasks) in Meta- world (26). where each task is given a specific manipulation objective such as opening a door or closing window. The tasks share common primitive functions such grasp and moving, they can be seen as general multi tasks with shared subtasks, which are consistent with our task decomposition strategy. The implementation detail can be found in Appendix.
Performance on MT10 benchmark Table shows the performance the average success rate on MT10 under mixed configurations of different datasets (MR, RP, ME). We compare the performance
vii

---

Dutasets
Comparison PCGGaad
Our model
MR
RP
ME
TD3+BC
SoftMod
SRTD
SRID+ID
10
o
19.73 1: 2.71%
20.66 1 4.27%
13.43 1 2.67%
21.24 1 1.40%
23.87 1 2.22%
0
10
0
25.93 1. 5.91%
27.31 1. 3.15%
29.04 1 0.58%
38.97 1. 3.38%
41,91
5,88%
0
0
10
23.93 1 3.90%
33.06 1 3.69%
39.61 1 1.02%
46,60 1: 3.11%
49.29 1 3,35%
7
0
22.13 = 1.05%
23.23 # 1.94%
20.80 = 4.97%
28.67 = 1.51%
32.53 = 4.90%
5
3
2
25.13 + 1.49%
25.10 + 1.72%
28.73 = 0.56%
33.60 + 6.24%
32. 13 + 3.57%
5
0
5
16.53 + 4.71%
22.17 + 3.68%
28.13 + 4.59%
35. 13 + 3.36%
36.80 + 5.27%
4
3
3
27.60 + 3.25%
23.53 + 8.40%
25.87 + 1.26%
36.80 + 4.67%
43.53 + 3.32%
3
0
23.53 + 1.98%
25.60 +: 5.01%
31.77 + 3.52%
42. 13 + 19%
44.93
5,35%
0
7
24.93 1 2.44%
26.50 1 4.06%
30.33 1 1.31%
43.27
1 3.27%
43.73
3.88%
0
5
24.70 1 3.90%
27. 1 3.69%
32.06 1 1.02%
42.46 1 3.11%
44.42
3,35%
Table I: Performance on MTI0 the success rate with 95% confidence intervals (with different random seeds). The Datasets column specifies the mixed configurations of different datasets; e.g.. the row (MR 10, RP O, ME 0) corresponds to a specific configuration where each of 10 task in MT10 has MR, and the row of (MR 5, RP 3, ME 2) corresponds to another mixed configuration where each of 5 tasks, tasks, and tasks has MR, RP, and ME, respectively.
50
52
rate (%)
Suceess rate 0%)
48-
TE
SRID
16-
SRID
SKID
SKTD ID
(MR 10,
(MB OCR M. (0)
04 O.
R. JO 9) RPS, Configures
Configures
(a) Comparison of task embedding methods
(b) Comparison of data augmentation methods
Figure 5: Effects of (a) quality-aware skill regularization and (b) imaginary demonstrations
achieved by our model (SRTD, SRTD+ID) and other methods (TD3+BC, PCGrad, SoftMod). Note that Skill Regularized Task Decomposition (SRTD) is trained as in Section 3 and SRTD with Imaginary Demonstrations (SRTD+ID) is trained by our whole model introduced in Sections Bl 4 Our model (SRTD, SRTD+ID) yields the best performance consistently for all configurations, demonstrating its superiority with 3.67%~~7.66 competitive comparison method, SoftMod. SRTD consistently shows robust performance, while the comparison methods do not. TD3+BC and PCGrad show better performance for the configurations of low-quality datasets, c.g., the of (MR 10, RP O, ME 0), but SoftMod shows better performance for the configurations of high-quality datasets c.g., the row of (MR O, RP O, ME 10).
higher success rates, compared to the most
TD3+BC and PCGrad explore the orthogonality of tasks by accumulating task-specific knowledge separately without much interference when learning different tasks, and SoftMod rather exploits the commonality of the tasks by learning shared skills and dynamically extracting task-specific knowledge by the combination of its modules 77 (). Specifically, our TD3+BC implementation with one-hot task encoding tends to learn individual tasks separately, considering that the task encoding does not represent the semantic relation of different tasks explicitly. PCGrad intends for geometric separation of model updates for minimizing both the individual loss of each task and the distance between the weights different tasks. SoftMod utilizes the shared weights of a modular network for jointly optimizing the multiple loss functions for the tasks (28). Our SRTD tends to adjust both orthogonality and commonality using the quality-aware joint learning, so it can achieve robust performance for different mixed configurations. SRTD+ID improves the performance clarifying the benefit of imaginary demonstrations in offline datasets.
SRTD by 2.97% average,
Ablation study Figure 5) shows the effect of our skill regularization, where TE denotes the task embedding without skill regularization (i.e., only LTE in (3) is used), and SRTD-Q denotes SRTD without the quality weighted term R(s, a) in ④ SRTD shows better performance than the others consistently for all configurations. However, SRTD-Q shows worse performance than
viii

---

Dutasets
Comparison PCGrad
Our model
MR
RP
ME
TD3+BC
SoftMod
SRTD
SRID+1D
6
0
0
12.71 1. 2.27%
15.70 1 0.34%
16.76 1 3.58%
22.34 1 0.98%
24.60) 1 2.25%
0
6
0
13.06 1 2.93%
13.45 1 0.70%
21.19 1 1.98%
29.34 1 0.32%
30,77 1 2.16%
0
0
14.82 1 1.99%
16,93 1: 2.15%
26.35 1 2.35%
30.36 1: 2.61%
35,83 1 0.80%
2
2
2
14.66 = 2.55%
16.38 # 4.63%
24.07 = 2.28%
29.08 + 2.36%
28.37 = 1.09%
1
2
3
13.23 = 0.47%
14.12 # 3.09%
22.80 = 1.37%
27.78 + 2.21%
34.18 = 1.16%
3
2
12.18 + 2.14%
12.28 + 1.92%
18.67 + 2.89%
25.47 + 2.61%
27.51 + 1.79%
Table Performance on Airsim-based drone navigation in the normalized returns with 95% confi- dence intervals. Each return is normalized based on the maximum episodic return by a fully trained (online) RL agent that directly interacts with the environment. The Datasets column specifies the mixed configurations, as in T for 6 individual tasks in multi-task drone navigation.
TE, which specifies the benefit of suulity-awarr regularization. Figure ((b) shows the effect of our data augmentation method, where SRTD+N denotes SRTD with the Gaussian noise-base data augmentation commonly used in offline RL [23). We observe that SRTD+N rarely achieves performance gains, compared to SRTD. This clarifies the advantage of SRTD+ID that leverages high-quality skills to generate trajectories.
5.2 A Case Study for Airsim-based Drone Navigation
Here, we verify the applicability of our model in real-world problem scenarios by conducting a case study with autonomous quad-copter drones in the Airsim simulator [27]. We configure various realistic maps in PEDRA [29) and diverse wind patterns to build a multi-task drone flying environment. Regarding the RL formulation, the drone agent is set observe its lidar data, position, speed, and angle of rotation, while it is set not to observe any data about wind patterns. With continuous observations, the drone agent conducts control actions by manipulating the 3-dimensional acceleration, and receives rewards based on the goal distance. For evaluation, we measure the normalized episodic return based on the maximum episodic return obtained by a fully trained online RL agent. Table @oolleecoppree the performance by our model and the others. For all mixed configurations, our model shows the best performance, outperforming the competitive SoftMod by 5.01%%~11.37%
Related Work
Multi-task RL Multi-task RL has been investigated for sample-efficiently dealing with complex control problems in real-world settings (30 6 31). By training a deep neural network with multiple tasks jointly, multi-task RL algorithms drive agents to learn how to share, reuse, and combine the knowledge across correlated tasks. Yang et al. (8) presented an explicit modular architecture with soft routing network for training an integrated multi-task policy. This, called soft modularization. addresses the issue of unclear task relation in a single network such that which shared parameters are related to which tasks. Yu et al. (7] proposed gradient surgery methodology that directly removes the negative effects of mulli-task learning in a single policy, identifying and adjusting geometric conflicts of calculated gradients when learning different tasks.
Task and skill embeddings in multi-task RL Several approaches using task embeddings have been
multi-task RL 66 35). imitation learning (36] 37), and non-stationary RL 38 33). Pertsch et al. 44) demonstrated that with pretrained low-level policy that readily achieves given skills, high-level policy yielding appropriate skills can facilitate the learning efficiency, where skills are embedded in the latent space with expert data, Sodhani et al. (6] used additional metadata for learning multi-task policy, exploiting task descriptions in natural language to represent the semantics and relation of tasks in the latent space. While these prior works rely online interaction and they rarely consider heterogencous datasets and different behavior polices, which are common in multi-task offline RL, our model employs the quality-aware regularization to handle the mixed configurations of multi-task datasets. We also devise joint learning mechanism for skill and task representation in offline settings.
introduced in the context of meta RL [32 33, 34
Data augmentation in offline RL To alleviate the issue of limited datasets and unseen states, several works exploited data augmentation [23] 41), data sharing [15] TI 42 and model-based approach 43 41 in offline RL. For example, Sinha et al. (23 tested several data augmentation
ix

---

schemes, demonstrating the possible performance gain with offline RL algorithms. Yu et al. [T] presented the conservative Q-function that can judge which transitions are relevant for learning specific task, thus establishing a conditional data sharing strategy upon data scarcity situations. Our data augmentation by imaginary demonstrations in the same vein, but it focuses on exploiting common skills to generate trajectories that are likely to be generated by expert policies.
7 Conclusion
In this study, we proposed a novel multi-task offline RL model to tackle the problem of heterogeneous datasets of different behavior quality across tasks. In the model, skills and tasks are jointly learned with quality- regularization so that achievable subtasks are found and aligned with high-quality skills. Our model consistently yields robust performance upon various mixed configurations of different-quality datasets without requiring additional interaction with the environment. The direction of our future works is to investigate the hierarchy of skill representation with different temporal abstraction levels in multi-task offline RL. This will tackle the limitation of our model that considers only fixed exed-length sub-trajes uubetrajectories for task and skill embeddings.
8 Acknowledgement
We would like to thank anonymous reviewers for their valuable comments and suggestions. This work was supported by Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2022-0-00688, 2022-0-01045, 2022-0- 00043, 2019-0-00421) and by the National Research Foundation Korea (NRF) grant funded by MSIT (No. 2020M3CIC2A010800
References
[1] Aviral Kumar et al. "Conservative q-learning for offline reinforcement learning". In: Proceed- ings of the 33rd Advances Neural Information Processing Systems (NeurIPS). 2020.
[2] Scott Fujimoto and Shixiang Shane Gu. "A minimalist approach to offline reinforcement learning". In: Proceedings of the 34th Advances in Neural Information Processing Systems (NeurIPS). 2021.
[3] Scott Fujimoto, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration". In: Proceedings of 36th International Conference on Machine Learning (ICML). PMLR. 2019, pp. 2052-2062.
[4] Rich Caruana. "Multitask learning". In: Machine learning 28.1 (1997), pp. 41-75.
[5] Diana Borsa, Thore Graepel, and John Shawe-Taylor. "Learning shared representations in multi-task reinforcement learning". In: arXiv preprint: 1603.02041 (2016).
[6] Shagun Sodhani, Amy Zhang, and Joelle Pineau. "Multi-task reinforcement learning with costext-based representations" In: Proceedings of 38th International Conference on Machine Learning (ICML). PMLR. 2021, pp. 9767-9779.
[7] Tianhe Yu et al. "Gradient surgery for multi-task learning". In: Proceedings of the 33rd Advances in Neural Information Processing Systems (NeurIPS). 2020.
[8] Ruihan Yang et al. "Multi-task reinforcement learning with soft modularization". In: Proceed- ings of the 33rd Advances Neural Information Processing Systems (NeurIPS). 2020.
[9] Maximilian Igl et al. "Multitask soft option learning". In: Proceedings of 36th Conference on Uncertainty in Artificial Intelligence (UAI). PMLR. 2020, pp. 969-978.
[10] Jacob Andreas, Dan Klein, and Sergey Levine. "Modular multitask reinforcement learning with policy sketches". In: Proceedings of 34th International Conference on Machine Learning (ICML). PMLR. 2017, pp. 166-175.
[11] Tianhe Yu et al. "Conservative data sharing for multi-tast offline reinforcement learning". In: Proceedings of the 34th Advances in Neural Information Processing Systems (NeurIPS). 2021.
[12] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
X

---

[13] Finale Doshi-Velez and George Konidaris. "Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations" In: proceed- ings of 25th international joint conference artificial intelligence (ICJAI). Vol. 2016. NIH Public Access. 2016, p. 1432.
[14] Ron Dorfman, Idan Shenfeld, and Aviv Tamar. "Offline Meta Reinforcement Learning- Identifiability Challenges and Effective Data Collection Strategies". In: Proceedings of the
34th Advances in Neural Information Processing Systems (NeurIPS). 2021.
[15] Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. "Transfer of samples in batch reinforcement learning". In: Proceedings of the 25th international conference on Machine learning (ICML). 2008, pp. 544-5541
[16] Tya Tolstikhin et al. "Wasserstein cittoeenccders". In: arXiv preprint arXiv:1711.01558 (2017).
[17] Karl Pertsch, Youngwoon Lee, and Joseph J Lim. "Accelerating reinforcement learning with learned skill priors". In: arXiv preprint: 2010. 19944 (2020).
[18] Taewook Nam et al. "Skill-based Meta-Reinforcemeent Learning". In: Proceedings of 10th
International Conference on Learning Representations (ICLR). 2022.
[19] Sham M. Kakade and John Langford. "Approximately Optimal Approximate Reinforcement Learning".. In: Proceedings of the 19th International Conference on Machine Learning (ICML). 2002, pp. 267-274.
[20] John Schulman et al. "Trust Region Policy Optimization", In: Proceedings of the 32nd Inter- national Conference on Machine Learning (ICML). Vol. 37. 2015, pp. 1889-1897.
[21] Xue Bin Peng et al. "Advantage-weighhed regression: Simple and scalable pollcy reinforce- ment learning". In: arXiv preprint: 1910.00177 (2019).
[22] Aviral Kumar, Xue Bin Peng, and Sergey Levine. "Reward-conditioned policies". In: arXiv preprint: 1912.1346 (2019).
[23] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. "S4RL: Surprisingly simple self- supervision for offline reinforcement learning in robotics". In: Proceedings of 5th Conference
on Robot Learning (CoRL). PMLR. 2022, pp. 907-917.
[24] Chaochao Lu et al. "Sample--fficient reinforcement learning via counterfactual-basse data augmentation". In: arXiv preprint: 2012.09092 (2020).
[25] Jianhao Wang et al. "Offline Reinforcement Learning with Reverse Model-based Imagination".
In: Proceedings of the 34th Advances in Neural Information Processing Systems (NeurIPS). 2021.
[26] Tianhe Yu et al. "Meta-World, A Benchmark and Evaluation for Multi-Task and Meta Re- inforcement Learning". In: Proceedings of the 3rd Conference on Robot Learning (CoRL). PMLR. 2019.
[27] Shital Shah al. "AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles". In: arXiv preprint: 1705.05065 (2017).
[28] Michael Crawshaw. "Multi-task learning with deep neural networks: A survey". In: arXiv
preprint:2009. 09900990999664 (2020).
[29] Aqeel Anwar and Arijit Raychowdhury. " "Autonomous Navigation via Deep Reinforcement Learning for Resource Constraint Edge Nodes using Transfer Learning". In: arXiv preprint: 1910.05547 (2019).
[30] Lerrel Pinto and Abhinav Gupta. "Learning to push by grasping: Using multiple tasks for effective learning". In: Proceedings of the 34th International Conference on Robotics and Automation (ICRA). IEEF. 2017, pp. 2161-2168.
[31] Yee Teh et al. "Distral: Robust multitask reinforcement learning". In: Proceedings of the 30th Advances in Neural Information Processing Systems (NeurIPS). 2017.
[32] Luisa Zintgraf et al. "Varibad: A very good method for bayes-adaptive deep rl via meta- learning". In: arXiv preprint: 1910.08348 (2019).
[33] Jan Humplik et al. "Meta reinforcement learning as task inference". In: arXiv preprint: 1905. 064.00644 (2019).
[34] Kate Rakelly et al. "Efficient off- tff-policy meta-reinforcemen learning via probabilistic context variables". In: Proceedings of the 36th International Conference on Machine Learning (ICML). PMLR. 2019, pp. 5331-5340.
xi

---

[35] Andrew Silva et al. "LanCon-Lear: Learning With Language to Enable Generalization in Multi-Task Manipulation", In: IEEE Robotics and Automation Letters 7.2 (2022), pp. 1635- 1642.
[36] Yan Duan al. "One-shot imitation learning". In: Proceedings of the 30th Advances in Neural Information Processing Systems (NewrIPS). 2017.
[37]
Ziyu Wang et al. "Robust imitation of diverse behaviors". In: Proceedings of the 30th Advances in Neural Information Processing Systems (NeurIPS). 2017.
[38] Annie Xie, James Harrison, and Chelsea Finn. "Deep Reinforcement Learning amidst Con- tinual Structured Non-Stationarityy In: Proceedings of the 38th International Conference Machine Learning (ICML). PMLR. 2021, pp. 11393-11403.
[39] Honguk Woo, Gwangpyo Yoo, and Minjong Yoo. "Structure Learning-Baan BBasdd Task Decompo- sition For Reinforcement Learning In Non-Stationary Environments" In: Proceedings of the 40th AAAI Conference on Artificial Intelligence. 2022.
[40] Karl Pertsch et al. "Demonstration-Cin Guideed Reinforcement Learning with Learned Skills". In: Proceedings of the 5th Conference on Robot Learning (CoRL.). Vol. 164. PMLR. 2022.
[41] Jianhao Wang et al. "Offline Reinforcement Learning with Reverse Model-based Imagination". In: Proceedings of the 34th Advances in Neural Information Processing Systems (NeurIPS). 2021.
[42] Tianhe Yu et al. "Data Sharing without Rewards in Multi-Task Offline Reinforcement Learn- ing". In: Deep RL Workshop of 34th Advances Neural Information Processing Systems (NeurIPS). 2021.
[43] Philip J Ball et al. "Augmented world models facilitate zero-shot dynamics generalization from a single offline environment". In: Proceedings of 38th International Conference on Machine Learning (ICML). PMLR. 2021, pp. 619-629.
xii

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
xiii


Reviews:
[{"review_id": "vg7osmNd6o7", "reviewer": "Reviewer_qkkL", "summary": "This paper proposes an offline multi-task RL algorithm which introduces a novel skill-level and task-level encoder, which is used to encode subtrajectories from offline data as well as tasks into a shared latent space. This generative model can be used to generate imaginary data for improved training. The method is tested on simulated Metaworld environments (MT-10) as well as a drone navigation task in the Airsim simulator and shows improved performance compared to prior works TD3+BC, PCGrad, and SoftMod.", "questions": "- I don’t quite understand the difference between TE and SRTD-Q in the ablation study section of the paper. The paper says TE is the task embedding without skill regularization (only $L_{TE}$ is used), but isn’t the only additional term the quality-aware term from Equation (4) (which is what SRTD-Q removes?) I would appreciate a clarification.\n- I can’t find any mention in the main paper that the learned subtask and subskill embeddings are later used in training an offline RL agent using TD3+BC, and had to look in the appendix for this – I strongly recommend this be included in the main text for clarity.\n", "limitations": "The limitations of the method don't appear to be explicitly discussed, although a direction of future work is discussed. I would encourage the authors to address the current limitations and potential negative societal impact of the work. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The skill and task encoding strategy is very novel to me and has appealing properties such as encoding skills and tasks into the same latent space, where the distance between skills and tasks is meaningful.\n- The experimental evaluation compares to sensible baselines (TD3+BC, PCGrad, SoftMod) and achieves strong results. It’s nice that the methods are compared across different configurations for the dataset quality as well, and demonstrates that the SRTD method is able to handle cases with lower quality data as well as higher-quality datasets. \n- The ability to generate additional imaginary high-quality data is a nice property of the formulation, and Figure 4 in the paper is a nice qualitative result in this direction. However, the additional imaginary data doesn’t improve performance as much as I may have expected when looking at Figure 5b, but does seem to help a bit. Based on the ablations, It seems that most of the performance improvements compared to the TD3+BC baseline are coming from using a high level skill embedding at all, which is also an interesting result (but please let me know if I am misinterpreting the results).\n\nWeaknesses:\n- One weakness of the current formulation is that the subtrajectories and skill embeddings represent sequences of states and actions with a fixed time length. How is the subtask length chosen, and how are the subtrajectories sampled during training of the WVAEs? \n- While the quality-aware skill regularization seems like a reasonable heuristic, it seems like estimating the quality of a particular sub-trajectory based on the episodic returns is likely error-prone if a trajectory is not of uniform quality. This could work well in settings where trajectories are collected such that they contain either solely low-quality or high-quality behaviors such as the ones examined in this work, but may not be able to pick good behaviors from mixed quality data. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "o6JiiPUuJ5U", "reviewer": "Reviewer_5BuZ", "summary": "This paper proposes a skill-based multi-task RL technique to learn the shareable knowledge across multiple offline datasets that are generated by behavior policies of different quality. Specifically, they develop a task decomposition method where the Wasserstein auto-encoder (WAE) is used to represent both skills and tasks in the same latent space. Then they use the quality-weighted loss as a regularization term to induce tasks to be decomposed into subtasks that are more consistent with high-quality skills than others. To further improve the performance, they augment datasets with imaginary trajectories relevant to high-quality skills for each task. They evaluate their method on two types of Offline RL tasks, manipulation tasks and drone navigation tasks, and the results show that their method outperforms other state-of-the-art algorithms.", "questions": "* What does the red region mean in Figure 1?\n\n* What’s the advantage of generating imaginary data using skill embedding? It is hard to tell that the imaginary trajectories are more like high-quality datasets in Figure 4. It would be better if the authors can compare their method with a baseline that uses normal counterfactual data augmentation methods.\n\n* I notice that the three baselines used in the experiments are not designed for tackling the Multi-task Offline RL problems and are published in 2020. Since I am not familiar with SOTA algorithms in this field, I can’t recommend suitable baselines. But if there exist similar methods, please add more baselines to make the comparison fair.\n", "limitations": "Limitations are not fully discussed in the paper. One limitation is that the application is limited if the skills and tasks are complicated and can’t be represented only by state-action pairs.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* This paper has a clear motivation for solving the multi-task problem in the Offline RL setting.\n* The proposed method is complicated but novel.\n* The paper is well-written and easy to follow.\n\n### Weaknesses\n* Some figures are not intuitive enough and do not provide enough information. For example, Figure 1 and the right part of Figure 2.\n* There might be some missing baselines in the experiment part.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "EaKZcoHFVba", "reviewer": "Reviewer_ttxU", "summary": "The authors present a method for multitask offline RL called skill regularized task decomposition. The authors learn autoencoders that embed skills (sequence of actions) and tasks (transition and reward models) into the same latent space, with greater weight given to higher-reward transitions. The authors also use these autoencoders to generate artificial demonstrations to augment the training dataset. In a 10-task Meta-world offline RL setup, the authors show that their method improves upon prior algorithms (TD3+BC, PCGrad, and SoftMod) for various different offline data mixture compositions.", "questions": "It’s not clear to me exactly how the TD3+BC baseline was trained. TD3+BC is a single-task algorithm, but appendix line 45 seems to say that it uses a “one-hot task encoding.” Are you considering a version of TD3+BC that has no knowledge of different tasks? Or are you one-hot encoding the tasks? It would be helpful if the paper clarified this.", "limitations": "I don’t see any negative societal implications that the authors failed to address.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe authors compare against strong baselines: TD3+BC (a state-of-the-art single-task offline RL method) and two multitask RL methods (PCGrad and Soft modularization). The authors also test a variety of different offline dataset compositions, changing the proportion of data that is medium-replay (MR), replay (RP), and medium-expert (ME).\n\nThe authors split up their method into SRTD and SRTD+ID. This is helpful to help understand how much of the benefit comes from Skills Regularized Task Decomposition and how much comes from Imaginary Demonstrations.\n\nOverall, I think multitask offline RL is relevant to the community. The authors give clear exposition for a new method and show that it outperforms strong baselines from prior work.\n\n## Weaknesses\nIt is only in the appendix that the authors actually describe how they train an RL agent on top of the SRTD(+ID) embeddings. It would be helpful if the authors could fit this into the main text.\n\nThe authors don’t motivate MT10. Why are the tasks in MT10 interesting? It would be helpful if the authors included in the main text more qualitative information about the MT10 tasks and why they are interesting.\n\nAt line 243, the authors write: “It is because TD3+BC and PCGrad explore the orthogonality of tasks and SoftMod exploits the commonality of tasks.” I don’t fully understand this explanation and think it would be helpful if the authors elaborated on it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept