Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an offline multi-task RL algorithm which introduces a novel skill-level and task-level encoder, which is used to encode subtrajectories from offline data as well as tasks into a shared latent space. This generative model can be used to generate imaginary data for improved training. The method is tested on simulated Metaworld environments (MT-10) as well as a drone navigation task in the Airsim simulator and shows improved performance compared to prior works TD3+BC, PCGrad, and SoftMod.
strengths_and_weaknesses: Strengths:
- The skill and task encoding strategy is very novel to me and has appealing properties such as encoding skills and tasks into the same latent space, where the distance between skills and tasks is meaningful.
- The experimental evaluation compares to sensible baselines (TD3+BC, PCGrad, SoftMod) and achieves strong results. It’s nice that the methods are compared across different configurations for the dataset quality as well, and demonstrates that the SRTD method is able to handle cases with lower quality data as well as higher-quality datasets. 
- The ability to generate additional imaginary high-quality data is a nice property of the formulation, and Figure 4 in the paper is a nice qualitative result in this direction. However, the additional imaginary data doesn’t improve performance as much as I may have expected when looking at Figure 5b, but does seem to help a bit. Based on the ablations, It seems that most of the performance improvements compared to the TD3+BC baseline are coming from using a high level skill embedding at all, which is also an interesting result (but please let me know if I am misinterpreting the results).

Weaknesses:
- One weakness of the current formulation is that the subtrajectories and skill embeddings represent sequences of states and actions with a fixed time length. How is the subtask length chosen, and how are the subtrajectories sampled during training of the WVAEs? 
- While the quality-aware skill regularization seems like a reasonable heuristic, it seems like estimating the quality of a particular sub-trajectory based on the episodic returns is likely error-prone if a trajectory is not of uniform quality. This could work well in settings where trajectories are collected such that they contain either solely low-quality or high-quality behaviors such as the ones examined in this work, but may not be able to pick good behaviors from mixed quality data. 

questions: - I don’t quite understand the difference between TE and SRTD-Q in the ablation study section of the paper. The paper says TE is the task embedding without skill regularization (only $L_{TE}$ is used), but isn’t the only additional term the quality-aware term from Equation (4) (which is what SRTD-Q removes?) I would appreciate a clarification.
- I can’t find any mention in the main paper that the learned subtask and subskill embeddings are later used in training an offline RL agent using TD3+BC, and had to look in the appendix for this – I strongly recommend this be included in the main text for clarity.

limitations: The limitations of the method don't appear to be explicitly discussed, although a direction of future work is discussed. I would encourage the authors to address the current limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a skill-based multi-task RL technique to learn the shareable knowledge across multiple offline datasets that are generated by behavior policies of different quality. Specifically, they develop a task decomposition method where the Wasserstein auto-encoder (WAE) is used to represent both skills and tasks in the same latent space. Then they use the quality-weighted loss as a regularization term to induce tasks to be decomposed into subtasks that are more consistent with high-quality skills than others. To further improve the performance, they augment datasets with imaginary trajectories relevant to high-quality skills for each task. They evaluate their method on two types of Offline RL tasks, manipulation tasks and drone navigation tasks, and the results show that their method outperforms other state-of-the-art algorithms.
strengths_and_weaknesses: ### Strengths

* This paper has a clear motivation for solving the multi-task problem in the Offline RL setting.
* The proposed method is complicated but novel.
* The paper is well-written and easy to follow.

### Weaknesses
* Some figures are not intuitive enough and do not provide enough information. For example, Figure 1 and the right part of Figure 2.
* There might be some missing baselines in the experiment part.

questions: * What does the red region mean in Figure 1?

* What’s the advantage of generating imaginary data using skill embedding? It is hard to tell that the imaginary trajectories are more like high-quality datasets in Figure 4. It would be better if the authors can compare their method with a baseline that uses normal counterfactual data augmentation methods.

* I notice that the three baselines used in the experiments are not designed for tackling the Multi-task Offline RL problems and are published in 2020. Since I am not familiar with SOTA algorithms in this field, I can’t recommend suitable baselines. But if there exist similar methods, please add more baselines to make the comparison fair.

limitations: Limitations are not fully discussed in the paper. One limitation is that the application is limited if the skills and tasks are complicated and can’t be represented only by state-action pairs.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a method for multitask offline RL called skill regularized task decomposition. The authors learn autoencoders that embed skills (sequence of actions) and tasks (transition and reward models) into the same latent space, with greater weight given to higher-reward transitions. The authors also use these autoencoders to generate artificial demonstrations to augment the training dataset. In a 10-task Meta-world offline RL setup, the authors show that their method improves upon prior algorithms (TD3+BC, PCGrad, and SoftMod) for various different offline data mixture compositions.
strengths_and_weaknesses: ## Strengths
The authors compare against strong baselines: TD3+BC (a state-of-the-art single-task offline RL method) and two multitask RL methods (PCGrad and Soft modularization). The authors also test a variety of different offline dataset compositions, changing the proportion of data that is medium-replay (MR), replay (RP), and medium-expert (ME).

The authors split up their method into SRTD and SRTD+ID. This is helpful to help understand how much of the benefit comes from Skills Regularized Task Decomposition and how much comes from Imaginary Demonstrations.

Overall, I think multitask offline RL is relevant to the community. The authors give clear exposition for a new method and show that it outperforms strong baselines from prior work.

## Weaknesses
It is only in the appendix that the authors actually describe how they train an RL agent on top of the SRTD(+ID) embeddings. It would be helpful if the authors could fit this into the main text.

The authors don’t motivate MT10. Why are the tasks in MT10 interesting? It would be helpful if the authors included in the main text more qualitative information about the MT10 tasks and why they are interesting.

At line 243, the authors write: “It is because TD3+BC and PCGrad explore the orthogonality of tasks and SoftMod exploits the commonality of tasks.” I don’t fully understand this explanation and think it would be helpful if the authors elaborated on it.
questions: It’s not clear to me exactly how the TD3+BC baseline was trained. TD3+BC is a single-task algorithm, but appendix line 45 seems to say that it uses a “one-hot task encoding.” Are you considering a version of TD3+BC that has no knowledge of different tasks? Or are you one-hot encoding the tasks? It would be helpful if the paper clarified this.
limitations: I don’t see any negative societal implications that the authors failed to address.
review_title: null

Meta-review: metareview: The reviewers appreciated the authors' response and clarifications. Given the feedback from the reviewers and the discussion, I would like to recommend this paper for acceptance and congratulate them on a strong submission. I encourage the authors to address the reviewers' comments for the final version of the paper.
confidence: Certain
Decision: Accept