Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a data-sharing strategy called CDS to tackle the distributional shift of the offline RL algorithm in the multitask setting. The authors reveal the problem of naive sharing strategies experimentally and theoretically. CDS achieves an overall advantage against naive sharing strategies in various multitask offline environments.
main_review: $\textbf{Originality}$: This paper targets the multitask offline RL problem which has not been well studied, as far as I know, this is the first work to introduce the idea of data sharing into offline RL. Although the method proposed in this paper is still based on relabelling, the authors give a clear theoretical support for CDS under offline RL setting.

$\textbf{Quality}$: The quality of this paper is generally very good, it has solid theoretical and experimental support for its claims. The authors make a comprehensive evaluation on the proposed method under different settings, comparing with strong baselines. 

$\textbf{Clarity}$: This paper is well-written but I have several comments. In Equation (4), why the argmax output is a tuple, I think $\pi_{\beta}^{\text{eff}}$ should not appear on the left side. In Table 2, what are the meaning of each number in the results column. In Figure 4, it's kind of weird to leave out the task image for AntMaze.

$\textbf{Significance}$: The technique proposed is simple and shown to be effective, and the problem addressed is meaningful. However, I have a concern why CDS is solely based on CQL, and the authors make the claim: "Although these results are only for CQL, we expect that any offline RL method would, insofar as it combats distributional shift by staying close to the data, would exhibit a similar problem". This is not so straightforward to me, and it'll be better if the authors could demonstrate the similar results (CDS, no sharing and sharing all) using another base offline RL algorithm.


limitations_and_societal_impact: The authors have addressed all of them.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Summary

In this paper, the authors study the problem of offline RL from a different angle: how do we pick the appropriate data samples from certain offline tasks so that the distribution shift between source tasks and target tasks is minimized.
Conservative data sharing (CDS) is proposed, which proposes a criteria of selecting the data samples from a theoretical inspiration.
main_review: 
Main Review

Pros:
1. Very good paper writing
The paper provides a comprehensive and exhaustive related work and background section.
It would be easy for readers to catch up the flow and understand the algorithm

2. Good mathematical support
The algorithm comes from a well supported mathematical ground. 
The intuition is very natural, and the algorithm itself boils down to a simple and neat practical algorithm. It would provide valuable insight for future research

3. good experiment results
The experiments cover a sufficient spectrum, which includes experiments from low dimensional toy experiments to high dimensional simulated robotics environments.
And the baselines are adequate, showing the proposed algorithms improving upon previous work.

4. The algorithm is novel, and the topic of offline reinforcement learning is quite interesting and valuable to many real life applications.

5. code is released.

In general I am no expert in this area, but I think the quality of the paper is good. 
However, for the experiments, the authors didn’t experiment with real life robotics tasks.
Considering the importance of offline RL for real life applications, these experiments will further improve the experiment section.

limitations_and_societal_impact: No potential negative societal impact found.
The limitations are addressed in the paper
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses the problem of data sharing in multi-task offline RL, and show that naive data sharing scheme can be harmful for some tasks where data sharing could lead to severe distribution shift. To effectively utilize data-sharing in such tasks, this paper proposes a conservative data sharing (CDS), that re-labels data whose conservative Q-value is higher than the expected conservative Q-value in the target dataset.
main_review: - Problem setup is a bit limited but important, the proposed idea is simple and interesting, and considering distribution shift for utilizing multi-task datasets seems very reasonable. 

- I have a strong concern on the significance of the experimental results. First of all, it's not clear why the standard deviations are only reported in the appendix and bolds for similar performance are only made in the tables from appendix (also missing the results on visual domain). It seems that there are many tasks where there is a overlap if we consider standard deviations, and this is important for evaluating the significance of the experimental results. I think there should be more runs for making it possible to make a conclusion on the effectiveness of the proposed method. Also, reporting learning / evaluation curves would be more helpful for understanding the performance.

-  Curious about how conservative Q-function generalizes to unseen state-action pairs in different datasets. Are the estimated Q-values at such unseen regime reliable estimate for filtering out 'bad' transitions? As CQL only regularizes unseen actions, it could not handle distribution shift from unseen states. Is there supporting analysis or experiments to show that the proposed method works as intended, filtering out 'bad' or 'not relevant' state-action pairs whose 'ground-truth' Q-values are low? Also, how does the proposed method compare to the baseline of just randomly sampling transitions from datasets of other tasks? 

- In that sense, more qualitative analysis like how much of the datasets are assigned to each task should be more helpful. For example, showing that transitions from the similar tasks are assigned to the target task but transitions from tasks which are not relevant are not assigned would be interesting.

- Problem setup of sharing data across multiple tasks seems very important and I like the observations in Table 1, but it's a bit limited in that this paper only deals with the setup of shared state/action space, and reward function is known (as authors already stated in the paper). In that sense, important but missing baseline (related work) should be utilizing other datasets for learning representations, as in [Yang'21].

[Yang'21] Representation Matters: Offline Pretraining for Sequential Decision Making, ICML 2021
limitations_and_societal_impact: Addressed in the Section 7.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper considers multi-task RL under the offline setting where the underlying system dynamics is the same but different task has a different reward function. Through experimental analysis, the authors argue that naive data sharing, while intuitive, can potentially degrade the performance due to distributional shift. To address the issue, this work proposes a simple, conservative data sharing scheme that only shares data among tasks that could potentially have a higher conservative Q-value estimate. Experimental results on several tasks validate that the scheme can often be effective.
main_review: The paper is well organized with clear presentation and meaningful motivation/analysis on data sharing. This makes the paper easy to follow and overall sound. While I enjoyed reading it, I think the significance of its contribution might be limited. The intuition is straightforward and the results are exactly what one would expect. In a sense, naively augmenting data that are too far from the original task could increase the effect of distributional shift and hurt the performance. The current work hence uses the "conservative approaches", common in offline RL, to regularize the data to be shared. These being said, I think the paper does  articulate the arguments well and the resulting algorithm seems to be an effective approach based on the experiments. 

Additional comments:
1. the presented tables seem to base on experiments from a single run. Given how noisy the results could be, few more repeated experiments are helpful in understanding the gaps among approaches.
2. the authors mention the limitation of CDS on requiring functional forms of rewards. In lines 144-145, it is stated that this is a common assumption in goal-conditioned RL (or otherwise, through the use of learned classifiers). Since the authors also mention [7] as the offline RL work for this setting, it seems reasonable to compare for goal-conditioned problem if appropriate.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper presents a data-sharing strategy called "conservative data sharing" CDS to tackle the distributional shift of the offline RL algorithm in the multitask setting. All the reviewers evaluated positively about the work, and this would make a solid constribution to NeurIPS. I hope that reviewer comments help the authors for future work.
Decision: Accept (Poster)