Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new offline multi-task RL algorithm named conservative unsupervised data sharing (CUDS). Prior works on multi-task RL require the functional form of the reward functions or are limited to goal-conditioned setting. On the other hands, CUDS shares task-agnostic transitions via simply relabeling their rewards as a constant.
The authors first prove that $\hat{Q}_\text{UDS}\le \hat{Q}_\text{sharing All}$ for all state-action pairs. Furthermore, they argue that the CUDS can achieve better performance through weighting transitions.
Finally, CUDS outperforms or performs competitively with the previous works. Surprisingly, CUDS achieves competitive results compared to CDS, which uses true reward functions for relabeling

main_review: Strength:
Two remarks in Section 4 provide an interesting intuition about how to use unlabeled transitions depending on whether each transition can reach a labeled state-action pair.
This work has empirically shown that a simple constant reward (zero reward) is enough to achieve good performance in practice.

Weakness:
Considering extending CDS into multi-task RL without oracle reward function, the simplest direction is relabeling unlabeled transitions with minimum reward. But it is not clear whether the direction is valid. Therefore, the readers expect justification of using simple relabeling. However, CUDS is only supported empirically without any theoretical analysis. 

Proposition 4.1 is proposed to prove the Q-value function is underestimated by UDS. However, it is not clear why such lower estimation is desirable in offline multi-task RL. Specifically, it seems too conservative way to estimate unlabeled rewards. For example, if we consider the $(s,a)$ is in both labeled and unlabeled data, then the reward of the state-action pair may be underestimated.
In addition, it is trivial since $\hat{r}(s,a)\le r(s,a)$ for all $(s,a)$ pairs.

summary_of_the_review: The remarks and empirical results are interesting. However, CUDS is a simple variant of CDS – the only difference is how to set the rewards. Therefore, I think CUDS needs additional meaningful theoretical analysis in addition to empirical results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a method for multi-task offline reinforcement learning setting where it can assign shared and annotate the reward, where its major strategy is to label only useful transitions. Specifically, the proposed method followed the conservative learning setting with the hope of easing the distribution shift issue. The authors also study the behavior of their proposed methods with s set of arguments and experimental results. 
main_review: Pros:
- The problem addressed in the paper is indeed critical in offline RL. It has a reasonable motivation. 

Issues:
- The biggest concern is the novelty. The proposed CUDS and UDS methods share similarities with one existing study (Yu et al. 2021a). Indeed the proposed method is more sophisticated than this previous paper; however I would expect the authors to emphasize that there exists a substantive difference between the two different works.

- In this work’s multi-task offline RL setting, the authors assume the dynamics to be the same across all tasks and use a single task-conditioned policy. It is okay to assume the tasks only differ in reward distributions but I would expect a clearer justification.

- How does the proposed method differ from learning environment dynamics with more data (in a maybe model-based offline RL setting)? More insights on this point would be appreciated. 

summary_of_the_review: The major issue of this paper is it seems to be an incremental study from existing works. I would expect the authors to address this issue and add more insights.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work targets useful data sharing between multi-tasks in an offline reinforcement learning setting. Motivated by the same transition kernel $P$, for each task, data sharing from other tasks is considered to be beneficial for finding an optimal policy. Different from previous works, the author claims that under some assumptions, data sharing using a constant reward/no reward can be better than using a reward predictor and competitive with using ground-truth rewards relabelled. Towards this, the author proposes two methods called CUDS and UDS and conducts experiments to test its performance.
main_review: Advantages:
1. This work proposed two easily implemented approaches called CUDS and UDS to achieve data sharing in offline reinforcement learning settings, without the requirement of a reward function approximation.

Some of the questions under concern:
1. One of the most important claims in this work is that under some assumptions, the proposed data sharing methods (CUDS, UDS) will be beneficial even suppress the oracle. However, there is not a clear statement or discussion about the intuition when and why it can achieve this? Although there are some experiments such as meta-world, etc., it is still confusing whether these data-sharing methods can be utilized in other application scenarios.
2. There is an assumption in the targeted MDP is that the reward is binary. Is it necessary for CUDS/UDS to work?
3. In section 4.2, the author tried to provide the intuition of how data sharing influences Q-estimates. However, remark 4.2 seems to be unclear and can't lead to the later conclusion that " CUDS does not have to push down Q-values at every state-action pair". Can you give more explanation about it?
4. The writing of the paper needs to be polished a lot. I will provide extra comments later.

Some minor comments about writing:
1. A lot of descriptions and equations are too similar to the most related work [1]? It may need more revision. Such as the equ 2 and 3, even Algorithm 1 looks all similar.
2. What's the definition of '[N]' and it has been used as a distribution in Section 3?
3. Typo in Section 3 as 'D>'?
4. The definition in Equ (1) is not clear? Such as what is (s, a), what is $\pi$, and what the $\mathcal{D}_i^{eff}$ may need the input of $\pi$, since different policies may lead to different sharing datasets?
5. More omits of definitions after equation (3), such as what's the exact formulation of the notation $J$ and $D(\pi,\pi_\beta)$
6. The text "CQL" is also introduced without reference, although it may refer to conservative Q-learning.
7. Undefined $\hat{Q}_{UDS}^\pi(s,a)$ in Proposition 4.1
8. $w$ is a binary matrix as defined above of equation (7), not a vector. Moreover, what is the difference between $w(s,a)$ and $w(s,a,I)$?
summary_of_the_review: I recommend that this paper be below the acceptance threshold. 

1. This paper release the strong assumption with a reward relabelling function in [1], which archives data sharing between multi-tasks without reward. It seems like an extension with a minor revision of CDS in [1]. 
2. The intuition of the great performance is also straightforward since more data yields a better estimation of the transition kernel $P$ in MDP, even without rewards. 
3. Although in the examined experiments the proposed methods show competitive performances compared to CDS and other baselines, it is not convincing enough that without an estimation of the reward, CUDS/UDS can achieve great results in more general scenarios.


[1] Yu, Tianhe, et al. "Conservative data sharing for multi-task offline reinforcement learning." arXiv preprint arXiv:2109.08128 (2021).
review_title: null

Meta-review: Although sharing data between tasks benefits multitask RL, this requires that rewards be relabeled across tasks. This paper shows that, for binary rewards, directly reusing data from other tasks with constant reward relabels is effective, and the paper develops a method around this idea that is highly effective.  The reviewers found that the idea and execution were impressive, that the paper was well written, and that the empirical analysis was convincing. 

In response to concerns in the preliminary reviews about certain shortcomings in the empirical analysis and some lack of theoretical analysis, the authors provided substantial revisions to the paper. Due to some lack of reviewer response to the discussion, this meta-reviewer examined whether those revisions were sufficient to address the reviewers' concerns. The authors did a good job in providing the requested improvements and the analysis is stronger, but remaining similarities to existing methods (CDS) means that this paper still remains borderline. These same concerns were also shared by reviewers that continued to engage in discussion with the authors. To remedy this, the authors are encouraged to better and more substantially address differences with prior work in the writing and motivation throughout the entire paper. In addition, although space is a concern, it would be beneficial to integrate the high-level takeaways from the new analyses in the appendices into the main paper.
Decision: Reject