Data Sharing without Rewards in Multi-Task Offline Reinforcement Learning
Authors: Tianhe Yu
Aviral Kumar
Yevgen Chebotar
Chelsea Finn
Sergey Levine
Karol Hausman
Venue: ICLR 2022 Submitted (2022)
Keywords: offline reinforcement learning
multi-task reinforcement learning

Abstract: Offline reinforcement learning (RL) bears the promise to learn effective control policies from static datasets but is thus far unable to learn from large databases of heterogeneous experience. The multi-task version of offline RL enables the possibility of learning a single policy that can tackle multiple tasks and allows the algorithm to share offline data across tasks. Recent works indicate that sharing data between tasks can be highly beneficial in multi-task learning. However, these benefits come at a cost -- for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks. This is particularly expensive and unscalable, since the manual effort in annotating reward grows quadratically with the number of tasks. Can we retain the benefits of data sharing without requiring reward relabeling for every task pair? In this paper, we show that, perhaps surprisingly, under a binary-reward assumption, simply utilizing data from other tasks with constant reward labels can not only provide a substantial improvement over only using the single-task data and previously proposed success classifiers, but it can also reach comparable performance to baselines that take advantage of the oracle multi-task reward information. We also show that this performance can be further improved by selectively deciding which transitions to share, again without introducing any additional models or classifiers. We discuss how these approaches relate to each other and baseline strategies under various assumptions on the dataset. Our empirical results show that it leads to improved performance across a range of different multi-task offline RL scenarios, including robotic manipulation from visual inputs and ant-maze navigation.

Under review as a conference paper at ICLR 2022
DATA SHARING WITHOUT REWARDS IN MULTI-TASK OFFLINE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Offline reinforcement learning (RL) bears the promise to learn effective control policies from static datasets but is thus far unable to learn from large databases of heterogeneous experience. The multi-task version of offline RL enables the possibility of learning single policy that can tackle multiple tasks and allows the algorithm to share offline data across tasks, Recent works indicate that sharing data between tasks can be highly beneficial in multi-task learning. However, these benefits come at cost - for data to be shared between tasks, each transition must be annotated with reward labels corresponding to other tasks. This is particularly expensive and unscalable, since the manual effort in annotating reward grows quadratically with the number of tasks. Can we retain the benefits of data sharing without requiring reward relabeling for every task pair? In this paper, we show that, perhaps surprisingly, under binary-reward assumption, simply utilizing data from other tasks with constant reward labels can not only provide substantial improvement over only using the single-task data and previously proposed success classifiers, but it can also reach comparable performance to baselines that take advantage of the oracle multi-task reward information. We also show that this performance can be further improved by selectively deciding which transitions to share, again without introducing any additional models or classifiers. We discuss how these approaches relate to each other and baseline strategies under various assumptions on the dataset. Our empirical results show that it leads to improved performance across range of different multi-task offline RL scenarios, including robotic manipulation from visual inputs and ant-maze navigation.
INTRODUCTION
Offline reinforcement learning (RL) provides the promise of a fully data-driven framework for learning performant policies. To avoid costly active data collection and exploration, offline RL methods utilize previously collected dataset to extract the best possible behavior, making it feasible to use RL to solve real-world problems where active exploration expensive, dangerous, or otherwise infeasible (Zhan et al., 2021; de Lima & Krohling, 2021; Wang et al., 2018; Kalashnikov et al., 2018). However, this concept is only viable when a significant amount of data for the target task is available in advance. A more realistic scenario might allow for much smaller amount of task-specific data, combined with a large amount of aas-stin data, that is not labeled with task rewards and some of which may not be relevant. For example, if our goal is to train robot perform new manipulation task (e.g., cutting an onion), we might have some data of the robot (suboptimally) attempting that task, perhaps collected under human teleoperation and manually labeled with rewards, combined with plentiful data of other tasks, some of which might be structurally related (e.g., picking up an onion, or cutting& carrot). This scenario presents several questions: How do we decide which prior data should be included when learning the new task? And how do we determine which reward labels to use for this prior data?
Prior methods have offered several potential answers to these two questions, typically in isolation. For the first question, it has been recently observed that a naive sharing strategy of sharing data from all tasks can be highly suboptimal (Kalashnikov et al., 202 I), and some works have proposed both man- ual (Kalashnikov et al., 2021) and automated (Yu et al., 2021a; Eysenbach et al., 2020) data-sharing strategies that prioritize the most structurally similar prior data. Most such methods assume that this shared data can be automatically relabeled with the reward function for the new task (Kalashnikov et al., 2021; Yu et al., 2021a; Eysenbach et al., 2020), but the assumption that we have access the

---

Under review as a conference paper at ICLR 2022
functional form of this reward is strong one: for example, in many real- world settings, the reward might require human labeling or human-provin examples (Cabi et al., 2019; Finn et al., 2016b). To this end, some prior works have proposed learning classifiers for reward labeling (Fu et al., 2018b; Xie et al., 2018; Singh et al., 2019), or other automated mechanisms (Konyushkova et al., 2020). But these mechanisms themselves add complexity and potential brittleness to the pipeline. Thus, we aim to devise a simple unified method that determines which data to share and which rewards to use, with minimal supervision and no additional modeling and learning.
In this paper, we make the potentially surprising observation that data from other tasks can be utilized with naive constant reward labels, when the MDP consists of binary rewards. We show that this simple method, which does not involve learning any additional models or classifiers, can outperform more sophisticated techniques in practice. Our approach simply utilizes data from other tasks with a constant reward label (e.g., 0), and uses value-aware strategy to decide which prior transitions to include for the new task. This strategy, based on the conservative data sharing (CDS) technique proposed in prior work (which assumes oracle reward access) (Yu et al., 2021a), also does not require learning any additional model and simply uses the Q-function that is already learned as part of the RL process.
Our main contribution, which we call conservative unsupervised data sharing (CUDS), is a technique for sharing data in multi-task salti--tskk offline RL that does not require any reward labels or reward function access for the aask-agnostic data, and requires no additional model or classifier. To achieve that, our method assumes a particular form of the MDP that consists of binary rewards. We discuss the behaviors of our methods, showing that, even without ground truth reward labels, our simple data sharing scheme achieves Q-values that are lower-bounded by the Q-values obtained with sharing all data with the ground-truth rewards and can be combined CDS to selectively filter out potentially irrelevant data under different assumptions on the structure of the dataset. Our empirical evaluation conducted over various multi-task offline RL scenarios such as robotic manipulation from visual inputs and ant-maze navigation shows that this approach improves over the performance of more sophisticated techniques that either learn the reward function explicitly, or utilize other methods to propagate reward labels. In addition, we show that the proposed approach is comparable to an oracle baseline that has access to true multi-task rewards.
2 RELATED WORK
Offline RL. Offline RL (Ernst et al., 2005; Riedmiller, 2005; Lange et al., 2012; Levine et al., 2020) considers the problem of learning policy from a static dataset without interacting with the environment, which has shown promises in many practical applications such as robotic con- trol (Kalashnikov et al., 2018; Mandlekar et al., 2020; Rafailov et al., 2021), NLP (Jaques et al., 2019), healthcare (Shortreed et al., 2011; 1; Wang et al., 2018), 18), education (de Lima & Krohling, 2021), electricity supply (Zhan et al., 2021) and UT design (Apostolopoulos et al., 2021). The main challenge of offline RL is the distributional shift between the learned policy and the behavior policy (Fujimoto al., 2018; Kumar et al.. 2019), which can cause erroneous value backups due to -iittr-disttiittion actions generated by the learned policy. To address this issue, prior methods have constrained the learned policy to not deviate much from the behavior policy via policy regularization (Liu et al., 2020; Jaques et al., 2019; Wu et al., 2019; Zhou et al., 2020; Kumar et al., 2019; Siegel et al., 2020; Peng et al., 2019; Zhou et al., 2020; Kostrikov et al., 2021: Ghasemipour et al., 2021), conservative value functions (Kumar et al., 2020; Sinha & Garg, 2021), an auxiliary behavioral cloning loss (Fujimoto & Gu, 2021) and based training with conservative penalties (Yu et al., 2020d: Kidambi et al., 2020; Argenson & Alnald, 2020; Swazinna et al., 2020; Matsushima et al., 2020; Lee et al., 2021; Yu et al., 2021b).
Multi- Task RL and data sharing. Multi-task RL (Wilson et al., 2007; Parisotto et al., 2015; Teh et al., 2017: Espeholt et al., 2018; Hessel et al., 2019; Yu et al., 2020a; Xu et al., 2020; Yang et al., 2020; Kalashnikov et al., 2021; Sodhani et al., 2021; Stooke et al., 2021) enables the goal of learning a single policy that solves multiple skills efficiently. Despite the promising results, multi- task RL suffers from three main challenges, optimization difficulties (Schaul et al., 2019; Hessel et al., 2019; Yu et al., 2020a), effective weight sharing for learning shared representations (Parisotto et al., 2015; Teh et al., 2017; Espeholt et al., 2018; Xu et al., 2020; D' Eramo et al., 2019; Sodhani et al., 2021; Stooke et al., 2021), and sharing data across different tasks (Eysenbach et al., 2020; Kalashnikov et al., 2021; Yu et al., 2021a). We consider the multi-task offline RL. setting and focus
2

---

Under review as a conference paper at ICLR 2022
on the challenge of sharing data across different tasks. Prior works share data across tasks based on metrics such as learned Q-values (Eysenbach et al., 2020; Li et al., 2020; Yu et al., 2021a), human domain knowledge (Kalashnikov et al., 2021), the distance to the target goals in goal-conditioned settings (Andrychowicz et al., 2017; Pong et al., 2018; Nair et al., 2018; Liu et al., 2019; Sun et al., 2019; Lin al., 2019; Huang et al., 2019; Lynch & Sermanet, 2020; Yang et al., 2021; Chebotar et al., 2021), and the learned distance with robust inference in the offline meta-RL setting (Li et al., 2019). However, all of these either require access to the fuunctiona form of the reward functions of each task in order to relabel the rewards or are limited to condicondi settings. Therefore, they are not applicable to the multi-task offline RL problem that we consider where only the reward label of the eegiinally-xxcuteer task is provided. Our work addresses this issue via simply relabeling the data shared from other tasks with a constant value and uses the conservative data sharing strategy (Yu et al., 2021a) la) to further improve the performance.
RL with unlabeled data. Prior works tackle the problem of learning from data without reward labels via either directly imitating expert trajectories (Pomerleau, 1988; Ross & Bagnell, 2012: Ho & Ermon, 2016), learning reward functions from expert data using inverse RL (Abbeel & Ng, 2004; Ng & Russell, 2000; Ziebart et al., 2008; Finn et al., 20; Fu et al., 20;8;;;; Konyushkova et al., 2020), or learning a reward/ / value classifier that discriminates successes and failures (Xie et al., 2018; Singh al., 2021). These algorithms require online data collection and do not
et al., 2019; Eysenbach consider the offline RL setting. Singh et al. (2020) considers the single-task offline setting with both task-specific datasets and task-agnostic prior datasets and relabel the unlabeled prior data as failures since these prior transitions cannot solve the task. Our method is not limited to such single-task settings and instead considers the more general multi-task offline RL with data-sharing problem.
PRELIMINARIES
Multi-task RL. Standard multi-task RL considers multi-task Markov decision process (MDP), M = P, { RR, N), where S and A denote the state and action spaces respectively, P(s' (s, a) denotes the dynamics, 7 [0, 1) is the discount factor, and Ri.... ,Rx correspond to reward functions of different tasks i E [NV] for total number of N tasks where [N] is the shorthand for .N.. our setting. we assume a binary per-task R, E (0,1), where 1 denotes success of the task and 0 otherwise. Note that the dynamics are assumed to be the same across all tasks, which not entirely general but indeed practical in many problem settings as noted in Yu et al. (2021a) and stands as a common assumption in prior data sharing works (Yu et al., 2021a; Kalashnikov et al., 2021; Eysenbach et al., 2020). Regardless, there are many practical scenarios with changing rewards and invariant dynamics such as various object manipulation objectives (Xie et al., 2018), different goal navigation tasks (Fu et al., 2020), and distinct user preferences (Christiano et al., 2017). The goal of multi-task RL is to find condiiintt policy (alss i) that expected return in multi-tast MDP: x (als, == arg max, E,-[M] EE(1-.9 |E, 7 (s,,, a,) Note that it is possible to model the policies for each task independently as { (a)s), 1),. ... TT (a|s)} without any weight sharing. In our work, we use the single mos-coondtioned policy to study data sharing and do not consider the weight sharing aspect, which is orthogonal to the focus of the paper, which is also noted in Yu et al. (2021a)
Multi-task offline RL and data sharing. Multi-task offline RL considers the problem of learning the multi-tast policy T (a))s))) i) from a static multi-task dataset with D = ,, D, where D, = {(s,, ay, sj. } is the per-task dataset. Dj is generated by a behavior policy (als))) without any interaction with the environment. The most straightffrward approach to learn 5 (a]s, i) would be train it for task i only using D,. However, sharing data from different tasks to task has been shown to be conducive in the multi-task offline RL setting (Kalashnikov et al., 2021; Yu et al., 2021a). To do so, prior works (Bysenbach et al., 2021; Kalashnikov et al., 2021; Yu et al., 2021a) assume access to the functional form of the reward ri, which is a rather strong assumption that is usually impractical to specify in practical applications due to the challenge of reward specification. The next straightforward approach is to naïvely sharing data across all tasks, denoted as Sharing All. Formally, Sharing All defines the dataset of transitions relabeled from task j to task as Dz- and the method can be then defined as Drfff == D, U Du)): where Dr denotes the effective dataset for task i. While Sharing All improves over not sharing data, shown in Yu et al. (2021a), Sharing All leads to distributional shift that could degrade performance in certain situations ( (Yu et al., 2021a). In our work, focus on the CDS (Yu et al., 2021a), which relabels data that aims to mitigate the
3

---

Under review as a conference paper at ICLR 2022
distributional shift introduced by sharing other task data. CDS addresses such an issue by proposing a conservative data sharing strategy as follows:
sy. Dt = Di D (U)(((/((s)))))))))) ag r:) E Dj-H : (s, a) 2 0}),
(1)
where 8j a,, s denote the transition from Dj. ry denotes the reward of s,, a,, s/ relabeled for task i, denotes the dask-conditioned policy x (-|,,,), i), A (s, a,) is the condition that shares data only if the expected Q-value of the relabeled transition exceeds the top k-pprcentile of the Q-values of the original task data, i.e.
((s.a)= Pise Fa f a'', D....
Beyond controlling the distributional shift introduced in data sharing, multi-task offline RL also needs to address the main challenge in standard offline RL, which is the distributional shift between the learned policy and the behavior policy Te. To handle both types of distributional shifts, CDS (Yu et al., 2021a) combines the conservative data sharing and the constrained policy optimization problem and arrives the following objective:
Vi E [N], I (als, i) == arg max JTD (T)) - DD(TT, T
(3)
where 73"h (als, i) is the effective behavior policy for task i denoted as TB (als, i) := [Dr (s, (I)) (s) J'pa (T) denotes the average return of policy æ in the empirical MDP induced by the effective dataset, and D(T, ) denotes a divergence measure (e.g.. KL-divergence (Jaques et al., 2019; Wu al., 2019), fisher divergence (Kostrikov et al., 2021), MMD distance (Kumar et al., 2019) or DeQr. from conservative Q-learning (Kumar et al., 2020)) between the learned policy π and the effective behavior policy =. While optimizing Eq. 3 with Eq. 1 as the data sharing scheme is able to mitigate distributional shift and improve over multi-task offline RL without sharing data and naively sharing data across all tasks as shown in Yu et al. (202 la), it requires the assumption of the access to the functional form of the reward functions, which is rather strong and make application of data sharing real-world applications impractical. We will instead present a simple yet effective data sharing and relabeling scheme in the setting where we do not make such an assumption and instead, only have the reward labels for originally commanded task in the following section.
4 DATA SHARING WITHOUT REWARDS IN MULTI-TASK OFFLINE RL
The goal of our method is to enable effective data sharing across different tasks without access to the functional form of the reward functions for each task. Data from each task only labeled for that particular task, and we do not know priori which data is relevant each task. Effective data sharing therefore requires resolving two questions: (i) which data from other tasks should we use for given task? and (ii) how do we label this data with rewards? One simple approach is to annotate all available data from other tasks with some "proxy" reward signal, and treat it no differently from data that is already labeled. That is, after relabeling with the proxy reward, transitions into the replay buffer of value-based offline RL method. But how can we obtain reliable proxy reward signal? Next, we will discuss two variants of our method in Section 4.1, understanding of both variants in Section 4.2, and practical implementations in Section 4.3.
can simply put these
4.1 CONSERVATIVE UNSUPERVISED DATA SHARING
Prior work assumes that it is necessary to relabel prior data with some estimate of the true reward function that the proxy reward closely reflects the true reward. We take a different approach, and instead argue that, under some assumptions, we can obtain many of the benefits of data sharing simply by labeling the multi-task data with the lowest possible reward, which we assume to be 0 without loss of generality in the binary-rewand setting. We refer to this simple strategy as unsupervised data sharing (UDS). Naive UDS prescribes sharing data from every task every other task, and labels the shared data with a reward value of O. Formally, for each task i E [N]. we define the UDS procedure as follows:
D. -D,U [(8), aj, 81, E Dj- Vj E [N] (0)
(4)
4

---

Under review as a conference paper at ICLR 2022
Intuitively, UDS relabels data shared from other tasks with the lowest possible reward, hence making the leamned Q-functions more conservative that the data sharing scheme with the oracle rewards. We will show that UDS learns Q-values that Sharing All scheme with true reward relabeling, and can be information-theoreiicoe offline RL thanks to such conservatism. Our empirical results in Section 5 also suggests that the benefits from data sharing outweigh the downsides of reward bias in practice. Next, we move on from the choice of proxy reward, to the choice of which data should be shared.
ower-bounded by the Q-values learned by the naive
optimal in
While UDS is simple yet effective in multi-task data sharing without reward relabing, naively sharing data from all other tasks with zero rewards in the offline RL setting can result in overly conservative Q-functions and policies. To further refine this strategy, we can adapt the CDS algorithm (Yu et al., 2021a) detailed Section to filter out irrelevant transitions from other tasks, and only share those transitions that are likely to be informative. We call this strategy conservative unsupervised data sharing (CUDS). We define the CUDS strategy as follows:
D. = DD
ay, a,, E DJ- (s, a) 2 O Vj E [M] {d)
(5)
As we will discuss in the next subsection, CUDS is able to select potentially useful transitions under certain structural assumptions the multi-task offline dataset, and therefore produce Q-values that are not as excessively conservative as those produced by UDS. As shown in Section 5, our empirical evaluation further validates that CUDS improves over UDS and prior approaches.
Algorithm (Conservative) Unsupervised Data Sharing
Require: Multi-task offline datasets -. D.. I: Randomly initialize policy no ma(as, i).
2: for = 1, 2, 3,
do 3: Initialize Detl + f
4: for = L,... . do
5: D =D
aj 0) (M (UDS) or D =D:U((s), Dz-s:
A (s, 2 0 Vj E [M] (i)) using Eq. (CUDS).
6: Perform policy improvement by solving Eq. 3 by sampling data from Deft.
4.2 UNDERSTANDING THE BEHAVIOR OF UDS AND CUDS
In this section, we aim to understand the behavior of the UDS and CUDS. We first consider the UDS scheme, which simply shares all available data from other tasks, and labels the reward for each transition from other tasks as When instantiated with CQL as the offline RI. method, the Q-values of given policy learned by UDS for each task i are the fixed point of the recursion:
x (a)s, (a(s)) -1) (6)
2 + a, i) Pro
(5'. -a
where F(s, a, 0 for all (s. a) E Dz-vi # i, and F(s, a, is equivalent the empirical reward observed otherwise (Kumar et al., 2020). We will now try to understand how UDS compares to the No Sharing strategy, which only uses the labeled data for training. Note that this comparison is non-trivial since, while UDS utilizes a larger dataset, it can induce significant reward bias during training. However, by assumption, 0 is the lowest possible reward, we would intuitively expect that UDS should be more conservative, compared to Sharing All that relabels with the true reward. While we may surmise that being too conservative on unlabeled data may be suboptimal, conservatism has been shown to be cnpemmaiion--hooeeticae offline RL. and bandit problems. Even though an unlabeled dataset provides us with information about environment dynamics, it does not provide information about rewards, and any optimistic estimate of reward on this data may lead to poor performance in the worst case.
optimal (Jin et al., 2021; Rashidinejad et al., 2021) in
We formally derive the performance guarantee for UDS in Proposition F.1 using the framework of safe-policy improvement and discuss cases where can perform better than No Sharing. We discuss in Appendix F.2.3 that UDS can perform better than No Sharing in long horizon tasks well as in cases where the unlabeled dataset consists of similar proportions of various state-action pairs as the labeled dataset. Please refer to this section for the theoretical results. Our bounds utilize new
5

---

Under review as a conference paper at ICLR 2022
technique that allows us to prove tighther, non-trivial bounds for UDS. despite the pessimism which is discussed in Appendix F.2.1.
To understand the behavior of CUDS, we will consider simple abstract model of CUDS-style relabeling. In the tabular setting, this model updates the Q-function to match the target (conservative) Q-value if the transition is selected for the update, and retains the old table entry otherwise. Formally, consider binary vector w RISIx|A that indicates whether a corresponding state-action pair (s, is utilized for the backup or not. Then, our weighted scheme performs the following backups:
(s, a) ((s,a,))
At(a|ss -1) t. ar..aarrPPPlaa)a.. ( [ a',
i(als, i)
(1- (11-w(s) (Is, a)
Equation 7 can be intuitively understood as performing conservative backup from the actual transition observed in the dataset when the binary weight w(s, a) = 1, and simply truncating the Bellman backup and retaining the previous Q-values (5, a, i), otherwise. For example, CUDS performs a conservative backup with F(s,, a, S "Shhring Anl As, (s, a, only on transitions where w(s,a) A (s, a) 2 0].
To understand how this affects the resulting Q-function, we consider two structural conditions on the offline dataset: (1) scenario where no trajectory in the relabeled dataset for given target task Dj-: actually visits state-action tuples that were observed in D., and (2) when trajectories in Dj=;: overlap with at least a fraction of state-action tuples in the original labeled data for this task D;. We will abstract CUDS utilizing w (s. a) = I[ a, i) 2 i] for some threshold (see Eqn. 2).
Remark 4.1 (CUDS reduces to no sharing under condition (1)). When the trajectories in the unlabeled, relabeled dataset do not overlap with any trajectory in the labeled dataset for a given task, any backup performed by CUDS on an unlabeled transition will eventually drive its Q-value to O as > co. Thus, CUDS weights ("(s, a) will eventually take on 0 values for such transitions, and will not be selected by the future weights, i.e. wi a) 0 Vj 2 k 1.
Perhaps unsurprisingly, when the unlabeled data has no overlap with the labeled data, CUDS reduces to sharing. However, the more practically relevant case is when the unlabeled data overlaps with the labeled data. We consider the scenario when UDS has been run initially to obtain a starting set of Q-values, @0(s,, a, i), which defines the initial weight vector.
Remark 4.2 (CUDS selects more useful unlabeled transitions). Imagine a transitions (s, a, s',,'',0 0) e for which the next state (and the next policy action) (s', a') are observed in the labeled dataset (denoted D.). This transition will will attain large initial Q-values (* a, i) if executing the policy high reward of 1.0, due to the Bellman
after (s', a eventually reaches the state that corresponds backup component of CUDS. However, on the flip side, these backups performed by CUDS are conservative, and performing more backups can reduce the Q-value. Two scenarios might then arise: (i) the Q-values eventually decrease and CUDS is deactivated, i.e., 3k, w (s. O, in which case this transition is discarded and not used for learning anymore the backup in Equation 7 preserves the Q-value (the second term) when w (s, a) = 0, or (ii) the learning process reaches an equilibrium where w* w*(s,, a) -11% 1 k, meaning that this relabeled transition is used for learning.
We have now provided a theoretical analysis of CUDS and a comparison between CUDS and UDS in Appendix F.2, Additionally, we provide several new experiments to build insight into why UDS and CUDS work in Appendix G.
4.3 PRACTICAL IMPLEMENTATION
We present pseudocode for UDS and CUDS in Algorithm 1. We train the Q-values with CQL to obtain conservative Q-values, and use the conservative Q-values to compute A A"(s, a) defined in Eq. 2. For CUDS, in practice, instead of computing the hard threshold of A (s. a) 2 0 to determine data sharing, we follow Yu et al. (2021a) and transform the condition A ( a)
into soft weighting
scheme, with weights given by WCUDS (s, j + i) := (ss.ngj-)))
where 7 is a hyperparameter for the temperature of the sigmoid term in WCUDS that is automatically selected by the running average of (s,, j i). These weights are applied both critic and actor training. For both UDS

---

Under review as a conference paper at ICLR 2022
and CUDS, we train policy (aass, i) where It (aass i) could either be single task-cond task with weight sharing or separate policies for each task without weight sharing. For more details of the practice implementations, Appendix B.
EXPERIMENTS
In this section, we present our empirical evaluation, which aims to answer the following questions: (I) Can our simple approach outperform prior methods for utilizing unlabeled offline data on multi-task offline datasets? (2) Can the conservative data sharing strategy further improve the results achieved by our method? (3) Is our approach able to attain competitive result compared to the prior multi-task offline RL algorithm that have access to the true rewards? (4) How does CUDS compare to prior offline RL methods that directly learn representations from the multi-tass offline dataset and run offline training on top of the representation?
Comparisons. We compare UDS and CUDS to number of prior methods. We first evaluate: No Sharing, which performs applies standard offline RL algorithm to the multi-task setting without sharing data across tasks, Reward Predictor, which learns classifier that directly predicts the reward using supervised learning, VICE (Fu et al., 2018c), an inverse RL method that learns a reward classifier from the labeled data and then annotates the unlabeled data with the learned classifier, and RCE (Eysenbach et al., 2021), a method similar to VICE except that RCE represents the Q-function as a classifier and learns the reward for unlabeled data implicitly. We adapt VICE and RCE the multi-task offline RL setting by extracting transitions with reward labels equal to 1 and treating these datapoints as positives to learn the classifier for each task. We also train VICE and RCE, but adapt them to the offline setting using CQL., i.e. the same base offline RL method as in UDS and CUDS. Finally, to answer question (4), we conduct empirical evaluations on ACL (Yang & Nachum, 2021), which is recent offline RL algorithm that performs representation learning the offline dataset and trains the policy on top of the representation. For more details for experimental set-up and hyperparameter settings, please see Appendix B. We also include evaluations of our methods under different quality of the relabeled data in Appendix C, results of UDS and CUDS in dense-reward settings in Appendix D, comparisons to model-basee offline RI. approaches in Appendix E, and empirical analysis of the reasons that UDS and CUDS work in Appendix G.
5.1 MAIN EVALUATION
To answer questions (I), (2), and (3), we perform empirical evaluations on two state-based multi-task robotic manipulation and navigation datasets and one image-based multi-task manipulation dataset introduced in prior work (Yu et al., 2021a), which we will discuss below.
Figure 1: Environments (from left to right): Meta-Worke door and drawer open/close, AntMaze, and vision- based pick-place tasks.
Tasks and Datasets. Following the experimental setup in prior work (Yu et al., 2021a), we consider three domains shown in Fig. 1: (i) the Seta-World (Yu et al., 2020b) domain, which consists of four tasks of opening and closing doors and drawers; (ii) the Antmaze (Fu et al., 2020) domain, which consists of two sizes of mazes (medium and large) with 3 and 7 tasks respectively; and (iii) the multi-task visual manipulation domain, which consists of 10 tasks with different combinations of object-oriented grasping, with 7 objects (banana, bottle, sausage, milk box, food box, can and carrot), and placing the picked objects onto one of three fixtures (bowl, plate and divider plate). For all domains, we use binary rewards, where I denotes the successful completion of the task and 0 corresponds to failure. Note that for Meta- World, we use a fixed 200 timesteps for each episode and do not terminate the episode when receiving a reward of 1 at intermediate timestep. In Antmaze, we terminate the episode upon seeing a reward of with the maximum possible 1000 transitions per episode. We the same datasets as prior work (Yu al., 2021a). For
7

---

Under review as a conference paper at ICLR 2022
Environment
Tusks
(ours)
(ours)
VICE
RCE
No Sharing
Reward Predictor
door open door close
61.3%=7.99
51.9%-425.34 3%
0.0%
0.0%=2000%
14.5%-+12.7
0.0%-=20.0%
54.0% 1.2.5%
12.3% 6%
66.7%9%2144.15
0.0%
4.0% 1%
99.3% 10.9%
World
drawer open
73.8% 13.8%=9.68%
61.8%++16 3%
0.0%k=00.%%
0.0%-+0.0%
16.0% 55k
13.35-+18.99
drawer close
99.3%%=0.7%
7% 99.6%
19.39 3%
2
99. 99.0%%:+0.7%
50.35--135.86
average
71.2% = 11.3%
56.49/112.88
21.5%
0.7%
33.4%
11.0%:211.99
AntMaze
medioum maze tasles)
31.5% 1..5%--.0% 3.0%
26.55++9 19%
2.9%++1.0%
0.05-000%
21. 21.69-+7.18 1%
3.8%++3388 3.8%
large (7 tacks)
18.4%=6.11
14.2%++3.9%
1.5%
0.0%=+0.0%
13.3% 8.66
5.9%-
Table I: Results for multi-task robotic manipulation WorldWorldd) and navigation enviromments (AntMaze) with diweeminnionnll state inputs. Numbers are averaged across seeds, = the 95%-ccnnidence interval. We take the results of No Sharing directly from Yu et al. (2021a). We include per-task performance for Meta- World domains and the overall performance averaged across tasks (highlighted in gray) for all three domains. We bold the highest score across all methods. Both CUDS and UDS outperforms prior vanilla multi-task offline RL approach (No Sharing) and reward learning methods (Reward Predictor, VICE and RCE)
Task Name
CUDS (ours)
UDS
No Sharing
CDS (oracle)
Sharing All (oracle)
litt-banana
15.9%111.7%
48.6%:15.11% 1%
20.0%:16.005
53.1 1.133-3332999
4..8%
lift-bottle
72.9% :: 12.8%
58.1%.13.6%
49.7%.18.75 8.7%
74.0%-16.399
60.1%:110.22
1ift-sauaage
74.3% 8.3%
66.8% + 2.7%
60.9%116.%%
71.8%
70.0%
litt-milk
6.79%
74.5%
68.4%
83.4%615:.25 5.22%
72.5% 333%
lifL-food
66.3%=8..
53.8% 8%
39.1 39.0%%
61.4%+99.5%
58.5%
lift-can
67.1%
61.0%:1..88%
19.1%
65.5 65.5%%16.9%
57.7%:17.2%
litt-carrot
44.1%=3.68
73.4%
7.66%
83,8 33.5%%33%%%%
75.2%
place-bewl
13.4%=33.0
77.6%
8..3%%+t866%
31.0%
70.8%
place-plats
86.2% + 1.8%
78.7%%++2.2%
86.1% 7.7%%
85.8 6.66%
78.7%%++7.66 7.6%
place-divider-plee
89.0% 2.22%
80.2%%
5..9%
87.8 87.8%-1: 7
79.2%.-16.399
average
75.0% 3.33%
67.3%
60.8%++755%
74.8% +6.4%
66.4%%+7.2%
Table 2: Results for multi-task imaged-basee robotic manipulation domains in (Yu et al., 2021a). Numbers are averaged across 3 seeds, + the 95% confidence interval. UDS outperforms No Sharing in 7 out of 10 tasks as well as the average task performance, while performing comparably to Sharing All. CUDS further improves the performance of UDS and outperforms No Sharing all of the 10 tasks.
we use large datasets with wide coverage of the state space and 152K transitions for the door open and drawer close tasks and datasets with limited (2K transitions), but optimal demonstrations for the door close and drawer open tasks. For AntMaze, following Yu et al. (2021a), we modify the datasets introduced by Fu et al. (2020) by equally dividing the large dataset into different parts for different tasks, where each task corresponds manipulation, we directly use the dataset collected by Yu et al. (2021a), which contains a total of 100K RL episodes with 25 transitions for each episode, where the success rate is 40% and 80% for the picking and placing tasks, respectively. Note that the success rate of placing is higher because the robot is already holding the object at the start of the placing tasks, making the placing easier to solve.
different goal position. For image- based
Results of Question (1). The main results are in Table 1. UDS achieves better performance than vanilla multi-task offline RL without data sharing and compared to reward learning methods, sug- gesting that our simple relabeling method is effective in both multi-task manipulation and navigation domains. Since the reward learning approaches obtaín similar or worse results compared to no sharing, we only compare methods to No Sharing and the oracle methods in the image- image-based experiments. As shown in Table 2. UDS outperforms No Sharing in 7 out of 10 tasks as well as the average task performance by a significant margin. Therefore, UDS is able effectively leverage unlabeled data shared from other tasks and achieves potentially surprisingly strong results compared to more sophisticated methods that handle unlabeled offline data, answering question (1).
Results of Question (2). In both the state-based and vision-based experiments shown in Table I and Table 2, we find that CUDS further improves upon the performance of UDS, which empirically indicates that the less conservative policy learned from CUDS's s selective filtering scheme is more performant in practice. Additionally, we measure the success rates of the relabeled data in Table Appendix C, measured by the oracle multi-task reward function on the Meta- Meta-World and AntMaze domain. We see that the success rates of the relabeled data are above 0% by a significant margin in most of the tasks. This suggests that UDS and CUDS not simply relabeling with the true reward, since the relabeled data does not entirely consist of failures but rather has a significant number of successful transitions.
Results of Question (3). We also show results for oracle methods that receive true reward labels: Sharing All, which shares all data with ground truth rewards, and CDS, which uses the CDS strategy (Yu et al., 2021a) with ground truth reward relabeling. We present the results in Table 3 for state-based experiments and the last two columns on the right in Table 2 for the vision-based multi-
8

---

Under review as a conference paper at ICLR 2022
Environment
Tasks
CUDS (ours)
UDS (ours)
CDS (oracle)
Sharing All (oracle)
door open dour close
61.3% +7.9%
51.9%++25.8
58.4%+9.33%
34.3%-+ 17.9%
54.0% 1..2.5%
12.3%
65.3% 1.7.7%
48.3%:.1.7..3%
Mcta-Worid
drawer open
73.5% +99 6.%
31.8%+16..3%
57.9%=++16.25
drawer close
99.3%%+0..%%
99. 99.6% 0.7%
99. 99.0%%+0.7%%
98.8%%+0.7%
average
71.2% + 11.3%
56.4%:112.889
70.1%::8..
59.4%%:45.7%
medium maze (3 tusks)
31.5% 3.0%
26.5% 1%
36.7 % 1..2%
22.9% 6.6
AntMaze
large maze (7 tasks)
18.4%:1..1%
14.2%
22.8% 4.5%
16.7% = 7.0%
Table 3: Comparison between UDS/ / CUDS and the oracle data sharing strategies with access to the true reward functions for relabeling. We take the results CDS and Sharing All directly from Yu et al. (2021a). CDS (Yu al., 2021a) and Sharing All (Kalashnikov et al., 2021). UDS/ / CUDS achieve competitive results compared
to CUDS and UDS.
Enviromment
Tasks
CUDS (ours)
UDS (ours)
ACT.
open
61.3% 1.7.%%
51.99.11255
2.8%
door close
54.0% 4.42.5%
12.3% 6%
0.00%
Meta-World
drawer open
73.5% 6%
61.8%++ 16.3%
83.2%:114.28
drawer close
99.7%%
99.6%%
100,0%=0000
averuge
71.2% 1 11.3%
56.4%% 12.8%
46.4% 3.5%
Table 4: Comparison between UDS/ / CUDS and the ACL ( (Yang & Nachum, 2021) that performs representation learning on the unlabeled data instead of data sharing. Both UDS and CUDS outperforms ACL by significant margin in the average task result. suggesting that sharing the unlabeled data crucial improving the multi-task offline RL performance compared to only using the data for learning the representation.
task robotic manipulation problem. Both CUDS and UDS achieves competitive results compared to CDS and Sharing All, indicating that our simple relabeling scheme is able to remove the dependence of functional form of reward functions without much loss of performance due to lacking ground-truth reward access. This addresses question (3).
Results of Question (4). Finally, to answer question (4), on the Meta-World environment, we compare UDS and CUDS to ACI, (Yang Nachum, 2021). We use the version of ACL. without alternative to unlabeled sharing data scheme, which leverages unlabeled data for representation learning rather than sharing it directly. We show the comparison to ACL in Table 4. UDS and CUDS outperform ACL in the average task performance while ACL is only proficient on drawer-open and drawer-close, and it cannot solve door-open or door-close. This indicates that sharing the unlabeled data conservatively across all tasks is important in multi-task offline RL while pretraining representations on the whole multi-task offline dataset might have limited benefit. We note that UDS CUDS are complementary to ACL and these approaches can be combined together to further improve performance, which we leave as future work.
inputting reward labels. ACL can be viewed
CONCLUSION
In the paper, we present two new algorithms, UDS and CUDS, that handle the problem of how to share data across tasks without access to the functional form of the multi-task reward function in the multi-task offline RI setting. UDS lifts the strong assumption of having access to the reward of all tasks at each transition in previous works in multi-task data sharing via simply sharing data across all tasks and relabeling the reward of data from other tasks to the minimum reward in the MDP, which indicates failure of the task. CUDS further improves over UDS via applying a more sophisticated data sharing scheme (Yu et al., 2021a) that shares data only if the relabeled Q-values improve over the expected Q-values of the original task data. We justify that UDS obtains Q-values that are lower-bounded by the Q-values learned by data sharing with true reward labels and then discuss that under certain structures of offline datasets, CUDS can selectively apply conservative policy evaluation on only transitions with high Q-values, resulting in a less conservative algorithm. Empirically, we show that both CUDS and UDS significantly outperform vanilla multi-task offline RL without data sharing as well as more complex methods that learns the reward function either explicitly or implicitly on a range of robotic manipulation and navigation domains. CUDS also improves over UDS on all of the domains. Furthermore, CUDS and UDS achieve competitive results compared to data sharing methods with access to the oracle rewards. While our method removes the strong assumption on reward functions in data sharing for multi-task offline RL and enjoys both theoretical guarantees and good empirical results, it does have a few limitations. For example, UDS and CUDS are evaluated in MDPs with binary rewards. Exploring their effects in MDPs with continuous rewards will be exciting future avenue.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We provided the code, data, and instructions needed to reproduce the main experimental results of the experiments with low-dimensional imputs in the supplementary material. We include all assumptions and derivations of all claims in Section 4.2 and Appendix A. For datasets used in our experiments, we discuss the details of processing the datasets in Appendix B.
REFERENCES
Picter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv 2017.
preprint
Pavlos Athanasios Apostolopoulos, Zehui Wang, Hanson Wang, Chad Zhou, Kittipat Virochsiri, Norm Zhou, and Igor L Markov. Personalization for web-based services using offline reinforcement learning. arXiv preprint arXiv:2202.056// 2021.
Arthur Argenson and Gabriel Dulac-Arnold.
Model-based offline planning. arXiv preprint
arXiv:2008.055: 05556, 2020.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, al. Scaling data-driven roboties with reward sketching and batch reinforcement learning. arXiv preprint arXiv: /909. /2200, 2019.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.00749) 2021.
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint
1704.03746..,..
Leandro M de Lima and Renato A Krohling. Discovering an aid policy to minimize student evasion
using offline reinforcement learning. arXiv preprint arXiv:2 104. 10258, 2021.
Carlo D' Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement learning. In International Conference Learning Representa- tions, 2019.
Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pp. 2701-2709. PMLR, 2020.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron. Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukeuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, 2018.
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. arXiv preprint a0Xiv:2002.11089, 2020.
Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. arXiv preprint arXiv:2103.12656, 2021.
Chelsea Finn, Sergey Levine. and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49-58. PMLR, 2016a.
10

---

Under review as a conference paper at ICLR 2022
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEEE, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. International Conference on Learning Representations, 20.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: general framework for data-driven reward definition. Conference on Neural Information Processing Systems, 2018b.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. arXiv preprint arXiv: 16686, 2018c.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.0663 06860, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. arXiv preprini 1812 02900, 2018.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 1682-369) PMLR, 2021.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Conference on Neural Information Processing Systems, 2016
Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. Advances in Neural Information Processing Systems, 32:1942-19522 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1907.00455 2019.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096 PMLR, 2021.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651-673. PMLR, 2018.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. Conference on Robot Learning (CoRL), 2021.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005. 0055500050,9,,,,, 2020.
Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. Semi-supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.066 266999, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2022
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning Via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp. 1 11761-11771, 2019.
2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, volume 12. Springer, 2012.
Byung-Jun Lee, Jongmin Lee, and Kee-Fung Kim. Representation balancing offiine model-
reinforcement learning, In International Conference on Learning Representations, 2021. URL
nttps://openreiiin www.:/ettteeeeeeeeeeeeeeeeeeeaat wewrummiddpppzz--.... Ri2Y.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01645 2020.
Alexander C Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. arXiv preprint arXiv:2002.11708, 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Keith Ross, Henrik Iskov Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. arXiv preprint arXiv: 1909. 11373, 2019.
Xingyu Lin, Harjatin Singh Baweja, and David Held. Reinforcement learning without ground-truth state. arXiv preprint arXiv: 1905v 07866, 2019.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. arXiv preprint arXiv:1902.00526. 2019.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. arXiv preprint arXiv:2007.08202, 2020.
Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648. 2020.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4414-4420. IEEE, 2020.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.03647, 2020.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual 2018.
reinforcement learning with imagined goals. arXiv preprint
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, ICML '00, 2000.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforeement learning, arXiv preprint arXiv:1511.06342. 2015.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-po reinforcement learning. arXiv preprint 1rX10:1910.001177, 2019.
Dean A Pomerleau. Alvinn: an autonomous land vehicle a neural network. In Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 305-313, 1988.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep rl for masel-based control. arXiv preprint /rXivv/802.09088 2018
Rafael Rafailov, Tianhe Yu, A. Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. Learning for Decision Making and Control (L4DC), 2021.
12

---

Under review as a conference paper at ICLR 2022
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning; A tale of pessimism. arXiv preprint arXiv:2103. 12021, 2021.
Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforcement learning method. In European Conference on Machine Learning, pp. 317-328. Springer, 2005,
Stephane Ross and Drew Bagnell. Agnostic system identification for model-based reinforcement learning. ICML, 2012.
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: a source of plateaus in deep reinforcement learning. arXiv preprint 1etp:/9044.5555 2019.
Susan M Shortreed, Eric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Susan A Murphy, Informing sequential clinical deciisonnmmaknng through reinforcement learning: an empirical study, Machine learning. 84(1-2): (5(1-2):199-136: 2011.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002 2020.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.07833 2019.
Avi Singh, Albert Yu, Jonathan Yang. Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 14500, 2020.
Samarth Sinha and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning. arXiv preprint arXiv:2103.06326 2021.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. task reinforcement learning with context- based representations. arXiv preprint arXiv:2 1xXiv:202.06617,, 2021.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021.
Hao Sun, Zhizhong Li, Xiaotong Liu, Dahua Lin, and Bolei Zhou. Policy continuation with hindsight inverse dynamics. arXiv preprint
2019.
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. arXiv preprint arXiv:2008. 05533, 2020.
Yee Whye Teh, Victor Bapst, Wojeiech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. arXiv preprint arXiv: /707.04175, 2017.
L. Wang, Wei Zhang, Xiaofeng He, and H. Zha. Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
Aaron Wilson, Alan Fem, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning. pp. 1015-1022, 2007.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.1133 2019.
Annie Xie, Avi Singh, Sergey Levine, and Chelsea Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pp. 40-52. PMLR, 2018.
Zhiyuan Xu, Kun Wu, Zhengping Che, Jian Tang, and Jieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. 2020.
13

---

Under review as a conference paper at ICLR 2022
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. arXiv preprint mrXi:22102.05815, 2021.
Rui Yang, Jiafei Lyu, Yu Yang. Jiangpeng Ya, Feng Luo, Dijun Luo, Lanqing Li, and Xiu Li.
Bias-reduced multi-step hindsight experience replay. arXiv preprint arXiv:2102.12962, 2021.
Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. arXiv preprint arXiv:2003. /36611 2020.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06788 2020a.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning. pp. 1094-1100, PMLR, 2020b.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100. PMLR, 2020c.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020d.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. arXiv preprint 1xXiv:2109.08126 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363, 2021b.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Yusen Huo, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforce- ment learning. arXiv preprint arXiv:2102. 14492, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. arXiv preprint 1.002:20/11.022133 2020.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
14

---

Under review as a conference paper at ICLR 2022
A Q-VALUES LEARNED VIA UDS
Proposition A.1. For policy T,, let QuDs(:..;) denote the fixed point of Eqn. 6. Then, VuDs (s, a) lower-bounds the Q-function Q Sharing All that would be obtained had we used the true rewards.
Proof. Note that QUDs(S, (s. a) at iteration k defined Eq. 6, which we restate for convenience:
(Taass)))
Qbss
f(s. a, Ploaaa)) (s' a, (s.
-1] (8)
(alaa)))
whereas Tmaning All (s, a) at iteration k + 1 is defined as
Sharing Au (8, a, + r(s,(( a, i) + P (( Sharing An (s',''',' a',
(9)
π (als, a (a)ass. 1)
(10)
Assume Sharing Qins. iee same Q-value initialization and Qsharing All 2 U Using such induction hypothesis and the fact that F(s, a, & r(s, a, i) for all 8, a,a, we can conclude that @ūDS (s. a) S Sharing Au (s, a). Therefore, UDS learns the Q-value that lower-bounds the Q-values learned data sharing all tasks with the ground-truth rewards.
B DETAILS OF UDS AND CUDS
In this section, we include the details of training UDS and CUDS in Appendix B. as well as details the environment and datasets used in experiments in Appendix B.2. Finally, discuss the compute information of UDS and CUDS in Appendix B.3, For additional details, please see our anonymous website: https: // //sites....// google. com/viee htt//view//////// /ud/e//
B.1 DETAILS ON THE TRAINING PROCEDURE
Our practical implementation of UDS optimizes the following objectives for the Q-functions and policy:
h+1 - arg min Ei [8 (EA~iM) (BanD,annttinn) ((s, a.
BaaDD [((5. a,)
Bi Bj-NMiaaadd (Q(s, ( (1(s, () (10
and
π arg max Ei-IN]] BJ- (s,
Similarly, CUDS optimizes the following objectives for training the critic and the policy with a soft weight:
@++1 arg min E.JIN] B (Ey-IN) PanD,aaaiin (ecuus(s.:;) ())(() a.
- FeaND, wCUDs(s, ajj +
1) (aa::+i)((00.
0i]]
E, Em/(M,t wcuusss. a:j (Q(s. (r(s. +70(s ]
and
π - arg max Ei
( (ucns(s,a;) +)) (s,
where 3 is the coefficient of the CQL penalty distribution shift, μ is an action sampling distribution that covers the action bound as in CQL. We follow all the CQL hyperparameters used in Yu et al. (2021a).
15

---

Under review as a conference paper at ICLR 2022
To compute the weight WCUDs j + = a ((()):)))1)) we pick T, i.c. the temperature term, using the exponential running average of (ss, a; i) with decay 0.995 for each task following Yu et al. (2021a). Following Yu et al. (2021a) again, we clip the automatically chosen 7 with minimum and maximum threshold, which we directly use the values from Yu al. (2021a). We use [1, 50) and [10, oo] as the minimum and maximum threshold for the multi-task Meta- World and AntMaze domains respectively whereas the vision-based robotic manipulation domain does not require such clipping.
Following the training protocol in Yu al. (2021a), for experiments with low-dimensional inputs, we stratified batch with 128 transitions for each task to train the Q- D-functions and the policy. We also balance the numbers of transitions sampled from the original task and the number of transitions drawn from other task data. Specifically, for each task we sample 64 transitions from D; and the remaining 64 transitions from Uo In CUDS, for each task i E [N], we only apply WCUDS to environments and multi-task vision-based
data shared from other tasks on multi-task robotic manipulation tasks while we also apply the relabeling weight to transitions sampled from the original task dataset D; with 50% probability in the task AntMaze domain.
Regarding the choices of the architectures, for state- state-based domains, we use 3-layer feedforward neural networks with 256 hidden units for both the Q-networks and the policy. We condition the policy on a one-hot task ID, which is appended to the input state. In domains with high-dimensional image inputs, we adopt the multi-headed convolutional neural networks used in Kalashnikov et al. (2021); Yu et al. (2021a). We use images with dimension 472 x 472 x 3, extra state features (Frobot_staru theight) and the one-hot task vector as the observations similar Kalashnikov et al. (2021); Yu et al. (2021a). Following the set-up in Kalashnikov et al. (2018: 88 2021); Yu et al. (2021a), we use Cartesian space control of the end-effector of the robot in 4D space (3D position and azimuth angle) along with two binary actions to open/close the gripper and terminate the episode respectively to represent the actions. For more details, see Kalashnikov et al. (2018; 2021).
B.2 DETAILS ON THE ENVIRONMENT AND THE DATASETS
In this subsection, we include the discussion of the details the environment and datasets used for evaluating UDS and CUDS. Note that all of our environment and offline datasets are from prior work (Yu et al., 2021a). We will nonetheless discuss the details to make our work self-contained. We acknowledge that all datasets with low-dimensional inputs are under the MIT License.
Multi-task Meta-World domains. We use the door open, door close, drawer open and drawer close environments introduced in Yu et al. (2021a) from the public Meta-Worke (Yu et al., 2020c) repo'. In this multi-task Meta-World environment, a door and drawer are put on the same scene, which ensures that all four tasks share the same state space. The environment uses binary rewards for each task, which are adapted from the success condition defined in the Meta-World public repo. In this case, the robot gets a reward of if it solves the target task and 0 otherwise.
We direct use the offline datasets constructed in Yu et al. (2021a), which are generated by training online SAC policies for each task with the dense reward defined in the Meta-World repo for 500 epochs. The medium-replay datasets use the whole replay buffer of the online SAC agent until 150 epochs while the expert datasets are collected by the final online SAC policy.
Multi-task AntMaze domains. Following Yu et al. (2021a), we the antmaze-medd um-play and antmaze- weeaaeeellaae Large-play datasets from D4RL (Fu et al., 2020) and partitioning the datasets into multi-task datasets in an undirected way defined in Yu et al. (2021a). Specifically, the dataset is randomly splitted into chunks with equal size, and then each chunk is assigned to a randomly chosen task. Therefore, under such task construction scheme, the task data for each task is of low success rate for the particular task it is assigned to and it is imperative for the multi-task offline RL algorithm to leverage effective data sharing strategy to achieve good performance. binary reward, which provides the agent reward of +1 when the ant reaches position within a 0.5 radius the task goal, which is also the reward used default by Fu et al. (2020). The terminal of an episode is set to be true when a reward of +1 is observed.
AntMaze, we also use a
'The Meta-World environment can be found at the open-sourced repo https: / / thub.com/ 71 #1korkggroug//etaavvlldd
16

---

Under review as a conference paper at ICLR 2022
Environment
Tasks
Oracle Success Rate of the Shared data
drawer open door close drawer open drawer close
47.4%
99.2%
Meta-World
0.1%
91.6%
average
59.5%
medium maze (3 lasks) average
4.3%
AntMaze
large maze (7 tasks) average
1.6%
Table 5: Success of the data shared from other tasks the target task determined by the ground-truth multi task reward function.
Multi-fask image-based robotic picking and placing domains. Following Kalashnikov al. (2021); Yu et al. (2021a), we use sparse rewards for each task. That is, reward is assigned to episodes that meet the success conditions and otherwise. The success conditions defined in (Kalashnikov et al., 2021). We directly use the dataset used in Yu et al. (2021a). Such dataset is collected by first training policy for each individual task using QT-Opt (Kalashnikov al., 2018) until the success rate reaches 40% and 80% for picking tasks and placing tasks respectively and then combine the replay buffers all tasks as the multi-task offline dataset. The dataset consists total number of 100K episodes with 25 transitions for each episode.
B.3 COMPUTATION COMPLEXITY
We train UDS and CUDS on single NVIDIA Geforce RTX 2080 Ti for one day on state-based domains. For the rision-based robotic picking and placing experiments, it takes 3 days to train it on 16 TPUs.
C ADDITIONAL DETAILS ON THE QUALITY OF DATA SHARED FROM OTHER TASKS
We present the success rate of the data shared from other tasks to the target task computed by the oracle multi-task reward function in both the multi-task Meta-World and AntMaze domains in Table 5. Note that the success rate of drawer close and door close are particularly high since for other tasks, the drawer / door initialized to be closed and therefore the success rate of other task data for these two tasks are almost 100% as defined by the success condition in the public WorldWorld repo. Apart from these two particularly high success rates, the success rates of the shared data are consistently above 0% across all tasks in both domains. This fact suggests that UDS and CUDS are not relabeling with the ground truth reward where the relabeled data are actually all failures but rather performs the conservative bellman backups relabeled data that is shown to be effective empirically.
To better understand the performance UDS under different relabeled data quality, we evaluate the UDS under different success rates of the data relabeled from other tasks in the multi-task Meta- Meta-World domain. Specifically, we filter out data shared from other tasks to ensure that the success rates of the relabeled data are 5%, 50% and 90% respectively. We compare the results of UDS on such data compositions to the performance of UDS in Table 1 where the success rate of relabeled data is 59.6% shown in Table 5. The full results are in Table 6. UDS relabeled data with 50% and 90% success rates achieves similar results compared to original UDS whereas UDS on relabel data with 5% success rate is significantly worse. Hence, UDS can obtain good results in settings where the relabeled data is of high quality despite incurring high reward bias, but is not helpful in settings where the shared data of low quality and does not offer much information about solving the target task.
D EMPIRICAL RESULTS OF UDS AND CUDS IN MORE GENERAL DENSE REWARD SETTINGS
In this section, we evaluate UDS and CUDS in the dense reward setting in order to test if UDS and CUDS work in more general reward settings and not limited to binary rewards. We pick the multi-task walker environment as used in prior work (Yu et al., 2021a), which consists of three tasks, forward, run backward rd and jump. The reward functions of the three tasks r(s, a)
17

---

Under review as a conference paper at ICLR 2022
Environment
Tasks
UDS
UDS-5% relabel success
UDS-50% relabel success
UDS-90% relabel success
drawer open
51.99)-255.
0.0%==0.0%
18.9%
73.3%
door close
12.3%
0.0%
0.0%=0.0%
0.0%
Meta- World
drawer upen
11.8%:1116..99
19.4%:1127.3%
61.0%%:112.7%
56.3% 00.33%:::20.94
drawer close
99.6% 00.5%
66.0% 4-6
99.7%-t-0.5%
100.0% +..0%
average
56.4%:+12.88%
21.4%% +16.1
54,3% +2.0%
4 37.4%%-33334
Table 6: Performance of UDS under different actual success rates of the relabeled data.
Enviromment
Tasks Dataset type
CUDS (ours)
UDS (ours)
No Sharing
CDS (oracle)
Sharing All (oracle)
medium replay
880.1
590.1.18.66
1057.91121.
701.4147.0
walker2d
run backward/ medium
717.478..3
689.3116.3
614.7187.3
564.8-47.7
756.71:76.7
jump expert average
1487.7=177.6
1036.0=247.1
1575.2470.9
1418.2+13844
15529
1028.6476.88
196.
926.6-+37.7
1013:6=71.
781.0: 100.8
Table 7: Results for multi-task walker experiment with dense rewards. CUDS and UDS are able to outperform No Sharing while attaining competitive results compared to CDS and Sharing All with oracle rewards. This suggests that CUDS UDS are able to solve more general problems where rewards are not binary.
Environment
Tasks
CUDS (ours)
UDS (ours
COMBO (Yu et al., 2021b)
door open door close
61.3%
51.9%:1:25.3%
0.0%:110.0%
54.0% % +-42.5%
12.3%+427.6%%
1.1 1111111..99
Meta-World
drawer open drawer close
73.5% +-966%
61.8% 16.3%
15.7%
99.3% +..7%
99.6% % +00.%
85.7% 13.3%
average
71.2% + 11.3%
56.4%%--12.5
25.6%
Table 8: On the multi-tass task Meta- World domain, compare CUDS and UDS the model- based offline RL method COMBO (Yu al., 2021b) that trains dynamics model on all the data and performs model-based offline training using the learned model. CUDS and UDS are able outperform COMBO by a large margin,
and r(s, a) = --1lu,l] fotell-0.0011- * |all (1 init z) respectively where v, denotes the velocity along the x-axis and denotes the z-position of the half- cheetah and init denotes the initial z-position. In UDS and CUDS, we relabel the rewards routed from other tasks with the minimum reward value in the offline dataset of the target task. As shown in Table 7, CUDS and UDS outperform No Sharing by a large margin while also performing comparably to CDS and Sharing All. Therefore, CUDS and UDS are not limited to settings with binary rewards but are able to be applied to more general cases, in particular, environments with dense rewards.
8 0000.0 || r(s, r(s,))
0.000 ||a]]2 -
=
E COMPARISONS OF CUDS AND UDS TO MULTI-TASK MODEL-BASED OFFLINE RL APPROACHES
In this section, we compare CUDS and UDS to a recent, state-of-the-art model-based offline RI. method COMBO (Yu et al., 2021b) in the Meta-World domain. We adapt COMBO to the multi-task offline setting by learning the dynamics model on data of all tasks combined and and performing vanilla multi-task offline training without data sharing using the model learned with all of the data. As shown in Table 8, CUDS and UDS indeed outperform COMBO in the average task success rate. The intuition behind this is that COMBO unable to learn an accurate dynamics model for tasks with limited data as in our Meta- World setting.
18

---

Under review as a conference paper at ICLR 2022
F THEORETICAL ANALYSIS OF UDS AND CUDS
In this section, we will theoretically analyze UDS and CUDS to understand when these approaches perform well. We will first discuss notation, then present our theoretical results, then discuss the intuitive explanations of these results, and finally, provide proofs of the theoretical results.
F.1 NOTATION AND ASSUMPTIONS
Let (als) denote the behavior policy for task i (note that index i was dropped from Ta(a|s; i) for brevity). The dataset, D; is generated from the marginal state-action distribution of Tr, i.e., D a (a)rr, We define dz as the state marginal distribution introduced by the dataset 'D under и. For our analysis, we will abstract offline RL algorithms into generic constrained policy optimization problem (Kumar et al., 2020):
a (als) == arg max JD(
1-7 D(T,, ap)
(11)
(T) denotes the average return of policy и in the empirical MDP induced by the transitions in the dataset, and DT,,, RT) denotes divergence measure (e.g.. KL-divergence (Jaques al., 2019; Wu et al., 2019), MMD distance (Kumar et al., 2019) or DooL. (Kumar et al., 2020)) between the learned policy and the behavior policy kg. Let DeoL (p. q) denote the following distance between
two distributions p(x) and q(x) with equal support X::
p(x) ((x)
1).
DCoL (p.9) = Σp(x)
Unless otherwise mentioned, we will drop the subscript "CQL" from Door and use D and Deou interchangeably. Prior works (Kumar al., 2020; Yu al., 2021a) have shown that the optimal policy of that optimizes Equation 11 attains a high probability safe-policy improvement guarantee, i... J ( 2 J(TT) Se. where Çi is:
1
[V
Dool (T)) Ta (s) + [Di(s)
= O
E s
1-7 D(T mg).
(12)
71-
The first term in Equation 12 corresponds to the decrease in performance due to sampling error and this term is high when the single-task optimal policy л. visits rarely observed states in the dataset D; and/or when the divergence from the behavior policy Ty is higher under the states visited by the single-task policy d.. We will show that UDS and CUDS enjoy safe policy improvement. In our analysis, we assume r(s, a) E [0, 1]. Finally, as discussed in Section 3, let De denote the relabeled dataset for task i, which includes both D; and the transitions from other tasks relabeled with 0 reward.
Assumptions. To prove our theoretical results, following prior work (Kumar et al., 2020; Yu et al., 2021a) we assume that the empirical rewards and dynamics concentrate towards their mean. Assumption F.1. V s, a, the following relationships hold with high probability, 6
Cr,5 V)D(s,
Cp.s
[F a) r(s,a))
||
VID(s,a)
Similar to prior work (Kumar et al., 2020; eu al., 2021a), we also make a coverage assumption, i.e. we assume that each state-action pair observed in the dataset D, but the rewards and transition dynamics are stochastic, so, the occurrence of each state-action pair does not trivially imply good performance. To relax this assumption, we can extend our analysis to function approximation (e.g., linear function approximation (Duan et al., 2020)), where such coverage assumption is only required on all directions of the feature space, and not all state-action pairs. This would not significantly change the analysis, and hence opt for the simple but illustrative analysis in tabular setting here.
F.2 THEORETICAL RESULTS
We first provide a performance guarantee for UDS which is then used to show that under certain conditions the sizes of the labeled D; and the effective dataset, D.ff. UDS attains better policy improvement guarantee than naive no sharing. We first briefly discuss a novel component of our proof technique, then present the theoretical results, and then interpret it.
19

---

Under review as a conference paper at ICLR 2022
F.2.1 OUR PROOF TECHNIQUE
While there several techniques to provide guarantees for offline RL algorithms, we will build on the line of safe-policy improvement bounds, previously used in Kumar et al. (2020); Yu et al. (2021a). However, naively applying these guarantees to our UDS setting will give rise to very weak bounds, since number of these guarantees utilize bound on the value difference of the policy in the empirical MDP and the actual MDP (term (i)) as shown below:
J(x) (TT)) (m) J(T) + J(x) J(TT) + (ASs ((TT).
(3))
(sie)
Typically, term (i) depends on the sampling error on states that are visited by the learned policy x, and decays to 0 with infinite samples, but UDS can learn quite pessimistie Q-values due to the reward labeling procedure. However, this may not affect the policy performance since the relative ordering of actions might still be the same. This is not accounted for in any prior analysis
aware of.
Therefore, we introduce a novel analysis tool that, rather than decomposing J(xx) - ((T)) naïvely using the return in the empirical MDP, decomposes using the return of the policy in the best empirical MDP that still produces the policy ı as its optimal policy. One simple way to obtain this best empirical MDP is via affine transformations on the reward function that preserve optimality. So, for strengthening our bound, we shall compute the bound similar to the above equation for different affine transformations of the reward and pick the one that gives the tightest bound.
Formally, let g(-) be an affine function: g(x) = u x + v for some u > 0, u E IR and R. Then our decomposition looks like:
J(x) - J(T)) = ((x - 9 (π(T) + (Jm) (J(T)) + (J(ms)) J(TT),
(6)
(84)
(152)
Then, to obtain a strong lower bound on ((T) (Fz), we can first bound each of the terms for given choice of g (u, v), and then take the supremum over and This is reflected in the performance guarantee we present next.
F.2.2 PERFORMANCE GUARANTEE FOR UDS
Proposition F.1 (Policy improvement guarantee for UDS). Let Tuss denote the policy learned by UDS for a given task i, and let T off (als, i) denote the behavior policy for the combined dataset for task i, D.R. Then with high probability 2 1 6, vins is Ç-safe policy improvement over g i.e.
J (FiDs) 2 [
where Ç Çis:
Ç (= min Su,0 u20,e
Su,r
1
E
[1 f(s, a)]
au
D D( x
s.a pelf
(a): reward hias, but modified the best
(bj: policy improvement
2Cr,s7
Dool(wine 3 TT (s) +
2uCr.5
f(s. (s)
+ (1 (1-7)²
Eg,and
[D)
(1 1--7)
reward sampling erior scaled down
(e): dynamics sampling error
where we use the notation f(s. : D (s.a]
[D,(s.a)
proof of Proposition F.1 provided in Appendix F.3. intuitively interpret the various terms that appear, we note that term (b) corresponds to the standard policy improvement that arises as result of using an offline RL algorithm, term (c) corresponds to sampling error that arises as a result of performing offline RL on the dynamics induced by finite dataset, but note that this term depends on the size of the effective dataset, Deff and not only the labeled dataset D; for the task. Term (a) corresponds to the bias incurred result of labeling various transitions with 0 reward in the data, and term (d) corresponds to the sampling error the reward function, under the assumption of stochastic reward function.
20

---

Under review as a conference paper at ICLR 2022
F.2.3 How DOES UDS COMPARE TO No SHARING?
In the setting when no data shared across tasks, we attain the guarantee shown in Equation 12. Comparing Proposition F.I to this guarantee, note that under some scenarios, UDS yields tighter bound compared to No Sharing. Two such scenarios are given by:
Long-horizon tasks: Consider a scenario where tasks have a long horizon = 1 1-y and
(e(((s) H²2 (s)) In this case, dynamics sampling error term (term (c)) consists of one less factor of H when UDS is utilized, compared to when it is not. Since the dynamics sampling error grows quadratically in the horizon, whereas other terms grow linearly, reduction in this term by increasing sample size (i.e., denominator) can lead to a stronger guarantee for UDS than No Sharing. This reasoning does not even consider term (d), which can be trivially upper- upper-bounded by the corresponding term for No Sharing, even though UDS reduces this term as well.
2. The fraction [(s, a) is identical for all state-action pairs in the labeled D;, i.., the labeled dataset consists of equal proportions state-action pairs as the labeled dataset. Consider an extreme case when the unlabeled dataset consists of the trajectories in the labeled dataset such that f(s, a) = Co for all state-action tuples, just not annotated with rewards. In this case, reward bias takes on a constant value across all the transitions in the dataset, and by virtue of utilizing and in our bound in Proposition F.1, we note that the overall effect of this reward bias disappears, since o can compensate for this bias.
F.2.4 EXTENSION TO CUDS
Finally, we extend Proposition F.1 to performance guarantee for CUDS by integrating the technique above with the analysis from Yu al. (2021a). To analyze CUDS, we consider the abstract model of the conservative data sharing scheme developed by Yu et al. (2021a). This model suggests that CUDS approximates the following optimization in the empirical MDP generated by the relabeled dataset:
(Tt (alss, i), (a i)) arg
max 7 Elliredl
(
1-7 ((T,T(()))
(13)
Now, utilizing Proposition F.1 and Proposition 5.1 from Yu et al. (2021a), we obtain the following guarantee for CUDS:
Corollary F.1. Let (cuns(als, be the optimal policy found by CUDS (Equation 13) and let i)) denote the behavior policy that optimizes Equation 13 for task i E [N]. Then, with high
probability 2 1 - 8, ACuDs is Ç-safe policy improvement over Ti iee, J (ICcus) 2 (xrz)
ScuDs-
where SCuDS is given by:
SCUDS = min Su.v
u u>0,v
Su,
E
f(s, a)]
acl D D 33)
?
s,add"
Deff
(a):
(b): poiley improvement
2CRBY
Door Prcor(Tiun (FCUns: TH) +1
2uCr.s
[( D, a)
+ (1 7)
+ +
(1 (1-7) Baadd
(Delt (s)
[cj: dynamics sampling
reward sampling error, scaled down
Proof. The proof of this proposition follows directly from the proof of Proposition F.1 with the exception that this argument must be applied against the optimized behavior policy T3--
Comparing the bounds for CUDS and UDS. We now interpret the bound in Corollary F.1 com- paratively against the bound in Proposition F.1. First note that since the abstract model of CUDS (Equation 13) optimizes the behavior policy T.. we first note from Equation 14 of Yu et al. (2021a) that for any other behavior policy x'
D :T)) S D(FCUDDS n').
(14)
21

---

Under review as a conference paper at ICLR 2022
This means that the numerator of the sampling error term (term (c)) is smaller when CUDS is utilized as compared to when UDS is utilized. In addition, since CUDS relabels unlabeled data from Equation 13, this scheme also increases the dataset size, increasing the denominator of term (c). On the other hand, note that while UDS increases the denominator [Dett it may also increase the distributional shift D(TT" off appearing in the numerator of the sampling error term. Our practical version of CUDS (Equation 2), which approximates Equation 13 by relabeling only the top k percentile of the unlabeled data based on the objective in Equation 2, gives us control over the effective dataset size after relabeling [De while still ensuring reduced value of D("", (("",, TT)), and thus expected to reduce Ç compared to UDS.
Intuitively, note that the bounds in Proposition F.1 and Corollary F.I, guarantee safe-policy improve- ment over different base policies off vs T... Intuitively, we would expect that J(zz) 2 (m3*) in practice, especially for a large a, since CUDS optimizes the behavior policy towards high return, compared to simply relabeling all unlabeled transitions. Therefore, CUDS not only reduces com- pared UDS, but also, practice, expected to improve over z2. which performs better than reff Thus, we would expect CUDS be better in practice compared UDS.
F.3 PROOF OF PROPOSITION F.1
As mentioned in the beginning of Section 1.2.1, to strengthen the conventional safe policy improve- ment bound, we instead utilize a different form of loss decomposition of the improvement of the leamned policy relative to the behavior policy with the affine transformation g:
J(x) ( = ((T) (J(T) + (J()) 9 (J(TS) + (J(T))) - (TTO).
(1)
(Gxi)
Now we will discuss how to bound each of the terms: terms (i) and (ii) correspond to the divergence between a transformed empirical policy return and the actual return. While usually, this difference depends on the sampling eтror and distributional shift, in our case, it additionally depends on the reward bias induced on the unlabeled data and the transformation g. We first discuss the terms that contribute to this reward bias.
Bounding the reward bias. Denote the effective reward of a particular transition (s, a, r,,,s) Deff, as peff which considers contributions from both the reward f(ss, a) observed in dataset D,, and the contribution of ( reward from the relabeled dataset:
Poff (s, a) [D.(s. a)) f(s. a) + D (s, (,(s, -0
(15)
DD: a))
Define f(s. a)
[D (s.a] for notation compactness. Equation 15 and the form of the reward transformation g(z) u. + can then be used to derive the following difference against the true
[D.
rewards:
up. (s, as r(s, = uf(s, (f(s, a) r(s, a)) + (1 uf(s. a)) (0 (00-r(s,)) a)) + (16)
Cr.
< as) (iD,(s,a)
(1 a)u) r(s, +
(17)
C,
< af)(s,))
VID
where the last step follows from the fact that the ground-truth reward r(s, a) E [0, 1] and the fact that o will be chosen to minimize this upper bound. Now, we lower bound the reward bias as follows:
cefl (s. a) + r a) (s)(s)) (F(s, r(s,)) a)) + (1 [(s, a)w) (-r(s, + (18)
Crb (1 [(s a)w) +
2 - as)
ViD.ss,)
where the last step follows from the fact that r(s, a) < 1. To highlight the significance of this reward transformation, note that in the last step, if V's a, f(s, a) = cg. then the best reward transformation would choose Co. and that completely eliminates the excess bias induced in the bound.
22

---

Under review as a conference paper at ICLR 2022
(T). Next, using the upper and lower bounds on the reward bias, now derive upper bound on the difference between the value of policy computed under the empirical MDP and the actual MDP. To compute this difference, we follow the following steps
Upper bounding ([((T))
9 (J((m) - (T)
E ( (s)ss))))) (7: (s. a)) (((s)((a)))r(ssa)
(19)
Σ bye (s) (g (77 (s, a)) r(s. a)) + Σ
1
Th (s) d= x x (a)ss)r(ssa)
1
A
Following Kumar et al. (2020) (Theorem 3.6), we can bound the second term A₂ using:
PPps
V
1 S
E
D(T, RT (s) +
(20)
(a)
Deff (s)
To upper bound A,, we utilize the reward upper bound from Equation 16:
A
Σ
d (s)
Σ (aass) (wr (s, (s,
+ r(s, a))
(21)
Cr.8 T(als)
$ Detr (s)
Σ us a) VIDi(s) / (a(s)
(22)
=bi
Combining the results so far, we obtain, for any policy m:
J((Z) 2 9 (J((T))
A 1-7
(23)
Lower bounding 9 (I((T) - (T).. To lower bound this quantity, we follow the step shown in Equation 19, and lower bound the term A2 by using the negative of the RHS of Equation 20, and lower bound Ai by upper bounding its absolute value as shown below:
A
Σ Toaer (s)
[m(als)) T(as)) (ur (s, a) + - -r-ras,))))
(24)
Cr
(alss
Toga (s) Σ a)
- Σ T (s) Σ (als)) (1 a)))
[D((s)] V (als)
(25)
This gives rise to the complete lower bound:
g (Ji(z)) 2 J((m)
A
1
Σ (S) - ((s,
- ((s A
(26)
Policy improvement term (ii). Finally, the missing piece that needs to be bounded is the policy improvement term (ii) in the decomposition of g (J(z) - (J(mT)). Utilizing the abstract form of offline RL (Equation 11, we note that term (ii) is lower bounded as:
term (ii) > a#
D(T, mz).
(27)
Putting it all together. To obtain the final expression of Proposition F.I, we put all the parts together, and include some simplifications obtain the final expression. The bound we show is relative the
23

---

Under review as a conference paper at ICLR 2022
effective behavior policy Tz" Applying Equation 26 for term (i) policy म, Equation 27 for term (ii), and Equation 23 for the behavior policy nza we obtain the following:
(T) J (TT J(x)
(J(zz) + (J(+))
(J(mzt) + (J(mzT)
(1TgT)
27 Cr B3
VIA
DTT,, (ggh)) (s) +1 2uCrnd
[ a)
(1 71-
snnft
(8)
(s)
1 7 IE
s.and
(1):(ss
an + +1-1 (IT,, x3
18
s.and
[] f(s, a)u]
:=
Note that in the second step above, we upper bound the quantities A and A₂ corresponding to nef with twice the expression for policy и. This is because the effective behavior policy #3 consists of mixture of the original behavior policy f.y with the additional data, and thus the new effective dataset consists of the original dataset D; as its part. Upper bounding it with twice the corresponding term for π is valid bound, though bit looser, but this bound suffices for our interpretations.
T
Finally to finish the proof, we can take the supremum over the best choice of (u, v). Thus, we obtain the desired bound in Proposition F.1.
G EMPIRICAL ANALYSIS OF THE REASON THAT CUDS AND UDS WORK
In this section, we perform an empirical study on the Meta- World domain to better understand the reason that UDS and CUDS work well. Our theoretical analysis suggests that UDS will help the most on domains with limited data or narrow coverage or low data quality. To test these conditions in practice, we perform empirical analysis on two domains as follows.
G.1 WORLD DOMAINS
We first choose the door open task with three different combinations of dataset size and data quality of the task-specific data with reward labels:
2k transitions with the expert-level performance (i.e. high-quality data with limited sample size and narrow coverage)
2k transitions with medium-level performance (i.c. medium-quality data with limited sample size and narrow coverage)
medium-replay dataset with 152k transitions (i.e. medium-quality data with sufficient sample size and broad coverage).
We share the same data from the other three tasks, door close, drawer open and d drawer close as in Table 1, which are As shown in Table 9, both UDS and CUDS are able to outperform No Sharing in the three settings, suggesting that increasing the coverage of the offline data as suggested by our theory does lead to performance boost in wherever we have limited good-quality data (expert), limited medium- -uallityy data (medium) and abundant mediium-qullin data (medium- replay). It's worth noting that UDS and CUDS significantly outperform No Sharing in the limited expert and medium data setting whereas in the mediuy-replay setting with broader coverage, CUDS outperforms No sharing but UDS fails to achieve non-zero success rate. Such results suggest that UDS and CUDS can yield greater benefit when the target task doesn' 't have sufficient data and the number of relabeled data is large. The fact that UDS is unable to learn medium-replay datasets also suggests that data sharing without rewards is less useful in settings where the coverage of the labeled offline data is already quite broad.
G.2 D4RL HOPPER DATA QUALITY COVERAGE DIAGNOSTIC STUDY
To further understand the sensitivity of UDS to the data coverage and the data quality of both target task data (i.e. with reward labels) and relabeled data (i.e. without reward labels), we perform another empirical study using the hopper environment from the D4RL (Fu et al., 2020) benchmark. We consider the following different combinations varying the quality and amount of the labeled and unlabeled datasets:
24

---

Under review as a conference paper at ICLR 2022
Environment
Dataset type/ size
CUDS (ours)
UDS (ours
No Sharing
expert / 2k transitions
67.6% %
58.8%
31.3%
Meta-World door open
medium / 2k transitions
67.3%
74.2%
27.6%
medium-replay / 152k transitions
30.0% %
0.0%
14.8%
We perform an empirical analysis on the Meta-World door open task where we use varying data quality and dataset size larget task door open. We share the same dataset from the other three tasks in the multi-task Meta- World environment, door close, drawer open The numbers are averaged over three random seeds. CUDS and UDS are able to outperform No Sharing in most of the settings except that UDS fails to achieve non-2ero success rate in the medium- dataset with large number of transitions. Such results suggest that CUDS and UDS robust to the data quality of the target task and work the best in settings where the larget task has limited data.
Table 9:
drawer close to the target task.
1. 10k labeled data from high-quality narrow labeled data, low-quality + broad unlabeled data)
+ unlabeled IM data hopper-randon, (i.e.,
2. 10k labeled data from hopper-experss + unlabeled 1M data from hopper-medium (i.e., high-quality + narrow labeled data, quaium- + narrow unlabeled data)
3. 10k labeled data from hopper- mediimediii um + unlabeled 1M data from hopper-rance (i.e., medium-quality + narrow labeled data, low-quality + broad unlabeled data)
4. 10k labeled data from hopper um + unlabeled 1M data from hopper-expen (i.e., medium-quality narrow labeled data, high-quality + narrow unlabeled data)
5. 10k labeled data from hopper-randen + unlabeled 1M data from hopper mediium (i.e., low-quality + broad labeled data, medium-quality narrow unlabeled data)
6. 10k labeled data from hopper- + unlabeled 1M data from
(i.e., low-quality + broad labeled data, high-quality + narrow unlabeled data)
Results. In cases (1) and (2), adding the unlabeled random or medium data, should increase coverage, since the labeled data only consists of expert transitions. Moreover, the induced reward bias due to incorrect labeling of rewards on the medium unlabeled data should not hurt, since the 10k expert transitions retain their correct labels, and the medium/random data should only serve as negatives. Therefore, we expect the benefits of coverage to outweigh any reward bias, and as shown in Table 10, we find that UDS does help.
In cases (4), (5) and (6), when the relabeled data is better compared to the labeled data (i.e., expert or medium), we find that even the rewards on these transitions are incorrect, behavior regularization properties induced by offline RL algorithms allow UDS to attain better performance than no sharing by utilizing the unlabeled data.
In case (3), we find that UDS hurts compared to No Sharing. This is because the target task data as well as unlabeled data are both low-medium quality and medium data already provides decent coverage (not as high as random data, but not low as expert data). Therefore, in this case, we believe that the addition of unlabeled data neither provides trajectories of good quality that can help improve performance,
does it significantly improve coverage, and only hurts by incurring reward bias. We therefore believe that UDS may not help in such cases where the coverage does not improve, and added data is not so high quality.
G.3 SUMMARY OF EMPIRICAL ANALYSIS
Given results in Table 9 and Table 10, we summarize the applicability of UDS under different scenarios in Table 11 below.
25

---

Under review as a conference paper at ICLR 2022
Enviromment
Labeled dataset type/ size
Unlabeled dataset type size
UDS (ours)
No Sharing
expert 10k transitions
random / 1M transitions
90.8
77.1
expert/ transitions
medium/ transitions
87.6
77.1
DARL hopper (Fu al., 2020)
medium/ 10k transitions
random IM transitions
9.8
28.7
medium/ 10k transitions
expert 1M transitions medium/ transitions
106.1
28.7
random 10k transitions
9.6
random 10k transitions
expert transitions
Table 10:
We perform an empirical analysis on hopper environment from D4RI. (Fu al., 2020)
benchmark to test the sensitivity of UDS under the data quality and data coverage for both the labeled task data and unlabeled data. The numbers are averaged over three random seeds. UDS outperforms No Sharing in 5 of 6 settings, suggesting that UDS robust in different combinations of data quality and coverage of both labeled and unlabeled data. Note that UDS fails the setting where the labeled data is of medium data quality and the unlabeled data random, suggesting that sharing data settings where the labeled data is limited and of low quality and the unlabeled data is also of poor quality is not useful.
Scenarios
UDS
Intuition
L: limited high-quality narrow, U: abundant low-quality broad
coverage
L: limited high-quality narrow. U: abundant medium--uulity narrow
more negatives
L: limited medium-quulity narrow, U: abundant low-quality broud
reward bius outweighs high coverage
L: abundant medium- quality
abundant medium quality broad
bias outweighs high coverage
L. limited ceadium-qulliyy narrow, U: ahundant high-quallty narrow
increase data quality
L. limited + low-quality + broad, U: abundant + medium-quallee qualiity narrow
increase data quality
L: limited low-quality + broad, U: abundant high-quality narrow
increase data quality
Table 11: Summary scenarios where UDS expected to work and where not expected work.
denotes the characteristics of labeled data, U denotes characteristics of unlaheled data. momited/Ahundann refers the relative amount of data available (note that these not absolute numbers and hard to precisely quantify without access the problem domain. but highly skewed ratio of the amount of labeled and unlabeled data might help characterize as limited/abundant. performance of the behavior policy generating the datasets. Narrow/broad refers to the relative state coverage of the datasets that we study.
refers to the actual
26


Reviews:
[{"review_id": "w6DnidgCKar", "reviewer": "Reviewer_YPSx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nTwo remarks in Section 4 provide an interesting intuition about how to use unlabeled transitions depending on whether each transition can reach a labeled state-action pair.\nThis work has empirically shown that a simple constant reward (zero reward) is enough to achieve good performance in practice.\n\nWeakness:\nConsidering extending CDS into multi-task RL without oracle reward function, the simplest direction is relabeling unlabeled transitions with minimum reward. But it is not clear whether the direction is valid. Therefore, the readers expect justification of using simple relabeling. However, CUDS is only supported empirically without any theoretical analysis. \n\nProposition 4.1 is proposed to prove the Q-value function is underestimated by UDS. However, it is not clear why such lower estimation is desirable in offline multi-task RL. Specifically, it seems too conservative way to estimate unlabeled rewards. For example, if we consider the $(s,a)$ is in both labeled and unlabeled data, then the reward of the state-action pair may be underestimated.\nIn addition, it is trivial since $\\hat{r}(s,a)\\le r(s,a)$ for all $(s,a)$ pairs.\n", "summary_of_the_paper": "This paper proposes a new offline multi-task RL algorithm named conservative unsupervised data sharing (CUDS). Prior works on multi-task RL require the functional form of the reward functions or are limited to goal-conditioned setting. On the other hands, CUDS shares task-agnostic transitions via simply relabeling their rewards as a constant.\nThe authors first prove that $\\hat{Q}_\\text{UDS}\\le \\hat{Q}_\\text{sharing All}$ for all state-action pairs. Furthermore, they argue that the CUDS can achieve better performance through weighting transitions.\nFinally, CUDS outperforms or performs competitively with the previous works. Surprisingly, CUDS achieves competitive results compared to CDS, which uses true reward functions for relabeling\n", "summary_of_the_review": "The remarks and empirical results are interesting. However, CUDS is a simple variant of CDS – the only difference is how to set the rewards. Therefore, I think CUDS needs additional meaningful theoretical analysis in addition to empirical results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tIlZUTX6xp", "reviewer": "Reviewer_UDVV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The problem addressed in the paper is indeed critical in offline RL. It has a reasonable motivation. \n\nIssues:\n- The biggest concern is the novelty. The proposed CUDS and UDS methods share similarities with one existing study (Yu et al. 2021a). Indeed the proposed method is more sophisticated than this previous paper; however I would expect the authors to emphasize that there exists a substantive difference between the two different works.\n\n- In this work’s multi-task offline RL setting, the authors assume the dynamics to be the same across all tasks and use a single task-conditioned policy. It is okay to assume the tasks only differ in reward distributions but I would expect a clearer justification.\n\n- How does the proposed method differ from learning environment dynamics with more data (in a maybe model-based offline RL setting)? More insights on this point would be appreciated. \n", "summary_of_the_paper": "This paper proposed a method for multi-task offline reinforcement learning setting where it can assign shared and annotate the reward, where its major strategy is to label only useful transitions. Specifically, the proposed method followed the conservative learning setting with the hope of easing the distribution shift issue. The authors also study the behavior of their proposed methods with s set of arguments and experimental results. ", "summary_of_the_review": "The major issue of this paper is it seems to be an incremental study from existing works. I would expect the authors to address this issue and add more insights. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Ocv4VXsji3", "reviewer": "Reviewer_SFTg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n1. This work proposed two easily implemented approaches called CUDS and UDS to achieve data sharing in offline reinforcement learning settings, without the requirement of a reward function approximation.\n\nSome of the questions under concern:\n1. One of the most important claims in this work is that under some assumptions, the proposed data sharing methods (CUDS, UDS) will be beneficial even suppress the oracle. However, there is not a clear statement or discussion about the intuition when and why it can achieve this? Although there are some experiments such as meta-world, etc., it is still confusing whether these data-sharing methods can be utilized in other application scenarios.\n2. There is an assumption in the targeted MDP is that the reward is binary. Is it necessary for CUDS/UDS to work?\n3. In section 4.2, the author tried to provide the intuition of how data sharing influences Q-estimates. However, remark 4.2 seems to be unclear and can't lead to the later conclusion that \" CUDS does not have to push down Q-values at every state-action pair\". Can you give more explanation about it?\n4. The writing of the paper needs to be polished a lot. I will provide extra comments later.\n\nSome minor comments about writing:\n1. A lot of descriptions and equations are too similar to the most related work [1]? It may need more revision. Such as the equ 2 and 3, even Algorithm 1 looks all similar.\n2. What's the definition of '[N]' and it has been used as a distribution in Section 3?\n3. Typo in Section 3 as 'D>'?\n4. The definition in Equ (1) is not clear? Such as what is (s, a), what is $\\pi$, and what the $\\mathcal{D}_i^{eff}$ may need the input of $\\pi$, since different policies may lead to different sharing datasets?\n5. More omits of definitions after equation (3), such as what's the exact formulation of the notation $J$ and $D(\\pi,\\pi_\\beta)$\n6. The text \"CQL\" is also introduced without reference, although it may refer to conservative Q-learning.\n7. Undefined $\\hat{Q}_{UDS}^\\pi(s,a)$ in Proposition 4.1\n8. $w$ is a binary matrix as defined above of equation (7), not a vector. Moreover, what is the difference between $w(s,a)$ and $w(s,a,I)$?", "summary_of_the_paper": "This work targets useful data sharing between multi-tasks in an offline reinforcement learning setting. Motivated by the same transition kernel $P$, for each task, data sharing from other tasks is considered to be beneficial for finding an optimal policy. Different from previous works, the author claims that under some assumptions, data sharing using a constant reward/no reward can be better than using a reward predictor and competitive with using ground-truth rewards relabelled. Towards this, the author proposes two methods called CUDS and UDS and conducts experiments to test its performance.", "summary_of_the_review": "I recommend that this paper be below the acceptance threshold. \n\n1. This paper release the strong assumption with a reward relabelling function in [1], which archives data sharing between multi-tasks without reward. It seems like an extension with a minor revision of CDS in [1]. \n2. The intuition of the great performance is also straightforward since more data yields a better estimation of the transition kernel $P$ in MDP, even without rewards. \n3. Although in the examined experiments the proposed methods show competitive performances compared to CDS and other baselines, it is not convincing enough that without an estimation of the reward, CUDS/UDS can achieve great results in more general scenarios.\n\n\n[1] Yu, Tianhe, et al. \"Conservative data sharing for multi-task offline reinforcement learning.\" arXiv preprint arXiv:2109.08128 (2021).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although sharing data between tasks benefits multitask RL, this requires that rewards be relabeled across tasks. This paper shows that, for binary rewards, directly reusing data from other tasks with constant reward relabels is effective, and the paper develops a method around this idea that is highly effective.  The reviewers found that the idea and execution were impressive, that the paper was well written, and that the empirical analysis was convincing. 

In response to concerns in the preliminary reviews about certain shortcomings in the empirical analysis and some lack of theoretical analysis, the authors provided substantial revisions to the paper. Due to some lack of reviewer response to the discussion, this meta-reviewer examined whether those revisions were sufficient to address the reviewers' concerns. The authors did a good job in providing the requested improvements and the analysis is stronger, but remaining similarities to existing methods (CDS) means that this paper still remains borderline. These same concerns were also shared by reviewers that continued to engage in discussion with the authors. To remedy this, the authors are encouraged to better and more substantially address differences with prior work in the writing and motivation throughout the entire paper. In addition, although space is a concern, it would be beneficial to integrate the high-level takeaways from the new analyses in the appendices into the main paper.
Decision: Reject