Data Sharing without Rewards in Multi-Task Offline Reinforcement Learning

1 INTRODUCTION . Offline reinforcement learning ( RL ) provides the promise of a fully data-driven framework for learning performant policies . To avoid costly active data collection and exploration , offline RL methods utilize a previously collected dataset to extract the best possible behavior , making it feasible to use RL to solve real-world problems where active exploration is expensive , dangerous , or otherwise infeasible ( Zhan et al. , 2021 ; de Lima & Krohling , 2021 ; Wang et al. , 2018 ; Kalashnikov et al. , 2018 ) . However , this concept is only viable when a significant amount of data for the target task is available in advance . A more realistic scenario might allow for a much smaller amount of task-specific data , combined with a large amount of task-agnostic data , that is not labeled with task rewards and some of which may not be relevant . For example , if our goal is to train a robot to perform a new manipulation task ( e.g. , cutting an onion ) , we might have some data of the robot ( suboptimally ) attempting that task , perhaps collected under human teleoperation and manually labeled with rewards , combined with plentiful data of other tasks , some of which might be structurally related ( e.g. , picking up an onion , or cutting a carrot ) . This scenario presents several questions : How do we decide which prior data should be included when learning the new task ? And how do we determine which reward labels to use for this prior data ? Prior methods have offered several potential answers to these two questions , typically in isolation . For the first question , it has been recently observed that a naı̈ve sharing strategy of sharing data from all tasks can be highly suboptimal ( Kalashnikov et al. , 2021 ) , and some works have proposed both manual ( Kalashnikov et al. , 2021 ) and automated ( Yu et al. , 2021a ; Eysenbach et al. , 2020 ) data-sharing strategies that prioritize the most structurally similar prior data . Most such methods assume that this shared data can be automatically relabeled with the reward function for the new task ( Kalashnikov et al. , 2021 ; Yu et al. , 2021a ; Eysenbach et al. , 2020 ) , but the assumption that we have access to the functional form of this reward is a strong one : for example , in many real-world settings , the reward might require human labeling or human-provided examples ( Cabi et al. , 2019 ; Finn et al. , 2016b ) . To this end , some prior works have proposed learning classifiers for reward labeling ( Fu et al. , 2018b ; Xie et al. , 2018 ; Singh et al. , 2019 ) , or other automated mechanisms ( Konyushkova et al. , 2020 ) . But these mechanisms themselves add complexity and potential brittleness to the pipeline . Thus , we aim to devise a simple unified method that determines which data to share and which rewards to use , with minimal supervision and no additional modeling and learning . In this paper , we make the potentially surprising observation that data from other tasks can be utilized with naı̈ve constant reward labels , when the MDP consists of binary rewards . We show that this simple method , which does not involve learning any additional models or classifiers , can outperform more sophisticated techniques in practice . Our approach simply utilizes data from other tasks with a constant reward label ( e.g. , r = 0 ) , and uses a value-aware strategy to decide which prior transitions to include for the new task . This strategy , based on the conservative data sharing ( CDS ) technique proposed in prior work ( which assumes oracle reward access ) ( Yu et al. , 2021a ) , also does not require learning any additional model and simply uses the Q-function that is already learned as part of the RL process . Our main contribution , which we call conservative unsupervised data sharing ( CUDS ) , is a technique for sharing data in multi-task offline RL that does not require any reward labels or reward function access for the task-agnostic data , and requires no additional model or classifier . To achieve that , our method assumes a particular form of the MDP that consists of binary rewards . We discuss the behaviors of our methods , showing that , even without ground truth reward labels , our simple data sharing scheme achieves Q-values that are lower-bounded by the Q-values obtained with sharing all data with the ground-truth rewards and can be combined CDS to selectively filter out potentially irrelevant data under different assumptions on the structure of the dataset . Our empirical evaluation conducted over various multi-task offline RL scenarios such as robotic manipulation from visual inputs and ant-maze navigation shows that this approach improves over the performance of more sophisticated techniques that either learn the reward function explicitly , or utilize other methods to propagate reward labels . In addition , we show that the proposed approach is comparable to an oracle baseline that has access to true multi-task rewards . 2 RELATED WORK . Offline RL . Offline RL ( Ernst et al. , 2005 ; Riedmiller , 2005 ; Lange et al. , 2012 ; Levine et al. , 2020 ) considers the problem of learning a policy from a static dataset without interacting with the environment , which has shown promises in many practical applications such as robotic control ( Kalashnikov et al. , 2018 ; Mandlekar et al. , 2020 ; Rafailov et al. , 2021 ) , NLP ( Jaques et al. , 2019 ) , healthcare ( Shortreed et al. , 2011 ; Wang et al. , 2018 ) , education ( de Lima & Krohling , 2021 ) , electricity supply ( Zhan et al. , 2021 ) and UI design ( Apostolopoulos et al. , 2021 ) . The main challenge of offline RL is the distributional shift between the learned policy and the behavior policy ( Fujimoto et al. , 2018 ; Kumar et al. , 2019 ) , which can cause erroneous value backups due to out-of-distribution actions generated by the learned policy . To address this issue , prior methods have constrained the learned policy to not deviate much from the behavior policy via policy regularization ( Liu et al. , 2020 ; Jaques et al. , 2019 ; Wu et al. , 2019 ; Zhou et al. , 2020 ; Kumar et al. , 2019 ; Siegel et al. , 2020 ; Peng et al. , 2019 ; Zhou et al. , 2020 ; Kostrikov et al. , 2021 ; Ghasemipour et al. , 2021 ) , conservative value functions ( Kumar et al. , 2020 ; Sinha & Garg , 2021 ) , an auxiliary behavioral cloning loss ( Fujimoto & Gu , 2021 ) and model-based training with conservative penalties ( Yu et al. , 2020d ; Kidambi et al. , 2020 ; Argenson & Dulac-Arnold , 2020 ; Swazinna et al. , 2020 ; Matsushima et al. , 2020 ; Lee et al. , 2021 ; Yu et al. , 2021b ) . Multi-Task RL and data sharing . Multi-task RL ( Wilson et al. , 2007 ; Parisotto et al. , 2015 ; Teh et al. , 2017 ; Espeholt et al. , 2018 ; Hessel et al. , 2019 ; Yu et al. , 2020a ; Xu et al. , 2020 ; Yang et al. , 2020 ; Kalashnikov et al. , 2021 ; Sodhani et al. , 2021 ; Stooke et al. , 2021 ) enables the goal of learning a single policy that solves multiple skills efficiently . Despite the promising results , multitask RL suffers from three main challenges , optimization difficulties ( Schaul et al. , 2019 ; Hessel et al. , 2019 ; Yu et al. , 2020a ) , effective weight sharing for learning shared representations ( Parisotto et al. , 2015 ; Teh et al. , 2017 ; Espeholt et al. , 2018 ; Xu et al. , 2020 ; D ’ Eramo et al. , 2019 ; Sodhani et al. , 2021 ; Stooke et al. , 2021 ) , and sharing data across different tasks ( Eysenbach et al. , 2020 ; Kalashnikov et al. , 2021 ; Yu et al. , 2021a ) . We consider the multi-task offline RL setting and focus on the challenge of sharing data across different tasks . Prior works share data across tasks based on metrics such as learned Q-values ( Eysenbach et al. , 2020 ; Li et al. , 2020 ; Yu et al. , 2021a ) , human domain knowledge ( Kalashnikov et al. , 2021 ) , the distance to the target goals in goal-conditioned settings ( Andrychowicz et al. , 2017 ; Pong et al. , 2018 ; Nair et al. , 2018 ; Liu et al. , 2019 ; Sun et al. , 2019 ; Lin et al. , 2019 ; Huang et al. , 2019 ; Lynch & Sermanet , 2020 ; Yang et al. , 2021 ; Chebotar et al. , 2021 ) , and the learned distance with robust inference in the offline meta-RL setting ( Li et al. , 2019 ) . However , all of these either require access to the functional form of the reward functions of each task in order to relabel the rewards or are limited to goal-conditioned settings . Therefore , they are not applicable to the multi-task offline RL problem that we consider where only the reward label of the originally-executed task is provided . Our work addresses this issue via simply relabeling the data shared from other tasks with a constant value and uses the conservative data sharing strategy ( Yu et al. , 2021a ) to further improve the performance . RL with unlabeled data . Prior works tackle the problem of learning from data without reward labels via either directly imitating expert trajectories ( Pomerleau , 1988 ; Ross & Bagnell , 2012 ; Ho & Ermon , 2016 ) , learning reward functions from expert data using inverse RL ( Abbeel & Ng , 2004 ; Ng & Russell , 2000 ; Ziebart et al. , 2008 ; Finn et al. , 2016a ; Fu et al. , 2018a ; b ; Konyushkova et al. , 2020 ) , or learning a reward / value classifier that discriminates successes and failures ( Xie et al. , 2018 ; Singh et al. , 2019 ; Eysenbach et al. , 2021 ) . These algorithms require online data collection and do not consider the offline RL setting . Singh et al . ( 2020 ) considers the single-task offline setting with both task-specific datasets and task-agnostic prior datasets and relabel the unlabeled prior data as failures since these prior transitions can not solve the task . Our method is not limited to such single-task settings and instead considers the more general multi-task offline RL with data-sharing problem . 3 PRELIMINARIES . Multi-task RL . Standard multi-task RL considers a multi-task Markov decision process ( MDP ) , M = ( S , A , P , γ , { Ri , i } Ni=1 ) , where S and A denote the state and action spaces respectively , P ( s′|s , a ) denotes the dynamics , γ ∈ [ 0 , 1 ) is the discount factor , and R1 , · · · , RN correspond to reward functions of different tasks i ∈ [ N ] for total number of N tasks where [ N ] is the shorthand for { 1 , 2 , . . . , N } . In our setting , we assume a binary per-task Ri ∈ { 0 , 1 } , where 1 denotes success of the task and 0 otherwise . Note that the dynamics are assumed to be the same across all tasks , which is not entirely general but is indeed practical in many problem settings as noted in Yu et al . ( 2021a ) and stands as a common assumption in prior data sharing works ( Yu et al. , 2021a ; Kalashnikov et al. , 2021 ; Eysenbach et al. , 2020 ) . Regardless , there are many practical scenarios with changing rewards and invariant dynamics such as various object manipulation objectives ( Xie et al. , 2018 ) , different goal navigation tasks ( Fu et al. , 2020 ) , and distinct user preferences ( Christiano et al. , 2017 ) . The goal of multi-task RL is to find a task-conditioned policy π ( a|s , i ) that expected return in a multi-task MDP : π∗ ( a|s , · ) : = arg maxπ Ei∼ [ N ] Eπ ( ·|· , i ) [ ∑ t γ tRi ( st , at ) ] . Note that it is possible to model the policies for each task independently as { π1 ( a|s ) , · · · , πN ( a|s ) } without any weight sharing . In our work , we use the single task-conditioned policy to study data sharing and do not consider the weight sharing aspect , which is orthogonal to the focus of the paper , which is also noted in Yu et al . ( 2021a ) . Multi-task offline RL and data sharing . Multi-task offline RL considers the problem of learning the multi-task policy π ( a|s , i ) from a static multi-task dataset with D = ∪Ni=1Di where D〉 = { ( sj , aj , s′j , rj ) } Mj=1 is the per-task dataset . Di is generated by a behavior policy πβ ( a|s ) , without any interaction with the environment . The most straightforward approach to learn π ( a|s , i ) would be train it for task i only using Di . However , sharing data from different tasks to task i has been shown to be conducive in the multi-task offline RL setting ( Kalashnikov et al. , 2021 ; Yu et al. , 2021a ) . To do so , prior works ( Eysenbach et al. , 2021 ; Kalashnikov et al. , 2021 ; Yu et al. , 2021a ) assume access to the functional form of the reward ri , which is a rather strong assumption that is usually impractical to specify in practical applications due to the challenge of reward specification . The next straightforward approach is to naı̈vely sharing data across all tasks , denoted as Sharing All . Formally , Sharing All defines the dataset of transitions relabeled from task j to task i as Dj→i and the method can be then defined as Deffi : = Di ∪ ( ∪j 6=iDj→i ) , where Deffi denotes the effective dataset for task i . While Sharing All improves over not sharing data , as shown in Yu et al . ( 2021a ) , Sharing All leads to distributional shift that could degrade performance in certain situations ( Yu et al. , 2021a ) . In our work , we focus on the CDS ( Yu et al. , 2021a ) , which relabels data that aims to mitigate the distributional shift introduced by sharing other task data . CDS addresses such an issue by proposing a conservative data sharing strategy as follows : Deffi = Di ∪ ( ∪j 6=i { ( sj , aj , s′j , ri ) ∈ Dj→i : ∆π ( s , a ) ≥ 0 } ) , ( 1 ) where sj , aj , s′j denote the transition from Dj , ri denotes the reward of sj , aj , s′j relabeled for task i , π denotes the task-conditioned policy π ( ·|· , i ) , ∆π ( sj , aj ) is the condition that shares data only if the expected Q-value of the relabeled transition exceeds the top k-percentile of the Q-values of the original task data , i.e . ∆π ( s , a ) : = Q̂π ( s , a , i ) − Pk % { Q̂π ( s′ , a′ , i ) : s′ , a′ ∼ Di } . ( 2 ) Beyond controlling the distributional shift introduced in data sharing , multi-task offline RL also needs to address the main challenge in standard offline RL , which is the distributional shift between the learned policy π and the behavior policy πβ . To handle both types of distributional shifts , CDS ( Yu et al. , 2021a ) combines the conservative data sharing and the constrained policy optimization problem and arrives at the following objective : ∀i ∈ [ N ] , π∗ ( a|s , i ) : = arg max π JDeffi ( π ) − αD ( π , π eff β ) , ( 3 ) where πeffβ ( a|s , i ) is the effective behavior policy for task i denoted as πeffβ ( a|s , i ) : = |Deffi ( s , a ) |/|Deffi ( s ) | , JDeffi ( π ) denotes the average return of policy π in the empirical MDP induced by the effective dataset , and D ( π , πeffβ ) denotes a divergence measure ( e.g. , KL-divergence ( Jaques et al. , 2019 ; Wu et al. , 2019 ) , fisher divergence ( Kostrikov et al. , 2021 ) , MMD distance ( Kumar et al. , 2019 ) or DCQL from conservative Q-learning ( Kumar et al. , 2020 ) ) between the learned policy π and the effective behavior policy πeffβ . While optimizing Eq . 3 with Eq . 1 as the data sharing scheme is able to mitigate distributional shift and improve over multi-task offline RL without sharing data and naı̈vely sharing data across all tasks as shown in Yu et al . ( 2021a ) , it requires the assumption of the access to the functional form of the reward functions , which is rather strong and make application of data sharing to real-world applications impractical . We will instead present a simple yet effective data sharing and relabeling scheme in the setting where we do not make such an assumption and instead , only have the reward labels for originally commanded task in the following section .

Summary: This paper proposes a new offline multi-task RL algorithm named conservative unsupervised data sharing (CUDS). Prior works on multi-task RL require the functional form of the reward functions or are limited to goal-conditioned setting. On the other hands, CUDS shares task-agnostic transitions via simply relabeling their rewards as a constant. The authors first prove that $\hat{Q}_\text{UDS}\le \hat{Q}_\text{sharing All}$ for all state-action pairs. Furthermore, they argue that the CUDS can achieve better performance through weighting transitions. Finally, CUDS outperforms or performs competitively with the previous works. Surprisingly, CUDS achieves competitive results compared to CDS, which uses true reward functions for relabeling