Should I Run Offline Reinforcement Learning or Behavioral Cloning?

1 INTRODUCTION . Offline reinforcement learning ( RL ) algorithms aim to leverage large , existing datasets of previously collected data to produce effective policies that generalize across a wide range of scenarios , without the need for costly active data collection . Many recent offline RL algorithms ( Fujimoto et al. , 2018 ; Kumar et al. , 2019 ; Wu et al. , 2019 ; Kumar et al. , 2020 ; Yu et al. , 2020 ; Sinha et al. , 2021 ; Kostrikov et al. , 2021 ) can work well even highly suboptimal data . With recent advances , the performance of offline RL algorithms has improved significantly , and a number of these approaches have been studied theoretically ( Wang et al. , 2021 ; Zanette , 2020 ; Rashidinejad et al. , 2021 ) . While it is clear that offline RL algorithms are a good choice when the available data is either random or highly suboptimal , it is less clear if these methods should even be tried when the datasets consist of demonstration that come from expert or near-expert demonstrations . In these cases , imitation learning algorithms , such as behavorial cloning ( BC ) , can be used to train policies via supervised learning . It then seems natural to ask : When should we prefer to use offline RL over imitation learning ? To our knowledge , there has not been a rigorous characterization of when offline RL perform better than imitation learning . Existing empirical studies comparing offline RL to imitation learning have mixed conclusions . Some works show that offline RL methods appear to greatly outperform imitation learning , specifically in environments that require “ stitching ” parts of suboptimal trajectories ( Fu et al. , 2020 ) . In contrast , a number of recent works have argued that BC performs better than offline RL on both expert and suboptimal demonstration data over a variety of tasks ( Mandlekar et al. , 2021 ; Florence et al. , 2021 ; Hahn et al. , 2021 ) . This makes it confusing for practitioners to understand whether to use offline RL or simply run BC on collected demonstrations . Thus , in this work we aim to understand if there are conditions on the environment or the dataset under which an offline RL algorithm might outperform BC for a given task , even when BC is provided with expert data or is allowed to use reward as side information . Our insights can inform a practitioner with sufficient domain knowledge on whether offline RL is a good idea , even when expert or near-expert data is available and BC is considered to be a natural choice . Our contribution in this paper is a theoretical and empirical characterization of certain conditions when offline RL can outperform BC . Theoretically , we are the first to derive conditions on the environment or offline dataset where offline RL achieves better worst-case guarantees than even the best-case lower-bound for BC using expert demonstrations . These conditions are grounded in practical problems , and provide guidance to the practitioner as to whether they should use RL or BC . Concretely , we show that in the case of expert data , the error incurred by offline RL algorithms can scale significantly more favorably when the MDP enjoys some structure including horizonindependent returns ( i.e. , sparse rewards ) , or a low volume of states where it is “ critical ” to take the same action as the expert ( Section 4.2 ) Meanwhile , in the case of sufficiently noisy data , we show that offline RL again enjoys better guarantees on long-horizon tasks ( Section 4.3 ) . Finally , since traditional BC methods ignore rewards , we consider generalized BC methods that use the observed rewards to inform learning , and show that it is still preferable to perform offline RL ( Section 4.4 ) . Empirically , we validate our theoretical conclusions on diagnostic gridworld domains ( Fu et al. , 2019 ) and large-scale benchmark problems in robotic manipulation and navigation and Atari games , using human data ( Fu et al. , 2020 ) , scripted data ( Singh et al. , 2020 ) and data generated from RL policies ( Agarwal et al. , 2020b ) . We verify that in multiple long-horizon problems where the conditions we propose are likely to be satisfied , practical offline RL methods can outperform BC and generalized BC methods . Using careful offline tuning practices , we show that it is possible for offline RL to outperform cloning an expert dataset for the same task , given equal amounts of data . 2 RELATED WORK . Offline RL ( Lange et al. , 2012 ; Levine et al. , 2020 ) has shown promise in domains such as robotic manipulation ( Kalashnikov et al. , 2018b ; Mandlekar et al. , 2020 ; Singh et al. , 2020 ; Kalashnikov et al. , 2021 ) , NLP ( Jaques et al. , 2020 ) and healthcare ( Shortreed et al. , 2011 ; Wang et al. , 2018 ) . The major challenge in offline RL is distribution shift ( Fujimoto et al. , 2018 ; Kumar et al. , 2019 ) , where the learned policy might execute out-of-distribution actions . Prior offline RL methods can broadly be characterized into two categories : ( 1 ) policy-constraint methods that regularize the learned policy to be “ close ” to the behavior policy either explicitly ( Fujimoto et al. , 2018 ; Kumar et al. , 2019 ; Liu et al. , 2020 ; Wu et al. , 2019 ; Fujimoto & Gu , 2021 ) or implicitly ( Siegel et al. , 2020 ; Peng et al. , 2019 ; Nair et al. , 2020 ) , or via importance sampling ( Liu et al. , 2019 ; Swaminathan & Joachims , 2015 ; Nachum et al. , 2019 ) , and ( 2 ) conservative methods that learn a lower-bound , or conservative , estimate of return and optimize the policy against it ( Kumar et al. , 2020 ; Kostrikov et al. , 2021 ; Kidambi et al. , 2020 ; Yu et al. , 2020 ; 2021 ) . Our goal is not to devise a new offline RL algorithm , but rather to understand when existing offline RL methods can outperform BC . When do offline RL methods outperform BC ? Rashidinejad et al . ( 2021 ) derive a conservative offline RL algorithm based on lower-confidence bounds ( LCB ) that provably outperforms BC in the simpler contextual bandits ( CB ) setting , but do not extend it to MDPs . While this CB result signals the possibility that offline RL can outperform BC in theory , this generalization is not trivial , as RL suffers from compounding errors ( Munos , 2003 ; 2005 ; Wang et al. , 2021 ) . Laroche et al . ( 2019 ) ; Nadjahi et al . ( 2019 ) ; Kumar et al . ( 2020 ) ; Liu et al . ( 2020 ) ; Xie et al . ( 2021a ) present safe policy improvement bounds expressed as improvements over the behavior policy , which imitation aims to recover , but these bounds do not clearly indicate when offline RL is better or worse . Empirically , Fu et al . ( 2020 ) show that offline RL considerably outperforms BC for tasks that require “ stitching ” trajectory segments to devise an optimal policy . In contrast , Mandlekar et al . ( 2021 ) ; Brandfonbrener et al . ( 2021 ) ; Chen et al . ( 2021 ) ; Hahn et al . ( 2021 ) suggest that BC or filtered BC using the top fraction of the data performs better on other tasks . While results on Adroit domains in D4RL ( Fu et al. , 2020 ) show that offline RL outperforms BC even on expert data , Florence et al . ( 2021 ) reported superior BC results , making it unclear . Kurenkov & Kolesnikov ( 2022 ) emphasize the importance of the online evaluation budget for offline RL methods and show that BC is more favorable in a limited budget . We provide a characterization of certain scenarios where we would expect offline RL to be better than BC , and empirical results verifying that offline RL performs well on such problems , spanning robotics , navigation , games ( Fu et al. , 2020 ; Singh et al. , 2020 ; Bellemare et al. , 2013 ) . Our theoretical analysis combines tools from a number of prior works . We analyze the total error incurred by RL via an error propagation analysis ( Munos , 2003 ; 2005 ; Farahmand et al. , 2010 ; Chen & Jiang , 2019 ; Xie & Jiang , 2020 ; Liu et al. , 2020 ) , which gives rise to bounds with concentrability coefficients that bound the total distributional shift between the learned policy and the data distribution ( Xie & Jiang , 2020 ; Liu et al. , 2020 ) . We use tools from Ren et al . ( 2021 ) , which provide horizon-free bounds for standard ( non-conservative ) offline Q-learning but relax their strict coverage assumptions . While our analysis studies a LCB-style algorithm similar to Rashidinejad et al . ( 2021 ) , we modify it to use tighter Bernstein bonuses ( Zhang et al. , 2021 ; Agarwal et al. , 2020a ) , which is key to improving its suboptimality guarantee . Xie et al . ( 2021b ) consider a similar algorithm with Bernstein bonuses , but take a different approach to analysis it and use it for policy finetuning . 3 PROBLEM SETUP AND PRELIMINARIES . The goal in reinforcement learning is to learn a policy π ( ·|s ) that maximizes the expected cumulative discounted reward in a Markov decision process ( MDP ) , which is defined by a tuple ( S , A , P , r , γ ) . S , A represent state and action spaces , P ( s′|s , a ) and r ( s , a ) represent the dynamics and mean reward function , and γ ∈ ( 0 , 1 ) represents the discount factor . The effective horizon of the MDP is given by H = 1/ ( 1− γ ) . The Q-function , Qπ ( s , a ) for a given policy π is equal to the discounted long-term reward attained by executing a at the state s and then following policy π thereafter . Qπ satisfies the recursion : ∀s , a ∈ S×A , Qπ ( s , a ) = r ( s , a ) +γEs′∼P ( ·|s , a ) , a′∼π ( ·|s′ ) [ Q ( s′ , a′ ) ] . The value function V π considers the expectation of the Q-function over the policy V π ( s ) = Ea∼π ( ·|s ) [ Qπ ( s , a ) ] . Meanwhile , the Q-function of the optimal policy , Q∗ , satisfies the recursion : Q∗ ( s , a ) = r ( s , a ) + Es′∼P ( ·|s , a ) [ maxa′ Q∗ ( s′ , a′ ) ] , and the optimal value function is given by V ∗ ( s ) = maxaQ∗ ( s , a ) . Finally , the expected cumulative discounted reward is given by J ( π ) = Es0∼ρ [ V π ( s0 ) ] . In offline RL , we are provided with a dataset D of transitions , D = { ( si , ai , ri , s′i ) } Ni=1 of size |D| = N . We assume that the dataset D is generated i.i.d . from a distribution µ ( s , a ) that specifies the effective behavior policy πβ ( a|s ) : = µ ( s , a ) / ∑ a µ ( s , a ) . Note that holds even if the data itself is generated by running a non-Markovian policy πβ ( Puterman , 1994 ) . Let n ( s , a ) be the number of times ( s , a ) appear in D , and P̂ ( ·|s , a ) and r̂ ( s , a ) denote the empirical dynamics and reward distributions in D , which may be different from P and r due to stochasticity . Following Rashidinejad et al . ( 2021 ) , the goal is to minimize the suboptimality of the learned policy π̂ : SubOpt ( π̂ ) = ED∼µ [ J ( π∗ ) − J ( π̂ ) ] = ED [ Es0∼ρ [ V ∗ ( s0 ) − V π̂ ( s0 ) ] ] . ( 1 ) Dataset and MDP conditions . Here we introduce some conditions on the offline dataset and MDP structure that we make for our analysis . The first characterizes the distribution shift between the data distribution µ ( s , a ) and the normalized state-action marginal of π∗ , given by d∗ ( s , a ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s , at = a ; π∗ ) , via a concentrability coefficient C∗ . Condition 3.1 ( Rashidinejad et al . ( 2021 ) , Concentrability of the data distribution ) . Define C∗ to be the smallest , finite constant that satisfies : d∗ ( s , a ) /µ ( s , a ) ≤ C∗ ∀s ∈ S , a ∈ A . Intuitively , the coefficient C∗ scales with how suboptimal the data µ ( s , a ) is relative to the optimal π∗ , where C∗ = 1 corresponds to data from π∗ . The next condition we consider is that the discounted return for any trajectory in the MDP is bounded by a constant , which w.l.o.g. , we assume to be 1 . Condition 3.2 ( Ren et al . ( 2021 ) , the value of any trajectory is bounded by 1 ) . The infinite-horizon discounted return for any trajectory τ = ( s0 , a0 , r0 , s1 , · · · ) is bounded as ∑∞ t=0 γ trt ≤ 1 . This condition holds in sparse-reward tasks , particularly those where an agent succeeds or fails at its task once per episode . This is common in domains such as robotics ( Singh et al. , 2020 ; Kalashnikov et al. , 2018b ) and games ( Bellemare et al. , 2013 ) , where the agent receives a signal upon succeeding a task or winning . This condition also appears in prior work deriving suboptimality bounds for RL algorithms ( Ren et al. , 2021 ; Zhang et al. , 2021 ) . Notation . Let n ∧ 1 = max { n , 1 } . Denote ι = polylog ( |S| , H , N ) . We let ι be a polylogarithmic quantity , changing with context . For d-dimensional vectors x , y , x ( i ) denotes its i-th entry , and define V ( x , y ) = ∑ i x ( i ) y ( i ) 2 − ( ∑ i x ( i ) y ( i ) ) 2 .

Summary: Offline RL approaches are of quite great interest because of potentially easier real world applications. But as the paper points out in the extensive related work, there are a lot of conflicting results in the literature when it comes to comparison with plain old behavior cloning. To this end, the paper functions as fairly clear exposition of the current state of offline RL literature and proposes a formalization to allow comparison between these approaches. This formalization leads to a better characterization for the conditions for offline RL methods to outperform behavior cloning (and the importance of planning horizon and critical states).  The paper performs experiments on a wide number of domains which allows the readers to make their own inferences for the conditions for offline RL successes.