The Essential Elements of Offline RL via Supervised Learning

1 INTRODUCTION . Offline and off-policy reinforcement learning ( RL ) is typically addressed using value-based methods . While conceptually simple , such methods can be difficult to apply in practice and tend to require a complex tricks to stabilize learning and delicate tuning of many hyperparameters . Recent work has explored an appealing alternative approach : convert the RL problem into a conditional , filtered , or weighted imitation learning problem . Such methods typically use a simple insight : experience that may be suboptimal for one task may be optimal for another task . By conditioning on some piece of information , such as a goal , reward function parameterization , or reward value , such experience can be used for simple behavior cloning ( Codevilla et al. , 2018 ; Lynch et al. , 2020 ; Ding et al. , 2019 ; Srivastava et al. , 2019 ; Ghosh et al. , 2018 ; Peng et al. , 2019 ; Neumann et al. , 2009 ; Kumar et al. , 2019 ; Eysenbach et al. , 2020 ; Chen et al. , 2021 ; Janner et al. , 2021 ) . We will refer to this set of approaches as RL VIA SUPERVISED LEARNING ( RVS ) . These approaches most commonly conditioning on goals ( Lynch et al. , 2020 ; Codevilla et al. , 2018 ; Ghosh et al. , 2021 ) or reward values ( Kumar et al. , 2019 ; Srivastava et al. , 2019 ; Chen et al. , 2021 ) , though they can also involve reweighting or filtering ( Eysenbach et al. , 2020 ; Neumann et al. , 2009 ; Peng et al. , 2019 ; Kumar et al. , 2019 ) . RvS methods are appealing because of their algorithmic simplicity . However , prior works have put forward conflicting hypotheses about which factors are essential for their good performance , including online data ( Ghosh et al. , 2021 ) , weighting ( Kumar et al. , 2019 ) , or large Transformer sequence models ( Chen et al. , 2021 ) . It remains unclear where , how , and why they work well . Therefore , the first and most important question we aim to study is : what elements are essential for effective RvS learning ? Answering this question may help users understand when RvS methods are a generally-applicable replacement for dynamic programming methods . Beyond this , it also remains unclear how or when such methods can effectively exploit temporal compositionality . Dynamic programming methods can make use of this effectively , and it is important for offline RL tasks where few near-optimal trajectories are present in the dataset , as for example in the Ant Maze tasks in D4RL ( Fu et al. , 2020 ) . Therefore , the second question we aim to answer is : what are the limits of RvS learning , and does it scale effectively to settings that contain few near-optimal trajectories ? Our focus is not on proposing new RvS algorithms , but rather on empirically investigating the design decisions , performance , and limits of such methods . Prior works have proposed a number of factors for improving the performance of RvS learning methods , such as filtering or weighting ( Kumar et al. , 2019 ) , high-coverage “ play ” data ( Lynch et al. , 2020 ) , online collection ( Ghosh et al. , 2021 ) , or the use of large Transformer models ( Chen et al. , 2021 ; Janner et al. , 2021 ) . Our first result is that , based mostly on careful adjustment of model capacity ( e.g. , via regularization ) , simple feedforward models can match the performance of more complex and computationally expensive sequence models from prior work across a wide range of tasks . Our second result is that choosing to condition on reward values versus goals can have a large effect on performance , with different choices working better in different domains . The main contribution of this paper is a study of RvS methods on a wide range of offline RL problems , as well as a set of analyses about which design factors matter most for such methods . We show that relatively simple network designs can , in many cases , perform as well ( or better than ) both state-of-the-art offline RL algorithms and more complex architectures , if tuned correctly to have the right capacity . While this may be unsurprising , it contradicts hypotheses put forward in several prior works ( Kumar et al. , 2019 ; Chen et al. , 2021 ) . Additionally , to our knowledge the results that we demonstrate exceed , on average , previously reported results for any RvS method , when considered across the full range of tasks . We believe that these findings will be useful to the RL community , because they help to understand the essential ingredients and limitations of RvS methods , and provide a foundation for future work on simple and performant offline RL algorithms . 2 RELATED WORK . Recent offline RL methods use many algorithmic techniques , including value functions ( Kumar et al. , 2020 ; Fujimoto et al. , 2019 ) , dynamics estimation alongisde value functions ( Yu et al. , 2020 ; Shen et al. , 2021 ; Kidambi et al. , 2020 ) , dynamics estimation alone ( Depeweg et al. , 2017 ; Argenson & Dulac-Arnold , 2020 ; Swazinna et al. , 2021 ) , and uncertainty quantification ( Agarwal et al. , 2020 ; Wu et al. , 2021 ; Yu et al. , 2020 ) . In this paper , we focus on a class of offline RL methods based on conditional behavior cloning that avoid value functions entirely . The most common instantiation of these methods are goal-conditioned behavior cloning ( Ghosh et al. , 2021 ; Lynch et al. , 2020 ; Ding et al. , 2019 ) and reward-conditioned behavior cloning ( Kumar et al. , 2019 ; Srivastava et al. , 2019 ; Schmidhuber , 2019 ) . Prior work has also looked at conditioning on different information , such as entire trajectories ( Chen et al. , 2021 ; Janner et al. , 2021 ) , tasks inferred by inverse RL ( Eysenbach et al. , 2020 ) , and other task information ( Codevilla et al. , 2018 ; Hawke et al. , 2020 ) . These methods all eschew value estimation and instead solve a supervised regression problem that resembles behavioral cloning . While these methods can , in theory , be directly applied to the offline RL setting , some prior works only evaluate these methods with combined with iterative data collection ( Srivastava et al. , 2019 ; Ghosh et al. , 2021 ) , which is not permitted in the typical offline RL setting . Kumar et al . ( 2019 ) study reward-conditioned policies and do present experiments in the offline RL setting . However , in contrast with our work , the results from Kumar et al . ( 2019 ) suggest that good performance can not be achieved through RvS learning but only through additional advantageweighting ( Neumann et al. , 2009 ; Peng et al. , 2019 ) . More recent work ( Chen et al. , 2021 ) presents a conditional imitation learning method for the offline RL setting that avoids advantage weighting by introducing a higher capacity , autoregressive Transformer policy . In contrast to these prior works , we find that suitable design decisions in training enable simple conditioning with standard feedforward networks to attain state-of-the-art results across a range of offline RL benchmarks , without advantage weighting or high-capacity sequence models . 3 REINFORCEMENT LEARNING VIA SUPERVISED LEARNING . Algorithm 1 RvS-Learning 1 : Input : Dataset of trajectories , D = { τ } 2 : Initialize policy π ( a | s , ω ) . 3 : while not converged do 4 : Randomly sample trajectories : τ ∼ D. 5 : Sample time index for each trajetory , t ∼ [ 1 , H ] , and sample a corresponding outcome : ω ∼ f ( τt : H ) . 6 : Compute loss : L ( ω ) ← ∑ ( st , at , ω ) log πθ ( at | st , ω ) 7 : Update policy parameters : ω ← ω + η∇θL ( θ ) 8 : end while 9 : return Conditional policy π ( a | s , ω ) In this section , we will describe a formulation of reinforcement learning via supervised learning . While we will refer to these methods as “ RvS learning ” methods , we emphasize that we are not proposing any new algorithm , but rather summarizing existing methods under one common framework . In our experiments section , we will discuss the specific design decisions that we find important to make such methods work well . To formulate RvS methods in a general way , we assume an agent interacts with a Markov decision process with states st , actions at , initial state distribution ps ( s1 ) , and dynamics p ( st+1 | st , at ) . We assume that the agent chooses actions using a policy πθ ( at | st ) , parameterized by θ . For simplicity , we assume episodes have fixed length H , and use τ = ( s1 , a1 , r1 , s2 , a2 , r2 , · · · ) to denote a trajectory of experience . As some trajectories might be described using multiple outcomes , we will use f ( ω | τ ) to denote the distribution over outcomes that occur in this trajectory . We will study two types of outcomes . The first type of outcome is a state ( ω ∈ S ) that the agent visits in the future : f ( ω | τt : H ) = Unif ( st+1 , st+2 , · · · , sH ) . We will refer to RvS methods that learn goal-conditioned policies as goal-conditioned behavior cloning ( GCBC ) . The second type of outcome is the average return ( ω ∈ R ) achieved over some future number of time steps : f ( ω | τt : H ) = 1 ( ω = 1H−t+1 ∑H t′=t r ( st′ , at′ ) ) . We will refer to RvS methods that learn rewardconditioned policies as reward-conditioned behavior cloning ( RCBC ) . It is also possible to condition on other information , such as the parameters of a reward function inferred via inverse RL ( Eysenbach et al. , 2020 ) or the parameters of a task , such as turning left or right ( Codevilla et al. , 2018 ) . We will focus on RvS methods applied to the offline RL setting . These methods take as input a dataset of experience , D = { τ } , and output an outcome-conditioned policy , πθ ( at | st , ω ) , which as in ( Ghosh et al. , 2021 ) is the solution to the following optimization problem : max θ 1 |D|H For all trajectories : ∑ τ∈D For all states in that trajectory : ∑ 1≤t≤|τ | E For all achieved outcomes : ω∼f ( ω|τt : H ) [ log πθ ( at | st , ω ) ] . ( 1 ) While the basic formulation of this conditional policy is simple , instantiating RvS methods that attain excellent results has proven challenging ( Kumar et al. , 2019 ; Srivastava et al. , 2019 ) . In the remainder of this paper , we will present an empirical analysis of the design choices in RvS methods . Specifically , we will aim to understand which design choices are needed to make RvS methods perform well on benchmark tasks , including how two different choices for ω ( goal and reward values ) compare in practice across diverse tasks and datasets with varying levels of optimality ( random , mixed , and expert ) . We will first describe our tasks and datasets in Section 4 , discuss various design choices in Section 5 , and then present comparisons and analysis in Section 6 .

Summary: This paper considers design choices involved in offline RL approaches that consider a reduction to weighted/conditional behavior cloning, a class of approaches referred to as Reinforcement Learning via Supervised Learning (RvS). The paper studies various issues including expressivity of policy architecture, regularization, choice of conditioning variables (e.g. based on goals or rewards).   At a high level, the takeaways are (a) RvS approaches are successful when using neural networks with appropriate capacity and regularization, (b) with appropriate conditioning variables (goal/reward based conditioning), (c) RvS can obtain policies that exhibit compositional behavior by conditioning on appropriate events.