Where Off-Policy Deep Reinforcement Learning Fails

1 INTRODUCTION . Learning from only a batch of data without requiring further interactions with the environment is a crucial requirement for scaling reinforcement learning to tasks where the data collection procedure is costly , risky , or time-consuming . Off-policy batch reinforcement learning has important implications in many areas , such as robotics , imitation learning and exploration . In practical applications , it is often preferable for data collection to be performed by some secondary controlled process , such as a human operator or a carefully monitored program . If assumptions on the quality of the behavioral policy can be made , imitation learning can be used to produce good policies . However , imitation learning algorithms , such as behavioral cloning , are known to fail when exposed to noisy or suboptimal trajectories ( Ross et al. , 2011 ) , and more sophisticated approaches are generally unable to surpass the performance of the demonstrator without further interactions with the environment ( Ziebart et al. , 2008 ; Ho & Ermon , 2016 ; Hester et al. , 2017 ) . In most deep reinforcement learning algorithms , a single policy , or variations of a single policy , are used for both exploration and exploitation . This paradigm creates a trade-off between future learning and current performance , and limits the agent to suboptimal policies , unless careful annealing of the exploratory behavior is performed . In contrast , batch reinforcement learning offers a mechanism for learning from a decoupled exploratory process without suffering from degradation in current performance . Most modern off-policy deep reinforcement learning algorithms fall into the category of growing batch learning ( Lange et al. , 2012 ) , in which data is collected and stored into an experience replay dataset ( Lin , 1992 ) , which is used to train the agent before further data collection occurs . However , we find that these off-policy algorithms are generally unsuccessful , unless the available dataset is correlated to the current policy . We demonstrate that this inability to truly learn off-policy is largely due to extrapolation error , a phenomenon in which unseen state-action pairs in the learning target are erroneously estimated to have unrealistic values , which may make them more attractive than observed actions . In other words , the uncertainty of unseen state-action pairs and generalization from function approximation , can cause poor value estimates and suboptimal action selection . To overcome extrapolation error in off-policy learning , we introduce batch-constrained reinforcement learning , in which agents are trained to maximize reward by only selecting previously seen actions . We prove that this paradigm induces desirable properties in the discrete MDP setting , such as converging to the optimal policy without needing to experience all possible transitions . We generalize this notion to continuous control with neural networks by training a state-conditioned generative model to produce only previously seen actions . This generative model is combined with a network which aims to optimally perturb the generated actions in a small range , resulting in the agent using only actions similar to those previously seen . We test our algorithm on several MuJoCo ( Todorov et al. , 2012 ) environments , where extrapolation error is particularly problematic due to the high dimensional continuous action space , which makes it impossible to sample the action space exhaustively . Unlike any previous continuous control reinforcement learning algorithm , our approach is able to learn successfully in a variety of batch reinforcement learning settings . Our algorithm offers a unified view on imitation and off-policy learning , and is capable of learning from purely expert demonstrations , as well as from finite batches of suboptimal data . To ensure reproducibility , we provide precise experimental and implementation details and our code will be made available . 2 BACKGROUND . In reinforcement learning , an agent interacts with its environment , typically assumed to be a Markov decision process ( MDP ) ( S , A , p , r , γ ) , with state space S , action space A transition dynamics p ( s′|s , a ) , for each s′ , s ∈ S and a ∈ A . At each discrete time step , the agent receives a reward r ( s , a ) ∈ R for performing action a in state s. The goal of the agent is to maximize the expectation of the sum of discounted rewards , known as the return Rt = ∑∞ i=t+1 γ ir ( si , ai ) , which weighs future rewards with respect to the discount factor γ ∈ [ 0 , 1 ) . The agent selects actions with respect to a policy π : S → A , which has a corresponding value function Qπ ( s , a ) = Eπ [ Rt|s , a ] , estimating the expected return when following the policy π after taking action a in state s. Given Qπ , a new policy π′ of equal or better performance can be derived by greedy maximization π′ = argmaxaQ π ( s , a ) ( Sutton & Barto , 1998 ) . For a given policy π , the value function can be estimated using sampled versions of the Bellman operator T π : T πQ ( s , a ) = Es′ [ r + γQ ( s′ , π ( s′ ) ) ] . ( 1 ) The Bellman operator is a contraction for γ ∈ [ 0 , 1 ) with unique fixed point Qπ ( s , a ) ( Bertsekas & Tsitsiklis , 1996 ) . Q∗ ( s , a ) = maxπ Qπ ( s , a ) is known as the optimal value function , which has a corresponding optimal policy obtained through greedy action choices . For large or continuous state and action spaces , the value can be approximated with neural networks , e.g . using the DQN algorithm ( Mnih et al. , 2015 ) . In DQN , the value function Qθ is updated using the target : r + γQθ′ ( s ′ , π ( s′ ) ) , π ( s′ ) = argmax a Qθ′ ( s ′ , a ) , ( 2 ) Q-learning is an off-policy algorithm ( Sutton & Barto , 1998 ) , meaning the target can be computed without consideration of how the experience was generated . As a result , off-policy reinforcement learning algorithms are able to learn from data collected by any behavioral policy . Typically , the loss is minimized over mini-batches of tuples of the agent ’ s past data , ( s , a , r , s′ ) ∈ B , sampled from an experience replay dataset B ( Lin , 1992 ) . For shorthand , we often write s ∈ B if there exists a transition tuple containing s in the batch B , and similarly for ( s , a ) ∈ B . To further stabilize learning , a target network with frozen parameters Qθ′ , is used in the learning target . The parameters of the target network θ′ are updated to the current network parameters θ after a fixed number of time steps , or by averaging θ′ ← τθ + ( 1− τ ) θ′ for some small τ ( Lillicrap et al. , 2015 ) . In a continuous action space , the analytic maximum of Equation 2 is intractable . In this case , actor-critic methods are commonly used , where action selection is performed through a separate policy network πφ , known as the actor , and updated with respect to a value estimate , known as the critic ( Sutton & Barto , 1998 ; Konda & Tsitsiklis , 2003 ) . This policy can be updated following the deterministic policy gradient theorem ( Silver et al. , 2014 ) : φ = argmax φ̃ Es∈B [ Qθ ( s , πφ̃ ( s ) ) ] , ( 3 ) which corresponds to learning an approximation to the maximum of Qθ , by propagating the gradient through both πφ and Qθ . When combined with off-policy deep Q-learning to learn Qθ , this algorithm is referred to as Deep Deterministic Policy Gradients ( DDPG ) ( Lillicrap et al. , 2015 ) . 3 EXTRAPOLATION ERROR . Off-policy algorithms commonly assume infinite state-action visitation for convergence or optimality guarantees ( Watkins , 1989 ; Precup et al. , 2001 ) , which is generally impossible to satisfy in any practical problem . In this section , we analyze how breaking this assumption can introduce large amounts of error into the value function by a phenomenon we call extrapolation error . We show how this error can lead to degenerate policies for even provably convergent batch algorithms , such as kernel-based reinforcement learning ( Ormoneit & Sen , 2002 ) . Furthermore , we find that state of the art deep reinforcement learning algorithms , such as DDPG ( Lillicrap et al. , 2015 ) and DQN ( Mnih et al. , 2015 ) , are generally incapable of learning from data uncorrelated to the current policy . Consequently , current deep reinforcement learning algorithms are unsatisfactory solutions for the batch setting , where no assumptions can be made on the data collection process . In Q-learning , an off-policy target r+γQ ( s′ , a′ ) is used to train the current value estimate , where a′ is selected by a policy π ( s′ ) . However , if the state-action pair ( s′ , a′ ) is not contained in the training set , the value Q ( s′ , a′ ) can be unpredictable due to extrapolation from other state-action pairs by function approximation . This means Q ( s′ , a′ ) can take on implausible values due to the lack of data near the state-action pair . Informally , extrapolation error comes from training the value estimate with state-action pairs that are not present in the training data . When combined with maximization in reinforcement learning algorithms , extrapolation error provides a source of noise that can induce a persistent overestimation bias ( Thrun & Schwartz , 1993 ; Van Hasselt et al. , 2016 ; Fujimoto et al. , 2018 ) . In an off-policy or batch setting , this extrapolation error may never be corrected , due to the inability to collect new data , resulting in disastrous value estimation and degenerate policies . 3.1 A SIMPLE EXAMPLE . This problem of extrapolation persists in traditional batch reinforcement learning algorithms , such as kernel-based reinforcement learning ( KBRL ) ( Ormoneit & Sen , 2002 ) . For a given batch B of transitions ( s , a , r , s′ ) , non-negative density function φ : R+ → R+ , hyper-parameter τ ∈ R , and norm || · || , KBRL evaluates the value of a state-action pair ( s , a ) as follows : Q ( s , a ) = ∑ ( saB , a , r , s ′ B ) ∈B κaτ ( s , s a B ) [ r + γV ( s ′ B ) ] , ( 4 ) κaτ ( s , s a B ) = kτ ( s , s a B ) ∑ s̃aB kτ ( s , s̃aB ) , kτ ( s , s a B ) = φ ( ||s− saB|| τ ) , ( 5 ) where states saB ∈ S corresponding to the action a ( sB , a ) ∈ B , and V ( s′B ) = maxa s.t . ( s′B , a ) ∈BQ ( s ′ B , a ) . At each iteration , KBRL updates the estimates of Q ( sB , aB ) for all ( sB , aB ) ∈ B following Equation ( 4 ) , then updates V ( s′B ) by evaluating Q ( s′B , a ) for all sB ∈ B and a ∈ A . Given access to the entire deterministic MDP , KBRL will provable converge to the optimal value , however when limited to only a subset , we find the value estimation susceptible to extrapolation . In Figure 1 , we provide a deterministic two state , two action MDP in which KBRL fails to learn the optimal policy when provided with state-action pairs from the optimal policy . Given the batch { ( s0 , a1 , r = 1 , s1 ) , ( s1 , a0 , r = 0 , s0 ) } , corresponding to the optimal behavior , and noting that there is only one example of each action , Equation ( 4 ) provides the following : Q ( · , a1 ) = 1 + γV ( s1 ) = 1 + γQ ( s1 , a0 ) , Q ( · , a0 ) = γV ( s0 ) = γQ ( s0 , a1 ) . ( 6 ) After sufficient iterations KBRL will converge correctly to Q ( s0 , a1 ) = 11−γ2 , Q ( s1 , a0 ) = γ 1−γ2 . However , when evaluating actions , KBRL erroneously extrapolates the values of each action Q ( · , a1 ) = 11−γ2 , Q ( · , a0 ) = γ 1−γ2 , and its behavior , argmaxaQ ( s , a ) , will result in the degenerate policy of continually selecting a1 . KBRL fails this example by estimating the values of unseen state-action pairs . In methods where the extrapolated estimates can be included into the learning update , such fitted Q-iteration or DQN ( Ernst et al. , 2005 ; Mnih et al. , 2015 ) , this can cause an increasing sequence of value estimates .

Summary: Proposes BCRL for learning from a fixed collection of off-policy experience (I'll call this the "training data"). BCRL attempts to avoid backing up values from states that are not present in the training data, on the assumption that the current estimates of these values are likely to be inaccurate. In the continuous state-action case, this is accomplished by training a generative model to propose, given a state `s`, an action `a` such that a transition similar to `(s, a)` is in the training data. A secondary policy is then trained to perturb the proposed action within a constrained region to maximize value. BCRL outperforms DDPG and DQN when learning from fixed data, but BCRL is slightly worse than behavior cloning at learning to reproduce an expert policy that does not take exploration actions.