Continuous Doubly Constrained Batch Reinforcement Learning

1 Introduction Deep RL algorithms have demonstrated impressive performance in simulable digital environments like video games [ 41 , 54 , 55 ] . In these settings , the agent can execute different policies and observe their performance . Barring a few examples [ 37 ] , advancements have not translated quite as well to real-world environments , where it is typically infeasible to experience millions of environmental interactions [ 11 ] . Moreover , in presence of an acceptable heuristic , it is inappropriate to deploy an agent that learns from scratch hoping that it may eventually outperform the heuristic after sufficient experimentation . The setting of batch or offline RL instead offers a more pertinent framework to learn performant policies for real-world applications [ 34 , 57 ] . Batch RL is widely applicable because this setting does not require that : a proposed policy be tested through real environment interactions , or that data be collected under a particular policy . Instead , the agent only has access to a fixed dataset D collected through actions taken according to some unknown behavior policy πb . The main challenge in this setting is that data may only span a small subset of the possible state-action pairs . Worst yet , the agent can not observe the effects of novel out-of-distribution ( OOD ) state-action combinations that , by definition , are not present in D. A key challenge stems from the inherent uncertainty when learning from limited data [ 28 , 36 ] . Failure to account for this can lead to wild extrapolation [ 17 , 29 ] and over/under-estimation bias in value 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . estimates [ 22 , 23 , 32 , 58 ] . This is a systemic problem that is exacerbated for out-of-distribution ( OOD ) state-actions where data is scarce . Standard temporal difference updates to Q-values rely on the Bellman optimality operator which implies upwardly-extrapolated estimates tend to dominate these updates . As Q-values are updated with overestimated targets , they become upwardly biased even for state-actions well-represented in D. In turn , this can further increase the upper limit of the extrapolation errors at OOD state-actions , which forms a vicious cycle of extrapolation-inflated overestimation ( extra-overestimation for short ) shown in Figure 1 . This extra-overestimation is much more severe than the usual overestimation bias encountered in online RL [ 22 , 58 ] . As such , we critically need to constrain value estimates whenever they lead to situations that look potentially ’ too good to be true ’ , in particular when they occur where a policy might exploit them . Likewise , naive exploration can lead to policies that diverge significantly from πb . This , in turn , leads to even greater estimation error since we have very little data in this un ( der ) -explored space . Note that this is not a reason for particular concern in online RL : after all , once we are done exploring a region of the space that turns out to be less promising than we thought , we simply update the value function and stop visiting or visit rarely . Not so in batch RL where we can not adjust our policy based on observing its actual effects in the environment . These issues are exacerbated for applications with a large number of possible states and actions , such as the continuous settings considered in this work . Since there is no opportunity to try out a proposed policy in batch RL , learning must remain appropriately conservative for the policy to have reasonable effects when it is later actually deployed . Standard regularization techniques are leveraged in supervised learning to address such ill-specified estimation problems , and have been employed in the RL setting as well [ 12 , 50 , 62 ] . This paper adapts standard off-policy actor-critic RL to the batch setting by adding a simple pair of regularizers . In particular , our main contribution is to introduce two novel batch-RL regularizers : The first regularizer combats the extra-overestimation bias in regions that are out-of-distribution . The second regularizer is designed to hedge against the adverse effects of policy updates that severly diverge from πb ( a|s ) . The resultant method , Continuous Doubly Constrained Batch RL ( CDC ) exhibits state-of-the-art performance across 32 continuous control tasks from the D4RL benchmark [ 14 ] demonstrating the usefulness of our regularizers for batch RL . 2 Background . Consider an infinite-horizon Markov Decision Process ( MDP ) [ 48 ] , ( S , A , T , r , µ0 , γ ) . Here S is the state space , A ⊂ Rd is a ( continuous ) action space , T : S × A × S → R+ encodes transition probabilities of the MDP , µ0 denotes the initial state distribution , r ( s , a ) is the instantaneous reward obtained by taking action a ∈ A in state s ∈ S , and γ ∈ [ 0 , 1 ] is a discount factor for future rewards . Given a stochastic policy π ( a|s ) , the sum of discounted rewards generated by taking a series of actions at ∼ π ( ·|st ) corresponds to the return Rπt = ∑∞ i=t γ i−tr ( si , ai ) achieved under policy π . The action-value function ( Q-value for short ) corresponding to π , Qπ ( s , a ) , is defined as the expected return starting at state s , taking a , and acting according to π thereafter , Qπ ( s , a ) = Est∼T , at∼π [ ∑∞ t=0 γ trt | ( s0 , a0 ) = ( s , a ) ] . Qπ ( s , a ) obeys the Bellman equation [ 5 ] : Qπ ( s , a ) = r ( s , a ) + γEs′∼T ( ·|s , a ) , a′∼π ( ·|s′ ) [ Qπ ( s′ , a′ ) ] ( 1 ) Unlike in online RL , no interactions with the environment is allowed here , so the agent does not have the luxury of exploration . D is previously collected via actions taken according to some unknown behavior policy πb ( a|s ) . In this work , we assume D consists of 1-step transition : { ( si , ai , ri , s′i ) } ni=1 where no further sample collection is permitted . In particular , our method , like [ 60 , 63 ] , only needs a dataset consisting of a single-step transitions and does not require complete episode trajectories . This is valuable , for instance , whenever data privacy and sharing restrictions prevent the use of the latter [ 34 ] . It is also useful when combining data from sources where the interaction is still in progress , e.g . from ongoing user interactions . We aim to learn an optimal policy π∗ that maximizes the expected return , denoting the corresponding Q-values for this policy as Q∗ = Qπ ∗ . Q∗ is the fixed point of the Bellman optimality operator [ 5 ] : T Q∗ ( s , a ) = r ( s , a ) + γEs′∼T ( ·|s , a ) [ maxa′ Q∗ ( s′ , a′ ) ] . One way to learn π∗ is via actor-critic methods [ 27 ] , with policy πφ and Q-value Qθ , parametrized by φ and θ respectively . Learning good policies becomes far more difficult in batch RL as it depends on the quality/quantity of available data . Moreover , for continuous control the set of possible actions is infinite , making it nontrivial to find the optimal action even for online RL . One option is to approximate the maximization above by only considering finitely many actions sampled from some π . This leads to the Expected Max-Q ( EMaQ ) operator of Ghasemipour et al . [ 18 ] : T Q ( s , a ) : = r ( s , a ) + γEs′∼T ( ·|s , a ) [ max { a′k } Q ( s′ , a′k ) ] . ( 2 ) Here a′k ∼ πφ ( ·|s′ ) for k = 1 , ... , N , i.e . the candidate actions are drawn IID from the current ( stochastic ) policy rather than over all possible actions . When drawing only a single sample from πφ , this reduces to the standard Bellman operator ( in expectation ) . Conversely , when N →∞ and πφ has support over A , this turns into the Bellman optimality operator . We learn Q by minimizing the standard 1-step temporal difference ( TD ) error . That is , we update at iteration t θt ← argmin θ E ( s , a ) ∼D [ ( Qθ ( s , a ) − T Qθt−1 ( s , a ) ) 2 ] ( 3 ) Throughout , the notation E ( s , a ) ∼D denotes an empirical expectation over dataset D , whereas expectations with respect to π are taken over the true underlying distribution corresponding to policy π . Next , we update the policy by increasing the likelihood of actions with higher Q-values : φt ← argmax φ Es∼D , â∼πφ ( ·|s ) [ Qθt ( s , â ) ] ( 4 ) using off-policy gradient-based updates [ 53 ] . Depending on the context , we omit t from Qθt and πφt . 2.1 Extrapolation-Inflated Overestimation . When our Q-values are estimated via function approximation1 ( with parameters θ ) , the Q-update can be erroneous and noisy [ 58 ] . Let Qθt ( s , a ) denote the estimates of true underlying Q ∗ ( s , a ) values at iteration t of a batch RL algorithm that iterates steps ( 3 ) and ( 4 ) , with πφt denoting the policy that maximizes Qθt . For a proper learning method , we might hope that the estimation error , ER : = Qθt ( s , a ) −Q∗ ( s , a ) , has expected value = 0 and variance σ > 0 for particular states/actions ( the expectation here is over the sampling variability in the dataset D and stochastic updates in our batch RL algorithm ) . However even in this desirable scenario , Jensen ’ s inequality nonetheless implies there will be overestimation error OE : = E [ maxaQθt ( s , a ) ] −maxaQ∗ ( a , s ) ≥ 0 for the actions currently favored by πφt . Here the expectation is over the randomness of the underlying datasetD and the learning algorithm . OE will be strictly positive when the estimation errors are weakly correlated and will grow with the ER-variance σ [ 32 , 58 ] . Under the Bellman optimality or EMaQ operator , these inflated estimates are used as target values in the next Q-update in ( 3 ) , which thus produces a Qθt+1 ( s , a ) estimate that suffers from overestimation bias , meaning it is expected to exceed the true Q value even if this was not the case for initial estimate Qθt ( s , a ) [ 13 , 16 , 22 , 23 , 30 , 33 , 58 ] . In continuous batch RL , ER may have far greater variance ( larger σ ) for OOD states/actions poorly represented in the dataset D , as our function approximator Qθt may wildly extrapolate in these data-scarce regions [ 25 , 34 ] . This in turn implies the updated policy πφt will likely differ significantly from πb and favor some action â = argmaxaQθt ( s , a ) that is OOD [ 17 ] . The estimated value of this OOD action subsequently becomes the target in the Q-update ( 3 ) , and its OE will now be more severe due to the larger σ [ 7 ] . Even though we only apply these Q-updates to non-OOD ( s , a ) ∈ D whose ER may be initially smaller , the severely overestimated target values can induce increased overestimation bias in Qθt+1 ( s , a ) for ( s , a ) ∈ D. In a vicious cycle , the increase in Qθt+1 ( s , a ) for ( s , a ) ∈ D can cause extrapolated Qθt+1 estimates to also grow for OOD actions ( as there is no data to ground these OOD estimates ) , such that overestimation at s , a ∈ D is further amplified through additional temporal difference updates . After many iterative updates , this extra-overestimation can eventually lead to the disturbing explosion of value estimates seen in Figure 1 . Several strategies address overestimation [ 13 , 16 , 22 , 23 , 30 , 33 ] . Fujimoto et al . [ 17 ] proposed a straightforward convex combination of the extremes of an estimated distribution over plausible Q values . Given a set of estimates Qθj for j = 1 , ... M , they combine both the maximum and the minimum value for a given ( s , a ) pair : Qθ ( s , a ) = νmin j Qθj ( s , a ) + ( 1− ν ) max j Qθj ( s , a ) ( 5 ) 1While overestimation bias has been mainly studied in regard to function approximation error [ 13 , 16 , 23 , 33 , 58 ] , Hasselt [ 22 ] shows that overestimation can also arise in tabular MDPs due to noise in the environment . Here ν ∈ ( 0 , 1 ) determines how conservative we wish to be , and the min/max are taken across M Q-networks that only differ in their weight-initialization but are otherwise ( independently ) estimated . For larger ν > 0.5 , Q may be viewed as a lower confidence bound for Q∗ where the epistemic uncertainty in Q estimates is captured via an ensemble of deep Q-networks [ 9 ] .

Summary: This paper presents two constraints for batch reinforcement learning. One is a Q regularizer; the other is a policy regularizer. The Q-value regularizer penalizes overestimation by ensuring out-of-distribution, policy-impacting Q values aren't too much larger than other Q values for a given state. The policy regularizer penalizes exploration by penalizing the divergence between the behavior and target policies.  These changes were tested on a suite of RL benchmarks using both ablation and comparison to other batch RL approaches. The performance of CDC was often superior.