Continuous Doubly Constrained Batch Reinforcement Learning

1 Introduction Deep RL algorithms have demonstrated impressive performance in simulable digital environments like video games [ 41 , 54 , 55 ] . In these settings , the agent can execute different policies and observe their performance . Barring a few examples [ 37 ] , advancements have not translated quite as well to real-world environments , where it is typically infeasible to experience millions of environmental interactions [ 11 ] . Moreover , in presence of an acceptable heuristic , it is inappropriate to deploy an agent that learns from scratch hoping that it may eventually outperform the heuristic after sufficient experimentation . The setting of batch or offline RL instead offers a more pertinent framework to learn performant policies for real-world applications [ 34 , 57 ] . Batch RL is widely applicable because this setting does not require that : a proposed policy be tested through real environment interactions , or that data be collected under a particular policy . Instead , the agent only has access to a fixed dataset D collected through actions taken according to some unknown behavior policy πb . The main challenge in this setting is that data may only span a small subset of the possible state-action pairs . Worst yet , the agent can not observe the effects of novel out-of-distribution ( OOD ) state-action combinations that , by definition , are not present in D. A key challenge stems from the inherent uncertainty when learning from limited data [ 28 , 36 ] . Failure to account for this can lead to wild extrapolation [ 17 , 29 ] and over/under-estimation bias in value 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . estimates [ 22 , 23 , 32 , 58 ] . This is a systemic problem that is exacerbated for out-of-distribution ( OOD ) state-actions where data is scarce . Standard temporal difference updates to Q-values rely on the Bellman optimality operator which implies upwardly-extrapolated estimates tend to dominate these updates . As Q-values are updated with overestimated targets , they become upwardly biased even for state-actions well-represented in D. In turn , this can further increase the upper limit of the extrapolation errors at OOD state-actions , which forms a vicious cycle of extrapolation-inflated overestimation ( extra-overestimation for short ) shown in Figure 1 . This extra-overestimation is much more severe than the usual overestimation bias encountered in online RL [ 22 , 58 ] . As such , we critically need to constrain value estimates whenever they lead to situations that look potentially ’ too good to be true ’ , in particular when they occur where a policy might exploit them . Likewise , naive exploration can lead to policies that diverge significantly from πb . This , in turn , leads to even greater estimation error since we have very little data in this un ( der ) -explored space . Note that this is not a reason for particular concern in online RL : after all , once we are done exploring a region of the space that turns out to be less promising than we thought , we simply update the value function and stop visiting or visit rarely . Not so in batch RL where we can not adjust our policy based on observing its actual effects in the environment . These issues are exacerbated for applications with a large number of possible states and actions , such as the continuous settings considered in this work . Since there is no opportunity to try out a proposed policy in batch RL , learning must remain appropriately conservative for the policy to have reasonable effects when it is later actually deployed . Standard regularization techniques are leveraged in supervised learning to address such ill-specified estimation problems , and have been employed in the RL setting as well [ 12 , 50 , 62 ] . This paper adapts standard off-policy actor-critic RL to the batch setting by adding a simple pair of regularizers . In particular , our main contribution is to introduce two novel batch-RL regularizers : The first regularizer combats the extra-overestimation bias in regions that are out-of-distribution . The second regularizer is designed to hedge against the adverse effects of policy updates that severly diverge from πb ( a|s ) . The resultant method , Continuous Doubly Constrained Batch RL ( CDC ) exhibits state-of-the-art performance across 32 continuous control tasks from the D4RL benchmark [ 14 ] demonstrating the usefulness of our regularizers for batch RL . 2 Background . Consider an infinite-horizon Markov Decision Process ( MDP ) [ 48 ] , ( S , A , T , r , µ0 , γ ) . Here S is the state space , A ⊂ Rd is a ( continuous ) action space , T : S × A × S → R+ encodes transition probabilities of the MDP , µ0 denotes the initial state distribution , r ( s , a ) is the instantaneous reward obtained by taking action a ∈ A in state s ∈ S , and γ ∈ [ 0 , 1 ] is a discount factor for future rewards . Given a stochastic policy π ( a|s ) , the sum of discounted rewards generated by taking a series of actions at ∼ π ( ·|st ) corresponds to the return Rπt = ∑∞ i=t γ i−tr ( si , ai ) achieved under policy π . The action-value function ( Q-value for short ) corresponding to π , Qπ ( s , a ) , is defined as the expected return starting at state s , taking a , and acting according to π thereafter , Qπ ( s , a ) = Est∼T , at∼π [ ∑∞ t=0 γ trt | ( s0 , a0 ) = ( s , a ) ] . Qπ ( s , a ) obeys the Bellman equation [ 5 ] : Qπ ( s , a ) = r ( s , a ) + γEs′∼T ( ·|s , a ) , a′∼π ( ·|s′ ) [ Qπ ( s′ , a′ ) ] ( 1 ) Unlike in online RL , no interactions with the environment is allowed here , so the agent does not have the luxury of exploration . D is previously collected via actions taken according to some unknown behavior policy πb ( a|s ) . In this work , we assume D consists of 1-step transition : { ( si , ai , ri , s′i ) } ni=1 where no further sample collection is permitted . In particular , our method , like [ 60 , 63 ] , only needs a dataset consisting of a single-step transitions and does not require complete episode trajectories . This is valuable , for instance , whenever data privacy and sharing restrictions prevent the use of the latter [ 34 ] . It is also useful when combining data from sources where the interaction is still in progress , e.g . from ongoing user interactions . We aim to learn an optimal policy π∗ that maximizes the expected return , denoting the corresponding Q-values for this policy as Q∗ = Qπ ∗ . Q∗ is the fixed point of the Bellman optimality operator [ 5 ] : T Q∗ ( s , a ) = r ( s , a ) + γEs′∼T ( ·|s , a ) [ maxa′ Q∗ ( s′ , a′ ) ] . One way to learn π∗ is via actor-critic methods [ 27 ] , with policy πφ and Q-value Qθ , parametrized by φ and θ respectively . Learning good policies becomes far more difficult in batch RL as it depends on the quality/quantity of available data . Moreover , for continuous control the set of possible actions is infinite , making it nontrivial to find the optimal action even for online RL . One option is to approximate the maximization above by only considering finitely many actions sampled from some π . This leads to the Expected Max-Q ( EMaQ ) operator of Ghasemipour et al . [ 18 ] : T Q ( s , a ) : = r ( s , a ) + γEs′∼T ( ·|s , a ) [ max { a′k } Q ( s′ , a′k ) ] . ( 2 ) Here a′k ∼ πφ ( ·|s′ ) for k = 1 , ... , N , i.e . the candidate actions are drawn IID from the current ( stochastic ) policy rather than over all possible actions . When drawing only a single sample from πφ , this reduces to the standard Bellman operator ( in expectation ) . Conversely , when N →∞ and πφ has support over A , this turns into the Bellman optimality operator . We learn Q by minimizing the standard 1-step temporal difference ( TD ) error . That is , we update at iteration t θt ← argmin θ E ( s , a ) ∼D [ ( Qθ ( s , a ) − T Qθt−1 ( s , a ) ) 2 ] ( 3 ) Throughout , the notation E ( s , a ) ∼D denotes an empirical expectation over dataset D , whereas expectations with respect to π are taken over the true underlying distribution corresponding to policy π . Next , we update the policy by increasing the likelihood of actions with higher Q-values : φt ← argmax φ Es∼D , â∼πφ ( ·|s ) [ Qθt ( s , â ) ] ( 4 ) using off-policy gradient-based updates [ 53 ] . Depending on the context , we omit t from Qθt and πφt . 2.1 Extrapolation-Inflated Overestimation . When our Q-values are estimated via function approximation1 ( with parameters θ ) , the Q-update can be erroneous and noisy [ 58 ] . Let Qθt ( s , a ) denote the estimates of true underlying Q ∗ ( s , a ) values at iteration t of a batch RL algorithm that iterates steps ( 3 ) and ( 4 ) , with πφt denoting the policy that maximizes Qθt . For a proper learning method , we might hope that the estimation error , ER : = Qθt ( s , a ) −Q∗ ( s , a ) , has expected value = 0 and variance σ > 0 for particular states/actions ( the expectation here is over the sampling variability in the dataset D and stochastic updates in our batch RL algorithm ) . However even in this desirable scenario , Jensen ’ s inequality nonetheless implies there will be overestimation error OE : = E [ maxaQθt ( s , a ) ] −maxaQ∗ ( a , s ) ≥ 0 for the actions currently favored by πφt . Here the expectation is over the randomness of the underlying datasetD and the learning algorithm . OE will be strictly positive when the estimation errors are weakly correlated and will grow with the ER-variance σ [ 32 , 58 ] . Under the Bellman optimality or EMaQ operator , these inflated estimates are used as target values in the next Q-update in ( 3 ) , which thus produces a Qθt+1 ( s , a ) estimate that suffers from overestimation bias , meaning it is expected to exceed the true Q value even if this was not the case for initial estimate Qθt ( s , a ) [ 13 , 16 , 22 , 23 , 30 , 33 , 58 ] . In continuous batch RL , ER may have far greater variance ( larger σ ) for OOD states/actions poorly represented in the dataset D , as our function approximator Qθt may wildly extrapolate in these data-scarce regions [ 25 , 34 ] . This in turn implies the updated policy πφt will likely differ significantly from πb and favor some action â = argmaxaQθt ( s , a ) that is OOD [ 17 ] . The estimated value of this OOD action subsequently becomes the target in the Q-update ( 3 ) , and its OE will now be more severe due to the larger σ [ 7 ] . Even though we only apply these Q-updates to non-OOD ( s , a ) ∈ D whose ER may be initially smaller , the severely overestimated target values can induce increased overestimation bias in Qθt+1 ( s , a ) for ( s , a ) ∈ D. In a vicious cycle , the increase in Qθt+1 ( s , a ) for ( s , a ) ∈ D can cause extrapolated Qθt+1 estimates to also grow for OOD actions ( as there is no data to ground these OOD estimates ) , such that overestimation at s , a ∈ D is further amplified through additional temporal difference updates . After many iterative updates , this extra-overestimation can eventually lead to the disturbing explosion of value estimates seen in Figure 1 . Several strategies address overestimation [ 13 , 16 , 22 , 23 , 30 , 33 ] . Fujimoto et al . [ 17 ] proposed a straightforward convex combination of the extremes of an estimated distribution over plausible Q values . Given a set of estimates Qθj for j = 1 , ... M , they combine both the maximum and the minimum value for a given ( s , a ) pair : Qθ ( s , a ) = νmin j Qθj ( s , a ) + ( 1− ν ) max j Qθj ( s , a ) ( 5 ) 1While overestimation bias has been mainly studied in regard to function approximation error [ 13 , 16 , 23 , 33 , 58 ] , Hasselt [ 22 ] shows that overestimation can also arise in tabular MDPs due to noise in the environment . Here ν ∈ ( 0 , 1 ) determines how conservative we wish to be , and the min/max are taken across M Q-networks that only differ in their weight-initialization but are otherwise ( independently ) estimated . For larger ν > 0.5 , Q may be viewed as a lower confidence bound for Q∗ where the epistemic uncertainty in Q estimates is captured via an ensemble of deep Q-networks [ 9 ] .

Summary: The authors propose Continuous Doubly Constrained (CDC) Batch RL. It adds two straightforward penalties, a policy constraint and a value-constraint. The policy constraint prevent the policy from taking actions that are not supported by the dataset, and the value-constraint prevents the value from being overly optimistic. Both penalties improve performance on the offline RL benchmarks compared to the baseline, but the two penalties combined work substantially better. They also perform competitively against several other offline RL baseline agents (BC, BCQ, BEAR, BRAC-V, BRAC-P, CQL).