Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble

1 Introduction . Over the recent years , deep reinforcement learning ( deep RL ) has achieved considerable success in various domains such as robotics [ 20 ] , recommendation systems [ 6 ] , and strategy games [ 26 ] . However , a major drawback of RL algorithms is that they adopt an active learning procedure , where training steps require active interactions with the environment . This trial-and-error procedure can be prohibitive when scaling RL to real-world applications such as autonomous driving and healthcare , as exploratory actions can cause critical damage to the agent or the environment [ 19 ] . Offline RL , also known as batch RL , aims to overcome this problem by learning policies using only previously collected data without further interactions with the environment [ 2 , 11 , 19 ] . Even though offline RL is a promising direction to lead a more data-driven way of solving RL problems , recent works show offline RL faces new algorithmic challenges [ 19 ] . Typically , if the coverage of the dataset is not sufficient , vanilla RL algorithms suffer severely from extrapolation ∗First two authors have equal contributions †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . error , overestimating the Q-values of out-of-distribution ( OOD ) state-action pairs [ 15 ] . To this end , most offline RL methods apply some constraints or penalty terms on top of the existing RL algorithms to enforce the learning process to be more conservative . For example , some prior works explicitly regularize the policy to be close to the behavior policy that was used to collect the data [ 11 , 15 ] . A more recent work instead penalizes the Q-values of OOD state-action pairs to enforce the Q-values to be more pessimistic [ 16 ] . While these methods achieve significant performance gains over vanilla RL methods , they either require an estimation of the behavior policy or explicit sampling from OOD data points , which themselves can be non-trivial to solve . Furthermore , these methods do not utilize the generalization ability of the Q-function networks and prohibit the agent from approaching any OOD state-actions without any consideration on whether they are good or bad . However , if we can identify OOD data points where we can predict their Q-values with high confidence , it is more effective not to restrain the agent from choosing those data points . From this intuition , we propose an uncertainty-based model-free offline RL method that effectively quantifies the uncertainty of the Q-value estimates by an ensemble of Q-function networks and does not require any estimation or sampling of the data distribution . To achieve this , we first show that a well-known technique from online RL , the clipped Q-learning [ 10 ] , can be successfully leveraged as an uncertainty-based penalization term . Our experiments reveal that we can achieve state-of-the-art performance on various offline RL tasks by solely using this technique with increased ensemble size . To further improve the practical usability of the method , we develop an ensemble diversifying objective that significantly reduces the number of required ensemble networks . We evaluate our proposed method on D4RL benchmarks [ 9 ] and verify that the proposed method outperforms the previous state-of-the-art by a large margin on various types of environments and datasets . 2 Preliminaries . We consider an environment formulated as a Markov Decision Process ( MDP ) defined by a tuple ( S , A , T , r , d0 , γ ) , where S is the state space , A is the action space , T ( s′ | s , a ) is the transition probability distribution , r : S × A −→ R is the reward function , d0 is the initial state distribution , and γ ∈ ( 0 , 1 ] is the discount factor . The goal of reinforcement learning is to find an optimal policy π ( a | s ) that maximizes the cumulative discounted reward Est , at [ ∑∞ t=0 γ tr ( st , at ) ] , where s0 ∼ d0 ( · ) , at ∼ π ( · | st ) , and st+1 ∼ T ( · | st , at ) . One of the major approaches for obtaining such a policy is Q-learning [ 12 , 20 ] which learns a state-action value function Qφ ( s , a ) parameterized by a neural network that represents the expected cumulative discounted reward when starting from state s and action a . Standard actor-critic approach [ 14 ] learns this Q-function by minimizing the Bellman residual ( Qφ ( s , a ) − BπθQφ ( s , a ) ) 2 , where BπθQφ ( s , a ) = Es′∼T ( ·|s , a ) [ r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) Qφ ( s′ , a′ ) ] is the Bellman operator . In the context of offline RL , where transitions are sampled from a static dataset D , the objective for the Q-network becomes minimizing Jq ( Qφ ) : = E ( s , a , s′ ) ∼D [ ( Qφ ( s , a ) − ( r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) [ Qφ′ ( s ′ , a′ ) ] ) ) 2 ] , ( 1 ) where Qφ′ represents the target Q-network softly updated for algorithmic stability [ 20 ] . The policy , which is also parameterized by a neural network , is updated in an alternating fashion to maximize the expected Q-value : Jp ( πθ ) : = Es∼D , a∼πθ ( ·|s ) [ Qφ ( s , a ) ] . However , as the policy is updated to maximize the Q-values , the actions a′ sampled from the current policy in Equation ( 1 ) can be biased towards OOD actions with erroneously high Q-values . In the offline RL setting , such errors can not be corrected by feedback from the environment as in online RL . To handle the error propagation from these OOD actions , most offline RL algorithms regularize either the policy [ 11 , 15 ] or the Q-function [ 16 ] to be biased towards the given dataset . However , the policy regularization methods typically require an accurate estimation of the behavior policy . The previous state-of-the-art method CQL [ 16 ] instead learns conservative Q-values without estimating the behavior policy by penalizing the Q-values of OOD actions by min φ Jq ( Qφ ) + α ( Es∼D , a∼µ ( ·|s ) [ Qφ ( s , a ) ] − E ( s , a ) ∼D [ Qφ ( s , a ) ] ) , where µ is an approximation of the policy that maximizes the current Q-function . While CQL does not need explicit behavior policy estimation , it requires sampling from an appropriate action distribution µ ( · | s ) . 3 Uncertainty penalization with Q-ensemble . In this section , we turn our attention to a conventional technique from online RL , Clipped Double QLearning [ 10 ] , which uses the minimum value of two parallel Q-networks as the Bellman target : y = r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) [ minj=1,2Qφ′j ( s ′ , a′ ) ] . Although this technique was originally proposed in online RL to mitigate the overestimation from general prediction errors , some offline RL algorithms [ 11 , 15 , 28 ] also utilize this technique to enforce their Q-value estimates to be more pessimistic . However , the isolated effect of the clipped Q-learning in offline RL was not fully analyzed in the previous works , as they use the technique only as an auxiliary term that adds up to their core methods . To examine the ability of clipped Q-learning to prevent the overestimation in offline RL on its own , we modify SAC [ 12 ] by increasing the number of Q-ensembles from 2 to N : min φi Es , a , s′∼D [ ( Qφi ( s , a ) − ( r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) [ min j=1 , ... , N Qφ′j ( s′ , a′ ) − β log πθ ( a′ | s′ ) ] ) ) 2 ] max θ Es∼D , a∼πθ ( ·|s ) [ min j=1 , ... , N Qφj ( s , a ) − β log πθ ( a | s ) ] , ( 2 ) for i = 1 , . . . , N . We denote this modified algorithm as SAC-N . Figure 1 shows the preliminary experiments on D4RL halfcheetah-medium and hopper-medium datasets [ 9 ] while varying N . Note that these datasets are constructed from suboptimal behavior policies . Surprisingly , as we gradually increase N , we can successfully find policies that outperform the previous state-of-the-art method ( CQL ) by a large margin . In fact , as we will present in Section 5 , SAC-N outperforms CQL on various types of environments and data-collection policies . To understand why this simple technique works so well , we can first interpret the clipping procedure ( choosing the minimum value from the ensemble ) as penalizing state-action pairs with high-variance Q-value estimates , which encourages the policy to favor actions that appeared in the dataset [ 11 ] . The dataset samples will naturally have lower variance compared to the OOD samples as the Bellman residual term in Equation ( 2 ) explicitly aligns the Q-value predictions for the dataset samples . More formally , we can regard this difference in variance as accounting for epistemic uncertainty [ 8 ] which refers to the uncertainty stemming from limited data and knowledge . Utilization of the clipped Q-value relates to methods that consider the confidence bound of the Q-value estimates [ 24 ] . Online RL methods typically utilize the Q-ensemble to form an optimistic estimate of the Q-value , by adding the standard deviation to the mean of the Q-ensembles [ 18 ] . This optimistic Q-value , also known as the upper-confidence bound ( UCB ) , can encourage the exploration of unseen actions with high uncertainty . However , in offline RL , the dataset available during training is fixed , and we have to focus on exploiting the given data . For this purpose , it is natural to utilize the lower-confidence bound ( LCB ) of the Q-value estimates , for example by subtracting the standard deviation from the mean , which allows us to avoid risky state-actions . The clipped Q-learning algorithm , which chooses the worst-case Q-value instead to compute the pessimistic estimate , can also be interpreted as utilizing the LCB of the Q-value predictions . Suppose Q ( s , a ) follows a Gaussian distribution with mean m ( s , a ) and standard deviation σ ( s , a ) . Also , let { Qj ( s , a ) } Nj=1 be realizations of Q ( s , a ) . Then , we can approximate the expected minimum of the realizations following the work of Royston [ 23 ] as E [ min j=1 , ... , N Qj ( s , a ) ] ≈ m ( s , a ) − Φ−1 ( N − π8 N − π4 + 1 ) σ ( s , a ) , ( 3 ) where Φ is the CDF of the standard Gaussian distribution . This relation indicates that using the clipped Q-value is similar to penalizing the ensemble mean of the Q-values with the standard deviation scaled by a coefficient dependent on N . We now move on to the empirical analysis of the clipped Q-learning . Figure 2a compares the strength of the uncertainty penalty on in-distribution and OOD actions . Specifically , we compare actions sampled from two types of policies : ( 1 ) the behavior policy which was used to collect the dataset , and ( 2 ) the random policy which samples actions uniformly from the action space . For each policy , we measure the size of the penalty from the clipping as Es∼D , a∼π ( ·|s ) [ 1N ∑N j=1Qφj ( s , a ) − minj=1 , ... , N Qφj ( s , a ) ] . Figure 2a shows that the clipping term penalizes the random state-action pairs much stronger than the in-distribution pairs throughout the training . For comparison , we also measure the standard deviation of the Q-values for each policy . The results in Figure 2b show that as we conjectured , the Q-value predictions for the OOD actions have a higher variance . We also find that the size of the penalty and the standard deviation are highly correlated , as we noted in Equation ( 3 ) . As we observe that OOD actions have higher variance on Q-value estimates , the effect of increasing N becomes obvious : it strengthens the penalty applied to the OOD samples compared to the dataset samples . To verify this , we measured the relative penalty applied to the OOD samples in Figure 2c and found that indeed the OOD samples are penalized relatively further as N increases .

Summary: In this work the authors propose a new algorithm for the problem of offline RL. One of the main difficulties in offline RL is estimating the Q-values of Out-Of-Distribution action state pairs.  To overcome this, they authors propose a new actor critic method, where an ensemble of Q values for the critic is trained  using clipped Q-learning. They show that this can be used to penalize OOD data  with high prediction uncertainties. The resulting algorithm achieves state-of-the-art performance in  a range of D4RL benchmarks. Furthermore, in order to reduce the size of the ensemble required to achieve good performance they propose a modification on the critic loss. They show that the resulting algorithm called Ensemble-Diversified Actor-Critic can match the performance of the naive approach while using only one tenth of the number of ensemble Q values.