Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble

1 Introduction . Over the recent years , deep reinforcement learning ( deep RL ) has achieved considerable success in various domains such as robotics [ 20 ] , recommendation systems [ 6 ] , and strategy games [ 26 ] . However , a major drawback of RL algorithms is that they adopt an active learning procedure , where training steps require active interactions with the environment . This trial-and-error procedure can be prohibitive when scaling RL to real-world applications such as autonomous driving and healthcare , as exploratory actions can cause critical damage to the agent or the environment [ 19 ] . Offline RL , also known as batch RL , aims to overcome this problem by learning policies using only previously collected data without further interactions with the environment [ 2 , 11 , 19 ] . Even though offline RL is a promising direction to lead a more data-driven way of solving RL problems , recent works show offline RL faces new algorithmic challenges [ 19 ] . Typically , if the coverage of the dataset is not sufficient , vanilla RL algorithms suffer severely from extrapolation ∗First two authors have equal contributions †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . error , overestimating the Q-values of out-of-distribution ( OOD ) state-action pairs [ 15 ] . To this end , most offline RL methods apply some constraints or penalty terms on top of the existing RL algorithms to enforce the learning process to be more conservative . For example , some prior works explicitly regularize the policy to be close to the behavior policy that was used to collect the data [ 11 , 15 ] . A more recent work instead penalizes the Q-values of OOD state-action pairs to enforce the Q-values to be more pessimistic [ 16 ] . While these methods achieve significant performance gains over vanilla RL methods , they either require an estimation of the behavior policy or explicit sampling from OOD data points , which themselves can be non-trivial to solve . Furthermore , these methods do not utilize the generalization ability of the Q-function networks and prohibit the agent from approaching any OOD state-actions without any consideration on whether they are good or bad . However , if we can identify OOD data points where we can predict their Q-values with high confidence , it is more effective not to restrain the agent from choosing those data points . From this intuition , we propose an uncertainty-based model-free offline RL method that effectively quantifies the uncertainty of the Q-value estimates by an ensemble of Q-function networks and does not require any estimation or sampling of the data distribution . To achieve this , we first show that a well-known technique from online RL , the clipped Q-learning [ 10 ] , can be successfully leveraged as an uncertainty-based penalization term . Our experiments reveal that we can achieve state-of-the-art performance on various offline RL tasks by solely using this technique with increased ensemble size . To further improve the practical usability of the method , we develop an ensemble diversifying objective that significantly reduces the number of required ensemble networks . We evaluate our proposed method on D4RL benchmarks [ 9 ] and verify that the proposed method outperforms the previous state-of-the-art by a large margin on various types of environments and datasets . 2 Preliminaries . We consider an environment formulated as a Markov Decision Process ( MDP ) defined by a tuple ( S , A , T , r , d0 , γ ) , where S is the state space , A is the action space , T ( s′ | s , a ) is the transition probability distribution , r : S × A −→ R is the reward function , d0 is the initial state distribution , and γ ∈ ( 0 , 1 ] is the discount factor . The goal of reinforcement learning is to find an optimal policy π ( a | s ) that maximizes the cumulative discounted reward Est , at [ ∑∞ t=0 γ tr ( st , at ) ] , where s0 ∼ d0 ( · ) , at ∼ π ( · | st ) , and st+1 ∼ T ( · | st , at ) . One of the major approaches for obtaining such a policy is Q-learning [ 12 , 20 ] which learns a state-action value function Qφ ( s , a ) parameterized by a neural network that represents the expected cumulative discounted reward when starting from state s and action a . Standard actor-critic approach [ 14 ] learns this Q-function by minimizing the Bellman residual ( Qφ ( s , a ) − BπθQφ ( s , a ) ) 2 , where BπθQφ ( s , a ) = Es′∼T ( ·|s , a ) [ r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) Qφ ( s′ , a′ ) ] is the Bellman operator . In the context of offline RL , where transitions are sampled from a static dataset D , the objective for the Q-network becomes minimizing Jq ( Qφ ) : = E ( s , a , s′ ) ∼D [ ( Qφ ( s , a ) − ( r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) [ Qφ′ ( s ′ , a′ ) ] ) ) 2 ] , ( 1 ) where Qφ′ represents the target Q-network softly updated for algorithmic stability [ 20 ] . The policy , which is also parameterized by a neural network , is updated in an alternating fashion to maximize the expected Q-value : Jp ( πθ ) : = Es∼D , a∼πθ ( ·|s ) [ Qφ ( s , a ) ] . However , as the policy is updated to maximize the Q-values , the actions a′ sampled from the current policy in Equation ( 1 ) can be biased towards OOD actions with erroneously high Q-values . In the offline RL setting , such errors can not be corrected by feedback from the environment as in online RL . To handle the error propagation from these OOD actions , most offline RL algorithms regularize either the policy [ 11 , 15 ] or the Q-function [ 16 ] to be biased towards the given dataset . However , the policy regularization methods typically require an accurate estimation of the behavior policy . The previous state-of-the-art method CQL [ 16 ] instead learns conservative Q-values without estimating the behavior policy by penalizing the Q-values of OOD actions by min φ Jq ( Qφ ) + α ( Es∼D , a∼µ ( ·|s ) [ Qφ ( s , a ) ] − E ( s , a ) ∼D [ Qφ ( s , a ) ] ) , where µ is an approximation of the policy that maximizes the current Q-function . While CQL does not need explicit behavior policy estimation , it requires sampling from an appropriate action distribution µ ( · | s ) . 3 Uncertainty penalization with Q-ensemble . In this section , we turn our attention to a conventional technique from online RL , Clipped Double QLearning [ 10 ] , which uses the minimum value of two parallel Q-networks as the Bellman target : y = r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) [ minj=1,2Qφ′j ( s ′ , a′ ) ] . Although this technique was originally proposed in online RL to mitigate the overestimation from general prediction errors , some offline RL algorithms [ 11 , 15 , 28 ] also utilize this technique to enforce their Q-value estimates to be more pessimistic . However , the isolated effect of the clipped Q-learning in offline RL was not fully analyzed in the previous works , as they use the technique only as an auxiliary term that adds up to their core methods . To examine the ability of clipped Q-learning to prevent the overestimation in offline RL on its own , we modify SAC [ 12 ] by increasing the number of Q-ensembles from 2 to N : min φi Es , a , s′∼D [ ( Qφi ( s , a ) − ( r ( s , a ) + γ Ea′∼πθ ( ·|s′ ) [ min j=1 , ... , N Qφ′j ( s′ , a′ ) − β log πθ ( a′ | s′ ) ] ) ) 2 ] max θ Es∼D , a∼πθ ( ·|s ) [ min j=1 , ... , N Qφj ( s , a ) − β log πθ ( a | s ) ] , ( 2 ) for i = 1 , . . . , N . We denote this modified algorithm as SAC-N . Figure 1 shows the preliminary experiments on D4RL halfcheetah-medium and hopper-medium datasets [ 9 ] while varying N . Note that these datasets are constructed from suboptimal behavior policies . Surprisingly , as we gradually increase N , we can successfully find policies that outperform the previous state-of-the-art method ( CQL ) by a large margin . In fact , as we will present in Section 5 , SAC-N outperforms CQL on various types of environments and data-collection policies . To understand why this simple technique works so well , we can first interpret the clipping procedure ( choosing the minimum value from the ensemble ) as penalizing state-action pairs with high-variance Q-value estimates , which encourages the policy to favor actions that appeared in the dataset [ 11 ] . The dataset samples will naturally have lower variance compared to the OOD samples as the Bellman residual term in Equation ( 2 ) explicitly aligns the Q-value predictions for the dataset samples . More formally , we can regard this difference in variance as accounting for epistemic uncertainty [ 8 ] which refers to the uncertainty stemming from limited data and knowledge . Utilization of the clipped Q-value relates to methods that consider the confidence bound of the Q-value estimates [ 24 ] . Online RL methods typically utilize the Q-ensemble to form an optimistic estimate of the Q-value , by adding the standard deviation to the mean of the Q-ensembles [ 18 ] . This optimistic Q-value , also known as the upper-confidence bound ( UCB ) , can encourage the exploration of unseen actions with high uncertainty . However , in offline RL , the dataset available during training is fixed , and we have to focus on exploiting the given data . For this purpose , it is natural to utilize the lower-confidence bound ( LCB ) of the Q-value estimates , for example by subtracting the standard deviation from the mean , which allows us to avoid risky state-actions . The clipped Q-learning algorithm , which chooses the worst-case Q-value instead to compute the pessimistic estimate , can also be interpreted as utilizing the LCB of the Q-value predictions . Suppose Q ( s , a ) follows a Gaussian distribution with mean m ( s , a ) and standard deviation σ ( s , a ) . Also , let { Qj ( s , a ) } Nj=1 be realizations of Q ( s , a ) . Then , we can approximate the expected minimum of the realizations following the work of Royston [ 23 ] as E [ min j=1 , ... , N Qj ( s , a ) ] ≈ m ( s , a ) − Φ−1 ( N − π8 N − π4 + 1 ) σ ( s , a ) , ( 3 ) where Φ is the CDF of the standard Gaussian distribution . This relation indicates that using the clipped Q-value is similar to penalizing the ensemble mean of the Q-values with the standard deviation scaled by a coefficient dependent on N . We now move on to the empirical analysis of the clipped Q-learning . Figure 2a compares the strength of the uncertainty penalty on in-distribution and OOD actions . Specifically , we compare actions sampled from two types of policies : ( 1 ) the behavior policy which was used to collect the dataset , and ( 2 ) the random policy which samples actions uniformly from the action space . For each policy , we measure the size of the penalty from the clipping as Es∼D , a∼π ( ·|s ) [ 1N ∑N j=1Qφj ( s , a ) − minj=1 , ... , N Qφj ( s , a ) ] . Figure 2a shows that the clipping term penalizes the random state-action pairs much stronger than the in-distribution pairs throughout the training . For comparison , we also measure the standard deviation of the Q-values for each policy . The results in Figure 2b show that as we conjectured , the Q-value predictions for the OOD actions have a higher variance . We also find that the size of the penalty and the standard deviation are highly correlated , as we noted in Equation ( 3 ) . As we observe that OOD actions have higher variance on Q-value estimates , the effect of increasing N becomes obvious : it strengthens the penalty applied to the OOD samples compared to the dataset samples . To verify this , we measured the relative penalty applied to the OOD samples in Figure 2c and found that indeed the OOD samples are penalized relatively further as N increases .

Summary: The paper presents a simple but rather effective idea (probably known by many but never formally tried): having an ensemble of N Q-networks and then using their *minimum* in the Bellman backup. This is roughly analogous to penalizing the objective function of fitted-Q iteration by the variance of the ensemble, which causes a somehow lower-confidence bound on the learning, which in turn causes conservative values for out-of-distribution actions at each seen state. This framework is then used as part of a soft actor-critic algorithm, called SAC-N. This significantly outperforms SOTA (but much slower in convergence rate). However, this setup can require an extremely large ensemble, which can be a limitation in practice. The authors next observe that the performance of SAC-N is highly correlated with the diversity of the Q-functions’ gradient w.r.t. actions (increased by the size of ensemble, N). To maximize diversity, they proposed to minimize the total cosine similarity matric, which gives rise to maximizing the variance for smallest eigen value of the variance-gradient matrix. As it gets into a computationally intractable problem, they instead maximize the sum of all eigenvalues, which is equal to the total variance. This final matric is called Ensemble Similarity and the resulting algorithm is named Ensemble-Diversified Actor Critic (EDAC). Finally, several experimental results are presented, largely outperforming previous methods.