Pessimistic Model Selection for Offline Deep Reinforcement Learning

1 INTRODUCTION . The success of deep reinforcement learning ( Mnih et al. , 2013 ; Henderson et al. , 2018 ) ( DRL ) often leverages upon executive training data with considerable efforts to select effective neural architectures . Deploying online simulation to learn useful representations for DRL is not always realistic and possible especially in some high-stake environments , such as automatic navigation ( Kahn et al. , 2018 ; Hase et al. , 2020 ) , dialogue learning ( Jaques et al. , 2020 ) , and clinical applications ( Tang et al. , 2020a ) . Offline reinforcement learning ( Singh & Sutton , 1996 ; Levine et al. , 2020 ; Agarwal et al. , 2020 ) ( OffRL ) has prompted strong interests ( Paine et al. , 2020 ; Kidambi et al. , 2020 ) to empower DRL toward training tasks associated with severe potential cost and risks . The idea of OffRL is to train DRL models with only logged data and recorded trajectories . However , with given observational data , designing a successful neural architecture in OffRL is often expensive ( Levine et al. , 2020 ) , requiring intensive experiments , time , and computing resources . Unlike most aforementioned applications with online interaction , Offline tasks for reinforcement learning often suffer the challenges from insufficient observational data from offline collection to construct a universal approximated model for fully capturing the temporal dynamics . Therefore , relatively few attempts in the literature have been presented for provide a pipeline to automate the development process for model selection and neural architecture search in OffRL settings . Here , model selection refers to selecting the best model ( e.g. , the policy learned by a trained neural network ) among a set of candidate models ( e.g . different neural network hyperparameters ) . In this work , we propose a novel model selection approach to automate OffRL development process , which provides an evaluation mechanism to identify a well-performed DRL model given offline data . Our method utilizes statistical inference to provide uncertainty quantification on value functions trained by different DRL models , based on which a pessimistic idea is incorporated to select the best model/policy . In addition , two refined approaches are further proposed to address the possible biases of DRL models in identifying the optimal policy . In this work , we mainly focus on deep Q-network ( Mnih et al. , 2013 ; 2015 ) ( DQN ) based architectures , while our proposed methods can be extended to other settings . Figure 1 demonstrates the superior performance of the proposed pessimistic model selection ( PMS ) method in identifying the best model among 70 DRL models of different algorithms on one navigation task ( See Appendix C for details ) , compared with the model selection method by ( Tang & Wiens , 2021 ) which uses three offline policy evaluation ( OPE ) estimates for validation . Specifically , based on the derived confidence interval of the OPE value for each candidate model , the final selected model by our PMS method is the one that has the largest lower confidence limit , which exactly has the largest true OPE value among all candidate models . In contrast , none of three OPE estimates used for model selection by Tang & Wiens ( 2021 ) can identify the best model due to the inevitable overfitting issue during the validation procedure . To close this section , we summarize the contributions of this work as follows : • We propose a novel PMS framework , which targets finding the best policy from given candidate models ( e.g. , neural architecture , hyperparameters , etc ) with offline data for DQN learning . Unlike many existing methods , our approach essentially does not involve additional hyperparameter tuning except for two interpretable parameters . • Leveraging asymptotic analysis in statistical inference , we provide uncertainty quantification on each candidate model , based on which our method can guarantee that the worst performance of finally selected model is the best among all candidate models . See Corollary 1 for more details . • To address potential biases of candidate models in identifying the optimal policy , two refined approaches are proposed , one of which can be shown to have regret bounded by the smallest error bound among all candidate models under some technical conditions ( See Corollary 2 ) . To the best of our knowledge , this is the first model-selection method in offline DRL with such a guarantee . • The numerical results demonstrate that the proposed PMS shows superior performance in different DQN benchmark environments . 2 RELATED WORK . Model Selection for Reinforcement Learning : Model selection has been studied in online decision-making environments ( Fard & Pineau , 2010 ; Lee & Taylor , 2014 ) . Searching nearly optimal online model is a critical topic for online bandits problems with limited information feed-backs . For linear contextual bandits , Abbasi-Yadkori et al . ( 2011 ) ; Chu et al . ( 2011 ) are aiming to find the best worst-case bound when the optimal model class is given . For model-based reinforcement learning , Pacchiano et al . ( 2020 ) introduces advantages of using noise augmented Markov Decision Processes ( MDP ) to archive a competitive regret bound to select an individual model with constraints for ensemble training . Recently , Lee et al . ( 2021 ) utilized an online algorithm to select a low-complexity model based on a statistical test . However , most of the previous model selection approaches are focused on the online reinforcement learning setting . Very few works including Farahmand & Szepesvári ( 2011 ) ; Paine et al . ( 2020 ) ; Su et al . ( 2020 ) ; Yang et al . ( 2020 ) ; Kuzborskij et al . ( 2021 ) ; Tang & Wiens ( 2021 ) ; Xie & Jiang ( 2021 ) are focused on the offline setting . In particular , ( Su et al. , 2020 ; Yang et al. , 2020 ; Kuzborskij et al. , 2021 ) focus on model selection for OPE problem . ( Farahmand & Szepesvári , 2011 ; Xie & Jiang , 2021 ) select the best model/policy based on minimizing the Bellman error , while the first approach requires an additional tuning and latter does not . ( Paine et al. , 2020 ; Tang & Wiens , 2021 ) proposed several criteria to perform model selection in OffRL and mainly focused on numerical studies . In this work , we provide one of the first model selection approaches based on statistical inference for RL tasks with offline data collection . Offline-Policy Learning : Training a DRL agent with offline data collection often relies on batchwise optimization . Batch-Constrained deep Q-learning ( Fujimoto et al. , 2019 ) ( BCQ ) is considered one OffRL benchmark that uses a generative model to minimize the distance of selected actions to the batch-wise data with a perturbation model to maximize its value function . Other popular OffRL approaches , such as behavior regularized actor-critic ( BRAC ) ( Wu et al. , 2019 ) , and random ensemble mixture ( Agarwal et al. , 2020 ) ( REM ) ( as an optimistic perspective on large dataset ) , have also been studied in RL Unplugged ( RLU ) ( Gulcehre et al. , 2020 ) benchmark together with behavior cloning ( Bain & Sammut , 1995 ; Ross & Bagnell , 2010 ) ( BC ) , DQN , and DQN with quantile regression ( Dabney et al. , 2018 ) ( QR-DQN ) . RLU suggests a naive approach based on human experience for offline policy selection , which requires independent modification with shared domain expertise ( e.g. , Atari environments ) for tuning each baseline . Meanwhile , how to design a model selection algorithm for OffRL remains an open question . Motivated by the benefits and the challenges as mentioned earlier of the model selection for offline DRL , we aim to develop a unified approach for model selection in offline DRL with theoretical guarantee and interpretable tuning parameters . 3 BACKGROUND AND NOTATIONS . Consider a time-homogeneous Markov decision process ( MDP ) characterized by a tuple M = ( S , A , p , r , γ ) , where S is the state space , A is the action space , p is the transition kernel , i.e. , p ( s′|s , a ) is the probability mass ( density ) of transiting to s′ given current state-action ( s , a ) , r is the reward function , i.e. , E ( Rt|St = s , At = a ) = r ( s , a ) for t ≥ 0 , and 0 ≤ γ < 1 is a discount factor . For simplifying presentation , we assume A and S are both finite . But our method can also be applied in continuous cases . Under this MDP setting , it is sufficient to consider stationary Markovian policies for optimizing discounted sum of rewards ( Puterman , 1994 ) . Denote π as a stationary Markovian policy mapping from the state space S into a probability distribution over the action space . For example , π ( a|s ) denotes the probability of choosing action a given the state value s. One essential goal of RL is to learn an optimal policy that maximizes the value function . Define V π ( s ) = ∑+∞ t=0 γ tEπ [ Rt|S0 = s ] and then the optimal policy is defined as π∗ ∈ argmaxπ { V ( π ) , ( 1 − γ ) ∑ s∈S V π ( s ) ν ( s ) } , where ν denotes some reference distribution function over S. In addition , we denote Q-function asQπ ( s , a ) = ∑+∞ t=0 γ tEπ ( Rt|A0 = a , S0 = s ) for s ∈ S and a ∈ A . In this work , we consider the OffRL setting . The observed data consist of N trajectories , corresponding to N independent and identically distributed copies of { ( St , At , Rt ) } t≥0 . For any i ∈ { 1 , · · · , n } , data collected from the ith trajectory can be summarized by { ( Si , t , Ai , t , Ri , t , Si , t+1 ) } 0≤t < T , where T denotes the termination time . We assume that the data are generated by some fixed stationary policy denoted by b . Among many RL algorithms , we focus on Q-learning type of methods . The foundation is the optimal Bellman equation given below . Q∗ ( s , a ) = E [ Rt + γ max a′∈A Q∗ ( St+1 , a ′ ) |St = s , At = a ] , ( 1 ) where Q∗ is called optimal Q-function , i.e. , Q-function under π∗ . Among others , fitted q-iteration ( FQI ) is one of the most popular RL algorithms ( Ernst et al. , 2005 ) . FQI leverages supervised learning techniques to iteratively solve the optimal Bellman equation ( 1 ) and shows competitive performance in OffRL . To facilitate our model-selection algorithm , we introduce the discounted visitation probability , motivated by the marginal importance sampling estimator in ( Liu et al. , 2018 ) . For any t ≥ 0 , let pπt ( s , a ) denote the t-step visitation probability Pr π ( St = s , At = a ) assuming the actions are selected according to π at time 1 , · · · , t. We define the discounted visitation probability function as dπ ( s , a ) = ( 1 − γ ) ∑ t≥0 γ tpπt ( s , a ) . To adjust the distribution from behavior policy to any target policy π , we use the discounted probability ratio function defined as ωπ , ν ( s , a ) = dπ ( s ) π ( a|s ) 1 T ∑T−1 t=0 p b t ( s , a ) , ( 2 ) where pbt ( s , a ) is the t-step visitation probability under the behavior policy b , i.e. , Pr b ( St = s , At = a ) . The ratio function ωπ , ν ( s , a ) is always assumed well defined . The estimation of ratio function is motivated by the observation that for every measurable function f defined over S ×A , E [ 1 T T−1∑ t=0 ωπ , ν ( St , At ) ( f ( St , At ) − γ ∑ a′∈A π ( a′ | St+1 ) f ( St+1 , a′ ) ) ] = ( 1− γ ) ES0∼ν [ ∑ a∈A π ( a | S0 ) f ( a , S0 ) ] , ( 3 ) based on which several min-max estimation methods has been proposed such as ( Liu et al. , 2018 ; Nachum et al. , 2019 ; Uehara & Jiang , 2019 ) ; We refer to ( Uehara & Jiang , 2019 , Lemma 1 ) for a formal proof of equation ( 3 ) . Finally , because our proposed model selection algorithm relies on an efficient evaluation of any target policy using batch data , we introduce three types of offline policy evaluation estimators in the existing RL literature . The first type is called direct method via estimating Q-function , based on the relationship that V ( π ) = ( 1 − γ ) ∑ s∈S , a∈A π ( a|s ) Q ( s , a ) ν ( s ) . The second type is motivated by the importance sampling ( Precup , 2000 ) . Based on the definition of ratio function , we can see V ( π ) = E [ 1T ∑T−1 t=0 ω π , ν ( St , At ) Rt ] , from which a plugin estimator can be constructed . The last type of OPE methods combines the first two types of methods and construct a so-called doubly robust estimator ( Kallus & Uehara , 2019 ; Tang et al. , 2020b ) . This estimator is motivated by the efficient influence function of V ( π ) under a transition-sampling setting and the model that consists of the set of all observed data distributions given by arbitrarily varying the initial , transition , reward , and behavior policy distributions , subject to certain minimal regularity and identifiability conditions ( Kallus & Uehara , 2019 ) , i.e. , 1 T T−1∑ t=0 ωπ , ν ( St , At ) ( Rt + γ ∑ a∈A π ( a|St+1 ) Qπ ( St+1 , a ) −Qπ ( St , At ) ) + ( 1− γ ) ES0∼ν [ ∑ a∈A π ( a|S0 ) Qπ ( S0 , a ) ] − V ( π ) . ( 4 ) A nice property of doubly robust estimators is that as long as either the Q-function Qπ ( s , a ) or the ratio function ωπ , ν ( s , a ) can be consistently estimated , the final estimator of V ( π ) is consistent ( Robins et al. , 1994 ; Jiang & Li , 2015 ; Kallus & Uehara , 2019 ; Tang et al. , 2020b ) . Furthermore , a doubly robust estimator based on ( 4 ) can achieve semiparametric efficiency under the conditions proposed by ( Kallus & Uehara , 2019 ) , even if nuisance parameters are estimated via black box models such as deep neural networks . Therefore such an estimator is particularly suitable under the framework of DRL . Our proposed algorithm will rely on this doubly robust type of OPE estimator .

Summary: This paper proposed a new algorithm for model-section in offline RL based on adding the pessimism principle on top of existing OPE algorithms. Under strong assumptions, the paper provides an asymptotic convergence result for the proposed algorithm. It also provides empirical experiments demonstrating that the proposed algorithm outperforms existing baselines.