Pessimistic Model Selection for Offline Deep Reinforcement Learning

1 INTRODUCTION . The success of deep reinforcement learning ( Mnih et al. , 2013 ; Henderson et al. , 2018 ) ( DRL ) often leverages upon executive training data with considerable efforts to select effective neural architectures . Deploying online simulation to learn useful representations for DRL is not always realistic and possible especially in some high-stake environments , such as automatic navigation ( Kahn et al. , 2018 ; Hase et al. , 2020 ) , dialogue learning ( Jaques et al. , 2020 ) , and clinical applications ( Tang et al. , 2020a ) . Offline reinforcement learning ( Singh & Sutton , 1996 ; Levine et al. , 2020 ; Agarwal et al. , 2020 ) ( OffRL ) has prompted strong interests ( Paine et al. , 2020 ; Kidambi et al. , 2020 ) to empower DRL toward training tasks associated with severe potential cost and risks . The idea of OffRL is to train DRL models with only logged data and recorded trajectories . However , with given observational data , designing a successful neural architecture in OffRL is often expensive ( Levine et al. , 2020 ) , requiring intensive experiments , time , and computing resources . Unlike most aforementioned applications with online interaction , Offline tasks for reinforcement learning often suffer the challenges from insufficient observational data from offline collection to construct a universal approximated model for fully capturing the temporal dynamics . Therefore , relatively few attempts in the literature have been presented for provide a pipeline to automate the development process for model selection and neural architecture search in OffRL settings . Here , model selection refers to selecting the best model ( e.g. , the policy learned by a trained neural network ) among a set of candidate models ( e.g . different neural network hyperparameters ) . In this work , we propose a novel model selection approach to automate OffRL development process , which provides an evaluation mechanism to identify a well-performed DRL model given offline data . Our method utilizes statistical inference to provide uncertainty quantification on value functions trained by different DRL models , based on which a pessimistic idea is incorporated to select the best model/policy . In addition , two refined approaches are further proposed to address the possible biases of DRL models in identifying the optimal policy . In this work , we mainly focus on deep Q-network ( Mnih et al. , 2013 ; 2015 ) ( DQN ) based architectures , while our proposed methods can be extended to other settings . Figure 1 demonstrates the superior performance of the proposed pessimistic model selection ( PMS ) method in identifying the best model among 70 DRL models of different algorithms on one navigation task ( See Appendix C for details ) , compared with the model selection method by ( Tang & Wiens , 2021 ) which uses three offline policy evaluation ( OPE ) estimates for validation . Specifically , based on the derived confidence interval of the OPE value for each candidate model , the final selected model by our PMS method is the one that has the largest lower confidence limit , which exactly has the largest true OPE value among all candidate models . In contrast , none of three OPE estimates used for model selection by Tang & Wiens ( 2021 ) can identify the best model due to the inevitable overfitting issue during the validation procedure . To close this section , we summarize the contributions of this work as follows : • We propose a novel PMS framework , which targets finding the best policy from given candidate models ( e.g. , neural architecture , hyperparameters , etc ) with offline data for DQN learning . Unlike many existing methods , our approach essentially does not involve additional hyperparameter tuning except for two interpretable parameters . • Leveraging asymptotic analysis in statistical inference , we provide uncertainty quantification on each candidate model , based on which our method can guarantee that the worst performance of finally selected model is the best among all candidate models . See Corollary 1 for more details . • To address potential biases of candidate models in identifying the optimal policy , two refined approaches are proposed , one of which can be shown to have regret bounded by the smallest error bound among all candidate models under some technical conditions ( See Corollary 2 ) . To the best of our knowledge , this is the first model-selection method in offline DRL with such a guarantee . • The numerical results demonstrate that the proposed PMS shows superior performance in different DQN benchmark environments . 2 RELATED WORK . Model Selection for Reinforcement Learning : Model selection has been studied in online decision-making environments ( Fard & Pineau , 2010 ; Lee & Taylor , 2014 ) . Searching nearly optimal online model is a critical topic for online bandits problems with limited information feed-backs . For linear contextual bandits , Abbasi-Yadkori et al . ( 2011 ) ; Chu et al . ( 2011 ) are aiming to find the best worst-case bound when the optimal model class is given . For model-based reinforcement learning , Pacchiano et al . ( 2020 ) introduces advantages of using noise augmented Markov Decision Processes ( MDP ) to archive a competitive regret bound to select an individual model with constraints for ensemble training . Recently , Lee et al . ( 2021 ) utilized an online algorithm to select a low-complexity model based on a statistical test . However , most of the previous model selection approaches are focused on the online reinforcement learning setting . Very few works including Farahmand & Szepesvári ( 2011 ) ; Paine et al . ( 2020 ) ; Su et al . ( 2020 ) ; Yang et al . ( 2020 ) ; Kuzborskij et al . ( 2021 ) ; Tang & Wiens ( 2021 ) ; Xie & Jiang ( 2021 ) are focused on the offline setting . In particular , ( Su et al. , 2020 ; Yang et al. , 2020 ; Kuzborskij et al. , 2021 ) focus on model selection for OPE problem . ( Farahmand & Szepesvári , 2011 ; Xie & Jiang , 2021 ) select the best model/policy based on minimizing the Bellman error , while the first approach requires an additional tuning and latter does not . ( Paine et al. , 2020 ; Tang & Wiens , 2021 ) proposed several criteria to perform model selection in OffRL and mainly focused on numerical studies . In this work , we provide one of the first model selection approaches based on statistical inference for RL tasks with offline data collection . Offline-Policy Learning : Training a DRL agent with offline data collection often relies on batchwise optimization . Batch-Constrained deep Q-learning ( Fujimoto et al. , 2019 ) ( BCQ ) is considered one OffRL benchmark that uses a generative model to minimize the distance of selected actions to the batch-wise data with a perturbation model to maximize its value function . Other popular OffRL approaches , such as behavior regularized actor-critic ( BRAC ) ( Wu et al. , 2019 ) , and random ensemble mixture ( Agarwal et al. , 2020 ) ( REM ) ( as an optimistic perspective on large dataset ) , have also been studied in RL Unplugged ( RLU ) ( Gulcehre et al. , 2020 ) benchmark together with behavior cloning ( Bain & Sammut , 1995 ; Ross & Bagnell , 2010 ) ( BC ) , DQN , and DQN with quantile regression ( Dabney et al. , 2018 ) ( QR-DQN ) . RLU suggests a naive approach based on human experience for offline policy selection , which requires independent modification with shared domain expertise ( e.g. , Atari environments ) for tuning each baseline . Meanwhile , how to design a model selection algorithm for OffRL remains an open question . Motivated by the benefits and the challenges as mentioned earlier of the model selection for offline DRL , we aim to develop a unified approach for model selection in offline DRL with theoretical guarantee and interpretable tuning parameters . 3 BACKGROUND AND NOTATIONS . Consider a time-homogeneous Markov decision process ( MDP ) characterized by a tuple M = ( S , A , p , r , γ ) , where S is the state space , A is the action space , p is the transition kernel , i.e. , p ( s′|s , a ) is the probability mass ( density ) of transiting to s′ given current state-action ( s , a ) , r is the reward function , i.e. , E ( Rt|St = s , At = a ) = r ( s , a ) for t ≥ 0 , and 0 ≤ γ < 1 is a discount factor . For simplifying presentation , we assume A and S are both finite . But our method can also be applied in continuous cases . Under this MDP setting , it is sufficient to consider stationary Markovian policies for optimizing discounted sum of rewards ( Puterman , 1994 ) . Denote π as a stationary Markovian policy mapping from the state space S into a probability distribution over the action space . For example , π ( a|s ) denotes the probability of choosing action a given the state value s. One essential goal of RL is to learn an optimal policy that maximizes the value function . Define V π ( s ) = ∑+∞ t=0 γ tEπ [ Rt|S0 = s ] and then the optimal policy is defined as π∗ ∈ argmaxπ { V ( π ) , ( 1 − γ ) ∑ s∈S V π ( s ) ν ( s ) } , where ν denotes some reference distribution function over S. In addition , we denote Q-function asQπ ( s , a ) = ∑+∞ t=0 γ tEπ ( Rt|A0 = a , S0 = s ) for s ∈ S and a ∈ A . In this work , we consider the OffRL setting . The observed data consist of N trajectories , corresponding to N independent and identically distributed copies of { ( St , At , Rt ) } t≥0 . For any i ∈ { 1 , · · · , n } , data collected from the ith trajectory can be summarized by { ( Si , t , Ai , t , Ri , t , Si , t+1 ) } 0≤t < T , where T denotes the termination time . We assume that the data are generated by some fixed stationary policy denoted by b . Among many RL algorithms , we focus on Q-learning type of methods . The foundation is the optimal Bellman equation given below . Q∗ ( s , a ) = E [ Rt + γ max a′∈A Q∗ ( St+1 , a ′ ) |St = s , At = a ] , ( 1 ) where Q∗ is called optimal Q-function , i.e. , Q-function under π∗ . Among others , fitted q-iteration ( FQI ) is one of the most popular RL algorithms ( Ernst et al. , 2005 ) . FQI leverages supervised learning techniques to iteratively solve the optimal Bellman equation ( 1 ) and shows competitive performance in OffRL . To facilitate our model-selection algorithm , we introduce the discounted visitation probability , motivated by the marginal importance sampling estimator in ( Liu et al. , 2018 ) . For any t ≥ 0 , let pπt ( s , a ) denote the t-step visitation probability Pr π ( St = s , At = a ) assuming the actions are selected according to π at time 1 , · · · , t. We define the discounted visitation probability function as dπ ( s , a ) = ( 1 − γ ) ∑ t≥0 γ tpπt ( s , a ) . To adjust the distribution from behavior policy to any target policy π , we use the discounted probability ratio function defined as ωπ , ν ( s , a ) = dπ ( s ) π ( a|s ) 1 T ∑T−1 t=0 p b t ( s , a ) , ( 2 ) where pbt ( s , a ) is the t-step visitation probability under the behavior policy b , i.e. , Pr b ( St = s , At = a ) . The ratio function ωπ , ν ( s , a ) is always assumed well defined . The estimation of ratio function is motivated by the observation that for every measurable function f defined over S ×A , E [ 1 T T−1∑ t=0 ωπ , ν ( St , At ) ( f ( St , At ) − γ ∑ a′∈A π ( a′ | St+1 ) f ( St+1 , a′ ) ) ] = ( 1− γ ) ES0∼ν [ ∑ a∈A π ( a | S0 ) f ( a , S0 ) ] , ( 3 ) based on which several min-max estimation methods has been proposed such as ( Liu et al. , 2018 ; Nachum et al. , 2019 ; Uehara & Jiang , 2019 ) ; We refer to ( Uehara & Jiang , 2019 , Lemma 1 ) for a formal proof of equation ( 3 ) . Finally , because our proposed model selection algorithm relies on an efficient evaluation of any target policy using batch data , we introduce three types of offline policy evaluation estimators in the existing RL literature . The first type is called direct method via estimating Q-function , based on the relationship that V ( π ) = ( 1 − γ ) ∑ s∈S , a∈A π ( a|s ) Q ( s , a ) ν ( s ) . The second type is motivated by the importance sampling ( Precup , 2000 ) . Based on the definition of ratio function , we can see V ( π ) = E [ 1T ∑T−1 t=0 ω π , ν ( St , At ) Rt ] , from which a plugin estimator can be constructed . The last type of OPE methods combines the first two types of methods and construct a so-called doubly robust estimator ( Kallus & Uehara , 2019 ; Tang et al. , 2020b ) . This estimator is motivated by the efficient influence function of V ( π ) under a transition-sampling setting and the model that consists of the set of all observed data distributions given by arbitrarily varying the initial , transition , reward , and behavior policy distributions , subject to certain minimal regularity and identifiability conditions ( Kallus & Uehara , 2019 ) , i.e. , 1 T T−1∑ t=0 ωπ , ν ( St , At ) ( Rt + γ ∑ a∈A π ( a|St+1 ) Qπ ( St+1 , a ) −Qπ ( St , At ) ) + ( 1− γ ) ES0∼ν [ ∑ a∈A π ( a|S0 ) Qπ ( S0 , a ) ] − V ( π ) . ( 4 ) A nice property of doubly robust estimators is that as long as either the Q-function Qπ ( s , a ) or the ratio function ωπ , ν ( s , a ) can be consistently estimated , the final estimator of V ( π ) is consistent ( Robins et al. , 1994 ; Jiang & Li , 2015 ; Kallus & Uehara , 2019 ; Tang et al. , 2020b ) . Furthermore , a doubly robust estimator based on ( 4 ) can achieve semiparametric efficiency under the conditions proposed by ( Kallus & Uehara , 2019 ) , even if nuisance parameters are estimated via black box models such as deep neural networks . Therefore such an estimator is particularly suitable under the framework of DRL . Our proposed algorithm will rely on this doubly robust type of OPE estimator .

Summary: The paper studies the model selection problem in offline reinforcement learning (ORL). The model selection of ORL is challenging due to insufficient observational data from offline collection. The paper proposes a novel model selection approach to automate ORL development process and to identify a well-performed model given offline data. The proposed pessimistic model selection (PMS) method leverages an uncertainty quantification on value functions and a pessimistic idea. Some experiments show the superior performance of the proposed model selection method.