Efficient Inference and Exploration for Reinforcement Learning

1 Introduction . We consider the classical reinforcement learning ( RL ) problem where the agent interacts with a random environment and aims to maximize the accumulated discounted reward over time . The environment is formulated as a Markov decision process ( MDP ) and the agent is uncertain about the true dynamics to start with . As the agent interacts with the environment , data about the system dynamics are collected and the agent becomes increasingly confident about her decision . With finite data , however , the potential reward from each decision is estimated with errors and the agent may be led to a suboptimal decision . Our focus in this paper is on statistically efficient methodologies to quantify these errors and uncertainties , and to demonstrate their use in obtaining better policies . More precisely , we investigate the large-sample behaviors of estimated Q-value , optimal value function , and their associated policies . Our results are in the form of asymptotic convergence to an explicitly identified and computable Gaussian ( or other ) distribution , as the collected data sizes increase . The motivation of our investigation is three-fold . First , these precise asymptotic statements allow us to construct accurate confidence regions for quantities related to the optimal policy , and , like classical statistical inference , they can assess the reliability of the current estimates with respect to the data noises . Second , our results complement some finite-sample error bounds developed in the literature ( Kearns & Singh , 1998 ; Kakade , 2003 ; Munos & Szepesvári , 2008 ) , by supplementing a closed-form asymptotic variance that often shows up in the first-order terms in these bounds . Our third and most important motivation is to design good exploration policies by directly using our tight error estimates . Motivated by recent autonomous-driving and other applications ( e.g. , Kalashnikov et al . ( 2018 ) ) , we consider the pure exploration setting where an agent is first assigned an initial period to collect as much experience as possible , and then , with the optimal policy trained offline , starts deployment to gain reward . We propose an efficient strategy to explore by optimizing the worst-case estimated relative discrepancy among the Q-values ( ratio of mean squared difference to variance ) , which provides a proxy for the probability of selecting the best policy . Similar criteria have appeared in the so-called optimal computing budget allocation ( OCBA ) procedure in simulation-based optimization ( Chen & Lee , 2011 ) ( a problem closely related to best-arm identification ( Audibert & Bubeck , 2010 ) in online learning ) . In this approach , one divides computation ( or observation ) budget into stages in which one sequentially updates mean and variance estimates , and optimizes next-stage budget allocations according to the worst-case relative discrepancy criterion . Our proposed procedure , which we term Q-OCBA , follows this idea with a crucial use of our Q-value estimates and randomized policies to achieve the optimal allocation . We demonstrate how this idea consistently outperforms other benchmark exploration policies , both in terms of the probability in selecting the best policy and generating the tightest confidence bounds for value estimates at the end of the exploration period . Regarding the problem of constructing tight error estimates in RL , the closest work to ours is Mannor et al . ( 2004 ; 2007 ) , which studies the bias and variance in value function estimates with a fixed policy . Our technique resolves a main technical challenge in Mannor et al . ( 2004 ; 2007 ) , which allows us to substantially generalize their variance results to Q-values , optimal value functions and asymptotic distributional statements . The derivation in Mannor et al . ( 2004 ; 2007 ) hinges on an expansion of the value function in terms of the perturbation of the transition matrix , which ( as pointed out by the authors ) is not easily extendable from a fixed-policy to the optimal value function . In contrast , our results utilize an implicit function theorem applied to the Bellman equation that can be verified to be sufficiently smooth . This idea turns out to allow us to obtain gradients for Q-values , translate to the optimal value function , and furthermore generalize to similar results for constrained MDP and approximate value iterations . We also relate our work to the line of studies on dynamic treatment regimes ( DTR ) ( Laber et al. , 2014 ) applied commonly in medical decision-making , which focuses on the statistical properties of polices on finite horizon ( such as two-period ) . Our infinite-horizon results on the optimal value and Q-value distinguishes our developments from the DTR literature . Moreover , our result on the non-unique policy case can be demonstrated to correspond to the “ non-regularity ” concept in DTR , where the true parameters are very close to the decision “ boundaries '' that switch the optimal policy ( motivated by situations of small treatment effects ) , thus making the obtained policy highly sensitive to estimation noises . In the rest of this paper , we first describe our MDP setup and notations ( Section 2 ) . Then we present our results on large-sample behaviors ( Section 3 ) , demonstrate their use in exploration strategies ( Section 4 ) , and finally substantiate our findings with experimental results ( Section 5 ) . In the Appendix , we first present generalizations of our theoretical results to constrained MDP ( A.1 ) and problems using approximate value iteration ( A.2 ) . Then we include more numerical experiments ( B ) , followed by all the proofs ( C ) . 2 Problem Setup . Consider an infinite horizon discounted reward MDP , M = ( S , A , R , P , γ , ρ ) , where S is the state space , A is the action space , R ( s , a ) denotes the random reward when the agent is in state s ∈ S and selects action a ∈ A , P ( s′|s , a ) is the probability of transitioning to state s′ in the next epoch given current state s and taken action a , γ is the discount factor , and ρ is the initial state distribution . The distribution of the reward R and the transition probability P are unknown to the agent . We assume both S and A are finite sets . Without loss of generality , we denote S = { 1 , 2 , . . . , ms } and A = { 1 , 2 , . . . , ma } . Finally , we make the following stochasticity assumption : Assumption 1 . R ( s , a ) has finite mean µR ( s , a ) and finite variance σ2R ( s , a ) ∀ s ∈ S , a ∈ A . For any given s ∈ S and a ∈ A , R ( s , a ) and S′ ∼ P ( ·|s , a ) are all independent random variables . A policy π is a mapping from each state s ∈ S to a probability measure over actions a ∈ A . Specifically , we write π ( a|s ) as the probability of taking action a when the agent is in state s and π ( ·|s ) as the madimensional vector of action probabilities at state s. For convenience , we sometimes write π ( s ) as the realized action given the current state is s. The value function associated with a policy π is defined as V π ( s ) = E [ ∑∞ t=0 γ tR ( st , π ( st ) ) |s0 = s ] with st+1 ∼ P ( .|st , π ( st ) ) . The expected value function , under the initial distribution ρ , is denoted by χπ = ∑ s ρ ( s ) V π ( s ) . A policy π∗ is said to be optimal if V π ∗ ( s ) = maxπ V π ( s ) for all s ∈ S. For convenience , we denote V ∗ = V π∗ and χ∗ = ∑ s ρ ( s ) V ∗ ( s ) . The Q-value , denoted by Q ( s , a ) , is defined as Q ( s , a ) = µR ( s , a ) +γE [ V ∗ ( S′ ) |s , a ] . Correspondingly , V ∗ ( s ) = maxaQ ( s , a ) and the Bellman equation for Q takes the form Q ( s , a ) = µR ( s , a ) + γE [ max a′ Q ( s′ , a′ ) |s , a ] , ( 1 ) for any ( s , a ) ∈ S × A. Denoting the Bellman operator as TµR , P ( · ) , Q is a fixed point associated with TµR , P , i.e . Q = TµR , P ( Q ) . For the most part of this paper we make the following assumption about Q : Assumption 2 . For any state s ∈ S , arg maxa∈AQ ( s , a ) is unique . Under Assumption 2 , the optimal policy π∗ is unique and deterministic . Let a∗ ( s ) = arg maxa∈AQ ( s , a ) . Then π∗ ( a|s ) = 1 ( a = a∗ ( s ) ) , where 1 ( · ) denotes the indicator function . We next introduce some statistical quantities arising from data . Suppose we have n observations ( whose collection mechanism will be made precise later ) , which we denote as { ( st , at , rt ( st , at ) , s′t ( st , at ) ) : 1 ≤ t ≤ n } , where rt ( st , at ) is the realized reward at time t and s′t ( st , at ) = st+1 . We define the sample mean µ̂R , n and the sample variance σ̂2R , n of the reward as µ̂R , n ( s = i , a = j ) = ∑ 1≤t≤n rt ( st , at ) 1 ( st = i , at = j ) ∑ 1≤t≤n 1 ( st = i , at = j ) , ( 2 ) σ̂2R , n ( s = i , a = j ) = ∑ 1≤t≤n rt ( st , at ) 21 ( st = i , at = j ) ∑ 1≤t≤n 1 ( st = i , at = j ) − µ̂R , n ( i , j ) 2 . ( 3 ) Similarly , we define the empirical transition matrix P̂n as P̂n ( s ′ = k|s = i , a = j ) = ∑ 1≤t≤n 1 ( st = i , at = j , s ′ t ( st , at ) = k ) ∑ 1≤t≤n 1 ( st = i , at = j ) ( 4 ) and its ms ×ms sampling covariance matrix ΣPs , a ( with one sample point of 1 ( st = s , at = a ) ) as ΣPs , a ( k1 , k2 ) = { P ( k1|s , a ) ( 1− P ( k1|s , a ) ) k1 = k2 −P ( k1|s , a ) P ( k2|s , a ) k1 6= k2 . , for 1 ≤ k1 ≤ ms , 1 ≤ k2 ≤ ms. With the data , we construct our estimate of Q , called Q̂n , which is the empirical fixed point of Tµ̂R , n , P̂n , i.e . Q̂n = Tµ̂R , n , P̂n ( Q̂n ) . Correspondingly , we also write V̂ ∗ n ( s ) = maxa∈A Q̂n ( s , a ) and χ̂∗n = ∑ s∈S ρ ( s ) V̂ ∗ n ( s ) . We shall focus on the empirical errors due to noises of the collected data , and assume the MDP or Q-value evaluation can be done off-line so that the fixed point equation for Q̂n can be solved exactly . .

Summary: This paper studies the inference problem of reinforcement learning. With a given exploration policy that satisfies some strong property to collect n data, the paper studies the distribution of the estimated optimal value function and Q-function when n goes to infinity. Both unique and non-unique optimal policy cases are studied. The non-unique case has a very different behavior as it is no longer Gaussian. The paper then uses these estimations to design a method that better explores and proposes a method Q-OCBA. Experiments were performed to compare this method with previous algorithms, e.g., UCRL.