Active Offline Policy Selection

1 Introduction . Reinforcement learning ( RL ) has recently proven to be successful in a range of applications from computer and board games [ 46 , 55 , 58 ] to robotics and chip design [ 37 , 10 , 45 ] . However , many challenges of real-world systems still prevent RL from being applied at scale in practice [ 16 ] . One of the limiting factors in real applications is that environment interactions are often expensive . Thus , training and evaluating agents becomes prohibitively slow and costly . Offline RL attempts to address this problem by training agents on a dataset without environment interactions [ 2 , 23 ] . This enables researchers to train policies with multiple algorithms and different hyperparameter settings [ 50 ] on a single dataset . The best policy can then be chosen with off-policy policy evaluation ( OPE ) approaches [ 53 , 40 ] ( Fig . 1 , left ) . These techniques attempt to estimate the expected agent performance by relying on the same pre-recorded dataset [ 28 , 41 , 36 , 72 ] . ∗equal contribution 2The paper website is at https : //sites.google.com/corp/view/active-ops and the code is at https : //github.com/deepmind/active_ops . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Unfortunately , OPE estimates are not yet precise enough to be relied upon for choosing which agent to deploy . Moreover , OPE is intrinsically limited by the pre-recorded dataset : When there is a distribution shift between a trained policy and a behavioural policy , i.e. , when the trained policy behaves very differently from the data collection policy , OPE is imprecise [ 38 ] . In many applications , however , a subset of policies can be evaluated online prior to final deployment [ 23 , 19 ] , even though this process can be hard and laborious . For example , in robotics a small subset of offline RL policies can be tested on the real hardware . Another example is recommender systems , which allow for restricted policy evaluation via A/B tests on a small fraction of user traffic . In this paper , we introduce the active offline policy selection problem . The goal is to identify the best policy where we benefit from both offline evaluation with logged data and online evaluation with limited interaction ( Fig . 1 , right ) . In our approach we advance a solution based on Bayesian optimization ( BO ) . It entails learning a Gaussian process ( GP ) surrogate function that maps policies to their expected returns . Then , we build on the GP statistics to construct an acquisition function to decide which policy to test next . To make BO successful in this problem setting , our approach has two key features . First , we incorporate existing OPE estimates as additional noisy observations ( Fig . 2 ) . This allows us to warm start online policy evaluation and to overcome the difficulties of GP hyper-parameter optimisation at the start . Second , we model correlation between policies through a kernel function based on actions that the policies take in the same states of the environment ( Fig . 3 ) . This makes our method data efficient [ 26 ] as the information about the performance of one policy informs us about the performance of similar behaving policies without costly execution in the environment . It is particularly valuable when the number of candidate policies is large ( or even larger than the interaction budget ) . The contributions of this paper are as follows . 1 . We introduce the active offline policy selection problem . This problem is important in practice , but so far it has not been studied in the literature to the best of our knowledge . 2 . We build a BO solution with an extended observation model to integrate both OPE estimates and interaction data . The use of OPE estimates enables us to warm-start learning a GP . 3 . We propose a novel GP kernel to capture the dependency between the policies through the actions that they take . As a result , our method infers the value of one policy from those of similar policies . The rest of this paper is organized as follows . Section 1.1 gives background on offline policy evaluation and selection . In section 2 , we propose a GP model for policy value estimation , explain how we approach the sequential decision making with BO and introduce a novel kernel based on the similarity of policy outputs . Section 3 discusses the related work . Section 4 shows that active policy evaluation can improve upon the OPE after merely a few interactions , thanks to the kernel that ensures data-efficiency . Additionally , our method works reliably with OPEs of varying quality and it scales well with the growing number of candidate policies . The paper is concluded with a discussion in section 5 . 1.1 Off-policy policy evaluation and selection . We consider a Markov decision process ( MDP ) defined by a tuple ( S , A , T , R , d0 , γ ) , with state space S , action space A , transition distribution T ( s′|s , a ) , reward function R ( s , a ) , initial state distribution d0 ( s ) , and discount factor γ ∈ [ 0 , 1 ] . A policy π ( a|s ) maps from S to A . The value of a policy π is measured by the expected sum of discounted rewards : µπ = E [ ∞∑ t=0 γtR ( st , at ) ] , with s0 ∼ d0 ( · ) , at ∼ π ( ·|st ) , st+1 ∼ T ( ·|st , at ) . ( 1 ) Typically , estimating the value of a given policy requires executing the policy in the environment many times . However , practical constraints make it difficult to run many policies of unknown quality in the environment . To address this , OPE estimates a value µ̂π using a dataset of trajectories D collected by a separate behavior policy πβ . In offline policy selection ( OPS ) the task is to select the policy with the highest value from a set of policies π1 , π2 , . . . , πK given access only to a dataset D ( Fig . 1 , left ) . A straightforward approach is to select the policy with the highest OPE estimate µ̂π1 , . . . , µ̂πK [ 20 ] , and alternative approaches can be used depending on the quality metric [ 71 ] . 2 Active offline policy selection . 2.1 Problem definition . We now formally define the active offline policy selection problem . Suppose we are given a set of K candidate policies πk , 1 ≤ k ≤ K. Denote by µk the unknown policy value and by ρk an OPE estimate computed from an offline dataset . At every step i , we can choose a policy ki to execute in the environment once and observe the trajectory with ( discounted ) episodic return ri , which is a noisy sample of the true policy value : E [ ri ] = µki . We would like to find the policy with the highest value , k∗ = arg maxk µk , with as few trajectories ( budget ) as possible . Policy selection algorithm estimates the mean as mk and recommends a policy that maximizes it : k̂ = arg maxkmk . Simple regret of the recommended policy measures how close we are to the best policy : regret = µk∗ − µk̂ . ( 2 ) To solve this sequential decision problem , we introduce a GP over the policy returns and OPE estimates ( subsection 2.2 ) , and design the GP kernel over policies to model their correlated returns ( subsection 2.3 ) . Then we use these as part of a BO strategy ( subsection 2.4 ) . 2.2 Gaussian process over policy values . Given a limited interaction budget and a large number of policies , it is impossible to obtain accurate estimates of their values by executing them many times . Fortunately , similar policies tend to have similar values . Thus , propagating the information between them helps to improve the value prediction . Intuitively , the number of policies that we need to execute to find a good policy should depend on their diversity instead of the total number of candidate policies ( see experiments supporting this in Fig . 7 ) . Formally , we use GP to predict a joint distribution over the policy values . < latexit sha1_base64= '' hXL3zgU/olAsQSEh0vHmEWgBWEM= '' > AAAB6HicbVDLSgNBEOyNrxhfUY9eBoOQU9gNoh4DXjwmkBckS5iddJIxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXEAuujet+O7mt7Z3dvfx+4eDw6PikeHrW1lGiGLZYJCLVDahGwSW2DDcCu7FCGgYCO8H0fuF3nlBpHsmmmcXoh3Qs+YgzaqzUKA+KJbfiLkE2iZeREmSoD4pf/WHEkhClYYJq3fPc2PgpVYYzgfNCP9EYUzalY+xZKmmI2k+Xh87JlVWGZBQpW9KQpfp7IqWh1rMwsJ0hNRO97i3E/7xeYkZ3fsplnBiUbLVolAhiIrL4mgy5QmbEzBLKFLe3EjahijJjsynYELz1lzdJu1rxbirVxnWp1sziyMMFXEIZPLiFGjxAHVrAAOEZXuHNeXRenHfnY9Wac7KZc/gD5/MHdK2MxQ== < /latexit > A GP is a stochastic process , that is an indexed collection of random variables , such that every finite collection of them has a multivariate normal distribution [ 54 ] . In our case , we assume that the policy value µ as a function of policy π follows a GP , with a kernel function K ( πi , πj ) over policies and mean m. Specifically , we consider the following generative process for the latent policy value µ ( π ) , OPE estimate ρ , and episodic return r : µ ( π ) ∼ GP ( m ( · ) , K ( · , · ) ) , ρ ∼ N ( µ ( π ) , σ2ρ ) , r ∼ N ( µ ( π ) , σ2r ) . ( 3 ) We use a constant mean m without loss of generality . We assume a flat prior for the hyper-parameter m , and weakly informative inverse Gamma prior for the variance σ2ρ and σ 2 r . Our probabilistic model is illustrated in Fig . 2 . As we have a finite number of policies , the input space is a finite set X = { π1 , . . . , πK } , where each index corresponds to a candidate policy . The joint distribution of the K policy values reduces to a multivariate normal distribution , µ1 , . . . , µK ∼ N ( m1K , K ) , with Kk , k′ = K ( πk , πk′ ) , ( 4 ) where 1K is a K-length vector of ones . For each policy πk , we have one noisy observation from an OPE method ρk , and zero or multiple noisy episodic return observations rik . As the policies are related through the covariance matrix K , observing a return sample from one policy updates the posterior distribution of all of them . Given the OPE estimates ρ = [ ρ1 , . . . , ρK ] and Nk return observations rk = [ r1k , . . . , r Nk k ] for each policy πk , the posterior distribution of the mean return is also a Gaussian , µ1 , . . . , µK |ρ , { rk } k ∼ N ( m , Σ ) , with m = K ( K + Λ ) −1y , Σ = K−K ( K + Λ ) −1K . yk = ( 1 σ2ρ + Nk σ2r ) −1 ( ρk σ2ρ + 1 σ2r Nk∑ i=1 rik ) , Λkk = ( 1 σ2ρ + Nk σ2r ) −1 , ( 5 ) where Λ is a diagonal matrix with diagonal entries defined above . 2.3 Kernel . A key component of the GP model is the kernel that measures our belief about the policy correlation . To obtain a kernel we make a key assumption : Policies that are similar in the actions that they take , yield similar returns . Then , our insight is to measure the distance between the policies through the actions that each of them takes on a fixed set of states from the offline dataset ( Fig . 3 , left ) . Formally , to measure the similarity between π1 and π2 , we obtain the policy actions at a given state s and measure the distance between these two action vectors d ( π1 ( s ) , π2 ( s ) ) 3 . We then define the distance between the two policies as the average distance over the states of the offline dataset D , d̄ ( π1 , π2 ) = Es∼D d ( π1 ( s ) , π2 ( s ) ) . ( 6 ) 3In this work , we use Euclidean distance for deterministic policies with continuous actions and Hamming distance for discrete actions . An extension to stochastic policies is straightforward . In practice we approximate d̄ using a randomly sampled subset of states D′ ⊂ D. An example of a pairwise distance matrix for a set of policies is shown in Fig . 3 , middle . Rows and columns are the policies which are arranged first by the type of the training algorithm , then by hyperparameters and finally by the training steps ( none of which is observed by our method ) . Note the block structure , which reflects the different nature of training algorithms . Smaller blocks of increasing policy distances correspond to policies with more training steps and the same hyperparameters . Additionally , Fig . 3 , right shows an example of policy distances to the behaviour policy . The values of policies produced by the same algorithm change gradually as a function of this distance . It provides indication that our kernel construction is informative for predicting the value of a policy . Finally , we compute Matérn 1/2 kernel as : K ( π1 , π2 ) = σ2k exp ( −d̄ ( π1 , π2 ) /l ) , where σ and l are the trainable variance and length-scale hyperparameters . This is a popular kernel choice [ 56 ] . As the distance metric is the average distance over states , using it in this kernel is equivalent to multiplication of kernels on these states and it is a valid positive semi-definite matrix .

Summary: This paper introduces a problem of active offline policy selection, where the goal is to select the best-performing policy among the fixed K policies using an abundant offline dataset and a limited additional interaction with the environment. This work proposes to solve this problem by Bayesian optimization (BO) with a Gaussian process (GP), where the input and output of the objective function are a policy and its expected return respectively, and function queries render noisy observations. To facilitate GP in this problem, a new GP kernel is introduced, which takes two policies as input, and its output is computed based on the similarities of policy distribution for each state in the offline dataset. Also, to make BO more data-efficient, off-policy evaluation (OPE) estimates through the offline dataset are used for a warm start of BO. Finally, UCB acquisition function is used for active policy selection. Experimental results show that the proposed BO with OPE-based warm start significantly outperforms the baselines. Also, ablation studies confirm that UCB selection is better than uniform selection, GP is better than methods that do not consider policy correlation, and using OPE is better than not using it.