Active Offline Policy Selection

1 Introduction . Reinforcement learning ( RL ) has recently proven to be successful in a range of applications from computer and board games [ 46 , 55 , 58 ] to robotics and chip design [ 37 , 10 , 45 ] . However , many challenges of real-world systems still prevent RL from being applied at scale in practice [ 16 ] . One of the limiting factors in real applications is that environment interactions are often expensive . Thus , training and evaluating agents becomes prohibitively slow and costly . Offline RL attempts to address this problem by training agents on a dataset without environment interactions [ 2 , 23 ] . This enables researchers to train policies with multiple algorithms and different hyperparameter settings [ 50 ] on a single dataset . The best policy can then be chosen with off-policy policy evaluation ( OPE ) approaches [ 53 , 40 ] ( Fig . 1 , left ) . These techniques attempt to estimate the expected agent performance by relying on the same pre-recorded dataset [ 28 , 41 , 36 , 72 ] . ∗equal contribution 2The paper website is at https : //sites.google.com/corp/view/active-ops and the code is at https : //github.com/deepmind/active_ops . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Unfortunately , OPE estimates are not yet precise enough to be relied upon for choosing which agent to deploy . Moreover , OPE is intrinsically limited by the pre-recorded dataset : When there is a distribution shift between a trained policy and a behavioural policy , i.e. , when the trained policy behaves very differently from the data collection policy , OPE is imprecise [ 38 ] . In many applications , however , a subset of policies can be evaluated online prior to final deployment [ 23 , 19 ] , even though this process can be hard and laborious . For example , in robotics a small subset of offline RL policies can be tested on the real hardware . Another example is recommender systems , which allow for restricted policy evaluation via A/B tests on a small fraction of user traffic . In this paper , we introduce the active offline policy selection problem . The goal is to identify the best policy where we benefit from both offline evaluation with logged data and online evaluation with limited interaction ( Fig . 1 , right ) . In our approach we advance a solution based on Bayesian optimization ( BO ) . It entails learning a Gaussian process ( GP ) surrogate function that maps policies to their expected returns . Then , we build on the GP statistics to construct an acquisition function to decide which policy to test next . To make BO successful in this problem setting , our approach has two key features . First , we incorporate existing OPE estimates as additional noisy observations ( Fig . 2 ) . This allows us to warm start online policy evaluation and to overcome the difficulties of GP hyper-parameter optimisation at the start . Second , we model correlation between policies through a kernel function based on actions that the policies take in the same states of the environment ( Fig . 3 ) . This makes our method data efficient [ 26 ] as the information about the performance of one policy informs us about the performance of similar behaving policies without costly execution in the environment . It is particularly valuable when the number of candidate policies is large ( or even larger than the interaction budget ) . The contributions of this paper are as follows . 1 . We introduce the active offline policy selection problem . This problem is important in practice , but so far it has not been studied in the literature to the best of our knowledge . 2 . We build a BO solution with an extended observation model to integrate both OPE estimates and interaction data . The use of OPE estimates enables us to warm-start learning a GP . 3 . We propose a novel GP kernel to capture the dependency between the policies through the actions that they take . As a result , our method infers the value of one policy from those of similar policies . The rest of this paper is organized as follows . Section 1.1 gives background on offline policy evaluation and selection . In section 2 , we propose a GP model for policy value estimation , explain how we approach the sequential decision making with BO and introduce a novel kernel based on the similarity of policy outputs . Section 3 discusses the related work . Section 4 shows that active policy evaluation can improve upon the OPE after merely a few interactions , thanks to the kernel that ensures data-efficiency . Additionally , our method works reliably with OPEs of varying quality and it scales well with the growing number of candidate policies . The paper is concluded with a discussion in section 5 . 1.1 Off-policy policy evaluation and selection . We consider a Markov decision process ( MDP ) defined by a tuple ( S , A , T , R , d0 , γ ) , with state space S , action space A , transition distribution T ( s′|s , a ) , reward function R ( s , a ) , initial state distribution d0 ( s ) , and discount factor γ ∈ [ 0 , 1 ] . A policy π ( a|s ) maps from S to A . The value of a policy π is measured by the expected sum of discounted rewards : µπ = E [ ∞∑ t=0 γtR ( st , at ) ] , with s0 ∼ d0 ( · ) , at ∼ π ( ·|st ) , st+1 ∼ T ( ·|st , at ) . ( 1 ) Typically , estimating the value of a given policy requires executing the policy in the environment many times . However , practical constraints make it difficult to run many policies of unknown quality in the environment . To address this , OPE estimates a value µ̂π using a dataset of trajectories D collected by a separate behavior policy πβ . In offline policy selection ( OPS ) the task is to select the policy with the highest value from a set of policies π1 , π2 , . . . , πK given access only to a dataset D ( Fig . 1 , left ) . A straightforward approach is to select the policy with the highest OPE estimate µ̂π1 , . . . , µ̂πK [ 20 ] , and alternative approaches can be used depending on the quality metric [ 71 ] . 2 Active offline policy selection . 2.1 Problem definition . We now formally define the active offline policy selection problem . Suppose we are given a set of K candidate policies πk , 1 ≤ k ≤ K. Denote by µk the unknown policy value and by ρk an OPE estimate computed from an offline dataset . At every step i , we can choose a policy ki to execute in the environment once and observe the trajectory with ( discounted ) episodic return ri , which is a noisy sample of the true policy value : E [ ri ] = µki . We would like to find the policy with the highest value , k∗ = arg maxk µk , with as few trajectories ( budget ) as possible . Policy selection algorithm estimates the mean as mk and recommends a policy that maximizes it : k̂ = arg maxkmk . Simple regret of the recommended policy measures how close we are to the best policy : regret = µk∗ − µk̂ . ( 2 ) To solve this sequential decision problem , we introduce a GP over the policy returns and OPE estimates ( subsection 2.2 ) , and design the GP kernel over policies to model their correlated returns ( subsection 2.3 ) . Then we use these as part of a BO strategy ( subsection 2.4 ) . 2.2 Gaussian process over policy values . Given a limited interaction budget and a large number of policies , it is impossible to obtain accurate estimates of their values by executing them many times . Fortunately , similar policies tend to have similar values . Thus , propagating the information between them helps to improve the value prediction . Intuitively , the number of policies that we need to execute to find a good policy should depend on their diversity instead of the total number of candidate policies ( see experiments supporting this in Fig . 7 ) . Formally , we use GP to predict a joint distribution over the policy values . < latexit sha1_base64= '' hXL3zgU/olAsQSEh0vHmEWgBWEM= '' > AAAB6HicbVDLSgNBEOyNrxhfUY9eBoOQU9gNoh4DXjwmkBckS5iddJIxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXEAuujet+O7mt7Z3dvfx+4eDw6PikeHrW1lGiGLZYJCLVDahGwSW2DDcCu7FCGgYCO8H0fuF3nlBpHsmmmcXoh3Qs+YgzaqzUKA+KJbfiLkE2iZeREmSoD4pf/WHEkhClYYJq3fPc2PgpVYYzgfNCP9EYUzalY+xZKmmI2k+Xh87JlVWGZBQpW9KQpfp7IqWh1rMwsJ0hNRO97i3E/7xeYkZ3fsplnBiUbLVolAhiIrL4mgy5QmbEzBLKFLe3EjahijJjsynYELz1lzdJu1rxbirVxnWp1sziyMMFXEIZPLiFGjxAHVrAAOEZXuHNeXRenHfnY9Wac7KZc/gD5/MHdK2MxQ== < /latexit > A GP is a stochastic process , that is an indexed collection of random variables , such that every finite collection of them has a multivariate normal distribution [ 54 ] . In our case , we assume that the policy value µ as a function of policy π follows a GP , with a kernel function K ( πi , πj ) over policies and mean m. Specifically , we consider the following generative process for the latent policy value µ ( π ) , OPE estimate ρ , and episodic return r : µ ( π ) ∼ GP ( m ( · ) , K ( · , · ) ) , ρ ∼ N ( µ ( π ) , σ2ρ ) , r ∼ N ( µ ( π ) , σ2r ) . ( 3 ) We use a constant mean m without loss of generality . We assume a flat prior for the hyper-parameter m , and weakly informative inverse Gamma prior for the variance σ2ρ and σ 2 r . Our probabilistic model is illustrated in Fig . 2 . As we have a finite number of policies , the input space is a finite set X = { π1 , . . . , πK } , where each index corresponds to a candidate policy . The joint distribution of the K policy values reduces to a multivariate normal distribution , µ1 , . . . , µK ∼ N ( m1K , K ) , with Kk , k′ = K ( πk , πk′ ) , ( 4 ) where 1K is a K-length vector of ones . For each policy πk , we have one noisy observation from an OPE method ρk , and zero or multiple noisy episodic return observations rik . As the policies are related through the covariance matrix K , observing a return sample from one policy updates the posterior distribution of all of them . Given the OPE estimates ρ = [ ρ1 , . . . , ρK ] and Nk return observations rk = [ r1k , . . . , r Nk k ] for each policy πk , the posterior distribution of the mean return is also a Gaussian , µ1 , . . . , µK |ρ , { rk } k ∼ N ( m , Σ ) , with m = K ( K + Λ ) −1y , Σ = K−K ( K + Λ ) −1K . yk = ( 1 σ2ρ + Nk σ2r ) −1 ( ρk σ2ρ + 1 σ2r Nk∑ i=1 rik ) , Λkk = ( 1 σ2ρ + Nk σ2r ) −1 , ( 5 ) where Λ is a diagonal matrix with diagonal entries defined above . 2.3 Kernel . A key component of the GP model is the kernel that measures our belief about the policy correlation . To obtain a kernel we make a key assumption : Policies that are similar in the actions that they take , yield similar returns . Then , our insight is to measure the distance between the policies through the actions that each of them takes on a fixed set of states from the offline dataset ( Fig . 3 , left ) . Formally , to measure the similarity between π1 and π2 , we obtain the policy actions at a given state s and measure the distance between these two action vectors d ( π1 ( s ) , π2 ( s ) ) 3 . We then define the distance between the two policies as the average distance over the states of the offline dataset D , d̄ ( π1 , π2 ) = Es∼D d ( π1 ( s ) , π2 ( s ) ) . ( 6 ) 3In this work , we use Euclidean distance for deterministic policies with continuous actions and Hamming distance for discrete actions . An extension to stochastic policies is straightforward . In practice we approximate d̄ using a randomly sampled subset of states D′ ⊂ D. An example of a pairwise distance matrix for a set of policies is shown in Fig . 3 , middle . Rows and columns are the policies which are arranged first by the type of the training algorithm , then by hyperparameters and finally by the training steps ( none of which is observed by our method ) . Note the block structure , which reflects the different nature of training algorithms . Smaller blocks of increasing policy distances correspond to policies with more training steps and the same hyperparameters . Additionally , Fig . 3 , right shows an example of policy distances to the behaviour policy . The values of policies produced by the same algorithm change gradually as a function of this distance . It provides indication that our kernel construction is informative for predicting the value of a policy . Finally , we compute Matérn 1/2 kernel as : K ( π1 , π2 ) = σ2k exp ( −d̄ ( π1 , π2 ) /l ) , where σ and l are the trainable variance and length-scale hyperparameters . This is a popular kernel choice [ 56 ] . As the distance metric is the average distance over states , using it in this kernel is equivalent to multiplication of kernels on these states and it is a valid positive semi-definite matrix .

Summary: This paper introduces the problem of active offline policy selection which consists of selecting the best policy from a set of candidates given a small budget to interact with the environment. The paper proposes a solution (A-OPS) based on Bayesian optimization using a kernel function to relate candidate policies and generalize between them. The proposed solution is thoroughly evaluated on simulated robotic control environments using benchmark datasets from the off-policy evaluation literature and several ablations are included to rationalize the choices of the proposed method.