Active Offline Policy Selection
Year: 2021

Abstract:
This paper addresses the problem of policy selection in domains with abundant logged data, but with a restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and recommendation domains among others. Several offpolicy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation in the real environment. Yet, large amounts of online interactions are often not possible in practice. To overcome this problem, we introduce active offline policy selection — a novel sequential decision approach that combines logged data with online interaction to identify the best policy. This approach uses OPE estimates to warm start the online evaluation. Then, in order to utilize the limited environment interactions wisely we decide which policy to evaluate next based on a Bayesian optimization method with a kernel function that represents policy similarity. We use multiple benchmarks with a large number of candidate policies to show that the proposed approach improves upon state-of-the-art OPE estimates and pure online policy evaluation 2.

heading: 1 Introduction
text: Reinforcement learning (RL) has recently proven to be successful in a range of applications from computer and board games [46, 55, 58] to robotics and chip design [37, 10, 45]. However, many challenges of real-world systems still prevent RL from being applied at scale in practice [16]. One of the limiting factors in real applications is that environment interactions are often expensive. Thus, training and evaluating agents becomes prohibitively slow and costly. Offline RL attempts to address this problem by training agents on a dataset without environment interactions [2, 23]. This enables researchers to train policies with multiple algorithms and different hyperparameter settings [50] on a single dataset. The best policy can then be chosen with off-policy policy evaluation (OPE) approaches [53, 40] (Fig. 1, left). These techniques attempt to estimate the expected agent performance by relying on the same pre-recorded dataset [28, 41, 36, 72].
∗equal contribution 2The paper website is at https://sites.google.com/corp/view/active-ops and the code is at
https://github.com/deepmind/active_ops.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Unfortunately, OPE estimates are not yet precise enough to be relied upon for choosing which agent to deploy. Moreover, OPE is intrinsically limited by the pre-recorded dataset: When there is a distribution shift between a trained policy and a behavioural policy, i.e., when the trained policy behaves very differently from the data collection policy, OPE is imprecise [38]. In many applications, however, a subset of policies can be evaluated online prior to final deployment [23, 19], even though this process can be hard and laborious. For example, in robotics a small subset of offline RL policies can be tested on the real hardware. Another example is recommender systems, which allow for restricted policy evaluation via A/B tests on a small fraction of user traffic.
In this paper, we introduce the active offline policy selection problem. The goal is to identify the best policy where we benefit from both offline evaluation with logged data and online evaluation with limited interaction (Fig. 1, right). In our approach we advance a solution based on Bayesian optimization (BO). It entails learning a Gaussian process (GP) surrogate function that maps policies to their expected returns. Then, we build on the GP statistics to construct an acquisition function to decide which policy to test next. To make BO successful in this problem setting, our approach has two key features. First, we incorporate existing OPE estimates as additional noisy observations (Fig. 2). This allows us to warm start online policy evaluation and to overcome the difficulties of GP hyper-parameter optimisation at the start. Second, we model correlation between policies through a kernel function based on actions that the policies take in the same states of the environment (Fig. 3). This makes our method data efficient [26] as the information about the performance of one policy informs us about the performance of similar behaving policies without costly execution in the environment. It is particularly valuable when the number of candidate policies is large (or even larger than the interaction budget).
The contributions of this paper are as follows.
1. We introduce the active offline policy selection problem. This problem is important in practice, but so far it has not been studied in the literature to the best of our knowledge.
2. We build a BO solution with an extended observation model to integrate both OPE estimates and interaction data. The use of OPE estimates enables us to warm-start learning a GP.
3. We propose a novel GP kernel to capture the dependency between the policies through the actions that they take. As a result, our method infers the value of one policy from those of similar policies.
The rest of this paper is organized as follows. Section 1.1 gives background on offline policy evaluation and selection. In section 2, we propose a GP model for policy value estimation, explain how we approach the sequential decision making with BO and introduce a novel kernel based on the similarity of policy outputs. Section 3 discusses the related work. Section 4 shows that active policy evaluation can improve upon the OPE after merely a few interactions, thanks to the kernel that ensures data-efficiency. Additionally, our method works reliably with OPEs of varying quality and it scales well with the growing number of candidate policies. The paper is concluded with a discussion in section 5.

heading: 1.1 Off-policy policy evaluation and selection
text: We consider a Markov decision process (MDP) defined by a tuple (S,A, T,R, d0, γ), with state space S, action space A, transition distribution T (s′|s, a), reward function R(s, a), initial state distribution
d0(s), and discount factor γ ∈ [0, 1]. A policy π(a|s) maps from S to A. The value of a policy π is measured by the expected sum of discounted rewards:
µπ = E [ ∞∑ t=0 γtR(st, at) ] , with s0 ∼ d0(·), at ∼ π(·|st), st+1 ∼ T (·|st, at) . (1)
Typically, estimating the value of a given policy requires executing the policy in the environment many times. However, practical constraints make it difficult to run many policies of unknown quality in the environment. To address this, OPE estimates a value µ̂π using a dataset of trajectories D collected by a separate behavior policy πβ . In offline policy selection (OPS) the task is to select the policy with the highest value from a set of policies π1, π2, . . . , πK given access only to a dataset D (Fig. 1, left). A straightforward approach is to select the policy with the highest OPE estimate µ̂π1 , . . . , µ̂πK [20], and alternative approaches can be used depending on the quality metric [71].

heading: 2 Active offline policy selection

heading: 2.1 Problem definition
text: We now formally define the active offline policy selection problem. Suppose we are given a set of K candidate policies πk, 1 ≤ k ≤ K. Denote by µk the unknown policy value and by ρk an OPE estimate computed from an offline dataset. At every step i, we can choose a policy ki to execute in the environment once and observe the trajectory with (discounted) episodic return ri, which is a noisy sample of the true policy value: E[ri] = µki . We would like to find the policy with the highest value, k∗ = arg maxk µk, with as few trajectories (budget) as possible. Policy selection algorithm estimates the mean as mk and recommends a policy that maximizes it: k̂ = arg maxkmk. Simple regret of the recommended policy measures how close we are to the best policy:
regret = µk∗ − µk̂ . (2) To solve this sequential decision problem, we introduce a GP over the policy returns and OPE estimates (subsection 2.2), and design the GP kernel over policies to model their correlated returns (subsection 2.3). Then we use these as part of a BO strategy (subsection 2.4).

heading: 2.2 Gaussian process over policy values
text: Given a limited interaction budget and a large number of policies, it is impossible to obtain accurate estimates of their values by executing them many times. Fortunately, similar policies tend to have similar values. Thus, propagating the information between them helps to improve the value prediction. Intuitively, the number of policies that we need to execute to find a good policy should depend on their diversity instead of the total number of candidate policies (see experiments supporting this in Fig. 7). Formally, we use GP to predict a joint distribution over the policy values.
<latexit sha1_base64="hXL3zgU/olAsQSEh0vHmEWgBWEM=">AAAB6HicbVDLSgNBEOyNrxhfUY9eBoOQU9gNoh4DXjwmkBckS5iddJIxs7PLzKwQlnyBFw+KePWTvPk3TpI9aGJBQ1HVTXdXEAuujet+O7mt7Z3dvfx+4eDw6PikeHrW1lGiGLZYJCLVDahGwSW2DDcCu7FCGgYCO8H0fuF3nlBpHsmmmcXoh3Qs+YgzaqzUKA+KJbfiLkE2iZeREmSoD4pf/WHEkhClYYJq3fPc2PgpVYYzgfNCP9EYUzalY+xZKmmI2k+Xh87JlVWGZBQpW9KQpfp7IqWh1rMwsJ0hNRO97i3E/7xeYkZ3fsplnBiUbLVolAhiIrL4mgy5QmbEzBLKFLe3EjahijJjsynYELz1lzdJu1rxbirVxnWp1sziyMMFXEIZPLiFGjxAHVrAAOEZXuHNeXRenHfnY9Wac7KZc/gD5/MHdK2MxQ==</latexit>
A GP is a stochastic process, that is an indexed collection of random variables, such that every finite collection of them has a multivariate normal distribution [54]. In our case, we assume that the policy value µ as a function of policy π follows a GP, with a kernel function K(πi, πj) over policies and mean m. Specifically, we consider the following generative process for the latent policy value µ(π), OPE estimate ρ, and episodic return r:
µ(π) ∼ GP(m(·),K(·, ·)), ρ ∼ N (µ(π), σ2ρ), r ∼ N (µ(π), σ2r). (3) We use a constant mean m without loss of generality. We assume a flat prior for the hyper-parameter m, and weakly informative inverse Gamma prior for the variance σ2ρ and σ 2 r . Our probabilistic model is illustrated in Fig. 2.
As we have a finite number of policies, the input space is a finite set X = {π1, . . . , πK}, where each index corresponds to a candidate policy. The joint distribution of the K policy values reduces to a multivariate normal distribution,
µ1, . . . , µK ∼ N (m1K ,K), with Kk,k′ = K(πk, πk′) , (4) where 1K is a K-length vector of ones. For each policy πk, we have one noisy observation from an OPE method ρk, and zero or multiple noisy episodic return observations rik. As the policies are related through the covariance matrix K, observing a return sample from one policy updates the posterior distribution of all of them. Given the OPE estimates ρ = [ρ1, . . . , ρK ] and Nk return observations rk = [r1k, . . . , r Nk k ] for each policy πk, the posterior distribution of the mean return is also a Gaussian,
µ1, . . . , µK |ρ, {rk}k ∼ N (m,Σ), with m = K(K + Λ)−1y, Σ = K−K(K + Λ)−1K.
yk =
( 1
σ2ρ + Nk σ2r )−1( ρk σ2ρ + 1 σ2r Nk∑ i=1 rik ) , Λkk = ( 1 σ2ρ + Nk σ2r )−1 , (5)
where Λ is a diagonal matrix with diagonal entries defined above.

heading: 2.3 Kernel
text: A key component of the GP model is the kernel that measures our belief about the policy correlation. To obtain a kernel we make a key assumption: Policies that are similar in the actions that they take, yield similar returns. Then, our insight is to measure the distance between the policies through the actions that each of them takes on a fixed set of states from the offline dataset (Fig. 3, left).
Formally, to measure the similarity between π1 and π2, we obtain the policy actions at a given state s and measure the distance between these two action vectors d(π1(s), π2(s)) 3. We then define the distance between the two policies as the average distance over the states of the offline dataset D,
d̄(π1, π2) = Es∼D d(π1(s), π2(s)) . (6) 3In this work, we use Euclidean distance for deterministic policies with continuous actions and Hamming
distance for discrete actions. An extension to stochastic policies is straightforward.
In practice we approximate d̄ using a randomly sampled subset of states D′ ⊂ D. An example of a pairwise distance matrix for a set of policies is shown in Fig. 3, middle. Rows and columns are the policies which are arranged first by the type of the training algorithm, then by hyperparameters and finally by the training steps (none of which is observed by our method). Note the block structure, which reflects the different nature of training algorithms. Smaller blocks of increasing policy distances correspond to policies with more training steps and the same hyperparameters.
Additionally, Fig. 3, right shows an example of policy distances to the behaviour policy. The values of policies produced by the same algorithm change gradually as a function of this distance. It provides indication that our kernel construction is informative for predicting the value of a policy.
Finally, we compute Matérn 1/2 kernel as: K(π1, π2) = σ2k exp(−d̄(π1, π2)/l), where σ and l are the trainable variance and length-scale hyperparameters. This is a popular kernel choice [56]. As the distance metric is the average distance over states, using it in this kernel is equivalent to multiplication of kernels on these states and it is a valid positive semi-definite matrix.

heading: 2.4 Active offline policy selection with Bayesian optimization
text: Under a GP formulation, we can employ BO [56] to search for the best policy efficiently. BO optimizes an unknown function µ(π) with π ∈ X using a limited number of function queries that render noisy observations. BO has been used successfully in many applications including experimental design [5], robotics [42, 43], hyper-parameter optimization [59, 27, 12], preference learning and interactive machine learning [9, 8]. The key component of a BO algorithm is an acquisition function ui(π) that balances exploration and exploitation when selecting a query. Widely used acquisition functions include upper confidence bounds (UCB), expected improvement (EI) and epsilon-greedy. Here, we simply use the UCB acquisition function [60, 30].4 Specifically, at every step i, we compute the score for a policy πk using
ui(k) = mk + √ βiΣkk, (7)
where βi is a constant depending on i. We then choose the policy with the highest score, ki = arg maxk ui(k), to execute next. After observing a new return rki , we update m and Σ in Eq. 5. We also update the hyper-parameters (m, σ2ρ, σ 2 r , σ 2 k and l) with the maximum a posteriori estimate (see details in the appendix).
Estimating the hyper-parameters with limited data is challenging in practice. Fortunately, in the active offline policy selection formulation, we can take advantage of the OPE estimates as prior observations to fit the hyper-parameters before launching the interactive optimization process. In some ablations where OPE is not available we instead execute 5 randomly sampled policies before fitting the hyperparameters.

heading: 3 Related work
text: Offline RL becomes increasingly popular due to its ability to leverage offline data to learn policies for RL in the real world. As a results, several benchmark datasets were recently announced [23, 19]. Offline RL methods include policy-constraint approaches that regularize the learned policy to stay close to the behavior policy [70, 21], value-based approaches that encourage more conservative estimates, either through regularization or uncertainty [35, 22], model-based approaches [73, 31], and standard RL methods used in off-policy RL [7].
OPE is studied extensively across many domains [39, 66, 29, 49]. It includes methods that use importance sampling [52] or stationary state distribution [41, 48, 67], value function [63, 44, 64], or learned transition models [74], as well as methods that combine these approaches [47, 15, 28, 18]. OPS is a related practical problem that receives increasing interest [50, 71, 20]. Our work has two notable distinctions in contrast to these prior works: 1) we consider similarities between policies when making value predictions, and 2) we focus on active setting, where a small number of evaluations in the environment can help to quickly improve the prediction of the best policy.
Similar to the way the kernel is computed based on policy’s actions, Policy Evaluation Networks [24] work considers a network fingerprinting that learns a policy embedding through the outputs of the
4Experiments with other acquisition functions can be found in the supplementary materials.
policy on probing states. The principal difference to our approach is that we only consider a very limited online policy evaluation budget (in the order of 100) while network fingerprinting needs many policy evaluations to train the policy embeddings (in the order of 100 000).
Applying BO in a finite set of indices reduces the problem to a Bayesian multiarmed bandit problem. Exploring the shared structure among arms using Bayesian models has been studied in the literature [60, 33, 26]. Our work builds upon the Bayesian bandit model to address the problem of policy selection additionally exploiting the offline dataset.
Active learning is used in RL for better exploration, learning reward or world models [17, 3, 14, 34, 6]. It has not been studied for the OPE or OPS problems to the best of our knowledge. Related to the policy representation through actions in the kernel, classifiers in active learning can be also characterised by their predictions on a subset of datapoints [32]. Furthermore, diversity in policies was measured through the action distribution for sampled states in population-based setting [51].

heading: 4 Experiments

heading: 4.1 Datasets and environments
text: In our experiments we consider three domains (Fig. 4) with different properties. Two domains are standard benchmarks in RL for continuous and discrete control and another one is motivated by the challenges of evaluating control policies in robotics. For each domain we describe the environments, datasets for offline RL and OPE, and sets of policies below. More details are provided in Appendix B.
DeepMind Control Suite (dm-control) This is a standard set of continuous control environments [65]. It contains 9 tasks of locomotion and manipulation, some of which are shown in Fig. 4, left. The observations are the state of the MPD, including joint angles and velocities. The dimensionality and semantics of the continuous actions vary across tasks. We use the offline RL policies from DOPE [20] (a benchmark for OPE and OPS) that are trained using the dataset from the RL Unplugged benchmark [23]. For each task, there are up to 96 policies with various algorithms (D4PG [7], ABM [57], and CRR [70]), hyperparameters, and number of training steps.
Manipulation Playground (MPG) This is a simulated robotics environment. The task is to control a Kinova Jaco arm with 9 degrees of freedom. Joint velocity control of 6 arm and 3 hand joints is used. The policies are learnt from proprioception and image input from frontal and in-hand cameras of 64× 64 dimensions. We perform 4 manipulation tasks within a 20× 20 cm basket (Fig. 4, middle). The dataset is generated by running an online RL algorithm as in [70] and the policies are generated as in [50]. For each task, there are up to 256 policies trained with various algorithms (including BC, D4PG [7], and CRR [70]), and hyperparameters.
Atari This is a popular benchmark with discrete actions in online and offline RL [23]. For each of 3 games we trained 210 policies on an offline dataset [23] from high dimensional pixel observations with six offline RL methods, including Double DQN [68], CQL [35], BCQ [21], REM [2] and BVE [22]. As the range of returns is quite different from the other two domains, we scale the discounted returns to have approximately the same mean and variance based on the offline RL dataset.
We obtain OPE estimates for each policy by running Fitted Q-Evaluation (FQE) [44, 36] on the offline RL data. FQE is the default OPE method used in our experiments unless stated otherwise.

heading: 4.2 Experimental setting and implementation
text: To evaluate the policy selection procedure in each experiment we randomly select a subset of K policies out of all trained policies. We set K = 50 for dm-control and K = 200 for MPG and Atari. Then, we repeat each experiment 100 times, and report the average results and the standard deviation of the mean estimate. As a performance metric we monitor the simple regret (Eq. 2) as a function of the number of executed episodes. The algorithms are progressively evaluated when each trajectory is added. To make the results comparable across domains, we first compute the gap between the best and the worst performing policy in each environment across all the policies. Then, we normalise the scores in each experiment by dividing them by this gap. The results by environment are discussed in the appendix.
Independent policy model To study the contribution of modeling the policy correlation with GP, we consider a baseline model with independently distributed value µk across policies. The data generating process for the k-th policy follows a hierarchical Gaussian distribution:
rik iid∼ N (µk, σ2k,r),∀i, with µk ∼ N (ρk, σ2k,ρ) , (8)
where we assume a weakly informative inverse Gamma prior for variance σ2k,r and σ 2 k,ρ as in GP. We refer to this model as independent policy model (IND). Such model can be used in combination with any policy sampling strategy.
All online and offline RL algorithms, as well as offline policy evaluation algorithms in this work are implemented with Acme [25] and Reverb [11], and run on GPUs on an internal cluster. We implement GP and IND together with all BO algorithms using the same TensorFlow [1] codebase, and run all the policy selection experiments using CPUs. Details about the tasks and policies, hyperparameter values, and additional BO algorithms and ablations are presented in the appendix.

heading: 4.3 Comparison to the existing methods
text: In our main experiments we would like to verify that the proposed method performs better than two strategies commonly employed in practice (see section 1). To this end, we compare:
1. A-OPS: Active-Offline Policy Selection, our proposed method that uses OPE estimates to bootstrap GP learning and performs UCB policy sampling as discussed in section 2. 2. OPE: standard offline policy selection technique that selects the policy with the highest OPE estimate [50]. 3. IND+UNIFORM: standard online selection where we execute policies uniformly and select the one with the highest return belief (estimated with independent policy model IND).
Fig. 5 presents the simple regret as a function of the number of executed trajectories. The results are averaged across 9 tasks from dm-control, 4 tasks from MPG and 3 games from Atari. Our proposed method A-OPS combines offline and online policy evaluations. It quickly outperforms OPE and improves with more trajectories. A-OPS estimate of the best policy may be erroneous initially due to noisy reward samples (e.g., in MPG), but it quickly updates, thus allowing for significant improvements over OPE only after a few actively selected trajectories.
Table 1: Design of the ablation experiments across three axes: the use of OPE, GP or independent policy model, active policy selection or uniform sampling. We refer to our method that combines all the components as A-OPS.
No OPE With OPE
Independent GP Independent GP
Uniform IND+UNIFORM GP+UNIFORM IND+UNIFORM+OPE GP+UNIFORM+OPE UCB IND+UCB GP+UCB IND+UCB+OPE A-OPS
0.0
0.2
0.4
sim pl
e re
gr et
W ith
O PE
dm-control
0.0
0.2
0.4
MPG OPE Ind+Uniform+OPE Ind+UCB+OPE GP+Uniform+OPE A-ops
0.0
0.1
0.2
Atari
0 100 200 # trajectories
0.0
0.2
0.4
sim pl
e re
gr et
No O
PE
0 200 400 # trajectories
0.0
0.2 0.4 Ind+Uniform Ind+UCB GP+Uniform GP+UCB
0 200 400 # trajectories
0.0
0.1
0.2
Figure 6: Ablations. In three domains, we ablate each of three components of our approach. 1) The use of OPE (with OPE in the first row and without in the second). 2) The choice of policy model: GP (purple and red) and independent (green and blue). 3) The policy selection strategy: active (dark, solid line) and uniform sampling (lighter, dashed line). In general, active is better than uniform, methods that exploit policy correlation are better than methods that do not, and methods that use an OPE prior are better than those that do not.
When comparing offline and online policy selection we notice that for the range of considered budgets, online policy selection IND+UNIFORM performs worse than OPE. It suggests that single trajectory returns are noisier than OPE estimates in these environments (more discussion on this is in the appendix). Naturally, we expect that given a larger budget the online baseline will eventually surpass its offline counterpart in all domains.

heading: 4.4 Component ablations
text: We are not aware of other works on active offline policy selection. Thus, in this section we investigate alternative solutions that help us to understand the importance of the proposed components of A-OPS. We design our ablation studies to test the contribution of the three components of A-OPS:
1) OPE: initialisation of algorithms with or without OPE estimates. 2) active learning: active policy selection (UCB) or uniform sampling (UNIFORM). 3) policy kernel: GP model or independent model baseline (IND).
Tab. 1 summarises the 8 possible combinations of the components. We refer to our full method (GP+UCB+OPE) simply as A-OPS. We also include the pure offline OPE estimates as before.
The results with all the baselines in two domains are presented in Fig. 6. First of all we observe that A-OPS outperforms all other methods across a wide range of budgets, thus, all the components are crucial for the performance of our method and excluding any of them leads to significantly worse results. Next, we answer three questions about each component individually.
1. How important is it to incorporate OPE estimates? Across all the experiments, incorporating OPE estimates (top row of Fig. 6) always considerably improves the results compared to starting policy selection from scratch (bottom row). For example, in MPG when A-OPS starts with OPE, it is able to cut the OPE regret by more than half before executing 100 trajectories. For this budget, GP+UCB, which starts from scratch, does not reach the same regret as OPE.
2. How informative is our kernel? To understand the value of modelling the correlation between policies through the kernel, we compare the GP method with the independent policy model (IND). In Fig. 6 purple and red lines use GP as the policy model and green and blue use the IND model. In the vast majority of settings (11 out of 12) the use of a kernel significantly improves the results and in one setting (IND+UNIFORM+OPE vs GP+UNIFORM+OPE in MPG domain) they perform on par. We conclude that the kernel is a key ingredient for improving data efficiency.
3. How important is the selection strategy? The active selection of policies is generally beneficial for identifying a good policy. For this we refer to the results of dashed lines that correspond to UNIFORM policy selection and solid lines that correspond to the use of UCB. Using IND+UCB yields a high regret in the initial exploration stage, but it improves substantially over IND+UNIFORM later. Moreover, incorporating OPE estimates and a kernel (resulting in A-OPS) significantly shortens the exploration stage.

heading: 4.5 More ablation studies
text: Incorporating the OPE estimates is clearly beneficial for policy selection. Here, we further study: How does the performance depend on the quality of OPE estimates? Fig. 7, left compares A-OPS using estimates from different OPE algorithms in DOPE [20] on dm-control with a small budget of 50 trajectories. A-OPS always improves upon the corresponding OPE estimate, and the performance clearly depends on the quality of the associated OPE estimate. The performance of GP+UCB with 50 trajectories that does not rely on any OPE metric is shown as a horizontal line. Some OPE estimates (e.g., IS, VPM and DICE) have large estimation errors and even a negative correlation with the ranks of policies [20]. As expected, A-OPS is negatively impacted by poor OPE estimates and ignoring them could yield better results. It should be possible to correct for this to a greater extent with more online observations. For now we follow the recommendation to use the FQE estimates.
In subsection 2.2 we hypothesised that A-OPS method should be data efficient as it would scale with the diversity of policies and not their number. We next try to verify it and answer the questions How does the policy selection algorithm scale with the growing number of candidate policies? In Fig. 7, right we show the simple regret for A-OPS and IND+UNIFORM in the MPG domain when we sample 25, 50, 100 and 200 policies. We see that when the number of policies is not very high (25), the algorithm that treats policies independently and samples them at uniform (IND+UNIFORM-25) can closely approach the performance of our method A-OPS-25 after 100 trajectories. However, when the number of policies grows, this small interaction budget is not enough for the naive algorithm: IND+UNIFORM-200 significantly degrades its performance. At the same time, A-OPS-200 is minimally affected by the increase in the number of policies as the diversity of the policies stays approximately the same.

heading: 4.6 Qualitative results
text: Finally, to understand the behaviour of A-OPS method we show its progress for one of the experiments with Robotank game in Fig. 8. For visualisation purposes, we use only 20 policies which are ordered by their ground truth returns (not observed by the method). Perfect value predictions would be aligned along the diagonal line. We show the initial OPE values, current selection of the best policy, current observation of the executed policy, past observations and the algorithm prediction with its standard deviation for steps 1, 10, 50, and 100. The A-OPS prediction is determined by the OPE observations and past observations of the policy returns. The kernel reinforces smoothness of the function. We can see that the initial selection is dominated by OPE prediction and the selected policy is quite far from the best policy. With more online observations, the predictions improve. Our algorithm is data-efficient because 1) it samples more promising policies more frequently (more observations with higher scores), 2) the knowledge from one policy is propagated to the policies related by the kernel (predictions for some policies are updated without any new observations of their values).

heading: 5 Conclusions
text: In this paper, we introduced the active offline policy selection problem. We proposed a BO solution that integrates OPE estimates with evaluations obtained by interacting with the environment. We also designed a novel kernel that models correlation over policies. Experiments on three control domains proved that our solution is very effective. Furthermore, our experiments ablated the different components of the solution, highlighting the value of modelling correlation via the kernel, active selection, and incorporation of OPE estimates in the observation model.
Limitations and future work Having tested the method and demonstrated its efficacy in simulation, our future plan is to conduct experiments in real domains.
The A-OPS approach could be of use in many safety critical applications. This work does not deal with this question explicitly as we assume safety constraints to be implemented directly on the hardware. It means that if an unsafe policy is attempted, the safety controller would terminate the policy before reaching the end of an episode, resulting in low reward and thus discouraging A-OPS from trying this policy again. Alternatively, we could alleviate this limitation by applying constrained BO techniques [62] to search for the best policy without violating safety constraints.
In this work, we explore the policy structure based on the offline dataset, and use every new trajectory only to obtain a noisy observation of policy values. This allows a clean Bayesian optimization formulation. Going forward, more information could be extracted from the new trajectories, for example, to further improve OPE estimates, GP kernel, or policies themselves. However, using this information would make the solution much more complex and we may need to take the additional computational cost into account. Finally, to construct a better policy kernel, we may investigate how to measure the policy similarity on the most informative subset of states in the offline dataset.

heading: Acknowledgements
text: We would like to thank Bobak Shahriari for the useful discussions and feedback on the paper.

Review summaries:
This paper considers a novel problem setting of "Offline Policy Selection", whereby there is access to an initial set of policies trained offline, and an algorithm must choose a policy to deploy in the online simulated environment. I am giving this paper the score of weak reject because despite introducing an important problem setting, I am not convinced the method is actually particularly effective. Hopefully this can be clarified during the rebuttal phase. As such, the emphasis of this review is on areas for improvement.

This paper introduces the problem of active offline policy selection which consists of selecting the best policy from a set of candidates given a small budget to interact with the environment. The paper proposes a solution (A-OPS) based on Bayesian optimization using a kernel function to relate candidate policies and generalize between them. The proposed solution is thoroughly evaluated on simulated robotic control environments using benchmark datasets from the off-policy evaluation literature and several ablations are included to rationalize the choices of the proposed method.

This paper introduces a problem of active offline policy selection, where the goal is to select the best-performing policy among the fixed K policies using an abundant offline dataset and a limited additional interaction with the environment. This work proposes to solve this problem by Bayesian optimization (BO) with a Gaussian process (GP), where the input and output of the objective function are a policy and its expected return respectively, and function queries render noisy observations. To facilitate GP in this problem, a new GP kernel is introduced, which takes two policies as input, and its output is computed based on the similarities of policy distribution for each state in the offline dataset. Also, to make BO more data-efficient, off-policy evaluation (OPE) estimates through the offline dataset are used for a warm start of BO. Finally, UCB acquisition function is used for active policy selection. Experimental results show that the proposed BO with OPE-based warm start significantly outperforms the baselines. Also, ablation studies confirm that UCB selection is better than uniform selection, GP is better than methods that do not consider policy correlation, and using OPE is better than not using it. 

This paper works on the policy selection problem in a setting characterized by features: available logged data and allowing a limited number of online interactions with the environment. To do so, this work proposes to use Bayesian optimization to combine existing OPE techniques and online interactions. For the setting with large number of policies, a kernel function is introduced to take the correlation between policies, which indicate the performances of different policies.