Active Offline Policy Selection
Authors: Ksenia Konyushkova
Yutian Chen
Thomas Paine
Caglar Gulcehre
Cosmin Paduraru
Daniel J Mankowitz
Misha Denil
Nando de Freitas
Venue: NeurIPS 2021 Poster (2021)
Keywords: policy selection
offline reinforcement learning
off-policy policy evaluation
active learning
Bayesian optimisation

Abstract: This paper addresses the problem of policy selection in domains with abundant logged data, but with a restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and recommendation domains among others. Several off-policy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation in the real environment. Yet, large amounts of online interactions are often not possible in practice. To overcome this problem, we introduce active offline policy selection --- a novel sequential decision approach that combines logged data with online interaction to identify the best policy. This approach uses OPE estimates to warm start the online evaluation. Then, in order to utilize the limited environment interactions wisely we decide which policy to evaluate next based on a Bayesian optimization method with a kernel function that represents policy similarity. We use multiple benchmarks with a large number of candidate policies to show that the proposed approach improves upon state-of-the-art OPE estimates and pure online policy evaluation.

Active Offline Policy Selection
Ksenia Konyushkova* Deep kksenia@deepmind.com com
Yutian Chen"
Tom Le Paine
DeepMind yutiancedeepmind.comm
DeepMind tpaine@deeppind.ccomm com
Caglar Gulcehre DeepMind caglargddeepmindd.cmm
Cosmin Paduraru DeepMind padurarruddeppmind. com
Daniel J Mankowitz
DeepMind
dmankooitzzódeepminnd com
Misha Denil DeepMind mdeni ndenil@deepmind.
Nando de Freitas
DeepMind
nandolefreeitas@deeemmnn..d com
Abstract
This paper addresses the problem of policy selection in domains with abundant logged data, but with a restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and recommendation domains among others. Several off- policy evaluation (OPE) techniques have been proposed assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation in the real environment. Yet, large amounts of online interactions are often not possible in practice. To overcome this problem, introduce active offline policy selection decision approach that combines logged data with online interaction to identify the best policy. This approach uses OPE estimates warm start the online evaluation. Then, in order to utilize the limited environment interactions wisely we decide which policy to evaluate next based on Bayesian optimization method with a kernel function that represents policy similarity. We use multiple benchmarks with large number of candidate policies to show that the proposed approach improves upon state-of-the-an OPE estimates and pure online policy evaluation
novel sequential
Introduction
Reinforcement learning (RL) has recently proven to be successful in a range of applications from computer and board games 46
55 58) to robotics and chip design [37] 10 45). However, many challenges of real-world systems still prevent RL from being applied scale in practice [16]. One of
the limiting factors real applications is that environment interactions are often expensive. Thus, training and evaluating agents becomes prohibitively slow and costly. Offline RL attempts to address this problem by training agents on dataset without environment interactions 23 This enables researchers train policies with multiple algorithms and different hyperparameter settings (50)
single dataset. The best policy can then be chosen with off-policy policy evaluation (OPE) approaches
53 40
Fig. I left). These techniques attempt estimate the expected agent performance by relying
on the same pre-recorded dataset 228 36 72
"qqual contribution
TThe paper website is at https://sites.. www.llleesss.ogeoeeeoooooooooeee google. wwt..///rr////e///ttteeeeeooo/// http:///e///eeeaooooooooooo/oo// httpp::/ggttub..comm// http://ccc/cceeeee/aaaaiiiiiiiee wwtp://cccttiippeeeeeeeeeeeeeeee
and the code is
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Data
Data
Env
T
T
Active
2
Offline
Offline
Policy
π
π
Policy
Selection
Selection
π KK-
π T
Figure 1: Left: Offline policy selection attempts to choose the best policy from set of policies, given only dataset of logged experiences. Right: Active offline policy selection additionally assumes restricted environment interaction budget which is assigned intelligently to evaluate the most promising policies.
Unfortunately, OPE estimates are not yet precise enough to be relied upon for choosing which agent to deploy. Moreover, OPE is intrinsically limited by the pre-recorded dataset: When there is distribution shift between a trained policy and behavioural policy, i.e., when the trained policy behaves very differently from the data collection policy, OPE is imprecise (38) In many applications,
however, subset of policies can be evaluated online prior to final deployment [23] T9 even though this process can be hard and laborious. For example, in robotics small subset of offline RL policies can be tested on the real hardware. Another example is recommender systems, which allow for restricted policy evaluation via A/B tests on small fraction of user traffic.
In this paper, we introduce the active offline policy selection problem. The goal is to identify the best policy where we benefit from both offline evaluation with logged data and online evaluation with limited interaction (Fig. right). In our approach we advance a solution based on Bayesian optimization (BO). It entails learning Gaussian process (GP) surrogate function that maps policies to their expected returns. Then, we build on the GP statistics to construct an acquisition function to decide which policy test next. To make BO successful in this problem setting, our approach has two key features. First, we incorporate existing OPE estimates as additional noisy observations Fig.. 2). This allows us to warm start online policy evaluation and to overcome the difficulties of GP hhper-paramete optimisation at the start. Second, we model correlation between policies through kernel function based actions that the policies take in the same states of the environment Fig. 3). This makes our method data efficient [26] as the information about the performance of one policy informs about the performance of similar behaving policies without costly execution in the environment. It is particularly valuable when the number candidate policies is large (or even larger than the interaction budget).
The contributions of this paper are as follows.
1. We introduce the active offfine policy selection problem. This problem is important in practice, but so far has been studied the literature to the best of knowledge.
2. We build BO solution with an extended observation model to integrate both OPE estimates and interaction data. The use OPE estimates enables us to warm-start learning GP.
3. We propose novel GP kernel capture the dependency between the policies through the actions that they take. As result, our method infers the value of one policy from those of similar policies.
The rest of this paper is organized as follows. Section I.1 gives background on offline policy evaluation and selection. In In|section 2] propose GP model for policy value estimation, explain how we approach the sequential decision making with BO and introduce a novel kernel based on the similarity of policy outputs. Section discusses the related work. Section 4 shows that active policy evaluation can improve upon the OPE after merely a few interactions, thanks to the kernel that ensures data-efficiency. Additionally, method works reliably with OPEs of varying quality and it scales well with the growing number candidate policies. The paper is concluded with a discussion in section
1.1 Off-policy policy evaluation and selection
We consider Markov decision process (MDP) defined by a tuple (S, A, T, R, do. S, action space A, transition distribution T(s')s (s, a). reward function R(s, a). initial state distribution
7), with state space

---

Episodic return OPE estimate p Posterior of J
Policy a
Figure 2: Gaussian process of the policy value function. We observe OPE estimates for all policies and noisy episodic returns for of them. Neighboring policies (measured kernel K.) have similar values, and the posterior variance lower where there are more observations. Policies are aligned ID visualization purposes, but practice they reside in high dimensional space.
do(s), and discount factor [0, 1]. A policy (als)) maps from S to A. The value of policy is measured by the expected sum of discounted rewards:
Ag E
Σ 7 M(ss, as) with so ~ do(-), ar ~ T(- T(-)))), 8041 ~ ()))),,a)).
(1)
Typically, estimating the value of a given policy requires executing the policy in the environment
many times. However, practical constraints make it difficult to run many policies of unknown quality in the environment. To address this, OPE estimates a value Ar using dataset of trajectories D collected by separate behavior policy Tp. In offline policy selection (OPS) the task is to select the policy with the highest value from a set of policies Tis n2; ... T given access only to a dataset D 1g left). A straightforward approach is to select the policy with the highest OPE estimate
Am....... Anx 20]. and alternative approaches can be used depending on the quality metric [71].
2
Active offline policy selection
2.1 Problem definition
We now formally define the active offline policy selection problem. Suppose we are given set of K candidate policies Tk. S S K. Denote by μı the unknown policy value and by Pk an OPE estimate computed from an offline dataset. At every step i, can choose policy k; to execute the environment once and observe the trajectory with (discounted) episodic return тı, which is noisy sample of the true policy value: [R[r:] = jk::- We would like to find the policy with the highest value, k* arg max (i, with as trajectories (budget) as possible. Policy selection algorithm estimates the mean as my and recommends a policy that maximizes it: = arg max; mg. Simple to the best policy:
regret of the recommended policy measures how close
regret P4 P4
To solve this sequential decision problem, we introduce GP over the policy returns and OPE estimates (subsection 2.2), and design the GP kernel over policies to model their correlated returns (subsection 2.3). Then we use these part of BO strategy (subsection 2.4).
2.2 Gaussian process over policy values
Given limited interaction budget and a large number of policies, is impossible to obtain accurate estimates of their values by executing them many times. Fortunately, similar policies tend to have similar values. Thus, propagating the information between them helps to improve the value prediction. Intuitively, the number of policies that we need to execute to find good policy should depend on their diversity instead of the total number candidate policies (see experiments supporting this in Fig. Formally, we use GP to predict joint distribution over the policy values.

---

22
CRR ABM D4PG
Policy return
30
CRR
RABM
20
D4PG
30
m
d( (T), m22 (d
Distance to behaviour policy
increasing training stops
Figure 3: Left: The distance between policies स and T2 computed as average distance across pairs action vectors that are selected by each policy on set of states, Middle: Pairwise distance matrix between policies on numanoid-rum task. High distance is shown in yellow and low distance in dark blue. The 3 block structure of the matrix reflects the 3 types of training algorithm. Smaller blocks correspond to the increasing number of training steps with the same hyperparameters. Right: Policy value (μz) versus action distance to the behavior policy d(m, dT). Policies the same training algorithm are in same color.
GP a stochastic process, that is an indexed collection of random variables, such that every finite collection of them has a multivariate normal distribution (54] In our case, we assume that the policy value 1r as function of policy ı follows GP, with a kernel function ([,m,, Ty) over policies and
mean m. Specifically, we consider the following generative process for the latent policy value μ(T), OPE estimate p. and episodic return
(T) ~ GP(m()),K(:)))
C,N(( (~N(U(TT),
(3)
We use a constant mean m without loss of generality. We assume a flat prior for the hyper-paramette m, and weakly informative inverse Gamma prior for the variance of and of. Our probabilistic model is illustrated in Fig. 2
As have finite number of policies, the input space is finite set X' { Tx) where each index corresponds to a candidate policy. The joint distribution of the policy values reduces to a multivariate normal distribution,
Mi.... HK ((11, K), with K = K(TTK
where 1 K-length vector of ones. For each policy Ty. we have one noisy observation from an OPE method pa. and zero or multiple noisy episodic return observations r-k As the policies are related through the covariance matrix K, observing return sample from one policy updates
the posterior distribution of all of them. Given the OPE estimates p (p,,...,PM ,K] and N₂. return observations r&
Nu for each policy me, the posterior distribution of the mean return is [r]] ......, r
also a Gaussian,
P1...., :.......kle. P. fru ~ (mm, E), with
K(K A)
Σ = K - K(KK + +A)
No
Pk
N
Yk
ARR
(5)
where A a diagonal matrix with diagonal entries defined above.
2.3 Kernel
A key component of the GP model is the kernel that measures our belief about the policy correlation. To obtain a kernel make key assumption: Policies that are similar in the actions that they take, yield similar returns. Then, our insight is to measure the distance between the policies through the actions that each of them takes on fixed set of states from the offline dataset (Fig. 3 left).
Formally, to measure the similarity between Mi and T2, we obtain the policy actions at a given state and measure the distance between these two action vectors di ( (s), (2) We then define the distance between the two policies as the average distance over the states of the offline dataset D,
d( ( T2) E,-D d(n [T] (s), T2 T2))))
II this work, we use Euclidean distance for deterministic policies with continuous actions and Hamming distance for discrete actions. An extension stochastic policies is straightforward.

---

In practice we approximate d using randomly sampled subset of states D' C C.
An example of pairwise distance matrix for set of policies is shown in Fig. 3 middle. Rows and columns are the policies which are arranged first by the type of the training algorithm, then by hyperparameters and finally by the training steps (none of which is observed by our method). Note the block structure, which reflects the different nature of training algorithms. Smaller blocks of increasing policy distances correspond to policies with more training steps and the same hyperparameters.
Additionally, Fig. 3] right shows an example of policy distances to the behaviour policy. The values of policies produced by the same algorithm change gradually indication that our kernel construction is informative for predicting the value of a policy.
function of this distance. It provides
Finally, we compute Matérn 1/2 kernel as: K(T,, x2) = of exp(-d(=1, m2)/l), where a and I are the trainable variance and length-scale hyperparameters. This a popular kernel choice [56]. As the distance metric is the average distance over states, using it in this kernel is equivalent to multiplication of kernels on these states and it a valid positive semi-definite matrix.
2.4 Active offiine policy selection with Bayesian optimization
Under a GP formulation, we can employ BO (56) to search for the best policy efficiently. BO optimizes an unknown function (u(z) with π E X using a limited number of function queries that render noisy observations. BO has been used successfully in many applications including experimental design [5]. robotics 442 43 hyper- parameterr optimization (59 27 12], preference learning and interactive machine learning (9 g The key component of BO algorithm is an acquisition function uy (T) that balances exploration and exploitation when selecting query. Widely used acquisition functions include upper confidence bounds (UCB), expected improvement (EI) and epsilon-greeds Here, we simply use the UCB acquisition function 660 30 Specifically, at every step we compute the score for policy nę using
Us (k) = m + VB
where 3r is a constant depending on i. We then choose the policy with the highest score, K; = arg max; u; (Re), to execute next. After observing new return rk, we update m and Σ in Eq. 5 We also update the details in the appendix).
(m, of of, of and 1) with the maximum a posteriori estimate (see
Estimating the pypeeeparameters with limited data is challenging in practice. Fortunately, in the active offline policy selection formulation, we can take advantage of the OPE estimates as prior observations to fit the dyper-parameter before launching the interactive optimization process. In some ablations where OPE not available we instead execute 5 randomly sampled policies before fitting the hyperparameters.
3 Related work
Offline RL becomes increasingly popular due to its ability leverage offline data to learn policies for RL in the real world. As a results, several benchmark datasets were recently announced [3] 19] Offline RL methods include policy-constraint approaches that regularize the learned policy to stay close to the behavior policy 770 21.. value-based approaches that encourage more conservative
estimates, either through regularization or uncertainty [35 22 model-based approaches
773
and standard RL methods used in off- policy RL 7
OPE is studied extensively across many domains
13
66
29
49
It includes methods that use value function 63
importance sampling [52) or stationary state distribution
[4
48
67
44
64 or
learned transition models (74), as well as methods that combine these approaches 47
15
28
18
OPS is a related practical problem that receives increasing interest [1 20 Our work has two
notable distinctions in contrast to these prior works: we consider similarities between policies when making value predictions, and 2) we focus on active setting, where small number of evaluations in the environment can help to quickly improve the prediction of the best policy.
Similar to the way the kernel is computed based on policy's 's actions, Policy Evaluation Networks [4] work considers network fingerprinting that leams policy embedding through the outputs of the
"xxpermments with other acquisition functions be found in the supplementary materials.
5

---

Cheetah run
Maniputstor Insert Deg
tumano
Box
Slide
Robotank
Figure 4: Environment used in the experiments. 9 environments from the dm-control domain (3 examples on the left), 4 from the MPG domain (2 examples in the middle), and 3 from Atari domain (1 example on the right).
policy on probing states. The principal difference to our approach is that we only consider a very limited online policy evaluation budget (in the order of 100) while network fingerprinting needs many policy evaluations to train the policy embeddings (in the order of 100 000).
Applying BO finite set of indices reduces the problem Bayesian multiarmed bandit problem. Exploring the shared structure among arms using Bayesian models has been studied in the literature Our work builds upon the Bayesian bandit model address the problem of policy selection additionally exploiting the offline dataset.
60 33 26
Active learning used in RL for better exploration, learning reward world models 17 It has not been studied for the OPE or OPS problems to the best of our knowledge. Related to the policy representation through actions in the kernel, classifiers in active learning can be also characterised by their predictions on a subset of datapoints [32]. Furthermore, diversity in policies was measured through the action distribution for sampled states population- based setting (51].
4 Experiments
4.1 Datasets and environments
In our experiments we consider three domains (Fig. 4 with different properties. Two domains are standard benchmarks in RL for continuous and discrete control and another one is motivated by the challenges of evaluating control policies in robotics. For each domain we describe the environments, datasets for offline RI. and OPE, and sets of policies below. More details are provided in Appendix B
DeepMind Control Suite (dm-control) This is standard set of continuous control environ- ments 65 It contains tasks of locomotion and manipulation, some of which are shown in anFi.. 4 left. The observations are the state of the MPD, including joint angles and velocities. The dimen- sionality and semantics of the continuous actions vary across tasks. We use the offline RL policies from DOPE (20) (a benchmark for OPE and OPS) that are trained using the dataset from the RL Unplugged benchmark (23]. For each task, there are up to 96 policies with various algorithms (D4PG 77,, ABM (57]. and CRR (70), hyperparameters, and number of training steps.
Manipulation Playground (MPG) This is simulated robotics environment. The task is control a Kinova Jaco arm with 9 degrees of freedom. Joint velocity control of 6 arm and 3 hand joints is used. The policies are learnt from proprioception and image input from frontal and in-hand cameras of 64 x 64 dimensions. We perform 4 manipulation tasks within a 20 x 20 cm basket (Fig. 4 middle). The dataset is generated by running an online RL algorithm as in (70] and the policies are generated as in [50]. For each task, there are up to 256 policies trained with various algorithms (including BC, D4PG 77, and CRR (70), and hyperparameters.
Atari This is popular benchmark with discrete actions in online and offline RL 23 For each of 3 games we trained 210 policies on an offline dataset (3 from high dimensional pixel observations with six offline RL methods, including Double DQN (68). CQL [35, BCQ (11, REM [2] and BVE [22]. As the range of returns is quite different from the other two domains, we scale the discounted returns have approximately the same
and variance based on the offline RL dataset.
We obtain OPE estimates for each policy by running Fitted Q-Evaluation (FQE) 4 3 on the
offline RL data. FQE is the default OPE method used in experiments unless stated otherwisc.

---

dm-cone control
MPG
Atari
simple regret
0.2
OPE
Ind+Uniform
0.2
0.1
A-ops
400
trajertories
trajectories
trajectories
Figure 5: Comparison of A-OPS approach, offline policy selection with OPE, online policy selection with on dm-control, MPG and Atari environments. The results are averaged across 9, 4 and 3 tasks and 100 experiments in each of them. Shaded areas show standard deviation of the mean. Our method A-OPS quickly surpasses the baselines and achieves the lowest regrets.
4.2 Experimental setting and implementation
To evaluate the policy selection procedure each experiment we randomly select a subset of K policies out of all trained policies. We set K 50 for dm-control and K 200 for MPG and Atari. Then, we repeat each experiment 100 times, and report the average results and the standard deviation of the mean estimate. As performance metric we monitor the simple regret (Eq. 2) as function of the number of executed episodes. The algorithms are progressively evaluated when each trajectory is added. To make the results comparable across domains, we first compute the gap between the best and the worst performing policy in each environment across all the policies. Then, we normalise the scores each experiment by dividing them by this gap. The results by environment are discussed in the appendix.
Independent policy model To study the contribution of modeling the policy correlation with GP, we consider a baseline model with independently distributed value μı across policies. The data generating process for the k-th policy follows hierarchical Gaussian distribution:
sid Tk ~
(uk, ofr) Vi, with μ4 ~ ((k,, ok.)),
(8)
where we assume a weakly informative inverse Gamma prior for variance of and 22 as in GP. We refer to this model as independent policy model (IND). Such model can be used in combination with any policy sampling strategy.
All online and offline RL algorithms, as well as offline policy evaluation algorithms in this work implemented with Acme (25) and Reverb [II, and run on GPUs on internal cluster. We implement GP and IND together with all BO algorithms using the same TensorFlow [ codebase, and run all the policy selection experiments using CPUs. Details about the tasks and policies, hyperparameter values, and additional BO algorithms and ablations are presented in the appendix.
4.3 Comparison to the existing methods
In our main experiments we would like to verify that the proposed method performs better than two strategies commonly employed in practice (see I). To this end, compare:
A-OPS: Active-Offline Policy Selection, our proposed method that uses OPE estimates to
bootstrap GP learning and performs UCB policy sampling as discussed in infsection 2)
2. OPE: standard offline policy selection technique that selects the policy with the highest
OPE estimate [50].
3. UND+ORI standard online selection where we execute policies uniformly and select the one with the highest return belief' (estimated with independent policy model IND).
Fig. 5 presennts the simple regret as a function of the number of executed trajectories. The results are averaged across 9 tasks from dm-coonrol, 4 tasks from MPG and 3 games from Atari. Our proposed method A-OPS combines offline and online policy evaluations. It quickly outperforms OPE and improves with more trajectories. A-OPS estimate of the best policy may be erroneous initially due to noisy reward samples (e.g., in MPG), but quickly updates, thus allowing for significant improvements over OPE only after few actively selected trajectories.
7

---

Table 1: Design the ablation experiments three the of OPE, GP independent policy model, active policy selection uniform sampling. We refer to our method that combines all the components as A-OPS.
No OPE
With OPE
Independent
GP
Independent
GP
Uniform
IND+UNIFORM IND+UCB
GP+UNIFORM GP+UCB
IND+UNIFORM+OPE IND+UCB+OPE
GP+UNIFORM+OPE A-OPS
UCB
din-control
MPG
Atari
0.4
0.2
With OPE
simple regret
Ind+Unifor++OPP
Ind Mnd+UCB++Ou
0.2
0.2
0.1
GP+Uniform+ OPE
A-ops
0.0
0.0
0.0
0.4
0.4
0.2
simple regret
No OPE
Ind+Uniffora
Ind+UCB
0.2
0.1
GP+Uniform
GP+UCB
0.0
0.0
100
200
200
400
200
400
# trajectories
trajectories
trajectories
Figure 6: Ablations. In three domains, we ablate each of three components of our approach. 1) The use of OPE (with OPE in the first row and without in the second). 2) The choice of policy model: GP (purple and red) and independent (green and blue). 3) The policy selection strategy: active (dark, solid line) and uniform sampling (lighter, dashed line). In general, active better than uniform, methods that exploit policy correlation are better than methods that do not, and methods that use an OPE prior are better than those that do not.
When comparing offline and online policy selection we notice that for the range of considered budgets, online policy selection IND+UNIFORM performs worse than OPE. It suggests that single trajectory returns are noisier than OPE estimates these environments (more discussion on this in the appendix). Naturally, expect that given larger budget the online baseline will eventually surpass its offline counterpart in all domains.
4.4 Component ablations
We are not aware of other works on active offline policy selection. Thus, in this section we investigate alternative solutions that help us to understand the importance of the proposed components of A-OPS. We design our ablation studies to test the contribution of the three components of A-OPS:
1) OPE: initialisation of algorithms with or without OPE estimates.
2) active learning: active policy selection (UCB) or uniform sampling (UNIFORM).
3) policy kernel: GP model or independent model baseline (IND).
Tab. I summarises the 8 possible combinations of the components. We refer to our full method (GP+UCB+OPE) simply as A-OPS. We also include the pure offline OPE estimates as before.
The results with all the baselines in domains are presented in Fig. First of all we observe that A-OPS outperforms all other methods across a wide range of budgets, thus, all the components are crucial for the performance of our method and excluding any of them leads significantly worse results. Next, we answer three questions about each component individually.
1. How important is it to incorporate OPE estimates? Across all the experiments, incorporating OPE estimates (top row of Fig. 6) always considerably improves the results compared to starting policy selection from scratch (bottom row). For example, in MPG when A-OPS starts with OPE, it able to cut the OPE regret by more than half before executing 100 trajectories. For this budget, GP+UCB which starts from scratch, does not reach the same regret OPE.

---

dm-coonra
MPG
0.6
GP+UCB
Ind+Unifarm-25
0.4
A-ops
OPE
Simple regret
0.4
Ind+
Ind Undffrrfom-
simple
Und+orroom-2200
0.2
A-ops-25
A-ops-50
A-ops-100
MB-AR
MB-
FQE-D
MB-FF
FOE-LE
DR
VPH
15
DICE
0.0
100
200
400
A-ops-200
trajectories
Figure 7: Left: The performance of A-OPS dm-control with varying type of OPE estimate with budget of 50 trajectories. The various OPE methods are Model -ased OPE with an auto nuto-regressive transition model (MB-AR). Fitted Q Evaluation with distributional Q function and distributional loss (FQE-D), D), Model-Basse OPE with a feedforward transition model (MB-FF), Fitted Q Evaluation with standard Q function and 1.2 loss (FQF-L22), Double Robust (DR), Variational Power Method (VPM), Importance Sampling (IS), Distribution Correction Estimation (DICE). A-OPS can improve upon the initial OPE estimates in all cases. Right: Performance of A-OPS and IND+UNIFORM with varying number policies available for execution. IND+UNIFORE affected a lot by the increase in the number of policies. Our approach A-OPS performs similarly when the number of policies is increase by factor of almost 10.
2. How informative is our kernel? To understand the value of modelling the correlation between policies through the kernel, we compare the GP method with the independent policy model (IND). In Fig. purple and red lines use GP as the policy model and green and blue use the IND model. In the vast majority of settings (11 out of 12) the use of a kernel significantly improves the results and in one setting (IND+ UNDFUUNNFOMMOPPP vs
in MPG domain) they perform on par. We conclude that the kernel is key ingredient for improving data efficiency.
3. How important is the selection strategy? The active selection of policies is generally beneficial for identifying good policy. For this we refer to the results of dashed lines that correspond to UNIFORM policy selection and solid lines that correspond to the of UCB. Using IND+UCD yields high regret in the initial exploration stage, but it improves substantially over IND+UNIFORM later. Moreover, incorporating OPE estimates and kernel (resulting in A-OPS) significantly shortens the exploration stage.
4.5 More ablation studies
Incorporating the OPE estimates is clearly beneficial for policy selection. Here, we further study: How does the performance depend on the quality of OPE estimates? Fig. left compares A-OPS using estimates from different OPE algorithms in DOPE (00 on dm-control with a small budget of 50 trajectories. A-OPS always improves upon the corresponding OPE estimate, and the performance clearly depends on the quality of the associated OPE estimate. The performance of GP+ with 50 trajectories that does not rely on any OPE metric shown as a horizontal line. Some OPE estimates (e.g., IS, VPM and DICE) have large estimation eтrors and even a negative correlation with the ranks of policies (00]. As expected, A-OPS is negatively impacted by poor OPE estimates and ignoring them could yield better results. It should be possible to correct for this to greater extent with more online observations. For now we follow the recommendation to use the FQE estimates.
In inisubsection 2.2 we hypothesised that A-OPS method should be data efficient as it would scale with the diversity of policies and not their number. We next try to verify and answer the questions How does the policy selection algorithm scale with the growing number of candidate policies? In Fig. 7| right we show the simple regret for A-OPS and IND+ IND+UNIFORM in the MPG domain when we sample 25, 50, 100 and 200 policies. We see that when the number of policies is not very high (25). the algorithm that treats policies independently and samples them at uniform (ND=-UNIFOR--25) can closely approach the performance of our method A-OPS-25 after 100 trajectories. However, when the number of policies grows, this small interaction budget not enough for the naive algo- rithm: IND+ UND+ORIFRRM-200 significantly degrades its performance. At the same time, A-OPS-200 minimally affected by the increase in the number of policies as the diversity of the policies stays approximately the same.

---

Robotank, step=1 step=10 step=50
step= 100
50
OPE selection ground truth current observatiens prediction
value
O6
Figure 8: Visualisation of the qualitative performance of A-OPS algorithm on Robotank environment with 20 randomly selected policies (ordered by ground truth only for the illustration purpose). From left to right we show the state 10, 50, 100 algorithm iterations. The initial OPE scores shown light blue, the policy selected by A-OPS highlighted with magenta star, the observation of currently executed policy shown orange circle, past observations are in pink and the prediction with its standard deviation is in purple.
4.6 Qualitative results
Finally, to understand the behaviour A-OPS method we show its progress for one the experiments with Robotank game in Fig. For visualisation purposes, by their ground truth returns (not observed by the method). Perfect value predictions would be aligned along the diagonal line. We show the initial OPE values, current selection of the best policy, current observation of the executed policy, past observations and the algorithm prediction with its standard deviation for steps 1, 10, 50, and 100. The A-OPS prediction is determined by the OPE observations and past observations of the policy returns. The kernel reinforces smoothness of the function. We can see that the initial selection dominated by OPE prediction and the selected policy is quite far from the best policy. With more online observations, the predictions improve. Our algorithm is data-efficient because 1) it samples more promising policies more frequently (more observations with higher scores), 2) the knowledge from one policy is propagated to the policies related by the kernel (predictions for some policies are updated without any new observations of their values).
only 20 policies which are ordered
Conclusions
In this paper, we introduced the active offline policy selection problem. We proposed BO solution that integrates OPE estimates with evaluations obtained by interacting with the environment, We also designed novel kernel that models correlation over policies. Experiments on three control domains proved that our solution is very effective. Furthermore, our experiments ablated the different components of the solution, highlighting the value of modelling correlation via the kernel, active selection, and incorporation OPE estimates the observation model.
Limitations and future work Having tested the method and demonstrated its efficacy in simulation, our future plan is to conduct experiments in real domains.
The A-OPS approach could be of use in many safety critical applications. This work does not deal with this question explicitly as we assume safety constraints to be implemented directly on hardware. It means that if an unsafe policy is attempted, the safety controller would terminate the policy before reaching the end of an episode, resulting in low reward and thus discouraging A-OPS from trying this policy again. Alternatively, we could alleviate this limitation by applying constrained BO techniques [62] to search for the best policy without violating safety constraints.
In this work, we explore the policy structure based on the offline dataset, and use every new trajectory only obtain a noisy observation of policy values. This allows a clean Bayesian optimization formulation. Going forward, more information could be extracted from the new trajectories, for example, to further improve OPE estimates, GP kernel, or policies themselves. However, using this information would make the solution much more complex and we may need to take the additional computational cost into account. Finally, to construct a better policy kernel, we may investigate how to measure the policy similarity on the most informative subset of states in the offline dataset.
10

---

Acknowledgements
We would like to thank Bobak Shahriari for the useful discussions and feedback the paper.
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, et al. TensorFlow: Large-scale machine lcarning on heterogeneous systems, 2015, Software available from tensorflow.org.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. International Conference on Machine Learning, 2020.
[3] Takayuki Akiyama, Hirotaka Hachiya, and Masashi Sugiyama. Efficient exploration through active learning for value function approximation in reinforcement learning. Neural Networks, 2010.
[4] Jean- Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In Conference Computational Learning Theory, 2010.
[5] Javad Azimi, Ali Jalali, and Xiaoli Zhang Fern. Hybrid batch bayesian optimization. In International Conference on Machine Learning, 2012.
[6] Philip Ball, Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Ready policy one: World building through active learning. In International Conference on Machine Learning, 2020.
[7] Gabriel Barth- Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference for Learning Representations, 2018.
[8] E. Brochu, T. Brochu, and N. de Freitas. A Bayesian interactive optimization approach to procedural animation design. In ACM SIGGRAPHI/Eurogaaphee Symposiun on Computer Animation, 2010.
[9] E. Brochu, N. de Freitas, and A. Ghosh. Active preference learning with discrete choice data. In Advances in Neural Information Processing Systems, 2007.
[10] Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zoina, Yusuf Aytar, David Budden. Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Robotics: Science and Systems Conference, 2020.
[11] Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, and Manuel Kroiss. Reverb: A framework for experience replay. arXiv preprint arXiv:2 102.04736, 2021.
[12] Yutian Chen, Aja Huang. Ziyu Wang, loannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in AlphaGo. arXiv preprint arXiv: 18/2.06855 2018.
[13] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quamtile networks for distributional reinforcement learning. In International Conference Machine Learning, 2018.
[14] Christian Daniel, Malte Viering. Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Robotics: Science and Systems Conference, 2014.
[15] Miroslav Dudik, John Langford, and Lihong Doubly robust policy evaluation and learning. arXiv preprint arXiv:1 4601, 2011.
[16] Gabriel Dulac-Amold, Daniel J. Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: 1904. /2901, 2019.
[17] Arkady Epshteyn, Adam Vogel. and Gerald DeJong. Active reinforcement learning. In International Conference on Machine Learning, 2008.
[18] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, 2018.
[19] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arX1v:2004.07219, 2020.

---

[20] Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, ziyu wang, Alexander Novikov, Mengjiao Yang. Michael R Zhang. Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off policy evaluation. International Conference for Learning Representations. 2021.
[21] Scott Fujimoto, David Meger, and Doina Precup. Off-policc deep reinforcement learning without explo- ration. In International Conference on Muchine Learning. 2019.
[22] Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang. Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen. Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation, arXiv preprint (rXiv:2113.099: 095755 2021.
[23] Caglar Gulcehre, Ziyu Wang. Alexander Novikov, Tom Le Paine, Sergio Gomez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RL Unplugged: A suite of benchmarks for offline reinforcement learning. arXiv preprint (tti::2000.188880 2020.
[24] Jean Harb, Tom Schaul, Doina Precup, and Pierre-Luc Bacon. Policy evaluation networks. arXiv preprint arXiv:2002. / 2020.
[25] Matt Hoffmann Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, et al. Acme: research framework for distributed reinforcement learning. arXiv preprint 2XXx:2006600975 2020.
[26] Matthew Hoffman, Bobak Shahriari, and Nando de Freitas. On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning. In Artificial Intelligence and Statistics, 2014.
[27] F. Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization, 2011.
[28] Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, 2016.
[29] Dmitry Kalashnikow, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Frie Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. QT-Opt: Scalable deep reinforcement learning for vision- robotic manipulation. In Conference on Robot Learning. 2018.
[30] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On Bayesian upper confidence bounds for bandit prohlems. In Artificial intelligence and statistics, 2012.
[31] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforeement leamnig. arXiv preprint 2005520005.09511, 2020.
[32] Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Discovering general- active learning strategics. arXiv preprint arXiv: 1810.04 /4, 2019.
[33] Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. Advances in Neural Information Processing Systems. 2012.
[34] David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observing rewards at a cost. arXiv preprint 2tt::2001,067000 2020.
[35] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-lcaming for offline reinforcement learning. arXiv preprint arXiv:2006.047 04779, 2020.
[36] Hoang Lee, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Muchine Learning, 2019.
[37] Sergey Levine. Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 2016.
[38] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644 2020.
[39] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual- approach to personalized news article recommendation. In World Wide Web, 2010.
[40] Lihong Li. Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual-bann based news article recommendation algorithms. In ACM international conference on Web search and data mining, 2011.
12

---

[41] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems, 2018.
[42] Daniel J Lizotte. Tao Wang. Michael H Bowling, and Dale Schuurmans. Automatic gait optimization with gaussian process regression. In International Joint Conference Artificial Intelligence, 2007.
[43] R. Martinez-Cantin, N. de Freitas, E. Brochu, J. Castellanos, and A. Doucet. A Bayesian exploration- exploitation approach for optimal online sensing and planning with a visually guided mobile robot. Autonomous Robots, 2009.
[44] Martino Migliavacca, Alessio Pecorino, Matteo Pirotta, Marcello Restelli, and Andrea Bonarini. Fitted policy search: Direct policy search using batch reinforcement learning approach. Workshop on Evolutionary and Reinforcement Learning for Autonomous Robor Systems, 2010.
[45] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang. Young-Joon Lee, Fric Johnson, Omkar Pathak, Sungmin Bae, et al. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004, 10746, 2020.
[46] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 13Xiv:13/2.5600,
2013.
[47] Remi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, 2016.
[48] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, 2019.
[49] Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to-treat policies. arXiv preprint
arXiv: 1905.0951, 2019.
[50] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.0005: 2020.
[51] Jack Parker- Holder. Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
[52] Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, 2000.
[53] Doina Precup, Richard S. Sutton, and Satinder P. Singh. Eligibility traces for off-policy policy evaluation. In International Conference Machine Learning, 2000.
[54] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[55] Julian Schritwieser, Toannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, Chess and Shogi by Planning with Learned Model. Nature, 2020.
[56] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 2015.
[57] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior mod- elling priors for offline reinforcement learning. In International Conference for Learning Representations, 2020.
[58] David Silver. Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016.
[59] J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, 2012.
[60] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning, 2010.
13

---

[61] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprini arXiv:0912.3995, 2009.
[62] Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with gaussian processes. In International Conference on Machine Learning. 2015.
[63] Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvári, and Eric Wiewiora, Fast gradient-descent methods for temporal-difference learning with linear function approximation. International Conference on Machine Learning, 2009.
[64] Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach the problem of off-policy temporal learning. Journal of Machine Learning Research, 2016.
[65] Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez. Timothy Lillicrap. and Nicolas Heess. dm_control: Software and tasks for continuous control. arXiv preprint arXiv:2006.1240 2020.
[66] Georgios Theocharous, Philip S Thomas, and Mohammad Ghavamzadeh, Personalized ad recommendation systems for life-time value optimization with guarantees. In International Joint Conference on Artificial Intelligence, 2015.
[67] Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and Q-function learning for off-policyy evaluation. In International Conference on Machine Learning, 2020.
[68] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. In National Conference on Artificial Intelligence (AAAI). 2016.
[69] Cameron Voloshin. Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint 1.1004491,.0684, 2019.
[70] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel. Josh Merel, Caglar Gulcehre, Nicolas Heess. and Nando de Freitas. Critic regularized regression. In Advances in Neural Information Processing Systems, 2020.
[71] Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. arXiv preprint wxXiv:2012.06112 2020.
[72] Mengjaao Yang. Ofir Nachum. Bo Dai, Lihong Li. and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. arXiv preprint arXiv:2007.03438, 2020.
[73] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
[74] Michael R Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. arXiv preprint 1xXx:21044.18777 2021.
14


Reviews:
[{"review_id": "u3bi55jzJtC", "reviewer": "Reviewer_68yo", "summary": "This paper introduces the problem of active offline policy selection which consists of selecting the best policy from a set of candidates given a small budget to interact with the environment. The paper proposes a solution (A-OPS) based on Bayesian optimization using a kernel function to relate candidate policies and generalize between them. The proposed solution is thoroughly evaluated on simulated robotic control environments using benchmark datasets from the off-policy evaluation literature and several ablations are included to rationalize the choices of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. The proposed problem seems important and not yet formalized in the community. Indeed much of the recent focus on offline RL and off-policy evaluation has neglected the possibility for a small budget of interaction with the environment. Thinking about how to effectively exploit this small budget is an important area for future work, and hopefully this paper can spur research in this area.\n2. The proposed solution is elegant. While not ground-breaking, the proposed solution does seem to be an intuitive and elegant way to approach the problem. With such a small budget of interaction, bayesian optimization is often a very strong approach, as this paper bears out.\n3. The experimental evidence is substantial. The paper runs on many environments with large datasets of policies. The advantage over the baselines is very evident. The ablations seem to do a comprehensive job of rationalizing each of the major components of the proposed approach. \n\n---\n\n### Weaknesses:\n\n1. The formalism is often imprecise. Here are a few examples:\n\n   (a) In Section 2.1, the notion of a budget which is often discussed informally in the text is never formally defined\n\n   (b) As a result $ \\hat k$ is used in the main definition of regret but is never defined. This makes it somewhat unclear if the paper is concerned only with the regret of the final policy or if it also cares about the cumulative regret during the evaluation in the environment (my impression was the former, but slightly unclear). \n\n   (c) In section 2.2 $ N_k $ is never defined, but often used. (It seems to be used to indicate the number of return observations of policy $ k$)\n\n2. Some important details are omitted in the experiments section.\n\n   (a) In Figure 7 the labels on the x-axis are never explained. The reader can try to guess (is MB-AR maybe model-based auto-regressive?), but these labels really ought to be explained in the text.\n\n   (b) For the plots in e.g. Figure 5, are the algorithms re-run from the start given each budget or are the algorithms just progressively evaluated after each trajectory is added? It seems to be the second, but this was a bit unclear. \n\n3. This is perhaps a more minor point, but based on the results in the appendix, it would seem useful to discuss the variability of the performance across tasks. While in aggregate the advantage of A-OPS over simple OPE is obvious, the advantage is much larger on some tasks than others (Fig 9, Fig 12). Moreover, for some tasks in the MPG domain, it seems that A-OPS performs worse than OPE even after 400 trajectories (Fig 12). Even if this discussion is just put in the appendix, it would be useful to discuss rather than just repeating that A-OPS is outperforming the other methods. \n\n---\n\n### Recommendation:\n\nAccept.\n\nI think this is a strong paper that introduces an interesting problem and provides a reasonable solution with substantial experimental evidence. The paper is sloppy about details in a few places, but these should be easily fixed. \n\n---\n\n### Minor comments/questions:\n\n1. Is there a way for the algorithm to incorporate knowledge of the size of the budget? Now the budget just determines the length of the loop, but it seems that the algorithm could modify its strategy with knowledge of the budget.\n\n2. Have you thought about how to select the set of candidate policies to best leverage the strengths of A-OPS? I understand that for this paper \n\n   \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Space is already a bit tight, but the paper would benefit from a more substantial discussion of the limitations in the current methods and the potential directions of improvement. I do not think a discussion of societal impact is necessary for this paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "lqKVqwEFkQZ", "reviewer": "Reviewer_6fix", "summary": "This paper introduces a problem of active offline policy selection, where the goal is to select the best-performing policy among the fixed K policies using an abundant offline dataset and a limited additional interaction with the environment. This work proposes to solve this problem by Bayesian optimization (BO) with a Gaussian process (GP), where the input and output of the objective function are a policy and its expected return respectively, and function queries render noisy observations. To facilitate GP in this problem, a new GP kernel is introduced, which takes two policies as input, and its output is computed based on the similarities of policy distribution for each state in the offline dataset. Also, to make BO more data-efficient, off-policy evaluation (OPE) estimates through the offline dataset are used for a warm start of BO. Finally, UCB acquisition function is used for active policy selection. Experimental results show that the proposed BO with OPE-based warm start significantly outperforms the baselines. Also, ablation studies confirm that UCB selection is better than uniform selection, GP is better than methods that do not consider policy correlation, and using OPE is better than not using it.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Overall, the paper is well-written and easy to follow. The introduced active offline policy selection is well-motivated by practical situations such as A/B tests, and the proposed method is sound with a suitable combination of Bayesian optimization and OPE estimates. I found introducing this setup is novel. The proposed solution itself is not surprising, but the experiments are thorough and the results are also convincing. The ablation studies are solid and appreciated.\n- The additional data collected during online interactions could be used to improve the OPE estimates since they are particularly on-policy samples for a candidate policy?\n- In Eq (3), the generative assumption may not be consistent with the actual episodic return distribution since it is likely that return distribution may not be Gaussian-distributed (e.g. highly skewed distribution). Would this inconsistency make a problem?\n- In Figure 3 (middle), why are the similarities between D4PG policies very low, even with the same hyperparameters?\n- It seems the proposed policy distance (Eq (6)) may not be optimal since it considers state distribution by the offline dataset, not the state distribution induced by the target policy. For example, if the target policies visit a very limited region of states, to measure the policy similarity, we only need to consider those states that will be visited while ignoring other states that will not be visited. The distance between stationary distributions induced by policies could be a more suitable distance measure?\n- What if the OPE estimates are significantly underestimated (or overestimated), compared to the true expected returns? Pure exploitation (or pure exploration) might happen during active policy selection?\n- In Figure 5, why doesn't the performance of A-opt at (# trajectories=0) match the performance of OPE (even worse than OPE)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have discussed the limitations and potential negative societal impact properly.", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gMBvGAdXL4w", "reviewer": "Reviewer_bUXf", "summary": "This paper considers a novel problem setting of \"Offline Policy Selection\", whereby there is access to an initial set of policies trained offline, and an algorithm must choose a policy to deploy in the online simulated environment. I am giving this paper the score of weak reject because despite introducing an important problem setting, I am not convinced the method is actually particularly effective. Hopefully this can be clarified during the rebuttal phase. As such, the emphasis of this review is on areas for improvement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1) Novel problem setting, definitely well motivated. Could be important in using Offline RL more broadly.\n\n2) The policy kernel idea seems intuitive, and the discussion around choice of kernels (A.5) is interesting. I think it would improve the paper if some of this discussion/additional experiments are included in the main body (more below).\n\n3) The authors did make efforts to include multiple ablations, and discussed limitations.\n\n4) The use of OPE + online is obvious in hindsight which is a common trait of valuable contributions.\n\nWeaknesses/Questions:\n\n1) The actual number of trials used is much larger than I would expect, with BO not converging until >100 trajectories. For DM control suite, *this is enough to train a near optimal policy from scratch using online data*. Why would we not just do this? I think a key baseline is OPE+online, since OPE always performs reasonably well zero-shot, we could then use the online data to fine tune the best OPE policy. \n\n2) Given 1), it seems that formulating this problem as blackbox optimization may not be the best way to go, since it is not making use of the structure of the task at all. Aside from using the episode returns, A-OPS does not leverage any of the trajectory data (as far as I am aware) which ends up being 200k timesteps worth of online samples by the end.\n\n3) What is the actual variance of the policy returns? The aggregate regret is definitely an effective overall metric to use, but if for example the min and max polices for a task are only 5-10% apart it is rather meaningless.\n\n4) It would be useful to include a plot showing the *min return* for each method. Using an exploratory acquisition function (such as UCB) likely leads to also testing some policies far away from the current best in the behavioral space, which could lead to very poor returns. For practical use this may be infeasible since if deploying on real hardware you could break the robot. This is alluded to in the Limitations section in the Conclusion, but it is not shown anywhere. I have a minor concern this may be because it looks bad for A-OPS. \n\n5) Looking at the *individual* task performance in the Appendix, it seems to be very hit and miss. Do the authors have any intuition why none of the methods or ablations work well for the manipulator dm control task? Similarly with MPG, for 2/4 of the tasks the OPE approach actually does better than A-OPS despite using zero online data. It seems only really the slide task which makes A-OPS outperform overall due to the poor performance of OPE, what happened there?\n\n6) How did you tune the UCB exploration parameter? \n\n7) I assume the answer to 6) and also other ablations is that the method was tuned on the testing data. Is this true? \n\nMinor comments/typos:\n\n-l.97: \"We can use GP\" → \"we can use a GP\"\n\n-l.204: \"We perform 4 several manipulation tasks\" (not clear what this means)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed sufficiently well but of course could always be expanded. There is no expected negative societal impact from this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "W_3afJTfvBG", "reviewer": "Reviewer_rn9y", "summary": "This paper works on the policy selection problem in a setting characterized by features: available logged data and allowing a limited number of online interactions with the environment. To do so, this work proposes to use Bayesian optimization to combine existing OPE techniques and online interactions. For the setting with large number of policies, a kernel function is introduced to take the correlation between policies, which indicate the performances of different policies. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem formulation is not typical. It is a mix of the common offline and online settings. On the one hand, it aims to leverage the logged datasets and OPE; on the other, it is not totally offline as the setting in the paper allows online interactions, although the number such interactions are limited. Given this, this work proposes a Bayesian optimization solution. While such a setting is possible in real-world, I doubt the importance of such a non-typically setting itself. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please also refer to the main review section. For high-stakes decision-making scenarios that offline RL aims to address, further online interactions with the environment is not available for reasons including ethical or cost concerns. Take healthcare and education for example, logged dataset can be large, but online interaction is typically infeasible. This work fails to show the importance of the proposed setting itself. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers felt this paper would be a nice contribution to the conference. It introduces a novel problem setting of active offline policy selection problem formulation, which relies both on logged data and limited online interactions with the environment.  The authors have done a good job replying to all concerns during the discussion.  To strengthen the contribution, authors are encouraged to open source components of the approach.
Decision: Accept (Poster)