Offline policy selection under Uncertainty

Abstract: The presence of uncertainty in policy evaluation  significantly complicates the process of policy ranking and selection in real-world settings. We formally consider offline policy selection as learning preferences over a set of policy prospects given a fixed experience dataset. While one can select or rank policies based on point estimates of their policy values or high-confidence intervals, access to the full distribution over one's belief of the policy value enables more flexible selection algorithms under a wider range of downstream evaluation metrics. We propose BayesDICE for estimating this belief distribution in terms of posteriors of distribution correction ratios derived from stochastic constraints (as opposed to explicit likelihood, which is not available). Empirically, BayesDICE is highly competitive to existing state-of-the-art approaches in confidence interval estimation. More importantly, we show how the belief distribution estimated by BayesDICE may be used to rank policies with respect to any arbitrary downstream policy selection metric, and we empirically demonstrate that this selection procedure significantly outperforms existing approaches, such as ranking policies according to mean or high-confidence lower bound value estimates.

Reviews and author responses:

## Review

Given as set of pre-specified policies, this paper proposes a Bayesian method to estimate the posterior distribution of their average values, by estimating posterior distributions of their discounted stationary distribution ratios. These posterior distributions are used for off-policy evaluation in various ways.


 
## Positives

+ The idea focusing on estimating nonlinear functionals of multiple policy values is appealing.


 
## Major concerns

+ The results in Figure 2 are a bit surprising. We expect that methods based on concentration inequalities like Bernstein or student-t to be somewhat conservative, but the results suggest that their confidence intervals are extremely wide. For example, in the Bandit case, even after 200 samples, the interval log-width would suggest that Bernstein's confidence intervals is more than 7x the confidence intervals suggested by BayesDICE. What explains these results?

+ Again on Figure 2, if "Bernstein" and "Student t" are unbiased methods, then having a very wide confidence interval should translate into over-coverage. However, they seem to be *under*-covering the true value. Are these methods somehow heavily biased? If not, what explains the under-coverage?

+ The paper proposes a method for evaluating non-linear functionals of policy values, such as ranking scores over their values. However, it seems to me that in order to evaluate such nonlinear functions one would require knowledge about the *joint* distribution of values over all policies of interest. In the notation of the paper, one would require knowledge of $q(\bar{\rho}_1, ..., \bar{\rho}_N)$. However, it is not clear from the method description in Section 3.2 how one is able to estimate this joint distribution. Instead, it seems to me that all we get is $q(\bar{\rho}_i)$ for each policy $i$ -- that is, their marginal distributions. If that is the correct interpretation of what's going on in Section 3.2, then that raises the question of whether these distributions are independent. 



 
## Minor concerns

+ In appendix C.1, I did not understand the description of the "bandit" environment. Are rewards binary?

+ Several symbols are not formally defined. E.g., on page 4, (lowercase) r(s,a) is not defined.

+ In Algorithm 1, what is the role of quantity L*? Why do we need it as a stopping rule?

+ Some notes on exposition.

  - The authors take some time to reveal what is their estimand --- the discounted stationary distribution ratios. As a reader, I would have benefited from having that explained much earlier, even before the conversation about ranking evaluation.

  - Section 3.2: the authors could have dedicated some more space developing the intuition for their method (e.g. an abridged version of Nachum and Dai 2020), even if that meant relegating some of the mathematical details to the appendix. As it stands, the section makes the paper incomprehensible as a standalone piece of research.


## Typos

+ The indices on the sum in the definition of "stationary visitation" (p.4) are wrong. On the next line, the last conditioning should have been s[i+1] ~ T(.|s[i],a[i]) instead of s[i+1] ~ T(.|s[t],a[t]).
+ Philip Thomas' "High-confidence off- policy evaluation" citation shows up twice in the bibliography.
+ indentical --> identical (Pg. 5)

Thank you for the feedback. Please let us know if our responses below address the issues raised in your review, or if any additional clarifications are needed.

> Bernstein’s wide confidence interval

The confidence intervals based on Bernstein’s inequality are expected to be quite conservative [3][4] due to the generality of the inequality. Therefore, we are not surprised that the CI of Bernstein is much wider than BayesDICE.

> Biased baselines and under-coverage

Bernstein, Student-t, and Bootstrapping in Fig. 2 of the original paper are indeed biased due to self-normalization across the entire dataset in the importance sampling estimator. It is known that self-normalization yields better empirical results in MDPs despite being biased [1, 2].  To maintain uniformity between the MDP and bandit settings, we therefore used self-normalization in both cases. However, to understand how self-normalization affects interval coverage on bandit, we did rerun these baselines in our bandit experiments without self-normalization.  The new bandit results are plotted in the updated appendix (Figure 5) for Bernstein, Student-t, and Bootstrapping. As expected, Bernstein is conservative, resulting in over-coverage, Student-t makes a distributional assumption, which if violated can lead to undercoverage, and Bootstrap is asymptotic and generally undercovers for small sample sizes, as we observe. It is true that removing self-normalization can improve the coverage of these estimators in bandit settings, although normalization remains advantageous for the MDP case.  We have clarified this issue in the paper.

> Independence assumption of $\{\bar{\rho}_i\}$

The last paragraph of Sec. 3.2 of the original submission pointed out that “we further assume that the distributions are independent for each policy”. This assumption allows the joint distribution of policy values to be expressed as a simple product of individual policy value distributions $q(\bar{\rho}_1,...,\bar{\rho}_N) = \prod_i q(\bar{\rho}_i)$, which makes computation much easier. This is a simplifying assumption that empirically works well.

> Bandit environment

The bandit environment is a Bernoulli bandit with binary rewards. Clarified in the updated appendix.

> The role of $L^*$

$L^*$ is a local variable for keeping track of the best ranking score (sum of $n$ sets of $\hat{\mathcal{S}}((\hat{\rho}_1^{(j)},...,\hat{\rho}_N^{(j)}), \mathcal{O})$) and the best ranking so far when iterating through the ranking permutations, as commented below. We have added the comments below in the algorithm to clarify this.

Algorithm 1 OfflineSelect\
...\
\# Local variable for keep tracking of the best ranking and score seen so far\
Initialize $O^∗$ ; $L^∗$\
...\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\# Ranking score as a sum of individual scores of samples from the posterior\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$L = L + \hat{S}((\hat{\rho}_1^{(j)},...,\hat{\rho}_N^{(j)}), O)$\
...\
&nbsp;&nbsp;&nbsp;&nbsp;if $L < L^∗$ then\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\# Update the best ranking and best ranking score seen so far\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$L^∗ = L$; $O^∗ = O$

> Exposition of the paper

The major goal of this submission is to solve the off-policy selection (OPS) problem, which is different from vanilla OPE and has not been seriously treated in the literature, as OPS introduces decision making to offline RL. We first define OPS in a formal way, which motivates the novel BayesDICE. The benefits of BayesDICE, namely the access to the full value posterior, are particularly well-aligned with the need of the OPS problem.

[1] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite horizon off-policy estimation. In Advances in Neural Information Processing Systems, pp. 5356– 5366, 2018.

[2] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pp. 2315–2325, 2019.

[3] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Conference on Machine Learning, pp. 2380–2388, 2015.

[4] Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence offpolicy evaluation. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015

We thank all reviewers for the feedback. We want to emphasize again that the major contribution of this paper is to formally consider the policy selection problem in a completely offline setting. To our knowledge, this work is the first to take decision making into account and the first to apply Bayesian decision theory to in offline policy evaluation, enabling learning once and selecting policies under any criteria. Meanwhile, we propose the posterior estimation from stochastic constraints when explicit likelihood is not available, which could be of independent interest.

We address individual reviews below and have fixed typos and updated the paper (with changes highlighted in blue).


Thank you for the positive feedback. Please see our response below.

> I am less convinced by the motivation of off-policy model selection, since if we want to find some policy to deploy, does off-policy learning address the problem?

Off-policy learning does not completely solve the problem for several reasons: 
(1) Current off-policy RL algorithms cannot be used out-of-the-box and require hyperparameter-tuning [1], thus precisely necessitating some mechanism for off-policy model selection.
(2) Off-policy RL algorithms learn policies to maximize expected return. As elaborated on in our submission, a practitioner may prefer a collection of policies optimized for some other metric, such as top-k precision or regret.
(3) Policies in practical applications may not be amenable to (policy)-gradient-based (or similar forms of) learning. For example, policies may have a combination of business logic and hard-coded rules. In these cases, it may be easier to rank a set of candidate policies rather than learn one from scratch.

> Do we need really to go though all permutations in Alg 1?

The reviewer is right, and we agree that depending on the evaluation metric chosen, its structural properties can be exploited to nullify the need to test all permutations in Alg 1. Such structural properties are present in many natural metrics (such as top-k precision or regret) and so we believe our method will easily scale to larger numbers of candidate policies.

> Just curious, any explanation about why it fails in some domain, say Taxi in Fig 2? It could be better to discuss more about when the method works, when it fails? Taxi in Fig 2 may be a good example to diagnose this?

Tabular Taxi [3] is a much harder environment than others in Fig. 2. The environment has 2000 states in total(25 × 24 × 5, corresponding to 25 taxi locations, 24 passenger appearance status and 5 taxi status (empty or with one of 4 destinations)). The large state-action space incurs additional stochasticity that results in larger estimation error in BayesDICE.

> In experiments, for baseline methods , the authors are using per-step IPS +/- conf level. However, I feel it is kind of unfair for baseline methods, since basically the BayesDICE are using marginalized IPS, is it possible to change the baseline estimate to marginalized IPS for a fair comparison?

First, we note that the original frequentist approach to CI estimation uses importance reweighting without marginalization [2]. Marginalized IPS for CI estimation has not been theoretically justified, but since the reviewer raised this question, we conducted additional tabular CI experiments using marginalized IPS and present the results in the updated appendix (Figure 6). Bandit is omitted since the horizon length is $1$. We found that the baselines still have large interval width or inaccurate coverage. Our most interesting finding is that Student-t undercovers in the Taxi environment with marginalized IPS, potentially due to its distributional assumption being violated.

> Can you comment how you address the stochasticity in the reward?

We restricted our analysis to deterministic reward (footnote 1 on pg. 2), but the proposed method can be easily extended to stochastic reward. BayesDICE is fitting the posterior of correction ratio, which is independent w.r.t. the reward. We could fit the posterior of the reward as a simple Bayesian regression. As a result, the final accumulated reward as well as its CI can be computed by sampling from the posteriors of stationary ratio and reward. 

[1] Paine T. Paduraru C., et al. Hyperparameter Selection for Offline Reinforcement Learning. 2020.

[2] Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence offpolicy evaluation. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015

[3] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite horizon off-policy estimation. In Advances in Neural Information Processing Systems, pp. 5356– 5366, 2018.



Thank you for appreciating the new perspective we bring to offline policy selection. Please see our response below.

> Choice of prior.

The prior of the ratio variables $\zeta$ is chosen to be a Gaussian distribution with $\mu=1$ and $\sigma=1$. Anecdotally, we have found the confidence intervals to be robust to small perturbations; specifically, we played with setting $\mu \sim [0.1, 10], \sigma \sim [0.1, 1]$ and observed the resulting confidence intervals to be similar to those in the paper.

> Multiple testing.

We assume the reviewer is asking about multiple hypothesis testing. Since the output of BayesDICE is a posterior distribution of policy values, we expect existing Bayesian frameworks for multiple testing (e.g., [1]) to apply to BayesDICE. The exact procedure for conducting such tests is beyond the scope of this paper.

> How the number of candidates influences the results.

How the number of candidates influences the results depends on the specific policy selection criteria and the characteristics of the policies at hand. For instance, if one policy’s mean value is much larger than all the other policies’ and the selection criteria is top-1 mean value estimate, additional candidate policies with small mean values will not affect the selection results. The OfflineSelect algorithm guarantees to select the “best” policies defined by a given selection criteria and a fixed number of policies, no matter what the selection criteria and policy characteristics are.

> Independence of data samples.

For mathematical simplicity, we assume that the dataset is sampled i.i.d. (see Section 2.1). This is a common assumption in the OPE literature [2] and may be relaxed in some cases by assuming a fast mixing time [3]. In practice, although the i.i.d. assumption may not hold, our empirical results show that the estimator’s accuracy is not strongly affected. Footnote 2 on pg. 2 of the paper is updated to reflect this.

[1] Berry D, Hochberg Y. Bayesian perspectives on multiple testing. J Stat Plan Inference. 1999;82(1–2):215–27.

[2] Uehara M., Huang J., Jiang, N. Minimax Weight and Q-Function Learning for Off-Policy Evaluation. ICML 2020.

[3] Nachum O., Chow Y., Dai B., and Li L. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pp. 2315–2325, 2019.



1. Significance:
- This paper studies off-policy model selection problem, which we aim to select a subset of policies based on different evaluation metric, such as top-k precision, top-k accuracy, etc. I am less convinced by the motivation of off-policy model selection, since if we want to find some policy to deploy, does off-policy learning address the problem? It would be great if the authors could give a more practical and clear motivation on this, especially compared with learning. 

2. Quality & Clarity:
- This paper is clearly written and easy to read, though I feel some details in the Appendix should be move into the main paper, especially some setup for the experiments, like the number of overall policies to be evaluated, what is the f in the experiments?
- The method is based on building posterior distribution of stationary distribution ratio, which tried to build a loss function of the Bellman residual on the average state-action visitation measure, and a KL regularization with the prior distribution. This idea is not new (see DualDice) however utilize them to learn the distribution of density ratio is novel. 
- I love the vision that when we have different evaluation metrics, we may need the whole distribution of the policy value to perform effective selection, and point estimate w. confidence interval may not be suffice in this setting. Regarding to this observation, I can see the importance/significance of this work.
- The method does make sense to me, and the empirical evaluation is well done (though some problems listed below). Important metrics are examined, like coverage and length of interval. Also, I like the experiments that the authors examine different metrics, and show how point estimate w. confidence estimation gives sub-optimal performance, this makes the motivation much clearer, about why we need the whole posterior distribution.

3. Questions:
- Regarding to the scale of the method, do we need really to go though all permutations in Alg 1? For some structured score S, can we just use sorting? Also, i see the experiments, there is only 5 policy to select, I am concerned about the scalability of the method. Could you justify more this?
- Just curious, any explanation about why it fails in some domain, say Taxi in Fig 2? It could be better to discuss more about when the method works, when it fails? Taxi in Fig 2 may be a good example to diagnose this?
- In experiments, for baseline methods , the authors are using per-step IPS +/- conf level. However, I feel it is kind of unfair for baseline methods, since basically the BayesDICE are using marginalized IPS, is it possible to change the baseline estimate to marginalized IPS for a fair comparison?
- Can you comment how you address the stochasticity in the reward?
- Some typo, say Eq4.

Overall, I like the problem it raises, that there exists some metric when a point estimate of policy value +/- conf level may not suffice to give optimal performance. The authors did a pretty good job in addressing this problem, though some problems remain and needs improvement. I am willing to improve my score if my questions are addressed. 


This paper proposes a method BayesDICE to estimate posteriors over candidate policy values, which can be used for downstream policy selection. Specifically, the authors estimate the posteriors over the correction ratios for state-action pairs, which optimize a combined metric of a chance constraint from collected data and KL from the prior. Computationally, the authors demonstrate the advantages of their approach by having better performances in both coverage and power for policy evaluation and better downstream ranking with respect to different metrics for policy selection.

I think the paper is well written and has made a good investigation of their approach. It would be better if the authors can talk about how different choices of prior influence the results. Besides, for policy selection, when we have a large pool of candidates, how would this be adapted to multiple testing or how would the number of candidates influence the results. Finally, solving eq.14 needs estimation of some expectation over the state action visitation, while the data itself is not independent, how would this influence the results?

some minor comments
- section 3.1, the first displayed equation: should it be summing from t=0 to infinity?
- typo, appendix B, the first sentence, `"consider exploitin" -> exploiting 

Meta-review: The review team appreciated the new Bayesian perspective offered by the submission, which lends itself well to selection and ranking, though some of them were still not convinced by the motivation (including in the private post-rebuttal discussion, R3). The reviewers also identified many points for improvement. The paper was borderline and, given the lack of enthusiastic support from the reviewers, the program committee decided to reject it. We strongly encourage you to address the raised concerns and resubmit to a future venue.

Decision: withdrawn-rejected-submissions