Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method BayesDICE to estimate posteriors over candidate policy values, which can be used for downstream policy selection. Specifically, the authors estimate the posteriors over the correction ratios for state-action pairs, which optimize a combined metric of a chance constraint from collected data and KL from the prior. Computationally, the authors demonstrate the advantages of their approach by having better performances in both coverage and power for policy evaluation and better downstream ranking with respect to different metrics for policy selection.

I think the paper is well written and has made a good investigation of their approach. It would be better if the authors can talk about how different choices of prior influence the results. Besides, for policy selection, when we have a large pool of candidates, how would this be adapted to multiple testing or how would the number of candidates influence the results. Finally, solving eq.14 needs estimation of some expectation over the state action visitation, while the data itself is not independent, how would this influence the results?

some minor comments
- section 3.1, the first displayed equation: should it be summing from t=0 to infinity?
- typo, appendix B, the first sentence, `"consider exploitin" -> exploiting 
review_title: good paper that brings new perspective for offline policy selection

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1. Significance:
- This paper studies off-policy model selection problem, which we aim to select a subset of policies based on different evaluation metric, such as top-k precision, top-k accuracy, etc. I am less convinced by the motivation of off-policy model selection, since if we want to find some policy to deploy, does off-policy learning address the problem? It would be great if the authors could give a more practical and clear motivation on this, especially compared with learning. 

2. Quality & Clarity:
- This paper is clearly written and easy to read, though I feel some details in the Appendix should be move into the main paper, especially some setup for the experiments, like the number of overall policies to be evaluated, what is the f in the experiments?
- The method is based on building posterior distribution of stationary distribution ratio, which tried to build a loss function of the Bellman residual on the average state-action visitation measure, and a KL regularization with the prior distribution. This idea is not new (see DualDice) however utilize them to learn the distribution of density ratio is novel. 
- I love the vision that when we have different evaluation metrics, we may need the whole distribution of the policy value to perform effective selection, and point estimate w. confidence interval may not be suffice in this setting. Regarding to this observation, I can see the importance/significance of this work.
- The method does make sense to me, and the empirical evaluation is well done (though some problems listed below). Important metrics are examined, like coverage and length of interval. Also, I like the experiments that the authors examine different metrics, and show how point estimate w. confidence estimation gives sub-optimal performance, this makes the motivation much clearer, about why we need the whole posterior distribution.

3. Questions:
- Regarding to the scale of the method, do we need really to go though all permutations in Alg 1? For some structured score S, can we just use sorting? Also, i see the experiments, there is only 5 policy to select, I am concerned about the scalability of the method. Could you justify more this?
- Just curious, any explanation about why it fails in some domain, say Taxi in Fig 2? It could be better to discuss more about when the method works, when it fails? Taxi in Fig 2 may be a good example to diagnose this?
- In experiments, for baseline methods , the authors are using per-step IPS +/- conf level. However, I feel it is kind of unfair for baseline methods, since basically the BayesDICE are using marginalized IPS, is it possible to change the baseline estimate to marginalized IPS for a fair comparison?
- Can you comment how you address the stochasticity in the reward?
- Some typo, say Eq4.

Overall, I like the problem it raises, that there exists some metric when a point estimate of policy value +/- conf level may not suffice to give optimal performance. The authors did a pretty good job in addressing this problem, though some problems remain and needs improvement. I am willing to improve my score if my questions are addressed. 

review_title: This paper studies off-policy model selection under various evaluation metrics. Basically a posterior distribution of the stationary density ratio is estimated, then utilize marginalized IPS to estimate the distribution of policy value.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Review

Given as set of pre-specified policies, this paper proposes a Bayesian method to estimate the posterior distribution of their average values, by estimating posterior distributions of their discounted stationary distribution ratios. These posterior distributions are used for off-policy evaluation in various ways.


 
## Positives

+ The idea focusing on estimating nonlinear functionals of multiple policy values is appealing.


 
## Major concerns

+ The results in Figure 2 are a bit surprising. We expect that methods based on concentration inequalities like Bernstein or student-t to be somewhat conservative, but the results suggest that their confidence intervals are extremely wide. For example, in the Bandit case, even after 200 samples, the interval log-width would suggest that Bernstein's confidence intervals is more than 7x the confidence intervals suggested by BayesDICE. What explains these results?

+ Again on Figure 2, if "Bernstein" and "Student t" are unbiased methods, then having a very wide confidence interval should translate into over-coverage. However, they seem to be *under*-covering the true value. Are these methods somehow heavily biased? If not, what explains the under-coverage?

+ The paper proposes a method for evaluating non-linear functionals of policy values, such as ranking scores over their values. However, it seems to me that in order to evaluate such nonlinear functions one would require knowledge about the *joint* distribution of values over all policies of interest. In the notation of the paper, one would require knowledge of $q(\bar{\rho}_1, ..., \bar{\rho}_N)$. However, it is not clear from the method description in Section 3.2 how one is able to estimate this joint distribution. Instead, it seems to me that all we get is $q(\bar{\rho}_i)$ for each policy $i$ -- that is, their marginal distributions. If that is the correct interpretation of what's going on in Section 3.2, then that raises the question of whether these distributions are independent. 



 
## Minor concerns

+ In appendix C.1, I did not understand the description of the "bandit" environment. Are rewards binary?

+ Several symbols are not formally defined. E.g., on page 4, (lowercase) r(s,a) is not defined.

+ In Algorithm 1, what is the role of quantity L*? Why do we need it as a stopping rule?

+ Some notes on exposition.

  - The authors take some time to reveal what is their estimand --- the discounted stationary distribution ratios. As a reader, I would have benefited from having that explained much earlier, even before the conversation about ranking evaluation.

  - Section 3.2: the authors could have dedicated some more space developing the intuition for their method (e.g. an abridged version of Nachum and Dai 2020), even if that meant relegating some of the mathematical details to the appendix. As it stands, the section makes the paper incomprehensible as a standalone piece of research.


## Typos

+ The indices on the sum in the definition of "stationary visitation" (p.4) are wrong. On the next line, the last conditioning should have been s[i+1] ~ T(.|s[i],a[i]) instead of s[i+1] ~ T(.|s[t],a[t]).
+ Philip Thomas' "High-confidence off- policy evaluation" citation shows up twice in the bibliography.
+ indentical --> identical (Pg. 5)
review_title:  

Meta-review: The review team appreciated the new Bayesian perspective offered by the submission, which lends itself well to selection and ranking, though some of them were still not convinced by the motivation (including in the private post-rebuttal discussion, R3). The reviewers also identified many points for improvement. The paper was borderline and, given the lack of enthusiastic support from the reviewers, the program committee decided to reject it. We strongly encourage you to address the raised concerns and resubmit to a future venue.
Decision: Reject