Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper is concerned with accurately estimating the logging policy in off-policy learning when it is unknown or unavailable. Specifically, the paper derives an uncertainty-aware inverse propensity score algorithm called UIPS that is more accurate for rarely-taken actions, and provides theoretical and empirical arguments in favour of the algorithm.
strength_and_weaknesses: **Strengths:**
+ The paper draws on work from several areas to build the proposed algorithm.
+ It seems like future algorithms for uncertainty estimation in neural networks could be dropped into the proposed method with only minor modifications, making the algorithm somewhat future-proof in this respect.
+ Strong performance in experiments.

**Potential Weaknesses/Questions:**
- Some of the citations do not seem to cite the correct papers. For example, the earliest use of probability ratios in off-policy RL that I'm aware of is Sutton and Barto (1998), and the earliest peer reviewed paper using probability ratios in off-policy RL that I'm aware of is Precup et al. (2000).
- The title of the paper seems overly broad, as the uncertainty is only in the estimated logging policy, not in the resulting policy or value estimates, and the proposed algorithm is only for the contextual bandit setting.
- ~~The example of news recommendation at the beginning of Section 2 seems a little weird. If the context summarizes the user's interaction with the recommender system, then their choice of action would affect future contexts, which would be reinforcement learning and not a contextual bandit problem. If that's true, why use the contextual bandit setting?~~
- It seems like the proposed algorithm is limited to the finite action case, and does not support continuous actions spaces, or generalization between actions.
- ~~Can the authors please comment on the computational complexity of the proposed method in comparison to the competitor methods used in the experiments? Does UIPS use more computation than the other methods?~~
- It would be good to explicitly specify the algorithm somewhere self-contained (even in an appendix would be fine if space is a concern), to make it easier to reproduce and easier for practitioners to implement.
- There's a lot of notation to remember which makes the paper harder to understand, and some of it is not necessary. For example, on page 5, why define the function $g(x_n,a_n)$ as the gradient of $f_\theta(x_n,a_n)$ instead of just writing $\nabla_\theta f_\theta(x_n,a_n)$?

**Minor comments/questions:**
- Page 3: "enlarges the this bias" has a typo.

**References:**
* Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An Introduction.
* Precup, D., Sutton, R. S., & Singh, S. P. (2000). Eligibility Traces for Off-Policy Policy Evaluation. ICML.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
The paper is somewhat clearly written, although there is a lot of mathematical notation to remember, which makes the paper harder to understand. In addition, there are frequent and numerous grammatical errors.

**Quality:**
I don't have any major complaints about the quality of the paper.

**Novelty:**
To the best of my knowledge, the proposed algorithm is novel, but I am also not up to date on the latest literature in this area.

**Reproducibility:**
The paper provides experimental details in an appendix, and should be reproducible without too much work.
summary_of_the_review: I am conflicted. The proposed algorithm is interesting and performs well in experiments, but my concerns with clarity and readability prevent me from wholeheartedly recommending acceptance. If the authors address my concerns, improve the clarity of the paper, and drastically reduce the grammatical errors, I would recommend acceptance.

**Update:** The authors have addressed enough of my concerns that I now recommend accepting the paper for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an extension to the classical inverse propensity score estimator for off-policy learning.
The authors propose considering the uncertainty in estimation to the logging policy when this policy is unknown, which is typically the case in many practical scenarios. They do so by introducing a new learnable parameter ( phi ), that increases for samples with low uncertainty, and decreases for samples with high uncertainty.
Through various datasets, the authors show that their proposed technique is able to outperform several standard methods for off-policy evaluation.

strength_and_weaknesses: Strengths:
- The paper is clearly written and well organized. It was pleasant to read, and the reasoning flowed nicely.
- The paper tackles a common problem that's encountered across several industries, how to apply off-policy evaluation when the logging policy is unknown. The typical approach of estimating a logging policy from data suffers from high bias, and this paper addresses that issue by including uncertainty when weighting samples with probability ratios.
- The experimental results show that UIPS consistently outperforms the chose benchmark techniques, when the logging policy is estimated.


Weakness:
- I would have liked to see a comparison in the synthetic dataset of how close the proposed technique gets to the ground-truth logging policy, when it is estimated. That oracle baseline would be very helpful to understand how significant the improvements of UIPS are in contrast to the performance of the other methods. 
For ex. if UIPS achieves P@5 of 0.5666 (table 1), the difference to CE 0.5559 is a lot more meaningful if the performance of using the true logging policy at P@5 is 0.5700, than if it is 0.988.

- In my experience, doubly-robust (DR) estimators are often some of the best performing benchmarks for off-policy evaluation. I would have liked to see the use of a DR estimator as part of the evaluations.

- For experimental results, please include standard deviation numbers as well as mean performance as it helps to identify how significant those improvements are.


Nitpick:
- Page 8, section 4.1, The second sentence makes reference to Table 3 showing the average MSE, but table 3 is actually showing P@5, R@5. Maybe a wrong table reference, or something got lost in editing?

- Table 2: Any comment on why would CE outperform other off-policy eval methods on low frequency actions? I'm very intrigued by that, as I would expect CE to be heavily biased specially in low frequency samples. 
clarity,_quality,_novelty_and_reproducibility: A high quality, well written paper.
As far as I can tell, the work presented is novel, and all code for reproducing results was submitted.
summary_of_the_review: A well-written paper that highlights a practical issue of estimating logging policies when these are not available.
This paper tackles the problem by taking into account uncertainty in estimating the logging policy, and making corrections to the classical propensity scores based on uncertainty.
Throughout several experiments on a number of datasets, the authors show that the propose technique improves results over other commonly used techniques.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies off-policy evaluation and learning, under unknown propensities, which is pretty common for most complicated systems. The paper proposes a novel Uncertainty-aware Inverse Propensity Score estimator, which builds upon the original IPS estimator by adding an additional weight to reflect the uncertainty of the estimated propensity score. This specific weight is derived by optimizing an upper bound of the MSE of the estimator. Empirical results on synthetic and real-world dataset shows better average performance compared to other baselines, in terms of MSE (for evaluation), Precision@K, Recall@K, etc (for learning). 
strength_and_weaknesses: Strength:
 
1). The unknown propensities are very common for real-world problems, especially when the underlying system is complex and the propensities are hard to calculate. This paper proposes efficient off-policy evaluation/learning method under this realistic setting.

2). The paper is well-written and easy to follow.

3). The method is well-motivated, solid, and the idea about reweighing based on uncertainty is neat. 

3). It has comprehensive ablation studies and empirical results, with both synthetic and real-world datasets.

Weakness:

1). The proposed method seems have several important hyper-parameters, that control the balance of bias and variance term, as well as the confidence bound. The methodology part does not give a clear guideline about how to choosing them. From empirical studies, it seems to be selected from the unbiased validation set? If I understand it correctly, it might be hard to get this validation set in real-world scenarios. Could the authors comment more on how these are selected for different experiments? 

2). The uncertainty estimate is crucial for the uncertainty weight in Theorem 2. The NTK theory provides a nice quantification about the uncertainty for deep neural networks. However, I am concerned about the dimensionality of the gradient vector $g$, as well as the inversion stability of the matrix M. What are their corresponding dimensionality for the experiments? [1] uses the inverse of the diagonal of the matrix M as approximations. Could the authors comment more on how the quality of the uncertainty estimate affects both the evaluation and learning? It should be reflected in the MSE of the estimator as well as the policy optimization regret bound.

3). All the results are average results, without standard deviation. It would be great if the std is provided, and this could provide us more information about whether the gain over previous method is significant. 

4). Could the authors comment more on the convergence criteria? For the policy optimization step, is it performed until full convergence? This part seems missing from the paper, and it is worth commenting more about the computational cost as well. 



Reference:
[1]. Neural Contextual Bandits with UCB-based Exploration

clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper is well-written and most of the parts are clear.

Quality:
The paper is a solid paper. The method is well-motivated and clearly explained. The idea is neat with good empirical performance, though it would be great if the authors could provide more evidence about the significance of the gain. Theoretically, it would be great to analyze both the evaluation and learning performance, in terms of the quality of the uncertainty estimate. Empirically, the paper contains comprehensive results over both synthetic and real-world experiments, as well as valuable ablation studies (though the ablation over $\eta_1$ and $\eta_2$ is missing)? One weakness is the number of hyper-parameters used is high, and it is well-known for off-policy problems, these hyper-parameters are hard to select. The paper does not provide a concrete method on this part, which might reduce the impact of the paper. 

Novelty:
The idea proposed by adding additional weight to quantify the uncertainty in IPS estimate is novel and neat. 

Reproducibility:
The code is included.
summary_of_the_review: I like the topic this paper studies, which seems to be relevant to lots of practitioners who work on OPE/OPL problems. This paper proposes a novel uncertainty-based IPS estimator, which is clearly motivated. Some minor theoretical guarantee of the proposed estimator and learning performance is missing. The empirical results are comprehensive, however it is unclear the significance of the gain, as well as the applicability of the method, due to hyper-parameter tuning as well as the training cost. Based on the current version of the paper, it is around the borderline from my perspective, but I would consider changing the score based on the authors' reply.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies off-policy learning when there is distribution shift between the data distribution and target distribution. The paper points out that the vanilla inverse propensity score (IPS) estimator suffers from large variance and bias when the estimated data distribution is inaccurate. To this end, this paper proposes to take into consideration the uncertainty in the logging probability estimation, and constructs an uncertainty-aware IPS estimator. A mathematical framework about how to solve for the 
UIPS estimator is described. Experimental results on both synthetic data and real-world data show that the proposed estimator performs better than a few benchmark algorithms on recommendation tasks.

strength_and_weaknesses: ***Strength***:

1. The paper is written very clearly. The problem is well-formulated. The theorems are stated clearly and there is no ambiguity. The experimental settings and results are presented with details.

2. The experimental results are extensive. Various datasets and cases are studied. The results corroborate the proposed estimator.
---
***Weakness***: 

1. My major concern is that the technical novelty is unclear. 

(i). The existence of bias for the IPS estimator when the data distribution estimation is not accurate is a well-known result in the area of offline policy evaluation. 

(ii). IPS-based estimator is quite thoroughly studied. This paper studies the single-step setting, while IPS-type estimator is already extensively studied in the case of dynamic decision making/reinforcement learning. For example, in the DICE line of work ([1-3]), a series of distribution correction estimators are proposed (Also, these works are not discussed). Therefore, the correction term in this paper does not seem novel. 

(iii). The most significant technical contribution seems to be the objective function equation 8. However, this follows directly from the robust optimization as mentioned by this paper. 

Overall, it is not clear what is the technical novelty and contribution of this paper.

2. There is no theoretical guarantee for the sample complexity of the proposed estimator. (minor issue)
---
[1] O. Nachum, Y. Chow, B. Dai, and L. Li: DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems 32 (NeurIPS), spotlight, 2019

[2] R. Zhang, B. Dai, L. Li, and D. Schuurmans: GenDICE: Generalized offline estimation of stationary values. In the 8th International Conference on Learning Representations (ICLR), 2020

[3] B. Dai, O. Nachum, Y. Chow, L. Li, Cs. Szepesvari, D. Schuurmans: CoinDICE: Off-policy confidence interval estimation. In Advances in Neural Information Processing Systems 33 (NeurIPS), spotlight, 2020

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is very clear. Theorems are clearly stated without ambiguity. Empirical results are described with details so that readers can easily understand.


Quality: The theorems are clear and the proof looks correct to me, though I did not check all the proof details. The experimental result is good and extensive.

Novelty: The technical novelty is unclear. Specifically, (1) the existence and bias when the data distribution is poorly estimated is well-known; (2) similar IPS correction term is also considered in other works; (3) no theoretical guarantee for the proposed estimator (minor issue).

Reproducibility: The reproducibility is good. Experimental details are provided. 

summary_of_the_review: My current recommendation is reject. 

The main reason is that the technical novelty is unclear to me. I did not see what the technical contribution of this paper is. 

However, this paper does have good and quite extensive experimental results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes uncertainty modeling for estimated logging policies in the IPS estimator. Specifically, the authors express the IPS estimator such that it involves the ratio of estimated and unknown logging policies, and then they overestimate it. The approach does not come with guarantees but it is comprehensively compared to 10+ baselines.

This paper had a wide range of scores, from accept to reject. All reviewers agree that the studied problem is novel, important in practice, and may motivate follow-up works. The most negative review is because the approach does not come with an error bound. While this may be concerning, I do not think that it would be a big issue if the algorithm design was clean and sound. The problem is that this is not clear. Here are a few things that came out during an online meeting with the reviewers:

* From Theorem 1 to (6), an unknown quantity is replaced with a constant $\lambda$. This constant needs to be properly bounded.

* Since (7) behave like logistic models, their confidence intervals should be chosen accordingly. This is unclear. The authors cite a linear bandit paper and their high-probability region does not seem additive in confidence interval widths. I would also expect something like one over the minimum derivative of the mean function term. See [Provably Optimal Algorithms for Generalized Linear Contextual Bandits](https://proceedings.mlr.press/v70/li17c.html) and [Randomized Exploration in Generalized Linear Bandits](https://proceedings.mlr.press/v108/kveton20a.html) for proper confidence intervals in related generalized linear models.

* The algorithm seems complicated. I say "seems" because it is never precisely stated. Specifically, what is the advantage of the upper bound in Theorem 2 over having a high-probability lower bound on $\beta^*(a \mid x)$ in an IPS estimator with $\beta^*(a \mid x)$? This one could be derived using the confidence interval on page 5, no?

Given the above concerns, no reviewer felt strongly about accepting the paper. It is a borderline with a bias to reject.
justification_for_why_not_higher_score: The paper is on a good topic but the algorithm design has issues.

This paper should be treated as a **weak reject**, reject and bump up if there are no better papers.
justification_for_why_not_lower_score: This paper addresses an important practical problem and the empirical results are good.
Decision: Reject