Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to learn a Mahalanobis distance metric using local information in order to reduce the bias of kernel-based off-policy estimators for deterministic policies in continuous action spaces. The paper proves that when the density is twice differentiable, then the optimal matrix can be computed in closed form from the Hessian, and that when the optimal bandwidth is used their proposed estimator has a faster convergence rate than the alternative without the learned metric. Finally, the paper shows in experiments on synthetic data and the Warfarin Dosing dataset that it consistently improves over baselines.
strengths_and_weaknesses: Strengths:
1) The empirical performance is quite strong, which makes it more significant.
2) In terms of significance, the proposed method is fairly general, and can be dropped into any setting where there is a kernel-based approximation to a deterministic policy in a continuous action space.
3) In terms of quality, the paper evaluates the baseline algorithms on the same experiments as were used in the original papers for the baseline, suggesting that the method is able to improve performance on problems where the baselines were already strong.
4) The paper is overall well-written.

Weaknesses:
1) (minor) Clarity could be improved by explaining more of how the proofs work instead of just referring to the appendix.
questions: 1) (minor) Both of the chosen baselines are pretty strong, and the Kallus & Zhou paper already compare to a discretized baseline, but it might be helpful for new audiences to see a comparison with a discretized baseline to establish that both of the chosen baselines beat a simple approach.
limitations: The paper discusses the limitation that they assume a fixed bandwidth rather than finding a new bandwidth to adapt to the learned metric, and leave it open for future work. The paper also notes that the method is only proven to work for twice-differentiable, but that it empirically seems to work in other cases.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Motivated by the fact that target policy should be deterministic in certain real-world scenarios, this paper works on off-policy policy evaluation of deterministic policies in the contextual bandit setting. This work proposes to use local kernel metric learning for importance sampling to derive OPE estimates. For a given state, this proposed metric measures the distance between a behavior policy and a target policy based on the difference of the rewards of them. 
strengths_and_weaknesses: This work points out a view that we could measure similarities of different actions based on the rewards generated by them, instead of the Euclidean distances between them. This view motivates the authors to propose to use local kernel metric learning for importance sampling (MIS) for OPE estimates. The overall presentation of the idea and method is not very clear. For example, at the beginning of the introcution section, the authors seems to use offline reinforcement learning (RL) and off-policy RL interchangeably; in line 47, the authors mention the "bias" but fails to define or explain it from lines 41 to 46 or even earlier paragraphs, which is confusing. 

While OPE is a critical question in the RL field, there are limited innovations in this paper. Besides, given the recent extensive exploration of OPE for both discrete and continuous action setting, this work misses a significant part of existing work on this topic. 
questions: The main idea that motivates the local kernel metric learning for IS is that measuring the distance between actions based on resulting rewards instead of the Euclidean distance. Following this, how could the proposed approach reconcile with the classic attribution issue in reinforcement learning? As we know, the reward following an action may be attributed to more than one previous actions instead of only the most recent action, especially in the sparse reward setting. 
limitations: The authors have mentioned some potential limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies off-policy evaluation in contextual bandits with continuous actions, where prior works estimate a deterministic policy using the estimated performance of a kernelized policy to address the infinity variance problem of inverse propensity weighting in this setting. In particular, they focus on scenarios where the action-space is multi-dimensional, where Euclidean distance might be a poor measure of similarity between actions since the relationship between the reward difference and the distance difference in each direction might be quite different. So they propose to learn a metric of the action space so that similar actions (in terms of rewards) have smaller distance and vice versa.The metric is learned to approximately minimize the mean squared error of the kernelized IPW estimator. They conduct empirical results to show the effectiveness of the proposed method. 
strengths_and_weaknesses: 1. Selecting a good distance metric for off-policy evaluation in contextual bandits with continuous actions is an interesting problem. 

2. The paper is well-written and easy-to-follow. 

3. The proposed distance metric learning approach seems to make sense and the paper provides some theoretical analysis on its (approximate) optimality. 

Weaknesses

1. Most of the weaknesses I thought of are already mentioned in the conclusion section in the paper. The algorithm seems to require athe optimal bandwidth as input, which is not known; and there are many approximations in the theoretical analysis and the Hessian matrix. 
 
2. I am confused about Algorithm 1. is the kernel and its bandwidth inputs to the algorithm? If yes, please write it explicitly. 


questions: See weaknesses. 
limitations: I did not find limitations nor potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on off-policy evaluation in contextual bandits with continuous action space. To measure the similarity of a behavior action to a target action according to the difference of rewards, the paper proposes local kernel metric learning for importance sampling (MIS). By analyzing the leading order MSE with optimal bandwidth, the author shows that applying the metric obtained in Theorem 1 to kernel-relaxed IS estimate reduces the MSE. They also show that metric learning increases the convergence speed. Their experimental results verify their statements.
strengths_and_weaknesses: Strengths:
- Interesting question. In many real-world problems, we need to measure the similarities of actions according to their resulting rewards.
- Convincing experimental results.
- Well-written and easy to follow. Though I am new in this area, it is not hard for me to get the main idea and follow the discussion in the paper because the paper is well-written.

Weaknesses:
- Lack of theoretical guarantees. See Question below.

questions: - By applying the metric learning to kernel-relaxed IS estimate, how much MSE can be reduced with respect to $D$ and the sample sizes $N$? Is there a theoretical guarantee?
- Algorithm 1 applies a neural network reward regressor. How does the error of this regressor affect the MSE and the convergence speed? The author should discuss more on this these and may give some theoretical analysis.
- How to choose $\beta$ and $\gamma$ in equation (11)?

limitations: Assuming an optimal bandwidth is provided.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the problem of off-policy evaluation of deterministic target policies for continuous actions. In this setting, the typical importance sampling approach does not work, so previous papers deal with the problem by relaxing the target policy via the kernel smoothing technique. The authors improve the existing kernel smoothing, especially for multi-dimensional continuous action setting, by proposing a method to data-drivenly learn a Mahalanobis distance metric in the action space. More specifically, the metric is learned to minimize the bias of the resulting estimator. The authors provide some theoretical analysis of the proposed MIS estimator showing that MIS converges faster than IS without metric learning. Moreover, some empirical analysis is performed demonstrating that MIS is more accurate in terms of MSE for a range of bandwidths compared to the baseline estimators.
strengths_and_weaknesses: Strengths
- makes a solid improvement of the kernel smoothing technique used for continuous action OPE by learning a distance metric from the data 
- theoretical and empirical evidence are provided to support the superior OPE performance of the proposed approach compared to the conventional approach without any distance metric

Weaknesses
- the proposed estimator needs an accurate estimation of the expected reward function, and there seems to be no theoretical analysis on how the estimation error of the expected reward function affects the OPE accuracy of the resulting estimator
- there is no theoretical and empirical comparison between the proposed method and the discretization approach of [Cai et al. 2021]
- Some important experiment configurations are missing (see below)

questions: - Could you compare the convergence rate of the proposed method and the estimator based on the discretization proposed by [Cai et al. 2021]? [Cai et al. 2021] compares their discretization approach with the kernel smoothing approach in their theoretical analysis.

- Did you perform an experiment to compare the estimators with increasing dimensions of the action space? There should be this kind of experiment given the discussion around Proposition 3.3 in Section 3.

- Did you use the same reward regressor for DM and MIS? I was wondering where the large difference between these two estimators comes from, even though both estimators rely on the estimated expected reward. Some explanation would be appreciated. 

- Related to the above question, how does MIS work when you gradually increase the noise on the reward? (the reward function estimation will become gradually difficult)

- Does the proposed method improve the bias or the variance in the experiments? I was curious about the bias-variance decomposition in the experiments. So, it would be great to have additional figures about the bias-variance decomposition, at least in the appendix. 

- Why does DM degrade its accuracy with increasing sample size in Figure 3 (a)?

[Cai et al. 2021] Hengrui Cai, Chengchun Shi, Rui Song, and Wenbin Lu. Deep jump learning for off-policy evaluation in continuous treatment settings. Advances in Neural Information Processing Systems, 34, 2021.

limitations: The proposed approach is not compared with the discretization approach of [Cai et al. 2021], neither theoretically nor empirically. 

There is also no related work section, even though there seems to be some space enough to include one additional section. It would be great to have some discussion on the comparison between the kernel smoothing method (authors’ approach) and discretization for continuous action OPE. Moreover, it would be useful to summarize a broader literature of OPE for contextual bandits and reinforcement learning.

An acronym “MIS” has already been used for “Marginalized Importance Sampling”, which is an OPE estimator for reinforcement learning proposed by [Xie et al. 2019].  

- [Xie et al. 2019] Tengyang Xie, Yifei Ma, Yu-Xiang Wang. Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling. Advances in Neural Information Processing Systems, 32, 2019.

Typos
- L46, a bug in citation
- Eq 5, an unnecessary bracket in the LHS of the equation
- L144, presented in 4 -> presented in Eq.(4) ?
- L246, $a_{min}, a_{max}$


=== after the response ===

The author response, additional experiments, and the new acronym of the proposed method seem reasonable to me, so I increase my score to 6. Please definitely add the related work section, in particular, a comparison with [Cai et al. 2021] would be helpful to the readers.
review_title: null

Meta-review: metareview: Reviewers and AC are satisfied by the authors' response and think the paper should be accepted. 
confidence: Certain
Decision: Accept