Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new approach to off-policy evaluation for contextual bandits with continuous action space based on adaptive discretization of the action space. The authors mainly compare the proposed approach with approaches based on fixed-bandwidth kernel smoothing of the action space and show both theoretically and empirically that the proposed approach performs better in certain classes of value functions.
main_review: While the differences between piecewise constant approximation and a smooth kernel are well-known, the use of piecewise constant action-value estimation in this particular setting seems new to me. I believe the contribution is significant enough for publication in NeurIPS.

For fixed-bandwidth kernels, there is only one parameter to estimate whereas the proposed approach needs to evaluate a potentially large number of action-space partitions. Is this the main price to pay for the proposed approach? 

What about adaptive kernels with multiple bandwidths? Also, what about a tree-based discretization as in [Majzoubi et al 2020 Efficient Contextual Bandits with Continuous Actions] ?

Perhaps the authors can elaborate on the above.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper considered an off-policy evaluation method in continuous treatment settings and corresponding asymptotic properties are established. The experimental results are exciting.
main_review: Overall, the paper is well-written, and is very potential to be published in NeurIPS. However, some mathematical expressions, while correct, are hard to interpret, e.g. equation (8) and (9). It would be nice if the authors could provide a brief explanation in words.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies off-policy evaluation (OPE) with continuous actions (not the prevalent discrete action settings). To handle continuous treatments, the paper develops a new estimation method for OPE using deep  jump learning. The key idea is to adaptively discretize the continuous treatment space based on deep learning and multi scale change point detection. The proposed method is compared with some previous related methods from both theoretical and empirical perspectives.
main_review: The paper studies the important problem of OPE with continuous actions, which is also of interest to the NeurIPS audience. The paper nicely argues the weakness of the previous kernel-based approaches in Section 3.1 “Toy Example”. This toy example well motivates the proposed approach. Then, a novel adaptive discretization using deep learning is proposed. One advantage of DJL is that it does not need the tuning of bandwidth. The proposed DJL is theoretically analyzed and shown to converge at a faster rate compared to kernel-based estimators. Overall, the paper is well-written and the motivation is easy to follow. 
limitations_and_societal_impact: One concern is that the paper does not touch on some recent related works such as [a] and [b]. I think that the paper still makes solid contributions given these additional related works. Thus, it would be good for the authors to just cite these papers and clarify the difference.

[a] Nathan Kallus and Masatoshi Uehara. Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies. In Advances in Neural Information Processing Systems, 33 (NeurIPS 2020).

[b] Mert Demirer, Vasilis Syrgkanis, Greg Lewis, and Victor Chernozhukov. Semi-Parametric Efficient Policy Learning with Continuous Actions. In Advances in Neural Information Processing Systems, 32 (NeurIPS 2019).


Another point that I would like to state here is that the simulation study is extremely simple. Specifically, the methods are tested only on small sample settings (i.e., n=50-300). Thus, there remains some concerns about the empirical performance of the proposed procedure. For example, the authors suggest that $m=n/10$ to achieve a good balance between the bias and the computational cost. This seems to work with n=50-300, but it is unclear that the same suggestion is applied to larger samples such as n=1000,10000,.... Showing how the method works in large sample settings is also critical as it is general in the personalized pricing problem in the web industry. The paper also states that a longer computational time (compared to Colangelo & Lee) might be a potential limitation. Arguing this potential limitation is nice, however, it is desirable to test how many samples DJL can handle to better see its applicability.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces an alternative to doubly robust policy evaluation using kernel methods, which are limited by an assumption that the estimated state-action value (Q) function is smooth and has a second derivative. The proposed estimator, built from multi-scale change-point detection, is able to account for possible discontinuities in the Q function in a way that both reduces bias and variance. The proposed estimation approach is effectively shown to improve over prior methods on a suite of synthetic experiments as well as a real-world dataset.
main_review: #### **High-level evaluation**
Altogether, I appreciated the work presented in this paper. Off-policy evaluation is a difficult challenge facing real-world implementation of decision support algorithms. While I generally view the work positively and consider it highly, there are several areas where the paper could better describe the details of the technical foundations underlying the proposed DJL estimation approach. For this reason, I am a little hesitant to recommend that this paper be published. I have posed several questions below about some points that were unclear to me as a reviewer and someone who would potentially use this approach. I am quite ready to raise my score if the authors are able to satisfactorily address the concerns I raise in the Quality/Clarity section below as well as in the Limitations field below.

#### **Originality**
The paper points out an important limitation of recent continuous-action Doubly Robust OPE approaches in that they rely on the existence of the second derivative of the action-value function Q. The paper rescopes the reliance on the continuity of Q by estimating the expected outcome as a piecewise constant function, using a parametrized form of change-point detection. This function is adaptively discretized so that it accounts for local discontinuities in the estimated value function. 

#### **Quality and Clarity**
In the introduction, the limitations of kernel-based methods are referred to but not specified. Ideally, the paper would more directly list these specific limitations as well as how the proposed DJL addresses them. The paper would be easier to follow if these were made more clear.

Some questions that remain after reading Section 2.1:
Are the feature-treatment-outcome observations fixed in time? In other words, is the problem set up to only consider single step treatment scenarios? As a follow-up, what are the limitations of DJL when considering the extension to sequential decision making? Does the formulation of DJL (relying on a piecewise constant Q-function) work in sequential settings, when assigned treatments affect the future expected outcome?
Is the observed behavior assumed to be derived by a single policy? Is it known? My guess toward the second question, based on the use of propensity scores, is that it is unknown and estimated.
Is the observed outcome discrete or continuous?

The use of DJL in the toy example in Section 3.1 isn’t informative as fairly few details have been shared about the approach by that point. It’s inclusion with limited vague details was slightly confusing. The discussion in lines 136-141 helps but I think it would have been better to introduce the technical concepts first.

When mentioning the two model assumptions in Sec. 3.2 what are some specific scenarios that each assumption may cover? Being more direct at this stage may help solidify the expressed limitations of the kernel-based assumption of a smooth Q function and expose the need for DJL. Dynamic pricing is mentioned in passing earlier in the paper in regards to piecewise functions. Would medication dosing be the implied application for the second model assumption? Also, are the functions $q_{\mathcal{I}, 0}$ in model assumption 1 constant? In line 154, what does it mean for the Q(x,a) to be approximately constant?

I really appreciated the efforts by the authors to make the DJL algorithm easier to follow by breaking out the specific steps. I however found several points that were unclear. I’ve tried to highlight where I was confused in each step.

**data splitting** Are the $\mathbb{L}_\ell$ subsets disjoint? If not, how are they sampled to ensure balance? What does it mean that the different estimates will be aggregated? Is this aggregation just summing over the value estimate in Eqt. 10 and then averaging by the number of samples?  More precision here is particularly necessary as this is where the technical understanding of $\hat{D}$ is established.

**deep discretization** This step seems to be the most critical component of the algorithm. However, it is possibly the hardest to understand based on the current state of the writing. It took me several passes through the section spanning lines 201-221 to have some basic understanding of what is expected to happen. For example, it’s not clear what the change points list $\tau(v^*)$ corresponds to? Given the definition on line 211, it seems that it may be a telescoping set of indices. The recursive relationship could be described much better. Additionally, it would be nice if there was a more careful description of what $v^1$ minimizes. I assume that it corresponds to $\mathcal{C}^{(\ell)}(\mathcal{I})$ but this should be clearer. Including the pseudocode for Algorithm 1 would greatly help the presentation of this part of the paper, it should be a part of the main paper if the language in this part can’t be improved any further.

One concern that I have in terms of this step of the DJL algorithm is that it appears the deep discretization step can only reduce the size of $\hat{D}$. Meaning that there’s a potential sensitivity to the number of partitions that the interval is initialized to. 

**cross-fitting** There is too little detail included with regards to how the propensity score is estimated. Additionally, the first sentence following Eqt. (10) is unclear. Is a heldout set used to estimate $\hat{V}$ that is independent from what was used to estimate the functions in the deep discretization step?

In Section 5.1, is there a typo in **S2** in the second term? I think there should be a `+` between the first indicator function and the following bracket.

On line 308, it is stated that the dimension of the features is set to 20 but in most scenarios only 2 dimensions appear to be used. Is this to add extensive noise to the estimation process? What is the purpose of this? It seems superfluous and potentially a typo?

#### **Significance**
The paper addresses relevant literature and appropriately frames its contributions within the prior work. The proposed method is well evaluated against relevant kernel-based OPE estimation approaches and is shown to have smaller bias and variance in a suite of instructive synthetic experiments as well as on a dataset derived from real-world medication dosing. The experimental and theoretical analysis of the proposed DJL show that it is an improvement over contemporary kernel-based estimation approaches.

limitations_and_societal_impact: The limitation that I foresee with the proposed DJL estimation approach is that it requires extensive computational approximations of the piecewise Q function, the partitioning of the treatment space, as well as the propensity estimator. The authors conveniently ignore the complexity of learning these separate MLP functions and do not address the time or computational complexity required to implement their approach. While likely scaling better than the competing kernel methods, I feel that the authors should devote some space in their response to address the challenges and complexity of training the various estimators used to form their proposed DJL approach. This should also be included in the paper to more completely present the method.

Another limitation, (possibly should be mentioned in the Quality/Clarity section above) is that there is no clear discussion surrounding how one would be able to determine what modeling assumption to use when selecting between a piecewise of continuous Q function approximation. The relationship and apparent changes needed to the proposed DJL in the face of one over the other wasn't very clear.
review_title: null

Meta-review: This paper presents an off-policy evaluation method motivated by medical treatment settings, and provides a theoretical analysis as well as simulation experiments.  The reviewers support the paper overall and I recommend acceptance; that said, many refined points were raised during discussion, and I highlight the reviews (and discussion) by reviewers Ru8S and 54Kf.  The authors responded to these concerns extensively below, but I request they incorporate these discussions and suggestions in their revisions.
Decision: Accept (Poster)