Deep Jump Learning for Off-Policy Evaluation in Continuous Treatment Settings

Abstract: We consider off-policy evaluation (OPE) in continuous treatment settings, such as personalized dose-finding. In OPE, one aims to estimate the mean outcome under a new treatment decision rule using historical data generated by a different decision rule. Most existing works on OPE focus on discrete treatment settings. To handle continuous treatments, we develop a novel estimation method for OPE using deep jump learning. The key ingredient of our method lies in adaptively discretizing the treatment space using deep discretization, by leveraging deep learning and multi-scale change point detection. This allows us to apply existing OPE methods in discrete treatments to handle continuous treatments. Our method is further justified by theoretical results, simulations, and a real application to Warfarin Dosing.


Reviews and author responses:

 We want to thank you again for your thoughtful suggestions and positive comments on our paper. And we are very glad that our response has addressed your concerns. We will add the discussion of computational complexity, and make the linear time complexity (i.e., the number of training a DNN) clear in the revised paper. Please, let us know if you have any other comments or questions. We would be happy to do our utmost to address them!
 

 We want to thank you again for your thoughtful suggestions and positive comments on our paper. And we are very glad that our response has addressed your concerns. Please, let us know if you have any other comments or questions. We would be happy to do our utmost to address them!

 We want to thank you again for your thoughtful comments and suggestions! We were wondering whether our response has sufficiently addressed your concerns. Please, let us know if you have any other comments or questions. We would be happy to do our utmost to address them!

 We want to thank you again for your thoughtful suggestions and positive comments on our paper. Please find our response to your additional valuable comments in the following. We will revise the paper by taking into accounts all your suggestions.

1. We greatly appreciate the reviewer's effort in validating the complexity of our algorithm. We will add the discussion of computational complexity, and make the linear time complexity (i.e., the number of training a DNN) clear in the revised paper.

2. We thank for your clarification and follow up about the dynamic pricing example. We will include more descriptions of why there is a discontinuity demand function. Specifically, a product could attract a new segment of customers if the seller reduces the price below a certain threshold. Thus, there will be a discrete upward jump in demand by a small price reduction, implying a discontinuity in the demand function.

In the context of policy evaluation with continuous treatments, this paper proposes to learn a discretization of the treatment space, and reduce the problem to learning with discrete treatments.  The core motivation is to avoid some of the difficulties associated with kernel-based methods, where a single bandwidth parameter may be insufficient to capture abrupt changes in the distribution of $E[Y | a, x]$ with respect to $a$.  To learn this discretization, the paper leverages ideas and algorithms from the change-point detection literature, and uses DNNs to learn the $Q$-function within each interval.  Theoretical and empirical results demonstrate advantages of this approach over kernel-based methods.
  ## Strengths

This paper introduces a novel and intuitive idea, and the technical motivation is well-explained, both in the introduction & through the use of a toy synthetic example.  Clear intuition is provided for the limitations of kernel-based methods, and how the proposed method overcomes those limitations.

Moreover, I found the experimental validation of the idea to be compelling, both in the synthetic experiments and the semi-synthetic warfarin experiment. Importantly in the latter case, the outcome is simulated by a learned model, not one that has been a-priori designed to showcase the strength of the proposed approach.  Nonetheless, the proposed approach yields substantial gains over existing approaches.

## Weaknesses

Particularly with respect to the theoretical results & algorithmic contributions, there are a few places where additional clarity would be welcome.

First, it is stated that Step 2 of the Algorithm is "inspired" by the pruned exact linear time (PELT) algorithm of Killick et al. 2012. What are the differences? My understanding is that the proposed change-point detection algorithm differs only in the cost function used (which incorporates the training of DNNs).  In particular, part III.3 of Algorithm 1 in Supplement A differs, but part III.4 is otherwise the same.  Is my understanding correct?  If so, perhaps this could be clarified further in the paper.

Second, the relationship to existing work in deep learning theory (e.g., Farrell et al. 2021) is somewhat unclear.  While the proofs do an adequate job of clarifying where previous results are used, I am missing the big picture: The unique challenges in deriving the results of Theorem 1 / 2, and the strategy used to overcome those challenges.  As an example of what I'm looking for:  It seems that one challenge in deriving Theorem 1 lies in controlling the width & number of the intervals, so that uniform guarantees can be derived across all intervals, and this is done by the choice of $\gamma$.  More discussion of intuition along these lines would be appreciated.

Third, more discussion may be warranted on computational complexity as a potential limitation; As I understand it, part III.4 of the algorithm (using the pruned exact linear time method) is linear in the number of samples, but part III.3 of the algorithm is quadratic in the number of candidate change-points $m$ (that is, $O(m^2)$ DNNs must be trained).  Since it is assumed that $m$ is of the same order as $n$, this would seem to imply that the computational complexity is at least $O(n^3)$, for $O(n^2)$ DNNs that must each look at $O(n)$ samples.  More discussion on this point would be appreciated.

Finally, with respect to the original motivation, it may be worth providing a simple example in the paper of where $Q(a, x)$ would be expected to be discontinuous in $a$, as this feels rather unintuitive as a general rule.  From a skim of den Boer & Keskin 2020 (cited in this work), it seems like the prime examples are pricing-related, where small changes in price can win access to a new segment of customers (e.g., if this causes your product to become featured on an e-commerce site).

## Minor typos

* Line 209: "requires to apply"
* Supplement, Equation below line 59 in the proof of Lemma D.1, should the first inequality should go the other way (already corrected in the following line)?
* Supplement, Line 121, $1 - O(n^2)$ should be $1 - O(n^{-2})$?

## Justification of score

I weighted most highly the novelty (and clear intuition) of the method, as well as the empirical comparisons to kernel-based methods. I also appreciated the theoretical results (particularly Section 4.3, comparing to kernel methods), but I weighted these less highly in my score, as I am less familiar with the relevant literature at the intersection of learning theory & DNNs;  This latter point also justifies my confidence score.

While I highlight (in some detail) a number of places where I feel that clarity can be improved, I do not feel that these concerns are insurmountable.  In particular, I am willing to revise my score upwards given clarifications on these points by the authors. Outside of my points above, I feel that the authors have adequately discussed the limitations of their work & potential societal impact.

 Thank you for the detailed clarifications!  In light of the clarifications & proposed changes, I am increasing my score to a 7.  Two notes: 

1. As reviewer fMSm stated, the main concern would be the number of times that "train a DNN" is invoked (not so much the time it takes to train each individual DNN).  I've taken a look at the source code (and run it to see what the # of iterations looks like) and I see your point here.  Thank you for clarifying, and for including code that's easy to run!  Please do make sure to clarify this point for future readers.
2. I apologize for any confusion re: "providing a simple example" of discontinuity - I only meant that e.g., the reference to dynamic pricing on lines 33-35 could be expanded to give some intuition for **why** discontinuities arise in that setting (in lieu of making an unfamilar reader chase down the reference).  Certainly a real-data example would strengthen the paper, but that was not the intent of my comment.

Authors have studied the off-policy evaluation (OPE) problem in continuous treatment setting. Their estimation is based on deep jump learning. They adaptively discretize the treatment space using deep learning and multiscale change point detection. Through this discretization, they are able to then apply OPE methods used for discrete treatment setting.  - One major limitation of this work is the restrictive assumption of having  a piecewise constant value function which is mentioned in Model 1. If we choose Model 2 and go with the continuous value function, the discretization will be a challenging problem itself. One thing that authors have emphasized is the superiority of their work compared to kernel-based methods since they do not need to determine the kernel bandwidth. However, in the case of continuous value function (Eq. 5), we have the regularization parameter that needs to be picked up. Therefore, although there is no kernel bandwidth, we have another hyperparameter to search for.

- Another issue is that authors have not really proposed a new OPE estimator. They have instead shifted their focus on continuous treatment setting for which they use some discretization to bring it back to the discrete treatment setting and then use a doubly-robust estimator to solve the OPE problem. This limits the contribution of this work in spite of all theoretical results provided.

- One thing which makes the evaluation of this paper more difficult is the lack of standard datasets used in other OPE papers. As an example, the SLOPE paper that authors have compared their results with, has reported results on different environments such as Mountain Car, GridWorld, etc.

-----------------------------------

Post-rebuttal:

Thanks to authors for addressing my concerns. I have updated my score accordingly. Authors have not provided discussions on limitations and societal impacts of their work.

The paper is concerned with performing policy evaluation from logged data (i.e. contextual bandit-style data) in settings where there is no unmeasured confounding and where treatments are continuous. Unlike previous works that deal with this problem by replacing indicator terms that observed and policy-prescribed treatments match replaced with kernel-smoothing terms, they instead propose to deal with this by learning a “good” discretization of the treatment space, such that the discretization allows for a good fit of the mean-outcome function. They propose a method for learning such a discretization involving fitting outcome functions with deep learning, and refining the partition using dynamic programming, and they evaluate their approach on a range of synthetic and real-data problems.   Overall, this is a fairly good paper. They provide a nice motivation for the issues with the existing kernel-based approaches for continuous treatments, with some clear explicit examples of where they may perform poorly (in particular where different kernel bandwidths are needed for different parts of the treatment space), and their proposed method seems to address these concerns. Also, the experimental results seem compelling. However, there are also some limitations of the paper as currently presented, which I would like to see addressed by the authors in order to increase my review.


Main Strengths:
- Motivation for the approach, and weaknesses of existing methods, is very clear and well presented
- Experiments seem to be very thorough, and the empirical performance of their proposed method seems to be very strong
- They also provide clear theoretical rate guarantees for the consistency of their methodology, including in cases where the true mean outcome function is actually continuous (and not given by a discretization of the treatment space)

Main Weaknesses:
- The biggest weakness to me is the presentation of their main dynamic programming algorithm in Section 3.3. I’m having a very difficult time understanding explicitly what is going on with the algorithm, and what the computation looks like. It would probably be very helpful to have it clearly presented in actual algorithm form, rather than in paragraphs. In particular, I would like to have a sense of how much computation (in terms of how many times the “fit a deep learning model” subroutine needs to be run) the algorithm involves, in terms of best-case, worst-case, typical-case, etc. Clearly there are a quadratic number of possible segments of the data, so this subroutine would need to be called no more than O(n^2) times, but a quadratic number of deep-learning optimizations would be extremely expensive and cumbersome when n isn’t tiny, so it would be helpful to know if/when we can expect better scaling (e.g. O(n) or O(n log n)).
- In the paragraph following Theorem 1, you give the rates for the bias and standard error terms. However, in general these rates are different, which would indicate to me that there is not an optimal bias/variance tradeoff. In particular, when $\beta < p$, bias would be strictly worse than $O_p(n^{-1/2})$, which would indicate that maybe there is a way of improving things by increasing variance and decreasing bias. It would be interesting to comment on / discuss this issue.
- Methods based on discretizing state or treatment spaces when things are actually continuous are definitely controversial. While arguably discretization shouldn’t be done when things are actually continuous, one could still argue that there could be various practical reasons / benefits of discretization. Furthermore, a very obvious approach to dealing with the issues of needing different bandwidths for different parts of treatment space in kernel methods would be to learn a bandwidth function (i.e. as a function of treatment), which would not require discretization. Although I think ultimately it would be unfair to penalize the paper for going down the discretization route on any kind of idealogical objection, there does need to be some discussion of these issues, which is currently lacking.

Other minor issues / comments / suggestions / typos:
- On lines 80/81, you say that the purpose of the DR formulation is to guard against misspecification. However, there are other reasons to use this DR formulation; for example, it can be semi-parametrically efficient in situations where the importance sampling or direct method formulations are not.
- On lines 99/100 you say that within each segment the expected outcome is a constant function. However, this could only be the case if you had enough data such that there was at least one data point lying within each segment (otherwise, no segments based on the data points Y_{1:n} could ensure this).
- There are a lot of issues with text going into the margins that should be fixed; e.g. with equations 5 and 10, or the equation after line 120.
- What does the notation $J(D)$ mean in Theorem 1, I don’t think this was defined anywhere? (I think this is meant to be the set of change-points within the discretization $D$, but this should be defined)
- In Theorem 1 you require that $\gamma_n$ decays to zero slower than some minimum rate, but in Theorem 2 it seems like instead you require $\gamma_n$ to decay at an \emph{exact} rate (rather than at least as slow as some rate). Is this intentional, or should the requirement in Theorem 2 been along the lines of $\gamma_n >> \ldots$? I have no issues with this.

 Thanks for the very detail response! Given the proposed changes, and extra details about running time of the PELT algorithm, I am willing to up my rating to a 7. Note that I wouldn't necessarily suggest including all of the detail about how long it takes to train a neural network, I think that is a bit extra and not the point; I think the main thing of interest is how many times the "train a deep neural network" subroutine is run.

 We sincerely appreciate the reviewer's great efforts in reviewing our paper and thank the reviewer's valuable comments, many of which will lead to a more readable and self-contained version of our paper. We attempt to address all the points one by one in the following. We will revise the paper by taking into accounts all the reviewers' comments/suggestions.

**Clarification on the DJL Algorithm**

We apologize for the confusion. We presented the DJL method in pseudocode in Appendix A, Algorithm 1 of the supplementary article due to page limit. We will move Algorithm 1 to the main paper to improve the readability and revise it accordingly to our response to *Reviewer # Dcr5* under the “computational complexity” section. 

In the following, we address the rest of your questions one by one.

1. ‘I would like to have a sense of how much computation (in terms of how many times the “fit a deep learning model” subroutine needs to be run) the algorithm involves, in terms of best-case, worst-case, typical-case, etc.’

First, we want to clarify that although there are a quadratic number of possible segments, the expected computation time is linear in $m$, since we use a pruning step to discard change points not relevant to obtain the final discretization (see Equation (9)). The worst-case computation time is quadratic in $m$, as the number of possible segments is $\mathcal{O}(m^2)$. The best-case computation time is linear in $m$ as well. 

We next analyze the computational complexity for the proposed method as follows. There are three main dominating parts of the computation: the adaptive discretization, the estimations of conditional mean function and the propensity score function, and the construction of the final value estimator.

First, for the adaptive discretization on the treatment space (the main part of DJL, see Algorithm 1 Part III.4), we use the pruned exact linear time (PELT) method in Killick et al. (2012) to solve the dynamic programming (see lines 206-208). This step requires at least (i.e., best-case) $\mathcal{O}(m)$ computing steps and at most (i.e., worst-case) $\mathcal{O}(m^2)$ steps (Friedrich et al. 2008). According to Theorem 3.2 in Killick et al. (2012), the expected (i.e., typical-case) computational cost is $\mathcal{O}(m)$. 

• *Killick, Rebecca, Paul Fearnhead, and Idris A. Eckley. "Optimal detection of changepoints with a linear computational cost." Journal of the American Statistical Association 107.500 (2012): 1590-1598.*

Second, for each step in the linear complexity of adaptive discretization, we need to train the deep neural network for the conditional mean function and the propensity score function to calculate the cost function. Here, the time and space complexity of training a deep learning model vary depending on the actual architecture used. In our implementation, we employ the commonly used multilayer perceptron (MLP) to estimate the function $Q$ and the propensity score in each segment. Suppose we use the standard fully connected MLPs of $w$ width and $d$ depth with feedforward pass and back-propagation under total $e$ epochs. Then according to the complexity analysis of neural networks (see the related link below), the computational complexity of modeling the function $Q$ and the propensity score is $\mathcal{O}(2 *n e (d-1) w^2 )$.

• *https://ai.stackexchange.com/questions/5728/what-is-the-time-complexity-for-training-a-neural-network-using-back-propagation/5730*

For the last part, the construction of the final value estimator based on $\mathcal{L}$-fold cross fitting, which repeats the above two steps $\mathcal{L}$ times. Therefore, by putting the above results together, the total expected computational complexity of the proposed DJL is $\mathcal{O}(\mathcal{L} *m *2 *n  e (d-1)w^2)$.
 
Note that the computation for the last part (i.e., cross-fitting) can be easily implemented in parallel computing, and thus the total expected computational complexity of the proposed DJL can be reduced to $\mathcal{O}( m *2 * n  e (d-1)w^2)$. 

We will include the above discussion on the computational complexity for the proposed DJL shall our paper be accepted.

**Optimal Bias/Variance Tradeoff in Theorem 1**

This is a very good and interesting point to discuss! When $\beta<p$, i.e., the underlying conditional mean function $Q$ is not smooth enough, we acknowledge that the bias will increase and the value might converge at a rate that is slower than the usual parametric rate. To address this concern, we can employ the A-learning method (see e.g., Schulte, Phillip J., et al. (2014)). The A-learning method is more robust and requires weaker conditions to achieve the parametric rate. Specifically, it only requires the difference $Q(x,1)-Q(x,0)$ to belongs to $\Phi(\beta,c)$. This is weaker than requiring both $Q(x,1)$ and $Q(x,0)$ to belongs to $\Phi(\beta,c)$. We will add the related discussion shall our paper be accepted.

• *Schulte, Phillip J., et al. "Q-and A-learning methods for estimating optimal dynamic treatment regimes." Statistical science: a review journal of the Institute of Mathematical Statistics 29.4 (2014): 640.).*

**Discussion under Continuous Settings**

Thank you very much for this excellent suggestion! First, as you have commented, there are various practical benefits of discretization. For instance, discretization would yield a set of dose intervals to be used for personalized dose recommendations. Recommending a dosing interval would be more flexible than recommending a single dose level, as it gives more options to implement in practice.  

Second, the proposed method can be understood as a kernel-based method with a boxcar kernel, as we adaptively discretize the action space into a set of nonoverlapping intervals. It is interesting to explore how to extend our proposal with general kernel functions. We leave it for future research.

**Clarifications on Minor Issues / Comments / Suggestions / Typos** 

1. Thank you for providing more reasons for using the DR formulation. We will include *‘it can be semi-parametrically efficient in situations where the importance sampling or direct method formulations are not’* in our revised paper.

2. As you have commented, we need some data points lying within each segment in order to consistently estimate the expected reward. Otherwise, the piecewise constant function cannot be consistently estimated. We will make this clearer shall the paper be accepted. 

3. Thank you for pointing it out. We will fix these margins in the revised paper.

4. Apologize for the confusion. We did define $\mathcal{J}(\mathcal{D})$ as the set of change point locations (see lines 145-146). We will remind readers of this notation in the statement of Theorem 1 to avoid confusion.

5. Please note the theoretical order of $\gamma_n$ is different in Theorems 1 and 2, due to the different model assumptions imposed on the conditional mean function $Q$. In Theorem 1, under the piecewise model assumption (see Model 1), the requirement of $\gamma_n$ is weaker. The optimal rate of convergence can be achieved under a wide range of $\gamma_n$. In contrast, in Theorem 2, the conditional mean function $Q$ is a smooth function. This corresponds to a harder scenario and we need $\gamma_n$ to decay at a specific rate to establish the optimal rate of convergence.

We will include all the above discussions and explanations shall the paper be accepted. 

We once again appreciate your great effort in reviewing our paper. We hope that the above discussions have addressed all your comments.


 We are grateful for the reviewer's effort and valuable comments on our work. We attempt to address all the points one by one in the following. We will revise the paper by taking into accounts all the reviewers' comments/suggestions.

**Advantages of DJL over Kernel-based Methods and the Challenge of Discretization under Model 2**

First, we would like to clarify the advantages of the proposed DJL over kernel-based methods. We acknowledge that our procedure also relies on certain tuning parameters. However, the bandwidth specification is not a major limitation of kernel-based methods. Instead, they suffer from two major limitations, as discussed in the introduction (see lines 31-38): 1. They require the mean outcome to be a smooth function over the treatment space, and thus cannot handle discontinuous functions such as in dynamic pricing example; 2. They typically use one single bandwidth and might be suboptimal in settings where the degree of smoothness of the conditional mean function $Q$ varies over the treatment space (under Model 2). Please refer to our discussions in the toy example in Section 3.1 for details. Our proposed method addresses those limitations by adaptively discretizing the action space. The discretization enables us to handle the discontinuous value functions. The adaptivity ensures the method has certain optimality under settings where the degree of smoothness of the conditional mean function $Q$ varies over the treatment space.

Second, we do not think discretization is a very challenging problem under Model 2. Actually, the proposed methodology is the same regardless of what model assumption is being imposed. We show that our method is consistent and derive the rate of convergence under Model 2. In the change point literature, researchers also proposed to use a piecewise constant function to approximate a smooth signal. 

**Contributions as New OPE Method**

With all due respect, we strongly disagree with your comment that our method is not a new OPE estimator. Following your logic, the double machine learning paper by Chernozhukov et al. (2018) has limited contribution as they did not introduce a new OPE estimator. Instead, they simply combine machine learning with the doubly-robust estimator to solve the causal inference problem.  In addition, the works of Kallus & Zhou (2018), Colangelo & Lee (2020), Su, Srinath & Krishnamurthy (2020) all rely on the doubly robust estimator. Their contributions are limited as well following the reviewer’s statement. 

We next summarize the contribution of our method. As commented by other reviewers, we propose a new method using deep jump learning that adaptively discretizes the treatment space, estimates a deep model in each discretization, and combines them using a doubly robust estimator. The core motivation is to avoid some of the difficulties associated with kernel-based methods, where a single bandwidth parameter may be insufficient to capture abrupt changes in the distribution of $E(Y|a,x)$ with respect to $a$. The paper leverages ideas and algorithms from the change-point detection literature, and uses DNNs to learn the conditional mean function within each interval. Theoretical and empirical results demonstrate the advantages of this approach over kernel-based methods.

**Clarification of Real Dataset** 

First, we would like to clarify that we use the same real dataset considered in Kallus & Zhou (2018) to compare different OPE methods. See Section 5.2 in our paper and Section 5.3 in Kallus & Zhou (2018). Second, we remark that the Mountain Car and GridWorld environments considered in Section 5 in the SLOPE paper by Su, Srinath & Krishnamurthy (2020) correspond to the setting of sequential decision making. This is different from the batched contextual bandits (single-stage decision-making) setting as considered in our paper. It would be practically interesting to extend our work under the reinforcement learning setting and compare these methods using the datasets that you mentioned.

We will include all the above discussions shall our paper be accepted. 

We once again appreciate your great effort in reviewing our paper. We hope that the above discussions have addressed all your comments.


 We sincerely thank the reviewer's great effort and valuable comments, many of which will lead to a more readable and self-contained version of our paper. We attempt to address all the points one by one in the following. We will revise the paper by taking into accounts all the reviewers' comments/suggestions.

**Clarification on PELT Algorithm**

Thank you for pointing this out! As you have mentioned, the difference only lies in the cost function where we incorporate the training of DNN in our proposal. We acknowledge that the word “inspired” is not proper. It might be more appropriate to say “we employed the PELT method to identify the change points with a cost function that involves DNN training”. We will revise the paper accordingly shall it be accepted.

**Relationship to Existing Work in Deep Learning Theory**

We next clarify our theoretical contributions compared with the deep learning theory established in Farrell et al. (2021). First, Farrell et al. (2021) considered a single deep neural network (DNN), whereas we established the uniform convergence rate of several DNN estimators, since our proposal requires training multiple DNN models. Establishing the uniform rate of convergence poses some unique challenges in deriving the results of Theorems 1 and 2. As you have commented, we need to control the initial number of the intervals $m$ to be proportional to $n$ and the order of penalty term $\gamma_n$, so that a uniform convergence rate can be established across all intervals. To address this difficulty, we derive the tail inequality to bound the rate of convergence of the DNN estimator and use Bonferroni’s correction to establish the uniform rate of convergence. We will include the above discussion and clarification shall our paper be accepted.

**Computational Complexity**

This is an excellent comment! First, we apologize that we did not present our algorithm correctly. Specifically, part III.3 of Algorithm 1 needs to be removed from the algorithm and we calculate the cost function $\mathcal{C}(\nu/m, \nu^*/m)$ only when it is needed in the recursive updating equation in (8). Please refer to our source code (in *DJL.py* lines 49-68) provided in the supplementary material. Consequently, the computation time is not quadratic in $m$, but linear in $m$ instead. We will revise Algorithm 1 accordingly shall the paper be accepted.

We next analyze the computational complexity for the proposed method as follows. There are three main dominating parts of the computation: the adaptive discretization, the estimations of conditional mean function and the propensity score function, and the construction of the final value estimator.

First, for the adaptive discretization on the treatment space (the main part of DJL, see Algorithm 1 Part III.4), we use the pruned exact linear time (PELT) method in Killick et al. (2012) to solve the dynamic programming (see lines 206-208). This step requires at least $\mathcal{O}(m)$ computing steps and at most $\mathcal{O}(m^2)$ steps (Friedrich et al. 2008). According to Theorem 3.2 in Killick et al. (2012), the expected computational cost is $\mathcal{O}(m)$. 

• *Killick, Rebecca, Paul Fearnhead, and Idris A. Eckley. "Optimal detection of changepoints with a linear computational cost." Journal of the American Statistical Association 107.500 (2012): 1590-1598.*

Second, for each step in the linear complexity of adaptive discretization, we need to train the deep neural network for the conditional mean function and the propensity score function to calculate the cost function. Here, the time and space complexity of training a deep learning model vary depending on the actual architecture used. In our implementation, we employ the commonly used multilayer perceptron (MLP) to estimate the function $Q$ and the propensity score in each segment. Suppose we use the standard fully connected MLPs of $w$ width and $d$ depth with feedforward pass and back-propagation under total $e$ epochs. Then according to the complexity analysis of neural networks (see the related link below), the computational complexity of modeling the function $Q$ and the propensity score is $\mathcal{O}(2 *n e (d-1) w^2 )$.

• *https://ai.stackexchange.com/questions/5728/what-is-the-time-complexity-for-training-a-neural-network-using-back-propagation/5730*

For the last part, the construction of the final value estimator based on $\mathcal{L}$-fold cross fitting, which repeats the above two steps $\mathcal{L}$ times. Therefore, by putting the above results together, the total expected computational complexity of the proposed DJL is $\mathcal{O}(\mathcal{L} *m *2 *n  e (d-1)w^2)$.
 
Note that the computation for the last part (i.e., cross-fitting) can be easily implemented in parallel computing, and thus the total expected computational complexity of the proposed DJL can be reduced to $\mathcal{O}( m *2 * n  e (d-1)w^2)$. 

We will include the above discussion on the computational complexity for the proposed DJL shall our paper be accepted.

**Discontinuous Example and Real Dynamic Pricing Dataset**

Thanks for this excellent suggestion! We do have data for dynamic pricing from an E-commerce platform. However, the dataset is very sensitive for external use due to its privacy and confidentiality. As such, we did not include such an example in our paper. We are working on the de-identification of the dataset for possible publication in the future. We remark that in our simulation study, we included Scenarios 1 and 2 (see also line 305 of the main text) to generate data under settings where the conditional mean function $Q$ is a piecewise constant function of the action.

**Clarification on Minor Typos**

Thank you for pointing out these typos in the main and the supplementary article. They will be corrected shall our paper be accepted. 

In addition, we will include all the above discussions and clarifications shall our paper be accepted. 

We once again appreciate your effort in reviewing our paper. We hope that the above discussion has addressed your comments.


 We greatly appreciate the reviewer's effort and valuable comments, many of which will lead to a more readable and self-contained version of our paper. We attempt to address all the points one by one in the following. We will revise the paper by taking into accounts all the reviewers' comments/suggestions.

**Function Classes in Section 4**

We focus on the Holder smooth function class in Section 4 because it is one of the most commonly imposed function classes in the current literature (Farrell, et al., 2021). As you have commented, it is a special example of the function classes that can be learned by neural nets. Our proposed DJL is also valid when the conditional mean function $Q$ belongs to a non-smooth function (as discussed in lines 226-228) introduced by Imaizumi & Fukumizu (2019). Our current proof can be further generalized to any function class that can be learned by neural nets at a certain rate. We will include more discussions after line 228 shall the paper be accepted.

• *Imaizumi, M. & Fukumizu, K. (2019), Deep neural networks learn non-smooth functions effectively, in ‘The 22nd International Conference on Artificial Intelligence and Statistics’, PMLR, pp. 869–878.*

**Tightness of Convergence Rate**

This is an excellent comment! We acknowledge that the convergence rate in Theorem 2 may not be tight. To the best of our knowledge, no formal lower bounds have been established in the literature in the continuous treatment setting. This is a very challenging task as we focus on the error bounds associated with the value, instead of the conditional mean function. In the literature on multi-scale change point detection, there are lower bounds on the estimated time series (Boysen et al., 2009). However, they considered settings without baseline covariates and it remains unclear how the rate of convergence of the estimated piecewise function can be translated into that of the value. It would be interesting to explore whether the rate of convergence in Theorem 2 is tight or not. However, it is beyond the scope of the current paper and we leave it for future research. 

• *Boysen, L., Kempe, A., Liebscher, V., Munk, A., Wittich, O. et al. (2009), ‘Consistencies and rates of convergence of jump-penalized least squares estimators’, The Annals of Statistics 37(1), 157–183.*

**Computational Time**

We make a few clarifications. First, the proposed DJL is much faster than the kernel method developed by Kallus & Zhou (2018). This is mainly because Kallus & Zhou (2018) developed a very complex procedure to find the optimal bandwidth. Specifically, they need to compute the kernel densities and the second derivative of these densities using the Hessian matrix on each observed data point. It was mentioned on page 6 of Section 5 in Kallus & Zhou (2018) that their method is computationally intensive. The long computation time is also mentioned in some other papers (see page 7 of Section 4.2 in Su, Srinath & Krishnamurthy (2020)). Second, if the kernel bandwidth is fixed, then the resulting kernel method (Colangelo & Lee, 2020) is much faster than the proposed method. Please refer to Table 1 in Appendix B for the detailed computation time (in minutes) of the proposed DJL and three kernel-based methods in Scenario 1. See also related discussions in lines 315-318. We do observe a slight difference in computational time across different scenarios. As commented by the reviewer, Scenarios 1 and 2 require relatively shorter time to find the best partition by the DJL due to the underlying piecewise function $Q(x,a)$, whereas it takes a relatively longer time in continuous settings including Scenarios 3 and 4 to identify the change points. The expected computational complexity in adaptative discretization based on the pruned exact linear time (PELT) method in Killick et al. (2012) is $\mathcal{O}(m)$.

• *Kallus, N. & Zhou, A. (2018), Policy evaluation and optimization with continuous treatments, in ‘International Conference on Artificial Intelligence and Statistics’, PMLR, pp. 1243–1251.*

• *Su, Y., Srinath, P. & Krishnamurthy, A. (2020), Adaptive estimator selection for off-policy evaluation, in ‘International Conference on Machine Learning’, PMLR, pp. 9196–9205.*

• *Killick, Rebecca, Paul Fearnhead, and Idris A. Eckley. "Optimal detection of changepoints with a linear computational cost." Journal of the American Statistical Association 107.500 (2012): 1590-1598.*

Finally, if the underlying conditional mean function $Q$ is smooth with the same degree of smoothness over the entire action space and the optimal bandwidth is identified, we would expect the kernel-based methods could outperform our method. In addition, as we have commented, if the kernel bandwidth is fixed, then the resulting kernel method is much faster than the proposed method.  

**Kernel Method with Local Bandwidth**

This is an excellent comment. First, to the best of our knowledge, no existing local kernel-based methods were proposed in the literature for the OPE problem. Second, suppose there exists a local kernel method that can adaptively identify the optimal local bandwidth, then we expect such a method works better in settings where the conditional mean function $Q$ is a continuous function of the action, whereas the proposed method will work better when the conditional mean function $Q$ is a piecewise constant function of the action. Third, our proposed method can be understood as a special local kernel method with the boxcar kernel function. It would be practically interesting to investigate how to couple our procedure with general kernel functions. We leave it for future research. We will include these discussions shall the paper be accepted.

**Response to ‘Clarity’ Section**

We are sorry for the confusion. The change points list $\tau(\nu^*)$ is the set of optimal change-point locations within the interval $[0,\nu^*/m]$. It is defined in a recursive procedure by $\tau(\nu^*)=\{\nu^1, \tau(\nu^1)\}$ where $\nu^1$ is defined in equation (8). In other words, $\tau(\nu^*)$ consists of the best change point $\nu^1$ (*that minimizes equation (8)*) and the set of optimal change-point locations within the interval $[0,\nu^1/m]$. We will define these notations more clearly in the appendix shall our paper be accepted.

And thank you for pointing out the typo in line 301. There should be a ‘+’ between the second indicator function and the following bracket. It will be corrected.

We once again appreciate your effort in reviewing our paper. We hope that the above discussion has addressed your comments.


This paper considers off-policy evaluation with continuous treatment. The authors propose a new method using deep jump learning that adaptively discretizes the treatment space, estimates a deep model in each discretization, and combines them using a doubly robust estimator. They show that the proposed algorithm converges when the expected outcome function is piecewise or Lipschitz continuous. Synthetic / semi-synthetic experiments show DJL can sometimes have better performance than kernel-based methods.  *Originality*

Introducing DJL to off-policy evaluation looks like a new task to me. The theoretical analysis combines ideas from multi-scale change point detection and cross-fitting. 

*Quality*

- This paper tells a complete story and compares DJL with kernel methods from different aspects.The proposed algorithm is intuitive and makes sense to me. It's good to see that it at least performs well in synthetic / semi-synthetic examples.
- I have the following comments and questions:
1. I don't know why the authors focus specifically on Holder classes in Section 4. It seems to me only "a special example of function classes that's learnable by neural nets". Can we generalize the proof to "any function class that can be learned by neural nets at a certain rate"? Or are there special properties of Holder functions used in the proof?
2. Based on my knowledge, the convergence rate in Theorem 2 seems not tight, and no formal lower bound is provided for it.
3. For numerical examples, why is the proposed algorithm much faster than some of the kernel methods? It seems that solving the DP involves frequent deep learning updates. Is it because the synthetic examples have few pieces so they are easy to learn by DJL? Also, are there other scenarios (besides global Holder functions) that we may expect kernel methods to have an advantage?
4. On a high-level, how do you compare DJL with kernel methods that use local bandwidth?


*Clarity*

- This paper is generally clear. 
- I think it's a bit notation-heavy and some of the notations may not be clearly-defined. For example, I didn't see the formal definition of $\tau(v^*)$, though I was able to guess its meaning from the text. It would be good if the authors can summarize the notations in the appendix. 
- A minor typo: missing a "+" in line 301.


*Significance*

I think it's good to have an alternative option to kernel-based methods for OPE problems, and I agree that it may have better performances in certain scenarios. There's no section that explicitly discusses limitations in the paper. My concerns and suggestions are listed in the main review section.

Meta-review: The reviewers appreciated the paper and agree it provides a useful new method for off-policy evaluation supported by appropriate theory and that the paper should be accepted. The authors are expected to address the points raised by reviewers in a final version as they outlined in their response, including additional discussion on tightness of results, computation, and comparisons to other work.

Decision: accept