Quantile-Optimal Policy Learning under Unmeasured Confounding

Authors: Zhongren Chen, Siyu Chen, Zhengling Qi, Zhuoran Yang, Xiaohong Chen
Primary area: reinforcement learning
Keywords: Quantile Treatment Effect, Causal Inference, Offline Contextual Bandit

Abstract:
We study quantile-optimal policy learning where the goal is to find a policy whose reward distribution has the largest $\alpha$-th quantile for some $\alpha \in (0, 1)$. We focus on the offline setting whose generating process involves unobserved confounders. Such a problem suffers from three main challenges: (i) nonlinearity of the quantile objective as a functional of the reward distribution,  (ii) unobserved confounding issue, and  (iii) insufficient coverage of the offline dataset. To address these challenges, we propose a suite of causal-assisted policy learning methods that provably enjoy strong theoretical guarantees under mild conditions. In particular, to address (i) and (ii), using causal inference tools such as instrumental variables and negative controls, we propose to estimate the quantile objectives by solving nonlinear functional integral equations. Then we adopt a minimax estimation approach with nonparametric models to solve these integral equations, and propose to construct conservative policy estimates that address (iii). The final policy is the one that maximizes these pessimistic estimates. In addition, we propose a novel regularized policy learning method that is more amenable to computation. Finally, we prove that the policies learned by these methods are $\tilde{O}(n^{-1/2})$ quantile-optimal under a mild coverage assumption on the offline dataset. To our best knowledge, we propose the first sample-efficient policy learning algorithms for estimating the quantile-optimal policy when there exists unmeasured confounding.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730230153609

thread_type:
official_review

<thread_object>
creator: reviewer_ump4

<content>
summary:
This work examines quantile-optimal policy learning, focusing on the whole distribution of rewards rather than just the mean. The authors consider a complex setting, addressing challenges such as the nonlinearity of the quantile objective, issues with unobserved confounding, and limited coverage in the offline dataset. They propose using causal inference tools, including instrumental variables (IVs), and adopt a minimax estimation approach with nonparametric models. Computationally, they provide two algorithmic solutions: the solution set algorithm and a regularized version, with the latter offering computational advantages. Theoretically, they analyze the regret rates of the policies derived from both algorithms. Lastly, they conduct a straightforward simulation to assess the performance of the regularized algorithm.

strengths:
1. This work considers reward distribution and unobserved confounding, a challenging yet highly practical setting, especially as unobserved confounding is prevalent in real applications.
2. The algorithms leverage the Pessimism Principle within minimax estimation to address issues arising from insufficient sample size. Furthermore, the authors introduce a regularized version to handle computational difficulty.
3. Theoretical results are substantial. The authors discuss both IV and NC scenarios in detail and provide in-depth regret analyses for the policies obtained from the two algorithms. The theoretical derivation is comprehensive.

weaknesses:
1. The motivation for studying quantiles in policy learning is not clear. The authors use an example of income in a job training program to illustrate the relevance of analyzing reward distribution, which has merit but remains insufficient. In particular, the related works section lacks any mention of quantile policy learning literature. Although work in this area is limited, introducing relevant studies on quantile treatment regimes and quantile treatment effects would help clarify the significance of studying quantiles in policy learning.
2. The experiments lack persuasiveness. 
- The authors should compare quantile policy learning with standard policy learning. For instance, using job training program example to highlight the benefits of studying the median rather than the mean would strongly emphasize the advantages of the proposed method, yet this is regrettably absent. 
- The authors only evaluate the performance of the regularized algorithm through simulations without comparing it to the solution set algorithm, making it difficult to illustrate the computational advantage of the regularized version. While the solution set algorithm may be computationally intractable, its performance could be evaluated in a simplified experiment. 
- The simulated DGP is overly simple, with one-dimensional contexts and IVs, and only a single quantile (0.2) selected, which is insufficiently compelling.
3. Readability is lacking. The theoretical analysis in Section 4 is overly detailed, occupying excessive space. The authors could condense Section 4, such as simplifying the paragraph following Condition 4.4, and instead elaborate on experimental results in Section 5 to improve readability. Additionally, symbols and indices are overly complex, and further simplification is advisable.
4. Numerous typos and errors exist in the writing. Here are a few examples:
- Line 061: "we can'' should be "which can''
- Line 066: "our methods works" should be "our method works''
- Line 087: "he'' should be "the''
- Line 112: The notation $O\in\mathcal O$ appears inconsistently, as in Line 361, $\mathcal O (\eta_n)$
- Line 139: "$\pi$'' should be "$\hat{\pi}$"
- Section 3.4 uses ":='', whereas Section 3.5 uses "=''
- Line 294: "the challenge two'' should be "challenge (2)''
- Line 312: Consider changing "function class $\mathbb{F}$'' to "$\mathcal{F}$'' for notational consistency
- Different notations for order: $\mathcal O$ and $\tilde{\mathcal O}$
- Certain theorems lack periods, such as Corollary 4.6
- Line 446: "$l_2$ norm'' should be "$\ell_2$ norm''
- Line 506: The values of $p$ are 20, 50, 70, while in Appendix I (Line 1699), they are 30, 50, 80.

The authors should take a thorough proofreading of the manuscript.

5. Theoretical results in Section 4 rely on the strong Condition 4.4. While the authors mention in Appendix E that Condition 4.4 is satisfied when $\eta_n = \mathcal O(n^{-1/2})$ and both $\mathcal{H}$ and $\Theta$ are linear spaces. However, they do not discuss Condition 4.4 under more complex cases, nor do they consider whether Condition 4.4 can hold without the assumption of $\mathcal{H}$ and $\Theta$ as linear spaces.

questions:
Could the authors provide a more detailed motivation for studying quantiles, like in which situations quantile policy learning offers a clear advantage over standard policy learning (mean)? Additionally, the authors could supplement the experiment part to highlight the advantages of QPL over standard policy learning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576598764

thread_type:
official_review

<thread_object>
creator: reviewer_uxa7

<content>
summary:
This paper investigates offline learning with partial observability under a quantile-based objective. The proposed algorithm uses minimax optimization and incorporates the principle of pessimism. Under various assumptions (including identifiability, compatibility of function approximation, coverage of the dataset, etc.), convergence rate of the proposed algorithm is provided.

strengths:
It is interesting to see how the tricks from the theory of offline RL also apply in this setting.

weaknesses:
**Readability of this paper.** The paper is difficult to read.

1. The notations are horrible. Letter W is both NCO variable and a function of (D,h). $\mathcal{O}$ stands for both observation space and big-O notation.

2. The bounds hide most dependency on the parameters, which makes it difficult how assumptions influence the results. For example, the dependency hidden in $o_p(\cdot)$ is never properly specified.

3. The assumptions are stated messily and lack sufficient explanation. For example: (1) It is not clear to me why the identifiability condition of Assumption 4.1 is reasonable, as E[W(D,h)|X,Z] takes expectation over A and can be under-determined. (2) I understand \eta_n corresponds to the complexity of the function class Q, but Condition 4.4 is stated in a unclear way. (3) Assumption 4.10 seems to assume certain coverage condition of the dataset, and should be made clearer (e.g. provide a simpler and sufficient condition for 4.10, possibly in terms of the concentrability). (4) Assumption 4.9 is also difficult to interpret.

4. It is not clear why the parameter eps_{\lambda_n} is introduced.

**Contribution.** It is hard to accurately evaluate the contribution of this paper given its readability  However, it seems to me that most tricks of this paper have already appeared in previous work. It could be beneficial to provide a more detailed comparison to previous works, e.g., Lu et al. 2022, and maybe also the paper of https://proceedings.mlr.press/v162/guo22a/guo22a.pdf.

questions:
See the discussion of weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730945033220

thread_type:
official_review

<thread_object>
creator: reviewer_k74w

<content>
summary:
The paper studies offline reinforcement learning with three main differentiators: 

1. the objective (cumulative rewards) is measured in $\alpha$-quantile, rather than expectation.

2. observations have latent confounders.

3. how to define sufficient coverage in this setting. 

The proposed method runs in two steps: (1) perform causal inference to get a nonparameteric model and (2) apply pessimism principle. Then for the practical use, a policy optimization method is proposed. It is claimed that this is the first sample-efficient policy learning algorithm in this setting.

strengths:
- This new setting sounds interesting, and authors did a good job of motivating the setting relatively well (though also see weakness on this point). 

- The writing quality is in general quite good, except the technical part (also see weakness on this point).

weaknesses:
- The authors motivated the unmeasured confounders with a healthcare example, and quantile optimization with job training programs. However, these are distinct cases, and I wonder whether there is one *unified* motivating example that requires both.

-  Section 4, while highly technical, is the most important part of the paper as the main contribution of the paper claims novel methodologies. Such section must be written with extra care on the clarity even though it conveys complex ideas. As it stands now, it is almost impossible to decipher this section. Here are a few main points that call confusions:

1. How strong is Assumption 4.1? What would this imply if you bring it to the tabular setting for instance?

2. Theorem 4.5 -- again, if you could explain this in the tabular setting, that might help a lot. 

3. Presentation became overly complicated by combining the quantile-objective formulation and nonparametric approaches, which I began to doubt why such complication was necessary in this work, and what does it have to do with novelty of the proposed method. 

4. I think not many readers, even the experts in RL, can go through the subsections here. 

 


- I think the experimental section could have been reinforced in the main text. The experimental setup is not clear: what are the "p% of random noises"? Does it mean a random exploration?       
 
- It is also not very clear what the plots are saying: small regret if sample size grows large? Why is that surprising?

questions:
- It is confusing whether we have two datasets, one with intervention and the other without intervention. If Z is an instrumental variable, does it appear in the dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731253734160

thread_type:
official_review

<thread_object>
creator: reviewer_6knv

<content>
summary:
This paper studies the problem of identifying the policy whose reward distribution with the largest $\alpha$-quantile. The authors are particularly interested in the setting when there is unmeasured confounding i.e. not all variables that affect the choice of the actions in the offline datasets are recorded in the dataset. The general setting of unmeasured confounding is difficult and often impossible and the authors focus on two cases --  (a) instrumental variables, and (b) negative controls. The proposed approach effectively combines conditional moment restrictions (popular in econometrics) with pessimism based policy optimization (popular in offline reinforcement learning). The authors provide theoretical guarantees and statistical rate of the proposed method under reasonable assumptions on the data generating process.

strengths:
1. I think the authors study an important problem as the issue of unmeasured confounding / partial observability is quite common in offline reinforcement learning. The proposed method also combines causal inference techniques with pessimism based policy optimization methods from offline reinforcement learning.

2. The theoretical results are interesting and provide statistical rates under the two settings -- (a) instrumental variables, and (b) negative controls. However, the results are proven under strong assumptions (see comments in the next section).

weaknesses:
1. Since the paper deals with causal inference with unmeasured confounding, it should include experiments on real-world datasets. Handling real-world datasets introduces many challenges e.g. discrete outcomes, missing covariates etc. The current experimental setup considers a simulation based setting that considers linear functionals, one-dimensional context, and is too simple to demonstrate the effectiveness of the proposed method.

2. The authors make several strong assumptions in order to prove theoretical results. First, it is assumed that the parameter $\eta_n = O(n^{-1/2})$ but this is only true for simple classes of functions (as shown in appendix E) and is not true in general. Second, assumption 4.7 assumes that the conditional density is continuous and it eliminates all causal inference problems with discrete outcomes. Third, I believe assumption 4.9 is quite strong and there is no justification as to why it should be true.

3. Overall, I found the paper hard to follow. Many steps require further explanation and the assumptions require further justification.

questions:
1. Can you please justify why assumption 4.9 should be true?

2. Solving offline reinforcement learning requires some type of coverage condition on the data e.g. single-policy concentrability. In the paper, I haven't seen any explicit mention of such coverage conditions. Can you please clarify what you assume regarding the overlap between the optimal policy and the data-generating policy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732508524138

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response I

comment:
We sincerely thank all four reviewers for their time and effort in carefully reading our paper and providing valuable feedback. In this global response, we aim to address common concerns raised by the reviewers and clarify key aspects of our work. Subsequently, we address each reviewer's specific concerns in detail and outline the corresponding revisions, all of which have been incorporated into the updated manuscript.

Several reviewers noted that the assumptions presented in Section 4 require further justification. In response, we added a separate section in the appendix to provide a detailed discussion of each assumption. We also clarify each of the assumptions here.

**A. Assumption 4.1 (Identifiability and Realizability).** This assumption, often referred to as a global identification assumption, is commonly made in nearly all nonparametric quantile IV literature. For instance, see Assumption 3.2(ii) in [Chen and Pouzo (2012)](#ref-chen2012), Assumption 1 in [Gagliardini and Scaillet (2012)](#ref-gagliardini2012), and Assumption 1 in [Horowitz and Lee (2007)](#ref-horowitz2007).

Regarding sufficient conditions for this assumption to hold, [Chernozhukov and Hansen (2005)](#ref-chernozhukov2005) provides such conditions in the case where the context $X$ is discrete. For more general setups, [Chen et al. (2014)](#ref-chen2014) discusses conditions for local identification in detail. The only study we are aware of that carefully outlines conditions for global identification is [Wong (2022)](#ref-wong2022). However, Wong's approach relies on a set of strong regularity conditions to establish these results.

**B. Assumption 4.7 (Regularity of Conditional Density).** This paper assumes that the reward variable $Y$ is continuous. The primary reason is that we are dealing with a **nonlinear** functional $\mathcal{T} \mathbf{h}$. To link the regret to $\mathcal{T} \mathbf{h}$, we conduct a local expansion of $\mathcal{T} \mathbf{h}$ around the neighborhood of the structural quantile function $\mathbf{h}^*_\alpha$. For this expansion, it is essential that the density of $Y$ is continuous and bounded, ensuring that the pathwise derivative $\mathcal{T}\mathbf{h}$ at $\mathbf{h} = \mathbf{h}^*_\alpha$ exists. We remark that this assumption is needed in most of the nonparametric quantile IV literature. For instance, see Condition 6.1(i), (iv) in [Chen and Pouzo (2012)](#ref-chen2012), Theorem 3.1 in [Abadie et al. (2002)](#ref-abadie2002), Condition A.1 in [Chernozhukov and Hansen (2005)](#ref-chernozhukov2005), Assumption A.1 in [Gagliardini and Scaillet (2012)](#ref-gagliardini2012). While our paper does not address discrete outcomes due to this constraint, we believe a similar algorithm could be applied in this case by first convolving the discrete outcome with Gaussian noise, which would smooth the distribution of $Y$. This smoothing allows for the use of techniques designed for continuous outcomes.

**References**
- [Chen and Pouzo (2012)](#ref-chen2012): Xiaohong Chen and Demian Pouzo. *Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals.* Econometrica, 80(1):277–321, 2012.
- [Abadie et al. (2002)](#ref-abadie2002): Alberto Abadie, Joshua Angrist, and Guido Imbens. *Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings.* Econometrica, 70(1):91–117, 2002.
- [Chernozhukov and Hansen (2005)](#ref-chernozhukov2005): Victor Chernozhukov and Christian Hansen. *An IV model of quantile treatment effects.* Econometrica, 73(1):245–261, 2005.
- [Chen et al. (2014)](#ref-chen2014): Xiaohong Chen, Victor Chernozhukov, Sokbae Lee, and Whitney K Newey. *Local identification of nonparametric and semiparametric models.* Econometrica, 82(2):785–809, 2014.
- [Gagliardini and Scaillet (2012)](#ref-gagliardini2012): Patrick Gagliardini and Olivier Scaillet. *Nonparametric instrumental variable estimation of structural quantile effects.* Econometrica, 80(4):1533–1562, 2012.
- [Horowitz and Lee (2007)](#ref-horowitz2007): Joel L Horowitz and Sokbae Lee. *Nonparametric instrumental variables estimation of a quantile regression model.* Econometrica, 75(4):1191–1208, 2007.
- [Van der Vaart and Wellner (2013)](#ref-van2013): A. van der Vaart and J. Wellner. *Weak Convergence and Empirical Processes: With Applications to Statistics.* Springer Series in Statistics. Springer New York, 2013.
- [Wong (2022)](#ref-wong2022): Wing Hung Wong. *An equation for the identification of average causal effect in nonlinear models.* Statistica Sinica, 32:539–545, 2022.
- [Miao et al. (2023)](#ref-miao2023): Rui Miao, Zhengling Qi, Cong Shi, and Lin Lin. *Personalized pricing with invalid instrumental variables: Identification, estimation, and policy learning.* arXiv preprint arXiv:2302.12670, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732508849120

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response II

comment:
**C. Condition 4.4:** We only verify that Condition 4.4 holds true for linear function classes because they already encompass a wide variety of functions, including polynomial splines, B-splines, wavelets, and Fourier series basis functions. These function classes can be represented as linear combinations of basis functions, making them instances of the linear function class. By examining our proof in Appendix E, we rely only on the property that linear function classes have a covering number and bracketing number bounded by $A_n \log \left(\frac{C_3}{t}\right)$ where $A_n$ is the dimension of the linear function classes. Therefore, we can actually obtain different convergence rates \(\eta_n\) under the supremum norm for the chosen function classes. The bracketing and covering numbers of various function classes can be found in different books. For example, [Van der Vaart and Wellner (2013)](#ref-van2013) provides an extensive discussion of covering numbers and bracketing numbers. They discuss these concepts for a variety of function classes, including Sobolev spaces, Besov spaces, Hölder classes, and reproducing kernel Hilbert spaces.

**D. Assumption 4.9 (Local Curvature for the Estimator of the Solution Set Algorithm).** In the initial version of our submission, this assumption was divided into two components. Upon reexamining the proof, we made slight modifications that allowed us to eliminate the need for Assumption 4.9(i). As a result, only Assumption 4.9(ii) is required, which asserts that the weaker pseudo-metric $\mathbf{h} - \mathbf{h}^*_\alpha$ is Lipschitz continuous with respect to the population criterion function, the $l_2$ norm of $\mathcal{T} \mathbf{h}$  within a small neighborhood around $\mathbf{h}^*_\alpha.$ This condition is relatively mild and is commonly employed in the nonlinear functional analysis literature [Chen and Pouzo (2012)](#ref-chen2012), [Miao et al. (2023)](#ref-miao2023). Condition 6.3(i) in [Chen and Pouzo (2012)](#ref-chen2012) provides a sufficient condition for the assumption to hold in a broad class of functional spaces $\mathcal{H}$, including the linear function spaces we consider in Appendix E.

**E. Assumption 4.10 (Change of Measure).** The Change of Measure Assumption 4.10 is actually a weaker substitute for the standard concentrability assumption commonly used in the RL literature. In the tabular setting, Assumption 4.10 is satisfied when the right-hand side of the assumption is uniformly bounded, and the Moore-Penrose inverse of the probability mass matrix $P(Z \mid A, X)^+$ exists. This condition holds if the instrumental variable (IV) satisfies the standard completeness assumption: $\text{rank}[P(Z \mid A, X)] \geq |\mathcal{A}| \times |\mathcal{X}|.$ We note that Theorem 4.11 and Theorem 4.13 remain valid if Assumption 4.10 is replaced with the usual single-policy concentrability assumption: there exists a constant $\tilde{c} > 0$ such that $\sup_{x \in \mathcal{X}, a \in \mathcal{A}} \frac{\tilde{p}(x)\pi^*(a \mid x)}{p(x, a)} \leq \tilde{c}$. Note $\frac{\tilde{p}(x)\pi^*(a \mid x)}{p(x, a)}$ represents the density ratio of the distribution under the interventional process with the oracle policy $\pi^*$ over that of the offline data collection process.



**References**
- [Chen and Pouzo (2012)](#ref-chen2012): Xiaohong Chen and Demian Pouzo. *Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals.* Econometrica, 80(1):277–321, 2012.
- [Van der Vaart and Wellner (2013)](#ref-van2013): A. van der Vaart and J. Wellner. *Weak Convergence and Empirical Processes: With Applications to Statistics.* Springer Series in Statistics. Springer New York, 2013.
- [Miao et al. (2023)](#ref-miao2023): Rui Miao, Zhengling Qi, Cong Shi, and Lin Lin. *Personalized pricing with invalid instrumental variables: Identification, estimation, and policy learning.* arXiv preprint arXiv:2302.12670, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732509173619

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response III

comment:
**1)** Several authors have argued that our experimental setting is naive and lacks a real-world data application. The primary reason we do not include a real data study is that our method relies on causal intervention based on the given context $X$ during the interventional process. Specifically, conducting a real data analysis would require access to the counterfactual potential outcomes $Y_i(a)$ for each of the $i$-th trial for some random action $a \in \mathcal{A},$ which depends on the realization of the randomized policy $\hat{\pi}(X_i)$. However, a fundamental challenge in causal inference is that we cannot observe all potential outcomes; for each trial, only one outcome is observable. As a result, no real dataset exists that allows us to evaluate the performance of our method. Second, we do not include comprehensive experiments as our work is primarily theoretical, focusing on a provably sample-efficient quantile-optimal policy learning algorithm. We introduce the first unified framework that integrates quantile causal inference with offline reinforcement learning.

However, we agree with the reviewers that experiments would strengthen the value of our results. In the camera-ready version, we would apply our method to a semi-synthetic dataset derived from the Job Search Intervention Study [Jo and Stuart (2009)](#ref-jo2009use)
 which investigates the quantile effect of a job training program on earnings. This dataset includes an eight-dimensional context for each participant. We will compare our algorithm against the standard policy learning algorithm that aims to maximize the average effect.

**2)** Several reviewers expressed concerns about the readability of the theoretical analysis in Section 4, noting that it is overly detailed and questioning whether such complexity is necessary. We apologize for the dense content filled with technical details. The analysis of the nonparametric quantile treatment effect is inherently technical, as it involves linking the regret to the nonlinear criterion function, $l_2$ norm of $\mathcal{T}\mathbf{h}$ through a local expansion around the unknown structural quantile function $\mathbf{h}^*_\alpha.$ We aimed to rigorously establish the statistical fast rate of regret within the ten-page limit of ICLR.

To improve readability, we addressed all notational issues that may cause confusion in the manuscript. Additionally, we moved many technical components to the appendix, including the detailed discussion of Condition 4.4, the treatment of pathwise derivatives, and the theoretical results. In their place, we will shift the explanation of the experimental setting and the real data study to the main sections in the camera-ready version, making the paper more accessible to readers.

**References**
- [Jo and Stuart (2009)](#ref-jo2009use): Booil Jo and Elizabeth A. Stuart. *On the use of propensity scores in principal causal effect estimation.* Statistics in Medicine, 28(23):2857–2875, 2009.
</content>
</thread_object>

</thread>


Metareview: