Optimal Treatment Regimes for Proximal Causal Learning
Authors: Tao Shen
Yifan Cui
Venue: NeurIPS 2023 poster (2023)
Keywords: Optimal treatment regimes
Policy-making
Proximal causal inference
Unmeasured confounding
Value function
TL;DR: This paper developes the sharpest individualized decision-making for recently proposed proximal causal learning

Abstract: A common concern when a policymaker draws causal inferences from and makes decisions based on observational data is that the measured covariates are insufficiently rich to account for all sources of confounding, i.e., the standard no confoundedness assumption fails to hold. The recently proposed proximal causal inference framework shows that proxy variables that abound in real-life scenarios can be leveraged to identify causal effects and therefore facilitate decision-making. Building upon this line of work, we propose a novel optimal individualized treatment regime based on so-called outcome and treatment confounding bridges. We then show that the value function of this new optimal treatment regime is superior to that of existing ones in the literature. Theoretical guarantees, including identification, superiority, excess value bound, and consistency of the estimated regime, are established. Furthermore, we demonstrate the proposed optimal regime via numerical experiments and a real data application.

Optimal Treatment Regimes for Proximal Causal Learning
Tao Shen
Yifan Cui* Zhejiang University
National University of Singapore
Abstract
A common concern when policymaker draws causal inferences from and makes decisions based on observational data is that the measured covariates are insuffi ciently rich to account for all sources of confounding, i.e., the standard no con- foundedness assumption fails hold. The recently proposed proximal causal inference framework shows that proxy variables that abound in real-life scenar- ios can be leveraged to identify causal effects and therefore facilitate decision- making, Building upon this line of work, we propose novel optimal individu- alized treatment regime based on so-called outcome and treatment confounding bridges. We then show that the value function of this optimal treatment regime is superior to that of cxisting ones in the literature. Theoretical guaran- tees, including identification, superiority, excess value bound, and consistency of the estimated regime, are established. Furthermore, we demonstrate the proposed optimal regime Via numerical experiments and real data application.
Introduction
Data-driven individualized decision-makinn has received tremendous attention nowadays due to its applications in healthcare, economics, marketing, etc. A large branch of work has focused max- imizing the expected utility of implementing the estimated optimal policy over a target popula- tion based randomized controlled trials observational studies, e.g.. Athey and Wager (2021): I); Chakraborty and Moodie (2013); Jiang et al. (2019); Kitagawa and Tetenov (2018); Kosorok and
Laber (2019); Murphy (2003); Qian and Murphy (201 1); Robins (1986 1994 1997 Tsiatis et al.
(019
Wu et al. (2019): Zhho et al. (2012
A critical assumption commonly made in these studies, known uncomfoundedness exchange- ability, precludes the existence of unmeasured confounding. Relying on an assumed ability of the decision-maker to accurately measure covariates relevant to a variety of confounding mechanisms present in a given observational study, causal effects, value functions, and other relevant quantities can be nonparametrically identified. However, such assumption might not always be realistic in observational studies or randomized trials subject to non-compliance (Robins) 1994 1997). There- fore, it is of great interest in recovering confounding mechanisms from measured covariates infer causal effects and facilitate decision-making. A prevailing strand of work has been devoted to us- ing instrumental variable (Angrist al. 1996 Imbens and Angrist
1994) a proxy variable in
dynamic treatment regimes and reinforcement learning settings (Cuil 2021 Cui and Tchetgen T'ch-
etgen
ab a Han 2023
Liao et al. 2021 Pu and Zhang 2021 Qiu et al. 2021 Stensrud and
Sarvet
2022
Recently, Tchetgen Tchetgen al. proposed the so-called proximal causal inference framework, formal potential outcome framework for proximal causal learning, which while explicitly acknowl- edging covariate measurements imperfect proxies of confounding mechanisms, establishes causal
"Correspondence Yifan Cui <cuiff@rzju..cu.ccm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

identification in settings where exchangeability on the basis of measured covariates fails. Rather than as current practice dictates, assuming that adjusting for all measured covariates, unconfoundedness can be attained, proximal causal inference essentially requires that the investigator can correctly classify a subset of measured covariates I. E L. into three types: i) variables X E X'' that may be common causes of the treatment and outcome variables; ii) treatment-inducing confounding proxies Z E Z; and ii)) outcome-inducing confounding proxies W E W.
There is a fast-growing literature on proximal causal inference since it has been proposed (Cui et al. Dukes et al., 2023: Ghassami et al. 2023] Kompa et al.) 2022] Li et al.
2023 Mastouri al. 2021 Miao et al. 2018b Shi et al. 2020b [2021] (hhpitser et al. 2023 Singh [0200] Tchetgen Tcch-
2023
etgen et 2020; Ying et al. 2023 2022 and many others). In particular, Miao et al. (2018a);
Tchetgen Tchetgen et al. (2020) propose identification of causal effects through an outcome con- founding bridge and et al. (2023) propose identification through a treatment confounding bridge. A doubly robust estimation strategy (Chernozhukov et al.] [2188] Robins et al. 1994] Rotnitzky et al.) 1998 [Scharfstein et al., 1999) further proposed in Cui et al. 2023). In addition, Ghassami et al. (202)) and Kallus et al. 2021) propose nonparametric estimation of causal effects through min- max approach. Moreover, by adopting the proximal causal inference framework, Qi et al. (2023) consider optimal individualized treatment regimes (ITRs) estimation, [Sverdrup and
con- sider learning heterogeneous treatment effects, and Bennett and Kallus (2023) consider off-policy evaluation in partially observed Markov decision processes.
estimation,Sverrrup
In this paper, we aim to estimate optimal ITRs under the framework of proximal causal inference. We start with reviewing two in-class ITRs that map from X' x W to A and X' x 2 to A, respectively, where A denotes the binary treatment space. The identification of value function and the learning strategy for these two optimal in-class ITRRs are proposed in Qi et al. (2023). In addition, Qi et al. (202)) also consider a maximum proximal learning optimal ITR that maps from X' X W x to A with the ITRs being restricted to either X' x W + A or X x Z + A. In contrast to their maximum proximal learning ITR, in this paper, we propose a brand new policy class whose ITRs map from measured covariates X' x W x 2 to A, which incorporates the predilection between these two in- class ITRs. Identification and superiority of the proposed optimal ITRs compared to existing ones are further established.
The main contributions of our work are four-fold. Firstly, by leveraging treatment and outcome confounding bridges under the recently proposed proximal causal inference framework, identifica- tion results regarding the proposed class D of ITRs that map X' x W 2 to A are established. generalization of existing ITR classes proposed in the
The proposed ITR class can be viewed literature. Secondly, an optimal subclass of DlD is further introduced. Learning optimal treat- ment regimes within this subclass leads to a superior value function. Thirdly, we propose learning approach to estimating the proposed optimal ITR. Our learning pipeline begins with the estima- tion of confounding bridges adopting the deep neural network method proposed by Kompa et al. (2022). Then we use optimal treatment regimes proposed in Qi et al. (2023) as preliminary regimes to estimate our optimal ITR. Lastly, establish an excess value bound for the value difference between the estimated treatment regime and existing ones in the literature, and the consistency of the estimated regime is also demonstrated.
2 Methodology
2.1 Optimal individualized treatment regimes
We briefly introduce some conventional notation for learning optimal ITRss. Suppose A is a binary variable representing a treatment option that takes values in the treatment space A {-1,1}, 13. Let L E L be a vector of observed covariates, and Y be the outcome of interest. Let (1) and Y (-1) 1) be the potential outcomes under an intervention that sets the treatment to values 1 and 11, respectively. Without loss of generality, we assume that larger values of Y are preferred.
Suppose the following standard causal assumptions hold: (1) Consistency: Y Y(A). That is, the observed outcome matches the potential outcome under the realized treatment. (2) Positivity: P(A L) for A almost surely, i.e., both treatments are possible to be assigned.
We consider an ITR class D containing ITRs that are measurable functions mapping from the covari- ate space L onto the treatment space A. For any d E D, the potential outcome under a hypothetical

---

intervention that assigns treatment according to d is defined as
Y(d(L)) (())((((L) (())))Y()) = 1] ++-+)(()))((( = -1,,
where 1f-} denotes the indicator function. The value function of ITR is defined as the expectation of the potential outcome, i.c.,
(dd) (d(L)))
It can be easily seen that an optimal ITR can be expressed as
d' (L) ( ((-1)(4)
d' arg max EY dED
There are many ways to identify optimal ITRs under different sets of assumptions. The most com- monly seen assumption the unconfoundedness: Y(a) A for a = =1, i.e., upon conditioning on L, there is no unmeasured confounder affecting both A and Y. Under this unconfoundedness assumption, the value function of a given regime d can be identified by (Qian and Murphy 2011)
YI[A =
(T{A=a(L)) d(L)}
vdd) = E
f(AIL)
where [(A denotes the propensity score (Rosenbaum and Rubin 1983), and the optimal ITR is identified by
YI(A= = d(L))
arg max Vdd) arg max E DED
[(A
We refer to Qian and Murphy 2011): : Zhang et al. (0012) Zhao et al. (2012) for more details of learning optimal ITRs in this unconfounded setting.
Because confounding by unmeasured factors cannot generally be ruled out with certainty in obser- vational studies or randomized experiments subject to non-compliance, skepticism about the uncon- foundedness assumption in observational studies is often warranted. To estimate optimal ITRs sub- ject to potential unmeasured confounding, Cui and I'chetgen Tchetgen (2netgen((021bb)) propose instrumental variable approaches to learning optimal ITRs. Under certain instrumental variable assumptions, the optimal ITR can be identified by
MAYIAA - d(L)) [P(A= = 1| P(A 1 --) 11)
arg
E dED
where M denotes a valid binary instrumental variable. Other works including Cuil (2021); I); Cui and Tchetgen Tchetgen (2021a); Han (2023); Puu and Zhang (2021) consider a sign or partial identifica- tion of causal effects to estimate suboptimal ITRs using instrumental variables.
2.2 Existing optimal ITRs for proximal causal inference
Another line of research in causal inference considers negative control variables as proxies to miti- gate confounding bias (Kuroki and Pearl) 2014 Miao et al., [2018a] (Shi et al. 2020a Tchetgen Tch- ccgen] 2014). Recently, formal potential outcome framework, namely proximal causal inference, has been developed by Tehetgen Tchetgen et al. (2020), which has attracted tremendous attention since proposed.
Following the proximal causal inference framework proposed in Tehetgen Tchetgen et al. 2020
suppose that the measured covariate L can be decomposed into three buckets L = (X, W. 2), where X E A' affects both A and Y. W E W denotes an outcome-induce inducing confounding proxy that is potential cause of the outcome which related with the treatment only through (U, X). and Z E 2 is treatment-inducing confounding proxy that is a potential cause of the treatment which related with the outcome Y through (U, X, A). We now summarize several basic assumptions of the proximal causal inference framework.
3

---

Assumption 1. We make the following assumptions:
(1) Consistency: Y Y (A, 2). W W Z).
(2) Positivity:
> 0, va A.
(3) Latent unconfoundedness:
(Z, A) 1 (Y(a), W) U, X, Va A.
The consistency and positivity assumptions conventional in the causal inference literature. The latent unconfoundedness essentially states that Z cannot directly affect the outcome Y, and W not directly affected by either A or Z. Figure deppicss classical setting that satisfies Assumption We refer to Shi et al. (2020b); Tchetgen Tchetgen et al. (2020) for other realistic settings for proximal causal inference.
Z
W
Figure I: A causal DAG under the proximal causal inference framework.
We first consider two in-class optimal ITRs that map from X' x to A and X to A, respectively. To identify optimal ITRs that map from X' x 2 to A. we make the following assumptions.
Assumption Completeness: For any a A, E X and square-integue
function g.
E[g(U) Z, A = o. x x] = O almost surely if and only if g(U)= = O almost surely.
Assumption 3. Existence of outcome confounding bridge: There exists an outcome confounding
bridge function h( hw, o, x)) that solves the following equation
Y A, X|
EW, A, X,
almost surely.
The completeness Assumption 2 a technical condition central to the study of sufficiency in foun- dational theory of statistical inference. It essentially assumes that Z has sufficient variability with respect to the variability of U.. We refer to tchhegn Tchetgen et al. (2020) and et al. (2022) for further diseussions regarding the completeness condition. Assumption 3 defines so-called inverse problem known as a Fredholm integral equation of the first kind through an outcome confounding bridge. The technical conditions for the existence of solution Fredholm integral equation can be found in Kress el al. 1989).
Let Dz be an ITR class that includes all measurable functions mapping from X' x Z to A. As shown in Qi et al. (2023), under Assumptions and for any de E Dz. the value function (dd.) can be nonparametrically identified by
V(d.) = B[,,W, d. 2), X)].
Furthermore, the in-class optimal treatment regime d E arg maxd,eD2 V(dz) is given by dx Z) (ign(E[h X) - 1(W, -1,, Z]}.
On the other hand, identify optimal ITRs that map from X' W A, make the following assumptions.
Assumption 4. Completeness: For any E A. E X and square- intterrln function g, E[g(U) W, A= x, x) = almost surely if and only if g(U)== = almost surely.
Assumption 5. Existence of treatment confounding bridge: There exists treatment confounding bridge function q(z, a, that solves the following equation
1
Ela(Z,a, A = X].
P(A a W. X)
almost surely.

---

Similar to Assumptions and 3 Assumption 4 assumes that W has sufficient variability relative to the variability of U. and Assumption defines another Fredholm integral equation of the first kind through a treatment confounding bridge q.
Let Dw be another ITR class that includes all measurable functions mapping from X' x W to A. As shown in Qi et al., (2023), under Assumptions I and for any du E Dw, the value function V (dw) can be nonparametrically identified by
V(dw) = E|y qZ, A, X, W) = =A}--
(2)
The in-class optimal treatment regime d% E arg maxd, V V(dw) is given by
de (x, W) =
1} Y Yg X)]{A 1]|X, W]}.
Moreover, Qi et al. (2023) consider the ITR class Dz U Dw and propose a maximum proximal
learning optimal regime based on this ITR class. For any daww E Dz U Ww under Assumptions
5 the value function V(dauw) ) for any dzww E zzz Dw can be identified by
V(dzuw) I[d E D2 ([W(W, dew (X, X)] + I[dzuw E Dw Tw)EElyn g A, (X. = A)]. (3)
The optimal ITR within this class is given by d=uw arg max EDzUDW V(dzuw), and they show that the corresponding optimal value function takes the maximum value between two optimal in-class ITRs, i.e.
V V(duww) = max V(d)).
2.3 Optimal recision-making based on two confounding bridges
As discussed in the previous section, given that neither ([Y(a) UJ nor (a) W, Z] for any E A may be identifiable under the proximal causal inference setting, one might nevertheless consider ITRRs mapping from X' x W to A. from X' x 2 to A, from X' x W x 2 to A as well as from X' A. Intuitively, policy-makers might want to use as much information as they can to facilitate their decision-makin information regarding (X, W. Z) is available.
Therefore, ITRs mapping from X' x W x Z to A are of great interest if
As a result, a natural question arises: is there an ITR mapping from X x W x Z to A which dominates existing ITRs proposed in the literature? In this section, we answer this question by proposing a novel optimal ITR and showing its superiority in terms of global welfare.
We first consider the following class of ITRs that map from X' W x Z to A.
DIw A {d_. d. W, 2) = (XX),, - (1 (1- (X)) (X, W), d2 E Dzz du E Dw, E 11), where II is the policy class containing all measurable functions X {0, 1} that indicate the individualized predilection between d₂ and d
Remark Note that Dz, Dw and Dz UDw are subsets Df Zw with a particular choice of For example, Dz is DI1 2w with restriction on π ((X)
Dw is Dill with restriction on (X) 0:
D2 U Dw is DTI Zw with restriction on x xXX 1 or (X) 0.
In the following theorem, we demonstrate that by leveraging the treatment and outcome confounding bridge functions, we can nonparametrically identify the value function over the policy class DJ i.e., (id for d E D zw-
2w
Theorem 1. Under Assumptions nonparametrically identified by
1
5
for any d D the value function VidT.) can be
v(d = ElE(()XX(VVV) x) (2
= A}]. (4)
One of the key ingredients of our constructed new policy class Diw is the choice of (--). It suggests an individualized strategy for treatment decisions between the two given treatment regimes. Because we are interested in policy learning, a suitable choice of TT-) that leads to a larger value function is more desirable. Therefore, construct the following म (X; d,, dw).
(X; dz, d I(E[ dx,, x) 2 E q Z, A XX = AHX1}. (5)
5

---

In addition, given any d, E Dz and dw E Dw. we define
d X W Z) X; d2, dew dx,
- (x,, W).
We then obtain the following result which justifies the superiority of æ.
Theorem 2. Under Assumptions for any d₂ E Dz and du E Dw-
V(d 2 max (V(d2), V(dw)).
Theorem 2 establishes that for the partieular choice of т given in (5), the value function of d no smaller than that of dz and dn for any d2 E DDz and d, Dw. Consequently, Theorem holds for dx and dy Hence, we propose the following optimal ITR d
ste
(X Z)
T(;d dih dd (x, Z) + (1 (X,, W),
and we have the following corollary. Corollary 1. Under Assumptions
5 we have that
(id 2 (ax(V V(
Corollary 1 essentially states that the value of d dominates that of d: d= as well as d'ow- Moreover, the proposition below demonstrates the optimality of d within the proposed class.
Proposition 1. Under Assumptions
we have that
d E arg
max
V(d
Therefore, d is an optimal ITR of policymakers' interest.
Statistical Learning and Optimization
3.1 Estimation of the optimal ITR d
The estimation of d consists of four steps: (i) estimation of confounding bridges h and q (ii)
estimation of preliminary ITRs d" and d= (iii) estimation of 7 (X; d and (iv) learning d
based on (ii) and (iii.. The estimation problem (i) has been developed by Cui et al. (2023): ); Miao
al. (2018b) using the generalized method of moments, (Gaasssmi et al. (2022) ; Kallus et (2021)
by min-max estimation (Dikkala et al.) 2020) using kernels, and Kompa et al. (0022) using deep learning; and (ii) has been developed by Qi et (2023). We restate estimation (i) and (ii) for completeness. With regard to (i), recall that Assumptions and 5 imply the following conditional moment restrictions
B[Y -W, =0, E - =(A a,, a WW, =0, Va E A.
respectively. Kompa et al. functions which avoids the reliance kernel methods. We adopt this approach in our simulation and details can be found in the Appendix.
2022) propose a deep neural network approach to estimating bridge
To estimate d' consider classification- based approaches according Zhang al. 2012): Zhao
ct al. (2012). Under Assumptions I minimizing the following classification error
2 and 3 maximizing the value function in is equivalent to
E((h(W,,,X) 1,
-WW ()W._
over de Dz. By choosing some measurable decision function gs Gz X Z R, we let (x,,, Z) = sign (ggn(gx(A) (X, Z)). We consider the following empirical version of (
min P, P,(( -1, -,, XX)) < 07}.
a. eg.

---

Due to the non-convexity and non moo-smoothness of the sign operator, we replace the sign operator with a smooth surrogate function and adopt the hinge loss Φ(x) max x, 0}. By adding penalty term P= P=119=1la to avoid overfitting, we solve
9= E arg min P FnIthWW, 9.E0.
(7)
tniig parameter. The estimated ITR then follows dz(X, Z) [ 2)). Similarly, under Assumptions [ and [ maximizing the value function in is equivalent to minimizing the following classification error
where P=
(,, 1}
-I), +1
over dw Dw. By the same token, the problem is transformed into minimizing the following empirical error
Fw E arg min ESw || + Pw 1l li.
(8)
The estimated ITR is obtained via (X,WW) = sign(ā.. (X, W)).
For problem (iii), given two preliminary ITRs, we construct an estimator # #(X; dz, dw.), that is, for x X.
= 1(6(1; de, dw) 2 0}.
where 8(x; dz, dw) denotes a generic estimator of 8(z; de ) A [W,(WW (X, - (,. A
where the expectation is taken with respect to everything except d, and d.y For example, the Nadaraya-Watee Watson kernel regression estimator (Nadaraya 1964) can be used, i.e., 8(a;; dz, dw) is expressed
(hn, Zi), - (Z. A, (dd W,) = A |1 X.
E K( X-lle)
where K IR + R is kernel function such Gaussian kernel, denotes the bandwidth.
|| denotes the L2-norm, and y
Finally, given dz. dm and (X; dhw,, di is estimated by the following plug-in regime,
(X, W, 2) = # #(X; (X. (:dd-#(A3 de, (x. W).
3.2 Theoretical guarantees for a
In this subsection, we first present an optimality guarantee for the estimated ITR di in terms of its value function
V(d
= =A],
where the expectation taken with respect to everything except म, d, and d...
We define an oracle optimal ITR which assumes (X;; dd.) known,
can (X, W. Z) _ (x: d. dx. Z) + #(x de, d,,, dd, (X. W).
The corresponding value function of this oracle optimal ITR is given by
v(d= d. d, Z), x) (1 - (X;d, d Y X)T W) AA],
where the expectation is taken with respect to everything except d, and d...
7

---

Then the approximation error incurred by estimating RXX; dz dw) is given by
K( - V(d V (ddww)
Moreover, define the following gain
G(T) min v V(ds), v(d))).
clear that this gain G()) by introducing always non-negative as indicated by Theorem Then we have the following excess value bound for the value of di.. compared to existing ones in the literature.
Proposition 2. Under Assumptions 75
V (( = max(V (dd)) K(+) + G(R) = V(deuw) - K(#)) + G(R).
Proposition 2 establishes link between the value function of the estimated ITR & and that of dz, du. and dzow- As shown Appendix G
(at diminishes as the sample size increases, therefore,
d has a significant improvement compared to other optimal ITRs depending on the magnitude of G(T).
Furthermore, we establish the consistency of the proposed regime based on the following assump- tion, which holds for example when d= and d. are estimated using indirect methods.
Assumption 6. For dz. dup. E[h(W, (X, Z), XI - ( (x, Z), x)xx = (n al- most surely and E[Y q(Z A, X ())(d W) (( on(n 9) almost surely.
(X, W) AX|IX]
Proposition 3. Under Assumptions
we have V(d 2n V.d
4 Numerical Experiments
The data generating mechanism for (X, A, Z, W,U) follows the setup proposed in Cui et al. (2023) and is summarized in Appendix To evaluate the performance of the proposed framework, vary by (X). (X). (x), b, and by. in E[Y X. Z, W. U to incorporate heterogeneous treatment effects including the settings considered in Qi Qi al. (0033) is compatible with the following h and g.
The adopted data generating mechanism
1+A A
1(W, A, X) = by (X) boW + (xX))
+ b W. + (X)X,,
1+ A
= exp Ato + At.Z ta
At,x
2
where in = 0..5.... f- = 0.5. l. 12. and f. (0.25, 0.25)T. We derive preliminary optimal ITRS d and d' Appendix from which we can see that X, Z, W are relevant variables for individualized decision-making
We consider six scenarios in total, and the setups of varying parameters are deferred to Appendix [ For each scenario, training datasets (Y,, A., X,, Z,, Wi are generated following the above mechanism with sample size n 1000. For each training dataset, we then apply the aforemen- tioned methods to learn the optimal ITR. particular, the preliminary ITRs d, and d,, are estimated using a linear decision rule, and (2; d d is estimated using a Gaussian kernel. More details can be found in the Appendix K
To evaluate the estimated treatment regimes, we consider the following generating mechanism for testing datasets: X ~ N(T,,
On aaPa + a. (Z. W.U ~ Nr{
o² Uew
0 O
0 2z
140 PaPa + M X
==
wry
Kg + KaPa + R
azu
Twn
8

---

where the parameter settings can be found in Appendix [1 The testing dataset is generated with a size 10000, and the empirical value function for the estimated ITR is used as a performance measure. The simulations are replicated 200 times. To validate our approach and demonstrate its superiority, we have also computed empirical values for other optimal policies, including existing optimal ITRs for proximal causal inference, as discussed in Section 2.2] along with optimal ITRs generated through causal forest (Athey and Wager 2019) and outcome weighted learning (Zhao et al. 2012).
32
S3
5.8
5.0
5.7-
4.8
5.7
4.6
5.5
ITR
5.-
5.4
Value
a
S5
S6
ag
4.35
6.10
5.60
d
4.30
5.55
6.00
5.50
5.90
ITR
Figure 2: Boxplots of the empirical value functions (dey and dowt denote estimated ITRs using causal forest and outcome weighted learning respectively).
Figure presents the empirical value functions of different optimal ITRs for the six scenarios. As expected, dz. dmw d=ww; and dit consistently outperform def and dowt. which highlights their ef- fectiveness in addressing unmeasured confounding. Meanwhile, across all scenarios, d yields superior comparable performance compared to the other estimated treatment regimes, which jus- tifies the statements made Sections 2
and 3 In addition, as can be seen in Scenario 5, all ITRs relying on the proximal causal inference framework perform similarly, which not surprising as d, (X, Z) and d (X, W) agree for most subjects. To further underscore the robust performance of our proposed approach, we include additional results with a changed sample size and a modified behavior policy Appendix
Real Data Application
In this section, we demonstrate the proposed optimal ITR via real dataset originally designed to measure the effectiveness right heart catheterization (RHC) for ill patients intensive care units (ICU), under the Study to Understand Prognoses and Preferences for Outcomes and Risks of Treat- ments (SUPPORT, Connors et al. (1996)). These data have been re-analyzed a number of papers in both causal inference and survival analysis literature with assuming unconfoundednss (Cui and Tchetgen Tchetgen, 2023 Tan 2006 2020 2019 Vermeulen and Vansteelandi 2013) or accounting
for unmeasured confounding (Cui al. 2023 Lin et al. 1998 Qi al. 2023 Tchetgen Tchetgen
et al. 2020 Ying et al. 2022).
There are 5735 subjects included in the dataset, in which 2184 were treated (with A 1) and 3551 1)). The outcome
the duration from admission to death or censor- ing. Overall, 3817 patients survived and 1918 died within 30 days. Following Tchetgen Tchetgen et al. (2020), we collect 71 covariates including demographie factors, diagnostic information, esti- mated survival probability, comorbidity, vital signs, physiological status, and functional status (see
were untreated (with

---

Hirano and Imbens (001) for additional discussion on covariates). Confounding in this study stems from the fact that ten physiological status measures obtained from blood tests conducted at the ini- tial phase of admission may be susceptible to significant measurement errors. Furthermore, besides the lab measurement errors, whether other unmeasured confounding factors exist unknown to the data analyst. Because variables measured from these tests offer only single snapshot of the under- lying physiological condition, they have the potential to act as confounding proxies. We consider total of four settings, varying the number of selected proxies from 4 to 10. Within each setting, rreatmen-innducing proxies are first selected based on their strength of association with the treatment (determined through logistic regression of A on L), and outcome-inducen inducing proxies are then chosen based on their association with the outcome (determined through linear regression of Y on and L). Excluding the selected proxy variables, other measured covariates included in X. We then estimate d=. dww, d.ww and d using the SUPPORT dataset in manner similar to that described in with the goal optimizing the patients' 30-day survival after their entrance into the ICU.
Section 4
FE
FrEET
du dgau
d
Figure 3: Graphical representation of concordance between estimated ITRs.
The estimated value functions of our proposed ITR, alongside existing ones, are summarized in Appendix M As can be seen, our proposed regime has the largest value among all settings. For visual representation of the concordance between the estimated optimal ITRs, we refer to Figure (results from Setting 1). The horizontal ordinate represents the 50 selected subjects and the vertical axis denotes the decisions made from corresponding ITRs. The purple and yellow blocks stand for being recommended treatment values of and respectively. For the subjects with purple or yellow columns, d, (X, Z)- = (X, W), which leads to the same treatment decision for the other two ITRs. For columns with mixed colors, dx(X, Z) and dan X, W) disagree. We see that in this case deum (X, W, 2) always agree with dx(X, (XX2) Z), while di (X, W, Z) take values from de (X, Z) or (X, W) depending on the individual criteria of the subjects indicated by म. In addition to the quantitative analysis, we have also conducted a qualitative assessment of the estimated regime to validate its performance. For further details, please refer Appendix M
6 Discussion
We acknowledge several limitations of our work. Firstly, the proximal causal inference framework relies on the validity of treatment- and outcome-inducing confounding proxies. When the assump- tions are violated, the proximal causal inference estimators can be biased even if unconfoundedness on the basis of measured covariates in fact holds. Therefore, one needs to carefully sort out proxies especially when domain knowledge is lacking. Secondly, while the proposed regime significantly improves upon existing methods both theoretically and numerically, it is not yet shown to be the sharpest under our considered model. It is still an open question to figure out if a more general pol- icy class could be considered. Thirdly, our established theory provides consistency and superiority of our estimated regime. It is of great interest to derive convergence rates for Propositions 2 and following Jiang (2017). In addition, it may be challenging to develop inference results for the value function of the estimated optimal treatment regimes, and further studies warranted.
Acknowledgments and Disclosure of Funding
Yifan Cui was supported by the National Natural Science Foundation of China.
10

---

References
J. D. Angrist, G. W. Imbens, and D. B. Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444 455, 1996.
S. Athey and S. Wager. Estimating treatment effects with causal forests: An application. Observa- fional Studies, 5(2):37-51, 2019.
Athey and S. Wager. Policy learning with observational data. Econometrica, 8911)133-111, 2021.
A. Bennett and N. Kallus. Proximal reinforcement learning: Efficient off-policy evaluation in par- tially observed markov decision processes. Operations Research, 09 2023.
B. Chakraborty and E. Moodie. Statistical methods for dynamic treatment regimes. Springer, 2013.
X. Chen and T. Christensen. Optimal uniform convergence rates for sieve nonparametric instrumen- tal variables regression. arXiv preprint arXiv: 13111.1111222 2013.
Y. Chen, D. Zeng, T. Xu, and Y. Wang. Representation learning for integrating multi-domain out- comes to optimize individualized treatment. Advances in Neural Information Processing Systems, 13:17976-17988 2020.
Y.-C. Chen. tutorial on kernel density estimation and recent advances. Biostatistics & Epidemiol- ogy, 111):111-87, 2017.
V. Chernozhukov. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68, 2018.
A. F. Connors, T. Speroff, N. V. Dawson, C. Thomas, F. E. Harrell, D. Wagner, N. Desbiens, L. Gold- man, A. W. Wu, R. M. Califf, al. The effectiveness of right heart catheterization in the initial care of critically iii patients. Jama, 276( 1):8898897, 1996.
Y. Cui. Individualized Decision-making under partial identification: Three perspectives, two opti- mality results, and one paradox. Harvard Data Science Review, 3(3), 2021.
Y. Cui and E. Tchetgen Tchetgen. On necessary and sufficient identification condition of optimal treatment regimes with an instrumental variable. Statistics & Probability Letters, 178:109 109180, 2021a. ISSN 0167-7152.
Y. Cui and E. Tchetgen Tchetgen. semiparametric instrumental variable approach to optimal treatment regimes under endogeneity. Journal of the American Statistical Association, 116(533): 162-173, 2021b.
Y. Cui and E. Tehetgen Tchetgen. Selective machine learning of doubly robust functionals. Biometrika, page asad055, 2023. ISSN 1464-3510.
Y. Cui, H. Pu, X. Shi, W. Miao, and F. Tchetgen Tchetgen. Semiparametric proximal causal infer- ence. Journal of the American Statistical Association, pages 1-12, 2023.
N. Dalmasso, T. Pospisil, A. B. Lee, R. Izbicki, P. E. Freeman, and A. I. Malz. Conditional density cstimation tools in python and r with applications to photometric redshifts and likelihood -free cosmological inference. Astronomy and Computing, 30:100332. 2020.
N. Dikkala, G. Lewis, L. Mackey, and V. Syrgkanis. Minimax estimation of conditional moment models. Advances in Neural Information Processing Systems, 33; 12248-12222, 2020.
L. Dinh, J. Sohl- Dickstein, and S. Bengio. Density estimation using real nvp. arXiv preprint arXi::1605.08803. 2016.
O. Dukes, I. Shpitser, and F. J. Tchetgen Tchetgen. Proximal mediation analysis. Biometrika, page asad015, 03 2023, ISSN 1464-3510.

---

N. Galie, M. M. Hoeper, M. Humbert, A. Torbicki, J.-L. Vachiery, J. A. Barbera, M. Beghetti, P. Corris, S. Gaine, J. S. Gibbs, al. Guidelines for the diagnosis and treatment of pulmonary hypertension: the task force for the diagnosis and treatment of pulmonary hypertension of the european society of cardiology (esc) and the european respiratory society (ers), endorsed by the international society of heart and lung transplantation (ishlt). European heart journal, 30(20): 2493-2537, 2009.
A. Ghassami, A. Ying. I. Shpitser, and E. Tchetgen Tchetgen. Minimax kernel machine learning for class of doubly robust functionals with application proximal causal inference. In International Conference on Artificial Intelligence and Statistics, pages 7210-7239. PMLR, 2022.
A. Ghassami, I. Shpitser, and E. T. Tchetgen. Partial identification of causal effects using proxy variables. arXiv preprint wwXiv:2304.04377. 2023.
Han. Optimal dynamic treatment regimes and partial welfare ordering. Journal of the American Statistical Association, pages 1-11, 2023.
K. Hirano and G. W. Imbens. Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes research method- ology, 2(3):259-22 278, 2001.
G. W. Imbens and D. Angrist. Identification and estimation of local average treatment effects. Econometrica, 62(2):467-475, 1994. ISSN 00129682, 14680262.
B. Jiang, R. Song. J. Li, and D. Zeng. Entropy learning for dynamic treatment regimes. Statistica Sinica, 29(4): 2019.
H. Jiang. Uniform convergence rates for kernel density estimation. In International Conference on Machine Learning, pages 1694-1703. PMLR, 2017.
N. Kallus, X. Mao, and M. Uehara. Causal inference under unmeasured confounding with negative controls: A minimax learning approach. arXiv preprint arXiv:2103.14029 2021. 2021.
T. Kitagawa and A. Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica, 86(2):591-616, 2018.
B. Kompa, D. Bellamy, T. Kolokotrones, A. Beam, et al. Deep learning methods for proximal inference via maximum moment restriction. Advances in Neural Information Processing Systems, 35:1 19:11189-11001 2022.
M. R. Kosorok and E. B. Laber. Precision medicine. Annual Review of Statistics and Its Application, 6:263-288 2203,28862222 2019.
R. Kress, V. Maz ya, and V. Kozlov. Linear integral equations, volume 82. Springer. 1989.
M. Kuroki and J. Pearl. Measurement bias and effect restoration in causal inference. Biometrika, 101(2):423-455 437, 2014.
K. Li, X. Shi, W. Miao, and E. Tchetgen Tchetgen. Double negative control inference test- negative design studies of vaccine effectiveness. Journal of the American Statistical Association, pages 1-12, 2023.
L. Liao, Z. Fu, Z. Yang, Y. Wang, M. Kolar, and Z. Wang. Instrumental variable value iteration for causal offline reinforcement learning. arXiv preprint arXiv:2102.09907. 2021.
D. Y. Lin, B. M. Psaty, and R. A. Kronmal. Assessing the sensitivity of regression results to unmea- sured confounders in observational studies. Biometrics, pages 948-96888943 1998.
A. Mastouri, Y. Zhu, L. Gultchin, A. Korba, Silva, M. Kusner, A. Gretton, and K. Muandet. Prox- imal causal learning with kernels: Two-stage estimation and moment restriction. In International Conference on Machine Learning, pages 7512-7523, PMLR, 2021.
W. Miao, Z. Geng, and J. Tehetgen Tchetgen. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika, 105(4):987-993, 201 8a.
12

---

W. Miao. X. Shi, and E. Tchetgen Tchetgen. A confounding bridge approach for double negative control inference on causal effects, arXiv preprint arXiv: 04945, 2018b.
W. Miao, W. Hu, E. L. Ogburn, and X.-H. Zhou. Identifying effects of multiple treatments in the presence of unmeasured confounding. Journal of the American Statistical Association, pages 1-15, 2022.
Murphy, Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65(2):331-355 2003.
E. A. Nadaraya. On estimating regression. Theory of Probability & Its Applications, 91):141-142, 1964.
H. Pu and B. Zhang. Estimating optimal treatment rules with an instrumental variable: A partial identification learning approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), (3(2):318-345 2021.
Z. Qi, R. Miao, and X. Zhang. Proximal learning for individualized treatment regimes under un- measured confounding. Journal of the American Statistical Association, pages 1-14, 2023.
M. Qian and S. A. Murphy. Performance guarantees for individualized treatment rules. Annals of Statistics, 39(2):1 1800, 2011.
H. Qiu, M. Carone, Sadikova, M. Petukhova, R. C. Kessler, and A. Luedtke. Optimal individ- ualized decision rules using instrumental variable methods. Journal of the American Statistical Association, 116633):1744-1999) 2021.
A. Raghu, M. Komorowski, L. Celi, P. Szolovits, and M. Ghassemi. Continuous state-space mod- els for optimal sepsis treatment: deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147-163. PMLR, 2017.
Robins. new approach to causal inference mortality studies with a sustained exposure pe- riod applicatio to control of the healthy worker survivor effect. Mathematical modelling, 7 (9-12):1393-1512, 1986.
Robins. Correcting for non-compliance in randomized trials using structural nested mean models. Communications in Statistics: Theory and Methods, 23(8):2379-244 2412, 1994.
J. M. Robins. Causal inference from complex longitudinal data. In Latent variable modeling and applications to causality, pages 69-117. Springer, 1997.
M. Robins, A. Rotnitzky, and L. Zhao. Estimation of regression coefficients when some regres- sors are not always observed. Journal of the American statistical Association, 894427):446866. 1994.
R. Rosenbaum and B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55) 1983.
A. Rotnitzky, J. M. Robins, and D. O. Scharfstein. Semiparametric regression for repeated outcomes with nonignorable nonresponse. Journal of the American Statistical Association, 93(444):1321- 1339, 1998.
D. O. Scharfstein, A. Rotnitzky, and J. M. Robins. Adjusting for nonignorable drop-out using semiparametric nonresponse models. Journal of the American Statistical Association, 94(448): 1096-1120, 1999.
D. W. Scott. Multivariate density estimation: theory, practice and visualization. John Wiley & Sons, 2015.
X. Shi, W. Miao, J. C. Nelson, and E. J. Tehetgen Tchetgen. Multiply robust causal inference with double-negative control adjustment for categorical unmeasured confounding. Journal of the Royal Statistical Society: Series (Statistical Methodology), 82(2):521-540, 2020a.
X. Shi, W. Miao, and E. Tehetgen Tchetgen. selective review of negative control methods in epidemiology. Current Epidemiology Reports, 1(4):190-202, 2020b.
13

---

X. Shi, W. Miao, M. Hu, and E. Tchetgen Tchetgen. Theory for identification and inference with synthetic controls: a proximal causal inference framework. arXiv preprint arXiv:2108. 13935, 2021.
1. Shpitser, Z. Wood-Doughty, and E. J. T. Tchetgen. The proximal id algorithm. Journal of Machine Learning Research, 23:1-46, 2023.
R. Singh. Kernel methods for unobserved confounding: Negative controls, proxies, and instruments. arXiv preprint arXiv:2012./0315, 2020.
K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems, 28, 2015.
M. J. Stensrud and A. L. Sarvet. Optimal regimes for algorithm-assiste human decision decision-making. arXiv preprint ar 00w:2203.03020, 2022.
E. Sverdrup and Y. Cui. Proximal causal learning of heterogeneous treatment effects. Interna- lional Conference on Machine Learning. 2023.
Z. Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101(4766;1619--1377 2006.
Z. Tan. Regularized calibrated estimation of propensity scores with model misspecification and high-dimensional data. Biometrika, 107(1):137-158, 12 2019. ISSN 0006-3444.
Z. Tan. Model-assisted inference for treatment effects using regularized calibrated estimation with high-dimensional data. The Annals of Statistics, 48(2):811--- 1-837, 2020.
E. Tchetgen Tchetgen. The control outcome calibration approach for causal inference with unob- served confounding. American journal of epidemiology, 179(5):633-640 2014.
E. J. Tchetgen Tchetgen, A. Ying. Cui, X. Shi, and W. Miao. An introduction to proximal causal learning. arXiv preprint arXiv:2009. 10982, 2020.
A. A. Tsiatis, M. Davidian, S. T. Holloway, and E. B. Laber. Dynamic treatment regimes: Statistical methods for precision medicine. Chapman and Hall/CRC, 2019.
K. Vermeulen and S. Vansteelandt. Bias-reduced doubly robust estimation. Journal of the American Statistical Association, 110(511):1044-15 2015.
Wang, Z. Qi, and C. Shi. Blessing from experts: Super reinforcement learning in confounded environments. arXiv preprint arXiv:2209. 15448, 2022.
P. Wu, D. Zeng, and Y. Wang. Matched learning for optimizing individualized treatment strategies using electronic health records. Journal of the American Statistical Association, 2019.
A. Ying, Y. Cui, and J. Tchetgen. Proximal causal inference for marginal counterfactual survival curves. arXiv preprint arXiv:2204.13144 2022.
A. Ying, W. Miao, X. Shi, and E. T. Tchetgen. Proximal causal inference for complex longitudinal studies. Journal of the Royal Statistical Society Series B: Statistical Methodology. 2023. In press.
J. Yoon, J. Jordon, and M. Van Der Schaar. Ganite: Estimation of individualized treatment effects using generative adversarial nets. International Conference Learning Representations, 2018.
B. Zhang, A. A. Tsiatis, E. B. Laber, and M. Davidian. A robust method for estimating optimal treatment regimes. Biometrics, 6814):001-- 2012.
Y. Zhao, D. Zeng, A. J. Rush, and M. R. Kosorok. Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association, 107(499): 1118, 2012.
Y.QQ. Zhao, E. B. Laber, Y. Ning. S. Saha, and B. E. Sands. Efficient augmentation and relaxation learning for individualized treatment rules using observational data, Journal of Machine Learning Research, 10(1):1821-1844 2019.
14


Reviews:
[{"review_id": "ivhvmSTRLG", "reviewer": "Reviewer_VJQR", "summary": "Most estimation methods for individualized treatment rules (ITRs) assume no unmeasured confounders for valid causal inference. However, such an assumption can be unreasonable, such as when estimating ITRs from observational data. Previous work has applied proximal causal inference to estimate ITRs when this assumption is violated, but is restricted to policy classes that either exclude treatment-inducing confounding proxies or exclude outcome-inducing confounding proxies [1]. To this end, the authors propose estimating a stochastic mixture of both policy classes from [1] to yield a more flexible ITR. Theoretical and simulation results demonstrate the superiority of the proposed method. ", "questions": "Questions that I feel the paper should address and general weaknesses of the paper can be found in the Weaknesses section. As to my suggestions, it may not be feasible to address weaknesses (1) and (4) prior to the camera-ready. However, I would suggest the authors at least acknowledge weakness (1) as a limitation, and discuss how to implement their method in a more data-adaptive manner, even if they don't change the implementation for the empirical experiments. Such a discussion should include how to tune hyperparameters for all functional classes, including estimators of $d_z$, $d_w$, $\\delta$, $h$ and $g$.  As to weaknesses (2), (3) and (5), I think it would be feasible to mostly address these weaknesses prior to the camera-ready, and my suggestions on how to address them can be found in the Weaknesses section. ", "limitations": "I did not notice the authors acknowledge any limitations of the proposed work. I would recommend adding a \"Discussion\" section that summarizes the results of the work, and addresses limitations by discussing important avenues for future work. In addition to what was discussed in the Weaknesses and Questions sections, another important limitation is that the proposed method assumes that assumptions (1)-(5) hold and that both $h$ and $g$ are correctly specified, while $d_z$ and $d_w$ appear to only require some of these assumptions to hold. \n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The assumption of no unmeasured confounders is nearly ubiquitous across ITR estimation methods, despite frequent violations when dealing with observational data. This makes the problem the authors are trying to solve - estimating ITRs when this assumption is violated - very significant. Moreover, the theoretical and simulation results are of sufficient quality to convince me that the method outperforms [1], the existing state-of-the-art in proximal learning.\n", "weaknesses": "While I believe the merits and potential contribution of the paper outweigh its limitations, the theoretical and empirical results of the paper are weaker than that of previous work, and the clarity of the paper can be improved. I go into more detail below:\n\n1. **Theoretical guarantees are much weaker than those of previous methods.** Convergence rates, finite-sample error bounds or asymptotic normality is often derived for ITR methods [2-5], including the method this work seeks to improve over [1]. However, while the authors prove that the proposed method will converge to a policy with better value than that of [1] asymptotically, they do not derive a rate of convergence or establish any finite-sample error bounds. Moreover, the asymptotic analysis from Appendix G assumes convergence of several estimators in $L\\_\\infty$ space. This is a much stronger assumption than the assumptions made in previous work, which only assumes convergence of estimators in $L\\_2$ space (e.g. see Assumption 12 from [1] or Condition B5 from [3]).\n\n2. **The real data analysis does not strengthen the validity of the proposed method.** When applying ITR estimators to real datasets, it is common practice to assess performance by using either (1) an estimator of the value [2-6] or (2) arguments based on domain knowledge that support the validity of the proposed method [1,6]. For example, in [1], the authors argue that the estimated policy is accurate by demonstrating that the estimated coefficients and interpretation of the policy is consistent with findings from previous literature. In contrast, the only conclusions that these authors draw from their real data analysis is that the proposed estimator gives different results than previously proposed methods. Such a conclusion says little about the validity or superiority of the proposed method. One way this analysis could be greatly improved is to look at the patients from Figure 3 where the recommended treatment differs between the proposed method and that of [1], and use domain knowledge or previous literature to argue that the recommendations given by the proposed method is more sound. Alternatively, the authors could make this conclusion by comparing the coefficients of the proposed policy with that of [1]. \n\n3. **Empirical comparisons were relatively limited.** The authors only benchmark the proposed method against those of a single previous work, [1]. To conclude that the proposed method achieves state-of-the-art performance, the authors should benchmark the proposed method against additional baselines as well. For example, there are many methods that assume no unmeasured confounders which the authors could evaluate to demonstrate the utility of using a proximal causal inference framework (e.g. [7,8]). There are also other methods that relax the no unmeasured confounders assumption or have shown robustness when such assumptions are violated, such as instrument variable (IV) methods [3,9] and M-learning [4]. How are the assumptions made by proximal learning less restrictive than those made by IV approaches, and can such methods be applied to the simulated datasets? If so, the authors should benchmark the proposed method against these method. And if these methods are not applicable, the authors should explain why in the paper. In addition to the number of competing baselines being limited, the simulated datasets from this work all have the same sample size and behavior policy. When deriving new ITR methods, it is common practice to evaluate the method on datasets of different sample sizes (and if observational data is of interest, varying behavior policies) so as to demonstrate robust performance [2-5,7-11]. \n\n4. **The implementation uses very simple estimators.** In theory, $d_z,d_w$ and $\\delta$ could be estimated by any weighted classification and regression algorithm. However, in their empirical experiments, the authors only explore estimating $d_z$ and $d_w$ with linear policies and estimating $\\delta$ with a Nadaraya-Watson estimator where the bandwidth is chosen based on a heuristic. Moreover, while $h$ and $g$ were estimated using neural networks, the architecture and number of training iterations was fixed a priori. This contrasts to previous works which use more cutting-edge machine learning approaches to estimate the ITR, such as kernel machines, random forests or neural networks, and adopts hyperparameters to the data at-hand using cross-validation [8]. Such works are especially prevalent in top-tier ML conferences [10,11], and better demonstrate broad applicability and flexibility of the proposed method. \n\n5. **Many parts of the paper need to be better written to avoid confusion and address some open questions.** For example, for the real data analysis, it is not clear what assumptions proximal learning is making and how it is useful for the analyzed dataset. It is mentioned that patients were arranged in a \"control group\" on line 289. Were patients randomized to receive a treatment? If so, wouldn't the no unmeasured confounders assumption hold, as treatment assignment is not being affected by any unmeasured covariates? Also, what is the logic behind the choice of $Z$ and $W$ on line 296 (e.g. why are the variables in $Z$ expected to affect treatment but not outcome), and what specifically are the unmeasured confounders $U$ that we are trying to adjust for? Finally, it is stated that the outcome is censored. Does this mean that 30-day survival rate is censored for some of the patients? It is well-known that optimizing censored outcomes without adjusting for the censoring mechanism can yield bias [5]. \n\n>> Here are some other suggestions to reduce points of confusion and improve readability:  \na. The explanation of how the proposed method improves upon [1] in the Introduction section (lines 56-65) is confusing. For example, it is stated on line 69-61 that [7] maps an ITR with domain $\\mathcal X\\times\\mathcal W\\times\\mathcal Z$ with the domain being restricted to $\\mathcal X\\times\\mathcal W$ to $\\mathcal X\\times\\mathcal Z$. While this makes more sense after reading section 2.2, these statements initially appear contradictory. Also on lines 59 and 63 \"two optimal in-class ITRs\" should be changed to \"these two optimal in-class ITRs\" to clarify that the authors are referring to the classes mentioned on line 57.  \nb. The paper has many typos. For example, \"Tchetgen Tchetgen et al\" in line 34 should include the year and a link to the reference, \"netwrok\" on line 73 should read \"network\", and on line 169 the authors should add $V(d_{z\\cup w})$ to the argmax.  \nc. On line 142-143 the authors state that Assumption 3 assumes \"Z has sufficient variability with respect to the variability of U\". But isn't assumption 3 actually assuming that U has sufficient variability with respect to the variability of Z?  \nd. On line 173 it is stated that $\\mathbb E[Y(a)|X,U]$ may not be identifiable under proximal causal inference. But doesn't assumptions 1-5 allow for such identifiability?  \ne. Remark 1 is not true. For example if $\\pi(X)=0.5$ then $\\pi$ is constant but the policy class will not be in $\\mathcal D_{\\mathcal Z}\\cup \\mathcal D_{\\mathcal W}$.   We actually need the restriction that $\\pi$ is both constant and in the set $\\\\{0,1\\\\}$.  \nf. The results of Appendices E and G should appear in the main paper as propositions, theorems or corollaries.  \ng. For sections where over 10 references are cited back-to-back, I think readability would be improved if these citations appeared in chronological order.  \nh. The authors should add a Discussion section that summarizes the results of the paper and proposes important avenues for future work (also see my comments on the Limitations section).\n\n\n\n\nReferences:\n\n1. Qi Z, Miao R and Zhang X. Proximal Learning for Individualized Treatment Regimes Under Unmeasured Confounding. JASA. 2023.\n\n2. Zhao Y, Zeng D, Rush AJ and Kosorok MR. Estimating Individualized Treatment Rules Using Outcome Weighted Learning. JASA 107 (499): 1106-1118. 2012. \n\n3. Qiu H et al. Optimal Individualized Decision Rules Using Instrumental Variable Methods. JASA 116 (533): 174-191. 2021.\n\n4. Wu P, Zeng D and Wang Y. Matched Learning for Optimizing Individualized Treatment Strategies Using Electronic Health Records. JASA 115 (529): 380-392. 2020.\n\n5. Zhao YQ, Zeng D, Laber EB, Song R, Yuan M and Kosorok MR. Doubly robust learning for estimating individualized treatment with censored data. Biometrika 102 (1): 151-168. 2015. \n\n6. Raghu et. al. Continuous State-Space Models for Optimal Sepsis Treatment: a Deep Reinforcement Learning Approach. MLHC 2017.\n\n7. Zhao YQ, Laber EB, Ning Y, Saha S and Sands BE. Efficient Augmentation and Relaxation Learning for Individualized Treatment Rules using Observational Data. JMLR 20: 1-23. 2019.\n\n8. Zhou X, Mayer-Hamblett N, Khan U and Kosorok MR. Residual Weighted Learning for Estimating Individualized Treatment Rules. JASA 112 (517): 169-187. 2017.\n\n9. Pu H and Zhang B. Estimating optimal treatment rules with an instrumental variable: A partial identification learning approach. JRSS-B 83 (2): 318-345. 2021. \n\n10. Yoon J, Jordon J and van der Shaar M. GANITE: Estimation of Individualized Treatment Effects using Generative Adversarial Nets. ICLR 2018. \n\n11. Chen Y, Zeng D, Xu T and Wang Y. Representation Learning for Integrating Multi-domain Outcomes to Optimize Individualized Treatment. NeurIPS 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive suggestions from the reviewer. \n\nW1: Theoretical guarantees are much weaker than those of previous methods. \n\nR: We express our gratitude for your invaluable suggestions regarding the exploration of convergence rates and finite sample error bounds associated with our proposed estimated ITR. In a revised version, we will enhance the theoretical foundation of our work by including an additional excess value bound (consistency) for the estimated regime (this is already presented in the global rebuttal). And we acknowledge that it is possible to derive convergence rates for the excess value bounds following Jiang (2017, ICML) and our newly proposed Proposition 2 in the global rebuttal. We will include this topic in the discussion section.\n\nMoreover, regarding the assumption we propose in Appendix G, the convergence of estimators in $L_{\\infty}$  space is applied as our intention is to guarantee that for each $X$, the conditional expected values based on $\\hat{h}$ and $\\hat{q}$ are consistent, which is not required by previous literature. Therefore, we employ this assumption to support our subsequent analysis.\n\nW2:  The real data analysis does not strengthen the validity of the proposed method.\n\nR: We sincerely value the constructive feedback provided by the reviewer. As a response, we have incorporated a table in the global rebuttal, detailing the estimated value functions of our proposed ITR, alongside existing approaches. This table further demonstrates the robustness and validity of our method.\n\nW3: Empirical comparisons were relatively limited.\n\nR: We will revise the numerical experiments following your suggestions by including additional baseline methods, varied sample sizes, and altered behavior policies. Due to the space/time limit, we have not presented all these results in the global rebuttal and we will include them in a revised version. As a preview of our progress, we have included a figure in the global rebuttal, showcasing empirical values of different ITRs with a broader spectrum of baseline methods, including ITRs obtained from causal forest (Wager and Athey, 2018 JASA) and outcome weighted learning (Zhao et al., 2012 JASA), as well as existing ITRs mentioned under proximal causal inference framework. \n\nFurthermore, with regard to the IV methods such as those proposed by Cui and Tchetgen Tchetgen (2021, JASA), Pu and Zhang (2021, JRSSB), and Qiu (2021, JASA), it's important to note that these methods operate under the assumption of binary instrumental variables (IVs), while, in our simulation and real data studies, the treatment-inducing confounding proxies $Z$ are not confined to binary values. \n\nW4: The implementation uses very simple estimators.\n\nR: We will include a discussion regarding more machine learning methods for estimating $d_z^*, d_w^*$, and $\\delta$ in the supplemental material, as well as the corresponding hyperparameter tuning and cross-validation procedure.\n\nW5: Many parts of the paper need to be better written to avoid confusion and address some open questions. \n\nR: Thank you. We will revise the text carefully to further improve the readability.\n\nFirstly, sorry for the confusion and “control” is a typo. The study is an observational study, and we meant by untreated group.\n\nSecondly, the source of confounding in this study arises from the fact that variables obtained from blood tests may be susceptible to significant measurement errors. Furthermore, besides the lab measurement errors, whether there exists other unmeasured confounding is unbeknownst to the data analyst. Since variables measured from the test offer a singular snapshot of the underlying physiological condition, they hold the potential to act as confounding proxies. Among the ten measures capturing physiological status, a subset of four (pafi1, paco21, ph1, hema1) is selected as they demonstrate a strong connection with both the treatment and the outcome. Then an approach proposed by Tchetgen Tchetgen et al. (2020) ranking proxies according to their strength of association in treatment and outcome models is leveraged to choose the sets of proxies $Z$ and $W$ respectively. \n\nLastly, as for the censoring problem you mentioned, the censoring in this data is administrative censoring, i.e., everyone is censored at 30 and no one is censored before 30. The outcome of our interest is the survival time up to 30 days, i.e., restricted mean survival time. No censoring bias occurred in this data analysis. \n\t\t\t\t\t\t\t\nMore discussion about the real data application, including assumption verification and sensitivity analysis can be found in Tchetgen Tchetgen et al. (2020), Cui et al. (2023, JASA), Sverdrup and Cui (2023, ICML) and Qi et al. (2023, JASA). \n\nFor the additional suggestions put forward in the gray box, we give our responses one by one below. \n\na. We will revise the text following your suggestion to make it clearer. \n\nb. Thank you. We will check the writing carefully to avoid typos.\n\nc. No. The completeness condition relating the range of $U$ to that of $Z$ states that the set of proxies $Z$ must have sufficient variability relative to the variability of $U$. For more discussion, one can refer to Canay et al. (2013, Econometrika).\n\nd. No. $E[Y(a)|X,U]$ is unidentifiable as we cannot access $U$. \n\ne. Remark 1 is correct as the range of the function $\\pi(X)$ is $\\{0,1\\}$ (defined on line 183). We will revise the text to highlight it.\t\n\nf. Thank you. We will convert Appendices E and G to formal propositions.\t\n\ng. We will follow your suggestion to revise the ordering of the citations.\n\nh. We will add a section to discuss limitations and future work (Please find the paragraph in the global rebuttal).\n\nWe extend our sincere gratitude for furnishing the reference list, and we will expand our literature review by including the sources you have kindly provided."}, {"review_id": "otb2eCIrxN", "reviewer": "Reviewer_NTDa", "summary": "The paper discusses the optimization of treatment rules in the context of observational data and under assumptions of proximal inference. Various theorems are introduced, and a real data analysis performed using a healthcare example.", "questions": "Is there quantitative evidence that \"proximal causes abound in real life scenarios\"? The matter would seem dependent on substance area, or knowing this would seem to require access to nature's true DAG.", "limitations": "I see no ethnical limitations here.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Below is a list of perceived weaknesses. \n\nThe paper is overall sound and the topic of importance. I appreciate the presence of the real data application. Assumptions and results clearly stated.", "weaknesses": "Below is a list of perceived weaknesses. \n\nIt was not clear to me how the empirical results compare to competing methodological baselines from other approaches (I don't believe the different values presented in the figure represent different algorithmic approaches). \n\nThe paper is quite heavy on notation and, at least to me, light on intuitive explanation for findings as they are discussed, limiting insight into the inner workings of why the method works.\n\nI don't know the proximal causal inference literature well so am not well-positioned to discuss the contribution in that subfield of causal inference. \n\nI don't see a discussion of uncertainty estimation in the theoretical or empirical results. Uncertainty estimation in optimized treatment effect regimes can be difficult (e.g., the bootstrap may not be appropriate or may have poor coverage) but may be important to usefulness in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for a careful reading of our work.\n\nW1: It was not clear to me how the empirical results compare to competing methodological baselines from other approaches (I don't believe the different values presented in the figure represent different algorithmic approaches).\n\nR: In Figure 2, we have presented the empirical values regarding different ITRs, including our proposed ITR which results in sharper values, and three ITRs proposed by Qi et al. (2023, JASA) that are baseline methods relying on proximal causal inference framework. For other methods which do not rely on the proximal causal inference framework, we have included a supplementary figure in the global rebuttal, showcasing results derived from causal forest (Wager and Athey, 2018 JASA) and outcome weighted learning (Zhao et al., 2012 JASA).  And we will add these results in a revised version. We also refer to Qi et al. (2023, JASA) for more baseline methods that do not rely on proximal causal inference.\n\nW2: The paper is quite heavy on notation and, at least to me, light on intuitive explanation for findings as they are discussed, limiting insight into the inner workings of why the method works.\n\nR: Sorry for the difficulty in reading caused by the notation. To facilitate your understanding, we summarize the main idea as follows:  we aim at providing the best decision-making strategy for the proximal causal inference framework (Tchetgen Tchetgen et al. 2020; Miao et al. 2018 Biometrika; Cui et al. 2023 JASA), and we focus on comparison with the recently developed methods proposed by Qi et al. (2023, JASA). Technically, on a high level, for each covariate strata $X$, we propose to choose the best decision between the one induced from the outcome confounding bridge $h$ and the one induced from the treatment confounding bridge $q$. We theoretically prove the consistency and superiority of our estimated regime.\n\nW3: I don't see a discussion of uncertainty estimation in the theoretical or empirical results. Uncertainty estimation in optimized treatment effect regimes can be difficult (e.g., the bootstrap may not be appropriate or may have poor coverage) but may be important to usefulness in practice.\n\nR: We will add the following discussion.\n\n“It may be overly challenging to develop inference results for the value function of the estimated optimal treatment regimes and further studies are warranted. To the best of our knowledge, some recent papers look at the jackknife in applied settings under unconfoundedness (there is some theory in these papers and supplements, though the theory is incomplete):\n \n- X. Jiang, A.E. Nelson, R.J. Cleveland, D.P. Beavers, T.A. Schwartz, L. Arbeeva, C. Alvarez. L.F. Callahan, S. Messier, R. Loeser, M.R. Kosorok (2021). A precision medicine approach to develop optimal exercise and weight loss treatment for overweight and obese adults with knee osteoarthritis. -Arthritis Care & Research 73:693-701. \n\n- G. Honvah, H. Cho, M.R. Kosorok (2022). Model selection for survival individualized treatment rules using the jackknife estimator. BMC Medical Research Methodology 22:328.”\n\nQ: Is there quantitative evidence that \"proximal causes abound in real life scenarios\"? The matter would seem dependent on substance area, or knowing this would seem to require access to nature's true DAG.\n\nR: We agree with the reviewer’s point. The choice of proxies relies on domain knowledge. For example, proxies abound in longitudinal studies as the future can not cause the past, and see Tchetgen Tchetgen et al. (2020) for many other examples. We acknowledge that it is important to correctly identify proxies and the framework when applying our method. Therefore, we add the following limitations of our method in the discussion section. \n\n“We acknowledge several limitations of our work. Firstly, the proximal causal inference framework relies on the validity of treatment- and outcome-inducing confounding proxies. When the assumptions are violated, the proximal causal inference estimators can be biased even if unconfoundedness on the basis of measured covariates in fact holds. Therefore, one needs to carefully sort out proxies especially when domain knowledge is lacking. Secondly, while the proposed regime significantly improves upon existing methods both theoretically and numerically, it is not yet shown to be the sharpest under our considered model. It is still an open question to figure out if a more general policy class could be considered. Thirdly, our established theory provides consistency and superiority of our estimated regime. It is of great interest to derive convergence rates for Proposition 1 and Proposition 2 following Jiang (2017, ICML).”"}, {"review_id": "fEiPmrIFdq", "reviewer": "Reviewer_193J", "summary": "The goal is to learn an optimal individual treatment rule (ITR) where the data suffer from unobserved confounding but where the researcher has a treatment proxy and an outcome proxy. \n\nWhile the general problem has been studied before by Qi et al (JASA 2023), this paper’s contribution is to broaden the class of ITRs. For a broader class of ITRs, the authors identify the value function and show that it exceeds the value function of the narrower class.\n", "questions": "Is it possible to derive rates for K(pi_hat) and G(pi_bar) in Proposition 1? Or can the authors at least pose this question for future work and give citations of where similar results are derived?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "My comments are brief because this is a strong paper.\n\nOriginality: The essence of the improvement is that, for different covariate values x, one may either use the “outcome” ITR or the “treatment” ITR. This departs from previous work, where either the “outcome” ITR or the “treatment” ITR is used across covariate values.\n\nQuality: The proofs look correct, and the results are easy to interpret. Rates for the objects in Proposition 1 would be an improvement; see the question below.\n\nClarity: The paper is well written and well referenced.\n\nSignificance: The paper contributes to two popular literatures: proxies and ITRs. While its theoretical contribution is modest, it does appear to have practical relevance.", "weaknesses": "The theoretical contribution is somewhat incremental.\n\nTake a pass to fix typos, e.g. “netwrok” on line 73.\n\nAn extra sentence in Remark 1 would be welcome, that explains the point summarized as “originality” above.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the constructive suggestions from the reviewer. \n\nW1: The theoretical contribution is somewhat incremental.\n\nR: We appreciate the reviewer's feedback and their assessment of our theoretical contribution. We acknowledge the importance of continually advancing the field, and we are trying our best to address the concern of incrementality.\n\nTheorem 2 and Corollary 1 stand as significant achievements in our work, introducing a policy that yields a notably sharper value function within the proximal causal inference framework. The implications of Proposition 1 are also noteworthy, highlighting an excess value bound that signifies the superiority of our optimal ITR when compared to other existing ITRs. To further strengthen our theoretical contributions, we will add an excess value bound (consistency) for the estimated regime in a revised version (this is already presented in the global rebuttal). This supplement will reinforce our theoretical framework.\n\nW2: Take a pass to fix typos, e.g. “netwrok” on line 73.\n\nR: Thank you for pointing out the typo. We will thoroughly review our writing to ensure that typos are minimized.\n\nW3: An extra sentence in Remark 1 would be welcome, that explains the point summarized as “originality” above.\n\nR: Thank you. We will revise the remark following your suggestion.\n\nQ: Is it possible to derive rates for $\\mathbb{K}(\\hat{\\pi})$ and $\\mathbb{G}(\\bar{\\pi})$ in Proposition 1? Or can the authors at least pose this question for future work and give citations of where similar results are derived?\n\nR: Your question is appreciated.  It is possible to derive convergence rates for these two components. For $\\mathbb{K}(\\hat{\\pi})$, we have discussed its asymptotic properties in Appendix G, and to the best of our knowledge, there is some theory in Jiang (2017, ICML) that might be applied when one tries to establish the convergence rate. For $\\mathbb{G}(\\bar{\\pi})$, the convergence rate can be inferred according to our newly proposed Proposition 2 in the global rebuttal and standard treatment regime literature. We will include these topics as an extension of our method in the discussion section."}, {"review_id": "EbLK72y4wv", "reviewer": "Reviewer_of5M", "summary": "The authors present a new optimal individual treatment regime (ITR) within the proximal causal inference framework, which avoids the strong assumption of no unmeasured confounding. Instead, one assumes the effect of the unmeasured confounders flows exclusively through proxy variables, as defined through outcome-inducing and treatment-inducing confounding bridges. Compared to prior work, this optimal ITR that is defined with respect to a more flexible function class that depends on known confounders X, treatment-inducing confounding proxies Z, and outcome-inducing confounding proxies W.", "questions": "1. How does one do statistical inference for the value of the estimated ITR?", "limitations": "The authors have not discussed limitations of the work.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed ITR is a natural extension of existing ITRs by using a function \\pi(x) that selectively chooses between two existing ITRs based on known confounders x. Under the proximal causal inference frameowrk, the proposed ITR is proven to be superior to existing ITRs in the literature. (Existing ITRs from Qi 2023 can be viewed as special cases of the proposed ITR.) The authors introduce a simple plugin estimator for the proposed ITR and show that the value of the resulting estimator is determined by approximation error of \\pi and the gain from using \\pi. Simulation studies show that the proposed ITR is either superior or comparable to existing ITRs. The manuscript is clearly written. The authors provide a nice review of prior work in this area and clearly describe how their work builds on existing work.", "weaknesses": "1. The proposed extension of the ITR function class appears quite incremental. The value of the proposed ITR follows directly from application of the tower rule. The paper would be greatly strengthened if the authors can show that this is the best one can do, e.g. showing that the value of a more complex ITR function class would be unidentifiable without much stronger assumptions.\n\n2. In the simulation studies, the improvement in mean value when using the proposed optimal ITR over existing ITRs is large only in scenario 2. In all other scenarios, the improvement is small. Can the authors explain the behavior in this simulation study? Also, can the authors explain settings in which the proposed ITR is expected to substantially improve over existing ITRs? My guess is that the gain is biggest when (i) there are large differences between expected value at each X for the ITR with domain (X,W) and the ITR with domain (X,Z) and (ii) the optimal pi function has high variance (e.g. pi(X) is equal to 1 half of the time). Does this correspond to scenario 2?\n\n3. The authors perform a real-data analysis in Section 5, which illustrates how the proposed ITR is different from existing ITRs. However, the authors do not calculate the values of the estimated ITR, so readers cannot compare the performance of the proposed ITR against existing ITRs. Do the authors have estimates of the values of the estimated ITRs?\n\n4. The number of treatment-inducing confounding proxies and outcome-inducing confounding proxies were small in both the simulation studies and real-world data analysis. However, the practical appeal of the proximal causal inference framework is its use of proxies for unmeasured confounders, which would suggest the use of many variables as potential proxies. Can the authors include simulations that reflect more realistic settings where more proxy variables are used? How does the proposed method perform as the number of these proxies increases?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive suggestions from the reviewer. \n\nW1: The proposed extension of the ITR function class appears quite incremental. The value of the proposed ITR follows directly from application of the tower rule. The paper would be greatly strengthened if the authors can show that this is the best one can do, e.g. showing that the value of a more complex ITR function class would be unidentifiable without much stronger assumptions.\n\nR: Thank you. We have pointed out that $E[Y(a)|X,W,Z]$ is not identifiable under the current assumptions and framework (Qi et al. 2023 JASA), and our proposed class is much more general than the classes considered in previous literature. However, at this juncture, whether a more general policy class could be considered remains an open question. We add this important point in the discussion section.\n\nW2: In the simulation studies, the improvement in mean value when using the proposed optimal ITR over existing ITRs is large only in scenario 2. In all other scenarios, the improvement is small. Can the authors explain the behavior in this simulation study? Also, can the authors explain settings in which the proposed ITR is expected to substantially improve over existing ITRs? My guess is that the gain is biggest when (i) there are large differences between expected value at each X for the ITR with domain (X,W) and the ITR with domain (X,Z) and (ii) the optimal pi function has high variance (e.g. pi(X) is equal to 1 half of the time). Does this correspond to scenario 2?\n\n\nR: We agree with your insightful guessing. As we indicate in Proposition 1, the value of the proposed estimator has a significant improvement compared to other optimal ITRs depending on the magnitude of $G(\\bar{\\pi})$, which is related to disparities between expected values at each $X$ and the variance of $\\bar{\\pi}$. A pertinent illustration can be drawn from our simulation results; for instance, in scenarios 1, 2, 4 & 6, the variance of $\\hat{\\pi}$ is large. On the other hand, scenarios 3 & 5 portray relatively limited variation in the two estimated expected values for each $X$.\n\nW3: The authors perform a real-data analysis in Section 5, which illustrates how the proposed ITR is different from existing ITRs. However, the authors do not calculate the values of the estimated ITR, so readers cannot compare the performance of the proposed ITR against existing ITRs. Do the authors have estimates of the values of the estimated ITRs?\n\nR: Thank you. We have added a table in the global rebuttal which describes the estimated value function of our proposed ITR and the existing ones. This table further demonstrates the robustness and validity of our method.\n\nW4: The number of treatment-inducing confounding proxies and outcome-inducing confounding proxies were small in both the simulation studies and real-world data analysis. However, the practical appeal of the proximal causal inference framework is its use of proxies for unmeasured confounders, which would suggest the use of many variables as potential proxies. Can the authors include simulations that reflect more realistic settings where more proxy variables are used? How does the proposed method perform as the number of these proxies increases?\n\nR: Following your suggestion, we have revised the real data experiment to include settings with more proxies in the global rebuttal. As we can see, our estimated regime has the largest values throughout and significantly improves upon other regimes including $d_z$ and $d_w$, which is consistent with what we have observed in simulation studies. For the pattern across rows, we note that the optimal values are not expected to be increasing as the number of proxies increases even if they are all compatible with Assumptions 1-5.\n\nQ: How does one do statistical inference for the value of the estimated ITR?\n\nR: Thank you for bringing up this question. We will add the following discussion.\n\n“It may be overly challenging to develop inference results for the value function of the estimated optimal treatment regimes and further studies are warranted. To the best of our knowledge, some recent papers look at the jackknife in applied settings under unconfoundedness (there is some theory in these papers and supplements, though the theory is incomplete): \n\n- X. Jiang, A.E. Nelson, R.J. Cleveland, D.P. Beavers, T.A. Schwartz, L. Arbeeva, C. Alvarez. L.F. Callahan, S. Messier, R. Loeser, M.R. Kosorok (2021). A precision medicine approach to develop optimal exercise and weight loss treatment for overweight and obese adults with knee osteoarthritis. Arthritis Care & Research 73:693-701. \n\n- G. Honvah, H. Cho, M.R. Kosorok (2022). Model selection for survival individualized treatment rules using the jackknife estimator. BMC Medical Research Methodology 22:328.”\n\nL: The authors have not discussed limitations of the work.\n\nR: We will discuss the limitations explicitly in the discussion section. The limitations include the following three aspects.\n\n“We acknowledge several limitations of our work. Firstly, the proximal causal inference framework relies on the validity of treatment- and outcome-inducing confounding proxies. When the assumptions are violated, the proximal causal inference estimators can be biased even if unconfoundedness on the basis of measured covariates in fact holds. Therefore, one needs to carefully sort out proxies especially when domain knowledge is lacking. Secondly, while the proposed regime significantly improves upon existing methods both theoretically and numerically, it is not yet shown to be the sharpest under our considered model. It is still an open question to figure out if a more general policy class could be considered. Thirdly, our established theory provides consistency and superiority of our estimated regime. It is of great interest to derive convergence rates for Proposition 1 and Proposition 2 following Jiang (2017, ICML).”"}]

Author rebuttal:
We extend our heartfelt gratitude for all the comments provided by the reviewers on our work. Within this global rebuttal, we address recurring concerns raised by multiple reviewers. Our response is organized into three key parts: refining the theoretical analysis, presenting updated experimental results, and introducing a discussion section on limitations.

Part I:
To bolster our theoretical contribution, we establish the consistency of the proposed regime based on the following Assumption 6. Assumption 6 holds for example when $\hat d_z$ and $\hat d_w$ are estimated using indirect methods.

Assumption 6. Suppose that $|E[h(W,\hat{d}_z(X,Z),X)|X]-E[h(W,d_z^*(X,Z),X)|X]|=o_p(n^{-\xi})$ almost surely and $ |E[Yq(Z,A,X)\mathbb{I}\\{\hat{d}_w(X,W) = A\\}|X]-E[Yq(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\}|X]|=o_p(n^{-\varphi})$ almost surely.

Proposition 2. Under Assumptions 1-6, we have $V(\hat{d}_{zw}^{\hat{\pi}})$ converges to 

$V(d_{zw}^{\bar{\pi}*})$ in probability.

Proof:

We start with defining four subsets of $\mathcal{X}$, i.e.,

$\mathcal{X}_{g1} = \\{x \in \mathcal{X}: \bar{\pi}(x,\hat{d}_z,\hat{d}_w) = 1, \bar{\pi}(x,d_z^*,d_w^*) = 0\\},$  

$\mathcal{X}_{g2} = \\{x \in \mathcal{X}: \bar{\pi}(x,\hat{d}_z,\hat{d}_w) = 0, \bar{\pi}(x,d_z^*,d_w^*) = 1\\},$

$\mathcal{X}_{gc1} = \\{x \in \mathcal{X}: \bar{\pi}(x,\hat{d}_z,\hat{d}_w) = \bar{\pi}(x,d_z^*,d_w^*) = 0\\},$ 

$\mathcal{X}_{gc2} = \\{x \in \mathcal{X}: \bar{\pi}(x,\hat{d}_z,\hat{d}_w) = \bar{\pi}(x,d_z^*,d_w^*) = 1\\}.$

Then we have

$V(d_{zw}^{\bar{\pi}*}) - V(\hat{d}_{zw}^{\bar{\pi}}) = $

$\int_{x \in \mathcal{X}} \mathbb{E}[\bar{\pi}(X;d_z^*,d_w^*) h(W, d_z^*(X,Z),X) + (1- \bar{\pi}(X;d_z^*,d_w^*)) Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\}$

$- \bar{\pi}(X;\hat{d}_z,\hat{d}_w) h(W, \hat{d}_z(X,Z),X) - (1- \bar{\pi}(X;\hat{d}_z,\hat{d}_w)) Y q(Z,A,X)\mathbb{I}\\{\hat{d}_w(X,W) = A\\}|X = x] f(x)dx$

$=\int_{x \in \mathcal{X}_{g1}} E[Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\} - h(W, \hat{d}_z(X,Z),X)|X=x] f(x)dx $

$+ \int_{x \in \mathcal{X}_{g2}} E[h(W, d_z^*(X,Z),X) - Y q(Z,A,X)\mathbb{I}\\{\hat{d}_w(X,W) = A\\}|X = x] f(x)dx $

 $+ \int_{x \in \mathcal{X}_{gc1}}  E[Y q(Z,A,X)\mathbb{I}\{d_w^*(X,W) = A\} - Y q(Z,A,X)\mathbb{I}\\{\hat{d}_w(X,W) = A\\}|X = x]f(x)dx $
 
$+ \int_{x \in \mathcal{X}_{gc2}}  E[h(W, d_z^*(X,Z),X) - h(W, \hat{d}_z(X,Z),X)|X = x]f(x)dx.$

Then it is easy to see that 
$\int_{x \in \mathcal{X}_{gc1}}  E[Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\} - Y q(Z,A,X)\mathbb{I}\\{\hat{d}_w(X,W) = A\\}|X = x]f(x)dx$ 

and $\int_{x \in \mathcal{X}_{gc2}}  E[h(W, d_z^*(X,Z),X) - h(W, \hat{d}_z(X,Z),X)|X = x]f(x)dx$ converge to 0 in probability according to Assumption 6. 

Therefore, we then essentially need to bound 
$\int_{x \in \mathcal{X}_{g1}} E[Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\} - h(W, \hat{d}_z(X,Z),X)|X=x] f(x)dx$.

We further split $\mathcal{X}_{g1}$ to 

$\mathcal{X}_{g1,1} = $

$\\{x \in \mathcal{X}_{g1}: E[Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\} - h(W, \hat{d}_z(X,Z),X)|X=x] \in (-C n^{-\zeta},C n^{-\zeta})\\}$

and $\mathcal{X}_{g1,2}=$

$\\{x \in \mathcal{X}_{g1}: E[Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\} - h(W, \hat{d}_z(X,Z),X)|X=x] \notin (-C n^{-\zeta},C n^{-\zeta})\\}$ 

where $\zeta = \min\\{\xi, \varphi\\}$. Then we can find

$\int_{x \in \mathcal{X}_{g1,1}} E[Y q(Z,A,X)\mathbb{I}\\{d_w^*(X,W) = A\\} - h(W, \hat{d}_z(X,Z),X)|X=x] f(x)dx$ 

is bounded by $O(n^{-\zeta})$ and $\mathbb{P}(\mathcal{X}_{g1,2})$ 

converges to 0 in probability based on Assumption 6 and the definition of $\mathcal{X}_{g1}$. 

A similar proof can also be conducted to obtain 

$\int_{x \in \mathcal{X}_{g2}} E[h(W, d_z^*(X,Z),X) - Y q(Z,A,X)\mathbb{I}\\{\hat{d}_w(X,W) = A\\}|X = x] f(x)$ is small enough.

We then have that $V(\hat{d}_{zw}^{\bar{\pi}})$ converges to 

$V(d_{zw}^{\bar{\pi}*})$ in probability. 

As we have proved that  

$\mathbb{K}(\hat{\pi}) = o(1)$ almost surely in Appendix G, 

we finally conclude that $V(\hat{d}_{zw}^{\hat{\pi}})$ converges to 

$V(d_{zw}^{\bar{\pi}*})$ in probability.

Part II:
In response to the insightful suggestions provided by the reviewers regarding our numerical experiments and data application, we add a figure and a table in the uploaded PDF file to further demonstrate the robustness and validity of our method.

In detail, for the simulation study, Figure 1 presents empirical values of different ITRs with a broader spectrum of baseline methods, adding causal forest and outcome weighted learning. As we can see, the proposed method consistently achieves the largest value.

For the real data analysis, Table 1 describes the estimated value functions of our proposed ITR, alongside existing approaches, and we also include settings with increasing numbers of proxies. As we can see from the table, the proposed regime has the largest value among all settings. 

Part III:
We will add a discussion section to address limitations, and discuss potential directions for future work:

“We acknowledge several limitations of our work. Firstly, the proximal causal inference framework relies on the validity of treatment- and outcome-inducing confounding proxies. When the assumptions are violated, the proximal causal inference estimators can be biased even if unconfoundedness on the basis of measured covariates in fact holds. Therefore, one needs to carefully sort out proxies especially when domain knowledge is lacking. Secondly, while the proposed regime significantly improves upon existing methods both theoretically and numerically, it is not yet shown to be the sharpest under our considered model. It is still an open question to figure out if a more general policy class could be considered. Thirdly, our established theory provides consistency and superiority of our estimated regime. It is of great interest to derive convergence rates for Proposition 1 and Proposition 2 following Jiang (2017, ICML).”

Reference:
H. Jiang. Uniform convergence rates for kernel density estimation. ICML, 2017.

Meta-review: This is a technically solid paper with a tangible contribution to the causal literature, tying together in a useful and non-trivial way recent results from proximal causal inference and from the literature learning individual treatment rules. Reviewers found the both the theoretical analysis and experiments to be of overall good quality, with some issues resolved during the discussion period.
Decision: Accept (poster)