SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The estimation of  $d_{zw}^{\overline{\pi}*}$  consists of four steps: (i) estimation of confounding bridges  $h$  and  $q$ ; (ii) estimation of preliminary ITRs  $d_{z}^{*}$  and  $d_{w}^{*}$ ; (iii) estimation of  $\bar{\pi}(X; d_{z}^{*}, d_{w}^{*})$ ; and (iv) learning  $d_{zw}^{\overline{\pi}*}$  based on (ii) and (iii). The estimation problem (i) has been developed by Cui et al. (2023); Miao et al. (2018b) using the generalized method of moments, Ghassami et al. (2022); Kallus et al. (2021) by a min-max estimation (Dikkala et al., 2020) using kernels, and Kompa et al. (2022) using deep learning; and (ii) has been developed by Qi et al. (2023). We restate estimation of (i) and (ii) for completeness. With regard to (i), recall that Assumptions 3 and 5 imply the following conditional moment restrictions
$$
\mathbb {E} [ Y - h (W, A, X) | Z, A, X ] = 0,
$$
$$
\mathbb {E} \left[ 1 - \mathbb {I} \{A = a \} q (Z, a, X) | W, X \right] = 0, \forall a \in \mathcal {A}.
$$
respectively. Kompa et al. (2022) propose a deep neural network approach to estimating bridge functions which avoids the reliance on kernel methods. We adopt this approach in our simulation and details can be found in the Appendix.
To estimate  $d_{z}^{*}$ , we consider classification-based approaches according to Zhang et al. (2012); Zhao et al. (2012). Under Assumptions 1, 2 and 3, maximizing the value function in (1) is equivalent to minimizing the following classification error
$$
\mathbb {E} \left[ \left\{h (W, 1, X) - h (W, - 1, X) \right\} \mathbb {I} \left\{d _ {z} (X, Z) \neq 1 \right\} \right] \tag {6}
$$
over  $d_z \in \mathcal{D}_{\mathcal{Z}}$ . By choosing some measurable decision function  $g_z \in \mathcal{G}_{\mathcal{Z}}: \mathcal{X} \times \mathcal{Z} \to \mathbb{R}$ , we let  $d_z(X, Z) = \mathrm{sign}(g_z(X, Z))$ . We consider the following empirical version of (6),
$$
\min  _ {g _ {z} \in \mathcal {G} _ {z}} \mathbb {P} _ {n} [ \{\hat {h} (W, 1, X) - \hat {h} (W, - 1, X) \} \mathbb {I} \{g _ {z} (X, Z) <   0 \} ].
$$
Due to the non-convexity and non-smoothness of the sign operator, we replace the sign operator with a smooth surrogate function and adopt the hinge loss  $\phi(x) = \max\{1 - x, 0\}$ . By adding a penalty term  $\rho_z||g_z||_{\mathcal{G}_z}^2$  to avoid overfitting, we solve
$$
\hat {g} _ {z} \in \arg \min  _ {g _ {z} \in \mathcal {G} _ {z}} \mathbb {P} _ {n} \left[ \left\{\hat {h} (W, 1, X) - \hat {h} (W, - 1, X) \right\} \phi \left(g _ {z} (X, Z)\right) \right] + \rho_ {z} \| g _ {z} \| _ {\mathcal {G} _ {z}} ^ {2}, \tag {7}
$$
where  $\rho_z > 0$  is a tuning parameter. The estimated ITR then follows  $\hat{d}_z(X,Z) = \mathrm{sign}(\hat{g}_z(X,Z))$ . Similarly, under Assumptions 1, 4 and 5, maximizing the value function in (2) is equivalent to minimizing the following classification error
$$
\mathbb {E} [ \{Y q (Z, 1, X) \mathbb {I} \{A = 1 \} - Y q (Z, - 1, X) \mathbb {I} \{A = - 1 \} \} \mathbb {I} \{d _ {w} (X, W) \neq 1 \} ]
$$
over  $d_w \in \mathcal{D}_{\mathcal{W}}$ . By the same token, the problem is transformed into minimizing the following empirical error
$$
\hat {g} _ {w} \in \arg \min  _ {g _ {w} \in \mathcal {G} _ {\mathcal {W}}} \mathbb {P} _ {n} [ \{Y \hat {q} (Z, 1, X) \mathbb {I} \{A = 1 \} - Y \hat {q} (Z, - 1, X) \mathbb {I} \{A = - 1 \} \} \phi \left(g _ {w} (X, W)\right) ] + \rho_ {w} \| g _ {w} \| _ {\mathcal {G} _ {\mathcal {W}}} ^ {2}. \tag {8}
$$
The estimated ITR is obtained via  $\hat{d}_w(X,W) = \mathrm{sign}(\hat{g}_w(X,W))$
For problem (iii), given two preliminary ITRs, we construct an estimator  $\hat{\pi} (X;\hat{d}_z,\hat{d}_w)$ , that is, for  $x\in \mathcal{X}$
$$
\hat {\pi} (x; \hat {d} _ {z}, \hat {d} _ {w}) = \mathbb {I} \{\hat {\delta} (x; \hat {d} _ {z}, \hat {d} _ {w}) \geq 0 \},
$$
where  $\hat{\delta} (x;\hat{d}_z,\hat{d}_w)$  denotes a generic estimator of
$$
\delta (x; \hat {d} _ {z}, \hat {d} _ {w}) \triangleq \mathbb {E} [ h (W, \hat {d} _ {z} (X, Z), X) - Y q (Z, A, X) \mathbb {I} \{\hat {d} _ {w} (X, W) = A \} | X = x ],
$$
where the expectation is taken with respect to everything except  $\hat{d}_z$  and  $\hat{d}_w$ . For example, the Nadaraya-Watson kernel regression estimator (Nadaraya, 1964) can be used, i.e.,  $\hat{\delta}(x; \hat{d}_z, \hat{d}_w)$  is expressed as
$$
\frac {\sum_ {i = 1} ^ {n} \{\hat {h} (W _ {i} , \hat {d} _ {z} (x , Z _ {i}) , x) - Y _ {i} \hat {q} (Z _ {i} , A _ {i} , x) \mathbb {I} \{\hat {d} _ {w} (x , W _ {i}) = A _ {i} \} \} K (\frac {| | x - X _ {i} | | _ {2}}{\gamma})}{\sum_ {i = 1} ^ {n} K (\frac {| | x - X _ {i} | | _ {2}}{\gamma})},
$$
where  $K: \mathbb{R} \to \mathbb{R}$  is a kernel function such as Gaussian kernel,  $||\cdot||_2$  denotes the  $L_{2}$ -norm, and  $\gamma$  denotes the bandwidth.
Finally, given  $\hat{d}_z, \hat{d}_w$  and  $\hat{\pi}(X; \hat{d}_z, \hat{d}_w)$ ,  $\hat{d}_{zw}^{\hat{\pi}}$  is estimated by the following plug-in regime,
$$
\hat {d} _ {z w} ^ {\hat {\pi}} (X, W, Z) = \hat {\pi} (X; \hat {d} _ {z}, \hat {d} _ {w}) \hat {d} _ {z} (X, Z) + (1 - \hat {\pi} (X; \hat {d} _ {z}, \hat {d} _ {w})) \hat {d} _ {w} (X, W). \tag {9}
$$
3.2 Theoretical guarantees for  $\hat{d}_{zw}^{\tilde{\pi}}$
In this subsection, we first present an optimality guarantee for the estimated ITR  $\hat{d}_{zw}^{\hat{\pi}}$  in terms of its value function
$$
V \left(\hat {d} _ {z w} ^ {\hat {\pi}}\right) = \mathbb {E} \left[ \hat {\pi} (X; \hat {d} _ {z}, \hat {d} _ {w}) h (W, \hat {d} _ {z} (X, Z), X) + \left(1 - \hat {\pi} (X; \hat {d} _ {z}, \hat {d} _ {w})\right) Y q (Z, A, X) \mathbb {I} \left\{\hat {d} _ {w} (X, W) = A \right\} \right],
$$
where the expectation is taken with respect to everything except  $\hat{\pi}$ ,  $\hat{d}_z$  and  $\hat{d}_w$ .
We define an oracle optimal ITR which assumes  $\bar{\pi} (X;\hat{d}_z,\hat{d}_w)$  is known,
$$
\hat {d} _ {z w} ^ {\bar {\pi}} (X, W, Z) \triangleq \bar {\pi} (X; \hat {d} _ {z}, \hat {d} _ {w}) \hat {d} _ {z} (X, Z) + (1 - \bar {\pi} (X; \hat {d} _ {z}, \hat {d} _ {w})) \hat {d} _ {w} (X, W).
$$
The corresponding value function of this oracle optimal ITR is given by
$$
V \left(\hat {d} _ {z w} ^ {\bar {\pi}}\right) = \mathbb {E} \left[ \bar {\pi} \left(X; \hat {d} _ {z}, \hat {d} _ {w}\right) h \left(W, \hat {d} _ {z} (X, Z), X\right) + \left(1 - \bar {\pi} \left(X; \hat {d} _ {z}, \hat {d} _ {w}\right)\right) Y q (Z, A, X) \mathbb {I} \left\{\hat {d} _ {w} (X, W) = A \right\} \right],
$$
where the expectation is taken with respect to everything except  $\hat{d}_z$  and  $\hat{d}_w$ .
Then the approximation error incurred by estimating  $\hat{\pi} (X;\hat{d}_z,\hat{d}_w)$  is given by
$$
\mathbb {K} (\hat {\pi}) \stackrel {\triangle} {=} V (\hat {d _ {z w} ^ {\pi}}) - V (\hat {d _ {z w} ^ {\pi}}).
$$
Moreover, we define the following gain
$$
\mathbb {G} (\bar {\pi}) \stackrel {\triangle} {=} \min  \{V (\hat {d} _ {z w} ^ {\bar {\pi}}) - V (\hat {d} _ {z}), V (\hat {d} _ {z w} ^ {\bar {\pi}}) - V (\hat {d} _ {w}) \}.
$$
It is clear that this gain  $\mathbb{G}(\bar{\pi})$  by introducing  $\bar{\pi}$  is always non-negative as indicated by Theorem 2. Then we have the following excess value bound for the value of  $\hat{d}_{zw}^{\hat{\pi}}$  compared to existing ones in the literature.
Proposition 2. Under Assumptions 1-5,
$$
V (\hat {d} _ {z w} ^ {\hat {\pi}}) = \max  \{V (\hat {d} _ {z}), V (\hat {d} _ {w}) \} - \mathbb {K} (\hat {\pi}) + \mathbb {G} (\bar {\pi}) = V (\hat {d} _ {z \cup w}) - \mathbb {K} (\hat {\pi}) + \mathbb {G} (\bar {\pi}).
$$
Proposition 2 establishes a link between the value function of the estimated ITR  $\hat{d}_{zw}^{\hat{\pi}}$ , and that of  $\hat{d}_z$ ,  $\hat{d}_w$ , and  $\hat{d}_{z\cup w}$ . As shown in Appendix G,  $\mathbb{K}(\hat{\pi})$  diminishes as the sample size increases, therefore,  $\hat{d}_{zw}^{\hat{\pi}}$  has a significant improvement compared to other optimal ITRs depending on the magnitude of  $\mathbb{G}(\bar{\pi})$ .
Furthermore, we establish the consistency of the proposed regime based on the following assumption, which holds for example when  $\hat{d}_z$  and  $\hat{d}_w$  are estimated using indirect methods.
Assumption 6. For  $\hat{d}_z, \hat{d}_w$ ,  $E[h(W, \hat{d}_z(X,Z), X)|X] - E[h(W, d_z^*(X,Z), X)|X] = o_p(n^{-\xi})$  almost surely and  $E[Yq(Z,A,X) \mathbb{I}\{\hat{d}_w(X,W) = A\}|X] - E[Yq(Z,A,X) \mathbb{I}\{d_w^*(X,W) = A\}|X] = o_p(n^{-\varphi})$  almost surely.
Proposition 3. Under Assumptions 1-6, we have  $V(\hat{d}_{zw}^{\hat{\pi}}) \xrightarrow{p} V(d_{zw}^{\bar{\pi}*})$ .
The data generating mechanism for  $(X,A,Z,W,U)$  follows the setup proposed in Cui et al. (2023) and is summarized in Appendix I. To evaluate the performance of the proposed framework, we vary  $b_{1}(X), b_{2}(X), b_{3}(X), b_{a}$  and  $b_{w}$  in  $\mathbb{E}[Y|X,A,Z,W,U]$  to incorporate heterogeneous treatment effects including the settings considered in Qi et al. (2023). The adopted data generating mechanism is compatible with the following  $h$  and  $q$ ,
$$
h (W, A, X) = b _ {0} + \left\{b _ {1} (X) + b _ {a} W + b _ {3} (X) W \right\} \frac {1 + A}{2} + b _ {w} W + b _ {2} (X) X,
$$
$$
q (Z, A, X) = 1 + \exp \left\{A t _ {0} + A t _ {z} Z + t _ {a} \frac {1 + A}{2} + A t _ {x} X \right\},
$$
where  $t_0 = 0.25, t_z = -0.5, t_a = -0.125$ , and  $t_x = (0.25, 0.25)^T$ . We derive preliminary optimal ITRs  $d_z^*$  and  $d_w^*$  in Appendix J, from which we can see that  $X, Z, W$  are relevant variables for individualized decision-making.
We consider six scenarios in total, and the setups of varying parameters are deferred to Appendix I. For each scenario, training datasets  $\{Y_{i},A_{i},X_{i},Z_{i},W_{i}\}_{i = 1}^{n}$  are generated following the above mechanism with a sample size  $n = 1000$ . For each training dataset, we then apply the aforementioned methods to learn the optimal ITR. In particular, the preliminary ITRs  $\hat{d}_z$  and  $\hat{d}_w$  are estimated using a linear decision rule, and  $\hat{\pi} (x;\hat{d}_z,\hat{d}_w)$  is estimated using a Gaussian kernel. More details can be found in the Appendix K.
To evaluate the estimated treatment regimes, we consider the following generating mechanism for testing datasets:  $X \sim \mathcal{N}(\Gamma_x, \Sigma_x)$ ,
$$
(Z, W, U) | X \sim \mathcal {N} \Bigg \{\left( \begin{array}{c} \alpha_ {0} + \alpha_ {a} p _ {a} + \alpha_ {x} X \\ \mu_ {0} + \mu_ {a} p _ {a} + \mu_ {x} X \\ \kappa_ {0} + \kappa_ {a} p _ {a} + \kappa_ {x} X \end{array} \right), \qquad \Sigma = \left( \begin{array}{c c c} \sigma_ {z} ^ {2} & \sigma_ {z w} & \sigma_ {z u} \\ \sigma_ {z w} & \sigma_ {w} ^ {2} & \sigma_ {w u} \\ \sigma_ {z u} & \sigma_ {w u} & \sigma_ {u} ^ {2} \end{array} \right) \Bigg \},
$$
where the parameter settings can be found in Appendix I. The testing dataset is generated with a size 10000, and the empirical value function for the estimated ITR is used as a performance measure. The simulations are replicated 200 times. To validate our approach and demonstrate its superiority, we have also computed empirical values for other optimal policies, including existing optimal ITRs for proximal causal inference, as discussed in Section 2.2, along with optimal ITRs generated through causal forest (Athey and Wager, 2019) and outcome weighted learning (Zhao et al., 2012).
Figure 2: Boxplots of the empirical value functions  $(\hat{d}_{cf}$  and  $\hat{d}_{owl}$  denote estimated ITRs using causal forest and outcome weighted learning respectively).
Figure 2 presents the empirical value functions of different optimal ITRs for the six scenarios. As expected,  $\hat{d}_z, \hat{d}_w, \hat{d}_{z\cup w}$ , and  $\hat{d}_{zw}^{\hat{\pi}}$  consistently outperform  $\hat{d}_{cf}$  and  $\hat{d}_{owl}$ , which highlights their effectiveness in addressing unmeasured confounding. Meanwhile, across all scenarios,  $\hat{d}_{zw}^{\hat{\pi}}$  yields superior or comparable performance compared to the other estimated treatment regimes, which justifies the statements made in Sections 2 and 3. In addition, as can be seen in Scenario 5, all ITRs relying on the proximal causal inference framework perform similarly, which is not surprising as  $\hat{d}_z(X,Z)$  and  $\hat{d}_w(X,W)$  agree for most subjects. To further underscore the robust performance of our proposed approach, we include additional results with a changed sample size and a modified behavior policy in Appendix L.
5 Real Data Application
In this section, we demonstrate the proposed optimal ITR via a real dataset originally designed to measure the effectiveness of right heart catheterization (RHC) for ill patients in intensive care units (ICU), under the Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments (SUPPORT, Connors et al. (1996)). These data have been re-analyzed in a number of papers in both causal inference and survival analysis literature with assuming unconfoundedness (Cui and Tchetgen Tchetgen, 2023; Tan, 2006, 2020, 2019; Vermeulen and Vansteelandt, 2015) or accounting for unmeasured confounding (Cui et al., 2023; Lin et al., 1998; Qi et al., 2023; Tchetgen Tchetgen et al., 2020; Ying et al., 2022).
There are 5735 subjects included in the dataset, in which 2184 were treated (with  $A = 1$ ) and 3551 were untreated (with  $A = -1$ ). The outcome  $Y$  is the duration from admission to death or censoring. Overall, 3817 patients survived and 1918 died within 30 days. Following Tchetgen Tchetgen et al. (2020), we collect 71 covariates including demographic factors, diagnostic information, estimated survival probability, comorbidity, vital signs, physiological status, and functional status (see
Hirano and Imbens (2001) for additional discussion on covariates). Confounding in this study stems from the fact that ten physiological status measures obtained from blood tests conducted at the initial phase of admission may be susceptible to significant measurement errors. Furthermore, besides the lab measurement errors, whether other unmeasured confounding factors exist is unknown to the data analyst. Because variables measured from these tests offer only a single snapshot of the underlying physiological condition, they have the potential to act as confounding proxies. We consider a total of four settings, varying the number of selected proxies from 4 to 10. Within each setting, treatment-inducing proxies are first selected based on their strength of association with the treatment (determined through logistic regression of  $A$  on  $L$ ), and outcome-inducing proxies are then chosen based on their association with the outcome (determined through linear regression of  $Y$  on  $A$  and  $L$ ). Excluding the selected proxy variables, other measured covariates are included in  $X$ . We then estimate  $\hat{d}_z, \hat{d}_w, \hat{d}_{z\cup w}$ , and  $\hat{d}_{zw}^{\hat{\pi}}$  using the SUPPORT dataset in a manner similar to that described in Section 4, with the goal optimizing the patients' 30-day survival after their entrance into the ICU.
Figure 3: Graphical representation of concordance between estimated ITRs.
The estimated value functions of our proposed ITR, alongside existing ones, are summarized in Appendix M. As can be seen, our proposed regime has the largest value among all settings. For a visual representation of the concordance between the estimated optimal ITRs, we refer to Figure 3 (results from Setting 1). The horizontal ordinate represents the 50 selected subjects and the vertical axis denotes the decisions made from corresponding ITRs. The purple and yellow blocks stand for being recommended treatment values of -1 and 1 respectively. For the subjects with purple or yellow columns,  $\hat{d}_z(X,Z) = \hat{d}_w(X,W)$ , which leads to the same treatment decision for the other two ITRs. For columns with mixed colors,  $\hat{d}_z(X,Z)$  and  $\hat{d}_w(X,W)$  disagree. We see that in this case  $\hat{d}_{z\cup w}(X,W,Z)$  always agree with  $\hat{d}_z(X,Z)$ , while  $\hat{d}_{zw}^{\pi}(X,W,Z)$  take values from  $\hat{d}_z(X,Z)$  or  $\hat{d}_w(X,W)$  depending on the individual criteria of the subjects as indicated by  $\hat{\pi}$ . In addition to the quantitative analysis, we have also conducted a qualitative assessment of the estimated regime to validate its performance. For further details, please refer to Appendix M.
We acknowledge several limitations of our work. Firstly, the proximal causal inference framework relies on the validity of treatment- and outcome-inducing confounding proxies. When the assumptions are violated, the proximal causal inference estimators can be biased even if unconfoundedness on the basis of measured covariates in fact holds. Therefore, one needs to carefully sort out proxies especially when domain knowledge is lacking. Secondly, while the proposed regime significantly improves upon existing methods both theoretically and numerically, it is not yet shown to be the sharpest under our considered model. It is still an open question to figure out if a more general policy class could be considered. Thirdly, our established theory provides consistency and superiority of our estimated regime. It is of great interest to derive convergence rates for Propositions 2 and 3 following Jiang (2017). In addition, it may be challenging to develop inference results for the value function of the estimated optimal treatment regimes, and further studies are warranted.
Yifan Cui was supported by the National Natural Science Foundation of China.