Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift
Authors: chao ying
Jun Jin
Yi Guo
Xiudi Li
Muxuan Liang
Jiwei Zhao
Venue: ICML 2025 poster (2025)
Keywords: semi-supervised learning
covariate shift
automated computational phenotype
efficiency gain
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning

Abstract: Collecting gold-standard phenotype data via manual extraction is typically labor-intensive and slow, whereas automated computational phenotypes (ACPs) offer a systematic and much faster alternative.
However, simply replacing the gold-standard with ACPs, without acknowledging their differences, could lead to biased results and misleading conclusions.
Motivated by the complexity of incorporating ACPs while maintaining the validity of downstream analyses, in this paper, we consider a semi-supervised learning setting that consists of both labeled data (with gold-standard) and unlabeled data (without gold-standard), under the covariate shift framework.
We develop doubly robust and semiparametrically efficient estimators that leverage ACPs for general target parameters in the unlabeled and combined populations. In addition,
we carefully analyze the efficiency gains achieved by incorporating ACPs, comparing scenarios with and without their inclusion.
Notably, we identify that ACPs for the unlabeled data, instead of for the labeled data, drive the enhanced efficiency gains. To validate our theoretical findings, we conduct comprehensive synthetic experiments and apply our method to multiple real-world datasets, confirming the practical advantages of our approach.

Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift
Chao Ying 12
Jun Jin Yi Guo Xiudi Li
Muxuan Liang
Jiwei Zhao 12
Abstract
1. Introduction
Automated computational phenotype (ACP) refers to the process of using advanced algorithms and models, includ- ing but not limited to, machine learning, natural language processing (NLP), large language model (LLM) and gener- ative AI, to automatically compute and define phenotypes from complex data such as electronic health records (EHRs). Though obtaining gold-standard phenotype data via man- ual extraction is labor-intensive and slow, ACP allows re- searchers to obtain phenotypes systemically and quickly, scaling their efforts in precision medicine and supporting the broader goal of turning raw data into actionable insights for research and healthcare improvements.
Collecting gold-standard phenotype data via man- ual extraction is typically labor-intensive and slow, whereas automated computational pheno- types (ACPs) offer systematic and much faster alternative. However, simply replacing the gold- standard with ACPs, without acknowledging their differences, could lead to biased results and mis- leading conclusions. Motivated by the complexity of incorporating ACPs while maintaining the va- lidity of downstream analyses, in this paper, we consider semi-supervise, learning setting that consists of both labeled data (with gold-standard) and unlabeled data (without gold-standard), un- der the covariate shift framework. We develop doubly robust and semiparametrically efficient estimators that leverage ACPs for general target parameters in the unlabeled and combined pop- ulations. In addition, we carefully analyze the efficiency gains achieved by incorporating ACPs, comparing scenarios with and without their in- clusion. Notably, we identify that ACPs for the unlabeled data, instead of for the labeled data, drive the enhanced efficiency gains. To validate our theoretical findings, we conduct comprehen- sive synthetic experiments and apply our method to multiple real-world datasets, confirming the practical advantages of our approach. Code: ?
Despite these benefits, simply replacing gold staddstandard data with ACPs downstream analyses introduces new chal- lenges. As computational phenotyping algorithms are of- ten trained to minimize prediction error such as the mean squared error (MSE), ACPs may contain non nog-neglie bias for important downstream tasks such as statistical in- ference. Therefore, the indiscriminate use of ACPs, without acknowledging their distinction from gold-standard pheno- type data, can lead biased results and misleading con- clusions. In this work, instead of replacing gold-standard phenotype data, we investigate the benefits of appropriately incorporating these ACPs, in order to enhance the estima- tion efficiency and to achieve the more powerful statistical inference.
In many applications, gold-standard phenotypes (labels) are only available for small subset of individuals due to the high cost of manual extraction, while unlabeled data are more readily accessible. Meanwhile, the selection of labeled individuals is often based on baseline characteristics rather than being purely random. This creates a semi-supervised learning setting, where the covariate distribution differs between labeled and unlabeled data. In this paper, we aim to integrate ACPs into classical semi-supervised learning frameworks to enable efficient estimation and inference.
Depaarmeen of Statistics, University of Wisconsin-Madissn,, Madison. Wisconsin, USA 'Department of Biostatistics & Medical Informatics, University of Wisconsin-Madisoon, Madi- son, Wisconsin, USA 'Henry Ford Hcalth, Detroit, Michi- gan, USA *Department of Health Outcomes & Biomedical In- formatics, University of Florida, Gainesville, Florida, USA "Biostatisties Division, University of California, Berkeley, Cal- ifornia, USA "Department of Biostatistics, University of Florida, Gainesville, Florida, USA. Correspondence to: Jiwei Zhao <ji- wei.zhao@ wisc.edu>.
Our motivating study is an analysis on diabetes among chil- dren and adolescents from the UF (University of Florida) health system (Li et al., 2025), where diabetes status is adju- dicated by medical experts through manual chart reviews for
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Inference by Incorporating ACPs
subset of patients selected via stratified random sampling, while ACPs are available for all patients in the EHR through previously validated decision-tree-basse algorithm. The stratified sampling introduces potential covariate shift be- tween the chart-review population and the general EHR population, Our work focuses on how to incorporate these ACPs with chart review data to improve parameter estima- tion and inference in a prediction model under possible covariate shifts.
et al. (2023); Kim et al. (2024); Lee al. (2025)). They are suitable assumptions under different contexts. Covariate shift, the focus in this paper, aligns with the causal learn- ing setting (Schölkopf et al., 2012) and has been studied comprehensively in the literature (Shimodaira, 2000; Huang et al., 2006; Sugiyama et al., 2008; Gretton et al., 2009; Sugiyama & Kawanabe, 2012; Kpotufe & Martinet, 2021; Aminian et al., 2022; Rodemann et al., 2023).
Prediction-powered inference. To our best knowledge, in- ference with predicted data derived from black-box AT/ML models started from Wang et al. (2020), followed by Mot- wani & Witten (2023). More recently, Angelopoulos et al. (2023a) proposed the approach termed prediction poweerdd inference (PPI), which yields valid inference even when the predictions were of low quality. However, PPI might be worse, in terms of estimation efficiency, than the bench- mark method that uses labeled data only. This motivates growing literature of research that investigates the statis- tical efficiency gains from different perspectives, such as Angelopoulos et al. (2023b); Miao et al. (2023); Gronsbell et al. (2024); Miao et al. (2024); Ji et al. (2025).
1.1. Related Work
Semi-supervised learning and inference. Semi- supervsuer learning has been popular in both machine learning and statistics in the past several decades (Zhu, 2005; Chapelle et al., 2009). Based on different assumptions, computational algorithms and estimation methods have been proposed in classification (Rigollet, 2006; Wang et al., 2022), non- parametric regression (Wasserman& & Lafferty, 2008), and semi-supervised regression (Kostopoulos et al., 2018), to understand the benefits of the abundant unlabeled data.
In particular, there are considerable progresses in the past several years on how to make use of the unlabeled data for parameter estimation, either with a faster convergence rate or with a smaller asymptotic variance. For example, Chakrabortty& & Cai (2018) and Azriel et al. (2022) pro- posed estimators that are more efficient than the ordinary least square (OLS) that uses labeled data only, under the essumption-lean regression framework (Buja et al., 2019; Berk et al., 2019). Similar conclusions could also be found under the general M-estimation framework (Song et al., 2023). The idea was also extended to the high-dimensional setting where the number of features is allowed to be greater than the sample size (Zhang et al., 2019; Zhang& & Bradic, 2022; Cai & Guo, 2020; Deng et al., 2024), as well as in the situation without the sparsity assumption (Livne et al., 2022; Hou et al., 2023).
Surrogacy in biostatisties and causal inference. In the literature of biostatistics especially clinical trails, there is line of research on identifying, evaluating and validating surrogate variables, motivated by the fact that primary end- point may be invasive, costly, or take a long time to measure. This dates back to Prentice (1989) who first proposed the definition and the criterion for surrogacy. Readers of interest could refer to Elliott (2023) for a comprehensive review of surrogate evaluation. This type of research usually seeks the replacement of primary endpoint with surrogate variable, enabling traditional or provisional approval of treatment in clinical trials. Not surprisingly, the replacement process often requires stringent, unverifiable assumptions, such as the strong surrogate assumption in Prentice (1989). More re- cently, surrogate variables were also analyzed in the causal inference framework enhance the efficiency of treatment effect estimation on long-term primary outcomes; see, e.g., Athey et al. (2019); Gupta et al. (2019); Athey et al. (2020); Cheng et al. (2021): :): Tran et al. (2023); Zhang et al. (2023); Kallus & Mao (2024); Imbens et al. (2024); Gao et al. (2024).
Distribution shift. Distribution shift (Quinonero-Candlla et al., 2008) refers to the phenomenon that the joint distri- butions between the training and testing data are different, or, in our context, between the labeled and unlabeled data. Therefore, the knowledge learnt from the labeled data may no longer be appropriate to be directly used in the unlabeled data or in the combined data. This also motivates the study of unsupervised domain adaptation (Kouw& & Loog, 2021), aiming to address the distributional shift between the labeled and unlabeled data domains.
Additionally, surrogate variable is relevant to the misclassifi- cation problem the label noise issue in machine learning (e.g.. Lawrence & Schölkopf 2001; Scott 2015; Li et al. 2021; Liu et al. 2023; Guo et al. 2024), also the measure- ment error issue in statistical literature (e.g., Carroll et al. 2006; Buonaccorsi 2010; Yi 2017; 2021).
Two dominating types of distribution shifts are covariate shift (the marginal distribution of the feature changes) and label shift (the marginal distribution of the label changes; see, c.g., Storkey (2009); Zhang et al. (2013); Du Plessis & Sugiyama (2014); Iyer et al. (2014); Nguyen et al. (2016); Tasche (2017); Lipton et al. (2018); Garg et al. (2020); Tian
1.2. Our Contributions
Firstly, we consider a semi-supervised learning setting under the covariate shift framework; that is, the labeling mecha-
2

---

Efficient Inference by Incorporating ACPs
nism may depend on features might necessarily be purely random. As discussed in motivating study, the la- beling mechanism is often unknown in applications, making it more flexible to assume dependence on features (covariate shift) rather than purely random. In PPI, Angelopoulos et al. (2023a) did consider scenarios involving covariate shift; however, their method assumed that the difference of the two distributions is completely known. Certainly this is not feasible in practice.
Table I. Data structure considered in this paper: Scenario I (w.o. ACP) vs Scenario Il (w. ACP).
SCENARIO
SCENARIO II
R
Y
X
R
Y
X
I
1
V
C
:
1
Secondly, our assumption on how the ACPs are generated is flexible. The ACP, denoted as Y in the paper, does not have to be an accurate prediction of the ground truth Y. Indeed, y does not have to in the same magnitude as Y, or even the same format as Y. For example, Y could be continuous variable even if is binary. addition, we assume the generating algorithm of Y is somewhat black- box, which, beyond the available feature X in the labeled and unlabeled data, might also depend on some additional unseen feature, say, Z.
n
0
V
0
V
u
V
V
V
n N
0
V
from 21 (R = 0). Please refer to Scenario I (w.o. ACP) in Table for the data structure.
We use p(-) and p(- |.) -) to denote the generic marginal and conditional distributions for the labeled data L, and q(.) and q -) for the unlabeled data U. Instead of assuming both sources of data follow exactly the same distribution, we only assume the conditional distribution of outcome Y given feature X remains the same, i.e., ply x) g(y x), while allowing the marginal distributions of X to differ between L and U, i.e.,
Under this general framework, we demonstrate that the proposed estimator in this paper ensures that incorporating ACPs never reduces estimation efficiency. The only scenario with no efficiency gain occurs when the generation of ACPs depends solely on the available feature X. This aligns with our intuition, simply because in such an extreme situation the ACP Y does not bring in any new information. Further- more, we highlight that the efficiency gain arises specifically from the ACPs in the unlabeled data. In contrast, ACPs in the labeled data do not contribute to the efficiency gain. This intuitive since the labeled data already contain the ground truth Y. so a less precise ACP Y provides no additional value. We rigorously support these efficiency comparison results by carefully analyzing and comparing the asymptotic variances of the estimators using closed-form formulas.
p(x) + q(x).
This covariate shift assumption implies the independence between Y and R conditional on X. That is,
pr(Rt = Y,X)= = pr R = X),
(1)
which is less stringent than the assumption that the sam- pling of the labeled data is purely random; i.c., pr(Ht = 1 Y. X) pr(R R 1).
Our contributions also include the development of semi- parametrically efficient and doubly robust estimators for the target parameter in both the unlabeled data population and the combined data population. By utilizing the cross- fitting technique. the proposed estimator achievesdouble robustness and, more importantly, attains the best possible estimation efficiency among all regular and asymptotically linear estimators.
Objectives and metrics. Under the covariate shift assump- tion, the distributions of the labeled data L, of the unlabeled data 21 and of the combined data U are all different. In the main paper, we focus on the d-dimensional parameter B for some characteristic the unlabeled data population U, defined
= argmin E(y, ()),,
2. Setup
where Eg represents the expectation with respect to the distribution of U4 and É(-) denotes a generic loss function. Equivalently, with a smooth loss function, we write (3 the solution of the estimating equation
Semi-supervised learning setting under covariate shift.
We adopt the standard semi-supervised learning setting where we have labeled data L that contains independent and identically distributed (i.i.d.) (y,, x.). i as well as unlabeled data 24 that contains i.i.d. x,. j I,... n + N M. We use binary indicator to
,,...,n,
(sY, X; 0.
(2)
For example, when the outcome mean EY) ) is of inter- est, one can define E(y, x; 33 as (y - 9)2/2 and the cor- responding s(Y, X: B) is Y - B. As another example,
denote whether one particular subject is from L (R 1) or
3

---

Efficient Inference by Incorporating ACPs
if the linear regression coefficient is of interest, one can define fly. x; 3) (y- sY,, 33 = (Y x)
to denote the best approximation of the truth within possibly misspecified model, e.g., m,(x) and m mx)x.. We assume the d x d symmetric matrix E (Y, X; B) /8B" evaluated at the true Go is invertible and denote this inverse as Ω. Clearly, Ω = Ω² For random vector Φ, we write E( B(<b") as E( )2
and, correspondingly,
The overarching goal of this paper is to understand how to best estimate the parameter (3 without incorporating ACPs, with incorporating ACPs (below) and their comparison on estimation efficiency.
3. Efficiency Gain Analysis
As an alternative, if some characteristic of the combined data population CUU is of interest, one can define the parameter 0, similarly as above. Indeed, all the results presented in this paper for 33 can be similarly developed for 0. For the interest of space, we only present some brief results for 0 in Appendix B.
In this section, mainly analyze the statistical benefits, in the sense of estimation efficiency, of the ACP y; 's Our first step is to understand the optimal estimation efficiency with and without incorporating these Ø 's, for estimating B. Here, the optimal estimation efficiency the so-called semi- parametric efficiency bound, characterized by the efficient influence function, also referred as canonical gradient, or, efficient influence curve (Bickel et al., 1993; Tsiatis, 2006; Fisher & Kennedy, 2021; Hines et al., 2022; Ichimura & Newey, 2022).
Incorporating ACPs. We assume that there exists black-box prediction model, and we have automated com- putational phenotypes (ACPs) Te for every subject i E (1. ... Here, this black-box model is usually trained from machine learning, natural language processing, or large language model (LLM) such as ChatGPT, where the input feature might contain not only X but also some other variable Z. For the data structure, please refer Scenario II (w. ACP) in Table 1.
3.1. Optimal Efficiency Bound with and without ACPs
We first briefly introduce the asymptotic representation of a regular asymptotically linear (RAL) estimator. In general, given i.i.d. copies of the random sample (di,..., dn. with sample size n, an estimator for the parameter of interest B, B, is a RAL estimator if
To understand the statistical benefits of these ACPs in the presence of covariate shift, we further assume
p(0 y)
g(ū x. y).
Vn(a - B) = nn2 Σ ((d.) + op)),
Together with the original covariate shift assumption in that ply x) = <4v x). it implies
where the zero-mean function Φ(-) is called the influence function of B, and the corresponding asymptotic variance is E( E(<b0") provided that it is finite and nonsingular. Among all RAL estimators for B, the influence function of the one with the smallest asymptotic variance is called the efficient influence function (EIF), Pere and the optimal efficiency bound is E(
pr R = Y v x) pr(R-= = x
(3)
Again, this assumption is still less stringent than the as- sumption that the sampling of the labeled data is purely random.
Remark 2.1. The assumption (3) is equivalent to stating that, (Y, Y) and R are conditional independent, given X. can be tested when Y is available in the unlabeled data
To proceed. we first consider the situation with the ACPs, Scenario II in Table 1. The result below, Proposition 3.1, presents the EIF for estimating 3 under this situation, as well as the efficiency bound. The proof of this result is contained in Appendix A. The main idea is to first derive the semiparametric tangent space T (Bickel et al., 1993; Tsiatis, 2006) under this situation, that defined as the mean squared closure of the tangent spaces of all parametric submodels spanned by the score vectors, and then derive the EIF using the orthogonality.
but cannot if otherwise. When it cannot be tested, one can decompose the assumption as p(7)x) = x00xx) and p(ylx, 7) = 4'vxx, V) It is clear that, the untestable part p(v(x, 7) = q(yl xx 77 is something similar to the covari- ate shift assumption p(y|x) = 4(v(x), which is, albeit untestable, popularly adopted in semi-supervised learning and distribution shift.
Notation.
We denote - = n/M, w(x) = g(x)
p(x) and ग (x) =
Proposition 3.1. With the ACP Th 's, the EIF for estimating (3 defined in (2) is Lt with Φ.. equals
pr(Rt = 1 x). Clearly, w(x) I- 7(x)
1-a(x) and I-
(-7(x)
(x) [x] We denote m(x) = B(s(Y, x: [x) and m xXs(Yxx;))Ixx
1-r
Wi (x) ( x: X; - mx (x,0)} + 1 - mx)
We also simply write
= (x),,, m(x), m(x D)). We use subscript to de- note the ground truth of nuisance functions and superscript
wx(x)
i (x, 1) mx)))
(4)
π + π T))o(x)
4

---

Efficient Inference by Incorporating ACPs
and the efficiency bound equals SVV St, where w is
Table 2. Data structure considered in this paper: Scenario III (in-
1
- (X, P)2
termediate scenario between Scenario I and Scenario II).
- Ep
SCENARIO III
+ (1 772 (2) $2
E [1 mo (m,, -mo (x)
R
Y
X
Y
1- Emo
1
V
V
C
:
Next we derive the efficiency bound for Scenario in Table I, without the ACPs. This can be regarded as a special case of Scenario Il in Table I, with the ACPs, so the proof is omitted.
n
V
V
v
n. +
V
u
V
11 N
V
Proposition 3.2. Without T 's the EIF for estimating B defined in (2) is 12 wo where
Proposition 3.5. For the intermediate scenario in Table 2, the estimation efficiency bound equals to the one in Proposi- tion 3.2, corresponding to Scenario I in Table 1.
- [x)(
mx 1- mx(),,
1-r
wo
and the efficiency bound equals S2V wos where
The proof of Proposition 3.5 is similar to Propo- sition 3.1 so omitted. It implies, the ACPs for the labeled data only were available, then this set of ACPs does not bring in any efficiency gain. Therefore, compared to Sce- nario I in Table 1, the efficiency gain of Scenario II in Table I comes from the ACPs for the unlabeled data. This is intu- itive. For the labeled data, the ground truth Y's exist and thus the ACPs Y'ss do not contribute; while for the unlabeled data, the ground truth Y's do not exist and their ACPs Y's do contribute.
Remark 3.6.
V wo
Ep (X) mo(X)
π 1 mm (X)
3.2. Efficiency Gain and its Source
To further understand the efficiency gain of the ACPs, which the contrast between Scenario and Scenario Table I, we have
Proposition 3.3. The difference of the two asymptotic vari-
4. Estimation
ances, SV wo SVV. equals
We mainly present the estimator 3 that incorporates the Ω ACPs. From the EIF presented in (4), to construct an effi- cient estimator, one needs to estimate the relevant nuisance functions p (w(x), m(xx 7), mx)). To accommodate estimations of nuisance functions using nonparametric or
To {3) To X)
(1 - m)² DE
(m (X, mx)X)
which is always positive semidefinite.
Remark 3.4. The proof of Proposition 3.3 is contained in Appendix A. From Proposition 3.3, it is clear that incor- porating ACPs will not attenuate the estimation efficiency. The extreme special case is that, the generation of y only depends on the available feature X; that is, Ý is a function of X. In this case, one can derive that (x, 87) = mx)x) and there is no efficiency gain. The intuition is that, in this case, Y does not bring any new information beyond X and thus does not bring in efficiency gain. If the generation of Y depends able Z predictive of Y, the estimator with ACPs does bring positive efficiency gain, compared to the estimator without ACPs.
Algorithm Estimator and Confidence Interval
Input: Labeled dataset D {(x,, Mi, 0i) i
......n). unlabeled dataset D" {(x,, 01) i n + 1,..., n N). To use cross cross-fitting. define D!
DI U... D'x and D = DY U... D with D
... = | = n and DD = .... D = N/K.
X as well as some other different vari-
Define D₁ = D UDD =1,
Output: Obtain 33 and confidence interval of 3 for any vector E Rdd
for 1: = to do Estimate Pk using DG. obtain Pk- end for
So, where exactly does the efficiency gain come from? The ACPs for the labeled data, or the ACPs for the unlabeled data, or both? To answer this question, we consider an intermediate scenario as shown Table below. For this scenario, we have the following
Plug p into equation (5) and solve the equation, obtain B.
Plug P and 3 into equation (6) and obtain ΩV S2.
Obtain the confidence intervals of 'T by equation (8).
5

---

Efficient Inference by Incorporating ACPs
machine learning methods, we adopt the cross-fitting tech- nique (Chernozhukov al., 2018) in our estimation.
al. (2024). The convergence rate is achievable for many ML methods such as in regression trees and random forests (Wager & Walther, 2015) and class of neural nets (Chen & White, 1999). One can refer to Chernozhukov et al. (2018) for more examples.
Take K-fold random partitions { {Di k=1 and { (D k== of the labeled and unlabeled data sets D' and D", respectively. Then D2- = D U D constitutes a K-fold random partition of the whole data set {(x;, rithi 11) For each k = M m and let Ps. -wx (x), m, (x, (), m2 (x)) denote the estimates of nuisance functions obtained using the data set D& only. Let
Theorem 5.3 (Double Robustness). Assume Assumption 5.1 holds. If either w" (x) = wo(x) or (m" (x, 0), m (x)) = (mo(X,)) D), mo mo(x)), then
1....,K, K. we define D {xx, rish. Di
| = 90
(Pi,..., px) Then the estimator B(p), shorthanded as 3. is the solution of the following equation:
Remark 5.4. Theorem 5.3 establishes the classic dou- ble robustness property, meaning that the proposed es- timator 33 remains consistent if w*(x)) (m" (x. 7), m (x)) (mo(x, fJ, mx()).
K
R Wy
wo(x) or
Ek
(X)
(x(, - m (X, 8)}
K
k=1
WE (X)
Consequently, we can develop the asymptotic representation of the proposed estimator 33 and further show that it achieves the optimal efficiency bound derived in Proposition 3.1.
m (X, 8) - mm (X))
π + (1 - T) (X)
R
+ 1 1 m (X)
= =0
(5)
Theorem 5.5 (Asymptotic Normality and Semiparametric Efficiency). Assume Assumption 5. holds. Further; if we assume 01 02M o(M-1/2), 01M50MM o(M -12)), and that all nuisance components are correctly specifieu, then as M + 00,
where Eg denotes the sample average over the k-th fold, and we denote the lent-hand side of the above equation N(p, 3). We briefly summarize the computational algo- rithm in Algorithm 1.
VM Bo) S N(O (VO.SVV...)
Remark 4. In our implementation, for the estimations of nuisance functions, we adopted super learner (Van der Laan et al., 2007), an ensemble learning approach that improves estimation by combining multiple machine learning models with data-adeptive weights.
where SVV,,. 2E ) is the semiparametric effi- ciency bound derived in Proposition 3.1.
Remark 5.6. Theorem 5.5 further establishes that if all nui- sance estimators converge to their true values at a suffi- ciently fast rate, the proposed estimator 3 attains a con- vergence rate of Op( and is asymptotically normal with the efficiency lower bound S2V as its limiting vari- ance. Further, it introduces the concept of rate double ro- bustness. As one can see from Theorem 5.5, what matters is the products a 41MM2M and diarda. instead of the in- dividual convergence rates 01M- 02M and A3M. Notably, this rate requirement is relatively mild, ensuring consis- tency and efficiency even when all nuisance estimators con- verge at op Op-M1//) which considerably slower than the parametric rate Op(M-1/2) In particular, we impose restrictions requiring nuisance estimators to belong to Donsker or bounded entropy classes (van der Vaart, 1998), thereby permitting the use of flexible machine learning methods. Furthermore, the product rate condition allows faster-converging estimators to offset the effects of slower- converging ones, enhancing practical applicability.
5. Theoretical Investigations
Now we develop the theoretical results for the proposed estimator 3, with all the proofs contained in Appendix A. Firstly, we need some assumptions on the nuisance functions and their estimates.
Assumption 5.1 (Requirements on Nuisance Estimators). For = I..... K, we assume positive density ratio models such that (x) and wx)x are bounded away from zero. We assume the nuisance estimators Āk (mj, (x), ma F), fu (x)) converges to the limit p²
(wo (x), m" (x, V) m= (x)) in MSE at the following rates:
(x) w" (x)ll2 = a1M;
[x) mn mx))]112 .2M,
m Ø- (x, 1l2 = agM:
Remark 5.7. An interesting scenario is when we have lot more unlabeled data in that N >> n so π + 0. Accordingly, one can derive that S2V.., = O instead of (1). Thus, the above asymptotic representation shall be more precisely written
where max max(aLM, a2M, (2M)) + as M- + +00.
Remark 5.2. The regularity conditions on nuisance estima- tors are standard and widely advocated in double/debiased ML literature, e.g., Van der Laan & Rose (2011), Cher- nozhukov et al. (2018), Kennedy (2024), Chernozhukov
Vn(ß - (b) 4 N(O, S).
6

---

Efficient Inference by Incorporating ACPs
This indicates, even if we have sufficiently many unlabeled data, the real sample size for deriving the efficient estimator is still n. Intuitively we do not have the ground truth Y in the unlabeled data, so they (with sample size N) cannot contribute to increasing the convergence rate.
Figure I. Variation of ARE for the estimation of (Y) under different sample sizes, signal strength, and correlation coefficient.
Finally we present how to construct the confidence interval for the linear combination vT 3 as well as its validity.
Theorem 5.8 (Construction of Confidence Interval). As- sume Assumption 5.1 holds. If we assume a 41M42M = )() dIMaaM = o(M)1/2), and that all nuisance components are correctly specified, then as M - xc,
S. =n áit Exlo (p (2:3) 3 DV (6)
Consequently, for any vector E Rd have
(B Bb) 4 1). VVVVVV.VVVl
Thus, we can construct (1-a) a)xx 100% confidence interval for 7 vT
CL, = 11:11-
2 -
(8)
where sd (VTVV wo Stv) and & is the cumulative dis- tribution function of the standard normal distribution. Fur- ther, this CI satisfies
Y. The vector (X, Z) follows a multivariate normal distri- bution N(0p+1. E). The covariance matrix Σ is given by Σ I + A, where A1(++1) other elements are zero. We vary Ç in {0, 0.3, 0.6, 0.9, 1} to characterize how much information in Y is overlapped with X. Two regression models are considered: for con- tinuous outcomes,
(p+1)1
Ç, and all
lim pr (vT E CT.) = 1 - M-000
(9)
Remark 5.9. Theorem 5.8 establishes that, under the same conditions, the efficiency lower bound be consis- tently estimated by constructing the sample estimator of E6 B,)} using cross-fitted nuisance estimators and the proposed estimator 3. Consequently, the confidence interval in equation (8) asymptotically attains the correct coverage probability. Moreover, since the proposed esti- mator v asymptotically achieves the smallest possible variance, the confidence interval in equation (8) tends to be shorter than those based on less efficient estimators.
Y, 1 1 <'x, where 6 N(0, 1), while for binary outcomes, set Y, ~ Bernoulli (logit (ooiiii(oooiii ((1 + ="x, + aZ,)) } for all = 1...., a is varied in 1,......, 5} to represent varying accuracy of the ACP. Additionally, we generate binary variable R2 ~ Bernoulli(logit (n²"',,) and n = 1 R, indicates labeled data with Y, observed, whereas R, indicates unlabeled data with Y, unavailable.
X, + aZ, + E..
N. In both cases, & = (1,).5,) and
For the parameters of interest, we consider the mean of Y on the unlabeled dataset (Y) and the regression coefficients solving the estimating equation ( - g {( = O where g(-) is the identify function for continuous outcomes compare and the expit function for binary outcomes. the performance the estimators with and without Y. we calculate their MSEs based on 500 simulation replications.
6. Numerical Evaluations
6.1. Synthetic Data
In this section, we demonstrate the efficiency gains achieved by incorporating ACPs into the estimation process compared to the estimation without ACPs through simulation studies.
The simulation setup is as follows. Recall that the total sam- plc size from the labeled and unlabeled data is M = n + N, and here we vary n, N E {300, 600, 900, 1200, 1500). The covariate vector has a dimension of p = 5, and we gen- erate an additional variable Z which used as the ACP
We report the results on (Y) in linear models in Figure 1, while deferring all other results to Appendix C. Specifically, Figure 1 shows the efficiency improvement by incorporat- ing ACPs, under varying values of labeled sample size (n), unlabeled sample size (N), signal strength from the ACP

---

Efficient Inference by Incorporating ACPs
Figure 2. Difference in distributions of the inpatient visit count for labeled (left) and unlabeled (right) dataset.
Figure 3. The 95% confidence interval for the coefficient corre- sponding each variable. Heavy comorbidity burden is defined as CCI2 2.
7300
Mothod
Eatimate
vist
(a), and the correlation between ACP and the covariates ((). Efficiency gain is quantified by Asymptotic Relative Efficiency (ARE), defined as the ratio of the MSE of the estimate without ACP to that with ACP.
Aigtt
Congration Heart Falion
B&n om 30
Heary Comeotially Buiden
Ippustent Vilit Coact
Type bott -paid
Gespostent Visc Coart
First, we note that ARE greater than indicates incorporat- ing ACPs results in smaller MSE and thus, more efficient estimation. This is indeed the case across all simulation settings, except when the ACP is not predictive of the true outcome (a 0) when it provides no additional informa- tion beyond what is captured in X (C- = 1).
our labeled dataset. Due to the stratified random sampling, we observed significant covariate shifts between the labeled and unlabeled datasets. Figure 2 provides strong evidence that the distribution of inpatient visit counts differs substan- tially. Patients in chart review are more likely to have more inpatient visit counts, which may be due to the need to have sufficient patient information for chart review. Other avail- able features include socio-demographic variables, e.g., sex and race/ethnicity, and clinical variables, e.g., family history of diabetes and Charlson Comorbidity Index (CCI) scores. A cohort summary associated with socio-demographic vari- ables is shown in Table 3, which shows that the distributions of many covariates differ substantially.
Second, ARE generally increases as ( decreases, with the maximum ARE achieved at ( = O. Note that (= = 0 indi- cates that Y is independent from the predictors, and thus the ACP provides the most additional information. We also observe that ARE increases with increasing a, as the ACP becomes more predictive of the true outcome. Given fixed values of a and S, when the sample size of the unlabeled dataset increases, the ARE generally increases, although the rate is relatively moderate; when the sample size of the labeled dataset increases, the ARE exhibits a quadratic de- creasing trend before gradually stabilizing, aligning with the results in our theoretical investigations.
Table 3. Comparison of the distributions of each X variable be- tween labeled and unlabeled data: t-test is used to assess whether the means of the two groups differ significantly for continuous variables, and chi-square test is used to evaluate whether the dis- tributions between the two groups are the same for categorical variables. A p-value less than 0.05 indicates the statistical signifi-
6.2. Diabetes Data
In this section, we implement our proposed method in our motivating study, identify risk factors for diabetes among children and adolescents.
cance.
Vartable
Labcled (n- 297)
Unlaheled (N 000)
p-value
Age
7.59 6:46
7.2+ 5.35
0.653
Inputient Visit Count
0.25 0.71
<.0000
Outparient Visit Count
3.43 6.67
2.18
0.070
The study population comprised individuals under 18 years of age who had at least one encounter recorded between January 1, 2012, and December 31, 2020. Using the criteria outlined in Li et al. (2025), we identified 3,000 patients who are suspected to have diabetes, including both Type I and Type II diabetes, as our unlabeled dataset. For these patients, we followed an ACP using decision- cerision-ttre-basee based algorithm to compute and identify diabetes status. Besides, through stratified random sampling, 297 patients were selected, and their documented visit summary and EHRs were sent to and reviewed by medical experts who provided binary (yes/no) assessments of diabetes status. We refer to this dataset as
Insurmne Type: Sclf-Paid
24 (8.1%)
(2%)
0.000
BMI2 (Yes)
(19.9%)
(92466)45)
0.000
Congestive Heart Pailure (Yes)
28 (9.448)
85 (2.8%)
0.001
Heavy Comorbidity Burden
20.66.75
85 (2.8%)
0.001
To identify driving factors for diabetes diagnosis. we first conducted marginal screening of all categorical variables using Fisher's exact test. Variables with a p- p-value < 0.05, ad- justed by the Benjamini-Hochherg procedure to control the false discovery rate, were selected. We adopted a logistic re- gression to predict diabetes status with all selected variables, and use the regression coefficients as association measures. We then implemented the proposed methods with and with-
8

---

Efficient Inference by Incorporating ACPs
out ACPs estimate these coefficients. For nuisance param- eter estimation, we employed super learner implemented via the R package SuperLearner with the following li- braries: generalized linear models, random forests, kemee support vector machines, and XGBoost. These algorithms were individually tuned using 10-fold cross- validation... To accommodate the flexible estimations for nuísance param- eters, we also applied cross- cross-fitting fitting algorithm. To obtain confidence intervals associated with the target parameters, perturbed bootstrap procedure with 500 repetitions was implemented. In each repetition, individual sample weights were generated from an independent Exp(1) distribution. Finally, the confidence interval was constructed using the 0.025 and 0.975 quantiles of the 500 bootstrap repetitions.
variate shift, we split the labeled and unlabeled data in 9 ratio, following the same procedure in the income data, where X = (1) with X1 representing hedge and a = (0.1)"
Wine data: Using the Wine Enthusiast review dataset, we investigate the association between wine rating (range from 80 to 100) and wine price, adjusted by wine region (Ji et al., 2025). Similar to the politeness data, the ACP Y is also generated by employing OpenAI's GPT-40 mini-model that produces predicted ratings with the same scale. To assess covariate shift, we follow the same procedure as in the previous experiments, splitting the labeled and unlabeled data in a 3 ratio. Here, X = (1, X2, X3. Xx)) where X1 represents price, X2 to X5 represents California, Washington, Oregon and New York, respectively, with a (0, 1, 0, 0,
Figure 3 summarizes the estimates of the coefficients and their 95% confidence intervals. Overall, compared with the method without ACPs, the proposed method with ACPs yields shorter intervals, which reflects the efficiency gain by incorporating ACPs in our proposed method. By incor- porating ACPs, we identified insurance type (self-paid) as an important factor associated with diabetes. The self- elff-paid patients often have limited access to preventative care and experience higher stress levels, which are known factors as- sociated with Type 2 diabetes among adults (Kelly & Ismail, 2015; Stark Casagrande & Cowie, 2012).
For each of these three data sets, we repeat the data splitting 50 times, and then compute the length of 95% confidence intervals of the regression coefficients. We compare the proposed method with PPI, PPI++ and RePPI, with results contained in Table 4. While there are some cases that the proposed method is tiny slightly less efficient than PPI++ or RePPI, it is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction in confidence interval length reaching approximately 60%.
6.3. Other Real-World Datasets
Table 4. Comparison of the length of 95% confidence intervals between proposed method and three alternatives: PPI, +PI++, RePPI, on three datasets: income, politeness, wine. Bolded values (less than 1) indicate that the proposed method has efficiency gain.
In this section we implement the proposed approach three other data sets, and also compare with some existing methods: PPI (Angelopoulos et al., 2023a), PPI++ (An- gelopoulos et al., 2023b), and RePPI (Ji et al., 2025).
Diraue
Valalle
PM
peppoee
Income data: Following Angelopoulos et al. (2023a) and Angelopoulos et al. (2023b), we analyze the relationship between wage (measured as log-income) and age, with sex as confounding variable, using U.S. Census data under covariate shift setting. The ACP prediction Y is obtained by fitting an XGBoost model to log-income using 14 covari- ates, including education, marital status, citizenship, race, and others (Ji et al., 2025). To induce covariate shift, we partition the labeled and unlabeled datasets based on the selection probability exp(a (p)(aXX//1 (1 + exp(a (X)), where = (0, 1, O)T and X (1, X1, TT, with X1 denoting age and X2 denoting sex. This yields a final labeled-to- unlabeled data ratio of approximately 2: 8.
Sncceso
0.0014
0.3622
00012
0.0012
0.115
031534
$4.51
57.14%
99.54%
18.73//
66.7%%
Hediy
5654
1.7894
1149
1.7118
65.17%
71.78%
73.77%
PMoce
I14007
0,2012
03391
0.164
41.55%
$7.31%
Caifirnic
1.5966
0,7935
0.788
677655
46 46.39%
08.57%
(0) 11.55%
Wase
Wadhingon
1.549
08636
08105
08005
46.0%%
96.81%
(00 00.006
1,8399
00487
03836
03963
47.30%
15.34%
101
7. Discussion
In this paper, we explore the benefits of incorporating ACPs in a semi-supervised learning framework under covariate shift, particularly in terms of improving estimation effi- ciency. We propose estimator that is both doubly robust and semiparametrically efficient. Additionally, our method allows for a rigorous quantification of the efficiency gain through closed-form expressions. In general, ACPs are typically generated by various machine learning models, in- cluding but not limited to NLP and LLM. These predictions often come with measures of uncertainty quantification. An intriguing direction for future research is investigating how to effectively integrate these uncertainty quantification mea- sures associated with ACPs into the learning process.
Politeness data: Using the data from Danescu- Mizil et al. (2013) that comprises texts from 5,512 online requests posted on Stack Exchange and Wikipedia, we try to understand the association between politeness score (range from 1 to 25) and binary indicator for hedging within the request (Gligorić et al., 2024). The ACP Y is gen- erated using OpenAl's GPT-4o mini-model that has the same range as the politeness score. To demonstrate the co-

---

Efficient Inference by Incorporating ACPs
Acknowledgement
Buja, A., Brown, L.., Berk, R., George, E., Pitkin, E.,
Traskin, M., Zhang, K., and Zhao, L.. Models as approxi- mations i: consequences illustrated with linear regression. Statistical Science, 34(4):523-55) 544, 2019.
We gratefully appreciate the ICML anonymous reviewers for their valuable feedback. The research supported in part by NSF (DMS 1953526, 2122074, 2310942), NIH (R01DC021431) and the American Family Funding Initia- tive of UW-Madison.
Buonaccorsi, J. P. Measurement Error: Models, Methods, and Applications. Chapman and Hal//CRC, 2010.
Cai, T. and Guo, Z. Semisupervised inference for explained variance high dimensional linear regression and its applications. Journal of the Royal Statistical Society: Series (Statistical Methodology), 82(2):391-419 2020.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. Our study does not involve human subjects, complies with all legal and ethical standards, and we do not anticipate any potential harmful consequences resulting from our work.
Carroll, R. J., Ruppert, D., Stefanski, L. A., and Crainiceanu, C. M. Measurement Error in Nonlinear Models: A Mod- ern Perspective. Chapman and Hall/CRC, 2006.
References
Chakrabortty, A. and Cai, T. Efficient and adaptive linear regression in semi-supervised settings. Ann. Statist., 46 (4):1541-1572, 2018. doi: 10.1214/1-AAS1150
Aminian, G., Abroshan, M., Khalili, M. M., Toni, L.,, and approach to semi-supervised learning under covariate-shift. In Inter- national Conference on Artificial Intelligence and Statis- tics, pp. 7433-7449, PMLR, 2022.
Rodrigues, M. An information-theoree
Chapelle, O., Scholkopf. B., and Zien, A. Semi-supervised learning. IEEE Trunsactions on Neural Networks, 20(3): 542-542, 2009.
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, rredictiio-powertt inference. Science, 382 (6671):669-677 674, 2023a.
Chen, X. and White, H. Improved rates and asymptotic normality for nonparametric neural network estimators. IEEE Transactions on Information Theory, 45(2);682- 691, 1999.
Angelopoulos, A. N., Duchi, J. C., and Zrnic, T. Ppi++: Efficient prediction- poweerd inference. arXiv preprint arXiv:2311.01453, 2023b.
Cheng, D., Ananthakrishnan, A. N., and Cai, T. Robust and efficient semi-supervised estimation of average treatment effects with application to electronic health records data. Biometrics, 77(2):413-42) 2021.
Athey, S., Chetty, R., Imbens, G. W., and Kang, H. The sur- rogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Technical report, National Bureau Economic Research, 2019.
Chernozhukov, V., Chetverikov, D.,. Demirer, M., Dutlo. E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters: Double/debiased machine learning. The Econometrics Journal, 21(1), 2018.
Athey, S., Chetty, R., and Imbens, G. Combining experimen- tal and observational data to estimate treatment effects on long term outcomes. arXiv preprint arXiv:2006.0966 2020.
Chemozhukov, V., Hansen, C., Kallus, N., Spindler, M., and Syrgkanis, V. Applied causal inference powered by mi and ai. arXiv preprint arXiv:2403.02467, 2024.
Azriel, D., Brown, L. D., Sklar, M., Berk, R., Buja, A., and Zhao, L. Semi-supervised linear regression. Journal of the American Statistical Association, 117(540):2238- 2022.
Danescu- Niczles C., Sudhof, M., Jurafsky, D., Leskovec, J., and Potts, C. computational approach politeness with application to social factors. arXiv preprini
2013.
Berk, R., Buja, A., Brown, L., George, E., Kuchibhotla, A., Su, W., and Zhao, L. Assumption lean regression. The American Statistician, pp. 1-23, 2019. doi: 10.1080/ 00031305 100113001009 305.2019.1552785
Deng, S., Ning, Y., Zhao, J., and Zhang, H. Optimal and safe estimation for high-dimensional semi-supervised learn- ing. Journal of the American Statistical Association, 119 (548):2748 001222782-799,,, 2024.
Bickel, P. J., Klaassen, J., Ritov, Y., and Wellner, J. A. Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins University Press Baltimore, 1993.
Du Plessis, M. C. and Sugiyama, M. Semi-supervised learn- ing of class balance under class-prior change by distribu- tion matching. Neural Networks, 50:1 10-1199 2014.
10

---

Efficient Inference by Incorporating ACPs
Elliott M. R. Surrogate endpoints in clinical trials. An- nual Review of Statistics and its Application, 10(1):75-96, 2023.
Imbens, G., Kallus, N., Mao, X., and Wang, Y. Long-term causal inference under persistent confounding via data combination. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. qkae095, 2024.
Fisher, A. and Kennedy, E. H. Visually communicating and teaching intuition for influence functions. The American Statistician, 75(2):162-177 2021.
Iyer, A., Nath, S., and Sarawagi. S. Maximum mean discrep- ancy for class ratio estimation: Convergence bounds and kernel selection. In International Conference on Machine Learning, pp. 530-538. PMLR, 2014.
Gao, C., Gilbert, P. B., and Han, L. On the role of surrogates in conformal inference individual causal effects. arXiv preprint arXiv:2412.12365 2024.
Ji, W. Lei, L., and Zrnic, T. Predictions as surrogates: Revisiting surrogate outcomes in the age of ai. arXiv preprint arXiv:2501.0973) 2025.
Garg, S., Wu, Y., Balakrishnan, S., and Lipton, Z. A uni- fied view of label shift estimation. Advances in Neural Information Processing Systems, 13:3290-33003 2020.
Kallus, N. and Mao, X. On the role of surrogates in the efficient estimation of treatment effects with limited out- come data. Journal of the Royal Statistical Society Series Statistical Methodology, pp- qkae099, 2024.
Gligorić, K., Zrncc, T., Lee, C., Candès, E. J., and Jurafsky, D. Can unconfident IIm annotations be used for confident conclusions? arXiv preprint ar arXiv:2408. 15204, 2024.
Kelly, S. J. and Ismail, M. Stress and type 2 diabetes: review of how stress contributes to the development of type 2 diabetes. Annual Review of Public Health, 36(1): 441 462, 2015.
Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borg- wardt, K., and Schölkopf, B. Covariate shift by kernel mean matching. Dataset Shift Machine Learning, 3(4): 5, 2009.
Kennedy, E. H. Semiparametric doubly robust targeted dou- ble machine learning: a review. Handbook of Statistical Methods for Precision Medicine, pp. 207-236, 2024.
Gronsbell, J., Gao, J., Shi, Y., McCaw, Z. R., and Cheng, D. Another look at inference after prediction. arXiv preprint 0rXiv:2411.19900 2024.
Kim, H., Zhang, X., Zhao, J., and Tian, Q. Retasa: A non- parametric functional estimation approach for addressing continuous target shift. In The Twelfth International Con- ference on Learning Representations, 2024.
Guo, H., Wang, B., and Yi, G. Label correction of crowd- sourced noisy annotations with an instance-dependen noise transition model. In Advances in Neural Informa- fion Processing Systems, 2024.
Kostopoulos, G., Karlos, S., Kotsiantis, S., and Ragos, O. Semi- suneisuper regression: recent review. Journal of Intelligent & Fuzzy Systems, 35(2): 2018.
Gupta, S., Kohavi, R., Tang, D., Xu, Y., Andersen, R., Bak- shy, E., Cardin, N.. Chandran, S., Chen, N., Cocy, D., et al. Top challenges from the first practical online con- trolled experiments summit. ACM SIGKDD Explorations Newsletter, 21(1):00-35, 2019.
Kouw, W. M. and Loog, M. A review of domain adaptation without target labels. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 43(3):766-785, 2021. doi: 10. (009//AAM..009..99959599
Hines, O., Dukes, O., Diaz-Ordaz, K., and Vansteelandt, S. Demystifying statistical learning based efficient influence functions. The American Statistician, 76(3): 292-304, 2022.
Kpotufe, S. and Martinet, G. Marginal singularity and the benefits of labels in covariate-shift. The Annals of Statistics, 49(6))
2021.
Hou, J.. Guo, Z., and Cai, T. Surrogate assisted semi- supervised inference for high dimensional risk prediction. Journal of Machine Learning Research, 24(265): 2023.
Lawrence, N. and Schölkopf, B. Estimating kernel Fisher discriminant in the presence of label noise. In Interna- tional Conference Machine Learning, 2001.
Huang, J., Gretton, A., Borgwardt, K., Schölkopf, B., and Smola, A. Correcting sample selection bias by unlabeled data. Advances in Neural Information Processing Sys-
Lee, S.-h., Ma, Y., and Zhao, J. Doubly flexible estimation under label shift. Journal of the American Statistical Association, 120(549):278-290 2025.
tems, 19, 2006.
Li, P., Spector, E., Alkhuzam, K., Patel, R., Donahoo, W. T., Bost, S., Lyu, T., Wu, Y., Hogan, W., Prosperi, M., et al. Developing an automated algorithm for identification of children and adolescents with diabetes using electronic
Ichimura, H. and Newey, W. K. The influence function of semiparametric estimators. Quantitative Economics, 13 (1):29-61, 2022.

---

Efficient Inference by Incorporating ACPs
health records from the oneftorida+ clinical research net- work. Diabetes, Obesity and Metabolism, 27(1):102-110, 2025.
Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E.,, Zhang. K., and Mooij, J. On causal and anticausal learning. In 29th International Conference Machine Learning (ICML 2012), pp. 1255-1262. Omnipress, 2012.
X., Liu, T., Han, B., Niu, G., and Sugiyama, M. Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning. 2021.
Scott, C. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In International Conference on Artificial Intelligence and Statistics, 2015.
Lipton, Z., Wang, Y.-X., and Smola, A. Detecting and correcting for label shift with black box predictors. In International Conference Machine Learning, pp. 3122- 3130. PMLR, 2018
Shimodaira, H. Improving predictive inference under covari- ate shift by weighting the log-likelihood function. Jour- nal of Statistical Planning and Inference, 90(2):2272244, 2000.
Liu, Y., Cheng, H., and Zhang, K. Identifiability of label noise transition matrix. In International Conference on Machine Learning, 2023.
Song, S., Lin, Y., and Zhou, A general m-estimation
theory in semi-supervised framework. Journal of the American Statistical Association, pp. 1-11, 2023.
Livne, I,, Azriel, D., and Goldberg. Y. Improved estimators for semi-supervised high-dimensional regression model. Electronic Journal of Statistics, 16(2):547755487, 2022.
Stark Casagrande, S. and Cowie, C. C. Health insurance coverage among people with and without diabetes the adult population. Diabetes Care, 35(11))2243-224 2012.
Miao, J., Miao, X., Wu, Y., Zhao, J., and Lu, Q. Assumption- lean and data-adaptive post-pre prediccteiioon inference. arXiv preprint (rXi:22311.14220 2023.
Storkey, A. When training and test sets are different: char- acterizing learning transfer. Dataset Shift in Machine Learning, 30:3-28, 2009.
Miao, J., Wu, Y., Sun, Z., Miao, X., Lu, T., Zhao, J., and Lu, Q. Valid inference for machine sourning-assisten gedome-wide association studies. Nature Genetics, 56 (11):2361-2369, 2024.
Sugiyama, M. and Kawanabe, M. Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation. MIT press, 2012.
Motwani, K. and Witten, Revisiting inference after prediction. Journal of Machine Learning Research, 24 (394):1-18, 2023.
Sugiyama, M., Suzuki, T., Nakajima, S., Kashima, H., von Bünau, P., and Kawanabe, M. Direct importance estima- tion for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4))699-766, 2008
Nguyen, D.. Christoffel, M., and Sugiyama, M. Contin- uous target shift adaptation in supervised learning. In Asian Conference Machine Learning, pp. 285-300 PMLR, 2016.
Tasche D. Fisher consistency for prior probability shift. The Journal of Machine Learning Research, 18( (8(1):3338- 3369, 2017
Prentice, R. L. Surrogate endpoints in clinical trials: defini- tion and operational criteria. Statistics in Medicine, 8(4): 431-440 1989.
Tian, Q., Zhang, X., and Zhao, J. Elsa: Efficient label shift adaptation through the lens of semiparametric models. International Conference on Machine Learning, pp. 34120-34142. PMLR, 2023.
J., Sugiyama, M., Schwaighofer, A.,
and Lawrence, N. D. Dataset Shift in Machine Learning. MIT Press, 2008
Tran, A., Bibaut, A., and Kallus, N. Inferring the long-term causal effeets of long-term treatments from short-term experiments. arXiv preprint arX1v:2311.08527, 2023.
Rigollet. P. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research, 8, 05 2006.
Tsiatis, A. A. Semiparametric Theory and Missing Data. New York: Springer, 2006.
Rodemann, J., Jansen, C., Schollmeyer, G., and Augustin, T. In all likelihoods: Robust selection of pseudo-labeled data. In International Symposium on Imprecise Proba- bility: Theories and Applications, pp. A12-425. PMLR, 2023.
Van der Laan, M. J. and Rose, S. Targeted Learning: Causal Inference for Observational and Experimental Data, vol- ume Springer, 2011.
12

---

Efficient Inference by Incorporating ACPs
Van der Laan, M. J., Polley, E. C.,, and Hubbard, A. E. Super learner. Statistical Applications in Genetics and Molecular Biology, 6(1), 2007.
van der Vaart, A. W. Asymptotic Statistics. Cambridge University Press, 1998.
Wager, S. and Walther, G. Adaptive concentration of re- gression trees, with application to random forests. arXiv preprint arXiv: /50: 06388, 2015.
Wang. S., McCormick, T. H., and Leek, J. T. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences, 11 30266-302266307755 2020.
Wang, Y., Chen, H., Fan, Y., Sun, W. Tao, R., Hou, W., Wang, R., Yang, L., Zhou, Z., Guo. L.-Z., et al. Usb: A unified semi-supervised learning benchmark for clas- sification. Advances in Neural Information Processing Systems, 35:3938-3961 2022.
Wasserman, L. and Lafferty, J. D. Statistical analysis of semi-supervised regression. In Platt, J. C.,, Koller, D., Singer, Y., and Roweis, S. (eds.), Advances in Neural Information Processing Systems 20, pp. 801-805 Curran Associates, Inc., 2008.
Yi, G. Y. Statistical Analysis with Measurement Error or Misclassification: Strategy, Method and Application. Springer, 2017.
Yi, G. Y. Likelihood methods with measurement error and misclassification. In Handbook of measurement error models, pp. 126. Chapman and Hall/CRC, 2021.
Zhang, A., Brown, L. D., and Cai, T. T. Semi-supervised inference: General theory and estimation of means. Ann. Statist., 4765)2253-2556, 10 2019. doi: 10.1214/ 18- A8-1551756.
Zhang, K., Schölkopf, B., Muandet, K., and Wang, Z. Do- main adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819- 827. PMLR, 2013.
Zhang, V., Zhao, M., Le, A., and Kallus, N. Evaluating the surrogate index as decision-making tool using 200 a/b tests at netflix. arXiv preprint arXiv:2311.11922 2023.
Zhang, Y. and Bradic, J. High-dimensional semi-supervised learning: in search of optimal inference of the mean. Biometrika, 109(2):387-403, 2022.
Zhu, J. Semi-supervised learning literature survey. Tech- nical report, University of Wisconsin-Madiin Depart- ment of Computer Sciences, 2005.
13

---

Efficient Inference by Incorporating ACPs
A. Technical Proofs.
Proof of Proposition 3.1. In the situation with incorporating ACPs, the i.idd data are {ra 1, Mr, X D.J Ure = 0, X k.).
2 1.... M, and the likelihood function of one generic observation is
{ply x, pp x)p(x))]"(pp (p(0 x)q(x) -)} x (())xx(xx"""((""""""""""
ply x, "7'pp x)p(()"((x)"">>>""""""""""""""""
T)--
then consider the Hilbert space of all d-dimensional zero-mean measurable functions with finite variance, equipped with the inner product (h1, h2) E{h h2())) where h1(-), h2(+) H. We first give an orthogonal decomposition of the semiparametric tangent space T (Bickel et al., 1993; Tsiatis, 2006) that is defined as the mean squared closure of the langent spaces of parametric submodels spanned by the score vectors. That is,
T Ar Ap B Ag Ags
where
Ag { 1-r
1- a Va
Ap = (rb(x) [p(((X)) = 0],
Ag [[1 -))((x) [c(c(()] 0].
Ag [d() x): :E x) |x] 0
Ag (re(y.x.)) 7) ((eeYY xx,0) x. 7) 0].
the tangent spaces with respect to p(x), (x). p(ū x) and ply x, 7), respectively. The notation represents the direct sum of two spaces that are orthogonal to each other.
Recognizing that the EIF for estimating ,3 is a special element in T, we can assume it has the form
rei (v, +
(1 (x)
EA,
EΛg
EA,
Define the score vectors as
alog ply x, TJ: an)
si (y, 0;)))
da
ölog p(ū x; or2)
S2 ([)))
dog
ölog q(x; ocs)
s3(x;as)
8o3
Then, based on the orthogonality satisfied by the EIF (Theorem 4.2 and Theorem 4.3 in Tsiatis (2006)), we have
From E(ss] R E(Rc,s] sT) one can derive that e; 2 (x)() mo (x, f)}]
From E(ss]]] R 0) E(d
E -
--
Sssp R ). one can derive that
no (x) {mo 3) - mo())]
d =
From E(ss]] R 0) EE(1 R)cis?), one can derive e₁ 5 - mo(x)
and this completes the proof.
14

---

Efficient Inference by Incorporating ACPs
Proof of Proposition 3.3.
The asymptotic variance without ACP is
1 q
S2E 0
Wa (X) mx)) x s2
+
Bg (Slmo(X))2g
1
By wo ( - - (X. 8))001
+ π
1 Eg
(xo(X))(mmoXX 8) - mo -mo (x))
+ Eg {Smmol X {
1 Eg
wol (X))))(ss(())))) - m. (x. 8) 000 12
Eg { { (X)mm())02)) ())
+
π
1 1
{1-me -o
+
0 fm (X, () mo (X))
To (X)
(1-
The third equation uses wx(x) 1-x mo(x)
1x
The asymptotic variance with ACP is
ZE ( 22
=
1
Eg
wol (X)s)(s(YY)):))))))))))))) - mo (), 8))82(? 22 + - Eg {semo (X) (2)
wil X
2
E
(0 8) -m m
+ (1 T T)wb
1 Eg
wo (X)))(s(Y,X;)))))))))) -mo 8) 82 22
By { {Semo (X) (
{
1 To [xx0ii 2
E
(1 (1-x) x) (X 8) m. m x)))(nn
The second equation uses
l- 1-
5+(1 - 13 (x)]
Therefore, we have
1
ΩE {1 (x)33 (mo(X, f) - mx) Ω
n ((B(d E(d 1152
(
(1 - T)²
To (X)
Proof of Theorem 5.3. When wx(x) is misspecified, and in* fx, 7)
and (x) are correctly specified. We have
R u
w* (X)
1 R mn
(X)(s(Y(X: m" (X, p)} (1 - n ) (X)
(X, P) mmm (X)) +
(X)
+ π
1 - π
R π
w" (X)
mm [ Q) ma (X)} + 1- R
= w² (X)(ssY,
m. x, p)} (1- x)w* (X) + π
(X)
1 π
When w² (x) is correctly specified, and m" (x, 7 and m (x) are misspecified. We have
R
(X)
1 R
w (X) - m (X, 8)}
(x, 8) mn (X))
+
m (x)
π
(1 )w (X) +
π
wol (X)
1 R
=
Wi ()(X))s(XXXx;))) - m' 8)}
+E
π r)wpp
X) + ग
[m* "x'(X,) 8) m mX)]
+
1 -
m (X)
R
1 R
=
Wo
(X) (s{, - ,x
+
{m" 2) - mm mX))
+
m (X)
π
1 -
R m (X, 8) =0
=
Wi [X (s() - MM (X (x,8))
+
The proof is completed.
15

---

Efficient Inference by Incorporating ACPs
Proof of TTheormm 5.5.
The efficient influence function is
wi (x)
( 1) - mo(x)} + 1 rr
(po, Bo) = [x(x))((u,xx;))) - roo (x, 07) + (1 woo (x) +
1 mo (x).
Then have
T (x)
(mx(x,)) 77) -mx(x)) + 1-
Pe (Pk:
({,, (x Ø)1
1 m (x).
(1 π uu (x) + n
Therefore,
1 K
Σ (Pk. (po
K Σ
| Bg) (po,
We only consider the following term,
Eg v (Px, Bo) - 30)
= (E. - ( (Pn. Bo) - (o) - (o)) (Es - (po, Bo)
sample splitting
bias
CLT
(i (Eg - () (2, ( (po. (M
Proof. Let (Pk: be function estimated from the sample DE. and let B& denote the empirical measure over Dx. First note that, conditional on Di. the term in question has mean zero since
E[E 1 (P,, 9o) (p. [o] = E (Px. 3o) (Po
The conditional variance is
var (Ex - B) Bu) - Φ (pu, Bo).B = var EE ( (pk, (pu. Bo
2 v (Pn, Bo) (eo 3o)l D3} < 21 ( Bo) ( 30)
M
Therefore using Chebyshev's inequality we have
[ (8 (Pk, Bo) (0 B1)
pr
/ の (Tu, Bo) (p0 B 212/ / M
[Ex E) (Px, Bu) & (po, (po, Bo)
1
1
E pr
v2 (Pc, 3,) (Po, 2 / 2 H
12
Thus for any > 0 we pick 1/VE so that the probability above is more than E, which yields the result.
(ii) (Px. (po. wo(x)]lallmid mo(x)]12) ap (M-) by rate conditions.
Op(1)x((x) wo(x)) 12 (x. ?)
m 0112) + Op(1lūx(x))
Proof.
E( . (pk, Bb) & (po, Bb)
E
Wh {s( Y. x; Bb) - m. (X, γ)} - (x)())s((,xx - 8)}} IDj
Wk (X)
wy X
+E
(1 T) (X) + {mg XX, P) - (x))
Mo (X, Ý) m. mx)))
(1 - T 7 X +
1 1 R
+E
- (x)
1 - mo (X)
(I)) +0)+(()) (III)
16

---

Efficient Inference by Incorporating ACPs
For the first term (I) have
R
R wo (Mo, Y) -mx(x, 8)
(1)
E
Wy
(X)
wo (XX (X,
F)H
+
R
+E
[ () wo( {mo [X, Y)
m& X DHDDE
π
R
R
E
wil
(X)
mig
X Y)
m. (x, D)
+
[wxx (X)
wo(X) (x (x,
For the second term (II), we have
wk (X)
(o(X)
(I)
E
[X, f) (x F)} (m, - mo mo
((X))))
(1 T) R (
+
(1 T) ww!! (X) +
Wk
wo (X)
+E
(mn(X, P) - mo mo (X))
(1 - π Diik X)
- π wwo (X)
+
Wo (X)
+E
mX P)) {, mo (X){)\|
(1 π) 2w (X)
WK (X)
wo X
= E
-1- wo X + Kmmx(X, 9) - mo (x, 8)} {mn (X) - mo mo))]]
(1
Wü (X)
+ .
Wo (X)
+E
[[mx(( (X, 8) mX B)) - (x) mo (x))II))))
[1 wo (X) +
For the third term (III), we have
1 R [mg (X) mx
(III E
1 π
Combine (I), (II) and (III), obtain
(1) + (II) + (II) = (1 wo (x) (m) 0) (11) + Op - wo(x)l2l|mxx(2 - mo
= (M1
The second equation by the rate conditions.
Therefore, we have
B(P) (o(p))= n
Σ
B. (Pk.
1. = -2
Σ
Bo) - 52E (Po, (po,Bo)
K k=1
K
k=1
Ω
1
EB B) Bb) o (po, +52 Σ
El (px, B) & (pos Bo)3 +
1
(Es () Bb)
K
K
sample splitting
Ω
BB
B)Φ (Po, 3o) (M-
K
CLT
Based on central limit theorem, We have
VM(3(() Bo(po)}
VMΩ
1
Σ
(Bs E) + Bo) (1)
K
NO.
The proof is completed.
17

---

Efficient Inference by Incorporating ACPs
Proof of Theorem 5.8. In this section, we divide it into two parts. First, we prove the consistency of the covariance matrix, and then prove the confidence interval.
First, we need show the SVV... is consistent. Here, we only show V.. that
Σ K k=
Bg ( is consistent, Note
Vall2 = VV -E( 1 (po: B,)
K
5 B)} - B 1r (po: B,)
K
k=1
We only need to prove that [Ex[o (Px: })} Bio (po: Bo)) (1). Consider the following decomposition:
B (0 B)
s 1 B)} Ex (po; (o + [Bx (p): E(o (po: Bo)lll
= R1 + R2.
Thus we only need to prove that both R₁ and R2 are op(1).
By the law of large numbers, we can easily obtain R2 = (1() Next we analyze the term Ri.
[Bu (px; ()) - En 10 02 (po: Bo)bll2
< Ex th (ix; (3 (po: B 802 ||2 2| (ow - (po How Bo)
s R2/2 x { ( 122/2 + 20
$2 (0o: B 30111/22 /
where Rg = |E { v (Px: 3) u
Since E0 Bp) (1), Markov inequality implies that (p = Op((). Moreover,
Ra s 2C Bo1l2 E (Pxi Bo) ow (pu: Bo) 10)1
Since ||3 B (1) by Theorem 5.3, thus we only prove [ (Pui 3p) o (po: (po: = (1( as well. We can further decompose this term:
Bal (p))Bo) (po: 02
Exl (Px: (o) (p(i Bb)) 02 E(d (p:i(Px @I(p: B,)
+B(0i (Pxs BB) - obi (po: },) (1) (2).
Note that E((1)) = 0, so by Markov inequality, we have (1) = (1). Moreover, it is easy to verify that (2) = (p(1) as (1 (x) wo (x) f) 0) mx 7)1 + Op (x) wo( mx(x) mo(x)]l2 op (1).
Putting all above together, we have
RR = =
Thus, we have V1 V1 o1). Further, we have V., V., + (1) by ñ Ω + (1). Second, we will show that v B, E CIg with probability 1 - a in the limit; that is,
lim pr (v't 3o E CI,) = 1 aa.
M
18

---

Efficient Inference by Incorporating ACPs
It can be easy to obtain, since we have
lim pr (v 30 Clas)
M0000
v" $ 11 (1-aa/2) S.v//M sv"p, S Sv33+00 + + (1-022/) V vT ..VVv..V/MM
lim pr M-x
VM -vT 5 a/2)
lim pr
-1- -1-1-001 (111)
M 100
VV Siv
B - vTB)
VM (M(vTB Bn -v < - <1 - a/2)
lim
< < (1-a/2)
a/2)
lim M-+06
M 000
VVTVVV.. Stv
VVrSVV...
(1-a1(1- 1 1 al2222a/2211
-8(--41----- $ 10-((11-11 (1-222)
$1-
The fourth equation holds because v VM B, - - a siv 4 (0, 1). The fifth equation holds because the standard normal distribution is symmetric. The proof is completed.
vTVV
B. Parallel Results for Parameter 0 in the Combined Population
To avoid repetition, we only present the results for the parameter 3, some characteristic in the unlabeled data population U. in the main paper. But, all the results can be generalized to the characteristic of the combind data population LUU if is of interest.
Similarly, we define the d-dimensional parameter 0 as
argmin B(((y, x; 0)),
and it equivalent to write 0 as the solution of the estimating equation
E E(u(Y,X;0)) O.
To proceed, we assume the d x d matrix B(,u(Y, X; 0)/00 evaluated at the true value On is invertible and denote the inverse r. We also denote B(u(Y, 0) x} h(x) and denote x(((Y,x;0)) xx 77) h(x,ū)
For estimating 0, in the situation with ACPs, the EIF equals I' with
9 xx(x) (u(y, x; 0o) nu(x, D) (x.
and the efficiency bound is rV, with
E To
Y, X; 0o) - xx P))
+ - +B (ho())))
- ho
In the situation without ACPs, the EIF is T wo with
(u(y,x;00) 0o) - -h(x))} + hx(x),
@ wa xx(x)
and the efficiency bound is rV T with
E No
(Y 0o) - (x 8)
+E
R
(x, ho(x)) + (x)
apo
To
Therefore, one can compute that the efficiency gain of using ACP y is
T(V... wo V...r
rE
R
}
(ho((, hx r
To
(X) Tu (X)
TE
(To 8) - (x)
r,
19

---

Efficient Inference by Incorporating ACPs
which is positive definite, long as (x) is bounded away from zero and and that the ACP Y does depend some other variable Z, beyond the available feature X in both labeled and unlabeled data. Clearly, this whole rationale the same as that for the parameter B, as well as the following theoretical results including double robustness and semiparametric efficiency.
C. Additional Numerical Results
In this section, we provide more numerical results for various parameters of interest across different models. For each parameter, demonstrate the efficiency improvement by incorporating ACPs under varying conditions, including varying labeled sample size (n), varying unlabeled sample size (N), varying signal strength from ACP (a), and varying correlation between ACP and the covariates (().
Figures 4-5 present the results for various parameters under the linear model, while Figures 6-8 illustrate the results for the logistic model. Similarly, Tables 5-8 and Tables 9-12 summarize the results for various parameters under the linear and logistic models, respectively.
In addition, we also present additional numerical results and comparisons with benchmark methods-PPI, PPI++, and RePPI-across various parameters of interest under different modeling scenarios. For each parameter, evaluate the efficiency gains achieved by incorporating ACP under range of conditions, including varying the labeled sample size (n), the unlabeled sample size (N), the signal strength of the ACP (a), and the correlation between the ACP and covariates (ς). Table 13 reports the comparison results for estimating the outcome mean under linear models, while Table 14 presents the corresponding results for coefficient estimation under linear models.
7
Method
MSB(Y)
MSE((1)
MSE(C2)
MSB(Ea)
MSE((A)
MSE(Es)
300
w.o. ACP
0.84
0.82
0.86
1.00
0.93
0.93
300
ACP
0.08
0.10
0.08
0.10
0.09
0.10
600
W.O. ACP
0.83
0.80
0.80
0.97
0.92
0.90
600
W. ACP
0.06
0.06
0.06
0.07
0.06
0.06
900
ACP
0.82
0.80
0.79
0.95
0.93
0.90
900
W. ACP
0.05
0.05
0.05
0.06
0.05
0.06
1200
W.O. ACP
0.81
0.79
0.79
0.94
0.94
0.90
1200
ACP
0.05
0.05
0.04
0.05
0.05
0.05
1500
W.O. ACP
0.81
0.79
0.78
0.93
0.92
0.91
1500
W. ACP
0.05
0.04
0.04
0.05
0.04
0.04
Table 5. Simulation results under different when Ç 0, N 330 under the linear model setting.
20

---

Efficient Inference by Incorporating ACPs
300, N changes
N changes
15-
-
ARE
0m2
0:5
300
600
900
1200
1500
300
900
1200
1500
Sample size
300, changes
300, changes
<
2=0.6
[=0
ARE
C=06
[-0.0
C <-0.3
C-0.9
<00.6
[ =0.9
300
600
900
1200
1500
300
600
000
1200
3500
Sample
Figure Variation of ARE for the estimation & under linear model setting different sample sizes, signal strength, correlation cnefficient.
21

---

Efficient Inference by Incorporating ACPs
300, N changes
N changes
20
-
ARE
0m2
0:5
300
600
900
1200
1500
300
600
900
1200
1500
Sample size
300, changes
300, changes
C C-0.5
15-
0-06
[=0
ARE
[=06
* [-0.
5-0.9
=0.3
00.6
[ -0.9
300
600
900
1200
1500
300
600
000
1200
3500
Sample
Figure Variation of ARE for the estimation under linear model setting different sample sizes, signal strength, correlation cnefficient.
22

---

Efficient Inference by Incorporating ACPs
300, changes
N 300, changes
25
-
ARE
0.2
0:5
1.5
1.0
300
600
900
1200
1500
300
600
900
1200
1500
Sample size
300, changes
changes
C-0.6
<0033
[=0
ARE
C-0
C=06
C -0.3
> (-0.
=0.66
1.5
2=0.9
[-0.9
1.0
300
600
900
1200
1500
300
600
900
1200
1500
Sample
Figure Variation for estimation E, (Y) under logistic model setting different sample sizes, signal strength, correlation coefficient.
23

---

Efficient Inference by Incorporating ACPs
300, changes
N 300, changes
-
ARE
2.0
0m2
0:5
1.5
Km
1.0
300
600
900
1200
1500
300
600
900
1200
1500
Sample size
300, changes
changes
<=0
[=0
ARE
C=06
* [-0.
C-0.9
C=0.0
300
600
900
1200
1500
300
600
900
1200
1500
Sample
Figure Variation of ARE estimation under logistic model setting different sample sizes, signal strength, correlation cnefficient.
24

---

Efficient Inference by Incorporating ACPs
300, changes
N 300, changes
3.0-
2.5
-
ARE
0m2
2.0
0:5
1.5
1.0
m0
300
600
900
1200
1500
300
600
900
1200
1500
Sample size
300, changes
changes
3.0
m0.6
25
[=0
ARE
[=06
* [-0.
<-0.3
<-0.9
--066
(-0.9
300
600
900
1200
1500
300
600
900
1200
1500
Sample
Figure Variation ARE estimation E under logistic model setting different sample sizes. signal strength, correlation cnefficient.
25

---

Efficient Inference by Incorporating ACPs
N
Method
MSE(Y)
MSE((1)
MSE(E2)
MSB((a)
MSE((1)
MSE((s)
300
ACP
0.84
0.82
0.86
1.00
0.93
0.93
300
W. ACP
0.08
0.10
0.08
0.10
0.09
0.10
600
W.O. ACP
0.42
0.55
0.45
0.57
0.52
0.54
600
ACP
0.07
0.07
0.07
0.07
0.07
0.08
900
W.O. ACP
0.29
0.38
0.36
0.43
0.39
0.33
900
W. ACP
0.05
0.06
0.06
0.06
0.06
0.06
1200
W.O. ACP
0.24
0.30
0.29
0.32
0.31
0.25
1200
W. ACP
0.05
0.06
0.05
0.05
0.06
0.06
1500
W.O. ACP
0.21
0.26
0.27
0.29
0.27
0.25
1500
W. ACP
0.04
0.05
0.05
0.05
0.05
0.05
Table 6. Simulation results under different when 5,
0,
300 under the lincar model setting.
Q
Method
MSE(Y)
MSE(E)
MSE((2)
MSE((s)
MSE(E.)
MSE (5s)
0
w.O. ACP
0.03
0.03
0.03
0.04
0.03
0.03
0
ACP
0.03
0.03
0.03
0.04
0.03
0.03
1
w.O. ACP
0.06
0.07
0.06
0.07
0.07
0.07
1
W. ACP
0.03
0.03
0.03
0.04
0.04
0.03
2
W.O. ACP
0.16
0.16
0.15
0.18
0.17
0.17
2
ACP
0.04
0.04
0.04
0.05
0.04
0.04
3
W.O. ACP
0.32
0.32
0.32
0.37
0.35
0.34
3
ACP
0.05
0.06
0.05
0.06
0.05
0.06
4
W.O. ACP
0.54
0.54
0.56
0.65
0.60
0.59
4
W. ACP
0.06
0.07
0.06
0.08
0.07
0,08
5
W.O. ACP
0.84
0.82
0.86
1.00
0.93
0.93
5
ACP
0.08
0.10
0.08
0.10
0.09
0.10
Table Simulation results under different a when Ç 0 and 300 under linear model setting.
26

---

Efficient Inference by Incorporating ACPs
Method
MSE(Y)
MSE((1)
MSE(E2)
MSE(C3)
MSE((A)
MSE((s)
0
ACP
0.84
0.82
0.86
1.00
0.93
0.93
ACP
0.08
0.10
0.08
0.10
0.09
0.10
0.3
W.o. ACP
0.72
0.70
0.79
0.64
0.70
0.63
0.3
ACP
0.09
0.09
0.09
0.08
0.09
0.09
0.6
W.O. ACP
0.51
0.52
0.61
0.40
0.50
0.49
0.6
W. ACP
0.10
0.07
0.07
0.06
0.07
0.07
0.9
W.O. ACP
0.25
0.15
0.17
0.16
0.16
0.14
0.9
ACP
0.11
0.04
0.05
0.05
0.04
0,04
1
W.O. ACP
0.12
0.03
0.04
0.02
0.03
0.03
1
ACP
0.12
0.03
0.04
0.02
0.03
0.03
Table Simulation results under different ( when and
300 under the linear model setting.
n
Method
MSE(Y)
MSE((1)
MSE 22)
MSB
MSE((1)
MSE (Es)
300
w.o. ACP
0.01
0.24
0.24
0.17
0.23
0.21
300
ACP
0.00
0.12
0.11
0.09
0.10
0.10
600
W.O. ACP
0.01
0.35
0.34
0.25
0.25
0.30
600
W. ACP
0.00
0.13
0.14
0.10
0.10
0.12
900
W.O. ACP
0.01
0.32
0.31
0.25
0.25
0.30
900
W. ACP
0.00
0.13
0.14
0.11
0.10
0.12
1200
W.O. ACP
0.01
0.39
0.44
0.28
0.28
0.37
1200
W. ACP
0.00
0.14
0.15
0.11
0.11
0.13
1500
W.O. ACP
0.01
0.41
0.48
0.30
0.28
0.35
1500
W. ACP
0.00
0.14
0.14
0.11
0.11
0.13
Table 9. Simulation results under different n when 5, < 0, N
300 under the logistic model setting.
N
Method
MSE(Y)
MSE(G)
MSE(E2)
MSB(Es)
MSE((.)
MSE((s)
300
w.o. ACP
0.01
0.24
0.24
0.17
0.23
0.21
300
w. ACP
0.00
0.12
0.11
0.09
0.10
0.10
600
W.O. ACP
0.00
0.10
0.11
0.08
0.10
0.09
600
W. ACP
0.00
0.06
0.06
0.05
0.06
0.06
900
W.O. ACP
0.00
0.07
0.07
0.06
0.07
0.06
900
W. ACP
0.00
0.04
0.04
0.03
0.04
0.04
1200
W.O. ACP
0.00
0.05
0.05
0.05
0.05
0.04
1200
W. ACP
0.00
0.03
0.03
0.03
0.03
0.03
1500
W.O. ACP
0.00
0.04
0.04
0.04
0.04
0.03
1500
ACP
0.00
0.02
0.02
0.02
0.02
0.02
Table Simulation results under different N when a = 5,
n
300 under logistic model setting.
27

---

Efficient Inference by Incorporating ACPs
a
Method
MSE(Y)
(SE(E)
MSB((2)
MSE (C3)
MSE((a)
MSE((a)
0
ACP
0.00
0.12
0.10
0.12
0.09
0.11
0
W. ACP
0.00
0.12
0.10
0.12
0.09
0.11
1
W.O. ACP
0.00
0.12
0.11
0.14
0.12
0.10
1
ACP
0.00
0.12
0.11
0.13
0.11
0.10
2
W.O. ACP
0.00
0.18
0.17
0.16
0.17
0.17
2
W. ACP
0.00
0.13
0.14
0.13
0.12
0.12
3
W.O. ACP
0.00
0.21
0.19
0.17
0.20
0.20
3
ACP
0.00
0.14
0.13
0.12
0.12
0.11
4
W.O. ACP
0.01
0.23
0.22
0.16
0.22
0.20
4
ACP
0.00
0.12
0.12
0.10
0.11
0.11
5
ACP
0.01
0.24
0.24
0.17
0.23
0.21
5
W. ACP
0.00
0.12
0.11
0.09
0.10
0.10
Table 11. Simulation results under different a when = 0 and n
N = 300 under the logistic model setting.
2
Method
MSE(Y)
MSE(C)
MSE(E2)
MSE((s)
MSE((L)
MSE((s)
w.o. ACP
0.01
0.24
0.24
0.17
0.23
0.21
0
w. ACP
0.00
0.12
0.11
0.09
0.10
0.10
0.3
w.o. ACP
0.00
0.25
0.23
0.19
0.19
0.22
0.3
ACP
0.00
0.11
0.11
0.09
0.09
0.10
0.6
W.O. ACP
0.00
0.25
0.21
0.19
0.22
0.20
0.6
W. ACP
0.00
0.13
0.10
0.10
0.10
0.09
0.9
ACP
0.00
0.70
0.15
0.13
0.14
0.12
0.9
W. ACP
0.00
0.65
0.11
0.12
0.13
0.11
1
W.O. ACP
0.00
7.38
0.15
0.17
0.17
0.16
1
ACP
0.00
7.11
0.15
0.16
0.17
0.16
Table 12, Simulation results under different Ç when a
5 n
300 under the logistic model setting.
28

---

Efficient Inference by Incorporating ACPs
Table 13. Comparison of mean squared error (MSE) for estimating outcome mean between proposed method and three alternatives: PPI. PPI++, RePPI, under the linear model setting. Bolded values (smallest across four methods) indicate proposed method performs the best.
Setting
Method
N 300
600
900
1200
1500
PPI
2.8976
2.7768
2.6824
2.8998
2.9021
a
PPI++
2.8689
2.7499
2.6573
2.8496
2.8497
= 0 Ç
RePPI
2.8639
2.7452
2.6528
3.8688
3.9909
n 300
proposed
0.0803
0.0756
0.0712
0.0481
0.0451
Method
n 300
600
900
1200
1500
PPI
2.8976
2.8811
2.8876
2.8897
2.9017
5
PPI++
2.8689
2.8711
2.8829
2.8899
2.9048
Ç 0
RePPI
2.8639
0.9078
0.4444
0.2822
0.1898
N 300
proposed 0.0803
0.0651
0.0545
0.0484
0.0441
Method
0
1
2
3
PPI
2.8273
2.8364
2.8475
2.8588
2.8757
Ç = 0
PPI++
2,8465
2.8470
2.8492
2.8524
2.8593
300
RePPI
0.0583
0.1344
0.4725
1.0425
1.8403
= 300
proposed
0.0327
0.0338
0.0393
0.0492
0.0630
Method
ç= 0
0.3
0.6
0.9
PPT
2.8976
5.7063
9.3864
13.8583
15.6780
a
PPI++ 2.8689
4.4729
6.2252
8.2971
9,0483
n 300
N 300
RePPI
2.8639
2.2030
1.6693
0.5678
0.1476
proposed
0.0803
0.0881
0.0990
0.1107
0.1231
29

---

Efficient Inference by Incorporating ACPs
Table 14. Comparison of mean squared error (MSE) for estimattnn regression coefficients between proposed method and three alternatives: PPI, PPI++, RePPI, under the linear model setting. Bolded values (smallest across all four methods) indicate that the proposed method performs the best.
N
Method
X1
X2
X3
X4
X5
PPI
0.1290
0.1136
0.1233
0.1158
0.1140
PPI++
0.1086
0.1086
0.1178
0.0987
0.1033
300
RePPI
0.2463
0.2741
0.2713
0.2676
0.2642
propoesd
0.0975
0.0844
0.0977
0.0889
0.0989
PPI 0.1208
0.1072
0.1156
0.1102
0.1091
PPI++ 0.1035
0.1029
0.1113
0.0962
0.0994
600
RePPI 0.2381
0.2613
0.2587
0.2542
0.2506
proposed
0.0913
0.0792
0.0934
0.0851
0.0948
PPT 0.1137
0.1015
0.1092
0.1045
0.1033
PPI++ 0.0992
0.0985
0.1067
0.0923
0.0951
900
RePPI
0.2297
0.2498
0.2471
0.2436
0.2401
proposed
0.0867
0.0751
0.0889
0.0813
0.0912
PPT 0.1219
0.1044
0.1166
0.1090
0. 001484
PPI++ 0.1219
0.1044
0.1166
0.1090
0.1084
1200
RePPI
0.3001
0.3396
0.3337
0.3255
0.3133
proposed
0.0493
0.0434
0.0548
0.0481
0.0483
PPI 0.1204
0.1038
0.1158
0.1080
0.1069
PPI++ 0.1204
0.1038
0.1158
0.1080
0.1069
1500
RePPI
0.3088
0.3465
0.3444
0.3370
0.3170
proposed
0.0431
0.0408
0.0486
0.0426
0.0417
30


Reviews:
[{"review_id": "1wgAtcG3vd", "reviewer": "Reviewer_mNmv", "summary": "This paper proposes an approach that leverages both labeled and unlabeled data to estimate target parameters. The approach first uses the pre-trained model to estimate $Y$ for the unlabeled data and then uses the estimated $\\hat{Y}$ to estimate the target parameters. The proposed approach is applied to both simulated and real-world data to demonstrate the efficiency gain as compared to the benchmark approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: However, it is a bit unclear...[from Methods and Evaluation Criteria]*\n\n**Response**: \n- Thanks for raising this question. In general, the outcome $Y$ and covariate $X$ are variables of scientific interest. The variable $Z$ represents additional information that may not be of direct scientific interest but is highly relevant to $Y$.\n- For example, in our diabetes dataset, $Y$ indicates whether a patient has diabetes, and $X$ includes 7 variables measured at the year of diagnosis: age, type of insurance (self-pay vs others), counts of inpatient and outpatients visits, BMI, congestive heart failure (yes vs no), and the Charlson Comorbidity Index (CCI, larger than 2 or not). Our primary scientific interest is in understanding the relationship between $Y$ and $X$, which captures patient characteristics. Meanwhile, ACP $\\hat Y$ is generated by a tree-based algorithm using diagnosis codes, medication history, and HbA1c lab test results from electronic health records---these variables form $Z$.\n- In another example using newly analyzed income data, we fit a least squares model to study the relationship between $Y$ (log-income) and $X$ (age, sex). The ACP $\\hat Y$ is obtained from an XGBoost model predicting log-income based on 14 variables, including education, marital status, citizenship, and race, among others---these variables consititute $Z$.\n- In some cases where $\\hat Y$ is generated by LLMs, defining $Z$ precisely is more challenging. For instance, in our politeness dataset, the goal is to study the relationship between $Y$ (politeness score) and $X$ (a binary indicator of hedging in requests), while $\\hat Y$ is generated by OpenAI's GPT-4o mini model. Here, $Z$ represents some latent information used to produce $\\hat Y$ but is not as explicitly defined as in previous examples. Similarly, in our wine dataset, the goal is to examine the relationship between $Y$ (rating) and $X$ (price and region), with $\\hat Y$ also generated by GPT-4o mini.\n- Finally, in a synthetic data example, consider the following data generating process $Y = \\xi^T X + \\alpha Z + \\varepsilon$, where $\\varepsilon \\sim N(0,1)$, $\\alpha>0$, as in our simulation studies. Suppose the parameter of interest is $\\xi$, and $\\hat Y = Z$. The magnitude of $\\alpha$ determines whether efficiency gain occurs: as long as $\\alpha\\neq 0$, $Z$ correlates with $Y$, leading to an efficiency gain.\n\n*Q2: However, I would appreciate...[from Experimental Designs or Analyses]*\n\n**Response**: \n- Yes. Thank you for this question. We implemented the proposed method on three new data sets, and compared with some existing methods: PPI (Angelopoulos et al. 2023a), PPI++ (Angelopoulos et al. 2023b) and RePPI (Ji et al. 2025).\n- Income data: We analyze the relationship between wage (measured by log-income) and age, confounded by sex, based on US census data, under covariate shift. The ACP $\\hat Y$ is generated by fitting XGBoost of log-income with 14 variables, including education, marital status, citizenship, and race. For covariate shift, we partition the labeled and unlabeled datasets using the probability $\\exp(\\alpha^T X)/\\{1+\\exp(\\alpha^T X)\\}$ where $\\alpha=(0,1,0)$ and $X =(1,X_1,X_2)$ with $X_1$ age and $X_2$ sex, resulting in a ratio of 2:8.\n- Politeness data: Using data that comprises texts from 5,512 online requests posted on Stack Exchange and Wikipedia, we understand the association between politeness score (range from 1 to 25) and a binary indicator for hedging within the request. The ACP $\\hat Y$ is generated using OpenAI's GPT-4o mini-model that has the same range as the politeness score. For covariate shift, we split the labeled and unlabeled data in a 1:9 ratio, following the same procedure as above, where $X=(1,X_1)$ with $X_1$ hedge and $\\alpha=(0,1)$.\n- Wine data: Using the Wine Enthusiast review dataset, we investigate the association between wine rating (range from 80 to 100) and wine price, adjusted by wine region. Similar to the politeness data, the ACP $\\hat Y$ is also generated by employing OpenAI's GPT-4o mini-model that produces predicted ratings with the same scale. To assess covariate shift, we follow the same procedure as in the previous experiments, splitting the labeled and unlabeled data in a 3:7 ratio. Here, $X=(1,X_1,X_2,X_3,X_4,X_5)$, where $X_1$ represents price, $X_2$ to $X_5$ represents California, Washington, Oregon and New York, respectively, with $\\alpha=(0,1,0,0,0,0)$.\n- For each of these three data sets, we present the comparison results of confidence interval length (the shorter, the more efficient, the better) for regression coefficients in Table 1 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing).\nAs you can see, while there are some cases that the proposed method is tiny slightly less efficient than PPI++ or RePPI, it is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction in confidence interval length reaching approximately 60%.", "claims_and_evidence": "The proposed approach is supported by theoretical analysis and empirical study.", "methods_and_evaluation_criteria": "It makes intuitive sense to augment the labeled data with unlabeled data to estimate target parameters. The paper shows in Section 3.2 that the unlabeled data can improve the estimation efficiency of target parameters. However, it is a bit unclear when the efficiency gain can actually happen, i.e., when the generation of $Y$ depends on covariates $X$ and other variables $Z$. Could the authors provide a concrete example? What are $X$, $Z$, and $Y$? When $Z$ is used to generate $Y$ but is not included in $X$ itself?", "theoretical_claims": "The theoretical claims look correct to me.", "experimental_designs_or_analyses": "The experimental results make sense. However, I would appreciate more real-world experiments (on other data sets) to demonstrate that the proposed approach has a substantial efficiency gain.", "relation_to_broader_scientific_literature": "This paper is related to the literature on semi-supervised learning and prediction-powered inference.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Please address my comments on methods and experiments.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JW7mZJkVlM", "reviewer": "Reviewer_6g7C", "summary": "This paper introduces a semiparametric framework for efficient inference under covariate shift by leveraging automated computational phenotypes (ACPs). The authors propose a doubly robust, semiparametrically efficient estimator for a target parameter $\\beta$ by integrating ACPs, density ratios, and conditional expectations.\n\nTheoretical results show that when ACPs offer extra predictive information beyond $\\boldsymbol{X}$, their inclusion strictly reduces the estimator's asymptotic variance. Simulation studies and a diabetes case study empirically validate that the proposed method achieves lower mean squared error and yields narrower confidence intervals compared to methods without ACPs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: [Methods and Evaluation Criteria]*\n\n**Response:**\n- Thank you. We provided a more comprehensive response in our answer to Q1 of reviewer mNmv. Please kindly refer to that section for details.\n\n*Q2: The simulations...[Experimental Designs or Analyses]*\n\n**Response:**\n- Yes. We conducted comparisons between the proposed method and PPI, PPI++, RePPI, across a variety of settings with different values of $n$, $N$, $\\alpha$ and $\\zeta$. The MSE results are presented in Tables 2 and 3 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing). As you can see, across all considered scenarios, the proposed method consistently outperforms the three alternatives.\n\n*Q3: Why SuperLearner...*\n\n**Response:**\n- We use Super Learner in our implementation, which integrates a library of flexible statistical learning tools to ensure consistent estimation of relevant nuisance functions. While these functions can be estimated using simple parametric models, such models are prone to misspecification, potentially introducing bias and reducing efficiency. More flexible machine learning methods mitigate this issue by avoiding strict parametric assumptions and enabling consistent estimation across a broader range of function classes. Instead of relying on a single estimation method, Super Learner leverages the strengths of multiple algorithms by constructing an optimally weighted combination, offering greater robustness and flexibility. Moreover, van der Laan et al. (2007) established a theoretical guarantee that the estimation error of Super Learner converges to that of the best-performing learner in the ensemble.\n- Alternatively, we may use individual learning algorithms, such as random forests or XGBoost. Neural networks are also a potential option, though the size of labeled data in our dataset---such as the diabetes data---is relatively small compared to typical deep learning applications.\n\n*Q4: For real dataset...*\n\n**Response:**\n- Yes we completely agree with you. We provided a more comprehensive response in our answer to Q2 of reviewer mNmv. Please kindly refer to that section for details. Briefly we newly analyze three datasets. For each of them, we present the comparison results of confidence interval length for regression coefficients in Table 1 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing). The proposed method is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction reaching approximately 60\\%.\n\n*Q5: For the diabetes dataset...*\n\n**Response:**\n- Figure 2 in our submitted paper demonstrates that the distribution of inpatient visit count indeed shifts between labeled and unlabeled data. To provide a more comprehensive comparison, Table 4 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing) presents the summary statistics of each variable in $X$. Overall, the covariate shift assumption appears reasonable in this dataset.\n\n*Q6: For the diabetes dataset...*\n\n**Response:**\n- Same as Q1 above.\n\n*Q7: [Supplementary Material]*\n\n**Response:**\n- Thank you for the suggestion, and we apologize for overlooking this issue. In the new version, we will make sure to appropriately reference the corresponding details in the supplementary material within the main paper.\n\n*Q8: [Essential References Not Discussed]*\n\n**Response:**\n- Thank you for pointing out these references! Both are highly exciting works. In the new version, we will be sure to include them and conduct a more thorough literature review, particularly on the topics of covariate shift and pseudo-labeling.\n\n*Q9: [Questions for Authors]*\n\n**Response:**\n- In the diabetes dataset, the scientific goal is to understand the relationship between $Y$ (diabetes status) and $X$ (7 variables representing patient characteristics). Since $Y$ is binary, we use logistic regression, which is arguably the most commonly used model for this type of analysis. More generally, other suitable regression or classification models, such as probit regression or support vector machines, could also be applied in this context.\n- In a broader sense, as long as the parameter of interest can be defined as the minimizer of a loss function, as presented in Lines 144-146, the proposed method can be used. This includes the case with MLP-based regressors or classifiers, high-dimensional models, or large deep learning models in general. To address this question more explicitly, we explore the implementation of the proposed method on the benchmark dataset MNIST where $X$ is image, $\\hat Y$ is generated using ResNet, and the comparison with the standard PPI method, for estimating the probability of the outcome being a certain label. In terms of the length of 95\\% confidence interval (the shorter, the more efficiency gain), the proposed method shortens about 15\\% (from 0.0448 to 0.0376 when $Y$=1/7, and from 0.0440 to 0.0370 when $Y$=6/8).", "claims_and_evidence": "Yes. Overall, the paper’s claims are supported by theoretical derivations and experiments.", "methods_and_evaluation_criteria": "Yes. I think the use of ACP and covariate shift problem should be a typical problem. And the experiment on real dataset looks like a reasonable setting. \nHowever, the author should have indicated what $\\boldsymbol{X}$, $\\boldsymbol{Y}$, and $\\boldsymbol{Z}$ mean in those experiments, especially for the diabetes experiment.", "theoretical_claims": "I reviewed the theoretical claims provided in the paper.  \nI did not perform a line-by-line check of every technical detail. \nI think the proofs for the main theoretical claims appear correct.", "experimental_designs_or_analyses": "1. The simulations compare non-ACP and using-ACP settings. While this is useful for demonstrating efficiency gains, a comparison with alternative methods from the literature (e.g., Prediction-Powered Inference) might further validate the practical benefits of the approach.\n2. Why SuperLearner is used? Are there other options?\n3. For real dataset, additional different datasets or disease contexts would strengthen the generalizability of the results.\n4. For the diabetes dataset, how well does the covariate shift assumption hold? \n5. For the diabetes dataset,  what $\\boldsymbol{X}$, $\\boldsymbol{Y}$, and $\\boldsymbol{Z}$ mean?", "relation_to_broader_scientific_literature": "The paper’s contributions are related to established statistical theories such as semiparametric theory, double robustness, and semi-supervised learning under covariate shift. This paper focuses specifically on the use of ACP.\nThis paper also relates to some work about data imputation, the use of pseudo-labeled data, and epistemic uncertainty.", "essential_references_not_discussed": "[1] proposes an approach for semi-supervised learning algorithms that can address covariate shift. Their framework also recovers some popular methods, including entropy minimization and pseudo-labeling.\n[2] discussed the use of pseudo-labeled data.\n\n[1] Aminian, Gholamali, et al. \"An information-theoretical approach to semi-supervised learning under covariate-shift.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\n\n[2] Rodemann, Julian, et al. \"In all likelihoods: Robust selection of pseudo-labeled data.\" International Symposium on Imprecise Probability: Theories and Applications. PMLR, 2023.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "In line 423-425, \"We adopted a logistic regression to predict diabetes status with all selected variables,\nand use the regression coefficients as association measures\"\n\nWhat linear/logistic regression is used? Did the authors try different models?\nWhat kind of models can be analyzed with the proposed theory? For example, can large deep learning models be explored with this theory? What if MLP-based regressors or classifiers?\nCan $\\boldsymbol{X}$ be images or other modalities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hPlbjyDfdB", "reviewer": "Reviewer_pTuV", "summary": "This paper explores the integration of automated computational phenotypes (ACPs) in semi-supervised learning settings. ACPs are used to derive phenotype data from electronic health records (EHRs) using machine learning models, reducing the labor-intensive nature of manual phenotype extraction. However, direct replacement of gold-standard phenotype data with ACPs can introduce bias.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1: [Claims and Evidence]*\n\n**Response:**\n- Thank you for this insightful comment! To understand the benefits of incorporating ACP $\\hat Y$, we introduce an assumption with a format similar to covariate shift, as presented in Lines 181-192. In real world scenarios, this assumption can be tested when $Y$ is available in the unlabeled data (e.g., in the income, politeness, wine datasets) but not when $Y$ is unavailable (e.g., in the diabetes dataset). In cases where the assumption cannot be tested, we agree that conducting a sensitivity analysis could be valuable and of help.\n- We also agree that evaluations on more datasets are worthwhile. We provided a more comprehensive response in our answer to Q2 of reviewer mNmv and to Q4 of reviewer 6g7c. Please kindly refer to that section for details. Briefly we newly analyze three datasets. For each of them, we present the comparison results of confidence interval length for regression coefficients in Table 1 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing). The proposed method is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction reaching approximately 60%.\n\n*Q2: [Methods and Evaluation Criteria]*\n\n**Response:**\n- This is definitely an excellent point that warrants further discussion in our revised version. In the presence of covariate shift, one must account for two sources of nuisance functions: the density ratio model and the regression model. Estimating both sources of nuisance functions is computationally more complex than estimating just one, as in some alternative methods. However, the inclusion of both sources forms the basis of double robustness, which is essential for achieving efficient estimation. This represents a tradeoff between estimation efficiency and computational complexity. In cases where the mechanism for selecting labels is known, such as in certain design-based studies, the proposed method only needs to estimate one source of nuisance function.\n\n*Q3: [Theoretical Claims]*\n\n**Response:**\n- The regularity conditions on nuisance estimators are standard and widely advocated in double/debiased ML literature, e.g., van der Laan (2011), Chernozhukov et al. (2018), Kennedy (2024), Chernozhukov et al. (2024). The convergence rate is achievable for many ML methods such as in regression trees and random forests (Wager and Walther 2015) and a class of neural nets (Chen and White 1999). One can refer to hernozhukov et al. (2018) for more examples.\n- As we briefly mentioned in our answer to Q1, the independence assumption for ACP generation can be tested when $Y$ is available in the unlabeled data but cannot if otherwise. When it cannot be tested, one can decompose the assumption as $p(\\hat y|x) = q(\\hat y|x)$ and $p(y|x, \\hat y) = q(y|x,\\hat y)$. It is clear that, the untestable part $p(y|x, \\hat y)=q(y|x,\\hat y)$ is something similar to the covariate assumption $p(y|x) = q(y|x)$. Note that this assumption is popularly adopted in semi-supervised learning, and distribution shift.\n\n*Q4: [Experimental Designs or Analyses]*\n\n**Response:**\n- We implemented our proposed method and compared with some alternatives on three new datasets (income, politeness, wine). We provided a more comprehensive response in our answer to Q2 of reviewer mNmv, to Q4 of reviewer 6g7c, and to Q1 above. Briefly, we present the comparison results of confidence interval length for regression coefficients in Table 1 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing). The proposed method is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction reaching approximately 60%.\n- We also expanded our simulation studies by comparing the proposed method with PPI, PPI++ and RePPI. We provided a more comprehensive response in our answer to Q2 of reviewer 6g7c. Briefly, the MSE results are presented in Tables 2 and 3 of [More Results](https://drive.google.com/file/d/1ihn6OAqb26TZ0KVaelSpWO1bQzqGDQaH/view?usp=sharing). Across all considered scenarios, the proposed method consistently outperforms the three alternatives.\n\n*Q5: [Relation to Broader Scientific Literature]*\n\n**Response:**\n- As in our answers to Q1 and Q4 above, we extensively compare the proposed method with PPI, PPI++ and RePPI in both real data and simulated data.\n\n*Q6: [Essential References Not Discussed]*\n\n**Response:**\n- Thank you for pointing out this important reference! Indeed, Yang et al. (2024) studies how to calibrate prediction sets in the presence of covariate shift and proposes a doubly robust approach to enhance the reliability and coverage of predictions. We also realized that there is work, such as Qiu et al. (2023), that explores prediction sets adaptable to unknown covariate shift. In the revised version, we will be sure to include these relevant, important, and exciting works!\n\n*Q7: [Other Strengths And Weaknesses]*\n\n**Response:**\n- Same as Q3 above.", "claims_and_evidence": "The paper provides rigorous mathematical justifications and asymptotic efficiency analyses to show the advantages of using ACPs.\nThe empirical results from simulations and a real-world application to diabetes prediction support the claim that ACPs improve inference efficiency. The paper assumes that the ACP prediction errors do not introduce systemic biases beyond covariate shift, which might not always hold in real-world scenarios. A sensitivity analysis could be conducted to investigate this.\nHowever, the practical effectiveness of these methods in more complex, real-world medical settings (beyond a single dataset) could be better validated.", "methods_and_evaluation_criteria": "The paper defines clear evaluation metrics (asymptotic efficiency bounds, mean squared error comparisons).\nThey compare scenarios with and without ACPs to show how the additional data impacts estimation.\nThe use of cross-fitting for nuisance parameter estimation ensures robustness when applying machine learning methods.\n It does not extensively discuss computational trade-offs (e.g., increased complexity due to doubly robust estimation).", "theoretical_claims": "I didn't see any issue.\nSome technical assumptions (e.g., regularity conditions on nuisance estimators, independence assumptions for ACP generation) are not explicitly tested.", "experimental_designs_or_analyses": "No issue, would be better to have more experiments.", "relation_to_broader_scientific_literature": "The paper is well-grounded in existing literature on: Semi-supervised learning (e.g., Zhu 2005, Rigollet 2006, Wang et al. 2022).\nInference with predicted data (prediction-powered inference, PPI) (Angelopoulos et al. 2023).\nCovariate shift and domain adaptation (Sugiyama et al. 2008, Gretton et al. 2009). Surrogacy in biostatistics and causal inference (Athey et al. 2019, Imbens et al. 2024).\nThe paper’s main contribution is combining semi-supervised learning, covariate shift adjustments, and ACPs in a unified framework. The paper does not extensively compare its approach to existing robust semi-supervised estimation methods beyond theoretical efficiency arguments. T", "essential_references_not_discussed": "The authors can perhaps also discuss the relationship with the paper Doubly Robust Calibration of Prediction Sets under Covariate shift (Yang, Kuchibhotla, Tchetgen Tchetgen 2024), which is highly relevant to their work.", "other_strengths_and_weaknesses": "The efficiency analysis is mathematically well-founded and there is a experiment on both real world and synthetic datasets.\nSome of the assumptions are too strong.", "other_comments_or_suggestions": "None", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper analyzes the efficiency of incorporating automated computational phenotypes (ACPs) in a semi-supervised learning setting, particularly in the case where a covariate shift exists between labeled and unlabeled data. For the estimation of a target parameter of the unlabeled(or combined) population, the authors derive the efficient influence function and the semiparametric efficiency bound under various scenarios, and rigorously show that ACPs can help improve efficiency when they are provided for the unlabeled data, and as long as they do not depend solely on available features. Finally, authors also propose a semi-parametrically efficient and doubly-robust estimator which also permits the use of flexible ML methods for estimation of nuisance functions. 

Reviewers acknowledge the theoretical contribution of the paper, and the additional experiments submitted during the rebuttal further support the theoretical claims.
Decision: Accept (poster)