Task-Agnostic Machine-Learning-Assisted Inference
Authors: Jiacheng Miao
Qiongshi Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: AI for Science
Scientific Machine Learning
Machine Learning-Assisted Inference
post-prediction inference
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We introduce a task-Agnostic statistical framework that integrates machine learning with classical statistical inference, allowing for valid and efficient post-prediction inference across various scientific disciplines.

Abstract: Machine learning (ML) is playing an increasingly important role in scientific research. In conjunction with classical statistical approaches, ML-assisted analytical strategies have shown great promise in accelerating research findings. This has also opened a whole field of methodological research focusing on integrative approaches that leverage both ML and statistics to tackle data science challenges. One type of study that has quickly gained popularity employs ML to predict unobserved outcomes in massive samples, and then uses predicted outcomes in downstream statistical inference. However, existing methods designed to ensure the validity of this type of post-prediction inference are limited to very basic tasks such as linear regression analysis. This is because any extension of these approaches to new, more sophisticated statistical tasks requires task-specific algebraic derivations and software implementations, which ignores the massive library of existing software tools already developed for the same scientific problem given observed data. This severely constrains the scope of application for post-prediction inference. To address this challenge, we introduce a novel statistical framework named PSPS for task-agnostic ML-assisted inference. It provides a post-prediction inference solution that can be easily plugged into almost any established data analysis routines. It delivers valid and efficient inference that is robust to arbitrary choice of ML model, allowing nearly all existing statistical frameworks to be incorporated into the analysis of ML-predicted data. Through extensive experiments, we showcase our method’s validity, versatility, and superiority compared to existing approaches. Our software is available at https://github.com/qlu-lab/psps.

Task-Agnostic Machine-Leaming--ssitte Inference
Jiacheng Miao University of Wisconsin-Madison jiacheng miao@wisc. edu
Qiongshi Lu University Wisconsin-Madison qlu@biostat. wisc
Abstract
Machine learning (ML) is playing an increasingly important role in scientific re- search. In conjunction with classical statistical approaches, ML-assisted analytical strategies have shown great promise in accelerating research findings, This has also opened whole field of methodological research focusing on integrative approaches that leverage both MI and statistics to tackle data science challenges. One type of study that has quickly gained popularity employs ML to predict unobserved outcomes in massive samples, and then uses predicted outcomes in downstream statistical inference. However, existing methods designed to ensure the validity of this type of post-prediction inference are limited to very basic tasks such as linear regression analysis. This is because any extension of these approaches to new, more sophisticated statistical tasks requires task-specific algebraic derivations and software implementations. which ignores the massive library of existing soft- ware tools already developed for the same scientific problem given observed data. This severely constrains the scope of application for post-prediction inference. To address this challenge, introduce novel statistical framework named PSPS for task-agnostic ML-assisted inference. It provides a post-prediction inference solu- tion that can be easily plugged into almost any established data analysis routines. It delivers valid and efficient inference that is robust to arbitrary choice of ML model, allowing nearly all existing statistical frameworks be incorporated into the analysis of MI-predicted data. Through extensive experiments, we showcase our method's ' validity, versatility, and superiority compared existing approaches. Our software is available at
com llu-lalllppppsss
1 Introduction
Leveraging machine learning (ML) techniques to enhance and accelerate research has become increasingly popular in many scientific disciplines [44]. For example, sophisticated deep learning models have achieved remarkable success in predicting protein structure and interactions, which has the potential significantly speed up the research process, costs, and revolutionize the field of structural biology 2 25). However, recent studies have pointed out that statistical inference using ML-predicted outcomes may lead to invalid scientific discoveries due to the lack of consideration of ML prediction uncertainty in traditional statistical approaches. To address this, researchers have introduced methods that couple extensive ML predictions with limited gold-standard data to ensure the validity of ML-assisted statistical inference
35 46)
Despite these advances, current ML-assisted inference methods can only address very basic statistical tasks, including mean estimation, quantile estimation, and linear and logistic regression 3 35). While the same mathematical principle behind existing ML-assisted inference methods can be generalized to a broader class of M-estimation problems, specific algebraic derivations and computational implementations
required for each new statistical task. Moreover, many tasks, such the Wilcoxon rank-sum test, do not fit into the M-estimation framework. These issues pose significant challenges to the broad application of ML-assisted inference across various scientific domains.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Historically, the field of statistics has faced similar types of challenges. Before the advent of resampling-based methods [99] it used require task-specific derivation and implementation to obtain the variance of any new estimator. This old problem mirrors the current state of MI.- assisted inference, where every new task requires non-trivial effort from researchers. However, with resampling-basee inference, the need to manually derive variance is reduced. Instead, resampling methods can be universally applied to many estimation problems and easily obtain variance [17-19]. Inspired by this, we seek universal approach that incorporates ML-predieted data into any existing data analysis routines while ensuring valid inference results.
We introduce a simple protocol named DoSt-Prediction Summary stwpsrr-sssttsssssbbaaattte (PSPS) inference (ig.. I. employs existing analysis routines to generate summary statistics sufficient for ML-assisted inference. and then produces valid and powerful inference results using these statistics. It has several key features:
Assumption-lean and data-adaptive: It inherits the theoretical guarantees of validity and efficiency from state-of-the of-tth--rtt-rttt ML-assisted inference methods 20 35 These guarantees hold with arbitrary ML predictions.
Task-agnostic and simple: Since our method only requires summary statistics from existing analysis routines, it can be easily adapted for many statistical tasks currently unavailable or difficult to implement in ML-assisted inference.
Federated data analysis: It does not need any individual-leve data as input. Sharing of Srivacy-preeervngg summary statistics is sufficient for real-world scientific collaboration.
2 Problem formulations
2.1
Setting
We focus on statistical inference problems for the parameter 8*
(PP) E RR defined on the joint distribution of (X, Y) ~ F. where Y E y is scalar outcome and X E X be a K K-dimensional vector representing features. We are interested in estimating 0" using labeled data = {(X, Y.), = 1, n + N) = X and a pre-trained ML model x Here, f()) a black-box function with unknown operating characteristics X and can be mis- miis-pecified. We also require algorithm A that inputs the labeled data L and returns consistent and asymptotically normally distributed estimator 0 for 0" There are three common ways in the literature to estimate 8":
1. n) = (X., Yz), unlabeled data U = {X..
Classical statistical methods apply algorithm A to only labeled data L (Xc, Yz), and returns the estimator and its estimated variance 10c, Var(O=)). Valid confidence intervals and hypothesis tests can then be constructed using the asymptotic distribution of the estimator. However. it ignores the unlabeled data and ML prediction.
Imputation- based methods treat MI. prediction f in the unlabeled data as the observed outcome, and apply algorithm A to 2 = (Xu, fu.). We denote the estimator and estimated variance as [M,, Var Var(Tz)) This has been shown to give invalid inference results and false scientific findings 3 35 36 46).
MI-assisted inference methods use both L and i as input. These approaches add debiasing term in the loss function (or estimating equation) for M-estimation problems, thus removing the bias from the imputation- estimators and producing results that are statistically valid and universally more powerful compared to classical methods 44 35 36).
Next, we use an example to provide intuition on Ml.-assisted inference and our protocol.
2.2 Building the intuition with mean estimation
We consider the mean estimation problem, where 0 = E[Y]] = arg Ein (Y; - 0)²1. The classical method only takes the labeled data Yc as input and yields an unbiased and consistent
estimator for H*: θz = arg ming 1 Σ (Y, 0)2 1 Σ y, The imputation- based method
only takes the unlabeled data Fu as input and returns îu arg ming 1 E 11 - N E fr. It is biased and inconsistent estimator for E[Y.] if the ML model is mis-specified.

---

To address this, Ms-assisted estimator takes both labeled data (Yc. fc) and unlabeled data fu as input and adds debiasing term to the loss function to rectify the bias caused by ML imputation
20 35 36
Ôn arg ming
100
1
Σ i=+t
++N
(h:
0)2
[00
1 Σ (Y. 0)91
N
Dehissing tonn
Wa
1 N
Σ 2m1---1
W
f 1
Debiasing serm
where the modified loss ensures the consistency of the ML-assisted estimator and the weight Wp
Covi (Y.]) /n
ensures that ML-aasisted estimator is no less efficient than the classical estimator
Varilf/ Varu JJ/N
with arbitrary ML predictions: Var(OmLa) Var(Oz) Var Varlf//N
<Vr(Oo)-
Our proposed method is motivated by the observation that the sufficient statistics of the ML-assisted estimator ONLA and its estimated variance Var Var(OnLa) are the following summary statistics:
Var [Y]/n
Covi fl/n
0
0nn
Σ
Î) and Var( Covi [Y, fl/n
Van [f/n
i=n+1
0
Var [F)/N
Morcover, they can be easily obtained by applying the same algorithm. AA (mean estimation)
labeled data with observed outcome A(Y2) (0c. Var Var(Oz)) [ Σ Ut. Var YY] 17/mm
labeled data with predicted outcome A(fc) + Iñc, Var(fc)) = 1 > Jr Vari [m/m]
n
:=1
unlabeled data with predicted outcome [iu Var(fn)) = I Σ Jú, Var [Л/N]
i==+1
bootstrap of labeled data Al(Y,, [Y, /m. Here, (Yc, Je),, represents the g-th bootstrap of labeled data.
(c)y, = 1....,Q) Q for estimation of Cov (0c, Mc) =
Combining these summary statistics for one-step debiasing WoTu (Zo - Qz) recovers G
To summarize, algorithm for mean estimation, coupled with resampling, is sufficient for MI.- assisted mean estimation. This observation inspired us to generalize this protocol for broad range of tasks. Our protocol illustrated in Fig. 1 only requires three steps: 1) using pre-trained ML model to predict outcomes for labeled and unlabeled data, 2) applying existing analysis routines to generate summary statistics, and 3) using these statistics in debiasing procedure to produce statistically valid results in ML-assisted inference.
Labeled
X
Analysis routines
Estimator
Åg. (ār((
Covarance
Labele
Pre-trained model
X
Analysis routines
Estimattor itc. (ir(
Rechonad
One-step debiasing
Uniabeled date
Pre-trained
ML model
fu
Xu
Analysis routines
Estimator Ru Victitul
ML-assisted inference
Outcome
Fostures
Summary statistics
Stept:
Step3: Employ debiasiing the summary statistics
pre-trined predict the gold-standard outsome in both labaled and unlabeled data
Apply the analysis summrrry
Figure 1: Workflow of PSPS for Task-Agnostic ML-Assisted Inference.
3

---

2.3 Related work
Our work is closely related to recent methods developed in the literature of ML-assisted in- [20 35 37 46 56), and is also related methods for handling missing data 40 42 and semi-supervised inference [6 16 50 52]. While current ML-assisted inference
ference 3
4
methods modify the loss function or the estimating equation, our protocol works directly on the summary statistics. For simple problems such as mean estimation, current methods yield closed-form solution the optimization problem. However, for more general statistical tasks, there is no such closed-form solution. Current methods typically require the algebraic form of the loss function, its first- and second-order derivatives, and the variance for the estimator, as well as newly implemented optimization algorithm to obtain the estimator. We use the logistic regression problem as an example. Here, 0 = arg ming B[-((e)) - (XX)) and
(t) = 1/(1 + exp(-t)). The ML-assisted estimator is BNLA = arg ming N
2+
Σ f
1=++1
* (x Σ 21 -1- 26 (X,0)) Σ PXO (X,+)]] with estimated asymptotic
[=1
variance V
where À C v (X,0) E 8" x - Var, ((V(XX,0) - + Cov (1 - (X,0)))))) + (Of: - X))?).
V(w) = N
and @ needs to be obtained by optimization to minimize the asymptotic variance. In contrast, our protocol simply applies logistic regression A to
labeled data with observed outcomes (Xc, Yz) to obtain 0c. Var
labeled data with predicted outcomes fc) to obtain Mic. Var( (7c)
unlabeled data with predicted outcomes (Xu, fu) obtain (ir. Var(fi))
bootstrap of labeled data (Xc, Yc, [c)g: = 1,...,Q Q for Cor (oz nc).
and returns (3) - nc 0c), where W. (Var(fic) Var " Nc.. For each new statistical task, as long as an existing analysis routine can produce an estimator that is asymptotically normally distributed, our protocol can be similarly applied. Additionally, the current mathematical principles guiding ML-assisted inference apply solely to M-estimation [3] 4 20 [3]]]]] [56.. Our protocol
extends beyond this limitation, addressing all estimation problems with an asymptotically normally distributed estimator.
Inference relying solely on summary statisties is widely used in the statistical genetics literature for practical reasons. Summary statistics-based methods have been developed for tasks such as variance component inference and genetic risk prediction [2 34 39 (3). In contrast our work, these applications do not leverage ML predictions, but instead focus on inference using summary statisties obtained from observed outcomes. An exception is a previous study for valid genome wide association studies (GWAS) on ML-predicted outcome [36]. However, it focused only on linear regression modeling with application to GWAS. The PSPS framework introduced in this paper aims to extend ML-assisted inference to general statistical tasks.
Our work is also related to semi-supervised learning, resampling-based inference, zero augmentation, and false discovery rate (FDR) control methods. Our protecol is designed for estimation and statistical inference using both labeled and unlabeled data, addressing a different problem from sum--supeeeieed learning [55]. which primarily focuses on prediction. Our protocol is inspired by the core principle of resampling-base
inference, which replaces algebraic derivation with computation [19]. The main difference is that we focus on how to use ML to support inference, whereas resampling- based inference focuses on bias and variance estimation, and type-I error control. The idea of zero augmentation has been used in augmented inverse propensity weighting estimators (38) and
in handling unmeasured confounders (48) and missing data for U-statistics [14]. These estimators do not incorporate ML, which is fundamental to our work. We also adapt techniques from the FDR literature 17-10] Our unique contribution to use MI. to support FDR control, thereby increasing its statistical power, in contrast to classical methods that rely solely on labeled data.

---

3 Methods
3.1 General protocol for PSPS
Building on Section 2. we formalized our protocol in Fig. for ML-assisted inference:
Algorithm 1 PSPS for ML-assisted inference
Input: A pre-trained MI. model f. labeled data (X c, Yz), unlaheled data 4 Xu
1: Use the ML model / to predict the outcome in both labeled and unlabeled data.
2: Apply the algorithm A in the analysis routine
labeled data (X Yz) and obtain [Oc, Var 0c)]
labeled data (Xe, c, Jc) and obtain [Mc, Var(fc))
unlabeled data with (Xu,, Jix) and obtain (Te, Var(fu))
Q bootstrap of labeled data (Xe, Yc, Sc),: 1y....9 Q and obtain Cov (0 nc). 3: Employ one-step debiasing to the summary statistics in step2:
OPEPs w6 nm
(w) ne Oc),
where Wo (ac) Var ) - Cov nc) and Var(Opsps) Cov(O2, 7c) [Var(fi) (mc) +
Var(Oz)
Cov nc)
Output: ML-assisted point estimator Opers, standard error
V Var (Orsrs)))) a-level confidence interval (Opsrs 21 -1/0/2 Var Var (Opsps) (k) and (two-sided) p-value ()), where the CDF of the standard normal distribution. 2(1 411 V Var(OOaaa )
for the k-th coordinate CPSPS
Oosps,
The only requirements for our protocol are: i) algorithm A, when applied to labeled data (Xc, Ye), returns a consistent and asymptotically normally distributed estimator of 0"; ii) labeled and unlabeled data are independent and identically distributed. Under these assumptions, the summary statistics have the following asymptotic properties:
0c 0"
{
0K
)
V(θc)
V nc)
0
1/2
NC -
AN
0K
v(0L,
V(ñe)
0
(I)
Tpu n
where =n(pp) E R is defined on (X,F) Pj. V(-) denotes the asymptotic variance and covariance of a estimator, and P N. The asymptotic approximation gives V(θc) a n Var(0z), V 7c) re n Cov îc), V(ñc) xe n Var(îc) and V(πx) ze N Var(fu)). Here, we do not require Ne and īuu to be consistent for 0*, thus allows arbitrary ML model.
With the summary statisties following a multivariate normal distribution asymptotically, the debiased estimator Opers = a6 th - (of Nr 0z) is consistent for 0* and asymptotically normally distributed Theorem 1p. Therefore, by plugging in consistent estimator for its asymptotic variance V (Opaps) RJ (ar(O)s)s) valid confidence interval and hypothesis testing can be achieved.
Remark 1. PSPS is more "task-agnostic" than existing methods in three aspects:
1. For M-estimation tasks, currently, only mean and quantile estimation, as well as linear, logistic, and Poisson regression, have been implemented in software tools and are ready for immediate application. For other M-estimation tasks, task-specific derivation of the ML- assisted loss functions and asymptotic variance via central limit theorem are necessary. After that, researchers still need to develop software packages and optimization algorithms to carry out real applications. contrast, PSPS only requires already implemented algorithms and software designed for classical inference based on labeled data.
5

---

2. For problems that do not fall under M- M-estimation but have asymptotically normally dis- tributed estimators, only PSPS can be applied, and all current methods would fail. The principles behind MI--assisted M-estimation do not extend to these tasks.
3. Even for M-estimation tasks that have already been implemented. PSPS offers the additional advantage of relying solely on summary statistics. The "rask-specffit derivations" refer not only to statistical tasks but also to scientific tasks. Real-world data analysis in any scientific discipline often involves conventions and nuisances that require careful consideration. For example, our work is partly motivated by GWAS 43 Statistically. GWAS is a linear regression that regresses an outcome on many genetic variants. While the regression- based statistical foundation is simple, conducting valid GWAS requires accounting for numerous technical issues, such as sample relatedness (i.e., study participants may be genetically related) and population structure (i.c., unrelated individuals of the same ancestry are both genetically and phenotypically similar, creating confounded associations in GWAS). Sophisticated algorithms and software have been developed to address these complex issues [31] will be very challenging if all these important features need to be reimplemented in an ML-assisted GWAS framework. With PSPS protocol, researchers can utilize existing tools that are highly optimized for genetic applications to perform ML-assisted GWASS. This adaptability not just limited GWAS, but is major feature of our approach across scientific domains. PSPS enables researchers to conduct MI-assisted inference using well-established data analysis routines.
Remark 2. The "federated data analysis" feature of PSPS refers to the fact that we only require summary statistics as input for inference, rather than individual-level raw data (features and label Y). For example, consider scenario where labeled data is in one center and unlabeled data is in another, yet researchers cannot access individual- such conditions, current ML-assisted inference, which relies accessing both labeled and unlabeled data to minimize a joint loss function, is not feasible. However, PSPS circumvents this issue by aggregating summary statistics from multiple centers, therehy performing statistical inference while upholding the privacy of individual-lever data.
data from both centers simultaneously. Under
3.2 Theoretical guarantees
In this section, we examine the theoretical properties PSPS. In what follows. P denotes con- vergence in probability and 22 denotes convergence in distribution. All proofs are deferred the Appendix A
The first result shows that proposed estimator is consistent, asymptotically normally distributed, and uniformly better in terms blement-wise asymptotic variance compared with the classical estimator based on labeled data only.
holds, then Opees P 0" and
Theorem 1. Assuming equation
n 1/2 ( 0²) 2 O. V V(Orass))
where ((ppsps) = ((())-vve ( (V (fu)) VV (0c,, Nc). Assume the k-th column of ( N))) is not zero vector and at least one of V (ñic) and (fru) are positive definite, then V ((@PSPs)RR ((0)) With V (0paps) ES (0psps)), lim, P(0) E CPSPS)
-aa
((Prss) can be obtained by applying the algebraic form of V ((ōpss)) using the bootstrap estimators for V(02). V(ñc), V(θz, nc,, and V(ñu). The regularity conditions for consistent bootstrap variance estimation outlined in Theorem 3.10 (i) of [T]. We also refer readers to (21). which showed that bootstrap-based variance provides valid but potentially conservative inference.
This result indicates that greater reduction in variance for the ML-assisted estimator is associated with larger values of V V me) and smaller values of V(fc). V V(fu). and p. The variance reduction term [V (0c, 7T) (V (fc) + (V(r)) -V ic)]E ka can also serve as a metric for selecting the optimal ML model in ML-assisted inference.

---

Our next result shows that three existing methods, i.e., PPI, PPI++ and PSPA, are asymptotically equivalent PSPS with different weighting matrices. A broader class for consistent estimator of e* ((w) = w nu (w 0c - 7c), where w is a K x K matrix. The consistency of ((w) for 0 only requires w (πu nc) P, Since (îu - nc) E 0, assigning arbitrarily fixed weights for will satisfy the condition. However, the choice of weights influences the efficiency of the estimator as illustrated Proposition 2 lattr..
Proposition 1. Assuming equation ① and regularity condition for the asymptotic normality of current ML-assisted estimator holds. For any M-estimation problem, we have
n (0 (diag(wae) C) D. 0, n² (0 C) 2 0, n (0 (d)=a 2 0.
Here, Wele [we.......... Wele. E RK and Wale minimizing the k-th diagonal element of V (Ō(w)). War is a scalar used minimize the trace of (0(w)), and C is matrix associated with the second derivatives of the loss function in M-estimation, with further details deferred to Appendix A
This demonstrates that for M-estimation problems, our method is asymptotically equivalent to PSPA, PPI++ and PPI with the respective weights diag(wel) C, diag(wr) C, and diag(1)C. Therefore, PSPS can be viewed generalization of these existing methods.
Our third result shows that the weights used in the Proposition are not optimal. Instead, our choice of wo represents the optimal smooth combination of (0c, nc, fu) in terms of minimizing the asymptotic variance, while still preserving consistency.
Proposition 2. Suppose n 7c, 0.) N00 Eg) and g is smooth function, then Egu 2 EpspSan
Together with Proposition 1 our results demonstrate that our protocol provides a more efficient estimator compared to existing methods for the M-estimation problems. Furthermore, the applicability of our protocol is not limited to M-estimation and only requires summary statistics as input. It also indicates that in a setting of federated data analysis [24] where individual-level data are not available, PSPS proves to be the optimal approach for combining shared summary statistics.
Remark 3. PPI++ 4 employs a power-tuning scalar for variance reduction in ML-assisted inference. This scalar is obtained by minimizing the trace possibly other scalarization of the estimator's variance-covariance matrix. However, the asymptotic variance of PSPS is always equal to or smaller than that of PPI++ irrespective of the scalarization chosen by researchers. This advantage arises because PSPS utilizes a K x K power tuning matrix, w, for variance reduction, where K repre- sents the dimensionality of parameters. This matrix facilitates information sharing across different parameter coordinates, thereby enhancing estimation precision. The choice of weighting matrix PSPS also allows for element variance reduction, reducing each diagonal element of the variance-covariance matrix. In contrast, the single scalar in PPI++ can only target overall trace reduc- tion variance reduction of a specific element. A detailed example is provided in AAppendix B Only in one-dimensional parameter estimation tasks, such as mean estimation, PPI+ and PSPS exhibit the same asymptotic variance.
3.3 Extensions
We also provide several extensions ensure the broad applicability of our method.
3.3.1
Labeled data and unlabeled data are not independent
Here, we relax the assumption that the labeled data and unlabeled data are independent. When they are not independent, this can lead to the non-zero covariance between the te and niu. Consider a broader class of summary statisties asymptotically satisfying
02 0"
0K
v
v V(OC. V(ñc) (Oc(Oc)
V fu) (oV(Oc, (V(fm))
n 1/2
nc m nu
D .
0x
V TC) V(ñc.
0x
iM)

---

We can similarly employ the one-step debiasing Pppe
no- Indep = 20 nu wo(nc - 0) where Wy
(V (V Ni) (ññ. nu)) (V ((nu)
mn) and (a no indop) Var(Oc)
PSPS
(V(Oc, 7c) V (ñc. nU)) theoretical guarantees of the proposed estimator can be similarly derived by Theorem
+ V(r)) - 2V (0c. n())) (V(0c. c. ne) Vñg Ru)). The
3.3.2 Sensitivity analysis for distributional shift between labeled and unlabeled data
The other assumption of our approach is that the labeled and unlabeled data are identically distributed so that we can ensure Пे -uu E O and validity of PSPS results. To address the potential violation of this assumption, we introduce a sensitivity analysis with hypothesis testing for the null Ho
nc.k 7 with test statisties
2 NN 1) to assess if me. and T significantly different. Here, the subscript & indicates the k-thh coordinate. We recommend using p-value < 0.1 as evidence for heterogeneity and caution the interpretation of results from ML-assisted inference.
fcc. -he.
Varr(fic +)+ V)
3.3.3 ML-assisted FDR control
The output from PSPS can used for ML-assisted FDR control, achieving greater statistical power compared to classical FDR control methods that solely rely on labeled data. We refer to our approach as PSPS-BH -B and SePS-knockoff. Briefly, PSPS-EB BH processes the p-value from ML-assisted linear regression through the Benjamini-Hochberg (BH) procedure [9], while
utilizes the
ML-assisted debiased Lasso coefficient [23 5T in the ranking algorithm of knockoff [7]. We present our algorithm in Appendix and evaluate their performance using experiments in Secciinn 4
3.3.4 ML-assisted inference with predicted features
We have discussed ML-assisted inference with outcomes predicted by ML. models. Here, we note that PSPS can also be applied when either features alone are predicted or both features and outcomes are predicted. The key idea is that the difference between point estimators obtained from applying A to predicted features in both labeled and unlabeled datasets is a consistent estimator for zero. This enables zero augmentation for estimators from observed features and outcomes. To implement this, modify step 2 in AAlgorithm to apply A to predicted features in both labeled and unlabeled data. A similar approach is applicable when both features and outcomes are predicted.
4
Numerical experiments and real data application
4.1
Simulations
We conduct simulations to assess the finite sample performance of our method. Our objectives are to demonstrate that 1) PSPS achieves narrower confidence intervals when applied to statistical tasks already implemented existing ML-assisted methods; 2) when applied to statistical tasks that have not been implemented for ML-assisted inference, PSPS provides confidence intervals with narrower width but correct coverage (indicating higher statistical power) compared to classical approaches rely solely on labeled data; 3) PSPS provides well-calibrated FDR control and achieves higher power compared to classical methods only using labeled data.
Tasks that have been implemented for ML-assisted inference We compare PSPS with the classical method using only labeled data, the imputation-baut based method using only unlabeled data, and three ML-assisted inference methods PPI, PPI++ and PSPA (3 4 35 mean estimation and linear and logistic regression. We defer the detailed data-generating process to Appendix D In short, we generated outcome Y; from feature X1: and X2s and obtained the pre-trained random forest that predict Y, using X1s and Xz.. We have 500 labeled samples (X1., Y., f() and unlabeled samples (X1,, f.) ranged from 1,000 to 10,000. Our goal is to estimate the mean of Y,, as well as the linear and logistic regression coefficient between Y, and X1s.
Fig. 2p show confidence interval coverage and Fig. 2- show confidence interval width. We find that the imputation-based method fails to obtain correct coverage, while all others including
PSPS have the correct coverage. PSPS has narrower confidence intervals compared the classical method and other approaches for ML-assisted inference.
8

---

a
Mean estimation
b
Linear regression
Logistic regression
1.00
1.00
1.00
Coverage
0.75
Covarage
0.75
Coverage
0.75
0.50
0.50
0.50
0.25
0.25
0.00
0.00
0.00
1000
2500
3000
10000
1000
2500
5000
10000
1000
2500
5000
10000
Nuriabeled
Nnnlabeled
Nurlibeled
d
Mean estimation
Linear regression
f
Logistic regression
0.18
Width of 959% CI
0.25
0.42
Width of 966%
0.16
0.38
0.14
0.20
0.12
0.10
0.15
Width
0.30
0.08
1000
2500
5000
10000
1000
2500
5000
10000
1000
2500
5000
10000
N Nuniabeled
N untaabetd
Nuriabeled
Methods
Classical
Imputation-bssed
PSPA
PSPS
PPI
PPI++
Figure 2: Simulation for tasks that have been implemented for MI.-assisted inference including mean estimation, linear regression, and logistic regression from left to right. Panel a-c present confidence interval coverage and panels d-f present confidence interval width.
Quantil Regrassion
Regrassion
NB Regression
Declased Lasto
W Icoxon Rank Sum Test
1.00
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.75
Coverage
Coverag
Coverage
0.50
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
0.00
1002
2000
5006 12300
1000
2500
5300
10000
3000
5000 30330 2003
1500
2000
2500
3300
:000
2530
5033 53000
Nunuaaied
Murkteke
Nasiaodeed
Murkateldd
N Aunideded
Quantie Regression
IV Reg resressionn
h
NB Regression
Destased Laszo
W Icoxon R Rank-Sum Test
1.00
0.27
0.33
witth of 85%
0.425
Witth C0%
0.16
0.75
0.24
0.400
0.32
0.14
0.50
0.21
0.375
0.31
0.12
0.00
1002 2000 5000 10000
1090 1353 5000 20033
2000
1353
5000 20300
1306
2000
2500
3330
2000
2330
5003 20000
Nasuadol
Narkttkk
N Naviboel
Methods
PSPS
Classical
Imputed
Figure 3: Simulation for tasks that have not been implemented for ML-assisted inference including quantile regression, instrumental variable (IV) regression, negative binomial (NB) regression, debi- ased Lasso, and Wilcoxon rank-sum test from left to right. Panels a-e present confidence interval coverage (I type-error for Wilcoxon rank-sum test) and panels f-j present confidence interval width power for Wilcoxon rank-sum test).
Tasks that have not been implemented for ML-assisted inference Next, we consider several commonly used statistical tasks that currently lack implementations for ML-assisted inference
including quantile regression 277, instrumental variable (IV) regression [S negative binomial (NB) regression (22), debiased Lasso [5 and the Wilcoxon rank-sum test (28]. Similar to our previous simulations, we utilize labeled data, unlabeled data, and pre-trained ML model. Detailed simulation settings are deferred to the Appendix Our goal is to estimate the regression coefficient between Y, and X1: for quantile (at quantile level 0.75), IV, and NB regression, between Y; and high dimensional features X, E R 150 for debiased Lasso, and to perform hypothesis testing on the medians of two independent samples X =1 and Y, X 0 using the Wilcoxon ramk-sum test.

---

Fig.3 l-d show confidence interval coverage and Fig. 3 show confidence interval width for parameter estimation, Fig. 3/ and Fng. 3] show the type-I error and statistical power for the Wilcoxon rank-sum test. We found that the imputation-based method fails to obtain correct confidence interval coverage and shows inflated type-I error, while PSPS and classical method have the correct coverage and well- calibrated type-I error control. PSPS has narrower confidence intervals width in all settings, and higher statistical power for the Wilcoxon rank-sum test compared to classical methods. Confidence intervals become narrower as unlabeled sample size increases, indicating higher efficiency gain.
FDR control We evaluate the finite sample performance of PSPS-BH and pSPS-knockoff in con- trolling the FDR compared with classical and imputation-baser methods as baselines. We consider n) linear regressions for PSPS-BH and
(KK < n) and high-dimensional(K respectively. We simulate the data such that only a proportion of the features are truly associated with the outcome. The data generating process is deferred to Appendix Our goal is to select the associated features while maintaining the target FDR level.
Fig. E. E- shows the estimated FDR Fig. E.Ip-d shows the statistical power for different methods. Imputation-based method failed to control FDR in either low-dimensional or high-dimensional settings. Classical approach, PSPS-BH, and PSPS-knockoff effectively controlled in both low- dimensional and high-dimensional settings. PH, and PSPS-knockoff achieve higher statistical power compared to the classical method.
These simulations demonstrate that PSPS outperforms existing methods and can be easily adapted for various statistical tasks not yet implemented in current MI--assisted inference methods.
4.2 Identify vQTLs for bone mineral density
We employed our method to carry out ML-assisted quantile regression to identify genetic variants associated with the outcome variability (vQTL) of bone mineral density derived from dual-energy X-ray absorptiometry imaging (DXA-BMI)
33 DXA-BMD is the primary diagnostic marker Identifying vQTL for DXA-BMI can provide insights
for osteoporosis and fracture risk 15 54.
into the biological mechanisms underlying outcome plasticity and reveal candidate genetic variants involved in potential cene-gene and gene-environment interactions [29] 32 45 47 [4]. We focused on total body DXA-BMD, which integrates measurements from multiple skeletal sites. We used data from the UK Biobank [13), which includes 36,971 labeled and 319,548 unlabeled samples with 9, 9,450,880 genetic variants after quality control. We predicted DXA-BMD values in both labeled and unlabeled samples using SoftImpute (30 with 466 other variables measured in the UK Biobank. Prediction in the labeled sample was implemented through vross-validttion avoid overfitting. The implementation detail is deferred to Appendix D We used the BH procedure to correct for multiple testing and considered FDR- 0.05 the significance threshold
No genetic variants reached statistical significance under the classical method with only labeled data. PSPS identified 108 significant variants with FDR < 0.05 spanning 5 independent loci, showcasing the superior statistical power of PSPS (Fig. E.2 and "Table E.T) Notably, these significant vQTL cannot be identified by linear regression [36], indicating the different genetic mechanisms controlling outcome levels and variability for DXA-BMD.
4.3 Computational efficiency
We compared the computational efficiency of PSPS with existing methods using a dataset of 500 labeled and 10,000 unlabeled data points. Results are shown in Table F.2 While PSPS is slower due to resampling, its overall runtime is still relatively short.
5 Conclusion
We introduced a simple, ags-ssin protocol for ML-assisted inference, with applications across broad range of statistical tasks. We established the consistency and asymptotic normality of the proposed estimator. We further introduced several extensions to expand the scope of our approach. Through extensive experiments, demonstrated the superior performance and broad applicability of our method across diverse tasks. Our protocol involves initially generating summary statistics using computationally efficient software tools in scientific data analysis, followed by integration of summary statistics to produce ML-assisted inference results, which achieves high computational efficiency while maintaining statistical validity. Future work could focus on developing a fast resampling algorithm to further improve computational efficiency.
10

---

Acknowledgements: We gratefully acknowledge research support from the National Institutes of Health (NIH; grant UOI HG012039) and support from the University of Wisconsin-Madde Office of the Chancellor and the Vice Chancellor for Research and Graduate Education with funding from the Wisconsin Alumni Research Foundation.
References
[1] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[2] Gustaf Ahdritz, Nazim Bouatta, Christina Floristean, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy J O' O'Donnell, Daniel Berenberg, lan Fisk, Niccolò Zanichelli, et al. Open- fold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. Nature Methods, pages 1-11, 2024.
[3] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zmic. Prediction-powertt inference. Science, 382(6671):669 6744 2023.
[4] Anastasios N Angelopoulos, John C Duchi, and Tijana Zrnic Ppi++: Efficient prediction-
powered inference. arXiv preprint arXiv:2311.01453, 2023.
[5] Joshua Angrist and Guido Imbens. Identification and estimation of local average treatment effects. Econometrica, 62:467 475, 1994.
[6] David Azriel, Lawrence D Brown, Michael Sklar, Richard Berk, Andreas Buja, and Linda Zhao. Semi-supervised linear regression. Journal of the American Statistical Association, 117(540): 2238 2251, 2022.
[7] Rina Foygel Barber and Emmanuel J Candès. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055-20883 2015.
[8] Rina Foygel Barber and Emmanuel J Candès. A knockoff filter for high- figh-dimensional selective inference. The Annals of Statisties, 47(5):2504-25331 2019.
[9] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series (Methodological), 57(1):289-300 1995.
[10] Yoav Benjamini and Daniel Yekuticli. The control the false discovery rate in multiple testing under dependency. The Annals of Statistics, pages 1165-1188, 2001.
[11] Brendan
Hilary K Finucane, Verneri Anttila, Alexander Gusev, Felix R Day. Po-Ru Loh, ReproGen Consortium, Psychiatric Genomics Consortium, Genetic Consortium for Anorexia Nervosa of the Wellcome Trust Case Control Consortium 3, Laramie Duncan, al. An atlas of genetic correlations across human diseases and traits. Nature genetics, 47(11):
1236-1241 2015.
[12] Brendan K Bulik-Sullivan, Po-Ru Loh, Hilary Finucane, Stephan Ripke, Jian Yang, Schizophrenia Working Group of the Psychiatric Genomics Consortium, Nick Patterson, Mark J Daly, Alkes Price, and Benjamin Neale. Ld score regression distinguishes confounding from polygenicity in genome- wide association studies. Nature genetics, (7(3):291-295 2015.
[13] Clare Bycroft, Colin Freeman, Desislava Petkova, Gavin Band, Lloyd T Elliott, Kevin Sharp, Allan Motyer, Damjan Vukcevic, Olivier Delaneau, Jared O' O'Connell et al. The uk biobank resource with deep phenotyping and genomic data. Nature, 562(7726):233-29, 2018.
[14] Timothy Cannings and Yingying Fan. The correlation-assistee missing data estimator. Journal of Machine Learning Research, 23(41):1-49, 2022.
[15] John J Carey and Miriam Delaney. Utility of dxa for monitoring, technical aspects of dxa bmd measurement and precision testing. Bone, 104:44-53, 2017.
11

---

[16] Siyi Deng, Yang Ning, Jiwei Zhao, and Heping Zhang. Optimal and safe estimation for high- dimensional sumi-supervised learning. Journal of the American Statistical Association, pages 1-12, 2023.
[17] Bradley Efron. The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
[18] Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569-593. Springer, 1992.
[19] Bradley Efron and Robert Tibshirani. An introduction to the bootstrap. Chapman and Hall/CRC, 1994.
[20] Feng Gan and Wanfeng Liang. arXiv:2312.06478, 2023.
Prediction de-correlated inference. arXiv preprint
[21] Jinyong Hahn and Zhipeng Liao. Bootstrap standard error estimates and inference. Economet- rica, 1934)116631197,,22 2021.
[22] Joseph M Hilbe. Negative binomial regression. Cambridge University Press, 2011.
[23] Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for high- dimensional regression. The Journal of Machine Learning Research, 15(1):2869-2909 2014.
[24] Michael Jordan, Jason D Lee, and Yun Yang. Communication--ffiicenn distributed statistical inference. Journal of the American Statistical Association, 2018.
[25] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin vZídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-55) 589, 2021.
[26] Zheng Tracy Ke, Jun S Liu, and Yucong Ma. Power of knockoff: The impact of ranking algorithm, augmented design, and symmetric statistic. Journal of Machine Learning Research, 25(3):1-67, 2024.
[27] Roger Koenker. Quantile regression, volume 38. Cambridge university press, 2005.
[28] Henry Mann and Donald R Whitney, On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50-60, 1947.
[29] Andrew R Marderstein, Emily R Davenport, Scott Kulm, Cristopher V Van Hout, Olivier Elc- mento, and Andrew Clark. Leveraging phenotypic variability to identify genetic interactions in human phenotypes. The American Journal of Human Genetics, 108( (08(1):49-67 67, 2021.
[30] Rahul Mazumder, Trevor Hastie, and Robert Tibshirani. Spectral regularization algorithms for learning large incomplete matrices. The Journal of Machine Learning Research, 11:2287-2322. 2010.
[31] Joelle Mbatchou, Leland Barnard, Joshua Backman, Anthony Marcketta, Jack A Kosmicki, Andrey Ziyatdinov, Christian Benner, Colm O' O'Dushlaine, Mathew Barber, Boris Boutkov, et al. Computationally efficient whole-genome regression for quantitative and binary traits. Nature genetics, 53(7): 1003, 2021.
[32] Jiacheng Miao and Qiongshi Lu. Identifying genetic loci associated with complex trait variability. In Handbook of Statistical Bioinformatics, pages 257-270. Springer, 2022.
[33] Jiacheng Miao, Yupei Lin, Yuchang Wu, Boyan Zheng, Lauren Schmitz, Jason M Fletcher, and Qiongshi Lu. A quantile integral linear model to quantify genetic effects on phenotypic variability. Proceedings of the National Academy of Sciences, 119(39))22129991199 19, 2022.
[34] Jiacheng Miao, Hanmin Guo, Gefei Song, Zijie Zhao, Lin Hou, and Qiongshi Lu. Quantifying portable genetic effects and improving cross-ancestry genetic prediction with gwas summary statisties. Nature Communications, 14(1) (:(12 2023.
[35] Jiacheng Miao, Xinran Miao, Yixuan Wu, Jiwei Zhao, and Qiongshi Lu. Assumption and data-adaptive post-prediction inference. arXiv preprint arXiv:2311.14200 2023.
12

---

[36] Jiacheng Miao, Yixuan Wu, Zhongxuan Sun, Xinran Miao, Tianyuan Lu, Jiwei Zhao, and Qiongshi Lu. Valid inference for machine learning-assistn assisted genome-wide association studies. Nature Genetics, pages 1-9, 2024.
[37] Keshav Motwani and Daniela Witten. Revisiting inference after prediction. Journal of Machine Learning Research, 24 2023.
[38] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association, 89(427):8468866 1994.
[39] Yunfeng Ruan, Yen-Feng Lin, Yen-Chen Anne Feng, Chia-Yen Chen, Max Lam, Zhenglin Guo, Lin He, Akira Sawa, Alicia R Martin, et al. Improving polygenic prediction in ancestrally diverse populations. Nature genetics, 54(5):573-580, 2022.
[40] Donald B Rubin. Multiple imputation after 18+ years. Journal of the American statistical Association, 91(434):473-489 1996.
[41] Jun Shao and Dongsheng Tu. The jackknife and bootstrap. Springer Science & Business Media, 2012.
[42]
Anastasios A Tsiatis. Semiparametric theory and missing data, volume 4. Springer, 2006.
[43] Emil Uffelmann, Qin Qin Huang, Nchangwi Syntia Munung, Jantina De Vries, Yukinori Okada, Alicia R Martin, Hilary C Martin, Tuuli Lappalainen, and Danielle Posthuma. Genome-wide association studies. Nature Reviews Methods Primers, 1(1):59, 2021.
[44] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60 60, 2023.
[45] Huanwei Wang, Futao Zhang, Jian Zeng. Yang Wu, Kathryn E Kemper, Angli Xue, Min Zhang, Joseph Powell, Michael E Goddard, Naomi R Wray, al. interactions inferred from genetic effects phenotypic variability in the uk biobank. Science advances, 5(8)=eaww3333 2019.
[46] Siruo Wang, Tyler H McCormick, and Jeffrey T Leek. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences, 117748):30266-302555 2020.
[47] Kenneth E Westerman, Timothy D Majarian, Franco Giulianini, Dong-Keun Jang, Jenkai Miao, Jose C Florez, Han Chen, Daniel Chasman, Miriam S Udler, Alisa K Manning, et al. Variance- -uaauncce-uantittiiieee trait loci enable systematic discovery of eeneeenviromment interactions for cardiometabolic serum biomarkers. Nature Communications, 13(1):3993, 2022.
[48] Shu Yang and Peng Ding. Combining multiple observational data sources to estimate causal effects. Journal of the American Statistical Association, 2020.
[49] Alexander I Young, Fabian I. Wauthier, and Peter Donnelly. Identifying loci affecting trait variability and detecting interactions in genome-wide association studies. Nature genetics, 50 (11):1608-1614, 2018.
[50] Anru Zhang, Lawrence D Brown, and T Tony Cai. Semi-supervised inference: General theory and estimation of means. 2019.
[51] Cun-Hui Zhang and Stephanie S Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76(1):217-242, 2014.
[52] Yuqian Zhang and Jelena Bradic. High-dimensional semissueervise learning: in search of optimal inference of the mean. Biometrika, 109(2)3387-433, 2022.
[53] Zijie Zhao, Tim Gruenloh, Meiyi Yan, Yixuan Wu, Zhongxuan Sun, Jiacheng Miao, Yuchang Wu, Jie Song, and Qiongshi Lu. Optimizing and benchmarking polygenic risk scores with gwas summary statistics. Genome Biology, 25(1):260, 2024.
13

---

[54] Hou-Feng Zheng, Vincenzo Forgetta, Yi-Hsiang Hsu, Karol Estrada, Alberto Rosello-Din Diez, Paul Leo, Chitra L Dahia, Kyung Hyun Park-Min, Jonathan Tobias, Charles Kooperberg. et al. Whole-genome sequencing identifies enl as determinant of bone density and fracture. Nature, Nature, 121177)) 2015.
[55] Xiaojin Jerry Zhu. suni-supervised learning literature survey. 2005.
[56] Tijana Zrnic and Emmanuel Candès.
inference. Proceedings of the
National Academy of Sciences, 121((55):2332201111,, 2024.
14

---

A Proofs
A.1 Proof the Theorem I
Proof. Since
2 0*
0K
)
v(0c) V V(0z, (c) 0
v(Oc. nc) V(ñc) 0
1/2
TC 1
0x
O
(2)
nu
OK
0c + 0* and πu Mc Pg O. Given weights Wo (V(nc) ((u)) (0c.
- (cc) where the
variances consistently estimated, Slutsky's theorem implies that, @g (fu 7c)) E 0.
Also by Slutsky's theorem.
OpSPS - 0c + WT (fRu nc.) P 0",
(3)
which completes the proof of consistency.
By multivariate delta methods, denoting h[[x, y.21") x wp (z - y), we have Vh([x,,,2]] 2)²) =
[1, wo, Wo). therefore by the consistency of Wo.
02 0" nC 17 iuu
ne /2/1
1= + w (Nu -7c]]
(4)
BNO (o
fc)), (5)
which completes the proof of asymptotic normality.
Denote V( 7c)..k as the k-th column of V(θ2. 7c), the k-th diagonal element of
(V (V mc) is quadratic form
v(Oc, (V + p (()) ( -V 7c)...
(6)
Here, by our assumption, (V + ((u)) is positive definite matrix. Therefore, quadratic -V is non-negative and is zero if only all ele- ments of ((2,, 7c);, is zero, which completes the proof of element-wiss variance reduction.
form V
O
A.2
Proof the Proposition
Proof. Given
0c 0*
)
v(θc) V(0c, nc)
v(θc, Nc) V(ñc)
0
} (7)
2/2
πc n
N
OK
O
iuu n
pV(fuu
the asymptotic variance of ((()) O(w) 0c wT (fu nT)) is
v(Ōww) v + T (fcc)w + wT (V - 2w V nc)
(8)
We first define the M-estimation (Z-estimation) problem. The goal is to estimate a K-dimensional parameter 0² defined through an estimating equation
( (Y,YY,X)))))) X; 0)} = 0,
(9)
where :b( 0) a user-defined function. By the theory of Z-estimation and a recent paper on ML-
assisted inference [35), we have V(0z) = Ag- M, V Nc) = A, M, V(ñc) =
A and V(ñu) A, M Here, M Var/ Y X; M2
15

---

(ar X; M. Var,lz/ (f. X; 0) M = (
n
BY X; //00,, A,, - B. X; n)/ /mn, and p= N
Rewritten V ((((w)) using the above notation, we have
v(Ō(w)) A M, A A M, Ag
A A2 " + pw A -
-ww AA A
M
(10)
~ (M2 + PM²) 2w A, M
(11)
Plug
w = w = (Var(fic) + Var(frr) Cov(Oc: Tc)
(12)
(V(ñic) (f (V 7c.)
(13)
M - A, -------MAA M.
(14)
=M + pM3) -M,)) AM
(15)
into the equation above, we have
v(θ(wo)) = Ag. A Ag. Az! A, (A, (M,) (A,A))
(16)
Ag. M A + pMa) -Ma)
"M -1
(17)
A {Mi Mf + PM3) -Ma) Ag
(18)
To connect our protocol with existing methods, we define
[ww) A M, Ag + W A + TT PA M3Ag.w Ag.
(19)
wi. == arg min Tr(I(w)rr) where War [witr....,, W E ER
(20)
w (4 kl RRK where weeck arg ming Elwelo)kit
(21)
By the theory of PSPA, PPI++, and PPI paper 3
35
we have
n 1/ OPsPA - 0*) 4 (O E (ddiag(wee))
(22)
n /2 Oppi.. - -0") 2, (0 [(diag(w)))
(23)
n 1/2 (0ppp - -Q*) 4 NO0, E(diag(1))
(24)
Based the proof of Theorem
have the
n (0(3) 0-) & (0, A M + + M,,,, + pw" M 2w A A
(25)
Plug in with ((ag((w.)))))) Ag- A,, diag(w A An and diag(1 A A,p we have
n 1/2 (d A.,) 0r) 3 N0, E(diag (w)))
(26)
n 1/2 (diag (ddag((w)))) A,)) - 0r) D N/O0, [ddiag(w)))
(27)
n 1/2 (0diagga A,)) 0¹) D A(0 E(diag(1))
(28)
A g- A n. we have PSPA, PPI++ and PPI is asymptotically equivalent to diag(w)) ( C, (iag(w ( C, and C respectively. This completes the proof.
Denote C =
A.3 Proof of Proposition 2
Proof. We apply the first-order Taylor expansion to g(0c, me. nu)) around (0", m. n):
g(0c, 7c, nu) Rs g00,, n) + Vo-g(0", n. n. 0*) (n , ( n) + V V0 (00000'')) n - 77),
(29)
16

---

where we used V, and Vn to denote the gradient of g(0", 77,77) w.r. the first and second respectively. This can be written as a linear function of (0",
g(0c, Tc, Mu) = - + 3 + B nc + Bi nu.
(30)
where μ g(0", 77. Ve g n. m) 2V (0n 17. 7. 31 Ve V00 n. 7), 32 V vn10(0" 7) n). A3 V n. 0" n, 7)) Since we require g(0 ne, nu)
& 0", have μ
31 = 1, and B2 + B3 0. This leads to
g(0c, ne, nM) = 0c -3 (īuu 7c),
(31)
Given
07 0" 1EC 1 Thu
OK
v(0c) V(0c, nc) 0
v(Oc. V(ñic) 0
1/2
D2 N
0K
(32)
1
0k
PV
have
Vg( Tc, Ru)) V(0c) ( + (V
7c),
(33)
which is quadratic function of B₂ and achieves it minimum when Bs (V(ñc)
pV ( Nc) = wh- This completes the proof.
B An example for understanding the difference between PSPS and PPI++
Consider a linear regression with two predictors: Y 01x, 02
The summary statistics for
T
PSPS can be expressed as: [bic. Poc, me, 9izc, Mu.. f1z and unlabeled data.
from linear regression analysis in labeled
For PSPS, since Öpsrs estimator for 01 is Órsps.1 01c
θ1c
Wi
W12
mc Í/cc
+
Wi
W12 W2
the final
82c
W12
W2
W12
Mic C
w
fiuu
W12f22 e + W12ffau-
In comparison, since Öppi.-
θ1c
w
0
in
w
iiu izu
its estimator
O2c
i22
0
for 01 is Oppitr,1 0ic whic wrnu-
Since êpers.1 involves two zero aurro-aummenattionn terms (i.e., -w1 -wi wwwccccuuuuu wrip and -w12f2c + w12fau) its asymptotic variance should be less than or equal to that PPI++ with one augmentation term. Therefore, PSPS borrows information from both coordinates, but PPI++ is restricted to information from only the first coordinate. Although the PPI++ can be used under a different scalarization, it still contains augmentation term.
C Algorithms for ML-assisted FDR control
Algorithm 2 PSPS-BH for linear regression
Input: Labeled data L (X Ye, .e)) unlabeled data 2 = (Xu: IT), FDR level E (0, 1). Obtain the p-value Pk for features
1,... K by ML-assisted linear regression
PSPS-LR(C,24
2: Sort the p-values in ascending order P(1) S P(2) S ... P(K)
3: Finds the p-value cutoff -BH = P(k); where lo = max {K: 1..... K: ::p S kg
Output: Discoveries Š {k: pk < T BH)
17

---

Algorithm 3 PSPS-knockoff with debiased Lasso
Input:
Labeled data & (X2, Yz, fc), unlabeled data 4 = (Xu, fu). FDR level E (0,
1: Obtain the augmented labeled and unlabeled data as L (Xe. Xx, Yc, and u
(Xu, Xu, fu) where X < knockoff -aaccoff-sample(XX and Xu www.ooof-sammlleAae..
2: Calculate the PSPS debiased Lasso coefficient plass-- t- PSPS (S)S-))aasooC,11)
3: We for h = 1,... K < knockoff -tttisstittisssicccccccco
4: Set the cutoff knockoff = { > '
#(k:W,
I 9)
#(:WW220v1
Output: Discoveries S {k: M& 74 k }
Here, we employ second-order multivariate Gaussian knockoff variables for knockoff- sample and use the difference between the absolute values of the k-th feature and its knockoff coefficient as the coockoff-statisee Alternative choices for these two steps also be readily integrated into our algorithm [26].
D Implementation details
D.1 Simulation
Here, we provide the details for our simulation. All our simulation is run R with version 4.2.1 (2022-06-22) in MacBook Air with an MI chip. For all the simulations, the ground truth coefficients are obtained using 10* samples; A pre-trained random forest with 500 trees to grow obtained from hold-out data. We bootstrap the labeled data for 200 times for covariance estimation. All simulations are repeated 1000 times.
Mean estimation, Linear regression, and Quantile regression: We simulate the data from Y = X1x31 + X2: - X2, X2 + where X; and X2; are independent simulated from (0, 1), 31 v0.8, B2 is set to be the value such that X2,32 + X7,33 + X2,3 explains 81% of the outcome variance, and e, is simulated from a mean zero normal We use X1 and X2; features to predict
distribution with variance such that Var(Y,) the Y, in the random forest. We consider labeled data with 500 individuals, and unlabeled data with sample size 1000, 2500, 5000, or 10000.
Logistic regression: We simulate the data from Y, = 1(; median (Yi)), where Y, X1,31 X2, 32 32 + Eg where X11 and X2: are independent simulated from (O, 1), 31 v0.08, A2 is set to be the value such that X2.32 + X7,32 + X2,02 explains 81% of the outcome variance, and is simulated from a mean zero normal distribution with variance such that Var(Y;) = 1. We use X1 and X2; features to predict the Y, in the random forest. We consider labeled data with 500 individuals, and unlabeled data with sample size 1000, 2500, 5000, or 10000.
Instrumental variable (IV) regression: We simulate the data by
Z4 ~ N(0,
(34)
X1: = 0.42, + 5,,8, ~ N.0,0.84),
(35)
X2: 0.3Z, -0.8Y, Ys where 7/8 ~ .N.O. such that Var(X2) = 1,
(36)
Y = v0.08X1+ + es, where e, ~ NO T.), such that Var(Y,) =
(37)
We use X1; and X2, features to predict the Y, in the random forest. We consider labeled data with 500 individuals, and unlabeled data with sample size 1000, 2500, 5000, or 10000. The Z, is used as instrument for X1a-
Negative binomial (NB) regression: We simulate the data by
X1 ~ 10, Xxx ~ (O,
(38)
#4 exp( v..(vv + 0.8.2;))
(39)
= NegativeBinomial (s k. = (1s), where s is the dispersion parameter and μ the rate.
(40)
18

---

We use X1; and X2s as features to predict the Y; in the random forest. We consider labeled data with 500 individuals, and unlabeled data with sample size 1000, 2500, 5000, or 10000.
Debiased Lasso: We simulate the data by
X X2001 (O 1)
(41)
0.9
0.,..., .,. ......,15
116....
,2000
(42)
v15
130
Y, Xm + E, ~ T.) such that Var(Y)) =
Σ
(43)
We use Xis...., 200 as features to predict the Y; in the random forest. We consider labeled data with 100 individuals, and unlabeled data with sample size 1500, 2000, 2500, or 3000.
Wilcoxon rank-sum test: We simulate the data by
X1s ~ (0r5o X2, N(,
(44) (45)
Y; = 31X 82X2, + X2 + Ei, N.),
where 31 = V0.0I to power simulation and 31 = for type-I error simulation, 33 set to be the value such that 32X2, - B2X2, explains 81 % of the outcome variance, and is set to be the value such that Var(Y,) = O. use Xıx and X2; as features to predict the Y; in the random forest. We consider labeled data with 500 individuals, and unlabeled data with sample size 1000, 2500, 5000, or 10000.
Benjamini-Hochherg (BH) procedure: We K 150 generate the features indepen- dently from (0, Σ), where E is symmetric Toeplitz matrix that has the structure:
0
PP1
1
PP-2
Σ
(46)
pP
The correlation is set to be 0.25. We then simulate the outcome Y, == 150 Xgaßa Bg Ci. where we randomly sample 15 3, to be 0.15 and let all other remaining 3x to be 0. e; is simulated from mean-zero normal distribution with variance
the value such that Var(Y;) 1. We further generate Z; = 0.9Y, + Σ Xs,0x + where 0g = 0.15 for all I = 1, ..... 150. We use Z; as features to predict the Y; in the random forest. We consider labeled data with 500 individuals, and unlabeled data with sample size 5000.
knockoff:
We used the same setting as described in the BH procedure above to generate the data. The only difference is that we set 9g 0.5 for features associated with the outcome and considered labeled data consisting of 100 individuals, along with unlabeled data comprising sample size of 1000.
D.2 Identify vQTLs for bone mineral density
Our prediction pipeline comprises two components: prediction for unlabeled data and prediction for labeled data. To predict bone mineral density in unlabeled data, we first selected predictive features by 1) calculating the correlation of bone mineral density with 466 other variables (sample size 200,000 from UK Biobank) using labeled data and 2) selecting the top 50 variables with the highest correlations as inputs for the SoftImpute algorithm [00 to predict bone mineral density in the unlabeled data. For the labeled data, we employ similar approach but incorporate 10-fold cross-validation to prevent overfitting. We select the predictive variables and train the SoftImpute model using 90% of the labeled data. We then perform predictions on the remaining 10% in each fold and repeat this process 10 times across all folds.
19

---

E Supplementary figures and tables
BH
b
Knockoff
BH
d
Knockaff
1.00
1.0E
1:0
1.0
Estimateed Power
Estimateed Power
Estimtted
Estimeted
8/30
0.35
0.30
0.15
0.05
0.30
0.15
0.05
0.10
0.15
0.05
0.10
0.15
Nominal FDR level
Nominal FOR level
Nominal FDR level
Nominal FDR level
Methods
PSPS
Classical
Imputatio-kkase
Figure E.1: Simulation for FDR control. Panel a-b shows the estimated FDR level given the expected
FDR. Panel c-d shows the power.
vQTLs for DXA-BMD
10
Chromrosame
Figure E.2: Manhattan plot of vQTLs for bone mineral density. The X-axis represents chromosomes (CHR), plotted by base pair positions (BP). Each point on the plot indicates a single nucleotide polymorphism (SNP). The Y-axis depicts -log (0pp-lle.s).
CHR
BP
SNP
A1
A2
EAF
BETA
SE
p-value
FDR
205359339
rs12139623
T
C
0.105
0.063
0.012
7.2e-08
0.033
14359045
18552582509
0.213
0.046
0.009
4.5e-07
0.045
132568586
1s79089873
G
0.073
-0.068
0.014
5.6e-07
0.049
9
11587905
rs 1s146938822
A
G
0.022
-0.134
0.024
4.0e-08
0.033
15
47672201
rs281258
0.393
-0.04
0.007
5.8e-08
0.033
Table E. Significant vQTLs for bone mineral density. Abbreviations: CHR, Chromosome; BP, Base
Pair: SNP, Single Nucleotide Polymorphism; A1, Allele (Effect Allele): A2, Allele 2 (Non-effect Allele): EAF, Effect Allele Frequency; BETA, Effect Size (Beta Coefficient); SE, Standard Error;
FDR, False Discovery Rate.
20

---

Method
Linear regression
Logistic regression
PSPS
I.62s
8.27s
PPI
0.024s
0.032s
PPI++
0.031s
0.077s
PSPA
0.049s
0.034s
Table E.2: Runtime experiments. Utilizing a dataset with 500 labeled and 10,000 unlabeled data points, PSPS required 1.62 seconds for linear regression and 8.27 seconds for logistic regression using 200 bootstrap resampling. The computation of one-step debiasing using summary statistics alone took 0.032 seconds for linear regression and 0.033 seconds for logistic regression. Current methods, which estimate asymptotic variance via the closed form derived by the Central Limit Theorem instead of resampling, ranged from 0.024 0.049 seconds for linear regression and 0.032 to 0.077 seconds for logistic regression.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provided extensive theoretical and experimental evidence support the main claims in our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations and future direction our paper Conclu- sion section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex. system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: The assumptions and complete (and correct) proof can be found in Section 3
and Appendixx A] respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes]
Justification: The detail the experimental can be found in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the code in the Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/pplile. wttp:ccpppppccceeeeeeeeee..c.ett httaa//gggeeee/eeeuuebiisssssss (Codeeoomdissssseeeeeeee ssilsiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The detail of the experimental can be found in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We reported the confidence interval width and p-value for statistical signifi- cance throughout the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The detail of the experiments compute resources can be found in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have followed the code Of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the positive societal impacts our method is accelerating scientific in the abstract and introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper introduced a statistical method and therefore poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The data collection from UK Biobank (UKB) was approved by UKB's Re- search Ethics Committee. Approval the UKB individual-level data in this work was obtained by the authors of this paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
27

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "0R1fxSSqxA", "reviewer": "Reviewer_TkNf", "summary": "The paper introduces a task-agnostic approach to inference with machine learning predictions. The basic idea follows a similar recipe as prediction-powered inference and related recent papers, but it makes use of resampling instead of the CLT with a plug-in estimate of the asymptotic covariance to avoid relying on analytical problem-specific derivations and expressions.", "questions": "- Can you elaborate on the point about the privacy-preserving feature of your method? I didn't understand the main point. A particular use case would be helpful.\n- I'm surprised that PSPS and PPI++ are not getting the same interval widths in linear and logistic regression in Figure 2. Is it because you are tuning PPI++ for the trace objective, as discussed above? The two methods (assuming the right tuning) should have exactly the same asymptotics in this problem. This should also be made clear in words.\n- I'm curious about the details behind the real-data application. Can you elaborate on how you computed the predictions and how you used cross-validation to avoid overfitting?", "limitations": "I don't think a further discussion is necessary.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper effectively demonstrates the applicability of the method beyond M-estimation (which is what most previous papers have focused on), giving several important applications. The method is simple and elegant. The real-data application is very compelling.", "weaknesses": "I agree with your point that the recent methods require task-specific derivations. However, it's worth noting that, since for M-estimators we know we get asymptotic normality, we can use the same estimators but instead of deriving the asymptotic variance through the CLT we can use resampling to estimate the asymptotic variance. This is just to say that for M-estimators we can use the old estimators and get inference without problem-specific derivations. (Your other criticisms still apply.)\n\nIn L144 it says that resampling-based inference focuses on bias and variance estimation. I don't really agree with this. Plenty of resampling-based inference focuses on type I error control (confidence intervals and p-values).\n\nThe mean-estimation result in Section 2.2 is completely borrowed from prior work. Please make this clear. Otherwise it looks like these are new results in this paper.\n\nIn Algorithm 1 it is unclear what the actual output is. Please write it out using mathematical symbols.\n\nRegarding Proposition 2, I would say that the claim looks a bit too strong. In PPI++ the authors state that they give the example with minimizing the trace as an example, but that other scalarizations of the covariance are clearly possible. Your Proposition 2 just chooses a different scalarization for the comparison so it is clearly better than the trace example from PPI++. But the PPI++ argument would give the same asymptotic variance under your scalarization.\n\nIt would be helpful to give an example of when the p-values in Proposition 3 will be PRDS.\n\nSome stylistic comments:\n- The grammatically correct way to spell the title would be \"Task-Agnostic Machine-Learning-Assisted Inference.\"\n- in abstract: constraints -> constrains\n- Very often throughout the paper you say \"standard error\" but you write the variance. For example, see L68 or L73. Please be consistent: either say variance and write Var(...) or write out the standard error.\n- In L75, you are clearly describing a particular class of ML-assisted inference methods. Not every method that is ML-assisted follows that description. Add references so it's clear what you are referring to.\n- L91: motivated the observation -> motivated by the observation\n- L99: inputs -> inputting\n- There are other typos and minor stylistic issues. Please go through the paper carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**: We first want to highlight that current methods and their implementations typically estimate asymptotic variance using the CLT rather than through resampling (for example, `Angelopoulos et al., 2023` PPI and PPI++). In addition, while resampling-based approaches can bypass the derivation of asymptotic variance, task-specific derivations for the loss function of a new ML-assisted M-estimator are still essential to obtaining the point estimator, which precedes resampling-based estimation of its uncertainty.\n\nWe also want to add that the “task-specific derivations” mentioned throughout our paper were not only referring to statistical tasks, but also scientific tasks. Real-world data analysis in any scientific discipline often involves conventions and nuisances that require careful consideration. For example, our work is partly motivated from genome-wide association studies (GWAS). Statistically, GWAS is a linear regression that regresses an outcome (e.g., height) on many genetic variants. While the regression-based statistical foundation is simple, conducting a valid GWAS requires accounting for numerous technical issues, such as sample relatedness (i.e., study participants may be genetically related) and population structure (i.e., unrelated individuals of the same ancestry are both genetically and phenotypically similar, creating confounded associations in GWAS). Sophisticated algorithms and software have been developed to address these complex issues (`Mbatchou et al. 2021`). It will be very challenging if all these important features need to be reimplemented in an ML-assisted GWAS framework. With our PSPS protocol, researchers can utilize existing algorithms and software that are highly optimized for genetic applications to perform ML-assisted GWAS. This adaptability is not just limited to GWAS but is a major feature of our approach across scientific domains. A main result of this paper is that PSPS enables researchers to conduct ML-assisted inference using well-established data analysis pipelines.\n\n**W2**: We have revised the texts to “whereas resampling-based inference focuses on bias and variance estimation, and type-I error control.”.\n\n**W3**: We have added relevant citations when we discuss the mean-estimation result in Section 2.2.\n\n**W4**: We have revised Algorithm 1 following the suggestion.\n\n**W5**: We would like to clarify that the asymptotic variance of PSPS will always be lower than that of PPI++, irrespective of the scalarization method used. The reason is that the weights matrix in PPI++ is a special case of the more general weights matrix used in PSPS, and we have demonstrated in Proposition 2 that the weighting matrix in PSPS is optimal. The weighting matrix $\\omega_0$ in PSPS is a Q×Q matrix with no constraint, where Q is the number of dimensions for the parameters. In contrast, PPI++ constrains its matrix to be diagonal with the same diagonal elements. Therefore, PSPS enables information sharing across different parameter coordinates, enhancing estimation precision. The choice of the weighting matrix in PSPS also facilitates element-wise variance reduction: each diagonal element of the variance-covariance matrix is reduced. In contrast, the single parameter in PPI++ can only target overall trace reduction or variance reduction of a specific element.\n\nTo provide further intuition, we consider a linear regression with two predictors:  $Y \\sim \\theta_1 X_1 + \\theta_2 X_2$. The summary statistics for PSPS can be expressed as: $[\\hat{\\theta} _{1L}, \\hat{\\theta} _{2L}, \\hat{\\eta} _{1L}, \\hat{\\eta} _{2L}, \\hat{\\eta} _{1U}, \\hat{\\eta} _{2U}]^T$. For PSPS, since $\\hat{\\theta} _{\\text{PSPS}}= \\begin{bmatrix}\\hat{\\theta} _{1L} \\\\\\ \\hat{\\theta} _{2L} \\end{bmatrix} - \\begin{bmatrix}w _1 & w _{12} \\\\\\ w _{12} & w _2 \\end{bmatrix}\\begin{bmatrix}\\hat{\\eta} _{1L} \\\\\\ \\hat{\\eta} _{2L} \\end{bmatrix} + \\begin{bmatrix}w _1 & w _{12} \\\\\\ w _{12} & w _2 \\end{bmatrix}\\begin{bmatrix}\\hat{\\eta} _{1U} \\\\\\ \\hat{\\eta} _{2U} \\end{bmatrix}$, the final estimator for $\\theta_1$ is $\\hat{\\theta} _{\\text{PSPS}, 1} = \\hat{\\theta} _{1L}-w _1 \\hat{\\eta} _{1L}+w _1 \\hat{\\eta} _{1 U}-\\mathrm{w} _{12} \\hat{\\eta} _{2L}+\\omega _{12} \\hat{\\eta} _{2U}$.\n\nIn comparison, since $\\hat{\\theta} _{\\text{PPI++}}= \\begin{bmatrix}\\hat{\\theta} _{1L} \\\\\\ \\hat{\\theta} _{2L} \\end{bmatrix} - \\begin{bmatrix}w & 0 \\\\\\ 0 & w \\end{bmatrix}\\begin{bmatrix}\\hat{\\eta} _{1L} \\\\\\ \\hat{\\eta} _{2L} \\end{bmatrix} + \\begin{bmatrix}w & 0 \\\\\\ 0  & w \\end{bmatrix}\\begin{bmatrix}\\hat{\\eta} _{1U} \\\\\\ \\hat{\\eta} _{2U} \\end{bmatrix}$, its estimator for $\\theta_1$ is $\\hat{\\theta} _{\\text{PPI}++, 1} = \\hat{\\theta} _{1L}-w \\hat{\\eta} _{1L}+w \\hat{\\eta} _{1U}$.\n\nSince $\\hat{\\theta} _{\\text{PSPS}, 1}$ involves two zero-augmentation terms (i.e., $-w _1 \\hat{\\eta} _{1L}+w _1 \\hat{\\eta} _{1 U}$ and $-\\mathrm{w} _{12} \\hat{\\eta} _{2L}+\\omega _{12} \\hat{\\eta} _{2U}$ ), its asymptotic variance should be less than or equal to that of PPI++ with one augmentation term. Therefore, PSPS borrows information from both coordinates, but PPI++ is restricted to information from only the first coordinate. Although the PPI++ can be used under a different scalarization, it still contains one augmentation term. Only in a one-dimensional parameter estimation task, PPI++ and PSPS will have the same asymptotic variance. We have clarified this in the revised manuscript. We have also added the example above and an example for a one-dimensional parameter estimation task to clearly state the difference between PSPS and PPI++.\n\n**Due to the character limit, we have placed our response to the remaining concerns in the comment section of the review.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rf1Z9kTGQO", "reviewer": "Reviewer_hhpT", "summary": "The paper proposes a novel statistical framework for ML-assisted inference. It describes how labeled data, together with unlabeled data and a pre-trained ML model, can be used for statistical inference. The paper establishes the asymptotic properties and optimality of the framework and then evaluates it empirically on simulated and real datasets.", "questions": "- What are the limitations of the proposed framework?", "limitations": "The authors state that the limitations are discussed in the Conclusion section of the paper, but it seems they are not discussed there.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written.\n- Framework is flexible. It can work with almost any established data analysis routine and machine learning model.\n- The framework is justified both theoretically and empirically.", "weaknesses": "- The limitations could have been discussed more thoroughly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses, Questions, and Limitations:**\n\nThank you for the suggestion. One limitation is the computational burden of the naive bootstrap approach. In our original submission, we discussed the future direction of improving the speed of resampling. In the revised manuscript, we have conducted additional experiments to compare the computational efficiency of PSPS with existing methods.  Utilizing a dataset with 500 labeled and 10,000 unlabeled data points, PSPS required 1.62 seconds for linear regression and 8.27 seconds for logistic regression using 200 bootstrap resampling. The computation of one-step de-biasing using summary statistics alone took 0.032 seconds for linear regression and 0.033 seconds for logistic regression. Current methods, which estimate asymptotic variance via the closed form derived by the Central Limit Theorem instead of resampling, ranged from 0.024 to 0.049 seconds for linear regression and 0.032 to 0.077 seconds for logistic regression. \n\n| Method | Linear regression  | Logistic regression |\n| ------------- | ------------- |------------- |\n| PSPS | 1.62s | 8.27s |\n| PPI  | 0.024s  | 0.032s |\n| PPI++  | 0.031s  | 0.077s |\n| POP-Inf  | 0.049s |0.034s |\n\nThese experiments demonstrate that while PSPS is slower than current methods due to its reliance on resampling to estimate the variance, the overall speed remains reasonable and fast. One potential solution to further improve speed could involve adopting more advanced methods for faster resampling such as Bag of Little Bootstraps (`Kleiner et al. 2014`). We have discussed this more thoroughly in the manuscript.\n\n**Final note**: We are excited that you find our method flexible and justified with both theoretical and empirical analysis. If you have any further questions, please do not hesitate to let us know. If our responses have resolved your concerns, we kindly request you to consider increasing your score and championing our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwyH8EC2pH", "reviewer": "Reviewer_DsrD", "summary": "This paper proposes a new unified framework for ML-assisted inference that reduces the general problem to essentially one of estimating normal means, and then applies simple operations (and a bootstrap step) to solve the normal means problem. In addition to the unifying framework’s simplicity, a key result is that the asymptotic efficiency of the proposed estimator dominates that of existing works under mild conditions.", "questions": "In what sense is your method more task-agnostic than others? It seems to still require \\mathcal{A}, which is task-specific. The authors mention in lines 129-131 that works have been developed that are general to M-estimators, but many (all?) the simulation tasks in Figure 3 which claim to be ones which “have not been implemented for ML-assisted inference” are M-estimation tasks (or reduce to them), so why couldn’t other methods have been applied to them? Explaining this more clearly would raise my score.", "limitations": "The authors have not addressed computational efficiency, which is a critical property—how fast is it relative to the other methods under consideration? Addressing this would raise my score.", "rating": 7, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "1. The problem is an important one\n2. The main idea presented in section 2.2 is quite compelling, reducing the ML-assisted inference problem into basically a means estimation problem, and also finding the optimal weighting to combine the components\n3. The results about dominating existing approaches in terms of statistical efficiency are exciting", "weaknesses": "1. Theorem 1 doesn’t give asymptotically valid inference, since it doesn’t address variance estimation. If the asymptotic variance were known, this result would be sufficient, but it is not, and what is used instead is an estimate of the variance, which must be proved to be consistent in order to conclude asymptotic validity of Algorithm 1’s confidence interval and p-value. This oversight is relevant in other areas, such as eq (35) in the appendix is unjustified and does not follow from eq (34).\n2. Prop 1 as stated does not say that these methods all have the same asymptotic variance, which seems to be the implication of the rest of section 3.2 after Prop 1. It seems the proof does prove the right thing, it’s just the statement is wrong (it is missing \\sqrt{n} multiplying the LHS of each of the three limits).\n3. Proof of prop 4 seems to just be rehashing the proof of knockoffs, except for a single sentence in lines 531-532 which simply states, without proof, the most important part, and the only part that involves the proposed method.\n\nIn my opinion, the paper cannot be published without these issues addressed, hence my current score. Item 2 seems to just be a typo, but item 1 is absolutely critical and central, and item 3 is just not rigorous, so it should either be fixed or deleted (personally, I don’t think section 4 strengthens the paper much, since it’s just applying known FDR control techniques to the output of the proposed method, so the validity should just follow from the validity of the proposed method and that of the known FDR control techniques). If they are addressed (and no other reviewers raise other issues that cause me concern), my score would go up quite a bit, as I think this paper has significant strengths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:**\nThank you for the comment. We have addressed the variance estimation in Theorem 1 by adding \n\n\"\nWith $\\hat{\\mathbf{V}}(\\hat{\\theta} _{\\mathrm{PSPS}}) \\xrightarrow{P} \\mathbf{V}(\\hat{\\theta} _{\\mathrm{PSPS}}), \\lim _n \\mathbb{P}(\\theta _k^* \\in \\mathcal{C} _{\\alpha, k}^{\\mathrm{PSPS}})=1-\\alpha$ where $\\mathcal{C} _{\\alpha, k}^{\\mathrm{PSPS}}=(\\hat{\\boldsymbol{\\theta}} _{\\mathrm{PSPS} _k} \\pm z _{1-\\alpha / 2} \\sqrt{\\hat{\\mathrm{V}}(\\hat{\\theta} _{\\mathrm{PSPS}, \\mathrm{k}}) / n})$.\n\"\n\nHere $\\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\theta}} _{\\text {PSPS}})=\\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\theta}} _{\\mathcal{L}})-\\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\theta}} \n _{\\mathcal{L}}, \\hat{\\boldsymbol{\\eta}} _\\mathcal{L})^{\\mathrm{T}}(\\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\eta}}  _\\mathcal{L})+\\rho \\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\eta}} _\\mathcal{U}))^{-1} \\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\theta}} _\\mathcal{L}, \\hat{\\boldsymbol{\\eta}}  _\\mathcal{L})$ can be obtained by applying the algebraic form of  $\\mathbf{V}(\\hat{\\boldsymbol{\\theta}} _{\\text {PSPS}})$ using the bootstrap estimators of $\\mathbf{V}\\(\\widehat{\\boldsymbol{\\theta}} _{\\mathcal{L}}\\), \\mathbf{V}(\\hat{\\eta} _{\\mathcal{L}}), \\mathbf{V}(\\widehat{\\boldsymbol{\\theta}} _{\\mathcal{L}}, \\hat{\\eta} _{\\mathcal{L}}) \\text {, and } \\mathbf{V}(\\hat{\\eta} _{\\mathcal{U}})$. Since bootstrap estimators of variance is consistent under certain regularity conditions, by Slutsky’s theorem, $\\hat{\\mathbf{V}}(\\hat{\\boldsymbol{\\theta}} _{\\text {PSPS}})$ is consistent for $\\mathbf{V}(\\hat{\\boldsymbol{\\theta}} _{\\text {PSPS}})$.\n\n**W2:**\nWe have added $\\sqrt{n}$ to Prop1. The new Prop 1 is \n$$n^{\\frac{1}{2}}\\left(\\widehat{\\boldsymbol{\\theta}}\\left(\\operatorname{diag}\\left(\\boldsymbol{\\omega} _{\\text {ele }}\\right) \\mathbf{C}\\right)-\\widehat{\\boldsymbol{\\theta}} _{\\text{POP-Inf}}\\right) \\xrightarrow{D} \\mathbf{0},\nn^{\\frac{1}{2}}\\left(\\widehat{\\boldsymbol{\\theta}}\\left(\\operatorname{diag}\\left(\\boldsymbol{\\omega} _{\\text {tr }}\\right) \\mathbf{C}\\right)-\\widehat{\\boldsymbol{\\theta}} _{\\text{PPI}++}\\right) \\xrightarrow{D} \\mathbf{0},\nn^{\\frac{1}{2}}\\left(\\widehat{\\boldsymbol{\\theta}}\\left(\\operatorname{diag}\\left(\\bf{1}\\right) \\mathbf{C}\\right)-\\widehat{\\boldsymbol{\\theta}} _{\\text{PPI}}\\right) \\xrightarrow{D} \\mathbf{0},\n$$\nwhich says these methods all have the same asymptotic variance.\n\n**W3:**\nThe statement in lines 531-532 can be proved below: For null $k$, since debiased lasso is consistent, the $k$-th debiased Lasso coefficient converges to 0. Therefore, $W_k$ is (asymptotically) symmetric around 0. Therefore, $Z_k=1(W_k)$ (asymptotically) follows the Bernoulli distribution with a probability of 0.5. There is a typo in the original proof though, that only requires the $Z_k$ (asymptotically) follows Bernoulli(0.5) for null feature $k (\\beta_k=0)$. \n\n**Overall:**\nWe really appreciate the reviewer for spotting these issues in our original submission. We have addressed these issues in details above. We have also changed Section 4 to a remark in Section 3 to indicate that the results for PSPS can be directly combined with existing FDR control techniques to achieve ML-assisted FDR control.\n\n**Q:**\nAmong the tasks illustrated in Figure 3, quantile regression and negative binomial regression are M-estimation problems, and the principles for applying ML-assisted inference to these tasks are available although no specific derivations and software implementations have been made available for broader use. Instrumental variable (IV) regression, while technically an M-estimation task, is typically solved using two-stage least squares, which are non-trivial to implement under a ML-assisted inference framework. debiased Lasso and Wilcoxon Rank-Sum test do not conform to minimizing a loss function. Hence, mathematical principles for the ML-assisted inference of these tasks are still underdeveloped, with no existing software implementations. \n\nPSPS is more task-agnostic than other methods in three aspects:\n* For M-estimation tasks\n  * Current methods: Currently, only mean and quantile estimation, as well as linear, logistic, and Poisson regression, have been implemented in software tools and are ready for immediate application. For other M-estimation tasks, task-specific derivation of the ML-assisted loss functions and asymptotic variance via the central limit theorem are necessary. After that, researchers still need to develop software packages and optimization algorithms to carry out real applications. \n  * In contrast, PSPS only requires already implemented algorithms and software designed for classical inference using labeled data. For example, implementing negative binomial regression with PSPS is straightforward using existing functions:\n```\nfrom statsmodels.formula.api import glm\nmodel = glm('count ~ x1 + x2', data=df, family=NegativeBinomial()).fit()\n``` \nin python\n\nor\n```\nlibrary(MASS)\nglm.nb(count ~ x1 + x2, data = data) \n```\nin R\n\n* For problems that are not considered M-estimation but have asymptotically normally distributed estimators, only PSPS can be applied and all current methods would fail. The principles that facilitate ML-assisted M-estimation are not applicable for these non-M-estimation tasks.\n\n**Due to the character limit, we have placed our response to the remaining concerns in the comment section of the reviews.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3pg9nhVNH", "reviewer": "Reviewer_5S3y", "summary": "This paper proposes a post-prediction inference solution (PSPS) that can be adapted into various established data analysis routine and delivers valid and efficient inference for most ML models. In particular, the paper uses both labelled and unlabelled data to derive estimators that are consistent and efficient and only utilizes 1st and 2nd order summary statistics.", "questions": "Please see above.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper provides a well-rounded analysis on the proposed method with sensitivity analysis on distributions shifts, to violations of independence assumption between labelled and unlabelled data and to better statistical power in false discovery control.", "weaknesses": "- The paper claims the proposed method offers better statistical power in various statistical tasks from mean estimation to quantile regression through various numerical experiments in Section 5.1. However, it is unclear how much of the statistical power is offered by the nature of transductive inference and how much is offered by the proposed protocol. For example, [1] shows that incorporating unlabelled data achieves lower test error than purely labelled data already. Could the author demonstrate the unique advantages offers by the proposed method?\n- The protocol proposed relies on assumptions on 1) i.i.d. distributions between labelled and unlabelled data and 2) finding an algorithm applied to labeled data returns a consistent and asymptotic normally distributed estimators. I am wondering how easy is it to find an algorithm that can produce such estimators? \n- The authors list one of the key features of the method is privacy-preserving, but did not discuss this point in the remaining paper. \n\n[1] Chapelle, O., Vapnik, V., & Weston, J. (1999). Transductive inference for estimating values of functions. Advances in Neural Information Processing Systems, 12.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:**\nReference [1] focuses on using unlabeled data to improve the estimation of regression function (i.e. E[Y|X]). This is a classic machine learning prediction problem which leverages semi-supervised learning to enhance prediction accuracy. In contrast, our study focuses on improving estimation and statistical inference using unlabeled data, emphasizing the need for estimator consistency and the validity of confidence intervals. This is a very new topic (`Angelopoulos, Anastasios N., et al. \"Prediction-powered inference.\" Science (2023)`) in the machine learning field. Because of this, we believe our paper and [1] address fundamentally different problems. In real-world scientific applications, many issues pertain to statistical inference (e.g., estimating the effects of genetic variants on height). The methods in [1] are not suitable for such applications, whereas our methods are specifically designed to tackle these issues.\n\nIn our initial submission, we demonstrated the validity and superior statistical efficiency of our method over existing approaches (Section 3.2), substantiated by theoretical guarantees:\n1. Theorem 1: PSPS achieves element-wise asymptotic variance reduction compared to classical statistical inference (based solely on labeled data) while ensuring estimator consistency.\n2. Theorem 1: PSPS guarantees valid inference (accurate confidence interval coverage) compared to imputation-based inference (which relies solely on machine learning predictions using unlabeled data).\n3. Propositions 1 and 2: PSPS has no larger asymptotic variance than existing ML-assisted inference methods that provide consistent estimators.\n\nFurthermore, a key feature of our method is its independence from task-specific derivations and implementations, allowing it to address a broader range of statistical problems compared to existing methods. \n* For M-estimation tasks, currently, only mean and quantile estimation, as well as linear, logistic, and Poisson regression, have been implemented in software and are ready for immediate application. For other M-estimation tasks, task-specific derivation of the ML-assisted loss functions and asymptotic variance are necessary. Next, researchers still need to develop software packages to carry out real applications. In contrast, PSPS only requires already implemented software designed for classical inference using labeled data.\n* For problems that are not considered M-estimation but have asymptotically normally distributed estimators, only PSPS can be applied and all current methods would fail. The principles that facilitate ML-assisted M-estimation are not applicable for these non-M-estimation tasks.\n\nTo summarize, our method, designed for estimation and statistical inference, addresses a different problem than [1], which focuses on prediction. We have also demonstrated the substantial advantages of our approach compared to existing methods for ML-assisted statistical inference through both theoretical and experimental analyses. We have also added the semi-supervised learning literature in the related work to avoid confusion.\n\n**W2:**\nConsistent and asymptotically normally distributed estimators are the fundamental properties of an estimator in statistical sciences, and significant research efforts have been devoted to developing such estimators for a wide range of statistical tasks. Generally, if a statistical task allows the application of the Central Limit Theorem. the resulting estimator is likely to be asymptotically normal. Specifically, M-estimator that calculates an estimator through empirical risk minimization, and U-statistics that can be expressed as averages of a symmetric function applied to subsets of a sample are both consistent and asymptotically normally distributed estimators. Common examples of M-estimators include but are not restricted to maximum likelihood estimation, mean and median estimations, linear regression models, generalized linear models, and quantile regression. U-statistics include but is not restricted to variance estimation and Wilcoxon-Rank Sum statistics. In summary, it is fairly straightforward to identify an algorithm capable of producing such estimators for a wide range of statistical tasks.\n\n**W3:**\nThe “privacy-preserving\" feature of PSPS refers to the fact that we only require summary statistics as input for inference, rather than individual-level raw data (features X and label Y). This terminology is commonly used in human genetics, healthcare, and multi-center electronic health record analysis to describe methods that enable statistical inference without directly accessing personal data. This approach is analogous to federated learning (`Kairouz, Peter, et al. 2021`) in the machine learning literature. For example, consider a scenario where labeled data is in one center and unlabeled data in another, yet researchers cannot access individual-level data from both centers simultaneously. Under such conditions, current ML-assisted inference, which relies on accessing both labeled and unlabeled data to minimize a joint loss function, is not feasible. However, PSPS circumvents this issue by aggregating summary statistics from multiple centers, thus performing statistical inference while upholding the privacy of individual-level data.\n\nWe also acknowledge that in the machine learning literature, \"privacy-preserving\" often specifically refers to techniques like differential privacy. To avoid confusion, we have revised the terminology from “privacy-preserving” to \"federated inference\" and provided an example to illustrate this use case more clearly.\n\n**Final note**: We are excited that you find our methods achieve better statistical power compared with current methods and our strategy to deal with distributional shifts. If you have any further questions, please do not hesitate to let us know. If our responses have resolved your concerns, we kindly request you to consider increasing your score and championing our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for providing valuable suggestions that helped us improve our paper. We are particularly encouraged that the reviewers have found that (i) the problem we study in this paper is important (R-DsrD), (ii) our method is simple and elegant (R- TkNf), (iii) our flexible statistical framework can be applied to almost any established data analysis routine (R-hhpT and R-Tknf) and machine learning model (R-hhpT), (iv) our method is justified both theoretically and empirically (R-hhpT), and the real-data application is very compelling (R-Tknf), (v) our method dominates existing approaches in terms of statistical efficiency (R-DsrD), (vi) the paper provides a well-rounded analysis on the proposed method with sensitivity analysis on distributional shifts (R-5S3y).

In response to the feedback, we have addressed each concern, added new experimental results and clarification, and updated our paper accordingly. 

A summary of our major changes is provided below.
1. We have explained the "task-agnostic" feature of our method.
2. We have clarified the "privacy-preserving" aspects of our method.
3. We have elucidated the connection between our ML-assisted statistical inference and semi-supervised learning in the machine learning literature.
4. We have addressed technical issues related to variance estimation in Theorem 1 and corrected typos throughout the paper.
5. We have included examples to illustrate how our method outperforms existing approaches in terms of statistical efficiency.

A detailed point-by-point response to each reviewer's comments is available in the rebuttal section corresponding to each reviewer.

Meta-review: The paper proposes a framework for statistical inference that combines labeled, unlabeled through a post-inference solution that can be used with different data analysis routines. The reviewers unanimously recommend its acceptance. They noted that the proposed method is simple and flexible, the evaluations were comprehensive through theoretical and empirical analysis and highlighted its application to important areas. I therefore recommend its acceptance.
Decision: Accept (poster)