Enhancing Statistical Validity and Power in Hybrid Controlled Trials: A Randomization Inference Approach with Conformal Selective Borrowing
Authors: Ke Zhu
Shu Yang
Xiaofei Wang
Venue: ICML 2025 poster (2025)
Keywords: causal inference
data fusion
randomization test
real-world data and evidence
small sample size
Primary area: general_machine_learning->causality
TL;DR: This paper proposes a randomization inference framework using conformal inference to selectively borrow external controls, ensuring exact type I error control and valid post-selection inference, with application to a lung cancer trial.

Abstract: External controls from historical trials or observational data can augment randomized controlled trials when large-scale randomization is impractical or unethical, such as in drug evaluation for rare diseases. However, non-randomized external controls can introduce biases, and existing Bayesian and frequentist methods may inflate the type I error rate, particularly in small-sample trials where external data borrowing is most critical. To address these challenges, we propose a randomization inference framework that ensures finite-sample exact and model-free type I error rate control, adhering to the “analyze as you randomize” principle to safeguard against hidden biases. Recognizing that biased external controls reduce the power of randomization tests, we leverage conformal inference to develop an individualized test-then-pool procedure that selectively borrows comparable external controls to improve power. Our approach incorporates selection uncertainty into randomization tests, providing valid post-selection inference. Additionally, we propose an adaptive procedure to optimize the selection threshold by minimizing the mean squared error across a class of estimators encompassing both no-borrowing and full-borrowing approaches. The proposed methods are supported by non-asymptotic theoretical analysis, validated through simulations, and applied to a randomized lung cancer trial that integrates external controls from the National Cancer Database.

Enhancing Statistical Validity and Power in Hybrid Controlled Trials: A Randomization Inference Approach with Conformal Selective Borrowing
Ke Zhu 12 Shu Yang' Xiaofei Wang?
Abstract
1. Introduction
Randomized controlled trials (RCTs) are the gold standard for making causal inferences on the treatment effect of new treatment relative to control treatment. However, large RCTs are often infeasible to conduct in practice when the indications of interest involve rare diseases (U.S. Food and Drug Administration, 2022) or common conditions where few patients are willing to participate due to lack of equipoise (Miller & Joffe, 2011). RCTs in such con- text often lack sufficient statistical power to detect realistic treatment effect sizes. Meanwhile, historical studies or large external databases provide real-world data under control conditions, often referred to as external controls (ECs). By integrating RCT with ECs, hybrid controlled trials have garnered significant interest as an effective approach to en- hance the power of RCTs with small sample sizes. However, most existing methods for hybrid controlled trials rely on model-based or asymptotic p-values, which can lead to in- flated type error rates when the randomized sample size is small, or the model is misspecified. Moreover, since ECs are not randomized, they may systematically differ from randomized controls, even after adjusting for measured con- founders. Directly incorporating these ECs may introduce hidden bias, compromising the validity of the statistical in- ference. Strictly controlling the type error rate in hybrid controlled trials, especially with small sample sizes and unmeasured confounding, remains an open problem.
External controls from historical trials or obser- vational data can augment randomized controlled trials when large-scale randomization is imprac- tical or unethical, such as in drug evaluation for rare diseases. However, non-randomized exter- nal controls can introduce biases, and existing Bayesian and frequentist methods may inflate the type I error rate, particularly in small-sample tri- als where external data borrowing is most criti- cal. To address these challenges, we propose a randomization inference framework that ensures finite-sample exact and model-free type I error rate control, adhering to the "analyze as you ran- domize" principle to safeguard against hidden biases. Recognizing that biased external con- trols reduce the power of randomization tests, we leverage conformal inference to develop an individualized test-then-pool procedure that se- lectively borrows comparable external controls to improve power. Our approach incorporates selec- tion uncertainty into randomization tests, provid- ing valid post-selection inference. Additionally, we propose an adaptive procedure optimize the selection threshold by minimizing the mean squared error across class of estimators encom- passing both no-borrowing and full-borrowing approaches. The proposed methods are supported by non-asymptotic theoretical analysis, validated through simulations, and applied a randomized lung cancer trial that integrates external controls from the National Cancer Database.
To address this problem, we extend the randomization infer- ence framework to hybrid controlled trials. To utilize ECs, we use doubly robust estimator of the average treatment effect (ATH) as the test statistic, which incorporates both RCT and EC data and effectively balances the measured confounders between RCT and EC (Li et al., 2023b). Then, Fisher randomization tests (FRTs) are performed using only the randomization in the RCT. In contrast to the asymp- totic inference Li et al. (2023b), which relies on (i) large sample sizes for both the RCT and EC, (ii) correct specifi- cation of least one of the two nuisance models, and (iii) unmeasured confounders, the FRT strictly controls the type I error rate without requiring any of these conditions, thus achieving model-free, finite-sample exact inference. The validity of the FRT relies solely on the randomization
'Department of Statistics, North Carolina State University, Raleigh, NC 27695, U.S.A. 'Department of Biostatistics and Bioin- formatics, Duke University, Durham, NC 27710, U.S.A.. Corre- spondence to: Shu Yang <syang24@ ncsu..uu>>
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
within the RCT, which is typically well-managed by the study design. Furthermore, we perform power analysis for FRT in hybrid controlled trials and show that incorporating unbiased ECs with correctly specified models can enhance statistical power. However, EC borrowing is not a free lunch, as including biased ECs may diminish power.
pirical MSE of CSB can be approximated leveraging the RCT-only estimator, making the procedure practically feasi- ble, and we provide non-asymptotic excess risk bound for its performance. The advantages of our approach are shown via simulations and a lung cancer RCT with ECs from the National Cancer Database.
The power issue motivates us to develop a method that selectively incorporates unbiased ECs rather than indiscrim- inately borrowing all ECs. Unlike observational studies, where the assumption of no unmeasured confounders is untestable, key advantage of hybrid controlled trials is that the bias in ECs can be identified by comparing EC units to randomized control units. Existing methods miti- gate hidden bias by penalized bias estimation and selective borrowing (Gao al., 2025), where selection consistency depends on asymptotic arguments, potentially leading to inferior performance in small samples.
1.1. Related work
Hybrid controlled trials aim to integrate ECs to boost RCT efficiency (Pocock, 1976). For an overview of RCT and RWD integration, see Colnet et al. (2024), A key challenge is biases in ECs, which stem from factors like selection bias, non-concurrency, and measurement error (U.S. Food and Drug Administration, 2023). Statistically, biases are catego- rized as measured and unmeasured confounding. Measured confounding, or covariate shift, refers to systematic differ- ences in observed covariates between RCs and ECs. To address measured confounding, covariate balancing tech- niques such as matching, inverse propensity score weighting, calibration weighting, and their augmented counterparts can be employed (Li et al., 2023b; Valancius al., 2024; Li & Luedtke, 2023). When there is unmeasured confounding between RCT and EC, rich body of literature addresses the hidden bias through various strategies, including test-then- pool (Viele et al., 2014; Yuan et al., 2019; Li et al., 2020; Ventz et al., 2022; Liu et al., 2022; Yang et al., 2023; Gao & Yang, 2023; Dang al., 2023), weighted combination (Chen et al., 2020; 2021a; Cheng & Cai, 2021: Li et al., 2022; Oberst et al., 2022; Rosenman et al., 2023; Chen et al., 2023; Karlsson et al., 2024), selective borrowing (Chen et al., 2021b; Li et al., 2023a; Zhai & Han, 2022; Gao et al., 2025; Huang et al., 2023), bias modeling (Stuart & Ru-
We propose a novel approach called Conformal Selective Borrowing (CSB), which tests the comparability of ECs and selectively incorporates them using conformal inference (Vovk et al., 2005; Lei et al., 2018). We measure the bias of each EC using score function that can flexibly accom- modate parametric machine-learnin models. We then calibrate this score to conformal p-value, which test the exchangeability of each EC. These conformal p-values are valid in finite samples, distribution- and do not depend on the asymptotic properties of models. CSB offers three advantages: (i) individual borrowing decisions for each EC, (ii) flexibility in using parametric or machine learning mod- els for bias estimation, and (iii) finite-sample guarantees with stable performance in small samples.
In summary. the proposed methods leverage the two key ad- vantages of hybrid controlled trials: (i) randomization within the RCT data allows us to use FRT to control the type I error rate, and (ii) the presence of randomized controls enables us to evaluate bias in ECs using conformal p-values, selectively borrow unbiased ECs, and enhance power. We account for selection uncertainty in FRT and offer valid post-selection inference. Both FRT and CSB are model-free distribution- free, and maintain finite-sample exact properties, allowing them to flexibly incorporate state-of-the ofee-----rrrrr ahe machine learn- ing methods while remaining valid for any sample size or data distribution. To ensure robust performance across vary- ing bias magnitudes, we propose iata-adaptive procedure for determining the selection threshold to minímize the MSE of the CSB estimator. Our MSE-guided adaptive threshold offers key advantages: (i) it improves FRT power over RCT- only analysis when EC bias is negligible or detectable; when the bias is neg-negligible yet difficult to detect, it may lead to power loss, though FRT still maintains valid Type er- ror control; (ii) it enables CSB to serve as both a powerful test statistic and accurate ATE estimator; (iii) the em-
bin, 2008; Cheng et al., 2023; Li & Jemielita, 2023; van der Laan et al., 2024; Yang et al., 2024; Gu et al., 2024), control variates or prognostic adjustment (Yang & Ding, 2020; Guo et al., 2022; Schuler al., 2022; Gagnon-Bartss Bartsch al., 2023), Bayesian methods (Hobbs et al., 2011; Schmidli et al., 2014; Jiang et al., 2023; K Kwiatkowski et al., 2024; Alt et al., 2024; Lin et al., 2024; 2025), and sensitivity analysis (Yi et al., 2023). None of them use randomization inference or conformal inference to address unmeasured confounding in hybrid controlled trials with small sample size.
Randomization inference, introduced by Fisher (1935), provides finite-sample exact p-values for any test statistic and is widely endorsed (Rosenberger et al., 2019; Proschan & Dodd, 2019; Young, 2019; Bind & Rubin, 2020; Carter et al., 2023). Randomization tests are useful for small sam- plc trials or complex designs, including cluster experiments with few clusters (Rabideau & Wang, 2021) and adaptive experiments (Simon & Simon, 2011; Plamadeala & Rosen- berger, 2012; Nair& & Janson, 2023; Freidling et al., 2024). Randomization tests have appeared in regulatory guidance documents to ensure type error rate control in adaptive
2

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
designs when conventional statistical methods fail (Euro- pean Medicines Agency, 2015; U.S. Food and Drug Ad- ministration, 2019; Carter et al., 2023). For overview of randomization inference, see Zhang & Zhao (2023) and Ritzwoller et al. (2024). Nevertheless, the randomization inference hasn't 't been applied to hybrid controlled trials, especially with selective borrowing to address unmeasured confounding.
((a) 11 A
1), a =,
Under Assumption 2.1, T is identifiable based on RCT data. We denote the conditional outcome mean functions by Pa (z) = E(Y X === A = a, S = a 0, 1. We estimate pa (z) and e(x) with only RCT data and denote the estimated functions by Ao, (a) and é(x), respectively. An RCT-only doubly robust estimator of r is
Conformal inference, conformal prediction, is a model- free method providing finite-sample valid uncertainty quan- tification for individual predictions (Vovk et al., 2005), par- ticularly useful in high-stakes scenarios with black-box ma- chine learning models (Angelopoulos & Bates, 2023). Two main applications are most relevant to this paper. The first involves using conformal inference infer individual treat- ment effects (Chernozhukov et al., 2021; Lei & Candès, 2021). The second line is in outlier detection (Guan & Tib- shirani, 2022; Bates et al., 2023; Liang et al., 2024). These studies inspire us to treat biased ECs as outliers and use con- formal p-vuuues to test their exchangeability. Our primary goal, however, to boost FRT power by selectively borrow- ing unbiased ECs with conformal p-values. The adaptive selection threshold that minimized the estimator's MSB is also novel approach.
n Es. Sc
Ai {Y: (),,((x)))) e(X,)
i (
TRR im1
1- A₁
(X. T- ((x)) (Y: - (x.
which is referred to as the No Borrowing (NB) approach hereafter. In RCTs, since the propensity score model e(xr) is known, ी. is consistent and asymptotically normal regard- less of whether Ha (x) is correctly specified for = 0. 1. Thus, नR serves as a model-assisted covariate-adjusted ATE estimator whose asymptotic variance attains the semipara- metric efficiency bound if (a is correctly specified for a = 0, The efficiency of fR could be further improved by borrowing information from EC data. To incorporate EC data for estimating т, many scholars have considered the following assumption (Li et al., 2023b).
Assumption 2.2 (Mean exchangeability). ({Y |
2. Randomization inference framework
x = EE (0) X, =
2.1. Preliminaries
Under Assumptions 2. and 2.2, t could be identified with both RCT and EC data. We estimate (o(x) with RCT and EC data and denote the estimated functions by ( Let (x)) denote the estimated sampling score. The vari- ance ratio between randomized controls and ECs is denoted by r(x) V(Y X = x, A = 0, = 13/V(Y (0) X = x, A = 0,S = 0}. Let fz) denote the estimated variance ratio. Li et al. (2023b) proposed a doubly robust estimator of
Consider nR patients in the RCT, ng patients in the EC group, and n = n& + ng patients in total. Let S for patients in RCT and S = for patients in the EC group. Let A denote the binary treatment, where A 1 stands for treatment and A = 0 stands for control. We denote T = fi 1}. R = C, and & = (i S; = 0}. Let X denote the baseline covariates, Y denote the observed outcome, and (0) and (1) denote the potential outcomes. In an RCT, we randomize nR patients into either the treatment or control groups based the known propensity score e(x) IPAA = 1 1 = S 1). This results in nn patients in the treatment group and no patients in the control group. For ne patients in the EC group. since all of them are under control, we have A = 0 for 5 = O. Let (xx = PSS = X = x) denote the sampling score of participating in the RCT. We consider the average treatment effect in the RCT population as our estimand (1) -Y (0) S - 1}. For RCT data, the following standard identification assumptions are considered (Imbens & Rubin, 2015).
A; = 1, S; = 1}. C = {i A; = S, =
1
Σ
A (Y, - (IR((X)) St (. ((X.)
ÎR-E
NR
- P. (X.) - W.YY - -x..R (I)
W = (X,) Te (, - A,) + (1 (1- S.)ee
+ (X.)(Fee)))
ÂR- is referred to as the Full Borrowing (FB) approach hereafter. The term "Full" here refers to incorporating the full set of ECs to construct ĀR-S. while down weighwiig those ECs based on similarity measured by X, thereby ad- dressing bias caused by observed confounders. PRtE is consistent and asymptotically normal if either (i) na (x) is correctly specified for = 0, 1, or (ii) both ((x) and e(x) are correctly specified. If all models for (a a = 0, ग (z). and e(x) are correctly specified, îि achieves the semi- parametric efficiency bound.
Assumption 2.1 (RCT identification). (i) (Consistency) Y = (1) + (1 - (0). (ii) (Positivity) 0 < e(x) < 1 for all x such that (x|s( > 0, where fx]s(2|1) is the conditional p.d.f. of X given S = s. (iii) (Randomization)
3

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
However, asymptotic inference for öR+S may be invalid due to three main reasons: (i) it assumes ng - oo, which contradicts the motivation for EC borrowing, where the sam- ple size of the RCT is typically small; (ii) it relies on the correct specification of at least one of the two nuisance mod- els, which may be violated because sophisticated models are difficult to work with under small sample sizes; and (iii) it depends on Assumption 2.2, which may be violated due unmeasured confounders. To address these issues, we consider finite-sample exact randomization inference framework that maintains strict type I error rate control even if all models are misspecified and Assumption 2.2 fails. We consider îR and ēR+E as candidate test statistics and pro- pose a new class of test statisties in Section to achieve improved power across various scenarios.
In practice, we use Monte Carlo to approximate pfRr, Based on the RCT's actual randomization, we generate the new assignment A for i E R. and set A = 0 for i E E since the randomization in the RCT does not affect the as- signments of the ECs. A caveat is that the assignment of ECs should not be permuted, as this would violate the "analyze as you randomize" principle and compromise the validity of the FRT. The new assignment vector is denoted as A' (A4, ................................ .A,)). We generate assignments for B times and obtain PFRT = Em= I(T(AA) 2 T(A)} + 1)/(BBBBB))B + 1), where the " "+1" term accounts for A itself.
Theorem 2.3 shows that FRT exactly controls the type/ I error rate in finite samples, regardless of Assumption 2.2, because, under Ho. the reference distribution is derived from true randomization, which is well-controleed in clinical trials. However, the power of FRT heavily depends on the choice of test statistic, making it the most critical decision in randomization inference.
2.2. Fisher randomization test
In the randomization inference framework, we are condi- tional on the potential outcomes Ya(a) and covariates X; for i E RR U E, and consider the randomized assignment A (A1...., An) as the sole source of randomness. Since A; for i E 'R is well controlled and known in the RCT, we can leverage this advantage to guarantee the validity of inference without any additional assumptions. Let A denote the set of all possible assignments generated by the actual RCT design. Since all external units are under control, we for E. Randomization inference accom- modates not only Bernoulli trials with A; ~ Bernoullipp) for i E R but also complex designs like covariate-adaptive randomization (Rosenberger & Lachin, 2015).
2.3. Model-based power analysis
There are two primary approaches for conducting a power analysis of FRT: model-based or simulation-based (Rosen- berger & Lachin, 2015). We first perform a model-based power analysis under Assumption 2.2, highlighting how low variance of consistent test statistic enhances the power of FRT. In the following section, we conduct a simulation- based power analysis for a more challenging scenario where Assumption 2.2 does not hold, showing that the bias of an inconsistent test statistic reduces the power of FRT.
have A;
i.i.d.
Let M denote the total number of possible assignments, (t) (T(( S t) denote the randomization distribution of T(A). and (t) = A (TA((T(A") S t) denote the reference distribution of T(A*) under Ho. Both Fi,M,M and Fo,,,M are discrete in finite samples. To ap- ply empirical process theory and derive asymptotic rates for testing power, we assume continuous super-population distributions Fi,, and Fo,n: with Fi,n,M and Fo rep- resenting the empirical distribution functions based on M independent samples drawn from Fi. and Fo.n. respec- tively. In cases where these assumptions do not hold, FRT still controls the type error rate, and we will investigate its power through simulation Section 4. Based on those notations, the p-value and the power can be expressed as FRT PA (T(A*) T(A))
Consider Fisher's s sharp null hypothesis Ho : (,) (L),,, Vi E R, which states treatment effect for any units in RCT. Based on Hg, we could impute all potential outcomes (0) yr:mp(() (1) Y; for i E RR. Let T(A) denote the test statistic, which depends on the assignment A E A. (AA) could be ( ( (A)). [AR+s(A)), or the estimator introduced in Section 3. The theoretical guarantee of type error rate control holds for any test statistic, including those involving ECs, even if these ECs have hidden biases. This is one of the key merits of randomization inference. We define the p- value for measuring the extremeness of the observed T(A) against Ho as
pd FRT_ PA-
Fo,n, (T(A)) and UM,M = PA PP PRT <a) = PAfl Fo.n..M (T(A)) S a) = 1 -F (Fo. a))-
where A' E A has the same distribution as A and is in- dependent of A, and PA is taken over the distribution of A'.
Theorem 2.3. Under Ho. for a (0,1), we have P PA PRT S a)) S a, where PA is taken over the distri- bution of A. If we further assume that T(A) takes distinct values for different A E A, then we have P A FRT < a)) = |a]AlI/|A|| > a A/I where [x] represents the greatest integer less than or equal to x.
Theorem 2.4. Fixd > 0. suppose
(a) There are continuous cumulative distribution distribution.iunctiios (c.df.)) Fo.n and F1,n. such that Fo.n,M and Fi.n. are the empirical distribution functions based on M independent samples drawn from Fo,, and Fi,,. respectively.
4

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
(b) There is on > 0 and a continuous c.d.f. F such that (t) = F(t/on) for all t R.
individualized test-then-pool approach that leverages con- formal inference to select comparable ECs. The conformal p-value pj E (0, 1] is used to test the exchangeability of each EC j E E. The selected EC set is then defined as
(c) For ATE 5, Fi.n = Fo (t) =(t (itt-r//om ( T)/On), for all t E R.
{()) = [i E p > 7), where y E [0, 1] is a selection threshold. Substituting & with E()) in (I), we obtain the Conformal Selective Borrowing (CSB) estimator:
For < 1 < 0.5 and sufficiently large M,
(( 21-
(MM-
n S,A1, + S: {(x,)
A (Y, - (x.)
where E is over M independent samples from Fo.n and Fi.
RR
St Po.R+ (X:) -ViY,, - Po. - (0.++(())
(2)
For a given A O. significance level a, and design with possible assignments M, Theorem 2.4 shows that the power of the FRT also depends on the variance of the test statistic, Fn- Under Assumptions 2.1 and 2.2, and with all working models correctly specified, fR+E is consistent and has a variance that is less than or equal to that of îR (Li et al., 2023b). Thus, when there is no hidden bias, using fR+S as the test statistic improves the power of the FRT compared to AR, shown in subplot (B) of Figure 1.
V; = (Xa)
S A.) A.) + (1 8 (())
*E()) (x,) -(x,)) + + - - (Xi) (fEr)))))))))))) (X,) (X:)
CSB represents a class of ATE estimators: when 7 = 1, no ECs borrowed and {1) = 2, we have f1 = pR: when 7 = 0, all ECs borrowed and {() = E, we have no = PR+E- For 0 < < 1, ł. balances the trade-off between borrowing more ECs with smaller y and discarding more ECs with larger 7. By using T(A) = |fy| as the test statistic for FRT and allowing E7)) to vary with resampling A in FRT could account for selection uncertainty and provide valid post-selection inference. The following sections introduce various conformal p-values and data-adaptive procedure for selecting to minimize the MSE of f..
2.4. Simulation-based power analysis
When unmeasured confounding exists between RCT and EC data, Assumption 2.2 is violated, rendering ēR+E incon- sistent. In such cases, asymptotic inference based on fR+E is invalid and fails to control the type I error rate. In contrast, since Theorem 2.3 holds for any test statistic, FRT can still control the type I error with the inconsistent test statistic fR+ highlighting core merit of FRTs. However, the vio- lation of Assumption 2.2 subsequently causes Assumption (c) in Theorem 2.4 to be unfulfilled, rendering FRT with FR+E unable to achieve a power improvement over FRT with fR. Furthermore, employing îR+E as the test statistic results in a substantial loss of power compared to using îR. as illustrated in subplot (D) of Figure 1.
3.2. Conformal p-value
Split conformal p-vallu We first consider split conformal inference (Papadopoulos et al., 2002). We randomly split C into a calibration set C1 and a training set C \ C1 accord- ing to a prespecified sample size ratio, for example, 1 3. We use a score function s(x, y) to measure the "noncon- formity" of (x, y). For example, we can residual as the score function: s; = YY; - fx,c().) for E C1 and 8j = IY, f-c, (X,)). where f-e, (x) is a prediction model fitted by the training set C \ C.. Intu- itively, if (X,. Y2) is not exchangeable (see Remark 3.2 for formal definition) with ((X,, Yi), 8, should be large compared to {siJiec, Thus, we define the split conformal p-value as the proportion of (si) that are larger than 8j, that is, p) where I is the indicator function, and the "+1" accounts for including 8j itself. If ppplit is smaller than threshold y, we reject the hypothesis of exchangeability and discard EC j. The following theoretical guarantee states that if EC j is exchangeable, the rejection rate less than 7-
The trade-off between नि and ēR+E generally arises be- tween a causal estimator that ignores additional informa- tion and assumptions and one that incorporates them but risks bias if the assumptions fail (Rothenhäusler, 2020; Rothenhäusler et al., 2021). In the next section, instead of choosing between ंि and îR+E, we construct a class of ATE estimators, fy. indexed by a tuming parameter ? and encompassing Åि and ÂR+E as special cases. We then pro- pose a data-adaptive procedure to select 7 that minimizes the MSE of f,p thereby enhancing the power of FRT by using f.. the test statistic.
the absolute
(Eree, I(s; 2 87) + 1)/([G] + 1),
3. Conformal Selective Borrowing
3.1. A class of estimators
Motivated by heterogeneous scenarios where some ECs sat- isfy Assumption 2.2 while others do not, we propose an
Proposition 3.1. For j E E, suppose that (X,, Y,) and ((,,, Y.)ee are exchangeable. For y E (0, 1), we have
5

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
(A) Ho. unbiased ECs
(B) Hj, unbiased ECs
(C) Ho, biased ECs
(D) H3. biased ECs
Type Leror 0.048
150
Power = 0.334
Type eerrrr = 0.048
Power 0.334
20
20-
10-
50
10
30-
0-
0-
0-
Type
0.038
Power 0.486
Type eeror 0.056
Power 0.186
20
20
100
60-
10-
50
10-
30-
0-
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Exact Exactp-value
exactp-valle
Exact p-vaue
Exact p-value
Method
No Borrew (y=1)
Full Borrow (y=0)
Figure 1. Simulated distributions of p-values under Ho and H7.
P( {) 7- 83 and (si)iec have distinct values, we have P(n split >) [[x((G)) + 1)1} (1C1)+ > 7 1/ + 1).
threshold y that controls the family-wise type I error rate or false discovery rate for testing the exchangeability of all ECs (Bates et al., 2023). However, in our context, the power of the conformal tests is of greater concern. The classical test-then-pon approach has been criticized for its low power in detecting hidden bias, especially with small randomized control sample sizes (Li al., 2020). Even with effective control of the family-wise type I error rate, low-power con- formal tests allow many biased ECs to be incorrectly borrowed, increasing the MSE of i,, and reducing the power of the FRT. Therefore, we propose a data-adaptive procedure to directly minimize the MSE of f.y
Remark 3.2 (Definition of exchangeability). The random variables 21, 21,..., 2n exchangeable if, for any permuta- tion w of I...., n. the random variables =w(1),... Rw(n) have the same joint distribution as 21...., 2n- The i.i.d assumption is stronger than exchangeability, as the latter can hold with dependence (Shafer& & Vovk, 2008). The ex- changeability required by conformal inference stronger than the mean exchangeability (Assumption 2.2), which al- lows the construction of a statistically valid estimator within the asymptotic inference framework.
We decompose MSF(?) = E(f+, - --)² = (B(+,) (B(t), -32 +
V(+-). The main challenge lies in estimating the squared bias (E(f,, 7) as the true 7 is unknown. Fortunately, since the NB estimator 2 is consistent for r, we approximate (B(f,- 2 by (1ff--1)) = E(A,, - 22 -+))2-(((t,-))) We then use (f., )² to estimate EEff =2 and apply
CV+ p-value. While split conformal p-values are com- putationally efficient, they lose statistical efficiency due to data splitting. CV+ (Barber et al., 2021) fully utilize training data and remain computationally feasible. We randomly split C into K disjoint folds: C = UK_ Cx. We use the training set C \ Cs to fit prediction models f-c, (I:) and use the absolute residual as the score function: 8, = |Y -f (X,)) and 5))) = |Y, - f-Cuco (X₃) for function that indi- cates Cg. Thus, for i * i' and k(t) k(t',, we have (f)) We define the CV+ p-luue as the proportion of = =b,
bootstrap to estimate V(f.,) and V(f, - f.). Combining
these provides the estimated MSE for each y over finite grids, and we select the y that minimizes it. The complete procedure is detailed in Algorithm 1.
i E C. where k(1) E fI.....,K K}
(6)
Algorithm 1: Adaptive Selection Threshold 1}:
of (siJiec that are larger than the corresponding (0))) Jiec.
Input: Grid r (0, .0). bootstrap times I., for y E do
that is, Dj 25+ 1Erec I1(:; 2 20)) 1)/(ICI + 1). Proposition 3.3. For j E E, suppose that (X,. and Y.),Ec are exchangeable. For E (0, 1), we have P(r) 57) 2 {11 (1-22))mm 11)-11))..11 (IC) + +m) < 2x 2x+ (1-K/(C) / (K+ + +1,, where m = |C|/ K assumed to be an integer for simplicity.
Compute 4., from the original sample.
for 1 1,..,,, L...,,Lde, do
L Compute ((4)) from the I-th bootstrap sample.
for y I do
Compute ((f - A using (1)) 211)
Compute (ft,) using (ll)
3.3. Adaptive selection threshold
MSE(1) (f. -2 Vi - F1) + ((+)).
Since we construct D individually and make borrowing de- cisions collectively, one might consider choosing selection
Output: 7 = arg minyer MSE(7)
6

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
We theoretically analyze the procedure from a non- asymptotic perspective (Wainwright, 2019). Decomposing f. = =T T- where 6, [ffx) -T and E(e,) 0. Let Re = V(f - -i) V(e,- - <1) and o V(f,) = V(e)).
treatment, randomized control, and EC groups are set as (n1; Ro, ng) = (50, 25, 50). Similar results for a larger EC sample size (ng = 300) are included in the Appendix. We generate covariates X ~ Unif(-2, 2) with dimension = 2. The sampling indicator 5 ~ (ernoulli(z(()) is generated with т ( = {1 + exp (No + X (x")))=² -)) where no is chosen to ensure E(5) = nR/n, and 1) = (0.1,0.1)
Theorem 3.4. For fixed n > and y E r, let c, be a centered Gaussian variable with parameter @4 > 0, i.e., Eexp(Xe,) I exp(o²,² X= (2) for all X R. For i > 0,
there exists > 0 such that with probability least 1 4u:
The assignment generated by A ~
/ for N(O,1) 1) denote the noise. For the RCT sample (S = 1), we generate the poten- tial outcomes as (0) = XT + e with 90 = (1, 1). and Y (1) = To + 31 + with TO = 0.4 and 3₁ = 2). For
S = 1 and A 0 for 5 0. Let
MS - MSE & cá1ôهl + cd / log (TT1/)
max yel'
+ max cDb Vlog (11)1), c0² log (111/0}
the EC sample (S = 0), we consider two scenarios: (i) the scenario without hidden bias, where ( = X 30 + 0.5e; (ii) the scenario where part of the ECs have hidden bias b, where random proportion p of the ECs is biased, with (() b + X So + 0.5z, and the remaining propor- tion (1 p) are unbiased, with Y()) x 8g + 0.5g. We consider proportions of biased ECs = 50% and mag- nitudes of hidden bias b =
+ max (( - f1) - -x2] 1+ 7er
max ((t,) og
TET'
where A ma 16y1. 4- ma p. d1 is the bias of tt and [T] is the candinality ofT.
Theorem 3.4 shows that the discrepancy between the es- timated and true MSE vanishes if Á is bounded and the bias of the consistent estimator c.. the maximum standard deviation proxy Φ, and the variance estimation errors are sufficiently small.
8. Note that hidden bias refers to bias that remains due to unmeasured con- founders, even after balancing the observed covariates. Un- der the alternative hypothesis, the observed outcome is Y == AY + (1 - A) (0); under the sharp null hypoth- esis, the observed outcome is Y = Y(0). We consider NB, FB, and CSB with the adaptive selection threshold as esti- mators of T and test statistics for FRT. We also consider Adaptive Lasso Selective Borrowing (ALSB) by Gao et al. (2025). Given its higher computational cost (approximately 10 times slower than CSB), we omit FRTs for this method and instead compare CSB+FRT with ALSB+asympootic in- ference in the Appendix. CV+ p-values are used with 10 folds. We set B = 5000 to approximate p FRT and replicate the simulation 500 times per scenario.
Theorem 3.5. Under the same assumptions as in Theorem 3.4, for any L >>0 O, there exists a constant e > O such that, with probability at least 1 St, the following holds:
(7g - -) min(?, - -)3² & 2n 52 + 2cA Vlog (TT)))
TET
+ max c0b² Vlog (1r1/1), c log (IT1/0)
+ 2 max TET
(ff, - -1) --))- -n, max gE
(ff)) - -g1-
Theorem 3.5 provides a bound for the excess risk of ity in comparison to the oracle estimator. Although tg generally outperforms é in terms of MSE, it may exhibit excess risk in certain challenging cases, as shown in Figure 2 (C) in the simulation. This phenomenon highlights that iqz behaves similarly to the Hodges estimator (Le Cam, 1953) and to in- tegrated estimators in data fusion (Yang et al., 2023; Oberst et al., 2022): improving upon the baseline estimator (here, the No Borrow estimator) in certain regions of the parameter space (where there is no bias in ECs) inevitably leads to worse performance in other regions (where the bias in ECs is difficult to detect). FRT still controls the type I error rate even if excess risk is present or the assumptions in Theorem 3.4 are not satisfied.
Figure 2 displays performance metrics for b = 0, 1,...,8.. 8. In the first case (b = 0): (i) all methods exhibit neg- ligible bias: (ii) FB and CSB reduce MSE by 42% and 20%, respectively, compared to NB; (iii) all methods ef- fectively control the type I error rate; and (iv) FB and CSB increase power by 46% and 45% respectively, compared to NB. In the following eight cases (b = 1,...,8), 8): (i) FB exhibits a large bias, approximately 125%-203% of its standard deviation (SD). The absolute bias of FB de- creases with when 6 2 because large 6 values increase V(Y X = x, A 0, S = 0}. causing FB to down- weight ECs with small f(X,) in (1). CSB performs better at bias control, with bias ranging from 0%-22% of its SD; (ii) compared to NB, FB increases MSE by up to 454%, and CSB decreases MSE by 13%- 16% (except when = 1,2, where MSE increases by 1%- 18%). (iii) In line with Theorem 2.3, all methods control the type I error rate well; (iv) compared to NB, FB decreases power by up 51 51%. In contrast, CSB in- creases power by 13%-36% (except when 6 2, 3, 4, where
4. Simulation
We conduct simulations to evaluate the repeated sampling performance of the proposed methods under small sam- ple sizes and varying magnitudes of hidden bias, includ- ing challenging cases where separating biased ECs is dif- ficult. Specifically, the sample sizes for the randomized
7

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
(A) Absolute Blas
(6) Variance
(C) MSE
(D) Type Error Rate
(E) Power
0.6
0.12
0.3-
0.4
0.4-
0.10-
0.10
0.05
0.2
0.0
0.00
Magnitude Hiddnn Blas
Magnitude of Hidden Blas
Magnitude of Hidden Blas
M Magnitude of Hidden Baas
Magnibude of Hidden Bias
M Method
Borrow tymi)
Bonow (y=0)
Conformal Selective Borrow (
b.
bi
b:
b:
b:
Outcome
0.6
06
0.8
0.5
0.7
0.8
0.5 0.7
0.7
0.6
0.8
0.5
06
0.7
0.0
05
0.7
0.7
5 Sampling Score
RCT Conretie
EC (Selected)
EC (Unse (Utselected)
Figure 2. Simulation results across different hidden bias magnitudes b.
power decreases by 70%).. In challenging cases where 0 < b S 4, the efficiency loss of CSB occurs because small biases make it hard to distinguish biased ECs from unbiased ECs. Such loss is inevitable when aiming to gain efficiency in scenarios without hidden bias, a phenomenon known in the transfer learning literature the cost of transferability detection (Cai et al., 2024). This phenomenon also occurs for other data integration estimators under hidden bias (see Figure 2 in Yang et al. (2023), Figure 4 in Oberst et al. (2022), and Figure 2 in Lin et al. (2024)). Finally, we ex- amine the selection performance of CSB. We do not expect CSB to perfectly separate biased ECs from unbiased ones due to (i) the small sample size of randomized controls and (ii) finite sample noise. As shown in Figure 2, CSB discards biased ECs and some unbiased ones that aren't sufficiently similar to randomized controls, demonstrating satisfactory selection performance.
(4 = 0,5 0) to improve CALGB 9633's statistical effi- ciency. The NCDB is a clinical oncology database sourced from hospital registry data, jointly run by the American Can- cer Society and the American College of Surgeons, covering 70% of U.S. cancer cases.
RMST and pseudo-observations We use the Restricted Mean Survival Time (RMST), Y = min(T, as the pri- mary endpoint, where T represents the survival time and 1" is the truncation time. RMST measures survival time up to a clinically relevant truncation point and serves as a compelling alternative to the hazard ratio when the pro- portional hazards assumption is violated (Hernán, 2010). We consider the difference in 3-year RMST between the treatment and control groups for the RCT population ( - Y(0) S = 1} as the estimand, where (a) = min(T((), 3} and T(a) is the potential survival time, a = 0, 1. Five baseline covariates in CALGB 9633 and NCDB are considered: sex, age, race, histology, and tumor size. The censoring rates of T in CALGB 9633 and NCDB are 42% and 48%, respectively. We use "once-for- all" approach transform right-censored survival times into pxeudo-obxervation. for RMST, allowing standard causal inference methods as if outcomes were non-censored (An- dersen et al., 2003; Overgaard et al., 2017). To address -eeeriete-dependentt censoring, we stratified by sex, race, and histology, applying transformations separately within each dataset (Andersen & Pohar Perme, 2010). The strat- ified Kaplan MMieer estimator is used to estimate survival functions, with pseudo-observations generated via the jack-
5. Real data application
The CALGB 9633 and NCDB data. We apply the proposed methods to an RCT conducted by the Cancer and Leukemia Group B (CALGB), known as CALGB 9633, which investigated the treatment effect of adjuvant chemotherapy in patients with stage IB non-small-cell lung cancer (Strauss et al., 2008). In CALGB 9633 (S = 1), n1 = 167 patients were randomized adjuvant chemother- apy (A- 1), and no 168 were randomized to obser- vation (A = 0). We extract data for 11,700 patients from the National Cancer Database (NCDB) as the EC sample
8

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
Table Analysis results for CAI.GB 9633 NCDB.
Method
Est
SE
CI
Asym p
Exact
#EC
No Borrow (Dif-in-Means)
0.135
0.072
(-0.007, 0.276)
0.062
0.060
No Borrow (AIPW)
0.142
0.074
(-0.003, 0.286)
0.055
0,051
Full Borrow
0.241
0.061
(0.122, 0.361)
<0.001
0.031
335
Conformal Selective Borrow
0.138
0.058
(0.024, 0.252)
0.018
0.046
264
"Est" is the estimate. "SE", "CT", and "Asym p" are the asymptotic standard emror, confidence interval, and p-value, respectively. "Exact p" is the exact p-value. "#EC" is the number of borrowed ECs.
knife method, as implemented in the R package eventylm (Sachs& & Gabriel, 2022). We treat the pseudo-observations for 3-year RMST as the outcome hereafter. More details about the real data are provided in Appendix Section D.
method can uniformly and significantly outperform RCT- only analysis across varying levels of hidden bias, although different approaches optimize the risk-reward trade-off from different perspectives. The most challenging scenarios are those where bias is non-negligible but complex to correct or difficult detect. Our key distinctions from existing literature are twofold: (i) we prioritize exact Type I error control in small samples before seeking power gains; (ii) we optimize the risk-reward trade-off between no borrowing and full borrowing through conformal selective borrowing, motivated by real data in which some ECs are unbiased while others are not.
Data analysis. We apply NB, FB, and CSB to estimate the ATE and perform FRTs. For comparison, we also apply NE without covariate adjustment, i.e., difference- -nnemaa in-means estimator. In addition to the proposed exact p-value, we also compute the standard error, confidence interval, and p-value based on asymptotic inference for all approaches (Li et al., 2023b). Since the outcome shows a high propor- years, resulting highly skewed
tion of truncation distribution, we apply the conformal quantile regression (Romano al., 2019) compute the conformal score. We use the Jackknife+ p-value (Barber et al., 2021) to achieve a better balance between statistical and computational effi- ciency. Table presents the analysis results. For NB using Dif-in-Means and AIPW, asymptotic and exact p-values range from 0.051 to 0.062. In contrast, FB (using all 335 ECs) gives an asymptotic p-value of 0.001 and an ex- act p-value of 0.031, indicating significantly positive ATE. Similarly, CSB (using 178 ECs) shows an asymptotic p-value of 0.018 and an exact p-value of 0.046, also indicating significantly positive ATE. The ATE estimate from CSB falls between NB and FB, indicating a trade-off between these two approaches.
Heterogeneity among data sources is common in integration and transfer learning, often leading to bias efficiency loss even after balancing measured confounders. While penal- ized bias estimation is a common solution, our work demon- strates that conformal inference provides greater stability and flexibility in finite samples. Extending this approach to tasks like developing individual treatment regimes (Chu et al., 2023), exploring treatment effect heterogeneity (Wu & Yang, 2022), and improving experimental design (Ruan et al., 2024) shows great potential.
Beyond the sharp null, FRTs can test the weak null asymp- totically using studentized or prepivoted statistics (Wu & Ding, 2021; Cohen & Fogarty, 2022). Randomization-bart confidence intervals can be constructed by inverting FRTs (Luo et al., 2021: Zhu & Liu, 2023; Fiksel, 2024), and ran- domization inference confidence intervals for treatment effect quantiles (Caughey et al., 2023). Extending these methods to hybrid controlled trials would be valuable.
6. Discussion
test bounded nulls and construct
This paper proposes using FRT in hybrid controlled trials and introduces CSB for selectively incorporating comparable ECs, mitigating hidden bias. FRT with CSB maintains type I error control and improves power compared to RCT-only analysis. The proposed CSB estimator with an adaptive se- lection threshold enhances efficiency over the approach.
Software and Data
A user-friendly R package, intFRT, is available at: https://github www/ce/////eeeeeeeeeee caake-zhuuiie
One limitation of our procedure is that, when the bias is non- negligible yet difficult to detect, it may incur some power loss, though it still maintains valid Type I eтror control. This no-free-lunch limitation is acknowledged in existing papers (Oberst et al., 2022; Lin et al., 2024), which point out that without assuming mean exchangeability of ECs, no
2ht//itFRT.. int.RT.
Acknowledgment
We thank the anonymous reviewers and meta-reviewers of ICML 2025 for their helpful comments, which significantly

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
improved the manuscript. This project supported by the Food and Drug Administration (FDA) of the U.S. Depart- ment of Health and Human Services (HHS) as part of a fi- nancial assistance award U01FD007934 totaling $1,674,013 over two years funded by FDA/HHS. It is also supported by the National Institute On Aging of the National Insti- tutes Health under Award Number R0I AG06688, totaling $1,565,763 over four years. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, FDA/HHS, the National Institutes of Health, the U.S. Government.
Cai, T., Li, M., and Liu, M. Semi-supervised triply ro- bust inductive transfer learning. Journal of the American Statistical Association, in press, 2024.
Carter, K., Scheffold, A. L., Renteria, J., Berger, V. W., Luo, Y. A., Chipman, J. J., and Sverdlov, O. Regulatory guidance on randomization and the use of randomization tests in clinical trials: A systematic review. Statistics in Biopharmaceutical Research, 16(4):428 440, 2023.
Caughey, D., Dafoe, A., Li, X., and Miratrix. L. Randomi- sation inference beyond the sharp null: Bounded null hypotheses and quantiles of individual treatment effects. Journal of the Royal Statistical Society Series B: Statisti- cal Methodology, 85(5):1471-1491 2023.
Impact Statement
This paper presents work aimed at advancing data integra- tion, conformal inference, and their applications in biomedi- cal science. The potential societal impact of this research is substantial, including fostering the reliable and efficient use of real-world data, accelerating drug development processes, improving the understanding of rare diseases, and ultimately enhancing patient outcomes.
Chen, C., Wang. M., and Chen, S. An efficient data integra- tion scheme for synthesizing information from multiple secondary datasets for the parameter inference of the main analysis. Biometrics, 79(4):2947-2966) 2023.
Chen, S., Zhang, B., and Ye, T. Minimax rates and adaptivity in combining experimental and observational data. arXiv preprint 1XXi::2009.10222 2021a.
References
Alt, E. M., Chang, X., Jiang, X., Liu, Q., Mo, M., Xia, H. A., and Ibrahim, J. G. LEAP: The latent exchangeabil- ity prior for borrowing information from historical data. Biometrics, 80(3)uuaa 2024.
Chen, W.-C., Wang, C., Li, H., Lu, N., Tiwari, R., Xu, Y., and Yue. L. Q. Propensity score-ine score-integrn core-integrated composite likelihood approach for augmenting the control arm of a randomized controlled trial by incorporating real-world data. Journal of Biopharnaceutical Statistics, 30(3):508- 520, 2020.
Andersen, K. and Pohar Perme, M. Pseudo-observations in survival analysis. Statistical Methods in Medical Re- search, 19(1):71-99) 2010.
Chen, Z., Ning, J.. Shen, Y., and Qin, J. Combining primary cohort data with external aggregate information without assuming comparability. Biometrics, 7703):0224-0066 2021b.
Andersen, P. K., Klein, J. P., and Rosthøj, S. Generalised linear models for correlated pseudo-observations, with applications to multi-state models. Biometrika, 90(1): 15-27,2003.
Cheng, D. and Cai, T. Adaptive combination of randomized and observational data. arXiv preprint arXiv:2111.1500 2021.
Angelopoulos, A. N. and Bates, S. Conformal prediction: A gentle introduction. Foundations and Trends Machine Learning, 16(4):494-591 2023.
Cheng, Y., Wu, L., and Yang. S. Enhancing treatment ef- fect estimation: A model robust approach integrating randomized experiments and external controls using the double penalty integration estimator. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artifi- cial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 381-390, 2023.
Barber, R. F., Candes, J., Ramdas, A., and Tibshirani, J. Predictive inference with the jackknife+. The Annals of Statistics, (9(1):486-507 2021.
Bates, S., Candès, E., Lei, L., Romano, Y., and Sesia, M. Testing for outliers with conformal p-values. The Annals of Statistics, 51(():149-178, 2023.
Chernozhukov, V., Wüthrich, K., and Zhu, Y. An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association, 1 16(533)):149-11664 2021.
Bind, M.-A. C. and Rubin, D. B. When possible, report a Fisher-exact P value and display its underlying null randomization distribution. Proceedings of the National Academy of Sciences, 11762):19151-199555 2020.
Chu, J., Lu, W., and Yang, S. Targeted optimal treatment regime learning using summary statistics. Biometrika, 110(4))913-931 2023.
10

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
Cohen, P. L. and Fogarty, C. B. Gaussian prepivoting for finite population causal inference. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 (2):295-320, 2022.
Guo, W. Wang, S. L., Ding, P., Wang, Y., and Jordan, M. Multi-source causal inference using control variates under outcome selection bias. Transactions on Machine Learn- ing Research, 2022. ISSN 2835-888 8856. URL https: / / //openreviie ew net. / forum? wiimmndddrmmmmii (BarimmjB66.
Colnet, B., Mayer, I., Chen, G., Dieng, A., Li, R., Varo- quaux, G., Vert, J.-P., Josse, J., and Yang, S. Causal inference methods for combining randomized trials and observational studies: A review. Statistical Science, 39 19:165-191,,024.
Hernán, M. A. The hazards of hazard ratios. Epidemiology, 15,)):13-1,.201
Ho, D., Imai, K., King, G., and Stuart, E. A. Matchit: Non- parametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8):1-28, 2011.
Dang, L. E., Tarp. J. M., Abrahamsen, T. J., Kvist, K., Buse, J. B., Petersen, M., and van der Laan, M. A cross- validated targeted maximum likelihood estimator for data- adaptive experiment selection applied to the augmentation of RCT control arms with external data, arXiv preprint arXiv:2210.058802: 2023.
Ho, D. E., Imai, K., King, G., and Stuart, E. A. Matching as nonparametric preprocessing for reducing model depen- dence in parametric causal inference. Political Analysis, 15(3):199-233 2007.
Hobbs, B. P., Carlin, B. P., Mandrekar, S. J., and Sargent, J. Hierarchical commensurate and power prior models for adaptive incorporation historical information in clinical trials. Biometrics, 67(3):1047-100 2011.
European Medicines Agency. Guideline on ad-
justment for baseline covariates in clinical tri-
als. https: / /www///www............. ema ww/eep..eeeen////oo..e seejusteentbassellee--ooee 2015.
de diiidennne Huang, Y., Huang, C.-Y., and Kim, M.-O. Simultaneous se-
lection and incorporation of consistent external aggregate information. Statistics Medicine, 42(30):5630-5644 2023.
Fiksel, J. On exact randomization- covariate-adjusted 2024. 2024.
confidence intervals. Biometrics, 80(2):ujae051,
Fisher, R. A. The Design of Experiments. Oliver and Boyd,
Imbens, G. and Rubin, B. Causal Inference in Statis- tics, Social, and Biomedical Sciences. Cambridge Uni- versity Press, 2015.
Edinburgh, 1st edition, 1935.
Freidling, T., Zhao, Q., and Gao, Z. Selective randomiza- tion inference for adaptive experiments. arXiv preprint arXiv:2405.0700 07006, 2024.
Jiang, L., Nie, L., and Yuan, Y. Elastic priors to dynamically borrow information from historical data in clinical trials. Biometrics, (9(1):49-600 2023.
Bagnon-Bartsch. J. A., Sales, A. C., Wu, E., Botelho, A. E., Erickson, J. A., Miratrix, L. W., and Heffernan, N. T. Precise unbiased estimation in randomized experiments using auxiliary observational data. Journal of Causal Inference, 11(1):2022011, 2023.
Karlsson, R., Wang, G., Krijthe, J. H.,, and Dahabreh, I. J. Robust integration of external control data randomized trials. arXiv preprint arXiv:2406. 17971, 2024.
Kwiatkowski, E., Zhu, J., Li, X., Pang. H., Lieberman, G., and Psioda, M. A. Case weighted power priors for hybrid control analyses with time-to-event data. Biometries, 80 (2) (2):ujee019, 2024.
Gao, C. and Yang, S. Pretest estimation in combining prob- ability and honprrobayilityy samples. Electronic Journal of Statistics, 17(1):1492 1546, 2023.
Le Cam, L. On some asymptotic properties of maximum likelihood estimates and related Bayes estimates. Univer- sity of California Publications in Statistics, 1:277-330, 1953.
Gao, C., Yang, S., Shan, M., Ye, W., Lipkovich, I., and Faries, D. Improving randomized controlled trial analysis via data- aata-adaptive borrowing. Biometrika, (2s2))aaae 2025.
Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. Distribution-fri free predictive inference for regression. Journal of the American Statistical Associa- tion, 109-5-33:.099411111 2018.
Gu, Y., Liu, H., and Ma, W. Incorporating external data for analyzing randomized clinical trials: A transfer learning approach. arXiv preprint arXiv:2409.04122 2024.
Guan, L. and Tibshirani, R. Prediction and outlier detection in classification problems. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(2):524 546, 2022.
Lei, L. and Candès, E. J. Conformal inference of counter- factuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodol- ogy, 83(5):111 1-398, 2021.

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
Li, H., Tiwari, R., and Li, Q. H. Conditional borrowing external data establish hybrid control arm in ran- domized clinical trials. Journal of Biopharnacentical Statistics, 32(6))954-99) 968, 2022.
Oberst, M., D' A'mourr, A., Chen, M., Wang, Y., Sontag, D., and Yadlowsky, S. Understanding the risks and rewards of combining unbiased and possibly biased estimators, with applications to causal inference. arXiv preprint arXiv:2205. 10467, 2022.
Li, L. and Jemielita, T. Confounding adjustment in the analysis of augmented randomized controlled trial with hybrid control arm. Statistics in Medicine, 42(16))2855- 2872, 2023.
Overgaard. M., Parner, E. T., and Pedersen, J. Asymptotic theory of generalized estimating equations based on jack- knife pseudo-obserrations. The Annals of Statistics, 45 (5):1988-22) 2015, 2017.
Li, R., Lin, R., Huang, J., Tian, L., and Zhu, J. A frequentist approach to dynamic borrowing. Biometrical Journal, 65 (7):2100406, 2023a.
Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. Inductive confidence machines for regression. Ma- chine learning: ECML 2002: 13th European conference machine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pp. 345-356. Springer, 2002.
Li, S. and Luedtke, A. Efficient estimation under data fusion. Biometrika, 110(4):04111054, 2023.
Plamadeala, and Rosenberger, W. F. Sequential monitor- ing with conditional randomization tests. The Annals of Statistics, 4:(1):30 44, 2012.
Li, W., Liu, F., and Snavely. D. Revisit of test-then-pou methods and some practical considerations. Pharmaceu- tical Statistics, 1955::49-551, 2020.
Pocock, S. J. The combination of randomized and historical controls in clinical trials. Journal of Chronic Diseases, 29(3):175-188, 1976.
Li, X., Miao, W., Lu, F., and Zhou, X.-H. Improving effi- ciency of inference in clinical trials with external control data. Biometrics, 79(1):394-44) 2023b.
Proschan, M. A. and Dodd, I.. E. Re-randomization tests in clinical trials. Statistics in Medicine, 38(12):2292-2330, 2019.
Liang, Z., Sesia, M., and Sun, W. Integrative conformal p-values for out-of-distribution testing with labelled out- liers. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. qkad138, 2024.
Puelz, D., Basse, G., Feller, A., and Toulis, P. A graph- theoretic approach to randomization tests of causal ef- fects under general interference. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 (1):174-204, 2022.
Lin, X., Tarp, M., and Evans, R. Data fusion for efficiency gain in ATE estimation: A practical review with simulations. arXiv preprint XxX::24007001886 2024.
Lin, X., Tarp, J.. and Evans, R. Combining experimen- tal and observational data through power likelihood. Biometrics, 2025.
Rabideau, D. and Wang. R. Randomization- confi- dence intervals for cluster randomized trials. Biostatistics, 22(4):913-927, 2021.
Liu, Y., Lu, B., Foster, R., Zhang. Y.. Zhong, Z. J., Chen, M.-H., and Sun, Matching design for augmenting the control arm of randomized controlled trial using real- world data. Journal of Biopharmaceutical Statistics, 32 (1):124-140, 2022.
Ritzwoller, D. M., Romano, P., and Shaikh, A. M. Ran- domization inference: Theory and applications. arXiv preprint arXiv:2406.09522 2024.
Romano, Y., Patterson, E., and Candès, E. J. Conformalized quantile regression. In Proceedings of the 33rd Inter- national Conference on Neural Information Processing Systems, pp. 3543-3553, 2019.
Luo, X., Dasgupta, T., Xie, M., and Liu, R. Y. Leveraging the Fisher randomization test using confidence distribu- tions: Inference, combination and fusion learning. Jour- nal of the Royal Statistical Society Series Statistical Methodology, 83(4):777-797, 2021.
Rosenberger, W. F. and Lachin, J. M. Randomization in Clinical Trials: Theory and Practice. John Wiley & Sons, 2015.
Miller, F. and Joffe, S. Equipoise and the dilemma of ran- domized clinical trials. The New England Journal of Medicine, 364(5):476 480, 2011.
Rosenberger, W. F., Uschner, D., and Wang, Y. Randomiza- tion: The forgotten component of the randomized clinical trial. Statistics in Medicine, 38(1):1-13 2019.
Nair, Y. and Janson, L. Randomization tests for adaptively collected data. arXiv preprint arXiv:2301.05365, 2023.
Rosenman, E. T., Basse, G., Owen, A. B., and Baiocchi, M. Combining observational and experimental datasets
12

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
using shrinkage estimators. Biometrics, 79(4):991122973, 2023.
U.S. Food and Drug Administration. Adaptive design clinical trials for drugs and biologics guidance for in- dustry. https: : /wwp// fda gtw//mdia//88955...5 download, 2019.
Rothenhäusler, D. Model selection for estimation of causal parameters. arXiv preprint arXiv:2008. 12892, 2020.
U.S. Food and Drug Administration. Rare Diseases FDA. https:/ /www. fda. wed...ooo.a.... gtw.gatiinntttte rare fare-disese deeeddissssssseeee Ida, 2022.
Rothenhäuster, D., Meinshausen, N., Bühlmann, P., and Peters, J. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2):215-244 2021.
U.S. Food and Drug Administration. Considerations for the design and conduct of externally controlled trials for drug and biological products guidance for indus- try. attpps//wwwt fda. www.:dww.......aaa.....eao wtt..///////////49/0// download, 2023.
Ruan, X., Wang, J., Wang, Y., and Wei, W. Electronic medical records assisted digital clinical trial design. In Proceedings of The 27th International Conference on Artificial Intelligence and Statisties, volume 238 of Pro- ceedings of Muchine Learning Research, pp. 2844, 2024.
Valancius, M., Pang. H., Zhu, J., Cole, S. R., Funk, M. J., and Kosorok, M. R. A causal inference framework for leveraging external controls in hybrid trials. Biometrics, 80(4):ujae095, 2024.
Sachs, M. C. and Gabriel, E. E. Event history regression with pseudo-obeervations: computational approaches and an implementation in R. Journal of Statistical Software,
van der Laan, M., Qiu, S., and van der Laan, L. Adaptive- TMLE for the average treatment effect based on ran- domized controlled trial augmented with real-world data. arXiv preprint arXiv:2405.077 076 2024.
102:1-34, 2022.
Schmidli, H., Gsteiger, S., Roychoudhury, S., O' Hagan, A., Spiegelhalter, D., and Neuenschwander, B. Robust meta- analytic-predictive priors in clinical trials with historical control information. Biometrics, 70(4): 1023-10223 2014.
Ventz, S., Khozin, S., Louv, B., Sands, J., Wen, Y., Rah- man, R., Comment, L., Alexander, B. M., and Trippa, L. The design and evaluation of hybrid controlled trials that leverage external data and randomization. Nature Communications, 13(1):5783, 2022.
Schuler, A., Walsh, D., Hall, D., Walsh, J., Fisher, C., for Alzheimer's Disease, C. P., Initiative, A. D. N., and Study, A. D. C. Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score. The International Journal of Biostatistics, 18(2):329-356, 2022.
Viele, K.,, Berry, S., Neuenschwander, B., Amzal, B., Chen, F., Enas, N., Hobbs, B., Ibrahim, J. G., Kinnersley, N., Lindborg. S., et al. Use of historical control data for as- sessing treatment effects in clinical trials. Pharmaceutical Statistics, 13(1):41-54, 2014.
Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9():3711-421, 2008.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning in Random World, volume 29. Springer, 2005.
Simon, R. and Simon, N. R. Using randomization tests to preserve type I error with response adaptive and covariate adaptive randomization. Statistics Probability Letters, 81(7):767777, 2011.
Wainwright, M. High-dimensional statistics: A non- asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
Strauss, G. M., Herndon, J. E., Maddaus, M. A., Johnstone, D. W., Johnson, E. A., Harpole, H., Gillenwater, H. H., Watson, M., Sugarbaker, J., Schilsky, R. L.. al. Adjuvant paclitaxel plus carboplatin compared with ob- servation in stage IB non-small-cell lung cancer: CALGB 9633 with the cancer and leukemia group B, radiation therapy oncology group, and north central cancer treat- ment group study groups. Journal of Clinical Oncology, 26(31))5043-500 2008
Wu, J. and Ding, P. Randomization tests for weak null hypotheses in randomized experiments. Journal of the American Statistical Association, 116(536):8988-1113 2021.
Wu, L. and Yang, S. Integrative R-learner of heterogeneous treatment effects combining experimental and observa- tional studies. In Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Pro- ceedings of Machine Learning Research, pp. 904-926. 2022.
Stuart, E. A. and Rubin, D. B. Matching with multiple control groups with adjustment for group differences. Journal of Educational and Behavioral Statistics, 33(3): 279-306, 2008
Yang, S. and Ding, P. Combining multiple observational
data sources to estimate causal effects. Journal of the
13

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
American Statistical Association, 11((531)11500-555 2020.
Yang. S., Gao, C., Zeng, D., and Wang, X. Elastic integra- tive analysis of randomised trial and real-world data for treatment heterogeneity estimation. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85 (3):575-596, 2023.
Yang. S., Liu, S., Zeng, D., and Wang, Data fusion methods for the heterogeneity of treatment effect and confounding function. Bernoulli, in press, 2024.
Yi, Y., Zhang, Y., Du, Y., and Ye, T. Testing for treatment effect twice using internal and external controls clini- cal trials. Journal of Causal Inference, 11(1):2022018, 2023.
Young, A. Channeling Fisher: Randomization tests and the statistical insignificance of seemingly significant experi- mental results. The Quarterly Journal of Economics, 134 (2):557-598, 2019.
Yuan, J., Liu, J., Zhu, R., Lu, Y., and Palm, U. Design of randomized controlled confirmatory trials using histori- cal control data augment sample size for concurrent controls. Journal of biopharnaceutical statistics, 29(3):
2019.
Zhai, Y. and Han, P. Data integration with oracle use exter- nal information from heterogeneous populations. Journal of Computational and Graphical Statisties, 31(4):1001- 1012, 2022.
Zhang, Y. and Zhao, Q. What is a randomization test? Journal of the American Statistical Association, 118(544): 2928-2942, 2023.
Zhu, K. and Liu, H. Pair-switching rerandomization. Bio- metrics, 792)22227--114, 2023.
14

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
A. Additional conformal p-values
Full conformal p-value. Full conformal inference (Vovk et al., 2005) fully utilizes all data in C for both training and calibration. We can still use the absolute residual as the score function: s; = IY; ff for i C and s, = IY, (X,]) where (z) is a prediction model fitted by the augmented set C U {i). To measure the extremeness of observing sj under the exchangeability, we define the full conformal p- p-value as the proportion of the elements in {sihee that are larger than or equal to 8,, that is, pf = (Eiec M(s; 2 83) + 13/(CI + 1).
Proposition A.I. For j E E, suppose that (X,, Yj) and {(X,, HiEE are exchangeable. For 7 E (0, 1), we have P(2) full S 7) S 7. If 8j and fec have distinct values, we have P(r)") >) = { La (IC) + 1) (IC) + +1 > ? 1/(IC| + 1).
To compute full conformal p-values for all ECs j E E, the prediction model must be refit ne times, which is time-consuming for large EC samples.
Jackknife+ p-values (Barber et al., 2021) is a special case of CV+ with K = C). We use the leave-one-out training set C \ {i} to fit prediction models (z) and use the absolute residual as the score function:
Jackknife+ p-value.
8 = - - fx))) and 5) = , fxx(Xx)] for i E C. We define the Jackknife+ p-value as the proportion of {sikee that are larger than the corresponding {s Hec. that is. pjackknilet =iiec I(s, 2 +/0)+ +0)++)/((() + 1).
Proposition A.2. For j E, suppose that (X,,Y, and Yi)vve are exchangeable. For & (0, 1), we have P(p) P(pjuckknife-- < < 2-- - 1/([c] + 1) < 2y.
Remark A.3. The factor of 2 cannot be reduced without further assumptions, as shown by pathological cases in Barber et al. (2021), though the empirical error rate is close to 7
B. Proofs
B.1. Proof of Theorem 2.3
Proof of Theorem 2.3. Under Ho. the imputed potential outcomes are the same as the true potential outcomes. Thus, the distribution of T = ={AA)) is the same as that of T = T(A). With simplified notations, we have
P FRT a) A AP (T" T) S o
In a finite sample, A can take only a finite set of values, which implies that T must also take on a finite set of values. Suppose these values are
R ... Tm > .... TM,
and
m. == I,...,.. M. PA(T- = Tm) = PA (T" = Tm)) = Om:
For T E (Ti......
}. we have a PA (T" 2 T) < M
am = 1. If0 < a < aj, we have
PA PRT aa = PP (T > T) < <a} 0 <
a <a< (I..... M - 1}, such that EdI Um S a and Em+1 am > a. Then, we have
M
PRTPRT PRT <a) = PA 2 5 = (T... Txl =
Σ
Om
S
a.
m=1
If T(A) takes distinct values for different A E A, p
FRT
uniformly distributed:
PA P PRRT=
a
=
1
a = 1,...,1441
A
[
Thus, we have
|a[A||
a|Al -
1
IP A ( FRT < <a)
A
a
A
15

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
Remark B.I. If is continuous random variable, suppose its distribution function is F(I) P(T s 1), then the proof could be simplified
PA PP (T*
sa} = (1 = -1- (1-
However, is discrete with finite values, and we provide rigorous proof in the finite-sample setting.
B.2. Proof of Theorem 2.4
We invoke two lemmas from the Supplementary Material of Puelz et al. (2022).
Lemma B.2 (Lemma 5 in Puelz et al. (2022)). Suppose Assumptions (b) and (c) in Theorem 2.4 hold, for some r (0.5, O(log M)), we have
((Pi,n(qn)) Fi,n (g), 2 -M(M"),
Lemma B.3 (Lemma 4 in Puelz et al. (2022)). Suppose Assumption (a) of Theorem 2.4 holds, for any 0 < < 0.5 and large enough M, we have
((Fi,,, (z)) = O0M 0(M-0.5+)) for any z E R.
Proof of Theorem 2.4. Let l Mo - ()) and 4a = For -a). Thus, we have
tN,M = - Fi P (1))
Fi (9a,M)
1 (ga) + (1,)( F1,n(uo,,)) - Fimn(yo,s)) - ai a
(3)
T2
Ts
By Assumptions (b) and (c), we have
T = Fi (Fotl - aa) =1 F P(F -a) - -T/ON).
Combined with Lemmas B.2 and B.3, we have
B( 2 F (F- - (000--00000 - O O(M -0.++). -.M5t.
The result follows from that 0.5 0 0.
B.3. Proof of Proposition A.1
Proof of Proposition A.I. Since the calibration set (X,, Yj)sec and external control (X,, Yj) are exchangeable, we have (si)xec and 8j exchangeable. Thus, have
Eiec [1s; 2 25))+ +1
P(p
full S 57)=1 =P
$T)
C +1+1
17 [1C] +111
C C1
+1
where the first inequality is due to exchangeability and the possibility of ties (si)tec and 8j- If sy and {sifiec have distinct values, pfull is uniformly distributed due exchangeability. That is.
full P9
1 a=
1.1,..,,((+1
C C1+11
C 1
Thus, we have
PP full < <>)
17 (IC) 1)J
(IC)
+ 1) -
1
[C| +1
C
+
C +
16

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
B.4. Proof of Proposition 3.1
Proof of Proposition 3.1. Since the calibration set (X,, Yf)iEC and external control (X3, Yÿ) are exchangeable, we have (si)iec, and 8j exchangeable. Thus, have
P(p |>) =P Eiee, T(x 2 83) +1
5T)
c1
|,([c|] 1)1
C1 +1
where the first inequality is due to exchangeability and the possibility of ties in (si)kec, and 8j- If sy and Hsc have distinct values,
pf is uniformly distributed due to exchangeability. That is,
TP
Pj
split
C1 c1|+1)
I,.... |C1
C1 +1'
Thus, we have
|r(|C]| + 1)1
>(1c]] + 1) -1
P(p
<)
7
[C1 ++
C1 +1
B.5. Proof of Proposition A.2
Lemma B.4, Consider a matrix R E R(n+1)x(x+)) with elements Raj. Define the set
S= jE fl....... +1
n+1
Σ
[Rig < Ryj) 2 (1 -1- -) 10 r 7E (0.1).
i=1
Then, we have
s 1) 2
where |S].
Proof. Since
a+1
n+1
Σ
I(Rg < <Rp))
[Ras 2 Rτi) S 7(n + 1),
by summing over all S, we have
[R 2 R sy
JES 1=1
For i # j. since I(R,y 2 RRi) + I(Rr Raj) 1. we have
ΣΣ
[(B 2 Rgi) Σ Σ
I(Ri 2 RB)) +
JES YES
JES iES
s(s 1)
2 + 8.
By combining these two inequalities, we obtain
s(s 1)
+
<
I(R.g 2 Rz))
2
JES iES
n+1
T(R 2 Ry)
JES 1=1
Ssy(n
17

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
Thus, have
(s - 1) 2 + +15((++1) + 8 25 - < 2y
Proof of Proposition A.2. For j' E ECU(U) we define
100
i=
Roy Yo- (xx) it t
where 52 is a prediction model fitted by the leave-two-our augmented set (C U (7}) \ (.,5'). For C, since (cu())) {i,5} C\ {i}, we have (x) f((())(z)) (z), thereby,
5; |Y; --i(X:)] Rij,
:/ 1Y, -f-(())) Ryt.
Thus, we have
Eiec I(s; 2 25 (8) +
P jackkniffe+
=P
57)
C +1
EiecuRsi T(Raj Ryj))
=P
57)
[C] +1
=P
Σ
IR < Rg)) 21 (1-) +1)
(ECU())
<2>
|C|
+1
<2%
where first inequality is due to exchangeability and Lemma B.4.
B.6. Proof of Proposition 3.3
Lemma B.5. Suppose m = =n/K is an integer, and the + units evenly divided into K + 1 sets, denoted by CI....,Cka Consider matrix R E (n+m) (a+m) with elements Rij = Rzi if and j belong to the same set. Define the sel
x+m
S = jE
jE fL,
++m]+
n m]
Σ 1=1
I(R < RDi) 2 (1 - (m 1m 7 (0,1).
Then, we have
S 27(n+ + 1) + m 2,
where (5).
Proof. For j S. by definition, we have
n
Σ
T(R 2 Ryd) S (n m) - (1 - (m + 1).
Since Rτg- = Ry: if i and j belong to the same set, we have
mm
Σ
T(Ri 2 R₁)= = Σ I(Rig 2 R5i)+ + Σ
I(Rij 2 RTi)
igCKC)
(ECwC)
Σ
T(Rey 2 Rgj) + m,
i4CK6)
18

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
where CRG) is the set containing unit j. Thus, we have
[(Ra Rgi) S(n + m) (1-7) + +1
(ÉCRG)
=y(n + +1) 11
By summing over all j & S, have
ΣΣ jES (ESC))
1(Rs 2 Rgi) S 8(7(1 + 1) - 13.
(4)
On the other hand, for * j. since (Ry 2 Rgj + +I(g 2 Rıj) 2 1, we have
[R RRi) 2 s(s - 1)
Σ
2
iES iES,işj
Since R, Rys if and j belong to the same set, we have
ΣΣ
T(Raj 2 RB))
Σ
Σ
[R 2 Rại) +
Σ Σ
T(Rij 2 R₃:)
JES (E8
JES (ES,
JES ic iE HEY
( 1)
Σ
Σ
[Rg Rτ))
JES (E5))
where 8 [Cx n|. Thus, we have
8(8 [ 1)
K+1 (sk
Σ
[(Ri 2 R2J) 2
2
JES iES,i
k=1
By combining (4) and (5), we have
K+1
s(s - 1)
(sk - 1)
Σ Σ
I(Ruj 2 Rya)
2
2
k=1
jES iES,iCCU()
$ Σ
Σ
I(Baj 2 R4j)
JES (gCn())
28(7(n + + 1}.
Since 8y S m, we have
K+1
( -
5(m - 1)
2
2
Thus, we have
8 < 2y(n + +)++ 4. m - 2.
Proof of Proposition 3.3. We consider m = |C|/K K is an integer for simplicity. Let CK+1 contain j and other m hypothetical points. For j E UK++c Cr. define
400
k(t') = k(j',,
Reg =
Yr - Rege (Xxr)
k(i') # k()",
19

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
where f-(Caar, (wert is prediction model fitted by the leave-two-set-our augmented set (U ( Since C UK k= and CRUS) Ck+1, have (UR-iCC C+) \ (CR(e) UCR(g)) C \ Cu(e) for i C. Thus, for i E C, we have f-ce(o (x) = =f (e()) (z)), thereby,
Cx) \ (Cu(P) U CR(P)).
8 |Y; f-ceco (Xi)l Raj,
( 1Y3 - -fccco (Xx)) Rjy-
Thus, we have
Eiec [s;; 2 28) (0) +1 $7)
P(p 57 =P
[C| +1
Eiecupy I(Rij 2 Rzi)
=P
5T)
c1+11
=P
Σ
I[Ry < Ríi) 2 (1 (1-7)(01+1)
jECuts
<T
Σ
I[Rg < (1)2(1101((111
NEU Ck
(x( 1) +m--
|C| + m
(1 27) 1)
527
[| + m
1- K/|C|
227
K
where the second inequality is due to exchangeability and Lemma B.5.
B.7. Proof of Theorem 3.4
Proof of Theorem 3.4. Since e, is centered sub-Gaussian variable with parameter dy,, we have E, Ei centered
sub-Gaussian variable with parameter 20, where & = max,er é,, Moreover, we have (c, 61) - Rn is centered
sub-exponential variable with parameters (c1 c1 2²), where e₁ is a constant. By f, न = (8,, 61) + (c, - =1) and using the concentration inequalities for sub-Gaussian and sub-exponential variables (Wainwright, 2019), it follows that, with probability at least 1 - 4c,
max (f 41 - 16, -012 x2)1 T
max 12(8.- - 8i)(Ey - E1) + (en -E1)2 - k T1r'
log (111)) + max { /v2c 22 2 (1T)/), 2c, 201 log ((F1/4)
(6)
where A max,er 18-1.
20

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials By (6), it follows that, with probability least 1 1- 41,
MSE(?)) - MSE(1))
max TET
(fy - 512 - V(ty - + - -g o2
max
YET
(f, n. 2² - x + o - 8 - 2 + max V(tt - {11) - -2g] + max (fxr)) - oΩ91 3ET
S max
TET
TET
[8, - 81)² - 8. + c log (11))) + max { cddp Vlog (1r1/(), cΦ2 log (111/())
< max
7ET
+ max [f +1) x max V(f,) - o 7er
7EL'
ScA1ói1 + cA Vlg (1/ + max ct2 / log (1T1)), c log (1T//))
+ max 7er
(V A) n max yer ((tr)) - oB11.
where c is a constant.
B.8. Proof of Theorem 3.5
Proof of Theorem 3.5. Since Ey is a centered sub-Gaussian variable with parameter Φy, we have e, o² as centered sub- exponential variable with parameter (c20², c2 where 02 is a constant. By ê. - -T= by + E and using the concentration inequalities for aub-Gaussian and sub-exponential variables (Wainwright, 2019), it follows that, with probability least 1 46
TET max (fqy -7, - o
max 7ET
128,c, + bg - -2
<2<2A0 / log ((T//)) + max { (222002 (IT//), 2c log (171/))
(7)
By (6) and (7), it follows that, with probability at least 1 - 86,
MSF(7) (fr - -7)²
max
TET
(fyy - Vi. - -1)++ + -((+)) (fr - -)
max
TET
< max
(f, - 32 - x2, + o - (f, - -) + max (VtT -i) -k2l + max (Trr) - 02 2
7ET
gET
TET
(6, - 61) 8.) + cA / log (1r + max { /oog2/lo (T]/)), < log (1T1/0)
max
YET'
+ max ((t - f1) - R + max ((tt)) -0?
YET
TE1
ScAl01 |6 + cA / Tog (11) + max x V Tog ([T),), cT2 co2log log ((T/)))
+ max TET
Vi +) 2 max
((Tx) o.21:
(8)
where is a constant.
Since
min MSE(7) 7Er
min(f, - -7)3
< max yer
MSE(2) (f, -7)2
yer
and
min MSE(>) - -f7--77 - 7)2 - rEl"
MSE(()) (fgg 7)2/
S max yE1
MSE(?) - (Fr- -7)
21

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
No Hidden Blas
Half ofECs Exhibi Hidden Bias
1.00
0.75
Power
0.50
0.25
0.00
0.0
0.3
0.0
0.3
0.6
T
Method
(y-0)
Conformal Selective Borrow (
Figure 3. Power curves when b = and = 8.
we have
>ET (fy - -)²² min (f. - 7)² < 2 max MSE(7) (fr -))
TET
The result follows from (8).
C. Additional simulation results
C.1. Power curve
For the scenario where there is no hidden bias (b = 0) and another where half of the ECs exhibit hidden bias with a magnitude of = 8, we vary т to plot the power curve, as shown in Figure 3. CSB outperforms NB in both cases, while FB demonstrates low power in the presence of hidden bias.
C.2. Adaptivity of the selection threshold
Figure illustrates how fy changes with the magnitude of (i) When there is no bias (b = 0). f approaches to borrow all ECs and maximize power; (ii) with moderate bias (b 1,, where distinguishing between biased and unbiased ECs challenging, increases help discard the biased ECs; (iii) when the bias large 2 4), decreases but remains non-zero, retaining more unbiased ECs, while easily discarding the biased ones.
C.3. Various selection thresholds
Figure shows the performance of the fixed selection threshold 7 and the adaptive selection threshold & when ne 50. As discussed in Section 3.3, smaller 7 selects more ECs but risks greater bias when distinguishing between biased and unbiased ECs is difficult. This creates power trade-off across different bias levels, similar to MSE simulation results in data integration (Yang et al., 2023; Oberst et al., 2022; Lin et al., 2024). We find that (i) CSB with 7 = 0.6 improves power compared to NB, except in extreme cases like 6 = 2, where it decreases power slightly, and (ii) CSB with f further improves power but also risks power loss in difficult scenarios. The power trade-off does not compromise the Type I error rate, which remains controlled with all selection thresholds.
C.4. Comparison to Adaptive Lasso Selective Borrowing
Figure 6 presents the simulation results for ALSB with asymptotic inference. Unlike CSB + FRT, ALSB with asymptotic inference fails to control the type I error rate in this small sample size scenario. Additionally, CSB demonstrates better estimation and selection performance in most cases.
22

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
3
b:
B
1.00
0.75
0.50
0.25
100 200 300
100
200
300
100
200
300
0
100
200
300
200 300 count
100 200 300
100 200
300
100
300 0
100 200
300
Figure & versus when ng 50.
(A) Absolute Bias
(B) Variance
[C) MSE
(D) Type Error Rate
(E) Power
0.12
0.5
0.30
0.10
0.10
0.05
0.09
0.2
0.3-
0.08
0.00
0.07
0.07
0.0
Wagnitude Hidden
Magnibude Hidden Blas
Magnitude Hidden
Magnitude Hidden
Magnitude Hidden Bias
ko Bonow Iy=1l
Contrnal Selective Borow (y-0.6)
Conformal Selective Bonow (
Method
Contormal Selective Bomow (y-8.8)
Conbrnal Selective Borow (y-0.4)
Figure Simulation results for various selection threshold y's when ng 50.
We further compared CSB with asymptotic inference to ALSB with asymptotic inference. Figure 7 shows that CSB+Asym Inf generally achieves better Type error control than ALSB+Asym Inf, while performing comparably when b = 1.
We did not compare to ALSB + FRT because, while CSB is compatible with FRT, ALSB is not readily applicable due to its computational complexity. This highlights an advantage of CSB when exact anite-sample inference is desired.
C.5. A larger sample size of ECs
Figures 8, 9, 10, 11, and 12 show the simulation results for = 300. The conclusion is similar to that in the main text.
C.6. Dependent covariates with p
We additionally consider p 5 and X ~ NO0, E), where Σ is Toeplitz matrix with p 0.6 introduce dependence among the coordinates of X. We did not consider larger since the sample size is small, with only 25 RCT controls. The simulation results (see Figure 13) show similar conclusions and demonstrate the robustness of our method.
D. More details about the real data
Pseudo-observations.. Figure 14 shows the pseudo-observations versus censored times for CALGB 9633 and NCDB, illustrating that (i) all pseudo-observations are less than equal to the truncation time of 3 years; (ii) when event occurs before 3 years, pseudd-obsserattins are generally equal to the event time; and (iii) when censoring occurs before 3 years, pseudo-observations are typically greater than the censored time.
Matching, We use nearest-neighbor matching to mitigate the covariate imbalance between CALGB 9633 and NCDB. Tumor size was imputed for eight missing values CALGB 9633 using the median of NCDB samples with missing values
23

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
(A) Absolute Bias
(6) Variance
(C) MSE
(D) Type Error Rate
(E) Power
0.12
0.7-
0.12
0.06
0.11
0.11
0.04
0.10
0.2
0.02
033
0.00
0.0
Magnitude Hidden Blas
Magnitude of Hidden Blas
Magnitude Hidden Blas
Magnitude Hidden
Magnibude Hidden Bias
Method
Acaptive Lasso Beroe
Figure Comparison of CSB FRT and ALSB asymptotic inference when 50.
Type Error Rate
Power
0.6
0.2
0.1
Magnitude of Hidden Bias
Magnitude Hidden
Method
(y=1) Asym
Adaptive
+ Asym
Conformal Selective Borrow Asym
Figure 7. Comparison of CSB + asymptotic inference and ALSB + asymptotic inference when ng 50.
covariates outside the CALGB 9633 range excluded, leaving 10,241 samples. We perform 1:1 nearest-neighhor matching using MatchIt (Ho et al., 2011), treating the sampling indicator S as "treatment" and targeting the average treatment effect on the treated (ATT). This preserves all RCT samples and matches 335 NCDB samples. Distributional balance for the baseline covariates and the estimated sampling score Tp(SS = 1(X) improves significantly after matching, with visual comparison in Figure 15. However, certain covariates, such as tumor size, remain imbalanced, which could not be addressed by matching without resorting methods that would undesirably discard RCT samples. This motivates the use of the doubly robust estimator in Sections 2.1 and Notably, while doubly robust estimator alone can address covariate imbalance, matching a pre-proceste pre-processing step reduces reliance correct model specification (Ho et al., 2007). A summary table of the pre-processed data is in Table 2.
Selection performance. Figure 16 shows that, given the observed confounder X, CSB tends to select ECs whose outcomes are more similar to randomized controls, reducing hidden bias that cannot be addressed by balancing alone.
24

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
(A) Absolute Blas
(B) Variance
(C) MSE
(D) Type Error Rate
(E) Power
0.175
3.0
0.150
0.50
1.0
0.125
0.3-
0.25
0.100
0.00
0.0
0.0
Magnitude Hidden Blas
Magnitude of Hidden Bias
Magnitude of Hidden Bias
Magnitude of Hidden Blas
Magnitude of Hidden Bias
Method
No Borrow tyml)
Bonow (ym0)
Conformal Selective Borrow
(9)
Figure Simulation results when me 300. ALSB's exact p-value unavailable due to computation.
Outcooe
-10
0.16
0.16
0.20
0.16 0.18
0.20
0.16 0.18
6.20
0.16 0.18 0.20
0.16 0.18 0.0
0.16 0.18 0.20
0.16 0.18
0.20
0.16
0.18
0.20
0.16 0.18 0.20
Sampling Score
RCT Controlled
EC (Selected)
(Uneelected)
Figure 9, Selection performance of CSB (f) when ne
300.
b:
b:
b: 3
b: 4
b:
b: B
1.00
0.75
0.50
0.25
0.00
100 200 3000
100
200
3000
100
200
3000
100 200
3000
100 200 3000
100
200
3600
100
200
3000
100
200
3000
100
200
300
count
Figure 10. 9 versus when
300.
(A) Absolute Bias
(B) Variance
(C) MSE
(D) Type E
(E)
0.6
0.7
0.15
0.6
0.10
0.10
0.4
0.30
0.05
0.00
0.2
0.3-
0.07
0.00
0.0
0.1
Magnitude Hidden Blas
Magnitude of Hidden Blas
Magnitude Hidden Blas
Magnitude Hidden Blas
Magnitude Hidden Blas
No Bonow |y=11
Conforaa Selective Borow 6)
Confornal Borrow
Method
Conformal Selective Donro (y=0.8)
Conformal Selective Dorow ()=
Figure 11. Simulation results for various selection threshold y's when ng = 300.
25

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
(A) Absolute Blas
(B) Variance
(C) MSE
(D) Type Error Rate
(E) Power
0.6
0.2-
0.2
0.09
0.2
0.50
0.1
0.0
0.0
Magnitude Hidenn Blas
Magnitude of Hidden Blas
Magnitude of Hidden Blas
Magnitude Hidden Blas
Magnitude of Hidden Bias
Acaptive Lasso Selective Borrew
Contormal Selective
Figure 12, Comparison of CSB FRT and ALSB asymptotic inference when 300.
(A) Absolute Bias
(B) Variance
(C) MSE
(D) Type Error Rate
(E) Power
0.5
0.200
0.6
0.175
0.10
0.150
0.2
0.2
0.05
0.0
4-
0.00
Magnitude Hfddd Blas
Magnitude Hidden Bias
Magnitude of Hidden Bias
Magnitude of Fiddenn Bias
Magnitude Hidden Bias
Method
Borrow ly=1)
Bonow (y=0)
Conformal S Selective Borrow 19)
Figure 13. Simulation results across different hidden bias magnitudes b for dependent covariates with p 5.
Table 2. Summary statistics of the pre-procesd pre-processed data.
C9633 Treated
C9633 Controlled
NCDB Controlled
(n1 167)
(Ng 168)
(ng 335)
Sex
Male
109 (65.3%)
106 (63.1 1%) 62 (36.9%)
219 (65.4%)
Female
58 (34.7%)
116 (34.6%)
Age (years)
Mean (SD)
60.4 (10.2)
61.2 (9.28) 62.0 [40.0, 81.0)
60.8 (9.69)
Median [Min, Max]
61.0 [34.0, 78.0]
61.0 [34.0, 80.0]
Race
White
151 (90.4%)
148 (88.1%) 1%)
300 (89.6%)
Non-white
16 (9.6%)
20 (11.9%)
35 (10.4%)
Histology
Squamous
66 (39.5%)
65 (38.7%)
131 (39.1%) 204 (60.9%)
Other
101 (60.5%)
103 (61.3%)
Tumor Size (cm)
Mean (SD)
4.60 (2.04)
4.56 (2.05)
4.77 (1.42)
Median [Min, Max]
4.00 [1.00, 12.0]
4.00 [1.00, 12.0]
4.50 [3.10, 12.0]
Outcome: 3-year RMST*
Mean (SD)
2.77 (0.596)
2.64 (0.720)
2.43 (0.947)
Median [Min, Max]
3.00 [0.383, 3.00]
3.00 [0.181, 3.00]
3.00 [0.0242, 3.00]
"Pseudo-observations transformed from censored survival time.
26

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
3
00
CALGB 9633
2
Pseudo observation
1
3-
2
NCDB
1
0
10
Censored Time
Censored
Yes
No
Figure 14, Pseudo-observation vs. Censored Time for CALGB 9633 and NCDB datasets.
27

---

Randomization Inference and Conformal Selective Borrowing in Hybrid Controlled Trials
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
0.6
0.04
0.75
Proportion
Proportion
Density
0.03
0.4
0.50
0.02
0.2
0.25
0.01
0.0
0.00
0.00
40
50
60
70
80
40
50
60
70
80
Sex
Age
Race
Unmatched
Matched
Unmatched
Matched
Unmatched
Matched
0.6
0.4
20
Proportion
0.4
Density
Density
15
0.2
5
0.0
2.5 5.0 7.510.0000 2.5 2.5 5.0 7.5
0.0
0.1
0.2
0.0
0.1
0.2
Histology
Tumor Size
Sampling Score
Sample
NCDB (S =0)
CALGB 9633 (S =1)
Figure 15. Distributional balance (unmatched and matched) between CALGB 9633 (S 1) and NCDB (S for baseline covariates and the estimated sampling score PPS X).).
3-
0000
Outcome -year RMST)
0.40
0.45
0.50
0.55
Sampling Score
CALGB 9633 Controlled
NCDB Controlled (Selected)
NCDB Controlled (Unselected)
Figure 3-year RMST (Outcome) vs. Sampling Score estimated by covariates. The shaded is constructed using quantile regression the CALGB 9633 controlled data.
28


Reviews:
[{"review_id": "anYGwrnxeY", "reviewer": "Reviewer_GG5D", "summary": "Authors study integration of external (historical) controls into randomized controlled trials in a principled way using conformal p-values, to guarantee type-1 error rates with finite samples under potential violation of the exchangeability assumption between trial and external controls.\n\n### update after rebuttal\n\nI thank the authors for their detailed response. In particular, the motivation/logic they outline in the beginning through 3 items: (i), (ii), (iii) are helpful and a more detailed and polished version should definitely be included in the updated manuscript. While those items helped me understand the contributions of the manuscript a little better, I still have concerns regarding sample splitting in the RCT to test ECs against: this  decreases power (reduced sample size in the RCT, as the samples we use for testing ECs should not be used again in downstream analyses). While there is some empirical evidence suggesting improved power overall, the fundamental tradeoff here should be more clearly analyzed theoretically. Therefore, I maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.\n\n**1. Clarifying the objective and the role of MSE minimization in improving FRT power**\n\nOur primary objective is to improve the power of the Fisher Randomization Test (FRT), which is limited when using only RCT data (No Borrowing). To achieve this, we rely on three key insights: \n\n(i) Variance and power: Theorem 2.4 (power analysis for consistent test statistics) shows that power can be improved by reducing the variance of a consistent test statistic, which can be achieved by borrowing unbiased ECs to augment the limited sample size of the RCT control arm; \n\n(ii) Bias and power: borrowing biased ECs renders the full borrowing estimator inconsistent, severely reducing the power of FRT; \n\n(iii) Bias-variance trade-off and power: these motivate the use of Conformal Selective Borrowing to improve FRT power of RCT-only analysis by borrowing unbiased ECs and discarding biased ones from a large EC pool.\n\nBased on above insights, we introduce our intermediate objective: testing the exchangeability of each EC. We acknowledge that the number of RCT controls statistically limits the power of this exchangeability testing. Therefore, we propose tuning the selection threshold $\\gamma$ (i.e., the significance level of the EC exchangeability test) to optimize our primary objective directly. While using empirical FRT power to tune $\\gamma$ would be ideal, this approach (i) requires specifying an alternative hypothesis and (ii) is computationally intensive, as computing FRT power across a grid of $\\gamma$ values is costly.\n\nThis leads to our final choice: using the MSE of the Conformal Selective Borrowing estimator as a proxy to optimize $\\gamma$. Our MSE-guided $\\gamma$ selection offers several advantages: (i) experiments show it achieves our primary goal of improving FRT power compared to RCT-only analysis, though we acknowledge that the theoretical link between MSE and FRT power is challenging due to the irregularity of the test statistic; (ii) it yields strong selection performance and supports our intermediate objective (see Fig. 2 and Fig. 14); (iii) with MSE-guided adaptive $\\gamma$, Conformal Selective Borrowing serves as both a powerful test statistic and an accurate ATE estimator; (iv) empirical MSE can be approximated by leveraging the No Borrowing estimator, and we provide a non-asymptotic excess risk bound for the adaptive procedure.\n\n**2. Finite-sample validity of FRT**\n\nFor the validity of FRT (Type I error control), we do not rely on asymptotic arguments such as estimator consistency, as shown in Theorem 2.3. This is a key advantage of FRT over existing integrative methods whose validity depends on asymptotic theory.\nFor power improvement, the optimality of the selected $\\gamma$ does rely on the consistency of the No Borrowing estimator, as shown in Theorem 3.8. However, our experiments show that Conformal Selective Borrowing with adaptive $\\gamma$ improves FRT power even in small-sample settings.\n\n**3. Significance in real-world experiments**\n\nThe real-world experiments show that Conformal Selective Borrowing (CSB) improves both robustness and efficiency over No Borrowing (NB) and Full Borrowing (FB): (i) Given that CSB+FRT theoretically controls Type I error in finite samples, which existing integrative methods do not, it yields a significant result (p < 0.05) compared to the borderline NB result (p = 0.055), addressing the underpower issue in the original study. (ii) CSB reduces the standard error by 20% compared to NB. (iii) CSB mitigates the bias of FB; the ATE estimate under CSB (0.138) is close to NB (0.142), while FB overestimates it (0.241). We acknowledge that the extent of improvement depends on the quality of the real data at hand. To further evaluate our method in finite samples, our simulations show that CSB can improve FRT power by up to 45% compared to NB.\n\n**4. Related work on historical controls**\n\nDue to the manuscript's space limitation at the original submission, we included the comprehensive literature review on historical control borrowing in Appendix A. We acknowledge the value of such a review in the main text and will move the related work on historical controls to the Introduction section at the resubmission. Please let us know if we have missed any relevant literature.", "claims_and_evidence": "The thing that bothers me the most is that it is not clear what authors claim to do and what they end up doing. I can understand the idea of using conformal inference to \"select\" external controls based on their conformity score, which is straightfoward. However the following question is not answered sufficiently in my opinion:\n\n- You use RCT controls to select in the first place. In that case, why would you not still be limited statistically by the number of controls in the RCT?\n\nThe question above is not answered/discussed verbally or as a theorem. By the latter, I mean the following. Authors claim that their new approach controls the type-1 error and improves power. I do not see a result where they \"prove\" they improve power. And they should be more clear that this \"power\" only relates to identifying whether or not the treatment has any effect, and not necessarily what is the size of the effect.\n\nA criticism to this conformal approach would be that each control is tested for pooling individually, and then the high probability guarantees are obtained straightforwardly by union bound. As authors briefly mention, this will suffer from low power and may introduce bias. This to me is the point where the paper is making its main contribution via an improved selection mechanism (correct me if I am wrong). However, my main issue here is the following:\n\n- You motivate minimizing MSE of the estimator as a proxy to guide better selection and leverage consistency of the estimators in doing that. The consistency is prone to the same issues you cite to be immune to: asymptotical validity, etc.\n- The results you have here again do not connect to an improved power for the FRT, but rather seem to be self-contained and remain as a proxy/heuristic to improve the power.\n\nPlease correct me if I am wrong. My main challenges with this paper is that I do not understand what is the one most important result its trying to prove (higher power for for FRT?) and how does the things you do/prove in the paper connect to that? I think the flow of methodology and the motivation of the paper is not very clear as it stands.", "methods_and_evaluation_criteria": "Real-world experiment is well described and interesting. It is a suitable experiment to run for this paper.", "theoretical_claims": "I did not check the proofs carefully. Most results seem to be standard from the conformal inference literature.", "experimental_designs_or_analyses": "The conformal selective borrowing approach in the real-world experiments seem to be not making a big difference compared to the no borrowing approach. An exception is slightly reduced p-vals and SE's, which might be a function of sample size alone. This falls in line with my earlier intuititon/concern regarding not having improved power as the selection is still limited by the RCT (See Claims And Evidence).", "relation_to_broader_scientific_literature": "Current manuscript focuses on a specific problem and claims to improve it using ideas from conformal inference. It does not relate to the broader scientific literature in a significant way.", "essential_references_not_discussed": "There is a vast body of work on historical controls, especially in the epidemiology literature. Current manuscript does a poor job covering that.", "other_strengths_and_weaknesses": "See Claims And Evidence", "other_comments_or_suggestions": "See Claims And Evidence", "questions_for_authors": "See Claims And Evidence", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C275oUOQtA", "reviewer": "Reviewer_f66S", "summary": "This paper proposes a randomization inference framework that can combine the data from randomized controlled trials with external controls. The proposed method controls the Type-I error in finite samples by leveraging conformal inference to select appropriate samples from external controls. In particular, the selection threshold provides an interpolation between no-borrowing and full-borrowing approaches, which can be tuned by minimizing the mean squared error. Some simulation results and real-world applications show the applicability of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.\n\n**1. Bound in Theorems 3.7 and 3.8**\n\nWe explain the key terms in Theorems 3.7 and 3.8 and their practical implications as follows: \n \n(i) The term $c\\Delta|\\delta_1|$ involves $\\delta_1$, the bias of the consistent No Borrowing estimator, which is of order $1/n$, and $\\Delta$, the maximum bias across all $\\gamma \\in \\Gamma$. This motivates us to apply pre-propensity-score matching on all ECs to prevent potentially large bias during the data preparation phase.\n\n(ii) The term $c\\Delta\\Phi\\sqrt{\\log (|\\Gamma|/\\iota)}$ involves $\\Phi$, the largest standard deviation proxy $\\phi$ across $\\gamma \\in \\Gamma$, which is of order $O(1/\\sqrt{n})$. The grid size $|\\Gamma|$ is fixed (e.g., 10 in our experiments), making this term well-controlled.\n\n(iii) The term $\\\\max (c\\Phi^2\\sqrt{\\log (|\\Gamma|/\\iota)}, c\\Phi^2\\log (|\\Gamma|/\\iota))$ arises from sub-exponential tail bounds and scales similarly to (ii).\n\n(iv) The terms $\\\\max\\_{\\gamma\\in\\Gamma} |\\hat{V}(\\hat{\\tau}\\_{\\gamma} - \\hat{\\tau}\\_1) -\\kappa_{\\gamma}^2|$ and $\\\\max\\_{\\gamma\\in\\Gamma} |\\hat{V}(\\hat{\\tau}\\_{\\gamma}) - \\sigma^2_{\\gamma}|$ corresponds to the estimation errors for $\\kappa^2_{\\gamma}$ and $\\sigma^2_{\\gamma}$. This motivates the use of a sufficiently large number of bootstrap replicates to ensure accurate variance estimation.\n\nOverall, Theorems 3.7 and 3.8 provide concrete guidance for implementing $\\gamma$ selection in practice and justify the design of our MSE-guided adaptive procedure, even if the bounds themselves are conservative due to their non-asymptotic and worst-case nature.\n\n**2. Clarifying the connection to super-efficiency-type behavior**\n\nThe reviewer questions the super efficiency of $\\hat{\\tau}_{\\hat{\\gamma}}$. We acknowledge that this was a misstatement; our intent was not to claim that it is super-efficient, but rather that it exhibits behavior similar to the Hodges estimator. We have revised the sentence as follows:\n\n\"This phenomenon highlights that $\\hat{\\tau}_{\\hat{\\gamma}}$ behaves similarly to the Hodges estimator (Le Cam, 1953) and to integrated estimators in data fusion (Yang et al., 2023; Oberst et al., 2022): improving upon the baseline estimator (here, the No Borrow estimator) in certain regions of the parameter space (where there is no bias in ECs) inevitably leads to worse performance in other regions (where the bias in ECs is difficult to detect).\"\n\n**3. Additional simulations**\n\nWe additionally consider $p = 5$ and $X \\sim N(0, \\Sigma)$, where $\\Sigma$ is a Toeplitz matrix with $(i,j)$-th entry $\\Sigma_{ij} = \\rho^{|i-j|}$ and $\\rho = 0.3$, to introduce dependence among the coordinates of $X$. We did not consider larger $p$ since the sample size is small, with only 25 RCT controls. The simulation results (see [here](https://anonymous.4open.science/r/doc-E6B4/sim_supp.pdf)) show similar patterns and demonstrate the robustness of our method.\n\n**4. $\\Phi$ and $\\epsilon_\\gamma$**\n\n$\\Phi$ is the largest standard deviation proxy $\\phi$ over $\\gamma \\in \\Gamma$. If $\\epsilon_\\gamma$ is Gaussian, then $\\phi$ equals its standard deviation. We previously mislabeled $\\phi$ as a variance proxy and have corrected this.\n\n**5.  Value and performance of $\\hat{\\gamma}$**\n\nSince $\\hat{\\gamma}$ is adaptive and varies across simulation replicates, we separately report its values in Figure 4 and analyze its behavior in Section C.2.\n\nWe do not expect $\\hat{\\gamma}$ to outperform all fixed $\\gamma$ values uniformly. Prior work has shown that no method can uniformly outperform No Borrowing (corresponding to $\\gamma = 1$) without additional assumptions (Oberst et al., 2022; Lin et al., 2024). Instead, our proposed $\\hat{\\gamma}$ is designed to improve power when the bias of ECs is either absent or detectable. When the bias is difficult to detect, it is reasonable that our method may incur some inevitable power loss, though within an acceptable range. Importantly, Conformal Selective Borrowing+FRT always controls the Type I error, even in such challenging cases.\n\n**6. Presentation**\n\nWe removed the sentence \"Based on...\", corrected the missing division symbols, and zoomed in Figure 2(D).\n\n**7. Ethical issue**\n\nAs per ICML guidelines, the Impact Statement is not counted toward the 8-page limit. We initially understood this to apply to the Software and Data section and had no intention of violating the page limit.\n\nWe did not disclose any identities or URLs in the submission. We mentioned the availability of the R package solely to highlight the applicability and reproducibility of our work. We sincerely apologize if our identity may have been inadvertently revealed in the package documentation, which is external to the submission; this was entirely unintentional and not meant to compromise anonymity. To avoid any potential concern, we have removed the Software and Data section from the manuscript.", "claims_and_evidence": "Basically, the claims in the submission are clear and supported by proofs or simulation studies. However, I feel that the bounds in Theorems 3.7 and 3.8 are loose and not easy to comprehend.\n\nIn addition, I don't quite understand why the authors can relate $\\hat{\\tau}_{\\hat{\\gamma}}$ to super efficiency on the second column of Line 295. Can the authors provide some theoretical justification to this claim?", "methods_and_evaluation_criteria": "The proposed methods make sense to me. However, for the simulation setups, the dimension of covariates is $p=2$, which is too small and won't lead to meaningful conclusions. Can the authors try some larger values for $p$? Also, the current covariates $X$ have independent coordinates. Can we introduce any dependence within the coordinates of $X$?", "theoretical_claims": "I basically checked all the proofs, and they looks correct. A minor question is how $\\Phi$ is related to variances of $\\epsilon_{\\gamma}$. Can the authors provide some short discussion on it?", "experimental_designs_or_analyses": "I checked the experimental results and already pointed out my concerns for the simulation setups above. \n\nMoreover, for Figures 5 and 10, the authors should also report the value of $\\hat{\\gamma}$. The reason is that the proposed method CBS under the \"optimal\" value $\\hat{\\gamma}$ performs not as good as the cases under other values of $\\gamma$. This is a severe issue that may limit the impact of this paper.", "relation_to_broader_scientific_literature": "This paper combines the approaches in conformal inference literature with other randomization tests to propose a new framework for combining the data from randomized controlled trials with external controls. The estimator relies on the one proposed by Li et al. (2023), but the authors also relax the mean exchangability condition imposed by Li et al. (2023).\n\n\nLi, X., Miao, W., Lu, F., and Zhou, X.-H. Improving efficiency of inference in clinical trials with external control data. Biometrics, 79(1):394–403, 2023.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "As I mentioned above for the simulation results, the proposed method under the optimal choices of selection parameter does not seem to perform as good as the case under other arbitrary choices.", "other_comments_or_suggestions": "1. Second column of Lines 178 to 182: The sentence \"Based on...\" seems to repeat what have been discussed in the last paragraph.\n\n2. Second column of Line 225: There should be a division in the definition of $p_j^{split}$. The same issues happen to $p_j^{jackknife+}$ and $p_j^{cv+}$.\n\n3. The plot in Figure 2(D) needs to be zoomed in.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Other expertise"], "ethical_review_concerns": "The paper exceeds the 8-page limits and potentially discloses the authors' identities."}}, {"review_id": "VguIRAHNX0", "reviewer": "Reviewer_U7hX", "summary": "This paper proposes a method for combining (potentially biased) external controls (ECs) with data from a randomized control trial, in an effort to improve power to detect causal effects, without sacrificing Type 1 error (false positive) control.  For controlling Type 1 errors, the key insight is to use a Fisher Randomization Test (FRT), which allows for computation of exact p-values for Fisher's \"sharp null\" hypothesis (potential outcomes are constant across treatment/control) with any test statistic, including those using potentially biased ECs.  For improving power, there is no \"free lunch\" (as noted by the authors and other related work), but an approach is proposed for \"selective borrowing\" of less biased ECs, which is illustrated empirically in simulated data and a real-data application to a chemotherapy randomized trial.\n\n**Update after rebuttal**: As stated below, I will keep my score.  My impression of the paper is somewhere between a 3 and a 4.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.\n\n**1. Clarification on sharp null hypothesis**\n\nThe sharp null hypothesis $Y_i(1) = Y_i(0)$ for all $i \\in$ RCT states that there is no individual treatment effect for any unit. Another common form of the sharp null is conditional independence, $Y_i \\perp A_i \\mid X_i$, meaning the observed outcome is independent of treatment assignment given covariates. In contrast, the hypothesis that the average treatment effect (ATE) is zero is often called the \"weak null.\"\n\nIn a finite-sample exact sense, FRT guarantees Type I error control under the sharp null but cannot guarantee it under the weak null. However, recent work has shown that FRT can also asymptotically control the Type I error under the weak null using studentized or pre-pivoted test statistics (Wu & Ding, 2021; Cohen & Fogarty, 2022). The RCT sample size is typically small in our context, such as trials for rare diseases. Asymptotic approximations may be unreliable, so we rely on the sharp null for a finite-sample exact test.\n\nWhile this reliance on the sharp null rather than the weak null may be viewed as a limitation of FRT, to our knowledge, there is currently no testing procedure that controls Type I error in a finite-sample exact sense under the weak null without additional distributional assumptions. Section 6 also discusses possible future directions beyond the sharp null.\n\n**2. Significance of the Conformal Selective Borrowing (CSB) and comparison to Adaptive Lasso Selective Borrowing (ALSB)**\n\nCompared to existing selective borrowing approaches such as ALSB, the significance of CSB can be summarized as follows:\n\n(i) Model-free flexibility: CSB is a model-free approach that allows flexible choice of conformal scores depending on data characteristics. For example, in our real-world application where the outcome exhibits heavy tails and heteroscedasticity (see Figure 14), we use conformalized quantile regression (Romano et al., 2019) for selection. Distance-based scores such as nearest-neighbor conformity scores (Shafer & Vovk, 2008) can be used for binary outcomes. This flexibility is difficult to achieve with model-based methods like ALSB.\n\n(ii) Estimation: CSB performs better than ALSB in terms of MSE under small and moderate sample sizes. This is demonstrated in Figures 6(C) and 11(C), where both methods are compared purely on estimation accuracy without involving inference procedures.\n\n(iii) Computation: CSB is compatible with the FRT, while ALSB is not readily applicable with FRT due to its computational complexity. This highlights an advantage of CSB when exact finite-sample inference is desired.\n\n(iv) Apples-to-apples comparison under asymptotic inference (Asym): We conducted a comparison between CSB+Asym and ALSB+Asym (see [here](https://anonymous.4open.science/r/doc-E6B4/sim_csb_asym.pdf)). CSB+Asym generally achieves better Type I error control than ALSB+Asym and performs comparably when $b=1$.\n\n**3. Novelty and implications of theoretical results**\n\nTheorem 2.3 builds on classical FRT results (e.g., Lehmann and Romano, 2005, Theorem 15.2.1), but to our knowledge, this is the first formal result establishing the validity of FRT in the context of hybrid controlled trials. It confirms that FRT can be applied in this setting and clarifies how it should be applied to ensure validity. Specifically, Theorem 2.3 provides the following practical guidance and important caveats:\n\n(i) Type I error control is guaranteed only when we permute assignments according to the actual experimental design, that is, permuting treatment assignments only within the RCT while keeping the EC assignments fixed. An important caveat is that permuting across all treated and control units, including ECs, would invalidate the theorem.\n\n(ii) To maintain validity, the test statistic should vary with the permuted treatment vector $A$, meaning the selected set of ECs should also be updated under each permutation to account for selection uncertainty. Fixing the EC selection across permutations would also invalidate the theorem.\n\nTheorems 3.7 and 3.8 provide novel non-asymptotic MSE bounds that guide the design of the adaptive selection threshold in finite samples. Please also see our response to Reviewer f66S, Point 1.\n\n**4. Post-selection inference**\n\nThe reviewer's understanding is correct. The selection uncertainty is fully incorporated into the reference distribution by allowing the selected set to vary with the resampled treatment vector $A$ in the FRT. Therefore, there is no post-selection inference concern when following this principle. However, as noted in the previous point, a key caveat is that the selected EC set should not be fixed during permutation, as this would ignore selection uncertainty and invalidate the test.\n\n**5. Presentation**\n\nWe moved the related work to the main text, revised Line 056, and enlarged Figure 1.", "claims_and_evidence": "The main claims as I understood them:\n* Their approach will have valid level / false positive rate in finite samples for the Fisher sharp null.\n* Their power can be better than \"no borrowing\" (i.e., just using the RCT) if the bias is small\n* Their approach is better suited to finite samples than other selective borrowing approaches like Gao et al. 2023.\n\nThe claims are supported by clear and convincing arguments, and backed up by synthetic experiments where the bias can be controlled.  I did find one claim a little difficult to understand (regarding post-selection inference), which I have mentioned in \"Questions for the Authors\" below, but I think this is a minor point.", "methods_and_evaluation_criteria": "Yes, for the most part (one important missing baseline, excluded due to computational concerns). See my comment on \"relation to the broader scientific literature\"", "theoretical_claims": "I did not see any issues in the proofs. I checked the proof of Theorem 2.3, which seemed fairly immediate from the definition of the FRT, and Theorem 2.4, which similarly seems to follow fairly directly from auxiliary lemmas from related work (which I did not check).  While I did not review the proofs of Propositions 3.1-3.6 in depth, they seem to follow from standard arguments, though I'm a little less familiar with the conformal prediction literature.  Theorems 3.7 and 3.8 follow from standard results (and some algebra, which I did not check in detail) in non-asymptotic statistics.", "experimental_designs_or_analyses": "The experimental design for the synthetic data seems sound to me, and is broadly similar to how I would have set up a synthetic data experiment for this method, probing failure modes as a function of the bias. The real-data experiment is more an \"illustrative application / case study\" rather than an experiment that tests a particular hypothesis, but that's a fairly standard approach for these types of papers in my experience.", "relation_to_broader_scientific_literature": "This paper adds yet another method to a growing literature (well-documented in Section A) on combining observational and experimental data in an adaptive fashion, without making assumptions on the validity of the observational data.\n\nThe core insight of this paper, in my view, is that Fisher Randomization Tests can be used with any test statistic, and therefore they are a good candidate for \"improvement\" with external controls, since the false positive rate is controlled exactly regardless of how good the test statistic is.\n\nSimilar to other work in this area, this paper identifies, at least experimentally, a no \"free lunch\" phenomenon, , i.e., there are moderate biases which can lead to loss of performance. However, beyond some intuition-building theory (e.g., Theorem 2.4), the authors do not formally analyze this relationship between bias and power.  That said, I am sympathetic to the difficulty in doing so.\n\nIt is harder to judge the significance of the conformal selective borrowing approach.  While it is novel, it's not clear (on it's own, without the FRT component), how this approach compares to other selective borrowing approaches, especially given the lack of experimental comparison (on an apples-to-apples basis) with the Adaptive Lasso Selective Borrowing (ALSB) approach.", "essential_references_not_discussed": "Overall, I found that the related work (in the Appendix) was quite comprehensive, but would have appreciated having more of that content in the main paper, if space permits.", "other_strengths_and_weaknesses": "Beyond the lack of direct apples-to-apples comparison with ALSB, I have two (relatively minor) concerns regarding clarity / significance.  \n\n1. I'm not sure how to square the null hypothesis tested under the FRT (the \"sharp null\") with the more conventional null hypothesis that e.g., the average treatment effect is equal to zero, which doesn't require that $Y_1 = Y_0$, but rather just that $E[Y_1] = E[Y_0]$.  Hence, while the FRT controls the false discovery rate under the sharp null, it's not clear that it controls the false discovery rate under more conventional null hypotheses.  It may be worth clarifying this point in the paper, or otherwise commenting on what is \"lost\" by relying on FRT.\n2. It's not clear to me how novel / original some of the theoretical results are, and it would be worth clarifying which theoretical results are considered most novel by the authors.  Theorem 2.3, for instance, seems like an obvious consequence of FRT, and I'm surprised that it's given as a Theorem in this paper, as opposed to being cited from elsewhere (though given my lack of familiarity with the FRT literature, I don't have a citation to provide).", "other_comments_or_suggestions": "Some minor grammar / presentation points:\n* On Line 056, right-hand column: \"Let the binary treatment denote by $A$\" -> \"Let $A$ denote the binary treatment\"\n* Figure 1 is extremely small.  Note that you can use figure* in two-column formats to get a figure that crosses both columns, which would be better for this figure.", "questions_for_authors": "I have listed these in priority order, with the first two questions being particularly important to clarify.\n\n1. I am less familiar with randomization inference (a la the FRT).  Does this procedure also provably control the false positive rate under the more conventional general null hypothesis that $E[Y_1 - Y_0] = 0$?  I would assume not, since it takes $Y_1, Y_0$ as fixed, as opposed to random?\n2. How does CSB compare to ALSB under an apples-to-apples comparison?  For context: The comparison in Figure 6 is not \"apples to apples\", instead it shows FRT + CSB versus ALSB with asymptotic inference.  It would be nice to see either (a) FRT+CSB versus FRT+ALSB, or (b) CSB and ALSB considered head-to-head where both use asymptotic inference.\n3. Which of the theoretical results are the most novel, in the view of the authors?  To me, it seems like most of the theory is relatively straightforward, and the main contributions are a bit more conceptual (e.g., the realization that one can use the FRT here), but I'm open to arguments that there are more novel theoretical contributions here.\n4. What is the concern with \"post-selection inference\" here?  I don't quite follow the claim (see 090-092 LHS, \"We account for selection uncertainty in FRT and offer valid post-selection inference\").  The only justification I see is on lines 222-225 (LHS), \"By using $T(A) = |\\hat{\\tau}_{\\gamma}|$ as the test statistic for FRT and allowing $\\mathcal{E}(\\gamma)$ to vary with resampling A in FRT could account for selection uncertainty and provide valid post-selection inference\". This is mainly just a conceptual argument for why there is no post-selection inference concern, right?  I.e., under the null, the entire procedure is run to get a sampling distribution of the test statistic, which can depend in arbitrary ways on the data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uL5fzmemHt", "reviewer": "Reviewer_uA9f", "summary": "This paper proposes to use Fisher Randomization Test (FRT) in RCTs when leveraging external controls (EC). Since FRT only uses the randomization distribution of a test statistic under the sharp null, it always provides valid type-I error control regardless of how the potentially biased ECs are incorporated. In this way, ECs are used as a tool for constructing more powerful test statistics. In doing so, the authors recognize the issue of bias in ECs, and propose to improve the test statistic by selectively borrowing ECs by thresholding their conformal p-values. To choose the threshold, a cross validation paradigm is also proposed. The proposed methods are demonstrated via simulations and real data illustrations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.\n\n**1. Connection between MSE of $\\hat{\\tau}_\\gamma$ and power of FRT**\n\n(i) Variance and power: Theorem 2.4 (power analysis for consistent test statistics) shows that a lower variance of a consistent test statistic leads to a higher power of FRT.  \n\n(ii) Bias and power: Simulation-based power analysis demonstrates that borrowing biased ECs can severely reduce FRT power.\n\n(iii) MSE and power: Taking both variance and bias into account, we observe that a lower MSE of the test statistic is associated with higher FRT power. While it remains theoretically challenging to link MSE and power when the test statistic is irregular, our experiments show that MSE-guided $\\gamma$ selection offers a unified solution that performs well for both estimation accuracy and power.\n\nWe also revised our storyline as follows:\n\nOur primary goal is to improve the power of the Fisher Randomization Test (FRT) for testing the null hypothesis of no treatment effect by borrowing external controls (ECs), compared to relying solely on RCT data (No Borrowing). This is guided by three key insights: (i) Theorem 2.4 shows that the power of FRT can be increased by reducing the variance of a consistent test statistic, which can be achieved by borrowing unbiased ECs to augment the small RCT control sample; (ii) borrowing biased ECs leads to inconsistency and significantly reduces the power of FRT; (iii) these insights motivate our use of Conformal Selective Borrowing to enhance power by borrowing unbiased ECs and discarding biased ones from a larger EC pool.\n\nTo this end, we introduce an intermediate objective: testing the exchangeability of each EC and selectively borrowing those deemed unbiased. We recognize that the power of this intermediate testing step is limited by the size of the RCT control group. To address this, we propose tuning the selection threshold $\\gamma$ (the significance level of the exchangeability test) to directly target our primary objective. Although tuning based on empirical FRT power would be ideal, it is impractical because (i) it requires specifying an alternative hypothesis and (ii) it is computationally expensive to compute power across a grid of $\\gamma$ values.\n\nAs a practical alternative, we use the MSE of the Conformal Selective Borrowing estimator as a proxy to guide $\\gamma$ selection. This MSE-guided approach offers several benefits: (i) experiments show it improves FRT power over RCT-only analysis, even though the theoretical connection between MSE and FRT power is challenging due to the irregular nature of the test statistic; (ii) it yields strong selection performance and supports our intermediate objective (see Fig. 2 and Fig. 14); (iii) with MSE-guided $\\gamma$, CSB serves as both a powerful test statistic and a accurate ATE estimator; (iv) empirical MSE can be approximated leveraging the No Borrowing estimator, and we provide a non-asymptotic excess risk bound for the adaptive procedure.\n\n**2. Conformal selective borrowing (CSB) as a powerful test statistic and a reliable, efficient estimator**\n\nAlthough our primary motivation is to improve the power of FRT, the final method, CSB with MSE-guided $\\gamma$ selection, applies to both estimation and testing:\n\n(i) Theorems 3.7 and 3.8 provide non-asymptotic excess risk bounds for the proposed estimator.\n\n(ii) In simulations, CSB shows better estimation performance than adaptive lasso selective borrowing (ALSB) in terms of MSE under small and moderate sample sizes. This is demonstrated in Figures 6(C) and 11(C), where both methods are compared purely on estimation accuracy without involving inference procedures.\n\n(iii) In real-world experiments, CSB improves both robustness and efficiency over No Borrowing (NB) and Full Borrowing (FB). Specifically, CSB reduces the standard error by 20% compared to NB and mitigates the bias of FB. The ATE estimate under CSB (0.138) is close to NB (0.142), while FB overestimates it (0.241).\n\nThese results suggest that CSB is not only effective for testing under sharp nulls but also serves as a reliable and efficient estimator of treatment effects.\n\n**3. Presentation**\n\nFollowing the reviewer’s suggestion, we moved the full conformal $p$-value (which is computationally infeasible) and jackknife+ (a special case of CV+) to the appendix. We kept the split conformal $p$-value and CV+ $p$-value in the main text, as both are used in our experiments.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I briefly checked the proofs which seem correct to me.", "experimental_designs_or_analyses": "I checked the simulation setups and they seem sound.", "relation_to_broader_scientific_literature": "- Propose the use of FRT in borrowing information from ECs in RCT analysis.\n- Demonstrate the use of selective borrowing in developing more powerful test statistics. \n- Demonstrate the use of conformal inference in selective borrowing.", "essential_references_not_discussed": "I didn't find important ones to my knowledge.", "other_strengths_and_weaknesses": "Strengths:\n- The results are rich. \n- Propose the importance of FRT in RCT analysis and suggest a new way of using ECs. \n\nWeakness: \nIn general, this paper contains a lot of results but the motivation and results can be organized in a better way. I list some confusions when reading the paper: \n- The comparison with existing methods which motivates the method is not fully reasonable. In particular, the hybrid doubly robust estimators are aimed for providing more accurate estimators, but the authors propose to switch gears to a randomization test for the sharp null, which makes it not fully comparable with the existing method. This makes the arguments that motivate the current proposal a bit weak. \n- The definition of conformal p-values are not new in this paper, and it takes too much space in the paper. \n- I'm not sure what is the connection between the MSE of $\\hat\\tau$ and the power of the resulting RCT. In reading the paper, I sometimes felt like the authors wanted to improve MSE of the test statistics but sometimes thought they want to improve the power of RCT. I would suggest the authors clean up the storyline.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Is there a connection between MSE of $\\hat\\tau$ and power of RCT?\n- Is it possible to get back to the estimation problem based on this approach?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method to integrate observational external controls with data from a RCT to estimate causal effects while controlling the risk that the external controls introduce bias into the estimates. Most reviewers found the introduction of Fisher randomization test, alongside an adaptively tuned selective borrowing step, to be worthwhile additions to the literature that would be of interest to others working on the problem of integrating RCTs and observational data. The main concern, raised by one reviewer, is that the tuning in the selective borrowing step targets a proxy objective (MSE) instead of the true power. In my view this is a reasonable choice and the concerns here can be addressed by revising the paper to be clearer about what strategy is being chosen and why.
Decision: Accept (poster)