Title: Adaptive Data-Borrowing for Improving Treatment Effect Estimation using External Controls
Venue: NeurIPS 2025
Accepted: yes
Keywords: Adaptive Selection

Causal Inference

Data Fusion

Efficiency Improvement
Abstract: Randomized controlled trials (RCTs) often exhibit limited inferential efficiency in estimating treatment effects due to small sample sizes. In recent years, the combination of external controls has gained increasing attention as a means of improving the efficiency of RCTs. However, external controls are not always comparable to RCTs, and direct borrowing without careful evaluation can introduce substantial bias and reduce the efficiency of treatment effect estimation. In this paper, we propose a novel influence-based adaptive sample borrowing approach that effectively quantifies the "comparability'' of each sample in the external controls using influence function theory. Given a selected set of borrowed external controls, we further derive a semiparametric efficient estimator under an exchangeability assumption. Recognizing that the exchangeability assumption may not hold for all possible borrowing sets, we conduct a detailed analysis of the asymptotic bias and variance of the proposed estimator under violations of exchangeability. Building on this bias-variance trade-off, we further develop a data-driven approach to select the optimal subset of external controls for borrowing. Extensive simulations and real-world applications demonstrate that the proposed approach significantly enhances treatment effect estimation efficiency in RCTs, outperforming existing approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to use external control data to improve inferential efficiency in estimating treatment effects for randomized controlled trials (RCTs), especially those with small sample sizes. The main difficulty arises from the individual heterogeneity; that is, the potential outcomes of external control units may be different from those of RCT control units. To address this issue, the authors proposed an influence-based adaptive sample borrowing approach that quantifies the comparability of each unit in the external control and adaptively adds external units to enhance the efficiency of estimating treatment effects. Specifically, the method calculates an influence score for each external unit based on the change of a pre-specified loss function when the external unit is added to learn the model parameters. Then, the authors derived a semiparametric estimator for the average treatment effect (a doubly robust estimator), analyzed its bias, and proved asymptotic normality of the estimation error.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1. The research problem is well-motivated. Due to small sample sizes of RCTs, one wants to combine external control data to improve the efficiency of estimating treatment effects. However, most existing works rely on a potentially implausible assumption that external control and RCT control units are exchangeable. The authors aim to drop this exchangeability assumption. The introduction is well written.\n\n2. The authors proposed a new influence-based method to adaptively borrow external control units. This new method can handle external outliers effectively.\n\n### Weaknesses:\n1. Compared with [1], it seems that the main difference appears to be how to select the external control units. The semiparametric estimator and the asymptotic normality of the estimation error seem similar to those in [1]. The authors could consider providing a more detailed comparison with [1].\n\n2. While the authors presented numerical evidence showing improvement over [1] when the external control data contain outliers, the authors could consider providing some theoretical results that specify the conditions under which their method has a smaller MSE.\n\n3. The authors should provide more details on how to estimate the parameters in their estimator, such as $\\hat{\\pi}$ (I understand that the authors mentioned $\\hat{m}$ is estimated using standard machine learning methods, but I don't think they discussed $\\hat{\\pi}$). In addition, I think the propensity score of the RCT units (i.e., $\\hat{e}_1$) is known by the RCT design.\n\n4. I am confused about Section 6.2 -- find the \"optimal\" borrowed set. Could the authors elaborate on how the borrowed set S that minimizes the MSE is selected? Do you greedily add external units based on their influence scores? If so, why does this method find the optimal S?\n\n5. The authors may also compare their method with other classical methods, such as matching.\n\n6. I think the authors should mention that they assume that the unit data are i.i.d. samples. This assumption is used in their proof of asymptotic normality of the estimation error.\n\nReferences:\n[1] Chenyin Gao, Shu Yang, Mingyang Shan, Wenyu YE, Ilya Lipkovich, and Douglas Faries. Improving randomized controlled trial analysis via data-adaptive borrowing. Biometrika, page asae069, 2024.\n\nQUESTIONS:\n1. I am confused about Proposition 1. What is the \"efficient influence function\"? It seems [1] also provides a formula for the efficient influence function under the same assumptions if one replaces $P$ with $P_S$. Is the formula in Proposition 1 different from that in [1]? If so, why are they different? In addition, what is the parameter $q$ in Proposition 1?\n\n2. In Section 5, when learning the model $\\hat{\\theta}_z$ parameters by the empirical risk minimizer, there is a regularizer term; however, when using the influence function to approximate the model change $\\hat{\\theta}$$\\epsilon, z$, there is no regularizer term. Could the authors explain this discrepancy?\n\n3. In the experiment setup for the NSW data, could the authors explain how the unobserved outcomes are generated? Every unit has a missing potential outcome. I assume that one needs all potential outcomes to calculate the MSE?\n\n4. In Proposition 2 and Theorem 1, should $(N_E + N_S) (\\hat{\\tau}_S - \\tau) \\rightarrow N(0, \\sigma^2)$ be $\\sqrt{N_E + N_S} (\\hat{\\tau}_S - \\tau) \\rightarrow N(0, \\sigma^2)$?\n\nReferences:\n[1] Chenyin Gao, Shu Yang, Mingyang Shan, Wenyu YE, Ilya Lipkovich, and Douglas Faries. Improving randomized controlled trial analysis via data-adaptive borrowing. Biometrika, page asae069, 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to improve treatment effect estimation in RCTs by adaptively borrowing external control samples without relying on the exchangeability assumption. The key innovation is an influence-function-based approach that quantifies each external sample’s comparability to the RCT controls through its impact on the outcome model, ranking samples by influence scores. The method constructs nested candidate borrowing sets and derives a semiparametric efficient estimator combining RCT data with these selected controls. The authors analyze asymptotic bias and variance under violations of exchangeability and propose data-driven selection of the optimal borrowed set by minimizing MSE. Simulations and real-world application (NSW/PSID) demonstrate the method’s superiority over existing approaches, including full and adaptive lasso-based borrowing (Gao et al).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper addresses a practically important problem of borrowing external controls when exchangeability may not hold.\n* The influence-based approach is relatively novel, theoretically grounded, and provides a principled way to assess sample comparability.\n* The paper presents rigorous asymptotic analysis of bias and variance.\n* Extensive simulations and real-data application demonstrate improvements over existing methods.\n* The paper is generally well-organized and clearly motivated, helping readers understand the method’s intuition.\n\nWeaknesses:\n* The paper lacks guidance on inference, including how to construct valid confidence intervals for the proposed estimator.\n* There is no practical strategy for choosing K, leaving practitioners without a clear selection procedure for the optimal number of borrowed samples.\n* The method’s robustness to other distributional shifts (e.g., outcome or treatment assignment mechanisms) beyond covariate shift is not explored.\n* The computational burden of influence score calculation, especially for large-scale or high-dimensional data, is not addressed in detail.\n* The introduction could better situate the work in the broader literature on generalizability, transportability, and federated causal inference.\n\nQUESTIONS:\n1. Inference and Confidence Intervals: The paper does not provide clear guidance on how to construct valid confidence intervals for the proposed estimator, especially given the adaptive selection of borrowed samples. Could the authors clarify whether the estimator is regular and root-n consistent? If not, how should practitioners approach uncertainty quantification?\n\n2. Choice of K (Number of Borrowed Samples): While the paper proposes ranking external controls by influence scores, there is no practical guidance on selecting the optimal K. Should practitioners perform a grid search over candidate Ks? Could the authors propose a data-driven or theoretically motivated stopping rule?\n\n3. Beyond Covariate Shift: The paper considers covariate shift (differences in X distributions) in simulations but does not address other forms of distribution shift between RCT and external controls, such as differences in Y|A,X (conditional outcome) or A|X (treatment assignment mechanism). Could the authors discuss the approach’s robustness to these more general types of shift in theory and simulations?\n\n4. Detectability of Bias and Limits of Borrowing: What are the limits on the degree of mismatch between RCT and external controls before the method’s bias detection and correction fail? Could the authors quantify scenarios where differences across data sources become too large for reliable borrowing?\n\n5. Comparison with Existing Methods: In what scenarios might the adaptive lasso-based approach (e.g., Gao et al.) outperform the influence-based method? A discussion of failure modes or trade-offs would help practitioners choose between methods.\n\n6. Related Literature: The introduction could better & more broadly be situated in the literature on generalizability/transportability of causal effects (e.g., work on extending causal inferences to new populations or data integration/fusion, see Dahabreh et al.). Relevant papers also include methods for federated causal inference (e.g., Xiong et al, 2023 SIM; Han et al., JASA 2025), etc.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an influence function-based adaptive sample borrowing method to enhance statistical efficiency in randomized controlled trials by incorporating external controls. The paper presents a data-driven selection strategy to optimally manage the bias-variance trade-off under potential exchangeability violation when identifying external control samples for borrowing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is clearly written and well-structured. It makes an effort to provide intuitions and examples to explain the motivation and the proposed methodology. \n\nWeakness:\n- Novelty: It appears that the proposed method is built upon the semiparametric efficiency results derived in Section 6.1. However, similar results can be found in Li et.al, 2023 (reference 9 cited in the paper). Could the authors provide more discussions on this connection? \n- Contributions: The top-k selective borrowing strategy and the relaxation of the exchangeability assumption appear to be the paper's primary contributions, but these points are not sufficiently highlighted. To clarify the selective borrowing component, it might be helpful to explicitly formulating the top-k selection as an objective function, perhaps under Eq. (5)? Regarding the relaxation of the exchangeability assumption, highlighting and elaborating further on lines 235 to 239 could strengthen the presentation of this contribution.\n- Theoretical results: It would be helpful to have some theoretical results (e.g., remarks) on the efficiency gain of the proposed methods.\n\nQUESTIONS:\n1. Section 5, line 181 to 186, how to decide the optimal $K$? What if there are ties among the influence scores? \n2. In practice, how to determine whether Assumption 3 is violated? Could the authors comment on this? \n3. Regarding the top-K selective borrowing approach, is the selected subset consistent? It would be helpful if the authors could provide further explanation or theoretical guarantees regarding the set selection consistency.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an influence-function-based framework for adaptively borrowing external control samples to improve treatment effect estimation in RCTs, particularly when sample sizes are small. The method assigns influence scores to external samples based on their perturbation to the RCT control model, allowing selection of the most comparable data without assuming full exchangeability. A semiparametric efficient estimator is then developed, and a data-driven approach selects the optimal subset of external controls by minimizing mean squared error. Through both simulation and real-world experiments, their proposed method demonstrates superior accuracy and robustness compared to existing approaches including adaptive lasso-based borrowing.\n\nSTRENGTHS AND WEAKNESSES:\nThe overall quality of the work is high. The authors provide both theoretical analysis and empirical results, establishing consistency, asymptotic normality, and semiparametric efficiency of their proposed estimator. They extend their results to accommodate violations of the exchangeability assumption, and use a bias-variance trade-off to drive a data-adaptive selection of borrowed samples. The experimental results are thorough, and the method demonstrates robustness across different covariate shifts and RCT sample sizes. One limitation is the computational burden associated with calculating influence scores, particularly for high-dimensional models requiring second-order derivatives. While this is acknowledged, a deeper exploration of scalability and approximate alternatives would strengthen the practical applicability of the method.\n\nThe clarity of the manuscript is generally strong. The structure is logical, with clear motivation, definitions, and progression through theoretical development and empirical evaluation. Mathematical notation is precise, though occasionally dense, and might benefit from simplified exposition in a few sections. Visuals such as borrowing diagrams and performance plots are well-designed and help convey the main results effectively. Implementation details and reproducibility criteria are addressed in full, with code and data availability confirmed.\n\nThe significance of the work is reasonably high. The method targets a well-known limitation in modern trial design and real-world evidence generation—the inability to use external data when the exchangeability assumption is questionable. By moving beyond global assumptions to local, sample-level borrowing decisions, the authors propose a strategy with clear practical implications for trials in rare diseases, oncology, and digital experimentation, where small sample sizes are common. However, the application remains focused on a single classical dataset (NSW/PSID). Demonstration of the method in more contemporary, high-dimensional settings such as electronic health records or biobanks would further enhance its relevance.\n\nThe originality of the paper is also fine. The use of influence functions to guide data borrowing in causal inference settings is a novel contribution that differentiates this work from prior lasso-based and bias-penalized approaches. The estimator unifies ideas from semiparametric efficiency theory, causal inference, and robust learning, and contributes new tools to an important methodological frontier. The method has the potential to inspire further work in areas such as transfer learning for treatment effect estimation or adaptive trial design. \n\nIn summary, this paper makes some contribution to the literature on treatment effect estimation and external data integration. It presents a theoretically grounded, empirically validated, and practically relevant methodology.\n\nQUESTIONS:\nit would be helpful to clarify 1) the influencial validity after adaptive selection; 2) how are the influence score estimation implemented in practice especially in high dimensional settings; 3) provide justification for the approximation of bias as the difference from the AIPW estimator, which tends to have high variability in small samples and hence unreliable.\n\nLIMITATIONS:\nThere is a notable limitation in their inference - although the author acknowledges a bias-variance-trade-off due to the selection of the comparable subset, ignoring this selection or simply minimizing it may not adequately address the post-selection-inference issue. There is no formal accounting for the variability introduced by the selection part. The theoretical results are derived as if the subset were fixed. The paper sidesteps this with asymptotics and MSE estimation, a rigorous treatment would require additional methods or caveats.  \n\nthe simulation settings are overly simplistic - in most RCTs, we observe complex covariates including counts, skewed distributions. It would also be important to consider heterogeneous treatment effects. The authors should compare to other methods such as AIPW, causal transfer or domain adaptation approaches. Authors should consider more modern real datasets such as UK Biobank or MIMIC.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm that borrows information from external controls using an influence-based criterion to improve the estimation of treatment effects. Some theoretical results are developed for the proposed algorithm, and its numerical performance is evaluated through a synthetic example and a real-data application. Overall, the paper is well written and the presentation is clear.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: the paper is well-written and the presentation is clear.\nWeakness: the theory part seems a bit weak.\n\nQUESTIONS:\n1. Assumption 3 appears to be very strong. It essentially assumes that the selection procedure can identify unbiased external controls with probability tending to one. However, the paper does not demonstrate that the sample-borrowing procedure described in Section 5 guarantees this assumption in practice. Given this, the resulting theoretical results in Section 6.1 seem to reduce to those for a standard AIPW estimator with an extended control set, which limits their novelty and practical value.\n\n2. Although Assumption 3 is relaxed in Theorem 1 (Section 6.2), the theory still treats the candidate set $\\mathcal{S}$ as deterministic. In reality, the selection process for $\\mathcal{S}$ is stochastic. Ignoring this stochasticity may lead to an inaccurate or overly optimistic theoretical characterization of the estimator’s properties.\n\n3. How was the propensity score estimated in the numerical experiments? Introducing external controls could potentially complicate the estimation of the propensity score, which may in turn increase the variability of the proposed borrowing algorithm. Some discussion or diagnostic results would help clarify this point.\n\n4. In Figures 2 and 3, why not compare the proposed algorithm to the standard AIPW estimator that does not use external controls? This would provide a clearer benchmark to evaluate the actual benefit of borrowing external information.\n\n5. In Table 3, why is there no comparison of mean squared error (MSE) across different estimators? In several cases, it appears that the proposed estimator performs no better — or even worse — than the regular AIPW estimator. \n\n6. In the experiments, there are 300 treatment units and 100 controls, so it is reasonable to expect that borrowing additional external controls may be beneficial. However, how would the method perform if the number of controls already exceeds the number of treatment units — for example, 100 treatment units versus 300 controls? Would you still expect any improvement in the estimation of the treatment effect? It would be useful to demonstrate, either theoretically or numerically, how the method performs in this setting. Understanding such limitations is important for practical applications.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}