Bayesian Sensitivity of Causal Inference Estimators under Evidence-Based Priors
Authors: Nikita Dhawan
Daniel Shen
Leonardo Cotta
Chris J. Maddison
Venue: Accepted by TMLR ()

Abstract: Causal inference, especially in observational studies, relies on untestable assumptions about the true data-generating process. Sensitivity analysis helps us determine how robust our conclusions are when we alter these underlying assumptions. Existing frameworks for sensitivity analysis are concerned with worst-case changes in assumptions. In this work, we argue that using such pessimistic criteria can often become uninformative or lead to conclusions contradicting our prior knowledge about the world. To demonstrate this claim, we generalize the recent s-value framework (Gupta & Rothenhäusler, 2023) to estimate the sensitivity of three different common assumptions in causal inference. Empirically, we find that, indeed, worst-case conclusions about sensitivity can rely on unrealistic changes in the data-generating process. To overcome this, we extend the s-value framework with a new sensitivity analysis criterion: Bayesian Sensitivity Value (BSV), which computes the expected sensitivity of an estimate to assumption violations under priors constructed from real-world evidence. We use Monte Carlo approximations to estimate this quantity and illustrate its applicability in an observational study on the effect of diabetes treatments on weight loss.

Published in Transactions on Machine Learning Research (01/2026)
Bayesian Sensitivity of Causal Inference Estimators under -assed Priors
Nikita Dhawan
miktto@cs.toronto.eel
University of Toronto Vector Institute
Daniel Shen University of Waterloo
d2shen Quwaterroaaa
Leonardo Cotta Vector Instiutte
leonardo. cotta @wwecforinstituteeeee
Chris J. Maddison
cmaddis@@cs.orrotto.co
University of Toronto Vector Institute
Reviewed on OpenReview: https: // openreview. net/ for id= 0zqt85Nuyk
Abstract
Causal inference, especially in observational studies, relies on untestable assumptions about the true data-generating process. Sensitivity analysis helps determine how robust our conclusions are when we alter these underlying assumptions. Existing frameworks for sensitivity analysis are concerned with worst-case changes in assumptions. In this work, we argue that using such pessimistic criteria can often become uninformative or lead to conclusions contradicting our prior knowledge about the world. To demonstrate this claim, we generalize the recent s-value framework (Gupta & Rothenhäusler 2023) to estimate the sensitivity of three different common assumptions in causal inference. Empirically, we find that, indeed, worst-case conclusions about sensitivity can rely on unrealistic changes the dota-generating process. To overcome this, we extend the s-value framework with new sensitivity analysis criterion: Bayesian Sensitivity Value (BSV), which computes the expected scnsitivity of an estimate to assumption violations under priors constructed from real-world evidence. We use Monte Carlo approximations to estimate this quantity and illustrale rate its applicability in an observational study on the effect of diabetes treatments on weight loss.
1 Introduction
Causal inference offers powerful tools for decision saaiionn-aiingg across high-stakes domains, but both its internal and external validity rely on assumptions that cannot be verified from data alone (Barcinboim et al. 2022
Pearl & Bareinboim 2011). Sensitivity analysis (Rosenbaum & Rubin 1983a) has emerged a critical
tool to assess the robustness of causal conclusions to violations of these assumptions. Existing frameworks
typically focus the unconfoundedness assumption (VanderWeele & Ding 2017 Zhao et al. 2019 Cinelli &
Hazlett 2020: Lu & Ding. 2023) - all confounding variables have been measured and controlled for. However, the internal validity of causal inference depends not only on unconfoundednesss but also on other model specification assumptions (Bang & Robins 2005). As for external validity, even randomized trials are exposed selection and outcome biases in their data collection mechanisms. For instance, participants included in a study may differ systematically from the broader population of interest, certain outcomes may be reported only by non-random subset of the study population.

---

Published in Transactions on Machine Learning Research (01/2026)
The key question asked by sensitivity analysis is: What eatent of violation of assumptions tolerale, before ils conclusions are reversed? This is often useful when deciding whether a study provides strong enough evidence to make decision, such as approving a drug for use. Further, practitioners are often interested in comparing subpopulations and prioritizing high-sensitivity ones when considering resource allocation and data collection for urt experimentation (Razavi et al. 2021p. However, since considering every possible combination of assumption violations usually infeasible, common approaches estimate sensitivity in a worst-case sense and search for the smallest violation that reverses the study's 's conclusion (Cinelli & Hazlett 2020 Gupta & Rothenhäusler 2023). When applying these approaches to multi-dimensional settings, practitioners often consider smaller spaces of possible assumptions, e.g. one confounder at time, a simplification that can render the analysis uninformative or even deceiving (Saltelli et al. 2019). Moreover, these practices can be entirely incompatible with real-world scenarios. First, worst-case violations may not be plausible in the real world, resulting in overly pessimistic results. Second, violations are likely to occur along multiple dimensions simultancously, producing combinatorial effects that single-dimensional analyses fail capture. Finally, trivially extending worst-case analyses to mulli-dimensional settings can be particularly problematic as they tend to become uninformative dimensionality increases.
experiment
In this work, we start by formalizing and showcasing the above challenges. To that end, we extend the s-value method of Gupta & Rothenhäusler (2023) to a more general sensitivity framework for causal inference. The s-value measures the min- imum amount of shift in the covariate distribution that changes the sign of an av- erage treatment effect estimate and hence reverses the corresponding causal conclu- sion. We generalize this to include other common assumptions in causal inference, including no unmeasured confounding, well-speeifted conditional models of out- comes given trealment and covariates, and external validity with respect to the dis- tribution over covariates. This extended framework defines an assumption variable, the space of its possible values, a divergence metric to measure violations this space, and constraints that enable practitioners to target particular sensitivity questions of interest.
riee <
Oberned for Wentsnse by
Sanpial -it
Figure 1: Let π be a prior distribution over the covariate distri- bution px. The closest (worst-case) distribution over covariates that reverses the causal decision defined by threshold 5 has low probability under म, and more realistic distributions are further away from the observed and assumed px (Left). Further, samples from this prior that reverse the decision highlight sensitive distri- butions that are different from the worst-case one (Right).
Through simulation studies, we find that standard worst-case sensitivity analyses often provide limited actionable information when trying to distinguish between and prioritize directions in the space of an assumption's 's possible values. These approaches tend to be overly pessimistic because they treat all possible violations as equally likely and rarely reflect. realistic violations. For instance, given our world knowledge, it is very unlikely to have a real-world shift both increasing family income and decreasing education level. Can we avoid attributing high scnsitivity to an experiment based on such violations that contradict well-established knowledge of real populations? To address these limitations, we propose a Bayesian alternative, called the Bayesian Sensitivity Value (BSV), including its Empirical Bayes version (EBSV), which computes the expected sensitivity to assumption violations under prior constructed from real-world evidence (data). Importantly, unlike existing Bayesian Causal Inference frameworks (McCandless et al. 2007 Li et al.] 2023 that assume priors to compute posteriors for the teettmntt effect, BSV defines a posterior over changes in the aata--gnneaaiig process, while using existing established frequentist estimators. Further, EBSV incorporates data-driven priors instead of arbitrary ones. We review related frameworks in detail in section
As an illustrative example, fig. 1 shows toy setting to assess the sensitivity of causal effect estimate with respect to the covariate distribution px, defined as the joint distribution over independent binary confounders Xi and X2. A worst-case analysis yields the closest distribution that reverses a causal decision based a threshold 6, despite having low probability under an empirical prior distribution. In contrast, distributions
2

---

Published in Transactions on Machine Learning Research (01/2026)
sampled from an empirical prior that reverse this decision indicate lower sensitivity in magnitude and lie in different direction in the space of possible assumptions on px- While the worst-case analysis might. lead practitioners to direct resources towards distributions with small P(X₁ = 1) and small P(X2 = 1), incorporating the empirical prior would instead prioritize those with small P(X1 1) and large P(X₂ = 1).
Our work provides strategies to leverage large demographic and medical databases as well as detailed subgroup analyses of previous studies construct empirical priors in a data-driven manner for a real-world diabetes study. The empirical priors allow us to analyze the sensitivity of an estimator to realistic violations of its assumptions. We provide practical algorithms to implement both, worst-case and Bayesian, sensitivity analyses. While worst-case analyses typically reduce to constrained optimization problems, Bayesian sensitivity computations reduce to a constrained sampling problem. Our empirical results demonstrate that sensitivity values in expectation differ substantially from worst-case scenarios, corroborating that conventional sensitivity analysis methods often fail reflect realistic conditions. Morcover, we show how BSV can distinguish between directions in assumption space better than worst-case analyses, hence providing more informative guidance for eritical experiment design decisions, such as defining appropriate study populations.
Contributions. The main contributions of our work are:
We present a sensitivity analysis framework to unify different types of assumptions required by a given estimator, that allows practitioners to ask targeted questions about the sensitivity of a study.
Our simulation studies and real-world application empirically demonstrate that worst-case analyses indeed often fail to reflect plausible scenarios, leading to overly pessimistic and uninformative results.
We propose a new sensitivity criterion, the Bayesian Sensitivity Value (BSV), which is grounded in real-world evidence, is more realistic, and remains useful in high-dimensional spaces of assumptions. Finally, we illustrate BSV's ability to address the above limitations in our experiments, along with strategies to construct empirical priors from real-world evidence and practical algorithms for both criteria.
2 Sensitivity Analysis
To formalize our study of the effect of assumption violations on a causal decision, we will first define required notation and state three typical assumptions under which the popular outcome imputation estimator can identify average treatment effects. We then present our framework for scnsitivity analysis with respect to each individual assumption in our notation, of which existing worst-case analyses are special cases.
a
2.1 Assumptions for Average Treatment Effect Identification
We consider the archetypal causal inference task: to estimate the causal effect of a treatment relative to either another treatment or no treatment (control) in population of interest. Then, we want to make a binary decision with respect to this estimate, e-g.,
binary treatment effect estimation. In this task, we want
treatment has relative positive negative effect. Concretely, consider binary treatment variable T E 1) and corresponding binary potential outcomes Y E 10, under each treatment, as defined in the Neyman- Boyman-Rubin model (Holland, 1986). In decision focused causal inference Jaeschke et al. Hedayat et al. (2055) we translate estimates of the statistic called average treatment effect (ATE),
(1989) );
= E E(Y
(1)
(7 (T rr>0 for user specified threshold 8. See a full list of notation in Appendix A
into binary choices such as II
The fundamental problem in causal inference is that we never observe the potential outcomes (O), (1)).
Instead, we observe the treatment T and an outcome Y - T- Y + (1 - T) (().. Therefore, different studies require assumptions of different forms and strengths to estimate the ATE consistently from observed data. Specifically, observational methods like inverse propensity score weighting (Horvitz & Thompson 1952. Rosenbaum 1987) and outcome regression [Rubin| 1977) convert eq. 11 into identifiable forms via a series of assumptions on its non-identifiable components. In this work, we treat these components as a collection of functions a, and parameterize the ATF function of these unknown functions: T(a).
3

---

Published in Transactions on Machine Learning Research (01/2026)
To illustrate this framework, consider the outcome imputation estimator. We start with the form of the ATE proposed in Lu & Ding 2023, Theorem 1) and then state the typical assumptions that make it identifiable from data. Given discrete covariates X E X, drawn from distribution px and observed jointly with treatment T and outcome Y, we have
| (T+1-12) 1-77 x= x E =((())) (1-111111 (1-T+Teo(e) x x= (1(x))
[
P(x Σ px(e)) ZEN
(2)
This equality holds, when the odds ratio function, E: - R2 is given by E(x) = E where
co(x) E| (0) 0, $ι B =
= x]
E|Y 0,
the conditional outcome function, p:: 1], is given by urxx) (2)(2), (z))
where
po(z) T 0, z], -= E| 1, x].
and the covariate distribution, Px A | gives the probability Px (z). See Lu & Ding (023) or appendix B
for proof of the equivalence between eqs. [ and (2). The functions (e, px) are either non-identifiabl
from observational data or challenging to estimate. For this reason, causal inference makes assumptions the possible values of (e, (x.)) Below, we discuss each function and its typically assumed value.
The common no unmeasured confounding (Rosenbaum & Rubin 1983b Imbens & Rubin 2015 assumption is that true odds ratios are identically 1, i.e., (1)()), = (1, 1). Equivalently, potential outcomes are independent of treatment assignment, given observed covariates, i.e. (Y(0), Y(1)) TIX. Note that this assumption holds true in randomized experiments, but may be violated in observational studies due to unobserved confounders. Under the no unmeasured confounding assumption, eq. 2 reduces to
T (1, = Σ (x)
(3)
xEX
where the conditional outcomes are defined as before and px is the target distribution of covariates X.
Assumptions of well-specifted conditional outcome models (e.g., Rubin 1977 typically state that the conditional outcome distributions μ can be learned from observed data by modeling the mapping from covariates and treatments to expected outcome, í.e., the risk-minimizing models of outcomes μ²" are the true conditional outcome distributions μ. This assumption may be violated due model misspecification reporting biases that modify the outcome distribution. Under unmeasured confounding and correct. specification of conditional outcome models, we have
T(1, px) = (xx( - =](x))
(4)
TEX
where px remains the target distribution over covariates.
External validity (Shadish et al.] 2002: Pearl Bareinboim 2011 assumptions typically state that the sampled population is the same as the target population of interest. In particular, in the context of a population distribution over covariates,
assumption of external validity would state that the sampled population's 's distribution over covariates qx is the same the target distribution px. This assumption be violated due to selection biases in data collection such that the obscrved population is not representative of the true target population. Under all three assumptions above, we have
(11 10 (xx) Exx())pp((> po(())
Σ
- - Ao(x
(5)
which the standard and popular outcome imputation estimator, computed using a study population, (x(0), 169, (0(y))))) (( 1 of n individuals and (usually) fitted regression models á for conditional outcome distributions.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Assumption
Function
Space A
Divergence D
Typical assumption
Unconfoundedness
R+
Euclidean
Conditional Outcome Distributions
[0,
1
KL
Covariate Distribution
Px
A
KL
qx
Table 1: Typical assumptions required for ATE identification.
While the example above corresponds to outcome imputation, it is straightforward to derive analogous parameterizations of assumptions for other estimators. As another example, we include the inverse propensity score weighting eslimator in appendix C Throughout this work, we present sensitivity criteria and algorithms with a focus on the setting with binary treatments and outcomes and discrete covariates. These may be applicd to contimuuus-aalue valuee variables via discretization or continuous extensions that are left to future work.
Note that the three assumptions above are fundamentally distinct that px and μ are observable, though it may be challenging access or represent all populations of interest, while is inherently unobservable.
2.2 Our Sensitivity Analysis Framework
To quantify how violations of different assumptions affect causal decisions, we generalize the s-value of Gupta. & Rothenhäusler (2023) with the following framework. For given choice of assumption functions, pxx), our goal is to quantify the impact of deviations from a on the downstream decision function
[(7(a) > >5) For ease of interpretation, we will consider each of these functions one at a time, each of which can individually belong to very high-dimensional spaces. For any given function, we define:
as a subset of possible triplets (s, (C,,, over which possible values that function range, and a divergence D(allā) to quantify the degree of violation from an assumed value if the correct one had been some A.
For binary outcomes and covariates that can take d possible values (|X*) d), the function of odds ratios be- long to IR?d conditionals μ are defined in [o, 1j2d and the covariate distribution lies in the simplex A Hence, A respectively corresponds to the subsets ((s, (x))ee {(1, p, xxxxxxx))))) ] and (1,,",,xx)xx" px)]xx 1 We take D the Kullback-Leibler (KL) divergence for distributions μ and px and Euclidean distance for the vectors Table summarizes these functions and their assumptions in sensitivity framework.
Assume, without loss of generality, that under default values, 7 T(1, n², ,xx) > 8. Then, the region in A where T()) < 8 represents a reversal of the causal decision.
2.3 Worst-case Sensitivity Criterion
A special case of the framework described in scction 2.2 the s-value, was introduced by Gupta & Rothenhäusler (2023) to measure the minimum amount of shift in the covariate distribution that changes the sign of an ATE estimate. In other words, they considered sensitivity to the px function with decision threshold 5 = 0. Generalizing this s-value, we can characterize the worst-case sensitivity criterion for a study for any function.
Definition (Worst-case Sensitivity). For given function that has possible values in A, its assumed value à and convez divergence measure D, e.g. Bregman divergences, if (()) > 8, the worst case sensitivity of study to this assumplion is defined as
sup(exp(-D(alla)) r(a)= S 8). aEA
(6)
Equation 6 ensures that the sensitivity values are bounded in [0, 1] for non-negative divergences and that higher values indicate higher sensitivity, since smaller deviations would be required to reverse the decision.
The analysis of odds ratio functions & by Lu Ding (2233) can also be reinterpreted under this framework. Lu & Ding (2023) visualized the ATE estimates under different values of Eg and E assuming each to be constant in X, i.e. RR. Hence, A was restricted to be 2-dimensional. In their work, the visual gap
5

---

Published in Transactions on Machine Learning Research (01/2026)
between 11, and the closest (Eo,E) E1) that changed the sign of the ATE estimate was used to characterize sensitivity of the estimate to unobserved confounding. Our framework quantifies this worst-case sensitivity to the unconfoundednesss assumption with a single scalar given by eq. 66 Importantly, it does not assume that E is a constant function of the covariates X, but instead belongs to R. Closely related also the sensitivily analysis of Franks el al. (2020), with respect the unconfoundedness assumplion. This work defines sensitivity parameters (No, m) which can be interpreted as the differences of expected conditional potential outcomes under the two treatments instead of the ratios (c0, used by Lu & Ding (0023), but are similarly treated being constant in covariates X for ease of visualization.
The absolute value of the scalar from eq- 66 is difficult to interpret. It is more practically useful to compare subsets of the (8, px) triplets that differ meaningfully e.g. different A range over changes in selected subsets of the covariates. Then, each subset A is a collection of possible subpopulations varying in these covariates and practitioners may accordingly direct resources towards the A with high sensitivity. We describe constrained optimization algorithms to compute worst-case scnsitivity values for all assumptions in section 4.1
3 Bayesian Sensitivity Value
While sensitivity analyses critical for decision-making under assumptions, they have not enjoyed widespread practical adoption Tarantola et al. 2024) and have not always proved to be practically useful (Saltelli el al. 2019). In this work, we take critical view of the worst-case sensitivity paradigm in particular, and highlight pitfalls that can render it uninformative or unrealistic in practical settings. The following real-world diabetes example is one such instance and motivates the need for alternate sensitivity criteria, beyond the worst-case.
Motivating Example. Worst-case analyses compute the solution to the optimization problem in eq. (6) and can compare this solution empirical realizations of a found in real populations. We did in diabetes application setting for the assumption the joint distribution over binarized covariates, Weight and HbA1c, described in section 6. We also calculated the empirical distribution over the same variables in diabetes patients according to data collected by the National Health and Nutritional Examination NHANES). Both distributions are visualized in fig.
Survey (National Center for Health Statistics, Note that the empirical distribution from the NHANES data puts highest probability on low HbА1c and low Weight values, in agreement with the belseed correlation between glycated hemoglobin reduction and weight loss Guummesson et al. 2017). However, the worst- case analysis suggests sensitivity due to a distri- bution that puts large probability on low HbA1c and high Weight and unlikely to appear in real diabetic populations. This exemplifies how worst-case analyses, once interpreted, may be based on optima that contradict known rela- tionships between variables of interest and fail to reflect realistic scenarios.
2023
7.9%
0.19
0.06
0.22
0.11
0.5
HbAlc, II
0.3
S 7.8%
0.11
0.64
0.47
0.20
S. 1bs lbs
>> 222
220 lbs
>> lbs
Weight, W
Figure 2: Optimum found by a worst-case analysis (Left) is significantly different from the real-world empirical dis- tribution over the same variables recorded in the NHANES dataset (Right) and contradicts prior knowledge.
lackle limitations of worst-case analyses, demonstrated above as well as further experiments in section 6 we argue favor of accounting for how likely any given assumption violation in settings of interest. To this end, we treat the each of the functions (e, p. as random variables and propose a novel sensitivity criterion, called the Bayesian Sensitivity Value (BSV), defined as follows.
Definition 2 (Bayesian Sensitivity Value). Given an assumplion à E A. divergence D, and random assumption A taking values in A with distribution TA, if T()) > 8, the Bayesian Sensitivity Value is
EANRa /exp (- D(Al(a)) r(A) <
The BSV given by eq. replaces the supremum of eq. G with an expectation computed with respect to a specified prior distribution the assumption variable. This criterion has several advantages for sensitivity

---

Published in Transactions on Machine Learning Research (01/2026)
analyses over its worst-case counterpart: (i) it is more pessimistic than the worst-case sensitivity value, by definition, and far less pessimistic in practice, (ii) the BSV is direct result of the chosen prior œл and different values can be computed and interpreted under different choices of this prior; (iii) when available, this critcrion allows practitioners to leverage large demographic, medical, or other domain-specific databases to construct evidence-based priors and compute an Empirical Bayesian Sensitivity Value (EBSV).
The EBSV grounds sensitivity analyses in real-world evidence and prior knowledge. For example, is often possible to derive empirical distributions over common covariates from large demographic databases and surveys to construct prior distribution over px. We can also use ATE estimates and subgroup analyses for different covariates and treatments from previous studies to construct priors over μ. The resulting EBSV quantifies the sensitivity of a new study under the prior of previous findings. Empirically, we find that it also distinguishes between different subpopulations better than worst-case sensitivity values can, especially when assumptions can take values in high-dimensional spaces, as illustrated in section G We describe constrained sampling algorithms to compute Bayesian sensitivity values for all assumptions in section 4.2
4 Practical Algorithms
Computation of the worst-case sensitivity value in eq. 66 and the BSV in eq. (7) requires solving constrained optimization and constrained sampling problems, respectively. Gupta & Rothenhäusler (2023) exploited the linearity of in px to derive closed-form solutions for sensitivity to px. We can generalize their solutions by noticing that the constrained problem in eq. 16 is convex in each of Px) (or one-to-one transformation of it), even if it isn't 't linear. We accordingly adapted existing optimization strategies via Lagrange multipliers to compute worst-case sensitivity values. The optimization algorithm presented here can be extended and applicd to any causal estimand that can be written as a convex function of its assumption a, divergence D that is convex in a, and convex assumption space A. We used existing constrained sampling schemes for the BSV, though more efficient samplers may be derived by exploiting the convexity of eq- 7
4.1 Constrained Optimization
We formulate eq. 6 as an equivalent minimization problem, with the following Langrangian.
max min D(alla) + A(T(a) - 8),
(8)
aEA
where A 2 0 is a Lagrange multiplier enforeing the threshold constraint on the ATE. We include any assumption-speciff constraints on a, e.g. so(x) 2 0, 2 0 for all covariate sets æ, with their own Lagrange multipliers. Due to its convexity, the above problem solved exactly by an ascent-descent algorithm that alternates between updating the primal variable a and performing gradient ascent on the dual variable X
[Boyd & Vandenberghe] 2004 Boyd et al. 2011). For assumptions that represent. probability distributions (u
and px), we used Entropic Mirror Descent (Beck & Teboulle 2003) to solve the optimization problem under
simplex constraints efficiently. For odds ratios E R we folded the non-negativity constraints into eq. (8)
with its own Lagrange multipliers and used Gradient Descent to update a. We state the corresponding convex optimization problem for each assumption function and the standard dual ascent algorithm in appendix
4.2 Constrained Sampling
We compute a rejection-based Monte Carlo estimate (Robert et al. 1999] of the conditional expectation in eq. 7 Specifically, we drew samples from the given prior distribution, A ~ TA i.i.d., accepted them if they satisfy the constraint T(A()) S 6, and repeated this procedure until we obtain M accepted samples, for some M. In all our experiments, we M 5000. Then, the BSV estimate given by
1 N
M Σ exp( exp(-D(A()a)) (T((10) <o) where N = min { Σ (7((()) 2 2 20)
{
(9)
Hence, samples of the assumption parameters are accepted with the prior probability that they reverse the causal decision. Note that if this probability is very small, rejection sampling can be very inefficient. In this
7

---

Published in Transactions on Machine Learning Research (01/2026)
case, more sophisticated sampling techniques like Markov Chain Monte Carlo methods (iiks et al. 1995
may be adapted, though we found rejection sampling sulficient for our experiments.
Samples from the prior TA also allow us to estimate the probability of reversal of causal decisions due to assumption violations drawn from AA; f.c. (A(((A) S 6) via the empirical probability R Ei= I (T(A S $6). for large K = 1000000. It is useful to jointly report the probability of decision reversal with the BSV and since the former accounts for how likely decision reversals are under the given prior and the latter provides information about the average "size" (via divergence D) of the assumption violation that reverses the decision. In our experiments, we additional report the acceptance rate N.
5 Related Work
Sensitivity analyses have long been recognized as critical for causal inference, particularly in observational studies (Rosenbaum 1987.. Traditional approaches, that typically consider worst-case violations of the
unconfoundedness assumption, include Tan (2006); Vander & Ding] (2017); Cinelli & Hazlett (2020);
Veitch & Zaveri 2020 Lu & Ding (2023 also proposed flexible analysis for the unconfoundedness assumption that may be applied to different estimators. There also Bayesian approaches to causal
inference (Li et al. 2023) and sensitivity analysis (McCandless et al. 2007| in the context of this assumption.
These works typically employ expert-elicited or uninformative priors. Gupta & Rothenhäusler (2023) tackled
sensitivity to covariate distribution shifts and proposed the s-value framework, which we generalized to other assumptions, including those on conditional outcome distributions and unconfoundedness.
Causal sensitivity analyses have been adapted to several applications in fairness (Fawkes et al., 2024 optimization of operating characteristics of clinical trial design (Han et al. 2024) and evaluation of con- strained selection biases (Cortes- Gomszz et al.) 2023). Recent critiques have highlighted limitations and misinterpretations associated with scnsitivity analyses in practice, emphasizing a need for more realistic and informative methodologies (Toannidis et al.] 2019 Saltelli et al. 2019 de Souza el al. 2016)_ Our work takes a step towards addressing these limitations by grounding sensitivity analyses in real-world evidence.
Several large databases, collected and curated nationally or globally, provide empirical data on demographic and medical variables (Clore et al.] 2014 CDC 2023). Additionally, post-hoc analyses of subgroup effects are
commonly found in medical literature (Tchang et al. 2025 Kadowaki et al. 2024 Ma et al. 2024 Ryan et al.
2024p. Our proposed Bayesian Sensitivity Value provides way to leverage these sources of information to understand sensitivity with respect to px and μx, respectively, while representing real-world populations.
6 Empirical Analysis
Our experiments use sensitivity analyses to compare and rank subsets of the (. triplets, denoted A, that differ meaningfully, so that high-sensitivity subgroups may be discovered, for instance, for resource prioritization. We computed worst-case sensitivity value, the BSV under a uniform or uninformative prior, and the EBSV under evidence-based priors, when they are available, investigate the following questions: (i) How do the different sensitivity analysis criteria scale with dimensionality of the space of possible assumptions A, specifically in their ability to distinguish between different A? (fi) Do BSV analyses, including EBBSV, differ meaningfully from worst-case analyses, specifically to uncover different sensitivity rankings across A? (iii) How does BSV perform in real-world applications, compared to worst-case analyses?
6.1 Experimental Settings
Simulation. We constructed a simulation study to generate data with binary covariates and heterogencous treatment effects, where the true treatment effect varies across subgroups defined by covariate values. Observational data was generated to reflect realistic biases commonly encountered in practice: confounding through treatment assignment and seloction bias based on outcomes, treatment assignment, and covariates. We used settings with varying numbers of covariates (4, 6,8) to explore how the dimensionality of the problem affects sensitivity analyses. See full details of the data generation in appendix D.I For BSV computations,
8

---

Published in Transactions on Machine Learning Research (01/2026)
considered an idealistic setting and constructed empirical prior distributions for px and j using samples from the true data-generation mechanism without confounding or selection biases.
Diabetes Application. As real-world application, we adapted the Semaglutide vs. Tirzepatide dataset and largelaagguage-mode (LLM) based causal estimator from Dhawan et al. (2024). This study leveraged patient- reported data from relevant subreddits, employed large language models to extract observational distributions, accounted for confounding due to Age, Sex, BMI (Body Mass Index), HbAic (Glycated Hemoollbiin) and Weight, and estimated the relative effect of diabetes treatments selection biases in both the data collection and modeling via LLMs, informative sensitivity analyses are crucial for reliable inference in this setting. See appendix D.2 for complete details.
weight loss outcome. Given the potential
This setting is also useful to demonstrate the construction of empirical priors and practical instantiation of an EBSV analysis when true priors are unavailable. Here, we leveraged previous databases and studies for information relevant the assumptions of interest, px and μs. Specifically, we extracted distributions over Age, Sex, HbA1c, and Weight across different races using the Diabetes dataset from the UCI repository CClore et al. 2014) and used them to fit Dirichlet distribution that serves prior over the covariate distribution px. We obtained distributions over the conditional outcome distributions for Semaglutide (T= 0) from subgroup analyses condueted in previous studies (Kadowaki et al. 2024) and similarly fit a Dirichlet distribution to use as the empirical prior. We note that constructing empirical priors on odds ratios & from data remains a challenge since they are inherently unobserved quantities. Hence, we use user-defined priors which are, in this case, truncated Gaussians, centered at 1, with scales E {1.0, 1.5].
Comparing different A. Following the compari- son of sensitivily values across covariates in Gupta & Rothenhhusler (2023), we constructed compar- isons between different A for all our assumptions, where possible elements of each A range over par- ticular covariates and ments. Concretely, this corresponds calculating sensitivity values when elements of A differ in their values of either the odds ratios Ex, for covariates Xj, outcome distri- butions P4,x; = [T = t, Xjj conditioned on a treatment and covariates Xj, or distributions px;
1.0
Sensilivity aCTOSS A
0.8
0.6
0.4
0.2
Worst-case Sensitivity
0.0
Empirical BSV
Unitorm BSV
0.2
2
24
2 26
27
over covariates Xy. Here, X, is any subset of all the covariates X, and including more covariates re- sults in higher dimensional A. For example, to find the pair of binary covariates whose joint distribu- tion T4, u4 px) is most sensitive to, we compared sensitivity values given by eq. (6) or eq. 7 with
Dimensionality of A
Figure 3: For high-dimensional A, worst-casc sensitiv- ity values for covariate distributions are all close to (mean se 1, variance x 0), while BSVs, both Uniform and Empirical, distinguish between different A better.
A A= (1,,"" pn"""" 4x _x,, ,xxx))px, EA, for different pairs of covariates Xj, where X-j includes all other covariates. This results in a sensitivity ranking according to either the worst-case or the Bayesian criterion.
6.2 Simulations
To study the behavior of sensitivity criteria at scale and answer item ii) we simulated a setting with eight: binary covariates and measured sensitivity the covariate distribution where A has increasing dimensionality: For l E (1, we considered every possible subset of covariates of size k. set A to be the space of possible joint distributions over this covariate subset, which is A and solved for the worst-case and Bayesian sensitivity values in each case. Figure 3 shows the mean and standard deviation of these sensitivity values against the dimensionality of A. As dimensionality increases, worst-case sensitivity values quickly tend toward a mean of 1 and standard deviation of 0, hence attributing highest possible sensitivity to all high nigh-dimensional A, regardless of the plausibility of the worst-case violation in practice. In contrast, the BSV does not appear to suffer from this curse of dimensionality in our settings, maintaining variability across different even in high dimensions and yielding sensitivity ranking that may be used to inform resource allocation in practice.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Worst-oom Seasitivity
0.9
Worst soee Sensitivity
0.0
0.8
0.8
Correlation: -0.35
Correlation: -0.14
A
A
06.6
Ag
0.6
= 0g
A
0.5
0.5
0.3
0.4
0.5
0.6
0.7
0.9
0.3
0.4
0.5
0.6
0.5
0.
1.1
Emparical BSV
Empirieal BSV
(a) Sensitivity to px, with four covariates.
(b) Sensitivity to px, with six covariates.
1.0
1.0
Worst-ose Seasitivity
0.9
Weest cuse Sousitzciyy
0.8-
0.7-
0.7
Correlation: -0.85
Convelation: 0.13
0.8-
A R
T= 0, A [0,1]"
= R!
T 1, 1, [0,1]
0.3
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.
0.6
0.7
0.8
0.9
1'a
B8V
1.5)
Empirical
(c) Sensitivity EX--
(d) Sensitivity Pe.X,"
Figure 4: For all assumptions and different choices of subsets A, Bayesian sensitivity uncorrelated negatively correlated with worst-case sensitivity, revealing different trends in scnsitivity rankings.
Next, we investigated the actual sensitivity rankings produced by BSV versus a worst-case analysis answer item [i) Figure shows the correlation between these rankings for assumptions on EX,, Mr,X,, and px, (with four or six total covariates) for different X, and t. Again, worst-case analyses are relatively pessimistic, assigning very similar and high scnsitivity values, especially when A is high-dimensional. Morcover, the worst-case sensitivity rankings are uncorrelated or even negatively correlated with BSV rankings. Hence, BSV produces different trends in sensitivity across Sensitivity values for different X, and under the worst-case, empirical BSV, and BSV with uniform priors are presented in figs. to of appendix F
For some intuition this behavior, consider the space of assumption violations. Adding more dimensions to this space allows violations in more variables. This is likely to increase the worst-case sensitivity, the feasible region of the constrained optimization problem bocomes larger. In contrast, adding dimensions to the space of assumption violations in the constrained sampling problem that yields the BSV also adds dimensions to the prior distríbution æ. Empirical joint priors over more variables can effectively constrain the space of likely assumption violations, especially when these variables are correlated with each other. Sampling from this prior results in BSVs that account for the choice of variables and provides more meaningful trends.
6.3 Diabetes application
We conducted similar sensitivity analyses under the worst-case criterion, the BSV with uniform priors, and the empirical BSV, on the diabetes study, with their results visualized in fig. 5 Here, we consider individual covariates as well as Covariate pair indices, which lexicographically index all possible pairs of covariates in (Age, Sex, BMI, HbA1c, Weight). In the analyses with respect to px and μ. shown in figs. 5b and 5c respectively, worst-case values and BSV always agree in zero-sensitivity settings expected. However, BSV highlights informative trends in others, which can support targeted future experimentation and data collection. Notably, they suggest that the estimated ATE is less sensitive than what a worst-case analysis would indicate in the Weight 220 1bs subpopulation. Similar to the simulation study, fig, 55a shows BSV for odds ratios under truncated Gaussian priors. While differences are small between choices of observed covariates, sensitivity is lower whenever Weight among them. Further, table 2 lists the sensitivity values given by worst-case, uniform Bayesian, and empirical Bayesian criteria, along with acceptance rates and probabilities of decision
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: In real-world diabetes experiment, the probabilities of decision reversal P (* S 8) and corresponding BSVs under uniform and empirical priors indicate how likely decision reversal is and the average size of assumption violation required for reversal, respectively. Here, AR = acceptance rate.
Worst-case
Uniform BSV (AR)
Pu(((A) s 6)
(E)BSV (AR) (T(((A) s d)
Covariate distributions
Px
Age
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
Sex
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
HbAle
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
Weight
0.52
0.30 (0.57)
0.58
0.14 (1.00)
0.99
Age, Sex
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
Age, HbAle
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
Age. Weight
0.57
0.31 (0.57)
0.57
0.10 (1.00)
1.00
Sex, HbAlc
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
Sex, Weight
0.64
0.27 (0.50)
0.51
0.08 (1.00)
1.00
HbAlc, Weight
0.68
0.29 (0.48)
0.49
0.09 (1.00)
1.00
Conditional outcome distributions
Ages 45,
0.97
0.97 (0.06)
0.06
0.94 (0.38)
0.37
Age> 45, T = 0
0.97
0.97 (0.03)
0.03
0.95 (0.66)
0.67
AgeS 45,
0.88
0.48 (0.90)
0.89
0.42 (0.83)
0.83
Age> 45, T
0.95
0.33 (0.95)
0.95
0.30 (0.88)
0.88
Bex=M, T
0.96
0.97 (0.06)
0.06
0.95 (0.37)
0.37
Sex=F, T' 0)
0.96
0.97 (0.03)
0.03
0.95 (0.41)
0.41
Sex=M, T
0.84
0.46 (0.91)
0.91
0.40 (0.84)
0.84
Sex=F, T==
0.85
0.32 (0.95)
0.95
0.30 (0.88)
0.88
BMI 28.5, T =
0.98
0.97 (0.08)
0.08
0.94 (0.51)
0.50
BMI> 28.5, =
0.99
0.98 (0.04)
0.04
0.96 (0.29)
0.29
BMI 28.5, T
0.95
0.48 (0.92)
0.92
0.51 (0.75)
0.76
BMI> 28.5, T
0.97
0.40 (0.96)
0.96
0.32 (0.90)
0.89
HbAles 7.5, T
0.96
0.97 (0.06)
0.06
0.94 (0.52)
0.52
HhAle> 7.5, ==
0.96
0.97 (0.03)
0.03
0.96 (0.12)
0.12
HbAles 7,5, T=
0.85
0.46 (0.91)
0.91
0.41 (0.84)
0.84
HbAle> 7.5, ==
0.84
0.32 (0.95)
0.94
0.29 (0.88)
0.88
Weight 220, T= 0
0.00
0.00 (0.00)
0.00
0.00 (0.00)
0.00
Weight> 220, T = 0
1.00
0.56 (1.00)
1.00
0.75 (1.00)
1.00
Weight 220, T = 1
0.30
0.10 (0.77)
0.76
0.08 (0.70)
0.71
Weight> 220, T= =
1.00
0.76 (0.52)
0.51
0.82 (0.51)
0.51
Odds ratio functions
Age
0.98
0.61 (0.37)
0.37
Sex
0.98
0.61 (0.38)
0.38
BMI
0.99
0.61 (0.38)
0.39
HbAlc
0.98
0.61 (0.38)
0.38
Weight
0.98
0.59 (0.42)
0.42
Age, Sex
0.99
0.57 (0.31)
0.32
Age, BMT
0.99
0.58 (0.30)
0.31
Age, HbAle
0.99
0.57 (0.31)
0.32
Age, Weight
0.99
0.56 (0.35)
0.35
Sex, BMI
0.99
0.57 (0.31)
0.31
Sex, HbAlc
0.99
0.56 (0.36)
0.36
Sex, Weight
0.99
0.56 (0.36)
0.36
BMI, HbAle
0.99
0.57 (0.32)
0.32
BMI, Weight
0.99
0.56 (0.36)
0.36
HbAle, Weight
0.99
0.56 (0.36)
0.36
11

---

Published in Transactions on Machine Learning Research (01/2026)
101
Ween
60
11.5
Seacibity
Seasiniitt
CS
0.6-
0.6-
Wanser
62
11.2
%6
R401 Observed conuriates
Ade
Sex
HeAlc
Weish
Covariate indices
Individuul conariates
Covariate pair indices
(a) Sensitivity to EX,'
(b) Sensitivity to px,,
Worst case Sensitvity
0.9-
Uniform BSV
0.5
Empirical BSV
0.3
0.5
0.5
0.5
0.4
0.0
0.0
0.3
SereoM
Sex-
BMIl 26.5
BMFS 28.3
HBA les 7.5
Weights 220
7.5
Waggrt 200
Sereo
BMIE 26.5
BM 28.9
HBA leg 7.0
HBA He 7.9
Wagins 200
Weight 200
Socerl
Sox-
BMDs 28.3
26.5
UbA tcg 7.9
Hottcs 7.5
Waigtes: 200
Raglit 220
Agost
Agos
HBA
Agest
Ages
Ste
Apoct
Appo
BMIts
Covariate values
Covariate values
Covariate values
(c) Sensitivity to Xy,
Figure 5: BSV for odds ratios in fig. 5a covariate distributions in fig. 5b and conditional outcome distributions fig. 5c in a real-world diabetes experiment can be more informative than worst-case analyses. In particular, BSV suggests relatively low sensitivity in the Weight > 220 1bs subpopulation, as compared to the worst- case analysis which assigns it the highest possible sensitivity value.
reversal under uniform and empirical priors. Notably, small BSV and large probability of decision reversal indicale that decision reversals are likely under the given prior but the corresponding assumption violations are far from the observed or assumed value of A, as measured by divergence D.
7 Limitations
While this work has taken a critical view of worst-case sensitivity analyses, the proposed Bayesian Sensitivity Value not without limitations in its real-world application. (i) Our experiments suggest that the empirical BSV under data-driven priors can be practically useful. However, constructing such priors odds ratios from data remains fundamentally challenging since they are inherently unobservbble quantities. (ii) We considered and compared different subscts A of possible assumptions that were of the same form and admitted the same divergence metric, e.g. KL, divergence for distributions lying in simplex. However, combining different spaces possible assumptions, along with their corresponding divergence metrics may allow practitioners ask more interesting questions about sensitivity and capture intcractions between different assumptions. (iii) The practical implementation of BSV computations is a simple rejection- based sampling technique, which can be very inefficient for small prior probability of reversing causal decision. More involved and efficient techniques would help improve the adoption of this sensitivity criterion in practice. (iv) Finally, while the BSV captures the variation and plausibility of assumptions under given prior, this also means that it is highly prior-dependent and may be susceptible to prior misspecification.
8 Conclusions
In this work, we generalized the s-value framework to three common assumptions causal inference and proposed new sensitivity criterion, the Bayesian Sensitivity Value, that can leverage real-world evidence. Our empirical studies demonstrated that worst-case sensitivity analyses can be overly pessimistic and uninformative, especially for high-dimensional spaces of possible assumptions, and rarely reflect realistic
12

---

Published in Transactions on Machine Learning Research (01/2026)
shifts in assumptions. However, the BSV distinguished between different A even in high-dimensional spaces and revealed trends that differ from worst-case analyses. There are several exciting directions for future research: (i) constructing informative priors for different assumptions, including odds ratios, using varied sources of real-world evidence, (ii) adapting more sophisticated sampling tochniques than rejoction sampling to impprve efficiency of BSV computations, and (iii) deriving mathematicalyy rigorous strategies operationalize sensitivity analyses for future experiment design. Given the critical importance of sensitivity analyses for causal decision-making in high-stakes domains like medicine, it is essential to improve their practically utility. While traditional analyses have typically focused on worst-case violations of assumptions, the BSV is step towards more informative criterion for real-world applications and improved resource prioritization.
Broader Impact Statement
Any sensitivity analysis framework has important societal implications for causal inference in high-stakes domains like healthcare. While the BSV can improve decision-making by providing more realistic sensitivity analyses and a better understanding of treatment effects across subpopulations, it also carries risks: the quality of analyses depends critically on the representativeness of real-world evidence used construct priors, and the method could be misused justify decisions that appear more robust than they actually are. To mitigate these risks, recommend transparent documentation of prior construction and use of multiple sensitivily criteria, including our Bayesian criterion as well as traditional worst-case analyses.
Acknowledgments
We would like to thank Ayoub El Hanchi for helpful discussions and feedback on the paper. Resources used in preparing this research were provided in part by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We acknowledge the support the Natural Sciences and Engineering Research Council of Canada (NSERC), RGPIN-2021-08445.
References
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 61(4)9962-973, 2005.
Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and causal inference: the works of judea pearl, pp. 507-556. 2022.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pp. $300839, 2020.
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167 175, 2003.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Pelcato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122) 122, 2011.
Stephen P Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
CDC. Diabetes health indicators dataset, 2023. URL http://archive. ics. uci http:///ree.rc..eeeeeeeeaaaattla Giabetesthealtt-iidaaae
Carlos Cinelli and Chad Hazlett. Making sense of sensitivity: Extending omitted variable bias. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):39 67, 2020.
John Clore, Krzysztof Cios, Jon DeShazo, and Beata Strack. Diabetes 130-us hospitals for years 1999-2008.
UCT Machine Learning Repository, 10:C5230J, 2014.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Santiago Cortes- Gomez, Mateo Dulce, Carlos Patino, and Bryan Wilder. Statistical inference under constrained selection bias. arXiv preprinl or arXim:2306.0340 03300, 2023.
Russell J de Souza, Rebecca B Eisen, Stefan Perera, Bianca Bantoto, Monica Bawor, Brittany B Dennis, Zainab Samaan, and Lehana Thabane. Best (but oft-forgotten) practices: sensitivity analyses in randomizod controlled trials. The American journal of clinical nutrition, 103(1):5-17, 2016.
Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul Krishnan, and Chris J Maddison. End-to-end causal effect estimation from unstructured natural language data. In The Thirty eighth Annual Conference on Neural Information. Processing Systems, 2024.
Jake Fawkes, Nic Fishman, Mel Andrews, and Zachary Lipton. The fragility of fairness: Causal sensitivity analysis for fair machine learning. Advances in Neurol Information Processing Systems, 37:137105 137134, 2024.
AlexanderM Franks, Alexander D'Amour, and Avi Feller. Flexible sensitivity analysis for observational studies without observable implications. Journal of the American Statistical Association, 2020.
Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. Markou chain Monte Carlo in practice. CRC pross, 1995.
Anders Gummesson, Elisabeth Nyman, Mikael Knutsson, and Martin Karpefors. Effect of weight reduction on glycaled haemoglobin in weight loss trials in patients with type 2 diabetes. Diabeles, Obesily and Metabolism, 19(9):1295-133) 1305, 2017.
Suyash Gupta and Dominik Rothenhäusler. The s-value: evaluating stability with respect distributional shifts. Advances in Neural Information Processing Systems, 36:72058-72070, 2023.
Larry Han, Andrea Arfè, and Lorenzo Trippa. Sensitivity analyses of clinical trial designs: selecting scenarios and summarizing operating characteristics. The American Statistician, 78(1):76-87, 2024.
AS Hodayat, Junhui Wang, and Tu Xu. Minimum clinically important difference in medical studies. Biometrics, (1(3):33 41, 2015.
Paul W Holland. Statistics and causal inference. Journal of the American statistical Association, 81(396): 945-960, 1986.
Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663 685, 1952.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015.
John PA Toannidis, Yuan Jin Tan, and Manuel Blum. Limitations and misinterpretatiins of e-values for sensitivity analyses of observational studies. Annals of internal medicine, 170(2):108 111-100811111---1111111111111111 2019.
Roman Jaeschke, Joel Singer, and Gordon H Guyatt. Measurement of health status: ascertaining the minimal clinically important difference. Controlled clinical trials, 10(4):407-415, 1989.
Takashi Kadowaki, Sang Yeoup Lee, Wataru Ogawa, Tomoyuki Nishida, Maria Overvad, Kazuyuki Tobe, Toshimasa Yamauchi, Soo Lim, et al. Clinical characteristics affecting weight loss in an east asian population receiving semaglutide: A step 6 subgroup analysis. Obesity Research & Clinical Practice, 18(6):457 464, 2024.
Fan Li, Peng Ding, and Fabrizia Mealli, Bayesian causal inference: a critical review. Philosophical Transactions of the Royal Society A, (2227)) 2023.
Sizhu Lu and Peng Ding. Flexible sensitivity analysis for causal inference in observational studies subject unmeasured confounding. arXiv preprint XXix:2305.17643. 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Jianhua Ma, Ming Liu, Rui Wang, Liying Du, and Linong Ji. Efficacy and safety of tirzepatide in people with type 2 diabetes by baseline body mass index: An exploratory subgroup analysis of surpass-ap-combo. Diabetes, Obesity and Metabolism, 26(4):1454 1463, 2024.
Lawrence McCandless, Paul Gustafson, and Adrian Levy. Bayesian sensitivity analysis for unmeasured confounding in observational studies. Statistics in medicine, 26(11):2331 2347, 2007.
National Center for Health Statistics. National health and nutrition examination survey data, 2023. URL
https://www. cdc. hoadd/gg///cchshhaaee//////
Judea Pearl and Elias Bareinboim. External validity and transportability: a formal approach. In JSM Proceedings, pp. 157-171, 2011.
Saman Razavi, Anthony Jakeman, Andrea Saltelli, Clémentine Prieur, Bertrand looss, Emanuele Borgonovo, Elmar Plischke, Samuele Lo Piano, Takuya Iwanaga, William Becker, et al. The future of sensitivity analysis: an essential discipline for systems modeling and policy support. Environseental Modelling & Softwore, 137:104954, 2021.
Christian P Robert, George Casella, and George Casella. Monte Carlo slatistical methods, volume 2. Springer, 1999.
Paul R Rosenbaum. Model-basod direct adjustment. Journal of the American statistical Association, 82(398): 387 394, 1987.
Paul R Rosenbaum and Donald B Rubin. Assessing sensitivity to an unobscrved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society: Series B (Methodological), 45(2):212-21) 218, 1983a.
Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):11-55, 1983b.
Donald B Rubin. Assignment to treatment group on the basis of a covariate. Journal of educational Statistics, 2(1):1-26, 1977.
Donna H Ryan, Ildiko Lingvay, John Deanfield, Steven E Kahn, Eric Barros, Bartolome Burguera, Helen M Colhoun, Cintia Cercato, Dror Dicker, Deborah B Horn, et al. Long- termttermmn weight loss effects of semaglutide in obesity without diabetes in the select trial. Nature medicine, 30(7):2049-2057, 2024.
Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, and Qiongli Wu. Why so many published scnsitivity analysos are false: systematic review of scnsitivity analysis practices. Environmental modelling & soflware, 114:29 39, 2019.
William R Shadish, Thomas D Cook, and Donald T Campbell. Experimental and quasi-experimental designs for generalized causal inference. Houghton, Mifflin and Company, 2002.
Zhiqiang Tan, distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 101 (476))):1619-1633 2006.
Stefano Tarantola, Federico Ferretti, Samuele Lo Piano, Mariia Kozlova, Alessio Lachi, Rossana Rosati, Arnald Puy, Pamphile Roy, Giulia Vannucci, Marta Kuc- -zarnczan,,,a, et al. An annotated timcline of sensitivily analysis. Environmental Modelling & Software, 174:105977, 2024.
Beverly G Tchang, Andreea Ciudin Mihai, Adam Stefanski, Luis-Emilio García- Donna Mojdami, Irina Jouravskaya, Sirel Gurbuz, Rebecca Taylor, Chrisanthi A Karanikas, and Julia P Dunn. Body weight reduction in women treated with tirzepatide by reproductive stage: post hoc analysis from the surmount program. Obesily, 33(5):851 860, 2025.
Tyler J VanderWeele and Peng Ding. Sensitivity analysis in observational research: introducing the e-value. Annals of internal medicine, 167(4))268 274, 2017.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Victor Veitch and Anisha Zaveri. Sense and sensitivity analysis: unobserved confounding. Advances in neural information processing systems, 33:10999- 11009, 2020.
Simple post-hoc analysis of bias due to
Qingyuan Zhao, Dylan Small, and Bhaswar Bhattacharya. Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap. Journal of the Royal Stalistical Sociely Series B: Statistical Methodology, 844):7335-71, 2019.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A
Notation
A
Assumption random variable whose value must be assumed for causal inference. Prior distribution over assumption variable A.
X
Random variable corresponding to features of an individual in causal inference dataset.
T
Random variable corresponding to treatment or intervention assigned to an individual in causal inference dataset.
Y
Random variable corresponding to outcome observed for an individual a causal inference dataset.
Possible instance of X from its support X'
Possible instance of T from its support T = {0, (binary treatments).
Possible instance of Y from its support y = {0, 1} (binary outcomes).
a
Possible instance or sampled value of A from its support A.
(t))
Random variable corresponding potential outcome observed for an individual after receiving treatment. t.
e(X)
Unconfoundednesss assumption parameters, s
p(X)
Conditional outcome distributions given covariates and treatment, (Ho(X), un (X)).
Px
Distribution over covariates X.
z00)
Sampled value of X for individual i.
(())
Sampled value of T for individual é.
(0)
Sampled value of y for individual i.
Average treatment effect (ATE) given by (E YO)), where the expectation is over some defined population of individuals.
(a))
ATE computed or estimated under the assumption a.
8
Decision threshold the (estimated) ATE to choose between treatments.
Total number of individuals.
17

---

Published in Transactions on Machine Learning Research (01/2026)
B ATE Identification Proof
We reproduce a proof of the result in Lu & Ding (2023 for eq.
1 in our notation, for completeness.
= (1 - Y()
E[((1) T ( -1] T - O 0)
-0 -|
0), P
= =E(Y T -
- (0
0 [ 0),
where the last equality follows from the definition of Y.
For PX( = T X] and (X) =
= =
E E|y T = 1| P(T= 1) = F Y 7 =
1, 1, T- 1] P(T =
= = [= =
B
EE 0, T P(T
= 10 (x) P(T 0).
(0) X and (X)
11
For (x()
E[Y = 0, X
( X
E Y [T) 0 = E E (L) (T= 0, 0 = 0)
(X)
= E
T P(T = 0).
(x
E (0) TT 1] P(T = E
F B (O) 1T= | = 1] P(T 1)
=X(uo(X)eo(N)) (-= =-1|
Plugging the above back into the equation for T, wC have,
T B[y T 1| 1)) T =
P(T P(T 0)
E[Y T -)] T 0) P(T =)
= ( (X)
T 0 P(T = 0)
(X)
-E( TT P(T |
- E((o((X) | T= P(T = 0)
= -Blu(x)
[ ( - no(X) (Tea(X)+1 +1 -7) 7)
(x)
which is equivalent to eq.
22 by the law of iterated expectations.
18

---

Published in Transactions on Machine Learning Research (01/2026)
C
Assumptions for Inverse Propensity Score Weighting
Consider the inverse propensity score weighting estimator. Equivalent to the form of the ATE proposed in
Lu & Ding 2023 Theorem 2), we have
(1/)) ty c(x))
(1--121)
f)y
T(e,e,xTTY)
Σ
(XTY
wi wi()
wo(w) -
( (l-e(z))
(10)
(b,
where wi(x) = e(z) + 1-e(z) and wo(x) e(z)=0(2) +1- E(x). (z)
This equality holds, when the odds ratio function, E: - R is given by 5(x) (z)( (1 (x))
where
co(x) L0 X = 51 B 1, = =x]
= -
(1) 0, 0,
the propensity scores, e: e + [0, 1)3, are given by
e(x) = = z].
and the joint distribution, PXTY 4 gives the probability PXty f, (z).
The functions (e, μr PxTTY) are either non-identifiable from observational data or challenging to estimate. So we must make the following assumptions on the possible values of (e,
Once again, the no unmeasured confounding assumption that true odds ratios are identically 1, i.e., (z) = (1, 1), discussed in section 2
Under this assumption, eq. (10) reduces to
(xTY(E,43) 4,) ty (1 - f)y
=(1,e,xxTV) = E
(11)
c(x) 1 e
ZEX
(0,1)
where the propensity scores and jont distribution pxry are defined before.
Assumptions of well-specifted propensity score models (e.g., Horvitz & Thompson 1952 typically state that the propensity scores e can be learned from observed data by modeling the mapping from covariates to treatments, f.e., the risk-minimizing models of outcomes e* are the true propensity scores This assumption may be violated due to model misspecification or reporting biases that modify the treatment assignment distribution. Under no unmeasured confounding and correct specification of propensity score models, we have
7(1,6" .XTY) = Σ (x,Y(2,t,
ty
(1 (1- t)y
(12)
(x(x)) 1- (x)
ZEX
(0.))
External validity of pxty assumptions typically state that the sampled population is the same as the target population of interest. Here, assumption of external validity would state that the sampled population's 's distribution over covariates, treatments, outcomes 4XTY is the same as the target distribution pxTY. This assumption can be violated due to selection biases in data collection such that the observed population is not representative of the true target population. Under all three assumptions above, we have
T(1, =
Σ
EXTY
ty (e(x))
(1 (1-t)y t)y
(13)
e- (z)
ZE
(0.12
(00 (1 (1- (0) ε(x(i)
(14)
1 1- ((x())
which the standard and popular inverse propensity weighting estimator, computed using study population, (z(()) 169), (1)) y of individuals and (usually) fitted regression models é for propensity scores.
19

---

Published in Transactions on Machine Learning Research (01/2026)
D Dataset Details
D.1 Simulation Study
We simulated datasets with four binary covariates and binary treatments and outcomes, as follows.
1. Sample covariates independently as X; ~ Ber(.), where there the Bernoulli parameter for each covariate is in [0.4, 0.6, 0.7] for i {1,2,3,4] 4,4.
2. Simulate obscrvational data by setting propensity score between and that depends on covariates dodot (t-coeff) ) and sampling treatments as T Ber Here, set. t-coeff = 00 -3 0, 0] and expit denotes the logistic sigmoid function.
X, given by ps expit
3. Assign outcomes based on treatments and covariates via a logistic model that includes non-lincar interactions between treatment and covariates. Outcome Y ~ Ber ( expit (logits) where logits beta T + d.t (gamma) d.dot (delta) T. Here, we set beta = 4, gamma = [1, 1,,1,,,00 and delta = 1-2, -3, -2.
4. To further simulate real-world selection biases, we sampled a mask Ber expit (sel-logits) ) with sel-logits delta-y Y delta-t
dot (delta-x), and selected only datapoints for which ((9 1. Here, we set delta-y 2, delta-t 1, and delta-x [10, 10,5,1,
For the setting with six covariates, take X5 and X5 copies of X3 and X4, respectively, and similarly repeat all the corresponding coefficients that determine relationships with T', Y, and S. Note that our choices of coefficients here ensure that treatment effects are heterogencous across subgroups.
D.2 Diabetes Application
We followed the dataset curation and set-up of the Semaglutide vs. Tirzepatide experiment in Dhawan et al. (2024).
1. There are five binary covariates constructed by binning values into discrete categories:
Age: [< 45, 45] years,
Sex: Femaleee
BMI (Body Mass Index): [] 28.5, 28.5] kgs per meter squared,
HbA1c (Glycated Hemoglobin): [< 7.5, 7.5] %, and
Weight: [ 220, > 220] lbs.
2. Possible treatments are Semaglutide (T - 0) and Tirzepatide (T=1).
3. Outcome is whether the user lost 5% or more their intital weight (Y = 1) or not -Y.
This dataset was constructed using unstructured text data found in the PushShift collection (Baumgartner et al. 2020) of Reddit posts upto December 2022. It has been curated to contain submissions and comments that describe users' lived experiences with the treatments and outcome above. Hence, it is naturally prone to the selection biases typical of such a platform, and exemplifies real-world use-case of the sensitivity analyses discussed in this work. Further, observational distributions were estimated by computing conditional probabilitics given by a large language model (LLM), before plugging them into standard causal estimators. Specifically, we followed Dhawan et al. (2024) to sample covariates for each each report in this datasel using GPT-4 and computed the conditional distributions of treatments and outcomes given the covariates and reports using LLAMA-2-70B, This leaves the estimated distributions prone biases of the LLMs as well.
20

---

Published in Transactions on Machine Learning Research (01/2026)
E
Dual Ascent for Constrained Optimization
Here we describe standard convex optimization problem with convex constraints, present standard dual ascent algorithm to solve it, and show that the constrained optimization problem for each of the assumptions in (e, (s, px) can be written in terms of this standard problem.
min ((a
s.t. (a(a) v 0, i= m, (j(a) = 0, j= -;.,..,, n
where (f,h,,E)) are all convex functions a. The corresponding Lagrangian function is given by
m
"
C(a:u.v) = f(a)+ Σ u (a)
E
u 120
the Lagrange dual objective is g(u, = min, C(a; v), and the Lagrange dual problem is
max a.v
s v)
s.t. u 120.
Under convexity and the Slater conditions (Boyd & Vandenberghe 2004), strong duality holds and the dual optimum equals the primal optimum. The following dual ascent algorith solves this optimization problem for differentiable g and small enough learning rate 17.
Algorithm 1: Dual Ascent
Require: Initial ( 2 0, v(0), learning rate 7
for k = 1,2.... do
a (k) E arg mina u(a;u( v(k-1))
((k) t max (1( + t (a(k))
v(k) + (kk- + (a))
if Slater's condition holds then return (k)), (K), ((k
The following define convex constrained optimization problems for each of our assumption functions.
1 1 (o(()), e1(x)) ): f(a) ||a 1112, h1(a) -(a,,,,xx)) hz(a)
n()), f(a) (aI hi ha(a) r(e,a,px)
px, f(a) (RL(a, gx), hi = T 11
Notice from eq. 2 that r(-) as given by eq. is convex in each a defined above. Further notice that L(-) is also convex in each a. The optimization problems for μ and px have additional simplex constraints, which we fold into the minimization in line Hence, we can use Gradient Descent for the unconstrained minimization of L with respect to [z((z), (ج()) and Entropic Mirror Descent for the simplex- contrried minimization of C with respect to p(æ) or Px.
21

---

Published in Transactions on Machine Learning Research (01/2026)
F Further Results
In our simulation study, we visualize subpopulation comparisons of sensitivity values with respect to E, M4, and px in figs. to respectively. When comparing distributions over covariate pairs or triplets in fig. 7 the empirical BSV uncovers trends that are significantly different from not only worst-case sensitivity but also the BSV under uniform priors, highlighting the usefulness of priors that reflect real populations.
1.0
Senstiivity
0.0
02
Wonk-une
BSV
RSV
(Na X1 (Xg, X2) (Ng, Ngt [X, (g)
Individual covariates
Covariate pains
Figure 6: Corst-case sensitivity to unconfoundedness parameter innrecasss as number of observed confounders, and hence, dimension of A, increases. BSV with respect. to e(X) in the simulation study reveals different. trend.
Sensiviity
Senstrvity
Unian BSY
Ustan Bay
Exgenai 30V
Eegenai 200
0.0 No
0.0
X
01
10
Individual covariates
Covariate pair indices
Individual covariates
Covariate poir indices
Covariate miller indices
(a) Simulation with four covariates
(b) Simulation with six covariates
Figure 7: Worst-casc sensitivity to shifts in px over single, pairs, triplcts of covariates becomes increasingly uninformative as the dimensionality of the corresponding assumption parameter spaces increases. Bayesian sensitivity is more informative than the worst-case, with empirical priors revealing different trends than uninformative ones.
Figures 6 and 7 show increasingly high sensitivity when considering joint distributions over more covariates and can not meaningfully distinguish between different sets covariates, espocially as the dimensionality of the corresponding assumption parameter space increases. fact, worst-case sensitivily to unconfoundedness is greater in settings with pairs of observed covariates, relative to single observed covariates, which is unintuitive and unlikely to reflect real scenarios. We attribute this limitation the large dimension of assumption parameter spaces, where the space of possible violations grows exponentially and worst-case analysis becomes increasingly pessimistic, regardless of the plausibility of the worst-case violation in practice. However, BSV does not suffer as much from this curse of dimensionality and better distinguishes between subpopulations even in high-dimensional parameter spaces.
High sensitivity scenarios are precisely the ones where we would like identify subpopulations for further experimentation or data collection. The assumption on conditional outcome distributions is especially prone to exhibiting high sensitivity because shifts in μ very easily shift the ATE across the decision threshold 8. Figure S shows worst-case sensitivity values for this parameter for different covariate and treatment values, indicating highest possible sensitivity in every case. Without incorporating any information on how likely the shifts in this parameter space are in practice, the worst-case analysis lends to be binary in nature, altributing either zero or full sensitivity to different subpopulations. This would be uninformative for practitioners trying decide where allocate resources for more robust ATE estimation. In contrast, BSV also reveals lower overall sensitivity for the treatment T 0, whereas worst-case sensitivity values are too pessimistic to find this insight.
22

---

Published in Transactions on Machine Learning Research (01/2026)
1.3
1.0
Unitomm BSV
Empirical BSV
0.0
0.5
62
0.2
-
-m
W
Vi
Covariate values
Covariate values
Covariate valucs
Figure 8: In the simulation study, BSV with respect to conditional outcome distributions ((X) attributes
lower scnsitivity to T = while worst-case analyses attribute maximum possible scnsitivity to all settings, making it impossible to meaningfully distinguish between subpopulations.
23


Reviews:
[{"review_id": "DX5LvHA9Vh", "reviewer": "Reviewer_UfqP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper recasts three common causal assumptions on unmeasured confounding, outcome models, and covariate distribution into a single assumption-space framework. The authors show that the worst-case sensitivity is a special case that relies on unrealistic changes in the data-generating process. They proposed a Bayesian sensitivity value that averages $\\exp(-D)$ over assumption draws from a prior restricted to the decision-reversing region, with an empirical Bayes version using external datasets such as NHANES, UCI, and subgroup analyses as priors. They provide optimization and sampling routines and a diabetes case study to argue BSV is more informative than worst-case, especially as the dimensionality of the assumption space grows.\n\nKey strengths:\nNicely argued motivation that worst-case optima can be implausible under empirical knowledge.\nA clear, unified notation of the sensitivity parameters across different models.\nAn optimization toolkit.\n\nWeakness:\nThe BSV as a scalar is not very well calibrated, prior‑sensitive, and hard to compare across assumptions.\nThe empirical case mixes heterogeneous evidence with minimal transport diagnostics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are mostly accurate and convincing. For example, the paper shows that the worst-case optima can be implausible and that the worst-case sensitivity tends to saturate in their setups. They also generalize s-value notation cleanly, and the \"expectation <= supremum\" observation correctly implies BSV will be no more pessimistic.", "audience_yn": "Yes", "audience_explanation": "I think so. The work touches a live pain point that worst-case sensitivity often leads to too conservative conclusions, and it offers a general, implementable alternative with appealing visuals. Hope the unifying formulation and the idea of empirical priors for sensitivity are attractive to causal ML and applied researchers.", "requested_changes": "I think the paper can be significantly strengthened by addressing the following substantive issues:\n\n1. The paper proposes to pool the three assumptions ($\\epsilon, \\mu, p_X$) and compute the generalized s-value jointly. However, these components differ fundamentally: $\\mu$ and $p_X$ can, at least in principle, be tested and/or validated using the observed data, while $\\epsilon$ is a causal identification assumption that is inherently untestable at all from the observed data. Treating all three within a single metric may obscure this asymmetry and make interpretation difficult. It would strengthen the paper to explicitly discuss this distinction, both conceptually and methodologically, and to consider whether it calls for different treatment (e.g., distinct priors, divergences, or sensitivity scales) for these assumption types.\n\n2. The empirical results illustrate that BSV can yield more discriminative values than worst-case sensitivity in the presented settings. However, the evidence of reliability or superiority beyond carefully chosen priors and discretizations is limited. The current conclusions seem heavily dependent on specific prior shapes (e.g., truncated Gaussians for $\\epsilon$) and binarized covariates. Including prior-sensitivity or stability analyses would greatly strengthen the methodological credibility and help clarify when BSV offers genuine robustness rather than prior-driven variation.\n\n3. Relevantly, the metric's semantics are weak, the role of priors is under-validated, and the diabetes case leans on LLM-extracted observational quantities from Reddit plus priors from unrelated cohorts, without transport diagnostics. I would suggest reporting $Pr(\\tau(A)\\leq \\delta)$ and BSV jointly, and add uncertainty and acceptance rates.", "broader_impact_concerns": "As mentioned above, I would be concerned if the prior for $\\mu$ is specified only for one treatment arm, since this could inadvertently favor the arm with a richer or better informed prior.\n\nAlso the BSV is conditional on $\\tau(A)\\leq \\delta$, stakeholders may misinterpret it as reflecting the overall robustness of the analysis while overlooking how often such decision reversals actually occur.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kvWo8NxSVZ", "reviewer": "Reviewer_BP7m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the recent s-value framework to a broader class of assumptions and introduce a new Bayesian sensitivity criterion: the Bayesian Sensitivity Value (BSV). BSV computes the expected sensitivity under prior distributions from real-world data. They provide algorithms for computing both worst-case and Bayesian sen", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. This paper is motivated by the limitations of existing sensitivity analysis frameworks, which typically focus on worst-case assumption violations. Such approaches can be overly pessimistic and may not reflect realistic scenarios. The motivation is effectively illustrated in Figure 1 and Figure 2.\n2. The authors extend the s-value framework beyond covariate distribution shifts by incorporating evidence-based priors. The proposed BSV is mathematically formulated in Equation 7, which generalizes the original s-value formulation to a broader class of assumptions.\n3. Two algorithmic approaches are proposed to compute sensitivity values under the BSV framework: constrained optimization for worst-case sensitivity and Monte Carlo sampling for Bayesian sensitivity. These practical algorithms are detailed in Section 4.\n4. The proposed framework is empirically validated through both simulation studies and a real-world diabetes treatment case study. As shown in Section 6, BSV consistently provides more nuanced and realistic sensitivity assessments compared to worst-case analyses.", "audience_yn": "Yes", "audience_explanation": "The causal inference community will be interested in this paper.", "requested_changes": "1. The paper demonstrates BSV primarily under empirical or data-driven priors, but does not evaluate how BSV behaves under prior misspecification, which is a realistic concern in applied settings. It is recommended to study the BSV under a range of prior distributions and the results under prior misspecification.\n2. It is interesting to discuss whether the proposed method is applicable to other unidentifiable causal inference conditions.", "broader_impact_concerns": "None", "additional_comments": "The paper mentions that observational distributions in the diabetes treatment case study are extracted using a large language model from Reddit data (Pushshift dataset). However, this procedure is not clearly presented. It is suggested to discuss this, such as the specific role and configuration of the LLM in the pipeline and evidence of reliability.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PMPVTqEIYY", "reviewer": "Reviewer_yiAZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework to estimate the sensitivity of causal estimators to assumptions pertaining to causal data. It introduces a new criterion, Bayesian Sensitivity Value, built upon the existing s-value framework, to measure the sensitivity of an estimator based on user-defined prior distribution. The prior may be specified using domain knowledge or empirically estimated from real studies relevant to the dataset of interest. This paper applies the new framework to both a simulated and a real-world dataset and showcases the differences in the sensitivity analyses performed using BSV and the s-value (worst-case) criteria. The experiments conclude that by using BSV, one may obtain a more informative and realistic sensitivity analysis when compared to the worst-case measure.   \n\nStrengths: \n- The paper is well-written with the motivation and the problem statement being very clear. The use of the real-world example and the corresponding analysis makes both the criteria and the experiments easy to follow. \n- The empirical analysis of BSV for both simulated and the real-world example is thorough and well-demonstrated.\n- The comparison of BSV and worst-case analysis is well-motivated and clearly articulated. \n\nWeaknesses: \n- Limited theoretical analysis of the proposed optimization (and sampling) problem: The following questions remain unanswered in the current paper: Is an exact solution always guaranteed to exist for the optimization problem? Under what conditions will this not hold, and how does that depend on the space of the assumptions $\\mathcal{A}$? The intended audience may not be as familiar with optimization and sampling, and it may help to provide a simple explanation of the conditions under which you are gauranteed to reach an exact solution.\n- Unclear if the benefits offered by the framework are equivalent across all types of causal estimators: the authors include the set of assumptions for the inverse propensity score weighting estimator in the Appendix, but provide limited explanation of whether the space of assumptions for the IPW estimators is equally estimable from real-world data and if the optimization problem is tractable. Similarly, a number of other estimators are commonly used (doubly robust methods, semi-parametric inference). There is no mention of whether this framework is applicable for all types of estimators and if not, what are the limitations? \n- Limited applicability to other causal estimation tasks: When covariates/outcome tend to be a mixture of continuous and discrete variables, it is unclear whether the BSV and the corresponding optimization and sampling algorithms are directly applicable (or scalable).   \n- Limited comparison with other sensitivity analysis frameworks: Currently the new critierion is only compared to the s-value framework, however there are other sensitivity analysis methods (e.g. Tukey factorization (Franks et al., 2020)) that can also be extended to other assumptions (apart from unconfoundedness). \n\nReferences\n- Franks, AlexanderM, Alexander D’Amour, and Avi Feller. \"Flexible sensitivity analysis for observational studies without observable implications.\" Journal of the American Statistical Association (2020).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims made by the authors in this paper are that worst-case analysis tends to be unrealistic and any sensitivity analysis framework considering only the covariate distribution may be incomplete for a causal estimation task. The authors support this claim by showing that, for an outcome imputation estimator, there are a number of additional assumptions that tend to be made in practice (unconfoundedness and well-specified outcome models) which can be incorporated into the sensitivity analysis frameworks. \n\nHowever, I would recommend that the authors be explicit about the set of causal estimators to which their framework applies as they state that their framework is a general sensitivity analysis framework for causal estimators. From the current version of the paper, it is not clear whether the space of assumptions for all estimators leads to a formulation of the constrained optimization problem which can be easily solved. (See details in the Requested Changes).", "audience_yn": "Yes", "audience_explanation": "This is an important and well-motivated problem in the causal inference literature. The practical application of any proposed framework is useful to consider, and this paper does a good job of highlighting the issues with considering sensitivity analyses that are based on worst case criteria.", "requested_changes": "Major Revisions (Prior to acceptance) \n\n- Comprehensive comparison to other sensitivity analysis frameworks: In the list of the contributions, the authors state \"We unify sensitivity analysis frameworks with respect to different types of assumptions in a general framework that allows practitioners to ask targeted questions about the sensitivity of a study.\" This seems more general than the scope of the current paper. There are other ways to do sensitivity analysis which treats the causal inference task as a missing data problem (e.g., Franks et al., 2020). Such methods provide a confidence interval for the causal effect based on the possible bounds of the parameters. These methods may also be extended to other assumptions (and not just unconfoundedness) and can also be translated to the decision-making setting. Therefore, I believe that the current framework, while extending the s-value framework to assumptions pertaining to causal estimators is still not unifying all possible ways of conducting sensitivity analysis. I would like to at least see a comparison of the advantages and disadvantages of the BSV criteria when compared to other sensitivity analysis methods, and if feasible (using the same set of assumptions) an empirical comparison as well. \n- Rigorous theoretical analysis of the constrained optimization problem: Is it guaranteed that we will reach the optima in the current setting? Under what conditions will that not hold true? I would like to see an explanation in the main text (and additional theorems in the appendix for any proofs, if required). \n- Clarification on the scope of the causal estimation task: While the paper explicitly focuses on the case of discrete variables for all (covariates/treatment/outcome), there is no mention of whether this framework is generalizable to other cases. Is the recommendation that all variables have to be discretized appropriately for this framework to be applied? If so, this has to be explicitly stated in Section 2.1. \n- Clarification on the types of causal estimators for which such a framework is applicable: There are a number of other causal estimators which are in use in practice today (e.g. BART, double machine learning). The paper focuses on two estimators (outcome imputation and IPW), but does not clarify whether it is generalizable, and what are the barriers to generalization, if any. This would be required for this framework to be a \"general framework for sensitivity analysis\". Specifically, include whether the space of assumptions for other estimators can be broken down to a discrete, bounded space for the optimization problem to be solvable.  \n\nMinor Revisions (Good to include)\n\nThe following parts can to be rewritten/improved for clarity\n- Introductory graphic. Figure 1 is difficult to follow along with the Introduction (as the meaning of $\\pi$ etc is not explained). I would improve the explanation in the caption or use a simpler graphic to explain the concept. \n- The current writing assumes that the users have an understanding of the s-value framework in the beginning, which may not be true. I would summarize the framework in a couple of sentences in the introduction. \n- Most of the introduction focuses on the usefulness of deriving empirical priors from real-world data, but this is not possible for all assumptions ($\\epsilon$). I would state that upfront in Section 2.1 (instead of in the experiments section). \n- Experiments: It is not immediately clear what the covariate pair indices are in Figure 5. \n\nProvide intuition for some of the observations: In the experiments, you mention that BSV is able to provide trends for $\\mathcal{A}$ even in high dimensions. It would be good to include some intuition for why this is the case (with respect to either the sampling or constrained algorithms).", "broader_impact_concerns": "The authors do a good job of stating the broader impact concerns when applying these methods to real-world problems.", "additional_comments": "Questions\n\n- One of the key advantages this paper offers is an interpretable sensitivity analysis criterion that can be applied across a set of assumptions ($\\epsilon, \\mu, p_X$). In a way, the BSV is a way of incorporating a joint prior over all these functions. However, any sensitivity analysis that is performed in the experiments are only shown for combinations of covariates (pairs/triplets) for each function. Is there a way to extend this across functions? Can you consider a combination of $\\sigma$ and $\\mu$ values for the decision boundary and show this in your experiments? It would strengthen the contributions of this paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)