SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Conditional probability mass functions: The estimation of the conditional probability mass function  $\mathbb{P}^{M_i}(\cdot \mid \mathbf{x},\bar{\mathbf{m}}_{i - 1},\mathbf{a})$  is a standard (multi-class) classification problem. We use feed-forward neural networks with 3 hidden layers, ReLU activation functions, and softmax output. For training, we minimize the standard cross-entropy loss by using the Adam optimizer [41]. We use the same approach to estimate the propensity scores  $\mathbb{P}^{\mathbf{A}}(\cdot \mid \mathbf{x})$  for discrete treatments  $\mathbf{A}$ .
Hyperparameter tuning:
We perform hyperparameter tuning for our experiments on synthetic data using grid search on a validation set. The tunable parameters and search ranges are shown in Table 2. For reproducibility purposes, we report the selected hyperparameters as .yaml files. $^3$
Table 2: Hyperparameter tuning details.
MODEL	TUNABLE PARAMETERS	SEARCH RANGE
CNFs	Epochs	50
Batch size	32, 64, 128
Learning rate	0.0005, 0.001, 0.005
Hidden layer size (hyper network)	5, 10, 20, 30
Number of spline bins	2, 4, 8
Feed forward neural networks	Epochs	30
Batch size	32, 64, 128
Learning rate	0.0005, 0.001, 0.005
Hidden layer size	5, 10, 20, 30
Dropout probability	0, 0.1
$^{3}$ Code is available in the supplementary materials and at https://github.com/DennisFrauen/SharpCausalSensitivity.
H Experiments using synthetic data
Here we provide details regarding our experiments using synthetic data. This includes data generation, obtaining oracle sensitivity parameters, and details regarding experimental evaluation.
Overall data-generating process: We first describe the overall data-generating process which we use as a basis to generate data for all settings (i)-(iii) and binary/continuous treatments. We construct an SCM following the causal graph in Fig. 1 (right) from the main paper. We have an observed confounder  $X \in \mathbb{R}$ , a (binary or continuous) treatment  $A$ , two binary mediators  $M_1$  and  $M_2$ , and a continuous outcome  $Y \in \mathbb{R}$ . Furthermore, we consider three unobserved confounders: (i)  $U_{M_1}$  confounding the  $A - M_1$  relationship, (ii)  $U_{M_2}$  confounding the  $A - M_2$  relationship, and (iii)  $U_Y$  confounding the  $A - Y$  relationship. Our data-generating process is inspired by synthetic experiments from previous works on causal sensitivity analysis [31, 35]. We start the data-generating process by sampling
$$
X \sim \text {U n i f o r m} [ - 1, 1 ], \quad \text {a n d} \quad U _ {M _ {1}}, U _ {M _ {2}}, U _ {Y} \stackrel {(i. i. d)} {\sim} \operatorname {B e r n o u l l i} (p = 0. 5) \tag {72}
$$
Depending on the setting, we either generate binary treatments  $A \in \{0,1\}$  via
$$
A \sim \operatorname {B e r n o u l l i} \left(\operatorname {s i g m o i d} \left(3 x + \gamma_ {\mathrm {M} _ {1}} \mathrm {u} _ {\mathrm {M} _ {1}} + \gamma_ {\mathrm {M} _ {2}} \mathrm {u} _ {\mathrm {M} _ {2}} + \gamma_ {\mathrm {Y}} \mathrm {u} _ {\mathrm {Y}}\right)\right) \tag {73}
$$
or continuous treatments  $A\in (0,1)$  via
$$
A \sim \operatorname {B e t a} (\alpha , \beta) \text {w i t h} \alpha = \beta = 2 + x + \gamma_ {M _ {1}} \left(u _ {M _ {1}} - 0. 5\right) + \gamma_ {M _ {2}} \left(u _ {M _ {2}} - 0. 5\right) + \gamma_ {Y} \left(u _ {Y} - 0. 5\right), \tag {74}
$$
where  $\gamma_{M_1},\gamma_{M_2}$ , and  $\gamma_{Y}$  are parameters controlling the strength of unobserved confounding. We then generate the mediators and outcome via functional assignments
$$
M _ {1} = f _ {M _ {1}} \left(X, A, U _ {M _ {1}}, \epsilon_ {M _ {1}}\right), \quad M _ {2} = f _ {M _ {2}} \left(X, A, M _ {1}, U _ {M _ {2}}, \epsilon_ {M _ {2}}\right) \tag {75}
$$
$$
Y = f _ {Y} \left(X, A, M _ {1}, M _ {2} U _ {Y}, \epsilon_ {Y}\right), \tag {76}
$$
where  $\epsilon_{M_1},\epsilon_{M_2},\epsilon_Y\sim \mathcal{N}(0,1)$  are standard normal distributed noise variables. The functional assignments are defined as
$$
f _ {M _ {1}} (x, a, u _ {M _ {1}}, \epsilon_ {M _ {1}}) = \mathbb {1} \left\{a \sin (x) + (1 - a) \sin (4 x) + \rho_ {M _ {1}} \left(\left(u _ {M _ {1}} - 0. 5\right) + \epsilon_ {M _ {1}}\right) > 0 \right\} \tag {77}
$$
for  $M_1$
$$
\begin{array}{l} f _ {M _ {2}} (x, a, m _ {1} u _ {M _ {2}}, \epsilon_ {M _ {2}}) = \mathbb {1} \left\{a m _ {1} \sin (x) + (1 - a) m _ {1} \sin (4 x) \right. (78) \\ - a (1 - m _ {1}) \sin (x) - (1 - a) (1 - m _ {1}) \sin (4 x) (79) \\ + \rho_ {M _ {2}} ((u _ {M _ {2}} - 0. 5) + \epsilon_ {M _ {2}}) > 0 \} (80) \\ \end{array}
$$
for  $M_2$ , and
$$
\begin{array}{l} f _ {Y} (x, a, m _ {1}, m _ {2}, u _ {Y}, \epsilon_ {Y}) = a m _ {1} m _ {2} \sin (x) + (1 - a) m _ {1} m _ {2} \sin (4 x) (81) \\ + a m _ {1} (1 - m _ {2}) \sin (8 x) + (1 - a) m _ {1} (1 - m _ {2}) \sin (x) (82) \\ - a \left(1 - m _ {1}\right) m _ {2} \sin (x) - (1 - a) \left(1 - m _ {1}\right) m _ {2} \sin (4 x) (83) \\ - a (1 - m _ {1}) (1 - m _ {2}) \sin (8 x) (84) \\ - (1 - a) \left(1 - m _ {1}\right) \left(1 - m _ {2}\right) \sin (x) (85) \\ + \rho_ {Y} ((u _ {Y} - 0. 5) + \epsilon_ {Y}) (86) \\ \end{array}
$$
for  $Y$ , where  $\rho_{M_1}, \rho_{M_2}$ , and  $\rho_Y$  are parameters that control the noise level.
Settings (i)-(iii): We define the settings (i)-(iii) in Sec. 5 via specific values of the confounding parameters  $\gamma_{M_1},\gamma_{M_2}$ , and  $\gamma_{Y}$ , and the noise parameters  $\rho_{M_1},\rho_{M_2}$ , and  $\rho_{Y}$  (see Table 3). Note that the settings are defined to mimic the causal graphs in Fig. 1 from the main paper. For example, the only unobserved confounder in setting (i) is  $U_{Y}$ , which means that we can ignore the mediators and use our data to evaluate our bounds for settings without mediators.
Table 3: Definition of settings (i)-(iii).
γM1	γM2	γY	ρM1	ρM2	ρY
Setting (i), binary A	0	0	1.5	0.2	0.2	2
Setting (i), continuous A	0	0	1.5	0.2	0.2	1
Setting (ii)	1.5	0	1.5	1	0.2	1
Setting (iii)	1.5	1.5	1.5	0.2	0.2	1
Obtaining  $\Gamma_W^*$ : We provide details regarding our approach to obtain oracle sensitivity parameters  $\Gamma_W^*$  for all  $W \in \{M_1, M_2, Y\}$ . By sampling from our previously defined SCM, we can obtain Monte Carlo estimates of the GMSM density ratio
$$
r \left(u _ {W}, x, a\right) = \frac {\mathbb {P} \left(u _ {W} \mid x , a\right)}{\mathbb {P} \left(u _ {W} \mid x , a\right)} \stackrel {*} {=} \frac {\mathbb {P} \left(a \mid x , u _ {W}\right)}{\mathbb {P} \left(a \mid x\right)} \tag {87}
$$
for all  $u_W \in \{0, 1\}$ ,  $a$ , and  $x$ , where (*) follows from Bayes' theorem. We then define
$$
r _ {W} ^ {+} (x, a) = \max  _ {u _ {W} \in \{0, 1 \}} r \left(u _ {W}, x, a\right) \quad \text {a n d} \quad r _ {W} ^ {-} (x, a) = \min  _ {u _ {W} \in \{0, 1 \}} r \left(u _ {W}, x, a\right). \tag {88}
$$
For binary treatment settings, we define parameters  $\Gamma_W^+ = \Gamma_W^+(x,a)$  and  $\Gamma_W^- = \Gamma_W^- (x,a)$  that attain the density ratio bounds in the MSM from Eq. (39), i.e.
$$
r _ {W} ^ {+} (x, a) = \frac {1}{\left(1 - \Gamma_ {W} ^ {+} {} ^ {- 1}\right) \mathbb {P} (a \mid x) + \Gamma_ {W} ^ {+} {} ^ {- 1}} \quad \text {a n d} \quad r _ {W} ^ {-} (x, a) = \frac {1}{\left(1 - \Gamma_ {W} ^ {-}\right) \mathbb {P} (a \mid x) + \Gamma_ {W} ^ {-}}. \tag {89}
$$
For continuous treatment settings, we define  $\Gamma_W^+$  and  $\Gamma_W^-$  as the sensitivity parameters that attain the density ratio bounds in the CMSM from Eq. (44), i.e.
$$
r _ {W} ^ {+} (x, a) = \Gamma_ {W} ^ {+} \quad \text {a n d} r _ {W} ^ {-} (x, a) = \frac {1}{\Gamma_ {W} ^ {-}}. \tag {90}
$$
Finally, we define  $\Gamma_W^*$  as the parameter corresponding to the maximum possible violation of unconfoundedness, i.e.,
$$
\Gamma_ {W} ^ {*} = \max  \left\{\Gamma_ {W} ^ {+}, \Gamma_ {W} ^ {-} \right\} \tag {91}
$$
By definition of  $\Gamma_W^*$ , our bounds should contain the oracle causal effect whenever we choose sensitivity parameters  $\Gamma_W \geq \Gamma_W^*$  for all  $W \in \{M_1, M_2, Y\}$ .
Weighted GMSM experiment (Table 1): For our experiment in Table 1, we modify the treatment assignment from Eq. (74) in setting (i) to
$$
A \sim \operatorname {B e t a} (\alpha , \beta) \tag {92}
$$
$$
\alpha = \beta = 2 + x + \mathbb {1} (x <   0) \left(\gamma_ {M _ {1}} \left(u _ {M _ {1}} - 0. 5\right) + \gamma_ {M _ {2}} \left(u _ {M _ {2}} - 0. 5\right) + \gamma_ {Y} \left(u _ {Y} - 0. 5\right)\right). \tag {93}
$$
Hence, unobserved confounding only affects individuals with  $x < 0$ . We then compare our bounds under the CMSM with our bounds under a weighted CMSM (Def. 4) with weight function  $q_{Y}(x) = \mathbb{1}(x > 0)$ .
We also provide results for the bounds from Jesson et al. [32] under the CMSM. We implemented the grid search algorithm from Jesson et al. [32] and used 5,000 samples for the search space. For a fair comparison, we also used 5,000 samples for our importance sampling estimators. Note that the method from Jesson et al. [32] requires estimation of both the conditional outcome density  $\mathbb{P}^Y (\cdot \mid \mathbf{x},\mathbf{a})$  and the conditional expectation  $\mathbb{E}[Y\mid \mathbf{x},\mathbf{a}]$ . For  $\mathbb{P}^Y (\cdot \mid \mathbf{x},\mathbf{a})$ , we use the same (normalizing flow-based) estimator as for our bounds. For  $\mathbb{E}[Y\mid \mathbf{x},\mathbf{a}]$ , we train a separate feed-forward neural network with linear output activation for continuous outcomes. Implementation and hyperparameter tuning are done the same way as described in Appendix G for the feed-forward neural networks.
I Experiment using real-world data
Data: We consider a setting from the COVID-19 pandemic where mobility in Switzerland (captured through telephone movement) was monitored to obtain a leading predictor of case growth. In total,  $\sim 1$ , 5 billion trips were monitored from 10 February through 26 April 2020. All data are recorded across 26 different states (cantons). For our analysis, we use an aggregated, de-identified, and pre-processed version of the data provided by Persson, Parie, and Feuerriegel [57]. The preprocessed data is publically available at https://github.com/jopersson/covid19-mobility/blob/main/Data. The code for our analysis is available at https://github.com/DennisFrauen/SharpCausalSensitivity.
We consider a binary treatment  $A$  in the form of a stay-at-home order, which bans gatherings with more than 5 people. We encode mobility as a single binary mediator  $M$ , which is 1 if the total number of trips on a specific day is larger than the median number of trips during the entire time horizon, and 0 otherwise. Our outcome is the 10-day-ahead case growth. We include the following observed variables as confounders  $\mathbf{X}$ : the canton code (swiss member state at a subnational level), the canton population, and whether the weekday is a Monday or not. After removing the first 10 recorded days for each canton (due to spillover effects from other countries) and rows with missing values, we obtain a dataset with  $n = 3276$  observations.
Analysis: We perform a causal sensitivity analysis for the natural directed effect (NDE) of the stay-at-home order  $A$  on the case growth  $Y$ . That is, we are interested in the part of the causal effect of  $A$  on  $Y$  that is not explained by the path via  $M$  (i.e., through the change in mobility). The NDE in an SCM  $\mathcal{M}$  is defined as
$$
N D E (\mathcal {M}) = \int Q (\mathbf {x}, (a _ {1} = 0, a _ {2} = 1), \mathcal {M}) - Q (\mathbf {x}, (a _ {1} = 0, a _ {2} = 0), \mathcal {M}) \mathrm {d} \mathbf {x}. \tag {94}
$$
Fig. 5 (main paper) shows causal sensitivity analysis for violations of the unconfoundedness between treatment  $A$  and mediator  $M$ . Hence, we consider a GMSM for binary treatments with sensitivity parameters  $\Gamma_{M}$  and  $\Gamma_{Y} = 0$ . For each  $\Gamma_{M}$ , we estimate our bounds for the expectation functional and the treatment combinations  $\bar{\mathbf{a}} = (0,1)$  and  $\bar{\mathbf{a}} = (0,0)$ . We then obtain bounds for the NDE as described in Appendix D.
J Additional experimental results
Here, we provide additional experimental results on synthetic data that extend the results from Sec. 5 in the main paper. We provide (i) results for additional treatment combinations and (ii) results for distributional effects. We follow the same experimental setup described in Sec. 5 (main paper) and Appendix. H.
J.1 Additional treatment combinations
Results for additional treatment combinations are shown in Fig. 6 (binary treatment settings) and Fig. 7 (continuous treatment settings). The results are similar to those in Sec. 5 in the main paper and empirically confirm the validity of our bounds. Hence, our results remain valid independently of the choice of treatment combination.
Figure 6: Results for additional treatments in the binary treatment setting. From left to right is shown: setting (ii) with  $\bar{\mathbf{a}} = (0,1)$ , setting (iii) with  $\bar{\mathbf{a}} = (0,1,0)$ , and setting (iii) with  $\bar{\mathbf{a}} = (0,0,1)$ . The top row shows the oracle sensitivity parameter  $\Gamma_W^*$  (depending on  $x$ ), and the bottom row shows the bounds.
J.2 Distributional effects
We also provide results for distributional effects, that is, we choose the  $\alpha$ -quantile functional  $\mathcal{D}\left(\mathbb{P}_+^Y (\cdot |\mathbf{x},\bar{\mathbf{m}}_\ell ,\mathbf{a})\right) = F_{\mathbb{P}^Y | \mathbf{x},\bar{\mathbf{m}}_\ell ,\mathbf{a}}^{-1}(\alpha)$ . Here, we consider three quantiles with  $\alpha = 0.7$ ,  $\alpha = 0.5$  (median), and  $\alpha = 0.3$ . We use our importance sampling estimator derived in Appendix. F (Eq. (68)) to estimate our bounds. The results are shown in Fig. 8 (binary treatment) and Fig. 9 (continuous treatment) for settings (i)-(iii) from Fig. 1 in the main paper. Again, our bounds cover the underlying oracle effect in regions where the chosen sensitivity parameters  $\Gamma_W$  are larger than the oracle sensitivity parameters  $\Gamma_W^*$ . This also confirms empirically the validity of our bounds for distributional effects.
J.3 Semi-synthetic data
We provide additional results for the semi-synthetic IHDP data with unobserved confounding from Jesson et al. [31]. Here, we demonstrate the effectiveness of our bounds for distributional effects for decision-making. We follow Jesson et al. [31] and evaluate our bounds by measuring the error rate of associated deferral policies, which defer a prespecified fraction of test samples with the largest uncertainty to an expert. We mimic a high-stakes decision-making problem where a wrong decision to prescribe treatment is much worse than a wrong decision not to prescribe treatment. To do so, we measure performance with a weighted error rate, that penalizes wrong
Figure 7: Results for additional treatments in the continuous treatment setting. From left to right is shown: setting (i) with  $\bar{\mathbf{a}} = 0.9$ , setting (ii) with  $\bar{\mathbf{a}} = (0.2, 0.4)$ , and setting (iii) with  $\bar{\mathbf{a}} = (0.4, 0.5, 0.3)$ . The top row shows the oracle sensitivity parameter  $\Gamma_W^*$  (depending on  $x$ ), and the bottom row shows the bounds.
treatment decisions twenty times heavier than wrong decisions not to treat. We then compare three different deferral policies: (i) the (standard) policy based on the expectation which treats whenever  $\mathbb{E}[Y|\mathbf{X} = \mathbf{x}, A = 1] > \mathbb{E}[Y|\mathbf{X} = \mathbf{x}, A = 0]$ , (ii) a more conservative quantile policy that treats whenever  $Q_{q}[Y|\mathbf{X} = \mathbf{x}, A = 1] > Q_{1 - q}[Y|\mathbf{X} = \mathbf{x}, A = 0]$  with  $q = 0.4$  ( $Q_{q}$  denotes the q-quantile of  $\mathbb{P}(Y|\mathbf{X} = \mathbf{x}, A = i)$ ), and (iii) the same quantile policy with  $q = 0.2$ . Note that our bounds for policy (i) coincide with sharp bounds for binary CATE from the literature [18]. As done in [31], we report means and standard deviations of 400 random initializations of the IHDP data. The results (Fig. 10) show that the conservative quantile policies improve over the expectation policy.
Figure 8: Results for distributional effects in the binary treatment setting using the same treatments as in Fig. 3 (main paper). Settings (i)-(iii) are ordered from left to right. The top row shows the oracle sensitivity parameter  $\Gamma_W^*$  (depending on  $x$ ). Rows 2, 3, and 4 show the bounds for the  $\alpha$ -quantiles of the interventional distribution with  $\alpha = 0.7$ ,  $\alpha = 0.5$ , and  $\alpha = 0.3$ .
Figure 9: Results for distributional effects in the continuous treatment setting using the same treatments as in Fig. 4 (main paper). Settings (i)-(iii) are ordered from left to right. The top row shows the oracle sensitivity parameter  $\Gamma_W^*$  (depending on  $x$ ). Rows 2, 3, and 4 show the bounds for the  $\alpha$ -quantiles of the interventional distribution with  $\alpha = 0.7$ ,  $\alpha = 0.5$ , and  $\alpha = 0.3$ .
Figure 10: results for the semi-synthetic IHDP data with unobserved confounding from Jesson et al. [31].