SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyperparameters. We use the Adam optimizer [71] with a learning rate  $\eta = 0.01$  and a minibatch size of  $b = 32$  to fit our APID. This  $b = 32$  is also used for all the sampling routines at other losses. For the residual normalizing flows, we use  $t = 15$  residual transformations, each with  $h_t = 5$  units of the hidden layers. We set the relative and absolute tolerance of the fixed point iterations to 0.0001 and a maximal number of iterations to 200. For variational augmentations, we set the number of units in the hidden layer to  $h_g = 5$ , and the variance of the augmentation to  $\varepsilon^2 = 0.5^2$ . For the noise regularization, we chose  $\sigma^2 = 0.001^2$ . This configuration resulted in a good fit (with respect to the test log-likelihood) for all the experiments, and, thus, we did not tune hyperparameters. We varied the coefficients for the losses,  $\lambda_Q$  and  $\lambda_\kappa$ , and report the results in Sec. 6 and Appendix H.
H Synthetic experiments
We conduct the experiments with datasets, drawn from two synthetic datasets. In the first dataset, we use
$$
\left\{ \begin{array}{l} Y \mid 0 \sim \mathbb {P} (Y \mid 0) = N (0, 1), \\ Y \mid 1 \sim \mathbb {P} (Y \mid 1) = N (0, 1), \end{array} \right. \tag {105}
$$
and, in the second,
$$
\left\{ \begin{array}{l} Y \mid 0 \sim \mathbb {P} (Y \mid 0) = \text {M i x t u r e} \left(0. 7 N \left(- 0. 5, 1. 5 ^ {2}\right) + 0. 3 N \left(1. 5, 0. 5 ^ {2}\right)\right), \\ Y \mid 1 \sim \mathbb {P} (Y \mid 1) = \text {M i x t u r e} \left(0. 3 N \left(- 2. 5, 0. 3 5 ^ {2}\right) + 0. 4 N \left(0. 5, 0. 7 5 ^ {2}\right) + 0. 3 N \left(2. 0, 0. 5 ^ {2}\right)\right). \end{array} \right. \tag {106}
$$
Importantly, the ground truth SCMs (and their  $\kappa$ ) for both datasets remain unknown, and we only have access to the observational distributions. This is consistent with the task of partial counterfactual identification.
H.2 Additional results
In Fig. 14, we provide additional results for the synthetic experiments. For our APID, we set  $\lambda_{Q} = 1.0$  and vary  $\lambda_{\kappa}\in \{0.0,0.5,1.0,5.0\}$ . Here, we see that estimated by APID bounds are located closer to (i) the theoretic upper and lower bounds of BGMs and (ii) closer to each other.
Figure 14: Additional results for partial counterfactual identification of the ECOU across two datasets. Reported: theoretical bounds of BGMs and mean bounds of APID over five runs. Here,  $\lambda_{Q} = 1.0$ .
Table 2 provides the duration of one training iteration of our APID for different training stages. Namely, we report the mean and std. across all the experiments (total of 720 runs) for the burn-in, the query, and the curvature-query stages. In our experiments, the long training times are attributed to the reverse transformations in the residual normalizing flows (as they require fixed point iterations) and to the computation of the Hessian to evaluate the curvatures.
Training stage	Number of training iterations	Evaluated losses	Duration per training iteration (in s)
Burn-in stage	nb=500	LNLL, LW	2.60 ± 1.72
Query stage	nQ=100	LNLL, LW, LO	11.04 ± 10.65
Curvature-query stage	ncQ=500	LNLL, LW, LO, Lκ	39.27 ± 27.26
Table 2: Total runtime (in seconds) for different stages of training for our APID. Reported: mean ± standard deviation across all the experiments, which amount to a total of 720 runs (lower is better). Experiments were carried out on 2 GPUs (NVIDIA A100-PCIE-40GB) with IntelXeon Silver 4316 CPUs @ 2.30GHz.
I Case study with real-world data
In our case study, we want to analyze retrospective "what if" questions from the COVID-19 pandemic. Specifically, we ask what the impact on COVID-19 cases in Sweden would have been, had Sweden implemented a stay-home order (which Sweden did not do but which is under scrutiny due to post-hoc legal controversies).
I.1 Dataset
We use multi-country data from [7].<sup>9</sup> It contains weekly-averaged number of recorded COVID-19 for 20 Western countries. The outcome,  $Y$ , is the (relative) case growth per week (in log), namely, the ratio between new cases and cumulative cases. The treatment,  $A \in \{0,1\}$ , is a binary variable referring to either no lockdown or enforcement of a stringent lockdown, respectively. We assume the data is i.i.d. and comes from a randomized control trial, i.e., there is no confounding between  $Y$  and  $A$ . We filtered out the weeks where the cumulative number of cases is smaller than 50, which resulted in  $n = n_0 + n_1 = 136 + 124$  observations in total, for both treatments, respectively. We plot a histogram of case growth per week depending on the treatment in Fig. 15, i.e., empirical densities of distributions  $\hat{\mathbb{P}}(Y \mid A = 0)$  and  $\hat{\mathbb{P}}(Y \mid A = 1)$ .
Figure 15: Empirical distributions of case growth per week (in log) for both treatments. Larger values mean a larger incidence.
We assume that the observational distributions  $\mathbb{P}(Y\mid a)$  are induced by some (unknown) SCM of class  $\mathfrak{B}(C^2,2)$ . Then, we fit our APID the same way as in synthetic experiments (see Appendix G). We additionally report the BGMs-EQTDs identification gap based on empirical CDF of the observational distributions.
I.2 Results
Fig. 16 shows the results for (upper) partial counterfactual identification of the ECOU for our method. Hence, the ECOU reports for the expected counterfactual case growth for countries without stay-home orders, i.e., the weekly case growth (in log) if a stay-home order would have actually been enforced. We also highlight the upper bounds on the ECOU for Sweden during weeks 10-13 of 2020, where no lockdown was implemented. Evidently, the ECOU would lie below the decision boundary for BGMs-EQTDs and for certain levels of  $\lambda_{\kappa}$ . This implies that Sweden could have had significantly lower case growth had it implemented a stay-home order, even when allowing for some level of non-linearity between outcome and independent latent noises that interact with the treatment, e.g., cultural distinctions, level of trust in government, etc. The results of the COVID-19 case study demonstrate that our CSM and its instantiation with APID could be applied for decision-making with real-world data.
The data is available at https://github.com/nbanho/npi-effectiveness_firstwave/blob/master/data/data_preprocessed.csv.
Figure 16: Case study where we analyze retrospective "what if" questions from the COVID-19 pandemic. Reported: BGMs-EQTDs identification gap and ECOU upper bounds of APID over five runs (in log). The gray-shaded area above denotes the decision boundary with the negative counterfactual effect, namely where a stay-home order would have led to an increase in case growth, respectively. The red lines show the observed case growth for Sweden during weeks 10–13 of 2020, where no stay-home order was implemented.