SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - g (\boldsymbol {\theta})) ^ {2} ] \geq \mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) ^ {2} ]
$$
for every function  $g(\pmb{\theta})$ . We begin by rearranging terms:
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - g (\boldsymbol {\theta})) ^ {2} ] =
$$
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ] + \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ] - g (\boldsymbol {\theta})) ^ {2} ] =
$$
$$
\mathbb {E} _ {\boldsymbol {\theta}, \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) ^ {2} + (\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ] - g (\boldsymbol {\theta})) ^ {2} ] + X
$$
$$
X = \mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) (\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ] - g (\boldsymbol {\theta})) ].
$$
By the law of iterated expectations, one can show that  $X = 0$ :
$$
X = \mathbb {E} _ {\boldsymbol {\theta} ^ {\prime} \sim p (\boldsymbol {\theta})} \mathbb {E} _ {\boldsymbol {\theta} \sim p (\boldsymbol {\theta})} \mathbb {E} _ {\mathbf {x} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) (\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ] - g (\boldsymbol {\theta})) ].
$$
The first term in the products reads a difference of the same term, and, thus, is zero.
Thus, since  $X = 0$  and since  $\mathbb{E}_{\mathbf{x}'\sim p(\mathbf{x}|\pmb {\theta})}[d(\mathbf{x}',\mathbf{x}_o)] - g(\pmb {\theta})^2\geq 0$  , we have
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\pmb {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - g (\pmb {\theta})) ^ {2} ] \geq \mathbb {E} _ {\theta , \mathbf {x} \sim p (\pmb {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \pmb {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) ^ {2} ].
$$
A3.2 Proposition 2 and proof
Proposition 2. Let  $p(\theta, \mathbf{x})$  be the joint distribution over parameters and data and let  $p(\mathbf{x}_t)$  be a distribution of target samples. Let  $\ell(\theta; \mathbf{x}_o)$  be a cost function and  $f_{\phi}(\cdot)$  a parameterized function as in proposition 1. Then, the loss function  $\mathcal{L} = \mathbb{E}_{p(\theta, \mathbf{x})p(\mathbf{x}_t)}[(f_{\phi}(\theta, \mathbf{x}_t) - d(\mathbf{x}, \mathbf{x}_t))^2]$  is minimized if and only if, for all  $\theta \in \text{supp}(p(\theta))$  and all  $\mathbf{x}_t \in \text{supp}(p(\mathbf{x}_t))$  we have  $f_{\phi}(\theta, \mathbf{x}_t) = \mathbb{E}_{p(\mathbf{x}|\theta)}[d(\mathbf{x}, \mathbf{x}_t)]$ .
$$
\mathbb {E} _ {\boldsymbol {\theta}, \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x}), \mathbf {x} _ {t} \sim p (\mathbf {x} _ {t})} [ (d (\mathbf {x}, \mathbf {x} _ {t}) - g (\boldsymbol {\theta}, \mathbf {x} _ {t})) ^ {2} ] \geq
$$
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x}), \mathbf {x} _ {t} \sim p (\mathbf {x} _ {t})} [ (d (\mathbf {x}, \mathbf {x} _ {t}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {t}) ]) ^ {2} ]
$$
for every function  $g(\pmb {\theta},\mathbf{x}_t)$  . We begin as in proposition 1:
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x}), \mathbf {x} _ {t} \sim p (\mathbf {x} _ {t})} \left[ \left(d (\mathbf {x}, \mathbf {x} _ {t}) - g (\boldsymbol {\theta}, \mathbf {x} _ {t})\right) ^ {2} \right] =
$$
$$
\mathbb {E} _ {p (\mathbf {x} _ {t})} \left[ \mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} \left[ \left(d (\mathbf {x}, \mathbf {x} _ {t}) - g (\boldsymbol {\theta}, \mathbf {x} _ {t})\right) ^ {2} \right] \right.
$$
Below, we prove that, for any  $\mathbf{x}_t$ , the optimal  $g(\pmb{\theta}, \mathbf{x}_t)$  is the conditional expectation  $\mathbb{E}_{\mathbf{x}' \sim p(\mathbf{x}|\pmb{\theta})}[d(\mathbf{x}', \mathbf{x}_t)]$ :
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} \left[ \left(d (\mathbf {x}, \mathbf {x} _ {t}) - g (\boldsymbol {\theta}, \mathbf {x} _ {t})\right) ^ {2} \right] =
$$
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\pmb {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {t}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \pmb {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {t}) ] + \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \pmb {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {t}) ] - g (\pmb {\theta}, \mathbf {x} _ {t})) ^ {2} ] =
$$
$$
\mathbb {E} _ {\boldsymbol {\theta}, \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {t}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {t}) ]) ^ {2} + (\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {t}) ] - g (\boldsymbol {\theta}, \mathbf {x} _ {t})) ^ {2} ] + X
$$
$$
X = \mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) (\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ] - g (\boldsymbol {\theta}, \mathbf {x} _ {t})) ],
$$
which, as above, is  $X = 0$  (proof is identical to proposition 1). Thus, since  $\mathbb{E}_{\mathbf{x}^{\prime}\sim p(\mathbf{x}|\boldsymbol{\theta})}[d(\mathbf{x}^{\prime},\mathbf{x}_t)] - g(\boldsymbol {\theta},\mathbf{x}_t)^2\geq 0$ , we have:
$$
\mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {t}) - g (\boldsymbol {\theta}, \mathbf {x} _ {t})) ^ {2} ] \geq \mathbb {E} _ {\theta , \mathbf {x} \sim p (\boldsymbol {\theta}, \mathbf {x})} [ (d (\mathbf {x}, \mathbf {x} _ {o}) - \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p (\mathbf {x} | \boldsymbol {\theta})} [ d (\mathbf {x} ^ {\prime}, \mathbf {x} _ {o}) ]) ^ {2} ]
$$
Because this inequality holds for any  $\mathbf{x}_t$ , the average over  $p(\mathbf{x}_t)$  will also be minimized if and only if  $g(\theta, \mathbf{x}_t)$  matches the conditional expectation  $\mathbb{E}_{\mathbf{x}' \sim p(\mathbf{x}|\theta)}[d(\mathbf{x}', \mathbf{x}_o)]$  for any  $\mathbf{x}_t$  within the support of  $p(\mathbf{x}_t)$ .
A4 Further experimental details
A4.1 Details on training procedure
Dataset construction We generate samples from the prior  $\theta_{i} \sim p(\pmb{\theta})$  and corresponding simulations  $\mathbf{x}_i \sim p(\mathbf{x}|\pmb{\theta}_i)$ , to obtain parameter-simulation pairs,  $\Theta, X = \{\pmb{\theta}_i, \mathbf{x}_i\}_{i=1\dots N}$ . Next, a dataset of target data points,  $X_{\mathrm{target}}$ , is constructed by concatenating in the batch-dimension: 1)  $X$ , 2) a random subset of  $X$  augmented with Gaussian noise with a specified variance, i.e.,  $\mathbf{x}_i + \epsilon_i, \epsilon_i \sim \mathcal{N}(0, \sigma^2\mathbf{I})$ , and 3) any number of real experimental observations the user wishes to include,  $X_{\mathrm{real}}$ ; in total,  $N_{\mathrm{target}} = N + N_{\mathrm{noised}} + N_{\mathrm{real}}$ . Note that  $X_{\mathrm{target}}$  does not technically need to include any simulated or real data, only 'realistic' targets in the neighborhood of the observed data that one eventually wants to perform inference for. Practically, 1) has already been simulated, and 2) and 3) does not require additional simulations while improving performance through noise augmentation and having access to real data targets. Finally, a pairwise distance matrix  $D$  is computed with elements  $d_{i,j} = d(\mathbf{x}_i, \mathbf{x}_t)$ ,  $\mathbf{x}_i \in X$ ,  $\mathbf{x}_t \in X_{\mathrm{target}}$ .  $D$  can either be pre-computed in full, or partially within the training loop.
Network optimization and convergence to GBI loss Given dataset  $\Theta, X, X_{\mathrm{target}}, D$ , a fully connected feed-forward deep neural network with weights  $\phi$  is trained to minimize the mean squared error loss:  $\frac{1}{N_{\mathrm{sim}}n_{\mathrm{targets}}}\sum_{i=1}^{N_{\mathrm{sim}}}\sum_{t=1}^{n_{\mathrm{target}}}\left(\mathrm{NN}_{\phi}(\pmb{\theta}_i,\mathbf{x}_t) - d(\mathbf{x}_i,\mathbf{x}_t)\right)^2$ . In other words, for a parameter configuration  $\pmb{\theta}_i$  and a target data point  $\mathbf{x}_t$ , the network is trained to predict the distance  $d_{i,t}$  between the corresponding single (stochastic) simulation  $\mathbf{x}_i$  and the target  $\mathbf{x}_t$ . In every training epoch,  $n_{\mathrm{target}}$  target simulations (usually 1-10, compared to  $N_{\mathrm{target}} > 10000$ ) are randomly sub-sampled per  $\pmb{\theta}_i$ , drastically reducing training time and allowing the relevant  $d_{i,t}$  to be computed on the fly.
Note that we do not wish to accurately predict the distance  $d(\mathbf{x}_i, \mathbf{x}_t)$  in data-space for individual simulations  $\mathbf{x}_i$ , but rather the loss  $\ell(\boldsymbol{\theta}_i; \mathbf{x}_t)$ . Conveniently, with mean squared error as the objective function for network training, for a fixed pair of  $\boldsymbol{\theta}_i, \mathbf{x}_t$ , the trained network predicts the mean distance,  $\mathbb{E}_{p(\mathbf{x}|\boldsymbol{\theta}_i)}[d(\mathbf{x}, \mathbf{x}_t)]$ , precisely the loss function we target in Eq.2 (proof in Appendix).
A4.2 Description of benchmark tasks, distance and cost functions
A4.2.1 Uniform 1D
A one-dimensional task with uniform noise to illustrate how expected distance from observation can be used as a cost function for inference, especially in the case of model misspecification:
Prior  $\mathcal{U}(-1.5,1.5)$
Simulator  $\mathbf{x}|\pmb{\theta} = g(z) + \epsilon$ , where  $\epsilon \sim \mathcal{U}(-0.25, 0.25)$ ,  $z = 0.8 \times (\pmb{\theta} + 0.25)$ , and  $x_{\mathrm{noiseless}} = g(z) = 0.1627 + 0.9073z - 1.2197z^2 - 1.4639z^3 + 1.4381z^4$ .
Dimensionality  $\pmb {\theta}\in \mathbb{R}^1,\mathbf{x}\in \mathbb{R}^1$
Cost function  $d(\mathbf{x},\mathbf{x}_o)$  : MSE;  $p(\mathbf{x}|\boldsymbol {\theta})$  computed exactly given uniform noise. For obtaining the true cost, Eq. 2 is analytically integrated over  $x_{\mathrm{noise}}\pm 0.25$
Posterior Ground-truth GBI posterior samples are obtained via rejection sampling. We used the prior as proposal distribution.
A4.2.2 Two Moons
A two-dimensional task with a posteriori that exhibits both global (bimodality) and local (crescent shape) structure to illustrate how algorithms deal with multimodality:
Prior  $\mathcal{U}(-1,1)$
$$
\begin{array}{l l} \textbf {S i m u l a t o r} & \quad \boldsymbol {x} | \boldsymbol {\theta} = \left[ \begin{array}{c} r \cos (\alpha) + 0. 2 5 \\ r \sin (\alpha) \end{array} \right] + \left[ \begin{array}{c} - | \theta_ {1} + \theta_ {2} | / \sqrt {2} \\ (- \theta_ {1} + \theta_ {2}) / \sqrt {2} \end{array} \right], \text {w h e r e} \alpha \sim \mathcal {U} (- \pi / 2, \pi / 2), \\ & \text {a n d} r \sim \mathcal {N} (0. 1, 0. 0 1 ^ {2}) \end{array}
$$
Dimensionality  $\theta \in \mathbb{R}^2, \mathbf{x} \in \mathbb{R}^2$
Cost function  $d(\mathbf{x},\mathbf{x}_o)$ : MSE; and  $p(\mathbf{x}|\boldsymbol {\theta})$  computed exactly. To obtain the true cost, Eq. 2 is numerically integrated over a 2D grid of  $d\mathbf{x}$  with 500 equal bins in both dimensions, where  $x^{(1)}\in [-1.2,0.4],x^{(2)}\in [-1.6,1.6]$ .
Posterior Ground-truth GBI posterior samples are obtained via rejection sampling with the prior as proposal distribution.
References [24, 8]
A4.2.3 Linear Gaussian
Inference of the mean of a 10-d Gaussian model, in which the covariance is fixed. The (conjugate) prior is Gaussian:
Prior  $\mathcal{N}(\mathbf{0}, 0.1 \odot \mathbf{I})$
Simulator  $\mathbf{x}|\boldsymbol {\theta}\sim \mathcal{N}(\mathbf{x}|\mathbf{m}_{\boldsymbol{\theta}} = \boldsymbol {\theta},\mathbf{S} = 0.1\odot \mathbf{I})$
Dimensionality  $\pmb {\theta}\in \mathbb{R}^{10},\mathbf{x}\in \mathbb{R}^{10}$
Cost function  $d(\mathbf{x},\mathbf{x}_o)$  : MSE; and  $p(\mathbf{x}|\boldsymbol {\theta})$  computed exactly. The true cost (Eq. 2) can be computed analytically.
Posterior Ground-truth GBI posterior samples are obtained following the procedure in Lueckmann et al. [24]: We first ran MCMC to generate 10k samples from the GBI posterior. We then trained a neural spline flow on these 10k samples and, finally, used the trained flow as proposal distribution for rejection sampling.
A4.2.4 Gaussian Mixture
The single-trial version of this task is common in the ABC literature. It consists of inferring the common mean of a mixture of two two-dimensional Gaussian distributions, one with much broader covariance than the other. In this study, we used this task to infer the common mean of the two distributions from five i.i.d. simulations:
Prior  $\mathcal{U}(-10,10)$
Simulator  $\mathbf{x}|\pmb{\theta}\sim 0.5\mathcal{N}(\mathbf{x}|\mathbf{m}_{\pmb{\theta}} = \pmb {\theta},\mathbf{S} = \mathbf{I}) + 0.5\mathcal{N}(\mathbf{x}|\mathbf{m}_{\pmb{\theta}} = \pmb {\theta},\mathbf{S} = 0.01\odot \mathbf{I})$
Cost function  $d(\mathbf{x},\mathbf{x}_o)$  :  $\mathrm{MMD}^2$  . The true cost (Eq. 2) is obtained by integrating the distance function on a grid for every trial independently and multiplying over the trials.
Posterior Ground-truth GBI posterior samples are obtained via rejection sampling. As proposal, we used a Normal distribution centered around the ground truth parameter and with variance  $\frac{50}{\beta}$ .
References [24, 53]
A4.3 Training, inference, and evaluation for benchmark tasks
ACE For the benchmark tasks, the cost estimation network is a residual network with 3 hidden layers of 64 units [54]. The training dataset was split 90:10 into training and validation sets, with  $n_{\text{target}} = 2$  and  $5 \times t$  randomly sampled each epoch for evaluating training and validation loss, respectively. We used a batchsize of 500, i.e.,  $500 \theta, 2 \times t$ , for 1000 cost targets per training batch. Networks usually converge within 500 epochs, with 100 epochs of non-decreasing validation loss as the convergence criterion. For inference, we ran multi-chain slice sampling with 100 chains to sample the potential function. For the Gaussian Mixture task with 5 i.d.d. samples per simulation/observation, we appended to the cost estimation network a fully connected, permutation-invariant embedding network [55] (2 layers, 100 units each), which preprocessed the i.d.d. datapoints into a single 20-dimensional vector (but was trained end-to-end).
Kernel ABC We compared our algorithm with a version of kernel ABC on the benchmark tasks. For kernel ABC, we accepted samples (from the fixed set of 10000 prior simulations) with probability  $\exp(-\beta \cdot d(\mathbf{x}, \mathbf{x}_o))$ , where  $d(\cdot, \cdot)$  is the distance function. In many cases, and especially for large  $\beta$ , this yielded very few or no accepted samples. We, therefore, reduced  $\beta$  until at least 50 samples were accepted.
NPE We used the implementation in the sbi toolbox [56], with neural spline flow [57] as density estimator (five transformation layers) to approximate the posterior. For tasks with bounded priors (all except Linear Gaussian), we appended a sigmoidal transformation to the flow such that its support matches the support of the prior [58]. For the Gaussian Mixture task, the same permutation-invariant embedding net was used as for ACE. Posterior samples were directly obtained from the trained flow. All other hyperparameters were the same as in the sbi toolbox, version 0.19.2 [56].
NLE and tempered NLE We used the implementation in the sbi toolbox [56], with neural spline flow [57] as density estimator (five transformation layers) to approximate the likelihood. For the Gaussian Mixture task with 5 i.d. simulations per sample/observation, they were split up as 5 independent simulations with the same  $\theta$  repeated 5 times, effectively having 5 times the training simulation budget. We ran multi-chain slice sampling with 100 chains to sample the potential function as the sum of log-prior probability and flow-approximated log-likelihood. Tempered NLE uses the same learned likelihood estimator, but the log-likelihood term in the potential function is scaled by  $\beta$  during MCMC sampling (same values as used for ACE). All other hyperparameters were the same as in the sbi toolbox, version 0.19.2 [56].
Noise augmentation For each task, we randomly subsampled 100 of all simulated datapoints  $(\mathbf{x})$  and added Gaussian noise with zero-mean and standard deviation equaling two times the standard deviation of all prior predictions, i.e.,  $\mathcal{N}(\mathbf{0}, (2\sigma_x)^2\mathbf{I})$ . We additionally varied noise amplitude  $(0, 2\sigma_x, 5\sigma_x)$  and found little to no effect on benchmark performance (see Appendix A3).
Synthetic misspecified observations For the Uniform 1D, 2 Moons, and Linear Gaussian task, we generated an additional 100,000 prior predictive simulations and defined the prior predictive bound as the minimum and maximum (of each dimension) of those simulations and  $\sigma_{x}$  as their standard deviation. Then, to create 20 misspecified observations, we generated 20 model-simulations and added Gaussian noise with zero-mean and standard deviation equaling  $0.5\sigma_{x}$  (i.e.,  $\epsilon \sim \mathcal{N}(\mathbf{0}, (0.5\sigma_x)^2\mathbf{I}))$  to the 20 observations iteratively until they are outside of the prior predictive bounds in every dimension. For the Gaussian Mixture task, we replaced the second Gaussian above by  $\mathcal{N}(12.5\times \mathrm{sign}(\theta),0.5^2\mathbf{I})$  i.e. displacing the mean to the corner of the quadrant whose signs match  $\pmb{\theta}$ , and slightly increasing the variance.
Metrics For the metrics in Fig. 3, in columns 1, 2, and 4 and all corresponding figures in the Appendix, results were aggregated across the 10 well-specified and misspecified samples separately, where marker and error bars represent mean and standard deviation over 10 observations. Columns 1 and 2 show average distance between observation and 5000 posterior predictive simulations from each algorithm (except ABC, for which there were 50 posterior samples). Column 3 shows, for each of the 10 well-specified observations and  $3\beta$  values, 3 random posterior samples for which we compare the ACE-estimated and ground-truth cost, i.e.,  $10\times 3\times 3 = 90$  points are shown for each
task. Column 4 shows C2ST score between 5000 ground-truth GBI posterior samples and ACE GBI posterior samples.
A4.4 Training procedure for Hodgkin-Huxley model
For NPE, we used the implementation in the sbi toolbox [56]. As density estimator, we used a neural spline flow [57] with five transformation layers. We used a batchsize of 5,000. We appended a sigmoidal transformation to the flow such that its support matches the support of the prior [58]. All other hyperparameters were the same as in the sbi toolbox, version 0.19.2 [56].
For ACE, we used a residual neural network [54] with 7 layers and 100 hidden units each. We used  $10\%$  of all simulations as held-out validation set and stopped training when the validation set did not decrease for 100 epochs. We used a batchsize of 5000. For sampling from the generalized posterior, we ran multi-chain slice sampling with 100 chains. As distance function, we used mean-absolute-error, where the distance in each summary statistic was z-scored with the standard deviation of the prior predictions. To generate  $\mathbf{x}_t$ , we used all simulations which generated between 5 and 40 spikes (since any reasonable experimental recording would fall within this range) and we appended 1000 augmented simulations to which we added Gaussian noise with two times the standard deviation of prior predictions that produce between 5 and 40 spikes. We did not use the experimental recordings from the Allen Cell Types database in  $\mathbf{x}_t$ .
A5 Hodgkin-Huxley model
We used the same model as Gonçalves et al. [10], which follows the model proposed in Pospischil et al. [34]. Briefly, the model contains four types of conductances (sodium, delayed-rectifier potassium, slow voltage-dependent potassium, and leak) and has a total of eight parameters that generate a time series which we reduce to seven summary statistics (spike count, mean resting potential, standard deviation of the resting potential, and the first four voltage moments mean, standard deviation, skew, curtosis, same as in Gonçalves et al. [10]).
A6 Supplementary figures
Figure A1: Benchmark performance for 'seen' observations Panels are the same as in Fig. 3, but for the  $20x_{o}$  that were included in the target dataset during training.
Figure A2: Cost estimation accuracy for all observations Columns are same as 3rd column in Fig. 3, but for all combinations of (unseen, seen) and (well-specified, misspecified) observations (10 each, 40 total) for each task. 1st column here is identical to Fig. 3 3rd column.
Figure A3: Benchmark performance for ACE trained with various augmentation noise levels. Results based on 10k training budget plus 100 noise augmented samples with varying noise amplitudes (blue, orange, green), as well as removing data augmentation altogether (i.e., no noise augmented simulated nor observed data, red). Overall, changing  $\sigma$  does not significantly impact performance compared to the original results ( $2\sigma_{x}$ , blue).
Figure A4: Benchmark performance with simulation budget of 200 Panels are the same as in Fig. 3, but all algorithms trained with simulation budget of 200.
Figure A5: Benchmark performance with simulation budget of 1000 Panels are the same as in Fig. 3, but all algorithms trained with simulation budget of 1000.
Figure A6: NPE performance with 1 million simulations. Panels are the same as in Fig. 4, but NPE was run with 1 million simulations. ACE is still run with 100K simulations and, thus, the ACE data is the same as in Fig. 4.
Figure A7: **Cornerplot of posterior distributions.** Diagonals are 1D-margins, upper diagonals are 2D-margins (68th-percentile contour from 5000 posterior samples). Orange: NPE with 100K simulations. Shades of blue: ACE with 100K simulations with  $\beta = 25$  (light blue),  $\beta = 50$  (medium blue),  $\beta = 100$  (dark blue).
Figure A8: Posterior predictive samples of ACE (with 100K simulations) with  $\beta = 25$ . Top row (black): 10 experimental recordings from the Allen Cell Types database. Below: nine predictive samples given each of the ten observations.
Figure A9: Posterior predictive samples of ACE (with 100K simulations) with  $\beta = 50$ . Top row (black): 10 experimental recordings from the Allen Cell Types database. Below: Nine predictive samples given each of the ten observations.