SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the context of this paper, we consider two instances of IPMs  $D(p^{\star}(\mathbf{x}), p(\mathbf{x}|\boldsymbol{\xi}))$  between the data generating distribution  $p^{\star}(\mathbf{x})$  and the likelihood  $p(\mathbf{x}|\boldsymbol{\xi})$  — the maximum mean discrepancy MMD and the squared 2-Wasserstein distance  $\mathcal{W}_2$ . To simplify the notation, we formulate the discrepancy between the pdfs  $p(\mathbf{x})$  and  $q(\mathbf{x})$  whose empirical probability distributions are denoted by  $\tilde{p}(\mathbf{x}) = 1/N\sum_{i=1}^{N}\delta_{\mathbf{x}_i}(\mathbf{x})$  and  $\tilde{q}(\mathbf{y}) = 1/M\sum_{j=1}^{M}\delta_{\mathbf{y}_j}(\mathbf{y})$ . Furthermore, we denote the cost between individual samples by  $c(\mathbf{x},\mathbf{y})$ .
Maximum mean discrepancy The MMD [25] can be formulated with respect to an evaluation kernel  $k(\mathbf{x},\mathbf{y})$  as the sum of three terms
$$
\operatorname {M M D} ^ {2} (p, q) = \underset { \begin{array}{c} \mathbf {x} \sim p (\mathbf {x}) \\ \mathbf {y} \sim p (\mathbf {y}) \end{array} } {\mathbb {E}} [ k (\mathbf {x}, \mathbf {y}) ] - 2 \underset { \begin{array}{c} \mathbf {x} \sim p (\mathbf {x}) \\ \mathbf {y} \sim q (\mathbf {y}) \end{array} } {\mathbb {E}} [ k (\mathbf {x}, \mathbf {y}) ] + \underset { \begin{array}{c} \mathbf {x} \sim q (\mathbf {x}) \\ \mathbf {y} \sim q (\mathbf {y}) \end{array} } {\mathbb {E}} [ k (\mathbf {x}, \mathbf {y}) ]. \tag {34}
$$
As the expectations are generally intractable, an unbiased estimate of MMD based on samples drawn from  $p$  and  $q$  is used [25]
$$
\mathrm {M M D} ^ {2} (\tilde {p}, \tilde {q}) \approx \frac {1}{N (N - 1)} \sum_ {i \neq i ^ {\prime}} ^ {N} k \left(\mathbf {x} _ {i}, \mathbf {x} _ {i ^ {\prime}}\right) - \frac {2}{N M} \sum_ {i, j = 1} ^ {N, M} k \left(\mathbf {x} _ {i}, \mathbf {y} _ {j}\right) + \frac {1}{M (M - 1)} \sum_ {j \neq j ^ {\prime}} ^ {M} k \left(\mathbf {x} _ {j}, \mathbf {x} _ {j ^ {\prime}}\right). \tag {35}
$$
Here,  $\mathbf{x}_i$  and  $\mathbf{y}_j$  represent samples drawn from the sampling distributions  $\mathbf{x}_{1:N} \sim p(\mathbf{x})$  and  $\mathbf{y}_{1:M} \sim q(\mathbf{y})$ . In this paper, a Gaussian kernel  $\exp(-1/(2\ell)c(\mathbf{x},\mathbf{y}))$  with bandwidth  $\ell$  is employed. The bandwidth is known to be very sensitive, which is why the kernel is evaluated over a variety of bandwidths  $\ell = \{1, 10, 20, 40, 80, 100, 130, 200, 400, 800, 1000\}$  by summing over the bandwidths  $k(\mathbf{x}_i,\mathbf{y}_j) = \sum_{\ell} k_{\ell}(x_i,y_j)$  [16].
Wasserstein distance. In the experiments, we consider the squared 2-Wasserstein, which can be formulated as the solution to the optimal transport problem
$$
\mathcal {W} _ {2} ^ {2} (p, q) = \inf  _ {\gamma} \int c (\mathbf {x}, \mathbf {y}) \gamma (\mathbf {x}, \mathbf {y}) d \mathbf {x} d \mathbf {y}. \tag {36}
$$
The problem is also known as the Kantorovich problem, searching for the optimal coupling  $\gamma \in \Gamma(p,q)$  in the set of joint distributions that admit  $p$  and  $q$  through marginalization. For the empirical measures  $\tilde{p}$  and  $\tilde{q}$ , the Kantorovich problem can be formulated as a linear program
$$
\mathcal {W} _ {1} = \min  _ {\mathbf {P} \in \mathbf {U}} \sum_ {i, j} P _ {i j} C _ {i j}. \tag {37}
$$
Here,  $C_{ij} = c(\mathbf{x}_i,\mathbf{y}_j)$  is the cost matrix containing the pairwise comparisons between the samples drawn from  $\tilde{p} (\mathbf{x})$  and  $\tilde{q} (\mathbf{y})$ . The linear program searches for the optimal coupling matrix  $\mathbf{P}$  among the set of doubly stochastic matrices  $\mathbf{U} = \{\mathbf{P}\in \mathbb{R}_{+}^{N\times M}:\mathbf{P}\mathbf{1}_{M} = 1 / N\mathbf{1}_{N},\mathbf{P}^{\intercal}\mathbf{1}_{N} = 1 / M\mathbf{1}_{M}\}$ . Peyré and Cuturi [44] show that introducing an entropy regularization term  $\varepsilon H(\mathbf{P})$  to the objective (37) leads to an iterative scheme that can be solved with the Sinkhorn algorithm [46]. The iterative procedure enables parallelization on hardware accelerators to efficiently solve the optimal transport problem. Furthermore, it can be seen that the 2-Wasserstein distance can be recovered in the limit  $\varepsilon \rightarrow 0$ . We leverage the JAX library, OTT [9], to approximate  $\mathcal{W}_2^2$  with Sinkhorn iterations for the computations.
C.2 Gaussian location
The Gaussian location model is a 10-dimensional Gaussian model. The ten dimensional parameters  $\pmb {\xi}\in [-1,1]^{10}$  define the means of the model  $\mathcal{N}(\mathbf{x}|\boldsymbol {\mu} = \boldsymbol {\xi},\boldsymbol {\Sigma} = 0.1\mathbf{I})$  . We choose a Gaussian prior  $p(\pmb {\xi}) = \mathcal{N}(\pmb {\xi}|\mathbf{0},0.1\mathbf{I})$  for which the posterior can be recovered in closed form. The ground-truth parameter is sampled uniformly within the posterior support  $\pmb{\xi}^{\mathrm{gt}}\sim \mathcal{U}(-\mathbf{1},\mathbf{1})$
Figure C.2: Slice of the posterior through the  $\xi_{1} - \xi_{2}$  plane. The upper row shows experiments conducted on  $N = 2$  reference observations. The lower row shows the approximate posteriors for  $N = 100$  reference observations. The dotted line represents the ground truth parameter  $\pmb{\xi}^{(\mathrm{gt})}$  that was used to generate  $\mathbf{x}_{1:N}^{\star}$ . All approaches show that the posterior becomes denser when conditioned on more data.
C.3 Gaussian mixture model
The task is to infer the mean parameters of a two-dimensional multivariate Gaussian mixture model [47]
$$
p (\mathbf {x} | \boldsymbol {\xi}) = \mathcal {N} (\mathbf {x} | \boldsymbol {\mu} = \boldsymbol {\xi}, \boldsymbol {\Sigma} = \mathbf {I}) + \mathcal {N} (\mathbf {x} | \boldsymbol {\mu} = \boldsymbol {\xi}, \boldsymbol {\Sigma} = 0. 0 1 \mathbf {I}) \tag {38}
$$
The two-dimensional observation space represents samples from the Gaussian mixture model. We assume a uniform prior  $p(\pmb{\xi}) = \mathcal{U}(-\mathbf{10}, \mathbf{10})$ .
C.4 Simple-likelihood complex-posterior
The Simple-Likelihood Complex-Posterior (SLCP) task consists of a 5-dimensional parameter space  $\Xi \in [-3,3]^5$  with a uniform prior  $\mathcal{U}(-3,3)$ . The ground-truth parameter, from which the reference observations are generated, is set to  $\pmb{\xi}^{(\mathrm{gt})} = (0.7,1.5, - 1.0, - 0.9,0.6)^{\mathrm{T}}$ . The observations represent four samples from a 2-dimensional Gaussian distribution
$$
\mathbf {x} = \left[ \mathbf {x} _ {1}, \mathbf {x} _ {2}, \mathbf {x} _ {3}, \mathbf {x} _ {4}, \right] ^ {\intercal}, \quad \mathbf {x} _ {i} = \mathcal {N} \left(\mathbf {x} _ {i}, \boldsymbol {\mu} (\boldsymbol {\xi}), \boldsymbol {\Sigma} (\boldsymbol {\xi})\right). \tag {39}
$$
For further information, we refer to the SBI benchmarking paper from Lueckmann et al. [32].
C.5 SIR
The SIR model is an epidemiological time-series model that models the spreading of a disease. The name derives from the three states, (i) susceptible, (ii) infectious, (iii) recovered, that an individual can be in. The parameters of the dynamics model are the contact rate  $\beta$  and the mean recovery rate  $\gamma$
$$
\boldsymbol {\xi} = \left[ \begin{array}{l} \beta \\ \gamma \end{array} \right] \in \left[ \begin{array}{l} (0, 2 ] \\ (0, 0. 5 ] \end{array} \right]. \tag {40}
$$
The prior is a log-normal distribution over  $\beta$  and  $\gamma$
$$
\beta \sim \operatorname {L o g N o r m a l} (\log (0. 4), 0. 5) \tag {41}
$$
$$
\gamma \sim \operatorname {L o g N o r m a l} (\log (0. 1 2 5), 0. 2) \tag {42}
$$
We rollout the dynamics over 160 timesteps and evaluate the simulation at 20 equidistant time-steps by taking a sample from the binomial,  $x_{i} \sim \mathrm{Binom}(1000, I_{i} / N)$ . Here  $N$  denotes the total population at the start of the simulation. See Lueckmann et al. [32] for details on the dynamics of the SIR model.
C.6 The Furuta pendulum
This inverted double pendulum can be described by the angular deflection  $[\theta_r, \theta_p]$  of the rods w.r.t. their equilibrium position  $[0, 0]$ . The equations of motion can be derived by formulating the Euler-Lagrange equation [37]:
$$
\begin{array}{l} \left[ \begin{array}{c} - d _ {r} \dot {\theta} _ {r} + \tau \\ - d _ {p} \dot {\theta} _ {p} \end{array} \right] = \left[ \begin{array}{c c} \frac {1}{1 2} m _ {r} l _ {r} ^ {2} + m _ {p} l _ {r} ^ {2} + \frac {1}{4} m _ {p} l _ {p} ^ {2} \sin^ {2} \theta_ {p} & \frac {1}{2} m _ {p} l _ {p} l _ {r} \cos \theta_ {p} \\ \frac {1}{2} m _ {p} l _ {p} l _ {r} \cos \theta_ {p} & \frac {1}{3} m _ {p} l _ {p} ^ {2} \end{array} \right] \left[ \begin{array}{c} \ddot {\theta} _ {r} \\ \ddot {\theta} _ {p} \end{array} \right] \\ + \left[ \begin{array}{c} \frac {1}{4} m _ {p} l _ {p} ^ {2} \sin 2 \theta_ {p} \dot {\theta} _ {r} \dot {\theta} _ {p} - \frac {1}{2} m _ {p} l _ {p} l _ {r} \sin \theta_ {p} \dot {\theta} _ {p} ^ {2} \\ - \frac {1}{8} m _ {p} l _ {p} ^ {2} \sin 2 \theta_ {p} \dot {\theta} _ {r} ^ {2} + \frac {1}{2} m _ {p} l _ {p} g \sin \theta_ {p} \end{array} \right]. \tag {43} \\ \end{array}
$$
Here, the mild assumption is made that the pole length is significantly greater than its diameter for which the moments of inertia of the poles around their pivot are  $J_{i} = 1 / 3 m_{i} l_{i}^{2}$ ,  $i \in \{r, p\}$ . The mass matrix contains entries from the translatory and rotational movement of the two poles. As the reference coordinate systems are constantly rotating w.r.t. the basis coordinate system, Coriolis forces occur. They are complemented by gravitation which works on the rotational pole. The left-hand side considers damping in the joints, represented by the damping coefficients  $d_{r}$  and  $d_{p}$ , and the torque  $\tau$  which is applied from a servo motor. For this paper, we omit external forces, i.e.,  $\tau = 0 \mathrm{Nm}$ . The Furuta pendulum is set into motion by perturbing the initial state around its unstable equilibrium.
For the system identification tasks we select the five system parameters
$$
\boldsymbol {\xi} = \left[ \begin{array}{c} g \\ l _ {r} \\ m _ {r} \\ l _ {p} \\ m _ {p} \end{array} \right] \in \left[ \begin{array}{c} [ 9, 1 1 ] \\ [ 0. 0 8, 0. 0 9 ] \\ [ 0. 0 8, 0. 1 ] \\ [ 0. 1 2, 0. 1 3 5 ] \\ [ 0. 0 2, 0. 0 3 ] \end{array} \right]; \qquad \boldsymbol {\xi} ^ {\mathrm {g t}} = \left[ \begin{array}{c} 9. 8 1 \\ 0. 0 8 5 \\ 0. 0 9 5 \\ 0. 1 2 9 \\ 0. 0 2 4 \end{array} \right]
$$
with a uniform prior on the predefined ranges.
Table C.3: Hyper-parameter settings of the SBI methods as used for the experiments in Section 4. Forward slashes symbolize layers of a neural network.
Parameter	Value
PLI (Ours)
Likelihood kernel	Exponential Kernel
Trust-region threshold ε	0.5
Model	Neural Spline Flow (NSF)
Bijector	Rational Quadratic Spline with param size D
# Bins	10
Conditioning MLP	input dim / 50 / 50 / 50 / D
# Birectors / Transforms	5
Base distribution	N(0,1)
Learning rate	1 × 10-5
Epochs	20
Train samples per iteration	5000
Batch size	125
PMC-ABC [31]
Likelihood kernel	Uniform Kernel
Likelihood update	α-Quantile, α = 0.1 (see Table A.1)
α	0.1
Reverse transition kernel Lt	reverse PMC kernel (see Table B.2)
Particles	1000
Epochs	200
Perturbation kernel	GMM with 5 components
APT [24]
Model	Conditional Neural Spline Flow (NSF)
Bijector	Rational Quadratic Spline with param size D
# Bins	10
# Birectors / Transforms	5
Conditioning MLP	input dim / 32 / 32 / 32 / D
Base distribution	N(0,1)
# Atoms	10
Learning rate	1 × 10-5
Epochs	20
Train samples per epoch	5000
Batch size	500
Figure C.3: Results of posterior inference on the SLCP task with  $N = 2$  reference observations. The unimodal distribution of the parameters  $\xi_{1}$  and  $\xi_{2}$  are depicted well by all approaches. On the contrary, the multi-modality is only represented properly by the APT posterior (Reference, MMD-PLI, APT, MMD-ABC).
Figure C.4: Results of posterior inference on the SLCP task with  $N = 100$  reference observations. Compared to the posterior given  $N = 2$  observations (Figure C.3), the posterior (Reference) is distributed tightly around distinct points. Here, MMD-PLI captures all modes of the posterior, MMD-ABC centers around a uni-mode, while APT cannot represent the multi-modality.
Figure C.5: Results of posterior inference on the Furuta pendulum with  $N = 2$  reference observations. All methods center around the ground truth parameter. - APT finds the expected correlations among the parameters while - MMD-PLI and - MMD-ABC remain more widespread.
Figure C.6: Results of posterior inference on the Furuta pendulum with  $N = 100$  reference observations. All models capture the ground-truth parameter well. In contrast to the  $N = 2$  setting (Figure C.5) MMD-PLI reveals pairwise correlations between the domain parameters, and MMD-ABC is less densely distributed. Note, that APT clusters outside of the ground-truth parameter.