SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} = \frac {\partial}{\partial x _ {n + 1}} \left(\sum_ {i = 0} ^ {d} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d} L _ {d, i} \cdot y (x _ {n + 1}) ^ {d + i}\right) \\ = \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i + 1} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p x _ {k} y ^ {\prime} (x _ {n + 1}) \cdot p ^ {i} x _ {k} ^ {i} (1 - p) ^ {d} L _ {d, i} y (x _ {n + 1}) ^ {d + i} \\ + \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} (1 - p) ^ {d} \cdot L _ {d, i} (d + i) y (x _ {n + 1}) ^ {d + i - 1} \cdot y ^ {\prime} (x _ {n + 1}) \\ = \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i + 1} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i + 1} x _ {k} ^ {i + 1} \cdot (1 - p) ^ {d + 1} y (x _ {n + 1}) ^ {2} \cdot L _ {d, i} y (x _ {n + 1}) ^ {d + i} \\ + \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d + 1} y (x _ {n + 1}) ^ {2} \cdot (d + i) L _ {d, i} y (x _ {n + 1}) ^ {d + i - 1} \\ = \sum_ {i = 0} ^ {d + 1} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d + 1} L _ {d, i - 1} y (x _ {n + 1}) ^ {d + i + 1} \\ + \sum_ {i = 0} ^ {d + 1} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d + 1} (d + i) L _ {d, i} y (x _ {n + 1}) ^ {d + i + 1} \\ = \sum_ {i = 0} ^ {d + 1} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d + 1} ((d + i) L _ {d, i} + L _ {d, i - 1}) y (x _ {n + 1}) ^ {d + i + 1} \\ = \sum_ {i = 0} ^ {d + 1} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d + 1} L _ {d + 1, i} y (x _ {n + 1}) ^ {d + i + 1} \\ \end{array}
$$
$$
\begin{array}{l} [   [ \text {o b s e r v e} d \sim \text {N e g B i n o m i a l} (X _ {k}, p) ]   ] _ {\mathbf {g f}} (G) \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} [   [ X _ {n + 1} \sim \operatorname {N e g B i n o m i a l} (X _ {k}, p) ]   ] _ {\mathbf {g f}} (G) (x _ {1}, \dots , x _ {n + 1}) \Bigg | _ {x _ {n + 1} = 0} \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} G (\boldsymbol {x} [ k \mapsto \frac {x _ {k} \cdot p}{1 - (1 - p) x _ {n + 1}} ]) \Bigg | _ {x _ {n + 1} = 0} \\ = \frac {1}{d !} \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p \cdot x _ {k} ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d} L _ {d, i} \\ \end{array}
$$
Theorem C.4. Observing from compound Bernoulli distributions can be implemented without introducing a new variable:
$$
\llbracket \text {o b s e r v e} d \sim \operatorname {B e r n o u l l i} (X _ {k}) \rrbracket_ {\mathbf {g f}} (G) = \left\{ \begin{array}{l l} G (\boldsymbol {x}) - x _ {k} \partial_ {j} G (\boldsymbol {x}) & d = 0 \\ x _ {k} \cdot \partial_ {j} G (\boldsymbol {x}) & d = 1 \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
Proof. We argue as follows:
$$
\begin{array}{l} [   [ \text {o b s e r v e} d \sim \operatorname {B e r n o u l l i} (X _ {k}) ]   ] _ {\mathrm {g f}} (G) (\boldsymbol {x}) \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} [   [ X _ {n + 1} \sim \mathsf {B e r n o u l l i} (X _ {k}) ]   ] _ {\mathbf {g f}} (G) (x _ {1}, \dots , x _ {n + 1}) \Bigg | _ {x _ {n + 1} = 0} \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} \left(G (\boldsymbol {x}) + x _ {k} \left(x _ {n + 1} - 1\right) \cdot \partial_ {k} G (\boldsymbol {x})\right) \Bigg | _ {x _ {n + 1} = 0} \\ \end{array}
$$
If  $d = 0$ , the right-hand side is  $G(\pmb{x}) - x_k \partial_k G(\pmb{x})$ . If  $d = 1$ , the right-hand side is  $x_k \partial_k G(\pmb{x})$ . For larger  $d \geq 2$ , the  $d$ -th derivative vanishes, which completes the proof.
C.3 Analysis of the running time
If the program makes observations (or compares variables with constants)  $d_{1}, \ldots, d_{m}$ , the running time of the program will depend on these constants. Indeed, each observation of (or comparison of a variable with)  $d_{i}$  requires the computation of the  $d_{i}$ -th partial derivative of the generating function. In our Taylor polynomial approach, this means that the highest order of Taylor polynomials computed will be  $d := \sum_{i=1}^{m} d_{i}$ . If the program has  $n$  variables, storing the coefficients of this polynomial alone requires  $O(d^{n})$  space. This can be reduced if the generating function is a low-degree polynomial, in which case we do not store the zero coefficients of the terms of higher order.
Naively, the time complexity of multiplying two such Taylor polynomials is  $O(d^{2n})$  and that of composing them is  $O(d^{3n})$ . However, we can do better by exploiting the fact that the polynomials are sparse. In the generating function semantics, we never multiply two polynomials with  $n$  variables. One of them will have at most two variables. This brings the running time for multiplications down to  $O(d^{n+2})$ . Furthermore, the GF semantics has the property that when composing two generating functions, we can substitute variables one by one instead of simultaneously, bringing the running time down to  $O(n \cdot d^{2n+1})$ . Even more, at most one variable is substituted and the substituted terms have at most two variables, which brings the running time down to  $O(d^{n+3})$ .
Theorem C.5. Our exact inference method can evaluate the  $k$ -th derivative of a generating function in  $O(s \cdot d^{n+3})$ , and  $O(sd^3)$  for  $n = 1$ , where  $s$  is the number of statements,  $d$  (for data) is  $k$  plus the sum of all values that are observed or compared against in the program, and  $n$  is the number of variables of the program.
Proof. The above arguments already analyze the bottlenecks of the algorithm. But for completeness' sake, we consider all operations being performed on generating functions and their Taylor expansions (cf. Appendix C.1).
The maximum order of differentiation needed is  $d$  and since there are  $n$  variables, the maximum number of Taylor coefficients that need to be stored is  $O(d^{n})$ . Addition of two such Taylor polynomials and scalar multiplications trivially take  $O(d^{n})$  time. Since in the GF semantics, a GF is only ever multiplied by one with at most two variables, the multiplications take  $O(d^{n + 2})$  time. Division only happens with a divisor of degree at most 1, which can also be implemented in  $O(d^{n + 1})$  time [13, Chapter 13]. Exponentiation  $\exp(\ldots)$ , logarithms  $\log(\ldots)$ , and powers  $(\ldots)^{m}$  are only performed on polynomials with one variable, where they can be implemented in  $O(d^{2})$  time [13, Table 13.2]. Finally, the only substitutions that are required are ones where a term with at most two variables is substituted for a variable, i.e. of the form  $p(\pmb{x}[k \mapsto q(x_1, x_2)])$  where  $p(\pmb{x}) = \sum_{|\alpha| \leq d} p_{\alpha} \pmb{x}^{\alpha}$ . Then
$$
p (\boldsymbol {x} [ k \mapsto q (x _ {1}, x _ {2}) ]) = p _ {0} (\boldsymbol {x}) + q (x _ {1}, x _ {2}) \cdot \left(p _ {1} (\boldsymbol {x}) + q (x _ {1}, x _ {2}) \cdot \left(p _ {2} (\boldsymbol {x}) + \dots\right)\right)
$$
where  $p_i(\pmb{x}) = \sum_{\alpha : \alpha_k = i} p_\alpha \pmb{x}^{\alpha[k \mapsto 0]}$ . This needs  $d$  multiplications of  $q(x_1, x_2)$  in 2 variables and a polynomial in  $n$  variables, each of which takes  $O(d^{n+2})$  time. In total, the substitution can be performed in  $O(d^{n+3})$  time. Finally, if there is only one variable, the substitution can be performed in  $O(d^3)$  time because  $q$  can only have one variable.
In summary, the input SGCL program has  $s$  statements, and each of them is translated to a bounded number of operations on generating functions. These operations are performed using Taylor expansions of degree at most  $d$ , each of which takes at most  $O(d^{n + 3})$  time (and at most  $O(d^3)$  time for  $n = 1$ ). So overall, the running time is  $O(s \cdot d^{n + 3})$  (and  $O(sd^3)$  for  $n = 1$ ).
Potential improvements of the running time Using various tricks, such as FFT, the running time of the composition of Taylor polynomials, i.e. substitution of polynomials, can be improved to  $O((d\log d)^{3n / 2})$  [2]. This is still exponential in the number of variables  $n$ , but would improve our asymptotic running time for  $n < 6$  program variables. In fact, we experimented with an implementation of multiplication and composition based on the fast Fourier transform (FFT), but discarded this option because it led to much greater numerical instabilities. It is possible that the running time could instead be improved with the first algorithm in [2], which relies on fast matrix multiplication instead
of FFT. In addition, it should often be possible to exploit independence structure in programs, so the GF can be factored into independent GFs for subsets of the variables. This is not implemented yet because it was not relevant for our benchmarks.
Performance in practice Note that, for many programs, the running time will be better than the worst case. Indeed, substitutions are the most expensive operation, if the substitute expression is of high degree. For some models, like the population model, the substitutions take the form of Theorem C.1, i.e.  $G(\pmb{x}[k \mapsto (1 - p)x_k])$ , which is of degree 1. Such a substitution can be performed in  $O(d^n)$  time, because one simply needs to multiply each coefficient by a power of  $(1 - p)$ . This explains why the population model is so fast in the experimental section (Section 5). In other cases as well, we can often make use of the fact that the polynomials have low degree to reduce the space needed to store the coefficients and to speed up the operations.
C.4 Numerical issues
If our implementation is used in floating-point mode, there is the possibility of rounding errors and numerical instabilities, which could invalidate the results. To enable users to detect this, we implemented an option in our tool to use interval arithmetic in the computation: instead of a single (rounded) number, it keeps track of a lower and upper bound on the correctly rounded result. If this interval is very wide, it indicates numerical instabilities.
Initially, we observed large numerical errors (caused by catastrophic cancellation) for probabilistic programs involving continuous distributions. The reason for this turned out to be the term  $\log(x)$  that occurs in the GF of the continuous distributions. The Taylor coefficients of  $\log(x)$  at points  $0 < z < 1$  are not well-behaved:
$$
\log (x) = \log (z) + \sum_ {n = 1} ^ {\infty} \frac {(- 1) ^ {n} z ^ {- n}}{n} (x - z) ^ {n}
$$
because the coefficients grow exponentially for  $z < 1$  (due to the term  $z^{-n}$ ) and the alternating sign exacerbates this problem by increasing the likelihood of catastrophic cancellations.
To fix this problem, we adopted a slightly modified representation. Instead of computing the Taylor coefficients of the GF  $G$  directly, we compute those of the function  $H(\boldsymbol{x}) \coloneqq G(\boldsymbol{x}^{\prime})$  where  $x_{i}^{\prime} \coloneqq x_{i}$  if  $X_{i}$  is discrete and  $x_{i}^{\prime} \coloneqq \exp(x_{i})$  if  $X_{i}$  is continuous, thus canceling the logarithm. (The reason we don't use  $x^{\prime} \coloneqq \exp(x)$  for all variables is that for discrete variables, we may need to evaluate at  $x_{i}^{\prime} = 0$ , corresponding to  $x_{i} = -\infty$ , which is impossible.) It is straightforward to adapt the GF semantics to this modified representation. Due to the technical nature of the adjustment (case analysis on whether an index corresponds to a discrete variable or not), we do not describe it in detail. With this modified representation, we avoid the catastrophic cancellations and obtain numerically stable results for programs using continuous distributions as well.
D Details on the Empirical Evaluation
All experiments were run on a laptop computer with a Intel® Core™ i5-8250U CPU @ 1.60GHz × 8 processor and 16 GiB of RAM, running Ubuntu 22.04.2 and took a few hours to complete overall. The code to run the benchmarks and instructions on how to reproduce our experiments are available in the supplementary material.
D.1 Comparison with exact inference
We manually patched the tools Dice, Prodigy, and PSI to measure and output the time taken exclusively for inference to ensure a fairer comparison and because the time a tool takes to startup and parsing times are not very interesting from a research perspective. We ran each tool 5 times in a row and took the minimum running time to reduce the effect of noise (garbage collection, background activity on
Note that this boils down to using the moment generating function (MGF)  $\mathbb{E}[\exp (x_iX_i)]$  for continuous variables  $X_{i}$  and the probability generating function (PGF)  $\mathbb{E}[x_i^{X_i}]$  for discrete variables  $X_{i}$ . In mathematics, the MGF is more often used for continuous distributions and the PGF more commonly for discrete ones. It is interesting to see that from a numerical perspective, this preference is confirmed.
Table 5: Summary of the benchmarks:  $n$  is the number of variables,  $o$  is the number of observations,  $d$  is the sum of all the observed values,i.e. the total order of derivatives that needs to be computed,  $s$  is the number of statements,and  $p$  the number of program paths.
Benchmark	n	o	d	s	p	continuous priors?
population model (Fig. 1a)	1	4	254	13	1	no
population (modified, Fig. 1b)	1	4	254	21	16	no
population (two types, Fig. 1c)	2	8	277	30	1	no
switchpoint (Fig. 2a)	2	109	188	12433	111	yes
mixture model (Figs. 2b and 2c)	2	218	188	329	\( 2^{109} \approx 6 \cdot 10^{32} \)	no
HMM (Fig. 2d)	3	60	51	152	\( 2^{30} \approx 10^9 \)	no
the computer). In rational mode, we ran Genfer with the option --rational and Dice with the option -wmc-type 1. For the "clinicalTrial" benchmark, Genfer required 400-bit floating-point numbers via --precision 400 because 64-bit numbers did not provide enough accuracy. For better performance, PSI was run with the flag --dp on finite discrete benchmarks.
The benchmarks were taken from the PSI paper [9, Table 1], excluding "HIV", "LinearRegression1", "TrueSkill", "AddFun/max", "AddFun/sum", and "LearningGaussian". While Genfer supports continuous priors, the excluded benchmarks use continuous distributions in other ways that are not supported (e.g. in observations). Note that Dice and Prodigy have no support for continuous distributions at all. These benchmarks were translated mostly manually to the tools' respective formats, and we checked that the outputs of the tools are consistent with each other.
D.2 Comparison with approximate inference
Experimental setup As explained in Section 5, we ran several of Anglican's inference algorithms. Each algorithm was run with a sampling budget of 1000 and 10000. The precise settings we used are the following:
- Importance Sampling (IS): has no settings
- Lightweight Metropolis Hastings (LMH): has no settings
- Random-Walk Metropolis Hastings (RMH): has two settings: the probability  $\alpha$  of using a local MCMC move and the spread  $\sigma$  of the local move. We used the default  $\alpha = 0.5$ ,  $\sigma = 1$  and another setting facilitating local moves with  $\alpha = 0.8$ ,  $\sigma = 2$ .
- Sequential Monte Carlo (SMC): number of particles  $\in$  {1, 1000}. The default is 1 and we also picked 1000 to see the effect of a larger number of particles while not increasing the running time too much.
- Particle Gibbs (PGibbs): number of particles  $\in$  {2, 1000}. The default is 2 and we also picked 1000 as the other setting for the same reason as for SMC.
- Interacting Particle MCMC (IPMCMC): it has the settings "number of particles per sweep", "number of nodes running SMC and CSMC", "number of nodes running CSMC", and "whether to return all particles (or just one)". We left the last three settings at their defaults (32 nodes, 16 nodes running CSMC, return all particles). For the number of nodes and the number of nodes we chose the default setting (2) and a higher one (1000) for the same reason as SMC.
A full list of Anglican's inference algorithms and their settings can be found at https://probprog.github.io/anglican/inference. Each inference algorithm was run 20 times on each benchmark and the running times were averaged to reduce noise. This took around an hour per benchmark.
D.3 Benchmarks with infinite support
A summary with important information on each benchmark can be found in Table 5, including some statistics about the probabilistic programs. We describe each benchmark in more detail in the following.
Population models The probabilistic program for the original population model from [26] is shown in Fig. 6a. This program can also be written without the extra variable New by writing the two
$N\coloneqq$  Poisson  $(\lambda_0)$ $N_{2}\sim \mathrm{Poisson}(\lambda_{0}^{(2)})$  .  $N\sim \mathrm{Binomial}(N,\delta);$ $N_{2} + \sim$  Binomial  $(N_{1},\gamma)$  .  $N_{1}\sim \mathrm{Binomial}(N_{1},\delta_{1})$  .  $N\sim \mathrm{Binomial}(N_2,\delta_2)$  .  $N_{1} + \sim \mathrm{Poisson}(\lambda_{1}^{1})$  .  $N_{2} + \sim \mathrm{Poisson}(\lambda_{1}^{2})$  .  $N\sim \mathrm{Binomial}(N,\delta);$  } else {  $N_{2} + \sim \mathrm{Poisson}(\lambda_{1}^{2})$  .  $N\coloneqq N e w + N;$  } observe  $y_{1}^{(1)}\sim \mathrm{Binomial}(N_{1},\rho)$  . observe  $y_{1}^{(2)}\sim \mathrm{Binomial}(N_{2},\rho)$  . ； ； (a) Original model from [26]. (b) Randomly modified arrival rate. (c) Two interacting populations.
Figure 6: The program code for the population model variations.
statements involving it as  $N + \sim$  Poisson( $\lambda_1$ ). Here we used the same parameter values as [26]:  $\delta = 0.2636$ ,  $\lambda = \Lambda \cdot (0.0257, 0.1163, 0.2104, 0.1504, 0.0428)$ , and  $\Lambda = 2000$  is the population size parameter. Note that the largest population size considered by [26] is  $\Lambda = 500$ . For the observation rate  $\rho$ , Winner and Sheldon [26] consider values from 0.05 to 0.95. For simplicity, we set it to  $\rho = 0.2$ . The  $y_k$  are  $k = 4$  simulated data points: (45, 98, 73, 38) for  $\Lambda = 2000$ .
The modified population example is shown in Fig. 6b, where the arrival rate is reduced to  $\lambda' := 0.1\lambda$  in the case of a natural disaster, which happens with probability 0.1. The rest of the model is the same.
The model of two interacting populations (multitype branching process) is programmed as shown in Fig. 6c, where  $\lambda^{(1)}\coloneqq 0.9\lambda$ ,  $\lambda^{(2)}\coloneqq 0.1\lambda$ ,  $\gamma \coloneqq 0.1$ . In this model, some individuals of the first kind (i.e. in  $N_{1}$ ) can turn into individuals of the second kind and are added to  $N_{2}$ . The other parameters of this model are the same as before. The data points are  $y^{(1)} = (35,83,78,58)$  and  $y^{(2)} = (3,6,10,4)$ .
Bayesian switchpoint analysis Bayesian switchpoint analysis is about detecting a change in the frequency of certain events over time. An example is the frequency of coal mining disasters in the United States from 1851 to 1962, as discussed in the PyMC3 tutorial [23]. We use the same model as in [23], and its probabilistic program is shown in Fig. 7a. This situation can be modeled with two Poisson distributions with parameters  $\Lambda_{1}$  (before the change) and  $\Lambda_{2}$  (after the change). Suppose we have observations  $y_{1},\ldots ,y_{n}$ , with  $y_{t}\sim \mathrm{Poisson}(\Lambda_{1})$  if  $t < T$  and  $y_{t}\sim \mathrm{Poisson}(\Lambda_{2})$  if  $t\geq T$ . The parameters  $\Lambda_1,\Lambda_2$  are given exponential priors Exponential(1). The change point  $T$  itself is assumed to be uniformly distributed:  $T\sim \mathrm{Uniform}\{1..n\}$ . The probabilistic program code is shown in Fig. 7a, where  $n = 111$  and  $y = (4,5,4,0,1,4,3,4,0,6,3,3,4,0,2,6,3,3,5,4,5,3,1,4,4,1,5,5,3,4,2,5,2,2,3,4,2,1,3,n/a,2,1,1,1,3,0,0,1,0,1,0,0,3,1,0,3,2,0,1,1,1,0,1,0,0,0,2,1,0,0,0,1,1,0,2,3,3,1,n/a,2,1,1,1,1,2,4,2,0,0,1,4,0,0,0,1,0,0,0,0,0,1,0,0,1,0,$  1) where "n/a" indicates missing data (which is omitted in the program).
In fact, this program can be rewritten to the more efficient form shown in Fig. 7b, to minimize the number of variables needed.
Mixture model In the binary mixture model, the frequency events are observed from an equal-weight mixture of two Poisson distributions with different parameters  $\Lambda_1, \Lambda_2$ , which are in discrete steps of 0.1. This is implemented by imposing a geometric prior and multiplying the rate by 0.1 in the Poisson distribution: Poisson  $(0.1 \cdot \Lambda_1)$ . The data is the same as for the switchpoint model. The task is to infer the first rate  $\Lambda_1$ . The probabilistic program is shown in Fig. 8a.
Figure 7: The program code for the Bayesian switchpoint analysis.
$T\sim$  Uniform{1..n};  $\Lambda \sim$  Exponential(1); if  $1\sim$  Bernoulli  $(1 / n)$  { observe  $y_{1}\sim$  Poisson(A);  $\Lambda \sim$  Exponential(1); observe  $y_{2}\sim$  Poisson(A);   
if  $1 <   T$  { observe  $y_{n}\sim$  Poisson(A);  $\boldsymbol{T}\coloneqq 1$  . } else { if  $1\sim$  Bernoulli  $(1 / (n - 1))$  { observe  $y_{1}\sim$  Poisson(A); observe  $y_{2}\sim$  Poisson(A);  $\Lambda \sim$  Exponential(1); observe  $y_{3}\sim$  Poisson(A); } else { observe  $y_{n}\sim$  Poisson(A); } else { } else { a) Switchpoint model from [23].
(b) More efficient version of the switchpoint model.
$\Lambda_{1}\sim \mathrm{Geometric}(0.1);$ $\Lambda_1\sim \mathrm{Geometric}(0.1);$ $\Lambda_2\sim \mathrm{Geometric}(0.1);$  if  $Z = 0$  { observe  $y_{1}\sim \mathrm{Poisson}(0.1\cdot \Lambda_{1})$ $Z\sim \mathrm{Bernoulli}(0.2)$  } else { observe  $y_{1}\sim \mathrm{Poisson}(0.1\cdot \Lambda_{2})$ $Z\sim \mathrm{Bernoulli}(0.8);\}$  ： if  $1\sim \mathrm{Bernoulli}(0.5)$  { if  $Z = 0$  { observe  $y_{m}\sim \mathrm{Poisson}(0.1\cdot \Lambda_{1})$ $Z\sim \mathrm{Bernoulli}(0.2);$  } else { observe  $y_{m}\sim \mathrm{Poisson}(0.1\cdot \Lambda_{2})$ $Z\sim \mathrm{Bernoulli}(0.8);\}$  (a) Program code for the mixture model. (b) Program code for the HMM.
Figure 8: The program code for the mixture and HMM model.
(a) Error of the mean
(b) Error of the standard deviation
Figure 9: Comparison of moments for the original population model.
(a) Error of the mean
Figure 10: Comparison of moments for the modified population model.
Hidden Markov model The hidden Markov example is based on [22, Section 2.2]. The hidden state  $Z$  can be 0 or 1 and transitions to the other state with probability 0.2. The rate of the Poisson-distributed number of observed events depends on the state  $Z$  and is  $0.1 \cdot \Lambda_1$  or  $0.1 \cdot \Lambda_2$ , where we impose a geometric prior on  $\Lambda_1$  and  $\Lambda_2$ . As for the mixture model, this discretizes the rates in steps of 0.1. The inference problem is to infer the first rate  $\Lambda_1$ . The probabilistic program is shown in Fig. 8b, where  $y_k$  are  $m = 30$  simulated data points from the true rates  $\lambda_1 := 0.5$  and  $\lambda_2 := 2.5$ . Concretely, we have  $y = (2, 2, 4, 0, 0, 0, 0, 0, 1, 1, 0, 2, 4, 3, 3, 5, 1, 2, 3, 1, 3, 3, 0, 0, 2, 0, 0, 2, 6, 1)$ .
In the main text, the data we presented was mainly in form of the total variation distance (TVD) between the true posterior (computed by our method) and the approximated posterior (computed by Anglican's inference algorithms). In this section, we present additional evidence in two forms:
- errors of the posterior moments (mean, standard deviation, skewness, and kurtosis) in Figs. 9 to 14, and
- histograms (Fig. 15) of the sampled distribution produced by the best MCMC algorithm, where the sampling budget is picked such that the running time is close to that of our method (e.g. 1000 for the (fast) population model and 10000 for the (slower) mixture model).
The additional plots confirm the conclusions from Section 5.