SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Generating function semantics of programming constructs
Language construct P	[[P]]gf(G)(x)
skip	G(x)
P1; P2	[[P2]]gf([[P1]]gf(G))(x)
Xk := a^T X + c	x_k^c \cdot G(x') where x'_k := x_k^{a_k} and x'_i := x_i x'_k^a_i for i ≠ k
if Xk ∈ A{P1} else {P2}	[[P1]]gf(G_{X_k\in A}) + [[P2]]gf(G - G_{X_k\in A})
where GX_k∈A(x) = ∑i∈A i ∂_k^iG(x[k←0]) / ∂_i^i x_k^i
Xk ~ D	G(x[k←1]) \cdot gf(D)(x_k)
Xk ~ D(X_j)	G(x[k←1, j←x_j \cdot gf(D(1))(x_k)]) for D ∈ {Binomial(-,p), NegBinomial(-,p), Poisson(λ·-)}
G(x[k←1]) + x_j(x_k-1) \cdot ∂_jG(x[k←1]) for D = Bernoulli(-)
observe X_k ∈ A	GX_k∈A(x) = ∑i∈A i ∂_k^iG(x[k←0]) / ∂_i^i x_k^i
Normalization	normalize(G) := G/G(1_n)
by the then-branch  $\llbracket P_1\rrbracket_{\mathrm{gf}}$ , the latter by the else-branch  $\llbracket P_2\rrbracket_{\mathrm{gf}}$ . The computation of  $G_{X_k\in A}$  is best understood by thinking of  $G$  as a power series where we keep only the terms where the exponent of  $x_{k}$  is in  $A$ . Sampling  $X_{k}\sim D$  from a distribution with constant parameters works by first marginalizing out  $X_{k}$  and then multiplying by the generating function of  $D$  with parameter  $x_{k}$ .
The first new construct is sampling  $X_{k} \sim D(X_{j})$  from compound distributions (see Appendix B for a detailed explanation). Observing events  $X_{k} \in A$  uses  $G_{X_{k} \in A}$  like in conditionals, as explained above. Just like the subprobability distribution defined by a program has to be normalized as a last step, we have to normalize the generating function. The normalizing constant is calculated by marginalizing out all variables:  $G(1, \ldots, 1)$ . So we obtain the generating function representing the normalized posterior distribution by rescaling with the inverse:  $\text{normalize}(G) := \frac{G}{G(1_n)}$ . These intuitions can be made rigorous in the form of the following theorem, which is proven in Appendix B.2.
Theorem 3.1. The  $GF$  semantics is correct w.r.t. the standard semantics: for any SGCL program  $P$  and subprobability distribution  $\mu$  on  $\mathbb{R}_{\geq 0}^n$ , we have  $\llbracket P\rrbracket_{\mathrm{gf}}(\mathsf{gf}(\mu)) = \mathsf{gf}(\llbracket P\rrbracket_{\mathrm{std}}(\mu))$ . In particular, it correctly computes the  $GF$  of the posterior distribution of  $P$  as normalize  $(\llbracket P\rrbracket_{\mathrm{gf}}(\mathbf{1}))$ . Furthermore, there is some  $R > 1$  such that  $\llbracket P\rrbracket_{\mathrm{gf}}(\mathbf{1})$  and normalize  $(\llbracket P\rrbracket_{\mathrm{gf}}(\mathbf{1}))$  are defined on  $\{\pmb {x}\in \mathbb{R}^n\mid Q_i < x_i < R\}$  where  $Q_{i} = -R$  if the variable  $X_{i}$  is supported on  $\mathbb{N}$  and  $Q_{i} = 0$  otherwise.
Novelty The semantics builds upon [17, 4]. To our knowledge, the GF semantics of the compound distributions Poisson  $(\lambda \cdot X_{j})$  and Bernoulli  $(X_{j})$  is novel, and the former is required to support most models in Section 5. While the GF of observations has been considered in the context of a specific model [26], this has not been done in the general context of a probabilistic programming language before. More generally, previous works involving GFs only considered discrete distributions, whereas we also allow sampling from continuous distributions. This is a major generalization and requires different proof techniques because the power series representation  $\sum_{i\in \mathbb{N}}\mathbb{P}[X = i]x^i$ , on which the proofs in [26, 17, 4] rely, is not valid for continuous distributions.
Example 3.1 (GF translation). Consider Example 2.1. We can find the posterior distribution mechanically by applying the rules from the GF semantics. We start with the GF  $A(x,y) = \mathbb{E}[x^0 y^0] = 1$  corresponding to  $X$  and  $Y$  being initialized to 0. Sampling  $X$  changes this to GF  $B(x,y) = A(1,y)e^{20(x - 1)} = e^{20(x - 1)}$ . Sampling  $Y$  yields  $C(x,y) = B(x(0.1y + 0.9),1) = e^{2x(y + 9) - 20}$ . Observing  $Y = 2$  yields  $D(x,y) = \frac{1}{2!} y^2\frac{\partial^2}{\partial y^2} C(x,0) = 2x^2y^2e^{18x - 20}$ . To normalize, we divide by  $D(1,1) = 2e^{-2}$ , obtaining  $E(x,y) = \frac{D(x,y)}{D(1,1)} = x^2y^2e^{18(x - 1)}$  since  $A(x,y) = 1$ .
As described above, we can extract from this GF the posterior probability of, for example, exactly 10 individuals  $\mathbb{P}[X = 10] = \frac{1}{10!} \frac{\partial^{10}}{\partial x^{10}} E(0,1) = 991796451840e^{-18}$  and the expected value of the posterior  $\mathbb{E}[X] = \frac{\partial}{\partial x} E(1,1) = 20$ .
4 Implementation & Optimizations
The main difficulty in implementing the GF semantics is the computation of the partial derivatives. A natural approach (as followed by [4, 18]) is to manipulate symbolic representations of the generating
functions and to use computer algebra for the derivatives. However this usually scales badly, as demonstrated by Winner et al. [27], because the size of the generating functions usually grows quickly with the data conditioned on. To see why, note that every observe  $X_{k} = d$  statement in the program is translated to a  $d$ -th partial derivative. Since probabilistic programs tend to contain many data points, it is common for the total order of derivatives to be in the hundreds. The size of the symbolic representation of a function can (and typically does) grow exponentially in the order of the derivative: the derivative of the product of two functions  $f \cdot g$  is the sum of two products  $f' \cdot g + f \cdot g'$ , so the representation doubles in size. Hence the running time would be  $\Omega(2^{d})$  where  $d$  is the sum of all observed values, which is clearly unacceptable.
Instead, we exploit the fact that we do not need to generate the full representation of a GF, but merely to evaluate it and its derivatives. We implement our own automatic differentiation framework for this because existing ones are not designed for computing derivatives of order greater than, say, 4 or 5. In fact, it is more efficient to work with Taylor polynomials instead of higher derivatives directly. Winner et al. [27] already do this for the population model (with only one variable), and we extend this to our more general setting with multiple variables, requiring multivariate Taylor polynomials. In this approach, derivatives are the easy part as they can be read off the Taylor coefficients, but the composition of Taylor polynomials is the bottleneck. Winner et al. [27] use a naive  $O(d^{3})$  approach, which is fast enough for their single-variable use case.
Running time For  $n$  variables, naive composition of Taylor polynomials takes  $O(d^{3n})$  time, where  $d$  is the sum of all observations in the program, i.e. the total order of differentiation, i.e. the degree of the polynomial. Note that this is polynomial in  $d$ , contrary to the symbolic approach, but exponential in the number of variables  $n$ . This is not as bad as it seems because in many cases, the number of variables can be kept to one or two, as opposed to the values of data points (such as the models from Section 5). In fact, we exploit the specific composition structure of generating functions to achieve  $O(d^3)$  for  $n = 1$  and  $O(d^{n + 3})$  for  $n \geq 2$  in the worst case, while often being faster in practice. Overall, our implementation takes  $O(sd^{n + 3})$  time in the worst case, where  $s$  is the number of statements in the program,  $d$  is the sum of all observed values, and  $n$  is the number of program variables (see Appendix C.3).
Reducing the number of variables Given the exponential running time in  $n$ , it is crucial to reduce the number of variables when writing probabilistic programs. For one thing, program variables that are no longer needed can often be reused for a different purpose later. Furthermore, assignment and sampling can be combined with addition:  $X_{k} + = \dots$  stands for  $X_{n + 1} \coloneqq \dots$ ;  $X_{k} \coloneqq X_{k} + X_{n + 1}$  and  $X_{k} + \sim D$  for  $X_{n + 1} \sim D$ ;  $X_{k} \coloneqq X_{k} + X_{n + 1}$ . The GFs for these statements can easily be computed without introducing the temporary variable  $X_{n + 1}$ . Similarly, in observe statements and if-conditions, we use the shorthand  $m \sim D$  with  $m \in \mathbb{N}$  for the event  $X_{n + 1} = m$  where  $X_{n + 1} \sim D$ . Probabilistic programs typically contain many such observations, in particular from compound distributions. Hence it is worthwhile to optimize the generating function to avoid this extra variable  $X_{n + 1}$ . Winner and Sheldon [26] can avoid an extra variable for a compound binomial distribution in the context of a specific model. We extend this to our more general setting with continuous variables, and also present optimized semantics for compound Poisson, negative binomial, and Bernoulli distributions. In fact, this optimization is essential to achieving good performance for many of the examples in Section 5. The optimized translation and its correctness proof can be found in Appendix C.2.
Implementation Our tool Genfer reads a program file and outputs the posterior mean, variance, skewness, and kurtosis of a specified variable. For discrete variables supported on  $\mathbb{N}$ , it also computes the posterior probability masses up to a configurable threshold. To have tight control over performance, especially for the multivariate Taylor polynomials, Genfer is written in Rust [20], a safe systems programming language. Our implementation is available on GitHub: github.com/fzaiser/genfer
Numerical issues Genfer can use several number formats for its computations: 64-bit floating point (the default and fastest), floating point with a user-specified precision, and rational numbers (if no irrational numbers occur). To ensure that the floating-point results are numerically stable, we also implemented interval arithmetic to bound the rounding errors. Initially, we found that programs with continuous distributions led to catastrophic cancellation errors, due to the logarithmic term in their GFs, whose Taylor expansion is badly behaved. We fixed this problem with a slightly modified representation of the GFs, avoiding the logarithms (details in Appendix C). For all the examples in Section 5.2, our results are accurate up to at least 5 significant digits.
Table 3: Comparison of inference times of tools for exact inference on PSI's benchmarks [9]
Tool	Genfer (FP)	Dice (FP)	Genfer (Q)	Dice (Q)	Prodigy	PSI
alarm (F)	0.0005s	0.0067s	0.0012s	0.0066s	0.011s	0.0053s
clickGraph (C)	0.11s	unsupported	3.4s	unsupported	unsupported	46s
clinicalTrial (C)	150s	unsupported	1117s	unsupported	unsupported	timeout
clinicalTrial2 (C)	0.0024s	unsupported	0.031s	unsupported	unsupported	0.46s
digitRecognition (F)	0.021s	0.83s	0.11s	2.7s	31s	146s
evidence1 (F)	0.0002s	0.0057s	0.0003s	0.0056s	0.0030s	0.0016s
evidence2 (F)	0.0002s	0.0056s	0.0004s	0.0057s	0.0032s	0.0018s
grass (F)	0.0008s	0.0067s	0.0044s	0.0067s	0.019s	0.014s
murderMystery (F)	0.0002s	0.0055s	0.0003s	0.0057s	0.0028s	0.0021s
noisyOr (F)	0.0016s	0.0085s	0.019s	0.0088s	0.21s	0.055s
twoCoins (F)	0.0002s	0.0054s	0.0003s	0.0057s	0.0032s	0.0017s
5.1 Comparison with exact inference methods
We compare our tool Genfer with the following tools for exact Bayesian inference: Dice [16], which uses weighted model counting, PSI [9], which manipulates density functions using computer algebra, and Prodigy [18], which is based on generating functions like Genfer, but uses computer algebra instead of automatic differentiation. We evaluate them on the PSI benchmarks [9], excluding those that only PSI supports, e.g. due to observations from continuous distributions. Most benchmarks only use finite discrete distributions (labeled "F"), but three feature continuous priors (labeled "C").
We measured the wall-clock inference time for each tool, excluding startup time and input file parsing, and recorded the minimum from 5 consecutive runs with a one-hour timeout. Dice and Genfer default to floating-point (FP) numbers, whereas Prodigy and PSI use rational numbers, which is slower but prevents rounding errors. For a fair comparison, we evaluated all tools in rational mode and separately compared Dice with Genfer in FP mode. The results (Table 3) demonstrate Genfer's speed even on finite discrete models, despite our primary focus on models with infinite support.
5.2 Comparison with approximate inference methods
It is impossible to compare our approach with other exact inference methods on realistic models with infinite support (Section 5.3): the scalable systems Dice [16] and SPPL [22] don't support such priors and the symbolic solvers PSI [9] and Prodigy [18] run out of memory or time out after an hour.
Truncation As an alternative, we considered approximating the posterior by truncating discrete distributions with infinite support. This reduces the problem to finite discrete inference, which is more amenable to exact techniques. We decided against this approach because Winner and Sheldon [26] already demonstrated its inferiority to GF-based exact inference on their graphical model. Moreover, it is harder to truncate general probabilistic programs, and even impossible for continuous priors.
Monte-Carlo inference Hence, we compare our approach with Monte Carlo inference methods. Specifically, we choose the Anglican [24] probabilistic programming system because it offers the best built-in support for discrete models with many state-of-the-art inference algorithms. Other popular systems are less suitable: Gen [6] specializes in programmable inference; Stan [3], Turing [8], and Pyro [1] mainly target continuous models; and WebPPL's [11] discrete inference algorithms are less extensive than Anglican's (e.g. no support for interacting particle MCMC).
Methodology The approximation error of a Monte Carlo inference algorithm depends on its settings $^3$  (e.g. the number of particles for SMC) and decreases with the number of samples the longer it is run. To ensure a fair comparison, we use the following setup: for each inference problem, we run several inference algorithms with various configurations (settings and sampling budgets) and
<sup>1</sup>We were not able to run Hakaru [21], which uses the computer algebra system Maple as a backend, but we do not expect it to produce better results than the other systems, given that PSI is generally more performant [9].
${}^{2}$  PSI was run with the experimental flag - -dp on finite discrete benchmarks for improved performance.
3See https://probprog.github.io/anglican/inference/ for a full list of Anglican's options.
(b) Modified arrival rate
Figure 1: Comparison of the population model and its modifications with approximate inference: mean and standard error of the computation time and TVD over 20 repeated runs.
(c) Two-type branching process
report the approximation error and the elapsed time. To measure the quality of the approximation, we report the total variation distance (TVD) between the exact solution and the approximate posterior distribution, as well as the approximation errors of the posterior mean  $\mu$ , standard deviation  $\sigma$ , skewness (third standardized moment)  $S$ , and kurtosis  $K$ . To ensure invariance of our error metrics under translation and scaling of the distribution, we compute the error of the mean as  $\frac{|\hat{\mu} - \mu|}{\sigma}$ , where  $\mu$  is the true posterior mean,  $\hat{\mu}$  its approximation, and  $\sigma$  the true posterior standard deviation; the relative error of the standard deviation  $\sigma$  (because  $\sigma$  is invariant under translation but not scaling), and the absolute error of skewness and kurtosis (because they are invariant under both translation and scaling). To reduce noise, we average all these error measures and the computation times over 20 runs and report the standard error as error bars. We run several well-known inference algorithms implemented in Anglican: importance sampling (IS), Lightweight Metropolis-Hastings (LMH), Random Walk Metropolis-Hastings (RMH), Sequential Monte Carlo (SMC), Particle Gibbs (PGibbs), and interacting particle MCMC (IPMCMC). For comparability, in all experiments, each algorithm is run with two sampling budgets and, if possible, two different settings (one being the defaults) for a total of four configurations. The sampling budgets were 1000 or 10000, because significantly lower sample sizes gave unusable results and significantly higher sample sizes took much more time than our exact method. We discard the first  $20\%$  of the samples, a standard procedure called "burn-in".
Note that this setup is generous to the approximate methods because we only report the average time for one run of each configuration. However, in practice, one does not know the best configuration, so the algorithms need to be run several times with different settings. By contrast, our method requires only one run because the result is exact.
5.3 Benchmarks with infinite-support distributions
Population ecology Our first benchmark comes from [26, 27] and models animal populations. We have seen a simplified version in Example 2.1. Here we model a population  $N_{k}$  at time steps  $k = 0, \dots, m$ . At each time step, there is a Poisson-distributed number of new arrivals, which are added to the binomially distributed number of survivors from the previous time step. Each individual is observed with a fixed probability  $\delta$ , so the number of observed individuals is binomially distributed:
$$
\text {N e w} _ {k} \sim \text {P o i s s o n} (\lambda_ {k}); \quad \text {S u r v i v o r s} _ {k} \sim \text {B i n o m i a l} (N _ {k - 1}, \delta);
$$
$$
N _ {k} := \text {N e w} _ {k} + \text {S u r v i v o r s} _ {k}; \quad \text {o b s e r v e} y _ {k} \sim \text {B i n o m i a l} (N _ {k}, \rho);
$$
where the model parameters  $\lambda_{k}\in \mathbb{R},\delta \in [0,1]$  are taken from [26]; the detection probability  $\rho$  is set to 0.2 ([26] considers a range of values, but we pick one for space reasons); and the observed number  $y_{k}\in \mathbb{N}$  of individuals in the population at time step  $k$  is simulated from the same ground truth as [26]. The goal is to infer the final number  $N_{m}$  of individuals in the population. We set the population size model parameter and hence the observed values which influence the running time to be 4 times larger than the largest in [26] (see details in Appendix D.3). The results (Fig. 1a) show that our method is superior to MCMC methods in both computation time and accuracy since it is exact.
Modifications While this model was already solved exactly in [26], our probabilistic programming approach makes it trivial to modify the model, since the whole inference is automated and one only needs to change a few lines of the program: (a) we can model the possibility of natural disasters affecting the offspring rate with a conditional:  $\text{Disaster} \sim \text{Bernoulli}(0.1)$ ; if  $\text{Disaster} = 1 \left\{ \text{New}_k \sim \text{Poisson}(\lambda') \right\}$  else  $\left\{ \text{New}_k \sim \text{Poisson}(\lambda) \right\}$ , or (b) instead of a single population, we can model populations of two kinds of individuals that interact, i.e. a multitype branching process (see Fig. 1c). None
(a) Switchpoint: TVD error
(b) Mixture: TVD error
(c) Mixture: histogram
(d) HMM: TVD error
Figure 2: Plots for the switchpoint, mixture, and hidden Markov model.
(a) Error of the mean
Figure 3: Comparison of moments for the switchpoint model: approximate inference vs our method.
(b) Error of the std. dev.
(c) Error of the skewness
(d) Error of the kurtosis
of these modifications can be handled by [26] or [27]. The results of the first modification (Fig. 1b) are very similar. The more complex second modification takes longer to solve exactly, but less time than approximate inference with 10000 samples, and achieves zero error.
Switchpoint model Our second benchmark is Bayesian switchpoint analysis, which is about detecting a change in the frequency of certain events over time. We use the model from [23] with continuous priors and its 111 real-world data points about the frequency of coal-mining accidents. We compare both the moment errors (Fig. 3) and the TVD errors (Fig. 2a). In both cases, the approximations are less accurate and take longer than our exact method.
Mixture model We consider a binary mixture model on the same data set, with equal mixture weights and a geometric prior for the rates: each data point is observed from a mixture of two Poisson distributions with different rates and the task is to infer these rates. Due to their multimodality, mixture models are notoriously hard for approximate inference methods, which is confirmed in Fig. 2b. Even the runs with the lowest error cover only one of the two modes (cf. the sample histogram in Fig. 2c).
Hidden Markov model We use a hidden Markov model based on [22, Section 2.2], but involving infinite (geometric) priors. It is a two-state system with known transition probabilities and the rate for the observed data depends on the hidden state. We run this model on 30 simulated data points. For this model as well, our method clearly outperforms approximate methods (Fig. 2d).
To our knowledge, our approach is the first to find an exact solution to these problems, except the very first problem without the modifications, which appeared in [26]. For brevity, we only presented the most important aspects of these benchmarks, relegating their encoding as probabilistic programs to Appendix D.3. Code and reproduction instructions are provided in the supplementary material.
By leveraging generating functions, we have developed and proven correct a framework for exact Bayesian inference on discrete models, even with infinite support and continuous priors. We have demonstrated competitive performance on a range of models specified in an expressive probabilistic programming language, which our tool Genfer processes automatically.
Future work It is a natural question how our method could be integrated with a more general probabilistic programming system. For example, in a sampling-based inference algorithm, one could imagine using generating functions to solve subprograms exactly if this is possible. More generally, it would be desirable to explore how the compositionality of the GF translation can be improved. As it stands, our GF translation describes the joint distribution of all program variables – we never reason “locally” about a subset of the variables. A compositional approach would likely facilitate the application of the GF method to functional probabilistic languages like Anglican.
We would like to thank Maria Craciun, Hugo Paquet, Tim Reichelt, and Dominik Wagner for providing valuable input on this work. We are especially grateful to Mathieu Huot for insightful discussions and very helpful feedback on a draft of this paper.
This research was supported by the Engineering and Physical Sciences Research Council (studentship 2285273, grant EP/T006579) and the National Research Foundation, Singapore, under its RSS Scheme (NRF-RSS2022-009). For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.