Title: Exact Bayesian Inference on Discrete Models via Probability Generating Functions: A Probabilistic Programming Approach
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian statistics

probabliistic programming

exact inference

discrete models

probability generating functions
Abstract: We present an exact Bayesian inference method for discrete statistical models, which can find exact solutions to a large class of discrete inference problems, even with infinite support and continuous priors.
To express such models, we introduce a probabilistic programming language that supports discrete and continuous sampling, discrete observations, affine functions, (stochastic) branching, and conditioning on discrete events.
Our key tool is *probability generating functions*:
they provide a compact closed-form representation of distributions that are definable by programs, thus enabling the exact computation of posterior probabilities, expectation, variance, and higher moments.
Our inference method is provably correct and fully automated in a tool called *Genfer*, which uses automatic differentiation (specifically, Taylor polynomials), but does not require computer algebra.
Our experiments show that Genfer is often faster than the existing exact inference tools PSI, Dice, and Prodigy.
On a range of real-world inference problems that none of these exact tools can solve, Genfer's performance is competitive with approximate Monte Carlo methods, while avoiding approximation errors.

Exact Bayesian Inference on Discrete Models via Probability Generating Functions: A Probabilistic Programming Approach

Abstract

We present an exact Bayesian inference method for discrete statistical models, which can find exact solutions to a large class of discrete inference problems, even with infinite support and continuous priors. To express such models, we introduce a probabilistic programming language that supports discrete and continuous sampling, discrete observations, affine functions, (stochastic) branching, and conditioning on discrete events. Our key tool is probability generating functions : they provide a compact closed-form representation of distributions that are definable by programs, thus enabling the exact computation of posterior probabilities, expectation, variance, and higher moments. Our inference method is provably correct and fully automated in a tool called Genfer , which uses automatic differentiation (specifically, Taylor polynomials), but does not require computer algebra. Our experiments show that Genfer is often faster than the existing exact inference tools PSI, Dice, and Prodigy. On a range of real-world inference problems that none of these exact tools can solve, Genfer's performance is competitive with approximate Monte Carlo methods, while avoiding approximation errors.

1 Introduction

Bayesian statistics is a highly successful framework for reasoning under uncertainty that has found widespread use in a variety of fields, such as AI/machine learning, medicine and healthcare, finance and risk management, social sciences, climate science, astrophysics, and many other disciplines. At its core is the idea of representing uncertainty as probability, and updating prior beliefs based on observed data via Bayes' law, to arrive at posterior beliefs. A key challenge in Bayesian statistics is computing this posterior distribution: analytical solutions are usually impossible or intractable, which necessitates the use of approximate methods, such as Markov-chain Monte Carlo (MCMC) or variational inference. In this work, we identify a large class of discrete models for which exact inference is in fact possible, in particular time series models of count data, such as autoregressive models, hidden Markov models, switchpoint models. We achieve this by leveraging probability generating functions (GFs) as a representation of distributions. The GF of a random variable X is defined to be the function G ( x ) := E [ x X ] . In probability theory, it is a well-known tool to study random variables and their distributions, related to the moment generating and the characteristic functions [14, Chapter 4]. In computer science, GFs have previously been used for the analysis of probabilistic programs [17, 4] and for exact inference on certain classes of graphical models [26, 27].

Here we apply them uniformly in a much more general context via probabilistic programming, enabling exact inference on more expressive Bayesian models.

We characterize the class of supported models with the help of a probabilistic programming language. Probabilistic programming [25] has recently emerged as a powerful tool in Bayesian inference. Probabilistic programming systems allow users to specify complex statistical models as programs in a precise yet intuitive way, and automate the Bayesian inference task. This allows practitioners to focus on modelling, leaving the development of general-purpose inference algorithms to experts. Consequently, probabilistic programming systems such as Stan [3] enjoy increasing popularity among statisticians and data scientists. We describe a programming language, called SGCL (statistical guarded command language), that extends pGCL (probabilistic GCL) [17]. This language is carefully designed to be simple yet expressive, and just restrictive enough to enable exact Bayesian inference on all programs that can be written in it.

Contributions We provide a new framework for exact inference on discrete Bayesian models.

(a) Our method is applicable to a large class of discrete models with infinite support , in particular time series models of count data, such as autoregressive models for population dynamics, Bayesian switchpoint models, mixture models, and hidden Markov models. To our knowledge, no exact inference method for all but the simplest population models was known before.

(b) The models are specified in a probabilistic programming language (PPL) , which provides flexibility in model definition, thus facilitating model construction. Our PPL supports stochastic branching, continuous and discrete priors, discrete observations, and conditioning on events involving discrete variables.

(c) Every program written in the language can be translated to a generating function that represents the posterior distribution in an automatic and provably correct way. From this generating function, one can extract posterior mean, variance, and higher moments , as well as the posterior probability masses (for a discrete distribution).

(d) We have built an optimized tool , called Genfer ('GENerating Functions for inFERence'), that takes a probabilistic program as input and automatically computes the aforementioned set of descriptive statistics about the posterior distribution.

(e) We demonstrate that (1) on benchmarks with finite support, Genfer's performance is often better than existing exact inference tools , and (2) on a range of real-world examples that no existing exact tool supports, Genfer is competitive with approximate Monte Carlo methods , while achieving zero approximation error.

Related Work Computing the exact posterior distribution of probabilistic programs is intractable in general as it requires analytical solutions to integrals [9]. For this reason, existing systems either restrict the programming language to allow only tractable constructs (this is our approach) or cannot guarantee successful inference. In the former category are Dice [16], which only supports finite discrete distributions, and SPPL [22], which supports some infinite-support distributions but requires finite discrete priors. Both are based on probabilistic circuits [5] or extensions thereof, which allow for efficient and exact inference. In the latter category are the systems PSI [9] and Hakaru [21], which do not impose syntactic restrictions. They rely on computer algebra techniques to find a closed-form solution for the posterior. Such a form need not exist in general and, even if it does, the system may fail to find it and the running time is unpredictable and unscalable [22]. None of the case studies featured in our evaluation (Section 5.2) can be handled by the aforementioned tools.

Probability generating functions are a useful tool in probability theory to study random variables with infinite support, e.g. in the context of branching processes [7, Chapter 12]. In the context of Bayesian inference, probabilistic generating circuits (PGCs) leverage them to boost the expressiveness of probabilistic circuits [28] and enable efficient inference [15]. However, PGCs cannot handle random variables with infinite support, which is the focus of our approach. Generating functions have also been applied to probabilistic graphical models: Winner and Sheldon [26] find a symbolic representation of the generating function for a Poisson autoregressive model and extract posterior probabilities from it. Subsequently, Winner et al. [27] extend this model to latent variable distributions other than the Poisson distribution, where symbolic manipulation is no longer tractable. Instead they evaluate generating functions and their derivatives using automatic differentiation, which enables exact inference for graphical models. Probabilistic programming is an elegant way of generalizing graphical models, allowing a much richer representation of models [10, 25, 12]. Our contribution here is a new framework for exact inference on Bayesian models via probabilistic programming.

In the context of discrete probabilistic programs without conditioning, Klinkenberg et al. [17] use generating functions to (manually) analyze loop invariants and determine termination probabilities. In follow-up work, Chen et al. [4] extend these techniques to automatically check (under certain restrictions) whether a looping program generates a specified distribution. Their analysis tool Prodigy recently gained the ability to perform Bayesian inference as well [18]. It supports discrete distributions with infinite support (but no continuous priors) and is less scalable than our automatic-differentiation approach (see Section 5) since it relies on computer algebra.

Limitations Exact posterior inference is already PSPACE-hard for probabilistic programs involving only finite discrete distributions [16, Section 6]. It follows that our method cannot always be performant and has to restrict the supported class of probabilistic programs. Indeed, our programming language forbids certain constructs, such as nonlinear transformations and observations of continuous variables, in order to preserve a closed-form representation of the generating functions (Section 2). For the same reason, our method cannot compute probability density functions for continuous parameters (but probability masses for discrete parameters and exact moments for all parameters are fine). Regarding performance, the running time of inference is polynomial in the numbers observed in the program and exponential in the number of program variables (Section 4). Despite these limitations, our approach shows that exact inference is possible in the first place, and our evaluation (Section 5) demonstrates support for many real-world models and efficient exact inference in practice.

2 Bayesian Probabilistic Programming

Probabilistic programming languages extend ordinary programming languages with two additional constructs: one for sampling from probability distributions and one for conditioning on observed values. We first discuss a simple program that can be written in our language using a simplified example based on population ecology (cf. [26]).

Suppose you're a biologist trying to estimate the size of an animal population migrating into a new habitat. The immigration of animals is often modeled using a Poisson distribution. You cannot count the animals exhaustively (otherwise we wouldn't need estimation techniques), so we assume that each individual is observed independently with a certain probability; in other words, the count is binomially distributed. For simplicity, we assume that the rate of the Poisson and the probability of the binomial distribution are known, say 20 and 0.1. (For more realistic population models, see Section 5.) As a generative model, this would be written as X ∼ Poisson (20); Y ∼ Binomial ( X, 0 . 1) .

Suppose you observe Y = 2 animals. The Bayesian inference problem is to compute the posterior distribution P ( X = x | Y = 2) given by Bayes' rule as P ( X = x | Y = 2) = P ( X = x ) P ( Y =2 | X = x ) P ( Y =2) , where P ( X = x ) is called the prior probability , P ( Y = 2 | X = x ) the likelihood and P ( Y = 2) the evidence or normalization constant .

Example 2.1. In our probabilistic programming language, this simplified population model would be expressed as:

The syntax looks similar to generative model notation except that the observations are expressible as a command. Such a program denotes a joint probability distribution of its program variables, which are viewed as random variables. The program statements manipulate this joint distribution. After the first two sampling statements, the distribution has the probability mass function (PMF) p 1 ( x, y ) = P [ X = x, Y = y ] = Poisson ( x ; 20) · Binomial ( y ; x, 0 . 1) . Observing Y = 2 restricts this to the PMF p 2 ( x, y ) = P [ X = x, Y = y = 2] , which equals P [ X = x, Y = 2] = p 1 ( x, 2) if y = 2 and 0 otherwise. Note that this is not a probability, but a subprobability , distribution because the total mass is less than 1. So, as a final step, we need to normalize , i.e. rescale the subprobability distribution back to a probability distribution. This corresponds to the division by the evidence in Bayes' rule, yielding the PMF p 3 ( x, y ) = p 2 ( x,y ) ∑ x,y p 2 ( x,y ) . To obtain the posterior P [ X = x | Y = 2] , which is a distribution of the single variable X , not the joint of X and Y , we need to marginalize p 3 and find ∑ y p 3 ( x, y ) = ∑ y p 2 ( x,y ) ∑ x,y p 2 ( x,y ) = p 1 ( x, 2) ∑ x p 1 ( x, 2) = P [ X = x,Y =2] P [ Y =2] = P [ X = x | Y = 2] , as desired.

Programming constructs Next we describe our probabilistic programming language more formally. It is based on the probabilistic guarded command language (pGCL) from [17] but augments

it with statistical features like conditioning on events and normalization, which is why we call it Statistical GCL (SGCL) . Each program operates on a fixed set of variables X = { X 1 , . . . , X n } taking values in R ≥ 0 . A program consists of a list of statements P 1 ; P 2 ; . . . ; P m . The simplest statement is skip , which does nothing and is useful in conditionals (like ♣❛/a115/a115 in Python). Variables can be transformed using affine maps, e.g. X 2 := 2 X 1 +7 X 3 +2 (note that the coefficients must be nonnegative to preserve nonnegativity of the variables). Programs can branch on the value of a (discrete) variable (e.g. if X k ∈ { 1 , 3 , 7 } { . . . } else { . . . } ), and sample new values for variables from distributions (e.g. X k ∼ Binomial (10 , 0 . 5) or X k ∼ Binomial ( X j , 0 . 5) ). The supported distributions are Bernoulli, Categorical, Binomial, Uniform (both discrete and continuous), NegBinomial, Geometric, Poisson, Exponential, and Gamma. They need to have constant parameters except for the compound distributions Binomial ( X,p ) , NegBinomial ( X,p ) , Poisson ( λ · X ) , and Bernoulli ( X ) , where X can be a variable. One can observe events involving discrete variables (e.g. observe X k ∈ { 3 , 5 } ) and values from (discrete) distributions directly (e.g. observe 3 ∼ Binomial ( X j , 0 . 5) ). Note that observe m ∼ D can be seen as a convenient abbreviation of Y ∼ D ; observe Y = m with a fresh variable Y . After an observation, the variable distribution is usually not a probability distribution anymore, but a subprobability distribution ('the numerator of Bayes' rule').

Syntax In summary, the syntax of programs P has the following BNF grammar:

where P , P 1 and P 2 are subprograms; a 1 , . . . , a n , c ∈ R ≥ 0 ; m ∈ N ; A is a finite subset of the naturals; and D is a supported distribution. To reduce the need for temporary variables, the abbreviations + ∼ , += , and if m ∼ D { . . . } else { . . . } are available (see Section 4). A fully formal description of the language constructs can be found in Appendix A.

Restrictions Our language imposes several syntactic restrictions to guarantee that the generating function of any program admits a closed form, which enables exact inference (see Section 3.1): (a) only affine functions are supported (e.g. no X 2 or exp( X ) ), (b) only comparisons between variables and constants are supported (e.g. no test for equality X = Y ), (c) only observations from discrete distributions on N and only comparisons of such random variables are supported (e.g. no observe 1 . 5 ∼ Normal (0 , 1) ), (d) a particular choice of distributions and their composites is supported, (e) loops or recursion are not supported. A discussion of possible language extensions and relaxations of these restrictions can be found in Appendix B.3.

3 Generating Functions

Consider Example 2.1. Even though the example is an elementary exercise in probability, it is challenging to compute the normalizing constant, because one needs to evaluate an infinite sum: P [ Y = 2] = ∑ x ∈ N P [ Y = 2 | X = x ] P [ X = x ] = ∑ x ∈ N ( x 2 ) 0 . 1 2 0 . 9 x -2 · 20 x e -20 x ! . It turns out to be 2 e -2 (see Example 3.1), but it is unclear how to arrive at this result in an automated way. If X had been a continuous variable, we would even have to evaluate an integral. We will present a technique to compute such posteriors mechanically. It relies on probability generating functions, whose definition includes an infinite sum or integral, but which often admit a closed form, thus enabling the exact computation of posteriors.

Definition Probability generating functions are a well-known tool in probability theory to study random variables and their distributions, especially discrete ones. The probability generating function of a random variable X is defined to be the function G ( x ) := E [ x X ] . For a discrete random variable supported on N , this can also be written as a power series G ( x ) = ∑ n ∈ N P [ X = n ] · x n . Since we often deal with subprobability distributions, we omit 'probability' from the name and refer to G ( x ) := E X ∼ µ [ x X ] , where µ is a subprobability measure, simply as a generating function (GF). For continuous variables, it is often called factorial moment generating function , but we stick with the former name in all contexts. We will use the notation gf ( µ ) or gf ( X ) for the GF of µ or X . Note that for discrete random variables supported on N , the GF is always defined on [ -1 , 1] , and for continuous ones at x = 1 , but it need not be defined at other x .

Probability masses and moments Many common distributions admit a closed form for their GF (see Table 1). In such cases, GFs are a compact representation of a distribution, even if it has infinite or continuous support like the Poisson or Exponential distributions, respectively. Crucially, one can

extract probability masses and moments of a distribution from its generating function. For discrete random variables X , P [ X = n ] is the n -th coefficient in the power series representation of G , so can be computed as the Taylor coefficient at 0 : 1 n ! G ( n ) (0) (hence the name ' probability generating function'). For a discrete or continuous random variable X , its expected value is E [ X ] = G ′ (1) , and more generally, its n -th factorial moment is E [ X ( X -1) · · · ( X -n +1)] = E [ d n d x n x X ] | x =1 = G ( n ) (1) (hence the name ' factorial moment generating function'). The raw and central moments can easily be computed from the factorial moments. For instance, the variance is V [ X ] = G ′′ (1)+ G ′ (1) -G ′ (1) 2 . We will exploit these properties of generating functions through automatic differentiation.

Multivariate case The definition of GFs is extended to multidimensional distributions in a straightforward way: for random variables X = ( X 1 , . . . , X n ) , their GF is the function G ( x ) := E [ x X ] where we write x := ( x 1 , . . . , x n ) and x X := x X 1 1 · · · x X n n . We generally follow the convention of using uppercase letters for random and program variables, and lowercase letters for the corresponding parameters of the generating function. Marginalization can also be expressed in terms of generating functions: to obtain the GF ˜ G of the joint distribution of ( X 1 , . . . , X n -1 ) , i.e. to marginalize out X n , one simply substitutes 1 for x n in G : ˜ G ( x 1 , . . . , x n -1 ) = G ( x 1 , . . . , x n -1 , 1) . This allows us to compute probability masses and moments of a random variable in a joint distribution: we marginalize out all the other variables and then use the previous properties of the derivatives.

3.1 Translating programs to generating functions

The standard way of describing the meaning of probabilistic programs is assigning (sub-)probability distributions to them. An influential example is Kozen's distribution transformer semantics [19], where each program statement transforms the joint distribution of all the variables X = ( X 1 , . . . , X n ) . Since Kozen's language does not include observations, we present the full semantics in Appendix A. We call this the standard semantics of a probabilistic program and write J P K std ( µ ) for the transformation of µ by the program P . As a last step, the subprobability distribution µ has to be normalized , which we write normalize ( µ ) := µ ∫ d µ . This reduces the Bayesian inference problem for a given program to computing its semantics, starting from the joint distribution Dirac ( 0 n ) in which all n variables are initialized to 0 with probability 1 . While mathematically useful, the distribution transformer semantics is hardly amenable to computation as it involves integrals and infinite sums.

Instead, we shall compute the generating function of the posterior distribution represented by the probabilistic program. Then we can extract posterior probability masses and moments using automatic differentiation. Each statement in the programming language transforms the generating function of the distribution of program states, i.e. the joint distribution of the values of the variables X = ( X 1 , . . . , X n ) . Initially, we start with the constant function G = 1 , which corresponds to all variables being initialized with 0 since E [ x 0 ] = 1 .

The generating function semantics of a program J P K gf describes how to transform G to the generating function J P K gf ( G ) for the distribution at the end. It is defined in Table 2, where the update notation x [ i ↦→ a ] denotes ( x 1 , . . . , x i -1 , a, x i +1 , . . . , x n ) and 1 n means the n -tuple (1 , . . . , 1) . The first five rules were already described in [4], so we only explain them briefly: skip leaves everything unchanged and P 1 ; P 2 chains two statements by transforming with J P 1 K gf and then J P 2 K gf . To explain linear assignments, consider the case of only two variables: X 1 := 2 X 1 +3 X 2 +5 . Then

For conditionals if X k ∈ A { P 1 } else { P 2 } , we split the generating function G into two parts: one where the condition is satisfied ( G X k ∈ A ) and its complement ( G -G X k ∈ A ). The former is transformed

̸

by the then-branch J P 1 K gf , the latter by the else-branch J P 2 K gf . The computation of G X k ∈ A is best understood by thinking of G as a power series where we keep only the terms where the exponent of x k is in A . Sampling X k ∼ D from a distribution with constant parameters works by first marginalizing out X k and then multiplying by the generating function of D with parameter x k .

The first new construct is sampling X k ∼ D ( X j ) from compound distributions (see Appendix B for a detailed explanation). Observing events X k ∈ A uses G X k ∈ A like in conditionals, as explained above. Just like the subprobability distribution defined by a program has to be normalized as a last step, we have to normalize the generating function. The normalizing constant is calculated by marginalizing out all variables: G (1 , . . . , 1) . So we obtain the generating function representing the normalized posterior distribution by rescaling with the inverse: normalize ( G ) := G G ( 1 n ) . These intuitions can be made rigorous in the form of the following theorem, which is proven in Appendix B.2.

Theorem 3.1. The GF semantics is correct w.r.t. the standard semantics: for any SGCL program P and subprobability distribution µ on R n ≥ 0 , we have J P K gf ( gf ( µ )) = gf ( J P K std ( µ )) . In particular, it correctly computes the GF of the posterior distribution of P as normalize ( J P K gf ( 1 )) . Furthermore, there is some R > 1 such that J P K gf ( 1 ) and normalize ( J P K gf ( 1 )) are defined on { x ∈ R n | Q i < x i < R } where Q i = -R if the variable X i is supported on N and Q i = 0 otherwise.

Novelty The semantics builds upon [17, 4]. To our knowledge, the GF semantics of the compound distributions Poisson ( λ · X j ) and Bernoulli ( X j ) is novel, and the former is required to support most models in Section 5. While the GF of observations has been considered in the context of a specific model [26], this has not been done in the general context of a probabilistic programming language before. More generally, previous works involving GFs only considered discrete distributions, whereas we also allow sampling from continuous distributions. This is a major generalization and requires different proof techniques because the power series representation ∑ i ∈ N P [ X = i ] x i , on which the proofs in [26, 17, 4] rely, is not valid for continuous distributions.

Example 3.1 (GF translation) . Consider Example 2.1. We can find the posterior distribution mechanically by applying the rules from the GF semantics. We start with the GF A ( x, y ) = E [ x 0 y 0 ] = 1 corresponding to X and Y being initialized to 0. Sampling X changes this to GF B ( x, y ) = A (1 , y ) e 20( x -1) = e 20( x -1) . Sampling Y yields C ( x, y ) = B ( x (0 . 1 y + 0 . 9) , 1) = e 2 x ( y +9) -20 . Observing Y = 2 yields D ( x, y ) = 1 2! y 2 ∂ 2 ∂y 2 C ( x, 0) = 2 x 2 y 2 e 18 x -20 . To normalize, we divide by D (1 , 1) = 2 e -2 , obtaining E ( x, y ) = D ( x,y ) D (1 , 1) = x 2 y 2 e 18( x -1) since A ( x, y ) = 1 .

As described above, we can extract from this GF the posterior probability of, for example, exactly 10 individuals P [ X = 10] = 1 10! ∂ 10 ∂x 10 E (0 , 1) = 991796451840 e -18 and the expected value of the posterior E [ X ] = ∂ ∂x E (1 , 1) = 20 .

4 Implementation & Optimizations

The main difficulty in implementing the GF semantics is the computation of the partial derivatives. A natural approach (as followed by [4, 18]) is to manipulate symbolic representations of the generating

functions and to use computer algebra for the derivatives. However this usually scales badly, as demonstrated by Winner et al. [27], because the size of the generating functions usually grows quickly with the data conditioned on. To see why, note that every observe X k = d statement in the program is translated to a d -th partial derivative. Since probabilistic programs tend to contain many data points, it is common for the total order of derivatives to be in the hundreds. The size of the symbolic representation of a function can (and typically does) grow exponentially in the order of the derivative: the derivative of the product of two functions f · g is the sum of two products f ′ · g + f · g ′ , so the representation doubles in size. Hence the running time would be Ω(2 d ) where d is the sum of all observed values, which is clearly unacceptable.

Instead, we exploit the fact that we do not need to generate the full representation of a GF, but merely to evaluate it and its derivatives . We implement our own automatic differentiation framework for this because existing ones are not designed for computing derivatives of order greater than, say, 4 or 5. In fact, it is more efficient to work with Taylor polynomials instead of higher derivatives directly. Winner et al. [27] already do this for the population model (with only one variable), and we extend this to our more general setting with multiple variables, requiring multivariate Taylor polynomials . In this approach, derivatives are the easy part as they can be read off the Taylor coefficients, but the composition of Taylor polynomials is the bottleneck. Winner et al. [27] use a naive O ( d 3 ) approach, which is fast enough for their single-variable use case.

Running time For n variables, naive composition of Taylor polynomials takes O ( d 3 n ) time, where d is the sum of all observations in the program, i.e. the total order of differentiation, i.e. the degree of the polynomial. Note that this is polynomial in d , contrary to the symbolic approach, but exponential in the number of variables n . This is not as bad as it seems because in many cases, the number of variables can be kept to one or two, as opposed to the values of data points (such as the models from Section 5). In fact, we exploit the specific composition structure of generating functions to achieve O ( d 3 ) for n = 1 and O ( d n +3 ) for n ≥ 2 in the worst case, while often being faster in practice. Overall, our implementation takes O ( sd n +3 ) time in the worst case, where s is the number of statements in the program, d is the sum of all observed values, and n is the number of program variables (see Appendix C.3).

Reducing the number of variables Given the exponential running time in n , it is crucial to reduce the number of variables when writing probabilistic programs. For one thing, program variables that are no longer needed can often be reused for a different purpose later. Furthermore, assignment and sampling can be combined with addition: X k += . . . stands for X n +1 := . . . ; X k := X k + X n +1 and X k + ∼ D for X n +1 ∼ D ; X k := X k + X n +1 . The GFs for these statements can easily be computed without introducing the temporary variable X n +1 . Similarly, in observe statements and if-conditions, we use the shorthand m ∼ D with m ∈ N for the event X n +1 = m where X n +1 ∼ D . Probabilistic programs typically contain many such observations, in particular from compound distributions. Hence it is worthwhile to optimize the generating function to avoid this extra variable X n +1 . Winner and Sheldon [26] can avoid an extra variable for a compound binomial distribution in the context of a specific model. We extend this to our more general setting with continuous variables, and also present optimized semantics for compound Poisson, negative binomial, and Bernoulli distributions. In fact, this optimization is essential to achieving good performance for many of the examples in Section 5. The optimized translation and its correctness proof can be found in Appendix C.2.

Implementation Our tool Genfer reads a program file and outputs the posterior mean , variance , skewness , and kurtosis of a specified variable. For discrete variables supported on N , it also computes the posterior probability masses up to a configurable threshold. To have tight control over performance, especially for the multivariate Taylor polynomials, Genfer is written in Rust [20], a safe systems programming language. Our implementation is available on GitHub: github.com/fzaiser/genfer

Numerical issues Genfer can use several number formats for its computations: 64-bit floating point (the default and fastest), floating point with a user-specified precision, and rational numbers (if no irrational numbers occur). To ensure that the floating-point results are numerically stable, we also implemented interval arithmetic to bound the rounding errors. Initially, we found that programs with continuous distributions led to catastrophic cancellation errors, due to the logarithmic term in their GFs, whose Taylor expansion is badly behaved. We fixed this problem with a slightly modified representation of the GFs, avoiding the logarithms (details in Appendix C). For all the examples in Section 5.2, our results are accurate up to at least 5 significant digits.

5 Empirical Evaluation

5.1 Comparison with exact inference methods

We compare our tool Genfer with the following tools for exact Bayesian inference: Dice [16], which uses weighted model counting, PSI [9], which manipulates density functions using computer algebra, and Prodigy [18], which is based on generating functions like Genfer, but uses computer algebra instead of automatic differentiation. 1 We evaluate them on the PSI benchmarks [9], excluding those that only PSI supports, e.g. due to observations from continuous distributions. Most benchmarks only use finite discrete distributions (labeled 'F'), but three feature continuous priors (labeled 'C').

We measured the wall-clock inference time for each tool, excluding startup time and input file parsing, and recorded the minimum from 5 consecutive runs with a one-hour timeout. 2 Dice and Genfer default to floating-point (FP) numbers, whereas Prodigy and PSI use rational numbers, which is slower but prevents rounding errors. For a fair comparison, we evaluated all tools in rational mode and separately compared Dice with Genfer in FP mode. The results (Table 3) demonstrate Genfer's speed even on finite discrete models, despite our primary focus on models with infinite support.

5.2 Comparison with approximate inference methods

It is impossible to compare our approach with other exact inference methods on realistic models with infinite support (Section 5.3): the scalable systems Dice [16] and SPPL [22] don't support such priors and the symbolic solvers PSI [9] and Prodigy [18] run out of memory or time out after an hour.

Truncation As an alternative, we considered approximating the posterior by truncating discrete distributions with infinite support. This reduces the problem to finite discrete inference, which is more amenable to exact techniques. We decided against this approach because Winner and Sheldon [26] already demonstrated its inferiority to GF-based exact inference on their graphical model. Moreover, it is harder to truncate general probabilistic programs, and even impossible for continuous priors.

Monte-Carlo inference Hence, we compare our approach with Monte Carlo inference methods. Specifically, we choose the Anglican [24] probabilistic programming system because it offers the best built-in support for discrete models with many state-of-the-art inference algorithms. Other popular systems are less suitable: Gen [6] specializes in programmable inference; Stan [3], Turing [8], and Pyro [1] mainly target continuous models; and WebPPL's [11] discrete inference algorithms are less extensive than Anglican's (e.g. no support for interacting particle MCMC).

Methodology The approximation error of a Monte Carlo inference algorithm depends on its settings 3 (e.g. the number of particles for SMC) and decreases with the number of samples the longer it is run. To ensure a fair comparison, we use the following setup: for each inference problem, we run several inference algorithms with various configurations (settings and sampling budgets) and

0.0

0.5

1.0

1.5

2.0

2.5

Time (in s)

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Error (TVD)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Original model

0

1

2

3

Time (in s)

0.0

0.1

0.2

0.3

0.4

Error (TVD)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Modified arrival rate

0

1

2

3

4

5

Time (in s)

0.0

0.1

0.2

0.3

0.4

Error (TVD)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Two-type branching process

report the approximation error and the elapsed time. To measure the quality of the approximation, we report the total variation distance (TVD) between the exact solution and the approximate posterior distribution, as well as the approximation errors of the posterior mean µ , standard deviation σ , skewness (third standardized moment) S , and kurtosis K . To ensure invariance of our error metrics under translation and scaling of the distribution, we compute the error of the mean as | ˆ µ -µ | σ , where µ is the true posterior mean, ˆ µ its approximation, and σ the true posterior standard deviation; the relative error of the standard deviation σ (because σ is invariant under translation but not scaling), and the absolute error of skewness and kurtosis (because they are invariant under both translation and scaling). To reduce noise, we average all these error measures and the computation times over 20 runs and report the standard error as error bars. We run several well-known inference algorithms implemented in Anglican: importance sampling (IS), Lightweight Metropolis-Hastings (LMH), Random Walk Metropolis-Hastings (RMH), Sequential Monte Carlo (SMC), Particle Gibbs (PGibbs), and interacting particle MCMC (IPMCMC). For comparability, in all experiments, each algorithm is run with two sampling budgets and, if possible, two different settings (one being the defaults) for a total of four configurations. The sampling budgets were 1000 or 10000, because significantly lower sample sizes gave unusable results and significantly higher sample sizes took much more time than our exact method. We discard the first 20% of the samples, a standard procedure called 'burn-in'.

Note that this setup is generous to the approximate methods because we only report the average time for one run of each configuration. However, in practice, one does not know the best configuration, so the algorithms need to be run several times with different settings. By contrast, our method requires only one run because the result is exact.

5.3 Benchmarks with infinite-support distributions

Population ecology Our first benchmark comes from [26, 27] and models animal populations. We have seen a simplified version in Example 2.1. Here we model a population N k at time steps k = 0 , . . . , m . At each time step, there is a Poisson-distributed number of new arrivals, which are added to the binomially distributed number of survivors from the previous time step. Each individual is observed with a fixed probability δ , so the number of observed individuals is binomially distributed:

where the model parameters λ k ∈ R , δ ∈ [0 , 1] are taken from [26]; the detection probability ρ is set to 0 . 2 ([26] considers a range of values, but we pick one for space reasons); and the observed number y k ∈ N of individuals in the population at time step k is simulated from the same ground truth as [26]. The goal is to infer the final number N m of individuals in the population. We set the population size model parameter and hence the observed values which influence the running time to be 4 times larger than the largest in [26] (see details in Appendix D.3). The results (Fig. 1a) show that our method is superior to MCMC methods in both computation time and accuracy since it is exact.

Modifications While this model was already solved exactly in [26], our probabilistic programming approach makes it trivial to modify the model, since the whole inference is automated and one only needs to change a few lines of the program: (a) we can model the possibility of natural disasters affecting the offspring rate with a conditional: Disaster ∼ Bernoulli (0 . 1); if Disaster = 1 { New k ∼ Poisson ( λ ′ ) } else { New k ∼ Poisson ( λ ) } , or (b) instead of a single population, we can model populations of two kinds of individuals that interact, i.e. a multitype branching process (see Fig. 1c). None

0

5

10

15

20

25

Time (in s)

0.0

0.2

0.4

0.6

0.8

Error (TVD)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Switchpoint: TVD error

0

10

20

30

40

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

Error (TVD)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Mixture: TVD error

0

20

40

60

result

0.000

0.025

0.050

0.075

0.100

0.125

0.150

probability

PGibbs (10000 samples)

ours (exact)

(c) Mixture: histogram

0

2

4

6

8

10

12

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

Error (TVD)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) HMM: TVD error

Figure 2: Plots for the switchpoint, mixture, and hidden Markov model.

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the std. dev.

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

of these modifications can be handled by [26] or [27]. The results of the first modification (Fig. 1b) are very similar. The more complex second modification takes longer to solve exactly, but less time than approximate inference with 10000 samples, and achieves zero error.

Switchpoint model Our second benchmark is Bayesian switchpoint analysis, which is about detecting a change in the frequency of certain events over time. We use the model from [23] with continuous priors and its 111 real-world data points about the frequency of coal-mining accidents. We compare both the moment errors (Fig. 3) and the TVD errors (Fig. 2a). In both cases, the approximations are less accurate and take longer than our exact method.

Mixture model We consider a binary mixture model on the same data set, with equal mixture weights and a geometric prior for the rates: each data point is observed from a mixture of two Poisson distributions with different rates and the task is to infer these rates. Due to their multimodality, mixture models are notoriously hard for approximate inference methods, which is confirmed in Fig. 2b. Even the runs with the lowest error cover only one of the two modes (cf. the sample histogram in Fig. 2c).

Hidden Markov model We use a hidden Markov model based on [22, Section 2.2], but involving infinite (geometric) priors. It is a two-state system with known transition probabilities and the rate for the observed data depends on the hidden state. We run this model on 30 simulated data points. For this model as well, our method clearly outperforms approximate methods (Fig. 2d).

To our knowledge, our approach is the first to find an exact solution to these problems, except the very first problem without the modifications, which appeared in [26]. For brevity, we only presented the most important aspects of these benchmarks, relegating their encoding as probabilistic programs to Appendix D.3. Code and reproduction instructions are provided in the supplementary material.

6 Conclusion

By leveraging generating functions, we have developed and proven correct a framework for exact Bayesian inference on discrete models, even with infinite support and continuous priors. We have demonstrated competitive performance on a range of models specified in an expressive probabilistic programming language, which our tool Genfer processes automatically.

Future work It is a natural question how our method could be integrated with a more general probabilistic programming system. For example, in a sampling-based inference algorithm, one could imagine using generating functions to solve subprograms exactly if this is possible. More generally, it would be desirable to explore how the compositionality of the GF translation can be improved. As it stands, our GF translation describes the joint distribution of all program variables - we never reason 'locally' about a subset of the variables. A compositional approach would likely facilitate the application of the GF method to functional probabilistic languages like Anglican.

Acknowledgments and Disclosure of Funding

Wewould like to thank Maria Craciun, Hugo Paquet, Tim Reichelt, and Dominik Wagner for providing valuable input on this work. We are especially grateful to Mathieu Huot for insightful discussions and very helpful feedback on a draft of this paper.

This research was supported by the Engineering and Physical Sciences Research Council (studentship 2285273, grant EP/T006579) and the National Research Foundation, Singapore, under its RSS Scheme (NRF-RSS2022-009). For the purpose of Open Access, the authors have applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.

References

E. Bingham, J. P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. A. Szerlip, P. Horsfall, and N. D. Goodman. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res. , 20:28:1-28:6, 2019. URL ❤/a116/a116♣✿✴✴❥♠❧/a114✳♦/a114❣✴♣❛♣❡/a114/a115✴✈✷✵✴✶✽✲✹✵✸✳ ❤/a116♠❧ .

R. P. Brent and H. T. Kung. Fast algorithms for manipulating formal power series. J. ACM , 25(4): 581-595, 1978. doi: 10.1145/322092.322099. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✷✷✵✾✷✳ ✸✷✷✵✾✾ .

B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. Stan : A probabilistic programming language. Journal of Statistical Software , 76(1), 1 2017. ISSN 1548-7660. doi: 10.18637/jss.v076.i01. URL ❤/a116/a116♣/a115✿✴✴✇✇✇✳♦/a115/a116✐✳❣♦✈✴❜✐❜❧✐♦✴✶✹✸✵✷✵✷ .

M. Chen, J. Katoen, L. Klinkenberg, and T. Winkler. Does a program yield the right distribution? - verifying probabilistic programs via generating functions. In S. Shoham and Y. Vizel, editors, Computer Aided Verification - 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part I , volume 13371 of Lecture Notes in Computer Science , pages 79-101. Springer, 2022. doi: 10.1007/978-3-031-13185-1_5. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳ ✶✵✵✼✴✾✼✽✲✸✲✵✸✶✲✶✸✶✽✺✲✶❴✺ .

Y. Choi, A. Vergari, and G. Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models, 2020. URL ❤/a116/a116♣✿✴✴/a115/a116❛/a114❛✐✳❝/a115✳✉❝❧❛✳❡❞✉✴♣❛♣❡/a114/a115✴ /a80/a114♦❜❈✐/a114❝✷✵✳♣❞❢ . Accessed: 2023-10-22.

M. F. Cusumano-Towner, F. A. Saad, A. K. Lew, and V. K. Mansinghka. Gen: a generalpurpose probabilistic programming system with programmable inference. In ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2019 . ACM, 2019. doi: 10.1145/3314221.3314642.

W. Feller. An introduction to probability theory and its applications, volume 1 . Wiley Series in Probability and Statistics. John Wiley & Sons, Nashville, TN, 3 edition, jan 1968.

H. Ge, K. Xu, and Z. Ghahramani. Turing: A language for flexible probabilistic inference. In A. Storkey and F. Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pages 1682-1690. PMLR, 09-11 Apr 2018. URL ❤/a116/a116♣/a115✿ ✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♠❧/a114✳♣/a114❡/a115/a115✴✈✽✹✴❣❡✶✽❜✳❤/a116♠❧ .

T. Gehr, S. Misailovic, and M. T. Vechev. PSI: exact symbolic inference for probabilistic programs. In S. Chaudhuri and A. Farzan, editors, Computer Aided Verification - 28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part I , volume 9779 of Lecture Notes in Computer Science , pages 62-83. Springer, 2016. doi: 10. 1007/978-3-319-41528-4_4. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✸✲✸✶✾✲✹✶✺✷✽✲✹❴✹ .

Z. Ghahramani. Probabilistic machine learning and artificial intelligence. Nat. , 521(7553):452459, 2015. doi: 10.1038/nature14541. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✸✽✴♥❛/a116✉/a114❡✶✹✺✹✶ .

N. D. Goodman and A. Stuhlmüller. The Design and Implementation of Probabilistic Programming Languages. ❤/a116/a116♣✿✴✴❞✐♣♣❧✳♦/a114❣ , 2014. Accessed: 2023-4-6.

A. D. Gordon, T. A. Henzinger, A. V. Nori, and S. K. Rajamani. Probabilistic programming. In J. D. Herbsleb and M. B. Dwyer, editors, Proceedings of the on Future of Software Engineering, FOSE 2014, Hyderabad, India, May 31 - June 7, 2014 , pages 167-181. ACM, 2014. doi: 10.1145/2593882.2593900. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✷✺✾✸✽✽✷✳✷✺✾✸✾✵✵ .

A. Griewank and A. Walther. Evaluating derivatives - principles and techniques of algorithmic differentiation, Second Edition . SIAM, 2008. ISBN 978-0-89871-659-7. doi: 10.1137/1. 9780898717761. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✸✼✴✶✳✾✼✽✵✽✾✽✼✶✼✼✻✶ .

A. Gut. Probability: A Graduate Course . Springer Texts in Statistics. Springer, New York, second edition, 2013. ISBN 978-1-4614-4707-8; 978-1-4614-4708-5. doi: 10.1007/ 978-1-4614-4708-5. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴✾✼✽✲✶✲✹✻✶✹✲✹✼✵✽✲✺ .

J. Harviainen, V. P. Ramaswamy, and M. Koivisto. On inference and learning with probabilistic generating circuits. In R. J. Evans and I. Shpitser, editors, Uncertainty in Artificial Intelligence, UAI 2023, July 31 - 4 August 2023, Pittsburgh, PA, USA , volume 216 of Proceedings of Machine Learning Research , pages 829-838. PMLR, 2023. URL ❤/a116/a116♣/a115✿✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♠❧/a114✳♣/a114❡/a115/a115✴ ✈✷✶✻✴❤❛/a114✈✐❛✐♥❡♥✷✸❜✳❤/a116♠❧ .

S. Holtzen, G. V. den Broeck, and T. D. Millstein. Scaling exact inference for discrete probabilistic programs. Proc. ACM Program. Lang. , 4(OOPSLA):140:1-140:31, 2020. doi: 10.1145/3428208. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✹✷✽✷✵✽ .

L. Klinkenberg, K. Batz, B. L. Kaminski, J. Katoen, J. Moerman, and T. Winkler. Generating functions for probabilistic programs. In M. Fernández, editor, Logic-Based Program Synthesis and Transformation - 30th International Symposium, LOPSTR 2020, Bologna, Italy, September 7-9, 2020, Proceedings , volume 12561 of Lecture Notes in Computer Science , pages 231-248. Springer, 2020. doi: 10.1007/978-3-030-68446-4\_12. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✵✼✴ ✾✼✽✲✸✲✵✸✵✲✻✽✹✹✻✲✹❴✶✷ .

L. Klinkenberg, C. Blumenthal, M. Chen, and J. Katoen. Exact bayesian inference for loopy probabilistic programs. CoRR , abs/2307.07314, 2023. doi: 10.48550/arXiv.2307.07314. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✹✽✺✺✵✴❛/a114❳✐✈✳✷✸✵✼✳✵✼✸✶✹ .

D. Kozen. Semantics of probabilistic programs. J. Comput. Syst. Sci. , 22(3):328-350, 1981. doi: 10.1016/0022-0000(81)90036-2. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✵✶✻✴✵✵✷✷✲✵✵✵✵✭✽✶✮ ✾✵✵✸✻✲✷ .

N. D. Matsakis and F. S. K. II. The rust language. In M. Feldman and S. T. Taft, editors, Proceedings of the 2014 ACM SIGAda annual conference on High integrity language technology, HILT 2014, Portland, Oregon, USA, October 18-21, 2014 , pages 103-104. ACM, 2014. doi: 10.1145/2663171.2663188. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✷✻✻✸✶✼✶✳✷✻✻✸✶✽✽ .

P. Narayanan, J. Carette, W. Romano, C. Shan, and R. Zinkov. Probabilistic inference by program transformation in hakaru (system description). In O. Kiselyov and A. King, editors, Functional and Logic Programming - 13th International Symposium, FLOPS 2016, Kochi, Japan, March 4-6, 2016, Proceedings , volume 9613 of Lecture Notes in Computer Science , pages 62-79. Springer, 2016. doi: 10.1007/978-3-319-29604-3_5. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴ ✶✵✳✶✵✵✼✴✾✼✽✲✸✲✸✶✾✲✷✾✻✵✹✲✸❴✺ .

F. A. Saad, M. C. Rinard, and V. K. Mansinghka. SPPL: probabilistic programming with fast exact symbolic inference. In S. N. Freund and E. Yahav, editors, PLDI '21: 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, Virtual Event, Canada, June 20-25, 2021 , pages 804-819. ACM, 2021. doi: 10.1145/3453483.3454078. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴✶✵✳✶✶✹✺✴✸✹✺✸✹✽✸✳✸✹✺✹✵✼✽ .

J. Salvatier, T. V. Wiecki, and C. Fonnesbeck. Probabilistic programming in python using pymc3. PeerJ Comput. Sci. , 2:e55, 2016. doi: 10.7717/peerj-cs.55. URL ❤/a116/a116♣/a115✿✴✴❞♦✐✳♦/a114❣✴ ✶✵✳✼✼✶✼✴♣❡❡/a114❥✲❝/a115✳✺✺ .

D. Tolpin, J. van de Meent, and F. D. Wood. Probabilistic programming in anglican. In European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2015 , volume 9286 of LNCS . Springer, 2015. doi: 10.1007/978-3-319-23461-8\_36.

J. van de Meent, B. Paige, H. Yang, and F. Wood. An introduction to probabilistic programming. CoRR , abs/1809.10756, 2018. URL ❤/a116/a116♣✿✴✴❛/a114①✐✈✳♦/a114❣✴❛❜/a115✴✶✽✵✾✳✶✵✼✺✻ .

K. Winner and D. Sheldon. Probabilistic inference with generating functions for poisson latent variable models. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 2640-2648, 2016. URL ❤/a116/a116♣/a115✿✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♥❡✉/a114✐♣/a115✳❝❝✴♣❛♣❡/a114✴✷✵✶✻✴❤❛/a115❤✴ ✻❝✶❞❛✽✽✻✽✷✷❝✻✼✽✷✷❜❝❢✸✻✼✾❞✵✹✸✻✾❢❛✲❆❜/a115/a116/a114❛❝/a116✳❤/a116♠❧ .

K. Winner, D. Sujono, and D. Sheldon. Exact inference for integer latent-variable models. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 3761-3770. PMLR, 2017. URL ❤/a116/a116♣✿ ✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♠❧/a114✳♣/a114❡/a115/a115✴✈✼✵✴✇✐♥♥❡/a114✶✼❛✳❤/a116♠❧ .

H. Zhang, B. Juba, and G. V. den Broeck. Probabilistic generating circuits. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 12447-12457. PMLR, 2021. URL ❤/a116/a116♣✿✴✴♣/a114♦❝❡❡❞✐♥❣/a115✳♠❧/a114✳♣/a114❡/a115/a115✴ ✈✶✸✾✴③❤❛♥❣✷✶✐✳❤/a116♠❧ .

A Details on the Probabilistic Programming Language SGCL

A.1 A minimal grammar

A minimal grammar of SGCL looks like this:

where a 1 , . . . , a n , c ∈ R ≥ 0 , p ∈ [0 , 1] are literals, and A is a finite subset of N . The X k for k = 1 , . . . , n are variables that can take on values in R ≥ 0 . To explain, skip leaves everything unchanged, P 1 ; P 2 runs P 1 and then P 2 , and X k := a ⊤ X + c performs an affine transformation where a ∈ R n ≥ 0 and c ∈ R ≥ 0 (to ensure that the support of X k is a subset of R ≥ 0 ). The if construct allows branching based on comparisons of a variable X k with constant natural numbers, with the requirement that X k be a discrete variable on N , and X k ∼ D samples X k from a primitive distribution (with constant parameters)

whereas X k ∼ D ( X j ) samples X k from a compound distribution (with a parameter that is again a variable)

A.2 Syntactic sugar

For convenience, we add some syntactic sugar, which is expanded as follows:

A.3 Standard measure semantics

Notation and Conventions Throughout the text, we always assume that N carries the discrete σ -algebra P ( N ) and R the Borel σ -algebra B ( R ) . Products of these spaces (e.g. N × R ) are endowed with the product σ -algebra. In the following, we write x for ( x 1 , . . . , x n ) and use the update notation x [ i ↦→ a ] to denote ( x 1 , . . . , x i -1 , a, x i +1 , . . . , x n ) . In fact, we even write updates x [ i 1 ↦→ a 1 , . . . , i k ↦→ a k ] , which stands for x [ i 1 ↦→ a 1 ] · · · [ i k ↦→ a k ] . Finally, we write 1 k for the k -tuple (1 , . . . , 1) and similarly for 0 k . We write 0 for the zero measure or the constant zero function. We also use Iverson brackets [ ϕ ] for a logical condition ϕ : [ ϕ ] is 1 if ϕ is satisfied and 0 otherwise. For example, the indicator function 1 S of a set S can be defined using Iverson brackets: 1 S ( x ) = [ x ∈ S ] . For a measure µ on R n , we define µ X k ∈ A ( S ) = µ ( { X ∈ S | X k ∈ A } ) .

Standard transformer semantics The standard semantics for such a probabilistic program uses distribution transformers [19]. Since our language includes conditioning, the distribution of program states will not always be a probability distribution, but a subprobability distribution µ on R n ≥ 0 for n variables, i.e. a function µ : B ( R n ≥ 0 ) → [0 , 1] with µ ( R n ≥ 0 ) ≤ 1 . The semantics of a program P is thus a measure transformer J P K std : Meas( R k ≥ 0 ) → Meas( R k ≥ 0 ) . We assume that the initial measure

is the probability distribution where every variable X k is 0 with probability 1, i.e. X ∼ Dirac ( 0 n ) . The semantics is shown in Fig. 4 where X is a vector of length n .

The first 5 rules were essentially given by [19], with slightly different notation and presentation. Kozen defines some of the measures on rectangles ( S = S 1 ×···× S n ) only, which uniquely extends to a measure on all sets in the product σ -algebra. By contrast, we directly define the measure on all sets in the σ -algebra using integrals.

We quickly explain the rules. The skip command does nothing, so the distribution µ of the program is unchanged. Chaining commands P 1 ; P 2 has the effect of composing the corresponding transformations of µ . Affine transformations of program variables yield a measure of S that is given by the original measure of the preimage of the event S under the affine map. For conditionals if X k ∈ A { P 1 } else { P 2 } , we consider only outcomes satisfying X k ∈ A in the first branch and the complement in the second branch and then sum both possibilities. Sampling from distributions X k ∼ D is a bit more complicated. Note that µ ( S ) = ∫ [ X ∈ S ] d µ ( X ) . To obtain the measure after sampling X k ∼ D , we care about the new state X [ k ↦→ Y ] being in S , where Y has distribution D . This is exactly expressed by the double integral. The rule for compound distributions X k ∼ D ( X j ) is essentially the same as the one for distributions with constant parameters X k ∼ D , except the dependency on X j needs to be respected.

The two new rules (not derivable from Kozen's work) are for fail and the implicit normalization normalize ( µ ) as a final step at the end of a program. The rule for fail departs from Kozen's semantics in the sense that it yields a subprobability distribution (i.e. the total mass can be less than 1). Intuitively, fail 'cuts off' or assigns the probability 0 to certain branches in the program. This is called hard conditioning in the PPL literature. For example, if X 1 = 5 { skip } else { fail } has the effect of conditioning on the event that X 1 = 5 because all other scenarios are assigned probability 0.

The normalization construct corresponds to the division by the evidence in Bayes' rule and scales the subprobability distribution of a subprogram P back to a probability distribution. The semantics is µ ∫ d µ ( X ) , so the distribution µ is scaled by the inverse of the total probability mass of µ , thus normalizing it.

B Details on Generating Functions

An exhaustive list of the supported distributions and their generating functions can be found in Tables 4a and 4b.

̸

̸

B.1 Generating function semantics

The full generating function semantics can be found in Fig. 5. The first five rules in Fig. 5 were presented in [17, 4]. The GF for sampling from Binomial ( X j , p ) was first given in [26]. Chen et al. [4] also implicitly describe the GF semantics of sampling from Binomial ( X j , p ) and NegBinomial ( X j , p ) by expressing it as summing X j iid samples from Bernoulli ( p ) and Geometric ( p ) , respectively.

Let us discuss each rule in detail. The skip command leaves the distribution and hence the generating function unchanged. Chaining commands P 1 ; P 2 is again achieved by composing the individual transformers for each command. To obtain some intuition for the affine transformation case X k := a ⊤ X + c , suppose we only have 2 variables: X 1 := 2 X 1 +3 X 2 +5 . The idea is that the transformed generating function is given by J X 1 := 2 X 1 + 3 X 2 + 5 K gf ( G )( x ) = E [ x 2 X 1 +3 X 2 +5 1 x X 2 2 ] = x 5 1 E [( x 2 1 ) X 1 ( x 3 1 x 2 ) X 2 ] = x 5 1 · G ( x 2 1 , x 3 1 x 2 ) .

The semantics of if X k ∈ A { P 1 } else { P 2 } uses G X k ∈ A . If G is the generating function for a measure µ then G X k ∈ A is the generating function for the measure µ X k ∈ A ( S ) := µ ( { x ∈ S | x k ∈ A } ) . To get intuition for this, remember that for discrete variables X 1 , X 2 , we have G ( x 1 , x 2 ) = ∑ X 1 ,X 2 µ ( { ( X 1 , X 2 ) } ) · x X 1 1 x X 2 2 . If we want to obtain the generating function for X 1 = 5 , we want to keep only terms where x 1 has exponent 5. In other words, we need the Taylor coefficient ∂ 5 1 G (0 ,x 2 ) 5! . This should give some intuition for G X k ∈ A . The semantics of if X k ∈ A { P 1 } else { P 2 } transforms G X k ∈ A in the then-branch and the complement G -G X k ∈ A in the else branch, summing the two possibilities.

For sampling, say X 1 ∼ D , we first marginalize out the old X 1 by substituting 1 for x 1 in G : E [ x X 2 2 ] = E [1 X 1 x X 2 2 ] = G (1 , x 2 ) . Then we multiply with the GF of the new distribution D . The rules for compound distributions D ( X j ) are more involved, so we offer the following intuition.

For the compound distributions D ( n ) = Binomial ( n, p ) , NegBinomial ( n, p ) , Poisson ( λ · n ) , we make essential use of their property that gf ( D ( n ))( x ) = ( gf ( D (1))( x )) n . So if G ( x ) := E [ x X ] is the GF of X and Y ∼ D ( X ) , then E X,Y [ x X y Y ] = E X [ x X E Y ∼ D ( X ) [ y Y ]] = E X [ x X gf ( D ( X ))( y )] = E X [ x X ( gf ( D (1))( y )) X ] = E [( x · gf ( D (1))( y )) X ] = G ( x · gf ( D (1))( y )) . This is not a fully formal argument, but it should give some intuition for the rule.

For D ( p ) = Bernoulli ( p ) , the intuition is that if G ( x ) := E [ x X ] is the GF of X and if Y ∼ Bernoulli ( X ) , then H ( x, y ) = E [ x X y Y ] = E [ x X ((1 -X ) y 0 + Xy 1 )] = E [ x X -( y -1) Xx X ] = G ( x ) -( y -1) xG ′ ( x ) .

The GF semantics of fail is the zero function because its distribution is zero everywhere. For normalization, note that substituting 1 n in the generating function computes the marginal probabilities, similarly to the substitution of 1 in the semantics of sampling. It scales the GF of the subprogram P by the inverse of the normalizing constant obtained by that substitution.

We hope these explanations give some intuition for the GF semantics, but of course, proof is required.

B.2 Correctness proof

For the correctness proof, we need to describe the set of x ∈ R n where the generating function is defined. For R > 0 and a measure µ on R n ≥ 0 , define it to be the product of open intervals

where Q i = -R if the i -th component of µ is supported on N (i.e. µ ( S ) = µ ( { X ∈ S | X i ∈ N } ) for any measurable S ) and Q i = 0 otherwise. The case split is necessary because the generating functions are better-behaved, and thus have a larger domain, if the measure is supported on N . In fact, it is important that x i = 0 is possible in the N -case because otherwise we would not be able to extract probability masses from the GF.

Theorem B.1 (Correctness) . The GF semantics is correct w.r.t. the standard semantics J -K std : for any SGCL program P and subprobability distribution µ on R n ≥ 0 , we have J P K gf ( gf ( µ )) = gf ( J P K std ( µ )) In particular, it correctly computes the GF of the posterior distribution of P as normalize ( P ( 1 ))

If gf ( µ ) is defined on T ( R,µ ) for some R > 1 then J P K gf ( gf ( µ )) is defined on T ( R , J P K std ( µ for some R ′ > 1 . In particular, the GFs J P K gf ( 1 ) and thus normalize ( J P K gf ( 1 )) are defined on T ( R, J P K std ( µ )) for some R > 1 .

. J K gf . ′ ))

Proof. As usual for such statements, we prove this by induction on the structure of the program. The program P can take one of the following forms:

Skip. For the trivial program, we have P gf ( G ) = G and P std ( µ ) = µ , so the claim is trivial.

J K J K Chaining. If P is P 1 ; P 2 , we know by the inductive hypothesis that J P 1 K gf ( gf ( µ )) = gf ( J P 1 K std ( µ )) and similarly for P 2 . Taking this together, we find by the definitions of the semantics:

Affine assignments. If P is X k := a ⊤ X + c , we have J P K std ( µ )( S ) = µ ( { X ∈ R n | X [ k ↦→ a ⊤ X + c ] ∈ S } ) and thus

The claim about the domain of J P K gf ( G ) follows directly from the inductive hypotheses.

̸

The claim that it is defined on some T ( R ′ , J P K std ( µ )) holds if we choose R ′ = min( √ R, min n i =1 2 a i √ R ) > 1 because then | x i x a i k | < √ R ( 2 a i √ R ) a i ≤ R for i = k and | x a k k | < ( 2 a k √ R ) a k ≤ √ R ≤ R .

Conditionals. If P is if X k ∈ A { P 1 } else { P 2 } , the GF semantics defines ( gf ( µ )) X k ∈ A , which is the same as gf ( µ X k ∈ A ) by Lemma B.2. Using this fact, the linearity of gf ( -) , and the induction hypothesis, we find

Sampling. If P is X k ∼ D , we find:

Also by Lemma B.2, we know that ( gf ( µ )) X k ∈ A (and thus ( gf ( µ ) -( gf ( µ )) X k ∈ A ) ) is defined on T ( R,µ ) , so by the induction hypothesis, both J P 1 K gf (( gf ( µ )) X k ∈ A ) and J P 2 K gf ( gf ( µ ) -gf ( µ X k ∈ A )) are defined on some T ( R ′ ) . Hence the same holds for J P K gf ( gf ( µ )) .

Regarding the domain of J P K gf ( gf ( µ )) , note that gf ( D ) is defined on all of R for any finite distribution and for the Poisson distribution (cf. Table 4a). For D ∈ { Geometric ( p ) , NegBinomial ( n, p ) } , the GF gf ( D ) is defined on ( -1 1 -p , 1 1 -p ) , so we can pick R ′ := min( R, 1 1 -p ) > 1 . The GF of

Exponential ( λ ) is defined for log x < λ , i.e. x < exp( λ ) , so we can pick R ′ := min( R,e λ ) > 1 . The GFof Gamma ( α, β ) is defined for log x < β , i.e. x < exp( β ) , so we can pick R ′ := min( R,e β ) > 1 . The GF of Uniform [ a, b ] is defined on R , so R ′ := R works.

Compound distributions. If P is X k ∼ D ( X j ) , we use the fact that gf ( D ( n ))( x ) = ( gf ( D (1))( x )) n for D ( n ) ∈ { Binomial ( n, p ) , NegBinomial ( n, p ) , Poisson ( λ · n ) } , which can easily be checked by looking at their generating functions (cf. Table 4b).

Regarding the domain of J P K gf ( gf ( µ )) , we have to ensure that | x j · gf ( D (1))( x k ) | < R . For the Binomial (1 , p ) distribution, we choose R ′ = √ R > 1 such that | x j || 1 -p + px k | < √ R (1 -p + p √ R ) ≤ √ R ((1 -p ) √ R + p √ R ) < R . For the NegBinomial (1) distribution, we choose R ′ = min( √ R, √ R -p √ R (1 -p ) ) = min( √ R, 1 + p ( √ R -1) √ R -p √ R ) > 1 because for any x ∈ ( -R ′ , R ) , we have

and thus ∣ ∣ ∣ x j p 1 -(1 -p ) x k ∣ ∣ ∣ < R . For the Poisson ( λ ) distribution, we choose R ′ = min( √ R, 1 + log( R ) 2 λ ) > 1 because then | x j · exp( λ ( x k -1)) | < √ R exp( λ log( R ) 2 λ ) = R . For the Bernoulli distribution D ( X j ) = Bernoulli ( X j ) , we reason as follows:

Note that the interchange of integral and differentiation is allowed by Lemma B.3. It also implies that J P K gf ( gf ( µ ))( x ) is defined for x j ∈ (0 , R ) by Lemma B.3.

Fail. If P is fail , then J P K std ( µ ) is the zero measure and J P K gf ( gf ( µ )) is the zero function, so the claim holds trivially. This GF is clearly defined everywhere.

Normalization. For normalization, we make use of the linearity of gf ( -) .

Furthermore, if gf ( µ ) is defined on T ( R,µ ) , then so is normalize ( gf ( µ )) on T ( R, normalize ( µ )) = T ( R,µ ) .

This finishes the induction and proves the claim.

Lemma B.2. Let A ⊂ N be a finite set. Let µ be a measure on R n such that its k -th component is supported on N , i.e. µ ( S ) = µ ( { X ∈ S | X k ∈ N } ) for all measurable S ⊆ R ≥ 0 . Define µ X k ∈ A ( S ) := µ ( { X ∈ S | X k ∈ A } ) . Let G := gf ( µ ) . Then

Furthermore, if G is defined on T ( R,µ ) then so is G X k ∈ A on T ( R,µ X k ∈ A ) = T ( R,µ ) .

Proof. Since µ X k ∈ A = ∑ i ∈ A µ X k = i where we write µ X k = i := µ X k ∈{ i } , we have:

The reason for the last step is that for the function f ( x ) = x l , we have

Since 0 l -i is 0 for l > i and 1 for l = i , we find that f ( i ) (0) = [ i = l ] · i ! . Above, we used this fact with l = X k and x = x k . We continue:

as desired. Note that the integral and differentiation operators can be interchanged by Lemma B.3.

Lemma B.3. If the integral ∫ x X d µ ( X ) for a measure µ on R n ≥ 0 is defined for all x ∈ T ( R,µ ) , we have

and both sides are defined for all x ∈ T ( R,µ ) .

Furthermore, the right-hand side has the form ∫ p ( X ) x X -w d µ ( X ) for a polynomial p in n variables and w ∈ N n , with the property that p ( X ) = 0 whenever X j < w j for some j , in which case we define p ( X ) x X -w := 0 , even if x X -w is undefined.

Proof. The proof is by induction on m . If m = 0 then the statement is trivial. For the induction step m> 0 , we may assume that

which establishes the induction goal with the polynomial ˜ p ( X ) := ( X i 1 -w i 1 ) p ( X ) and ˜ w := w [ i 1 ↦→ w i 1 +1] . The property ˜ p ( X ) = 0 whenever X j < ˜ w j for some j still holds due to the added factor ( X i 1 -w i 1 ) taking care of the case X i 1 = w i 1 = ˜ w i 1 -1 .

Lemma B.4. Let w ∈ N n , p be a polynomial in n variables, and µ a measure on R n ≥ 0 . We define p ( X ) x X -w := 0 whenever p ( X ) = 0 , even if x j ≤ 0 and X j < w j for some j . Suppose p ( X ) x X -w is defined µ -almost everywhere and µ -integrable for all x ∈ T ( R,µ ) . Then

holds and is defined for all x ∈ T ( R,µ ) .

Proof. The proof is about verifying the conditions of the Leibniz integral rule. The nontrivial condition is the boundedness of the derivative by an integrable function. The proof splits into two parts, depending on whether the i -th component of µ is supported on N . If it is, then x i ∈ ( -R,R ) by the definition of T ( R,µ ) and can be nonpositive, but X i -w i < 0 and p ( X ) = 0 happens µ -almost never. Otherwise, we have x ∈ (0 , R ) and it is thus guaranteed to be positive, but X i -w i may be negative. Hence the two cases need slightly different treatment.

̸

We first deal with the case that the i -th component is not supported on N . Let 0 < ϵ < R ′ < R ′′ < R . We first prove ∂ ∂x i ∫ p ( X ) x X d µ ( X ) = ∫ ∂ ∂x i p ( X ) x X d µ ( X ) for any x i ∈ ( ϵ, R ′ ) , using the Leibniz integral rule. For this purpose, we need to bound the derivative:

If X i > w i , we can choose M > 1 sufficiently large such that | X i -w i | x X i -w i i ≤ ( X i -w i ) R ′ X i -w i ≤ R ′′ X i -w i whenever | X i -w i | > M . (Such an M can be found because the exponential function y ↦→ ( R ′′ R ′ ) y grows more quickly than the linear function y ↦→ y .) If X i -w i ≤ M

then we also have | X i -w i | x X i -w i i ≤ M · R ′′ X i -w i . So the derivative is thus bounded on the set { X | X i > w i } by M ϵ · | p ( X ) | ( x [ i ↦→ R ′′ ]) X -w , which is integrable by assumption.

On the complement set { X | 0 ≤ X i ≤ w i } , we find | X i -w i | x X i -w i i ≤ | X i -w i | ϵ X i -w i ≤ w i ϵ X i -w i . Hence the derivative is bounded by w i ϵ · | p ( X ) | ( x [ i ↦→ ϵ ]) X -w , which is again integrable by assumption. So the derivative is bounded by an integrable function and thus, by the Leibniz integral rule, interchanging differentiation and the integral is valid for all x i ∈ ( ϵ, R ′ ) and thus for all x i ∈ (0 , R ) .

̸

Next, consider the case that the i -th component of µ is supported on N . Let 0 < R ′ < R ′′ < R . Note that the set { X | X i < w i ∧ p ( X ) = 0 } has measure zero because otherwise the integrand p ( X ) x X -w would not be defined for x i ≤ 0 due to the negative exponent X i -w i . Hence, for µ -almost all X , we have p ( X ) = [ X ≥ w ] p ( X ) and can bound the partial derivative for | x i | < R ′ :

because the factor | X i -w i | vanishes for X i = w i and thus ensures that negative values for the exponent X -w [ i ↦→ w i +1] don't matter, so that we can use the monotonicity of | x | X -w [ i ↦→ w i +1] in | x | .

Similarly to the first case, for a sufficiently large M > max(1 , w i ) , we have | X i -w i | R ′ X i -w i ≤ R ′′ X i -w i whenever X i -w i > M . If X i -w i ≤ M then [ X ≥ w ] | X i -w i | ≤ M and we also have [ X ≥ w ] | X i -w i | R ′ X i -w i ≤ M · R ′′ X i -w i . The derivative is thus bounded by M · | p ( X ) || x [ i ↦→ R ′′ ] | X -w , which is integrable by assumption. By the Leibniz integral rule, interchanging differentiation and the integral is valid for all x i ∈ ( -R ′ , R ′ ) and thus for all x i ∈ ( -R,R ) .

Remark B.1. As an example of what can go wrong with derivatives at zero if the measure is not supported on N , consider the Dirac measure µ = Dirac ( 1 2 ) . It has the generating function G ( x ) := gf ( µ )( x ) = √ x defined for x ∈ R ≥ 0 . Its derivative is G ′ ( x ) := 1 2 √ x , which is not defined for x = 0 .

B.3 Possible extensions to the probabilistic programming language

The syntax of our language guarantees that the generating function of the distribution of any probabilistic program admits a closed form. But are the syntactic restrictions necessary or are there other useful programming constructs that can be supported? We believe the following constructs preserve closed forms of the generating function and could thus be supported:

Additional distributions with constant parameters: any such distribution could be supported as long as its GF is defined on [0 , 1 + ϵ ) for some ϵ > 0 and its derivatives can be evaluated.

Additional compound distributions: we spent a lot of time trying to find additional compound distributions with a closed-form GF since this would be the most useful for probabilistic models, but we were largely unsuccessful. The only such distributions we found are Binomial ( m,X k ) , which could just be written as a sum of m iid Bernoulli ( X k ) -variables, and Gamma ( X k , β ) with shape parameter X k , which could be translated to a GF with the same idea as for Binomial ( X k , p ) , NegBinomial ( X k , p ) , and Poisson ( λ · X k ) .

Modulo, subtraction, iid sums: The event X k mod 2 = 0 , and the statements X k := max( X k -m, 0) (a form of subtraction that ensures nonnegativity), and X k := sum _ iid ( D,X l ) (i.e. X k is assigned the sum of X l independent random variables with distribution D ) can also be supported as shown in [4, 18].

Nonlinear functions on variables with finite support : if a variable has finite support, arbitrary functions on it could be performed by exhaustively testing all values in its domain.

Soft conditioning: another supportable construct could be score a X k for a ∈ [0 , 1] , which multiplies the likelihood of the current path by a X k . We think score q ( X k ) where q ( t ) := ∑ m i =0 q i t i is a polynomial with coefficients q i ∈ [0 , 1] and with q ≤ 1 on the domain of X k could also be supportable using similar techniques as for observations from Bernoulli ( X k ) .

None of these extensions seemed very useful for real-world models, in particular, they are not needed for our benchmarks. For this reason we did not formalize, implement, or prove them.

Support for loops is another big question. Bounded loops are easy enough to deal with by fully unrolling them. However exact inference for programs with unbounded loops seems extremely difficult, unless the inference algorithm is given additional information, e.g. a probabilistic loop invariant as in [4] or at least a loop invariant template with a few holes to be filled with real constants [18]. However, finding such a loop invariant for nontrivial programs seems exceedingly difficult. Furthermore, the variables of looping programs may have infinite expected values, so the generating function may not be definable at the point 1. But in our semantics, evaluating at 1 is very important for marginalization. For these reasons, we consider exact inference in the presence of loops an interesting but very hard research problem.

C Details on implementation and optimizations

Overview Our tool takes an SGCL program P with n variables as input, and a variable X i whose posterior distribution is to be computed. It translates it to the GF J P K gf ( 1 ) according to the GF semantics and normalizes it: G := normalize ( J P K gf ( 1 )) . This G is the GF of the posterior distribution of the program, from which it extracts the posterior moments of X i and its probability masses (if X i is discrete).

Computation of moments First, it computes the first four (raw) posterior moments from the factorial moments, which are obtained by differentiation of G :

From the raw moments, it computes the first four (centered/standardized) moments, i.e. the expected value ( µ := E [ X i ] ), the variance ( σ 2 := V [ X i ] ), the skewness ( E [( X i -µ ) 3 ] /σ 3 ) and the kurtosis ( E [( X i -µ ) 4 ] /σ 4 ):

Computation of probability masses If X i is a discrete variable, the tool computes all the probability masses P [ X i = m ] until the value of m such that the tail probabilities are guaranteed to be below a certain threshold, which is set to P [ X i ≥ m ] ≤ 1 256 . This is achieved by setting

by Markov's inequality. Note that in practice, the tail probabilities are typically much smaller than 1 256 , usually in the order of 10 -5 . For all k ∈ { 0 , . . . , m } , the tool computes the posterior probability P [ X i = k ] by marginalizing out all the other variables (substituting 1 for them) and computing the Taylor coefficient at x i = 0 :

It would be desirable to compute posterior densities for continuous distributions as well. In fact, there are mathematical ways of recovering the probability density function from a generating function via an inverse Laplace transform. However, this cannot be automated in practice because it requires solving integrals, which is intractable.

Implementation details Our tool Genfer is implemented in Rust [20], a safe systems programming language. The main reasons were low-level control and performance: the operations on the Taylor polynomials to evaluate derivatives of the generating function need to be fast and are optimized to exploit the structure of GFs arising from probabilistic programs (see Appendix C.3). C or C++ would have satisfied the performance criterion as well, but Rust's language features like memory safety, enums (tagged unions), and pattern matching made the implementation a lot more pleasant, robust, and easier to maintain. The first author's experience with Rust was another contributing factor.

The coefficients of the Taylor polynomials are stored in a multidimensional array provided by the ♥❞❛/a114/a114❛② library 4 . Arbitrary-precision floating-point numbers and unbounded rational numbers are provided by the /a114✉❣ library 5 , which is an interface to the GNU libraries GMP (for rationals) and MPFR (for floats). Our implementation is available on GitHub: github.com/fzaiser/genfer

C.1 Automatic differentiation and Taylor polynomials

The main difficulty in implementing the GF semantics is the computation of the (partial) derivatives. This seems to be a unique situation where we want to compute d -th derivatives where d is in the order of hundreds. The reason is that the observe X k = d construct is translated into a d -th (partial) derivative and if d is a real-world observation, it can be large. We have not come across another application of automatic differentiation that required derivatives of a total order of more than 10.

As a consequence, when we tried PyTorch , a mature machine learning library implementing automatic differentiation, the performance for derivatives of high order was very poor. Therefore, we decided to implement our own automatic differentiation framework. Our approach is to compute the Taylor expansion in all variables up to some order d of the generating function instead of a symbolic expression of the d -th derivative. This has the advantage of growing polynomially in d , not exponentially. Contrary to [27], it is advantageous to use Taylor coefficients instead of the partial derivatives because the additional factorial factors are can easily lead to overflows.

Taylor polynomials More formally, we define the Taylor polynomial Taylor d w ( G ) of a function G : R n → R at w ∈ R n of order d as the polynomial

where we used multi-index notation: | α | stands for α 1 + · · · + α n ; α ! for α 1 ! · · · α n ! ; and ∂ α for ∂ α 1 1 . . . ∂ α n n . The coefficients c α := 1 α ! ∂ α G ( w ) are the Taylor coefficients of G at w and are stored in a multidimensional array in our implementation.

Operations The operations on generating functions (Appendix B) are implemented in terms of their effect on the Taylor expansions. Let G,H : R n → R be two functions with Taylor expansions Taylor d w ( G ) = ∑ | α |≤ d g α ( x -w ) α and Taylor d w ( H ) = ∑ | α |≤ d h α ( x -w ) α . Addition F = G + H of two generating functions is implemented by adding the coefficients: Taylor d w ( G + H ) = ∑ | α |≤ d ( g α + h α )( x -w ) α . Scalar multiplication F = c · G is implemented by multiplying the coefficients: Taylor d w ( c · G ) = ∑ | α |≤ d ( c · g α )( x -w ) α . Multiplication F = G · H of two generating functions is implemented by the Cauchy product:

Division, exponentiation, logarithms, and powers can be implemented as well (see [13, Chapter 13] for details). Partial derivatives essentially correspond to shifting the index and a multiplication:

The most complicated case is composition/substitution, i.e. F ( x ) = G ( x [ k ↦→ H ( x )]) . For this, we let w ′ := w [ k ↦→ h 0 ] where Taylor d w ( H ) = ∑ | β |≤ d h β ( x -w ) β and we let Taylor d w ′ ( G ) = ∑ | α |≤ d g α ( x -w ′ ) α . Then we have (where 'h.o.t.' stands for 'higher order terms')

because by definition x [ k ↦→ h 0 ] -w ′ = ( x -w )[ k ↦→ 0] . This means that we can obtain Taylor d w ( F ) by substituting ∑ 1 ≤| β |≤ d h α ( x -w ) β for x k -w ′ k in Taylor d w ′ ( G ) .

Example C.1 (Calculations with Taylor polynomials) . Here we show how the expectation for Example 2.1 can be computed using Taylor polynomials. We use the same naming as in Example 3.1. Recall that to compute E [ X ] , we need to evaluate ∂ 1 E (1 , 1) , so we compute the Taylor expansion of E at (1 , 1) of order 1, i.e. Taylor 1 (1 , 1) ( E ) . To compute E ( x, y ) = D ( x,y ) D (1 , 1) , we need D ( x, y ) , so we need Taylor 1 (1 , 1) ( D ) . For D ( x, y ) = 1 2! y 2 ∂ 2 y C ( x, 0) , we need the Taylor expansion of C at (1 , 0) of order 1 + 2 = 3 . For C ( x, y ) = B ( x (0 . 1 y + 0 . 9) , 1) , we need the Taylor expansion of B at (1(0 . 1 · 0 + 0 . 9) , 1) = (0 . 9 , 1) of order 3. Since B ( x, y ) = exp(20( x -1)) = exp(20(( x -0 . 9) -0 . 1)) = exp(20( x -0 . 9) -2) , this Taylor expansion is

Since C (1 + ( x -1) , 0 + y ) = B ((1 + ( x -1))(0 . 1 y +0 . 9) , 1) = B (0 . 9 + 0 . 9( x -1) + 0 . 1 y + 0 . 1( x -1) y, 1) , the Taylor expansion of C at (1 , 0) is given by replacing ( x -0 . 9) with 0 . 9( x -1) + 0 . 1 y +0 . 1( x -1) y and we get

For D ( x, y ) = 1 2! y 2 ∂ 2 y C ( x, 0) , we compute the second partial derivative wrt. y , substitute y = 0 and then multiply by y 2 = (1 + ( y -1)) 2 , yielding

As a consequence, we find that D (1 , 1) = 4 e -2 , so Taylor 1 (1 , 1) ( E ) = 1 + 20( x -1) + 2( y -1) . So E [ X ] = ∂ x E (1 , 1) = 20 , as desired.

Memoizing intermediate results When computing the Taylor expansion of a function G , it is important to memoize the intermediate Taylor expansions of subexpressions of G if they occur more than once. For example, conditionals are translated to a generating function that uses the previous generating function twice. Evaluating it repeatedly would double the running time for each conditional. With memoization, it is possible to handle programs with lots of branching (e.g. > 2 100 paths in the mixture model, cf. Table 5) in a reasonable amount of time.

C.2 Optimizing Observations from Compound Distributions

Observations from compound distributions are generally the bottleneck for the running time because the construct observe d ∼ D ( X k ) is syntactic sugar for X n +1 ∼ D ( X k ); observe X n +1 = d , which introduces a new variable, which is immediately discarded after the observation. This expansion has the semantics

containing an extra variable x n +1 , which worsens the running time of the Taylor approach significantly. We can find more efficient ways of computing this semantics.

Theorem C.1. Observing from compound binomial distributions can be implemented without introducing a new variable:

Proof. A proof for the discrete setting was given in [27]. In our setting, which allows continuous distributions, we cannot use the usual argument about power series anymore. Instead, we reason as follows:

where ∗ follows by iterating the following argument:

which holds by the chain rule.

Theorem C.2. Observing from compound Poisson distributions can be implemented without introducing a new variable:

where

Proof. Let F : R n → R be a smooth function. Then

Inductively, we get

Substituting x n +1 ↦→ 0 in the final expression and dividing by d ! yields:

Theorem C.3. Observing from compound negative binomial distributions can be implemented without introducing a new variable:

J observe d ∼ NegBinomial ( X k , p ) K gf ( G ) = 1 d ! d ∑ i =0 ∂ i k F ( x [ k ↦→ p · x k ]) · p i x i k · (1 -p ) d L d,i where the numbers L d,i are known as Lah numbers and defined recursively as follows:

Proof. Let F : R n → R be a smooth function and y : R → R given by y ( z ) := 1 1 -(1 -p ) z . Then y ′ ( x ) = -1 (1 -(1 -p ) x ) 2 · ( -(1 -p )) = (1 -p ) y ( x ) 2 and

We claim that we can write

The claim is proved by induction. First the case d = 0 :

Next, the induction step:

Substituting x n +1 ↦→ 0 in the final expression and dividing by d ! yields:

Theorem C.4. Observing from compound Bernoulli distributions can be implemented without introducing a new variable:

Proof. We argue as follows:

If d = 0 , the right-hand side is G ( x ) -x k ∂ k G ( x ) . If d = 1 , the right-hand side is x k ∂ k G ( x ) . For larger d ≥ 2 , the d -th derivative vanishes, which completes the proof.

C.3 Analysis of the running time

If the program makes observations (or compares variables with constants) d 1 , . . . , d m , the running time of the program will depend on these constants. Indeed, each observation of (or comparison of a variable with) d i requires the computation of the d i -th partial derivative of the generating function. In our Taylor polynomial approach, this means that the highest order of Taylor polynomials computed will be d := ∑ m i =1 d i . If the program has n variables, storing the coefficients of this polynomial alone requires O ( d n ) space. This can be reduced if the generating function is a low-degree polynomial, in which case we do not store the zero coefficients of the terms of higher order.

Naively, the time complexity of multiplying two such Taylor polynomials is O ( d 2 n ) and that of composing them is O ( d 3 n ) . However, we can do better by exploiting the fact that the polynomials are sparse. In the generating function semantics, we never multiply two polynomials with n variables. One of them will have at most two variables. This brings the running time for multiplications down to O ( d n +2 ) . Furthermore, the GF semantics has the property that when composing two generating functions, we can substitute variables one by one instead of simultaneously, bringing the running time down to O ( n · d 2 n +1 ) . Even more, at most one variable is substituted and the substituted terms have at most two variables, which brings the running time down to O ( d n +3 ) .

Theorem C.5. Our exact inference method can evaluate the k -th derivative of a generating function in O ( s · d n +3 ) , and O ( sd 3 ) for n = 1 , where s is the number of statements, d (for data) is k plus the sum of all values that are observed or compared against in the program, and n is the number of variables of the program.

Proof. The above arguments already analyze the bottlenecks of the algorithm. But for completeness' sake, we consider all operations being performed on generating functions and their Taylor expansions (cf. Appendix C.1).

The maximum order of differentiation needed is d and since there are n variables, the maximum number of Taylor coefficients that need to be stored is O ( d n ) . Addition of two such Taylor polynomials and scalar multiplications trivially take O ( d n ) time. Since in the GF semantics, a GF is only ever multiplied by one with at most two variables, the multiplications take O ( d n +2 ) time. Division only happens with a divisor of degree at most 1, which can also be implemented in O ( d n +1 ) time [13, Chapter 13]. Exponentiation exp( . . . ) , logarithms log( . . . ) , and powers ( . . . ) m are only performed on polynomials with one variable, where they can be implemented in O ( d 2 ) time [13, Table 13.2]. Finally, the only substitutions that are required are ones where a term with at most two variables is substituted for a variable, i.e. of the form p ( x [ k ↦→ q ( x 1 , x 2 )]) where p ( x ) = ∑ | α |≤ d p α x α . Then

where p i ( x ) = ∑ α : α k = i p α x α [ k ↦→ 0] . This needs d multiplications of q ( x 1 , x 2 ) in 2 variables and a polynomial in n variables, each of which takes O ( d n +2 ) time. In total, the substitution can be performed in O ( d n +3 ) time. Finally, if there is only one variable, the substitution can be performed in O ( d 3 ) time because q can only have one variable.

In summary, the input SGCL program has s statements, and each of them is translated to a bounded number of operations on generating functions. These operations are performed using Taylor expansions of degree at most d , each of which takes at most O ( d n +3 ) time (and at most O ( d 3 ) time for n = 1 ). So overall, the running time is O ( s · d n +3 ) (and O ( sd 3 ) for n = 1 ).

Potential improvements of the running time Using various tricks, such as FFT, the running time of the composition of Taylor polynomials, i.e. substitution of polynomials, can be improved to O (( d log d ) 3 n/ 2 ) [2]. This is still exponential in the number of variables n , but would improve our asymptotic running time for n < 6 program variables. In fact, we experimented with an implementation of multiplication and composition based on the fast Fourier transform (FFT), but discarded this option because it led to much greater numerical instabilities. It is possible that the running time could instead be improved with the first algorithm in [2], which relies on fast matrix multiplication instead

of FFT. In addition, it should often be possible to exploit independence structure in programs, so the GF can be factored into independent GFs for subsets of the variables. This is not implemented yet because it was not relevant for our benchmarks.

Performance in practice Note that, for many programs, the running time will be better than the worst case. Indeed, substitutions are the most expensive operation, if the substitute expression is of high degree. For some models, like the population model, the substitutions take the form of Theorem C.1, i.e. G ( x [ k ↦→ (1 -p ) x k ]) , which is of degree 1. Such a substitution can be performed in O ( d n ) time, because one simply needs to multiply each coefficient by a power of (1 -p ) . This explains why the population model is so fast in the experimental section (Section 5). In other cases as well, we can often make use of the fact that the polynomials have low degree to reduce the space needed to store the coefficients and to speed up the operations.

C.4 Numerical issues

If our implementation is used in floating-point mode, there is the possibility of rounding errors and numerical instabilities, which could invalidate the results. To enable users to detect this, we implemented an option in our tool to use interval arithmetic in the computation: instead of a single (rounded) number, it keeps track of a lower and upper bound on the correctly rounded result. If this interval is very wide, it indicates numerical instabilities.

Initially, we observed large numerical errors (caused by catastrophic cancellation) for probabilistic programs involving continuous distributions. The reason for this turned out to be the term log( x ) that occurs in the GF of the continuous distributions. The Taylor coefficients of log( x ) at points 0 < z < 1 are not well-behaved:

because the coefficients grow exponentially for z < 1 (due to the term z -n ) and the alternating sign exacerbates this problem by increasing the likelihood of catastrophic cancellations.

To fix this problem, we adopted a slightly modified representation. Instead of computing the Taylor coefficients of the GF G directly, we compute those of the function H ( x ) := G ( x ′ ) where x ′ i := x i if X i is discrete and x ′ i := exp( x i ) if X i is continuous, thus canceling the logarithm. 6 (The reason we don't use x ′ := exp( x ) for all variables is that for discrete variables, we may need to evaluate at x ′ i = 0 , corresponding to x i = -∞ , which is impossible.) It is straightforward to adapt the GF semantics to this modified representation. Due to the technical nature of the adjustment (case analysis on whether an index corresponds to a discrete variable or not), we do not describe it in detail. With this modified representation, we avoid the catastrophic cancellations and obtain numerically stable results for programs using continuous distributions as well.

D Details on the Empirical Evaluation

All experiments were run on a laptop computer with a Intel® Core™ i5-8250U CPU @ 1.60GHz × 8 processor and 16 GiB of RAM, running Ubuntu 22.04.2 and took a few hours to complete overall. The code to run the benchmarks and instructions on how to reproduce our experiments are available in the supplementary material.

D.1 Comparison with exact inference

Wemanually patched the tools Dice, Prodigy, and PSI to measure and output the time taken exclusively for inference to ensure a fairer comparison and because the time a tool takes to startup and parsing times are not very interesting from a research perspective. We ran each tool 5 times in a row and took the minimum running time to reduce the effect of noise (garbage collection, background activity on

the computer). In rational mode, we ran Genfer with the option ✲✲/a114❛/a116✐♦♥❛❧ and Dice with the option ✲✇♠❝✲/a116②♣❡ ✶ . For the 'clinicalTrial' benchmark, Genfer required 400-bit floating-point numbers via ✲✲♣/a114❡❝✐/a115✐♦♥ ✹✵✵ because 64-bit numbers did not provide enough accuracy. For better performance, PSI was run with the flag ✲✲❞♣ on finite discrete benchmarks.

The benchmarks were taken from the PSI paper [9, Table 1], excluding 'HIV', 'LinearRegression1', 'TrueSkill', 'AddFun/max', 'AddFun/sum', and 'LearningGaussian'. While Genfer supports continuous priors, the excluded benchmarks use continuous distributions in other ways that are not supported (e.g. in observations). Note that Dice and Prodigy have no support for continuous distributions at all. These benchmarks were translated mostly manually to the tools' respective formats, and we checked that the outputs of the tools are consistent with each other.

D.2 Comparison with approximate inference

Experimental setup As explained in Section 5, we ran several of Anglican's inference algorithms. Each algorithm was run with a sampling budget of 1000 and 10000. The precise settings we used are the following:

Importance Sampling (IS): has no settings

Lightweight Metropolis Hastings (LMH): has no settings

Random-Walk Metropolis Hastings (RMH): has two settings: the probability α of using a local MCMC move and the spread σ of the local move. We used the default α = 0 . 5 , σ = 1 and another setting facilitating local moves with α = 0 . 8 , σ = 2 .

Sequential Monte Carlo (SMC): number of particles ∈ { 1 , 1000 } . The default is 1 and we also picked 1000 to see the effect of a larger number of particles while not increasing the running time too much.

Particle Gibbs (PGibbs): number of particles ∈ { 2 , 1000 } . The default is 2 and we also picked 1000 as the other setting for the same reason as for SMC.

Interacting Particle MCMC (IPMCMC): it has the settings 'number of particles per sweep', 'number of nodes running SMC and CSMC', 'number of nodes running CSMC', and 'whether to return all particles (or just one)'. We left the last three settings at their defaults (32 nodes, 16 nodes running CSMC, return all particles). For the number of nodes and the number of nodes we chose the default setting (2) and a higher one (1000) for the same reason as SMC.

A full list of Anglican's inference algorithms and their settings can be found at ❤/a116/a116♣/a115✿✴✴♣/a114♦❜♣/a114♦❣✳ ❣✐/a116❤✉❜✳✐♦✴❛♥❣❧✐❝❛♥✴✐♥❢❡/a114❡♥❝❡ . Each inference algorithm was run 20 times on each benchmark and the running times were averaged to reduce noise. This took around an hour per benchmark.

D.3 Benchmarks with infinite support

A summary with important information on each benchmark can be found in Table 5, including some statistics about the probabilistic programs. We describe each benchmark in more detail in the following.

Population models The probabilistic program for the original population model from [26] is shown in Fig. 6a. This program can also be written without the extra variable New by writing the two

(a) Original model from [26].

(b) Randomly modified arrival rate.

(c) Two interacting populations.

statements involving it as N + ∼ Poisson ( λ 1 ) . Here we used the same parameter values as [26]: δ = 0 . 2636 , λ = Λ · (0 . 0257 , 0 . 1163 , 0 . 2104 , 0 . 1504 , 0 . 0428) , and Λ = 2000 is the population size parameter. Note that the largest population size considered by [26] is Λ = 500 . For the observation rate ρ , Winner and Sheldon [26] consider values from 0 . 05 to 0 . 95 . For simplicity, we set it to ρ = 0 . 2 . The y k are k = 4 simulated data points: (45 , 98 , 73 , 38) for Λ = 2000 .

The modified population example is shown in Fig. 6b, where the arrival rate is reduced to λ ′ := 0 . 1 λ in the case of a natural disaster, which happens with probability 0.1. The rest of the model is the same.

The model of two interacting populations (multitype branching process) is programmed as shown in Fig. 6c, where λ (1) := 0 . 9 λ , λ (2) := 0 . 1 λ , γ := 0 . 1 . In this model, some individuals of the first kind (i.e. in N 1 ) can turn into individuals of the second kind and are added to N 2 . The other parameters of this model are the same as before. The data points are y (1) = (35 , 83 , 78 , 58) and y (2) = (3 , 6 , 10 , 4) .

Bayesian switchpoint analysis Bayesian switchpoint analysis is about detecting a change in the frequency of certain events over time. An example is the frequency of coal mining disasters in the United States from 1851 to 1962, as discussed in the PyMC3 tutorial [23]. We use the same model as in [23], and its probabilistic program is shown in Fig. 7a. This situation can be modeled with two Poisson distributions with parameters Λ 1 (before the change) and Λ 2 (after the change). Suppose we have observations y 1 , . . . , y n , with y t ∼ Poisson (Λ 1 ) if t < T and y t ∼ Poisson (Λ 2 ) if t ≥ T . The parameters Λ 1 , Λ 2 are given exponential priors Exponential (1) . The change point T itself is assumed to be uniformly distributed: T ∼ Uniform { 1 ..n } . The probabilistic program code is shown in Fig. 7a, where n = 111 and y = ( 4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6, 3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5, 2, 2, 3, 4, 2, 1, 3, n/a, 2, 1, 1, 1, 1, 3, 0, 0, 1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 3, 3, 1, n/a, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1 ) where 'n/a' indicates missing data (which is omitted in the program).

In fact, this program can be rewritten to the more efficient form shown in Fig. 7b, to minimize the number of variables needed.

Mixture model In the binary mixture model, the frequency events are observed from an equalweight mixture of two Poisson distributions with different parameters Λ 1 , Λ 2 , which are in discrete steps of 0.1. This is implemented by imposing a geometric prior and multiplying the rate by 0.1 in the Poisson distribution: Poisson (0 . 1 · Λ 1 ) . The data is the same as for the switchpoint model. The task is to infer the first rate Λ 1 . The probabilistic program is shown in Fig. 8a.

(b) More efficient version of the switchpoint model.

0.0

0.5

1.0

1.5

2.0

2.5

Time (in s)

0.00

0.05

0.10

0.15

0.20

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0.0

0.5

1.0

1.5

2.0

2.5

Time (in s)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the standard de-

viation

0.0

0.5

1.0

1.5

2.0

2.5

Time (in s)

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0.0

0.5

1.0

1.5

2.0

2.5

Time (in s)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

0

1

2

3

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0

1

2

3

Time (in s)

0.00

0.05

0.10

0.15

0.20

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the standard de-

0

1

2

3

Time (in s)

0.0

0.1

0.2

0.3

0.4

0.5

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0

1

2

3

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

viation

Hidden Markov model The hidden Markov example is based on [22, Section 2.2]. The hidden state Z can be 0 or 1 and transitions to the other state with probability 0.2. The rate of the Poissondistributed number of observed events depends on the state Z and is 0 . 1 · Λ 1 or 0 . 1 · Λ 2 , where we impose a geometric prior on Λ 1 and Λ 2 . As for the mixture model, this discretizes the rates in steps of 0.1. The inference problem is to infer the first rate Λ 1 . The probabilistic program is shown in Fig. 8b, where y k are m = 30 simulated data points from the true rates λ 1 := 0 . 5 and λ 2 := 2 . 5 . Concretely, we have y = ( 2, 2, 4, 0, 0, 0, 0, 0, 1, 1, 0, 2, 4, 3, 3, 5, 1, 2, 3, 1, 3, 3, 0, 0, 2, 0, 0, 2, 6, 1 ) .

D.4 Additional results

In the main text, the data we presented was mainly in form of the total variation distance (TVD) between the true posterior (computed by our method) and the approximated posterior (computed by Anglican's inference algorithms). In this section, we present additional evidence in two forms:

errors of the posterior moments (mean, standard deviation, skewness, and kurtosis) in Figs. 9 to 14, and

histograms (Fig. 15) of the sampled distribution produced by the best MCMC algorithm, where the sampling budget is picked such that the running time is close to that of our method (e.g. 1000 for the (fast) population model and 10000 for the (slower) mixture model).

The additional plots confirm the conclusions from Section 5.

0

1

2

3

4

5

Time (in s)

0.0

0.1

0.2

0.3

0.4

0.5

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0

1

2

3

4

5

Time (in s)

0.00

0.05

0.10

0.15

0.20

0.25

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the standard de-

0

1

2

3

4

5

Time (in s)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0

1

2

3

4

5

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

viation

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the standard de-

viation

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0

5

10

15

20

25

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

0

10

20

30

40

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0

10

20

30

40

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the standard de-

viation

0

10

20

30

40

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0

10

20

30

40

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

0

2

4

6

8

10

12

Time (in s)

0.0

0.5

1.0

1.5

2.0

Error of mean (in σs)

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(a) Error of the mean

0

2

4

6

8

10

12

Time (in s)

0.0

0.2

0.4

0.6

0.8

1.0

Rel. error of standard deviation

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(b) Error of the standard de-

viation

0

2

4

6

8

10

12

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of skewness

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(c) Error of the skewness

0

2

4

6

8

10

12

Time (in s)

0.0

0.5

1.0

1.5

2.0

Abs. error of kurtosis

ours

IS

IPMCMC

LMH

PGibbs

RMH

SMC

(d) Error of the kurtosis

0

50

100

150

200

250

result

0.00

0.01

0.02

0.03

0.04

probability

SMC (1000 samples)

ours (exact)

(a) Original population model.

0

50

100

150

200

250

result

0.00

0.01

0.02

0.03

0.04

0.05

probability

SMC (1000 samples)

ours (exact)

(b) Modified population model.

0

10

20

30

40

50

60

result

0.00

0.02

0.04

0.06

0.08

probability

PGibbs (10000 samples)

ours (exact)

(c) Two-type population model.

0

20

40

60

80

100

result

0.00

0.05

0.10

0.15

0.20

0.25

0.30

probability

LMH (10000 samples)

ours (exact)

(d) Switchpoint model.

0

20

40

60

result

0.000

0.025

0.050

0.075

0.100

0.125

0.150

probability

PGibbs (10000 samples)

ours (exact)

(e) Mixture model.

0

20

40

60

result

0.00

0.02

0.04

0.06

0.08

0.10

0.12

probability

SMC (10000 samples)

ours (exact)

Contents

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to exact inference in probabilistic programs. Existing approaches to exact inference attempt to compute marginal and normalized probabilities by efficiently computing the necessary (finite) sums, or using computer algebra to recognize and solve tractable integrals. By contrast, the present work constructs the moment-generating function ($G(x) = \\mathbb{E}_{X \\sim \\mu}[x^X]$) for the distribution represented by the program, and takes its (higher-order) derivatives (using AD) to compute exact posterior marginal probabilities, as well as exact moments. This enables automated exact inference in a broader class of models than handled by existing techniques. The approach is implemented in Rust, and experiments in four example models show that their implementation typically computes exact answers in less time than it takes for Anglican’s generic Monte Carlo algorithms (e.g., sequential Monte Carlo with the prior as the proposal) to converge to accurate estimates.\n\nSTRENGTHS:\nThe key strengths I see in this paper are:\n\n- It introduces a significant new approach to PPL inference (automatic differentiation on compositionally constructed moment-generating functions), and I believe it’s important enough that most people working in PPLs will want to understand the new technique. The exposition is clear and should be accessible to a broad probabilistic programming audience.\n- The generating function approach resolves a key limitation of other exact inference methods: it can marginalize latent variables with infinite support.\n- Limitations of the language, and performance concerns, are clearly outlined (e.g., in the Restrictions paragraph on L147, and the Limitations paragraph on L83).\n- The appendix contains rigorous correctness proofs and further exposition of the generating function semantics. There is also a Rust implementation that helps clarify the actual implementation of the technique (e.g., the data representation of Taylor polynomials used by the AD algorithm).\n\nWEAKNESSES:\nThe key weaknesses I notice are:\n\n- The novel capabilities (and limitations) with respect to the line of work by Klinkenberg et al. could be more clearly described. It would also be good to cite and explain the differences with their recent LAFI abstract, “Exact Probabilistic Inference Using Generating Functions” (https://arxiv.org/pdf/2302.00513.pdf). That workshop abstract is non-archival and much less thorough than the present work, so I don’t mean this as a critique of your work’s novelty. But I think readers will appreciate a clear-eyed description of exactly how their proposed approach differs from yours.\n- It is unclear to what extent the limitations of the language are fundamental limitations of the generating function technique, vs. concessions made to get to a simple first publication/implementation of the idea. Could the technique be extended to programs with loops, or does that seem unlikely? What about compound distributions with more than one random parameter (e.g., Binomial where both $n$ and $p$ is also random)? What about ‘score statements’ that multiply a non-negative (affine?) expression into the likelihood? (It seems this should be possible for scores < 1, because we can `observe 1 ~ Bernoulli(X)`, but can it be made to work for more general scores?)\n- The experiments are all on programs for which existing exact inference methods fail. But it would be nice to understand how the new approach compares to existing methods on the programs that both approaches can handle. (Even if it is much less efficient, I believe the method still warrants publication, but it would be useful to readers to understand the expressiveness / efficiency trade-offs clearly.)\n- L310-312 incorrectly state that WebPPL does not support SMC or particle MCMC. See the SMC and PMCMC implementations here: https://github.com/probmods/webppl/tree/master/src/inference.\n\nQUESTIONS:\n1. Are you 100% sure that systems like Hakaru and $\\lambda$PSI cannot handle your examples? I don’t have a great handle on their capabilities, but I expect Maple’s computer algebra simplifications can handle at least some infinite sums and integrals. Have you tried implementing the example programs and seeing what these tools spit out?\n2. What are the prospects for a generating-function semantics of a functional (rather than imperative) source language, such as Anglican (but suitably restricted)?\n\nLIMITATIONS:\nThe paper is up-front about both the expressiveness limitations of the probabilistic language they support, and the factors in which their algorithm scales exponentially.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a new probabilistic programming language called statistical guarded command language (SGCL) and an exact Bayesian inference method for any discrete statistical model that can be expressed in SGCL. Empirical evaluatios on a variety of different discrete probabilistic models demonstrate the effectiveness of the method, especially for challenging distributions with discrete variables that have infinite support.

STRENGTHS:
- The paper is well organized and clearly written.
- The proposed method seems novel, and is very useful especially for challenging discrete distributions involving variables with infinite support.

WEAKNESSES:
- The paper seems to miss discussions on probabilistic circuits which is a large body of work concerning probabilistic models with tractable inference.
- Given the limitations of the proposed PPL and its specialized use cases, it would also be helpful to add some discussions on how this can potentially be integrated into/work together with other more general PPLs to expand the applicability of the proposed method.
- The current title and abstract seems slightly misleading, in the sense that the proposed PPL is not generally applicable to any discrete distributions. It would be better to rephrase to make the scope of its applicability clearer, and also emphasize its targeted use case is distributions with variables having infinite support.

QUESTIONS:
See weaknesses.

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a restricted, first-order probabilistic programming language in which inference for infinite-support latent variables, with continuous or discrete sample spaces, admits a closed form.  For experiments they compare against importance sampling, Lightweight M-H, Random Walk M-H (RMH), sequential Monte Carlo, particle Gibbs sampling, and interacting particle MCMC (IPMCMC) in the Anglican universal PPL.  The authors achieve much lower error than Monte Carlo sampling across experiments, typically with significantly less compute.\n\nSTRENGTHS:\nThe paper demonstrates that when latent variables are subject only to affine transformations, a first-order (lacking general recursion) probabilistic programming language can admit tractable exact inference for certain important classes of discrete observations. Even when the resulting exact inference engine takes compute time comparable to the Anglican Monte Carlo inference routines, it tends to run in the middle of the pack while providing much lower approximation error.\n\nWEAKNESSES:\nSince the paper focuses on proving and benchmarking something that has an exact answer, it has few weaknesses as such.  It could perhaps have done a better job of making explicit the if-and-only-if conditions that must be met for generating functions to support closed-form inference.\n\nThe authors have addressed this concern in the rebuttal.\n\nQUESTIONS:\nHow could the exact inference algorithm in this paper be used in a larger probabilistic programming framework, such as when probabilistic circuits admitting exact density calculations are applied as proposals in a broader family of inference algorithms?  When the paper mentions outputting posterior probability masses for certain intervals of integer-valued latent variables, why not also include densities for continuous latent variables?\n\nThe authors have addressed these questions in their response.\n\nLIMITATIONS:\nThe authors have admitted the limitations of their work, and in exchange for those limitations in expressivity they gain performance in both error and compute time.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
A probabilistic programming framework for exact inference in discrete probabilistic programs with infinite support is introduced. The framework levegares probability generating functions as a first class attribute of discrete distributions. The probabilistic programming language is formally defined, inference is theoretically analysed for correctness. The performance and accuracy are empirically evaluated and compared with baselines on real-world examples.

STRENGTHS:
The paper brings the idea of using GF as a first-class property of a distribution into a practical implementation as a probabilistic programming language. Formal properties of the language and inference are rigorously analysed. The empirical evaluation demonstrates capabilities of the approach, and favorably compares to alternatives. The submission is accompanied by a working and well organized code based.

WEAKNESSES:
The PPL is implemented in Rust, a language which, while it certainly has its merits, less popular for statistical modelling and Bayesian inference, which may make adoption of the approach more difficult. This is a matter of taste though.

QUESTIONS:
You apparently unroll loops in your benchmarks (I assume because the language does not support loops). What prevents you from implementing restricted loops in the language?

The syntax in the examples is richer than the BNF in the paper. What is '+~' for example? Can you define the full syntax in the paper, at least briefly?

Why rust? Have you considered other options?

LIMITATIONS:
The paper adequately discusses limitations of the proposed solution.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07