SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] E. Bingham, J. P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh, P. A. Szerlip, P. Horsfall, and N. D. Goodman. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res., 20:28:1-28:6, 2019. URL http://jmlr.org/papers/v20/18-403.html.
[2] R. P. Brent and H. T. Kung. Fast algorithms for manipulating formal power series. J. ACM, 25(4): 581-595, 1978. doi: 10.1145/322092.322099. URL https://doi.org/10.1145/322092.322099.
[3] B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. Stan: A probabilistic programming language. Journal of Statistical Software, 76(1), 1 2017. ISSN 1548-7660. doi: 10.18637/jss.v076.i01. URL https://www.osti.gov/biblio/1430202.
[4] M. Chen, J. Katoen, L. Klinkenberg, and T. Winkler. Does a program yield the right distribution? - verifying probabilistic programs via generating functions. In S. Shoham and Y. Vizel, editors, Computer Aided Verification - 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part I, volume 13371 of Lecture Notes in Computer Science, pages 79-101. Springer, 2022. doi: 10.1007/978-3-031-13185-1_5. URL https://doi.org/10.1007/978-3-031-13185-1_5.
[5] Y. Choi, A. Vergari, and G. Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models, 2020. URL http://starai.cs.ucla.edu/papers/ProbCirc20.pdf. Accessed: 2023-10-22.
[6] M. F. Cusumano-Towner, F. A. Saad, A. K. Lew, and V. K. Mansinghka. Gen: a general-purpose probabilistic programming system with programmable inference. In ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2019. ACM, 2019. doi: 10.1145/3314221.3314642.
[7] W. Feller. An introduction to probability theory and its applications, volume 1. Wiley Series in Probability and Statistics. John Wiley & Sons, Nashville, TN, 3 edition, jan 1968.
[8] H. Ge, K. Xu, and Z. Ghahramani. Turing: A language for flexible probabilistic inference. In A. Storkey and F. Perez-Cruz, editors, Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1682–1690. PMLR, 09–11 Apr 2018. URL https://proceedings.mlr.press/v84/ge18b.html.
[9] T. Gehr, S. Misailovic, and M. T. Vechev. PSI: exact symbolic inference for probabilistic programs. In S. Chaudhuri and A. Farzan, editors, Computer Aided Verification - 28th International Conference, CAV 2016, Toronto, ON, Canada, July 17-23, 2016, Proceedings, Part I, volume 9779 of Lecture Notes in Computer Science, pages 62-83. Springer, 2016. doi: 10. 1007/978-3-319-41528-4_4. URL https://doi.org/10.1007/978-3-319-41528-4_4.
[10] Z. Ghahramani. Probabilistic machine learning and artificial intelligence. Nat., 521(7553):452-459, 2015. doi: 10.1038/nature14541. URL https://doi.org/10.1038/nature14541.
[11] N. D. Goodman and A. Stuhlmüller. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org, 2014. Accessed: 2023-4-6.
[25] J. van de Meent, B. Paige, H. Yang, and F. Wood. An introduction to probabilistic programming. CoRR, abs/1809.10756, 2018. URL http://arxiv.org/abs/1809.10756.
[26] K. Winner and D. Sheldon. Probabilistic inference with generating functions for poisson latent variable models. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2640-2648, 2016. URL https://proceedings.neurips.cc/paper/2016/bitnet/6c1da886822c67822bcf3679d04369fa-Abstract.html.
[27] K. Winner, D. Sujono, and D. Sheldon. Exact inference for integer latent-variable models. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3761-3770. PMLR, 2017. URL http://proceedings.mlr.press/v70/winner17a.html.
[28] H. Zhang, B. Juba, and G. V. den Broeck. Probabilistic generating circuits. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12447-12457. PMLR, 2021. URL http://proceedings.mlr.press/v139/zhang21i.html.
A Details on the Probabilistic Programming Language SGCL
A.1 A minimal grammar
A minimal grammar of SGCL looks like this:
$$
\begin{array}{l} P := \operatorname {s k i p} \mid P _ {1}; P _ {2} \mid X _ {k} := a _ {1} X _ {1} + \dots + a _ {n} X _ {n} + c \mid \text {i f} X _ {k} \in A \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\} \\ | \text {f a i l} | X _ {k} \sim D | X _ {k} \sim D (X _ {j}) \\ \end{array}
$$
where  $a_1, \ldots, a_n, c \in \mathbb{R}_{\geq 0}, p \in [0,1]$  are literals, and  $A$  is a finite subset of  $\mathbb{N}$ . The  $X_k$  for  $k = 1, \ldots, n$  are variables that can take on values in  $\mathbb{R}_{\geq 0}$ . To explain, skip leaves everything unchanged,  $P_1; P_2$  runs  $P_1$  and then  $P_2$ , and  $X_k := \mathbf{a}^\top \mathbf{X} + c$  performs an affine transformation where  $\mathbf{a} \in \mathbb{R}_{\geq 0}^n$  and  $c \in \mathbb{R}_{\geq 0}$  (to ensure that the support of  $X_k$  is a subset of  $\mathbb{R}_{\geq 0}$ ). The if construct allows branching based on comparisons of a variable  $X_k$  with constant natural numbers, with the requirement that  $X_k$  be a discrete variable on  $\mathbb{N}$ , and  $X_k \sim D$  samples  $X_k$  from a primitive distribution (with constant parameters)
$$
\begin{array}{l} \mathcal {D} = \left\{\text {B e r n o u l l i} (p), \text {C a t e g o r i c a l} \left(p _ {0}, \dots , p _ {m}\right), \text {B i n o m i a l} (m, p), \text {U n i f o r m} \{l.. m \}, \right. \\ \operatorname {N e g B i n o m i a l} (m, p), \operatorname {G e o m e t r i c} (p), \operatorname {P o i s s o n} (\lambda), \\ \text {E x p o n e n t i a l} (\alpha), \text {G a m m a} (\alpha , \beta), \text {U n i f o r m} [ a, b ] \\ | p \in [ 0, 1 ], \alpha , \beta \in \mathbb {R} _ {\geq 0}, l, m \in \mathbb {N}, a, b \in \mathbb {R} _ {\geq 0}, a \leq b \} \\ \end{array}
$$
whereas  $X_{k} \sim D(X_{j})$  samples  $X_{k}$  from a compound distribution (with a parameter that is again a variable)
$$
\begin{array}{l} \mathcal {D} \left(X _ {j}\right) = \{\text {B i n o m i a l} \left(X _ {j}, p\right), \text {N e g B i n o m i a l} \left(X _ {j}, p\right), \text {P o i s s o n} \left(\lambda \cdot X _ {j}\right), \text {B e r n o u l l i} \left(X _ {j}\right) \\ | p \in [ 0, 1 ], j \in \{1, \dots , n \}, \lambda \in \mathbb {R} _ {\geq 0} \}. \\ \end{array}
$$
A.2 Syntactic sugar
For convenience, we add some syntactic sugar, which is expanded as follows:
$$
X _ {k} + = \mathbf {a} ^ {\top} X + c \rightsquigarrow X _ {k} := \left(\mathbf {a} [ k \mapsto a _ {k} + 1 ]\right) ^ {\top} X + c
$$
$$
X _ {k} + \sim D \rightsquigarrow X _ {n + 1} \sim D; X _ {k} + = X _ {n + 1} \quad (X _ {n + 1} \text {f r e s h})
$$
$$
o b s e r e \phi \rightsquigarrow i f \phi \{\text {s k i p} \} e l s e \{\text {f a i l} \}
$$
$$
\begin{array}{l} \text {i f} m \sim D \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\}\rightsquigarrow X _ {n + 1} \sim D; \text {i f} X _ {n + 1} = m \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\} \quad (X _ {n + 1} \text {f r e s h}) \\ \text {i f} X _ {k} = m \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\}\rightsquigarrow \text {i f} X _ {k} \in \{m \} \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\} \\ i f \neg \phi \left\{P _ {1} \right\} e l s e \left\{P _ {2} \right\}\rightsquigarrow i f \phi \left\{P _ {2} \right\} e l s e \left\{P _ {1} \right\} \\ \text {i f} \phi_ {1} \wedge \phi_ {2} \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\}\rightsquigarrow \text {i f} \phi_ {1} \left\{\text {i f} \phi_ {2} \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\}\right\} \text {e l s e} \left\{P _ {2} \right\} \\ \text {i f} \phi_ {1} \vee \phi_ {2} \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\}\rightsquigarrow \text {i f} \phi_ {1} \left\{P _ {1} \right\} \text {e l s e} \left\{\text {i f} \phi_ {2} \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\}\right\} \\ \end{array}
$$
A.3 Standard measure semantics
Notation and Conventions Throughout the text, we always assume that  $\mathbb{N}$  carries the discrete  $\sigma$ -algebra  $\mathcal{P}(\mathbb{N})$  and  $\mathbb{R}$  the Borel  $\sigma$ -algebra  $\mathcal{B}(\mathbb{R})$ . Products of these spaces (e.g.  $\mathbb{N} \times \mathbb{R}$ ) are endowed with the product  $\sigma$ -algebra. In the following, we write  $\pmb{x}$  for  $(x_{1}, \ldots, x_{n})$  and use the update notation  $\pmb{x}[i \mapsto a]$  to denote  $(x_{1}, \ldots, x_{i-1}, a, x_{i+1}, \ldots, x_{n})$ . In fact, we even write updates  $\pmb{x}[i_{1} \mapsto a_{1}, \ldots, i_{k} \mapsto a_{k}]$ , which stands for  $\pmb{x}[i_{1} \mapsto a_{1}] \cdots [i_{k} \mapsto a_{k}]$ . Finally, we write  $\mathbf{1}_{k}$  for the  $k$ -tuple  $(1, \ldots, 1)$  and similarly for  $\mathbf{0}_{k}$ . We write  $\mathbf{0}$  for the zero measure or the constant zero function. We also use Iverson brackets  $[\phi]$  for a logical condition  $\phi$ :  $[\phi]$  is 1 if  $\phi$  is satisfied and 0 otherwise. For example, the indicator function  $\mathbf{1}_{S}$  of a set  $S$  can be defined using Iverson brackets:  $\mathbf{1}_{S}(x) = [x \in S]$ . For a measure  $\mu$  on  $\mathbb{R}^{n}$ , we define  $\mu_{X_{k} \in A}(S) = \mu(\{X \in S \mid X_{k} \in A\})$ .
Standard transformer semantics The standard semantics for such a probabilistic program uses distribution transformers [19]. Since our language includes conditioning, the distribution of program states will not always be a probability distribution, but a subprobability distribution  $\mu$  on  $\mathbb{R}_{\geq 0}^n$  for  $n$  variables, i.e. a function  $\mu : \mathcal{B}(\mathbb{R}_{\geq 0}^n) \to [0,1]$  with  $\mu(\mathbb{R}_{\geq 0}^n) \leq 1$ . The semantics of a program  $P$  is thus a measure transformer  $\llbracket P\rrbracket_{\mathrm{std}} : \mathrm{Meas}(\mathbb{R}_{\geq 0}^k) \to \mathrm{Meas}(\mathbb{R}_{\geq 0}^k)$ . We assume that the initial measure
$$
[   [ \mathrm {s k i p} ]   ] _ {\mathrm {s t d}} (\mu) = \mu
$$
$$
[   [ P _ {1}; P _ {2} ]   ] _ {\mathrm {s t d}} (\mu) = [   [ P _ {2} ]   ] _ {\mathrm {s t d}} ([   [ P _ {1} ]   ] _ {\mathrm {s t d}} (\mu))
$$
$$
\llbracket X _ {k} := \mathbf {a} ^ {\top} \boldsymbol {X} + c \rrbracket_ {\mathrm {s t d}} (\mu) (S) = \mu \left(\left\{\boldsymbol {X} \in \mathbb {R} ^ {n} \mid \boldsymbol {X} [ k \mapsto \mathbf {a} ^ {\top} \boldsymbol {X} + c ] \in S \right\}\right)
$$
$$
[   [ \text {i f} X _ {k} \in A \{P _ {1} \} \text {e l s e} \{P _ {2} \} ]   ] _ {\text {s t d}} (\mu) = [   [ P _ {1} ]   ] _ {\text {s t d}} \left(\mu_ {X _ {k} \in A}\right) + [   [ P _ {2} ]   ] _ {\text {s t d}} \left(\mu - \mu_ {X _ {k} \in A}\right)
$$
$$
w h e r e \mu_ {X _ {k} \in A} (S) = \mu (\{X \in S \mid X _ {k} \in A \})
$$
$$
\llbracket X _ {k} \sim D \rrbracket_ {\mathrm {s t d}} (\mu) (S) = \int \int [ \boldsymbol {X} [ k \mapsto Y ] \in S ] \mathrm {d} D (Y) \mathrm {d} \mu (\boldsymbol {X})
$$
$$
\llbracket X _ {k} \sim D (X _ {j}) \rrbracket_ {\mathrm {s t d}} (\mu) (S) = \int \int [ \boldsymbol {X} [ k \mapsto Y ] \in S ] \mathrm {d} D (X _ {j}) (Y) \mathrm {d} \mu (\boldsymbol {X})
$$
$$
\llbracket \text {f a i l} \rrbracket_ {\text {s t d}} (\mu) = \mathbf {0}
$$
$$
\operatorname {n o r m a l i z e} (\mu) = \frac {\mu}{\int \mathrm {d} \mu (\boldsymbol {X})}
$$
Figure 4: The standard measure semantics  $\llbracket -\rrbracket_{\mathrm{std}}$
is the probability distribution where every variable  $X_{k}$  is 0 with probability 1, i.e.  $\mathbf{X} \sim \mathrm{Dirac}(\mathbf{0}_n)$ . The semantics is shown in Fig. 4 where  $\mathbf{X}$  is a vector of length  $n$ .
The first 5 rules were essentially given by [19], with slightly different notation and presentation. Kozen defines some of the measures on rectangles  $(S = S_{1} \times \dots \times S_{n})$  only, which uniquely extends to a measure on all sets in the product  $\sigma$ -algebra. By contrast, we directly define the measure on all sets in the  $\sigma$ -algebra using integrals.
We quickly explain the rules. The skip command does nothing, so the distribution  $\mu$  of the program is unchanged. Chaining commands  $P_{1}; P_{2}$  has the effect of composing the corresponding transformations of  $\mu$ . Affine transformations of program variables yield a measure of  $S$  that is given by the original measure of the preimage of the event  $S$  under the affine map. For conditionals if  $X_{k} \in A\{P_{1}\}$  else  $\{P_{2}\}$ , we consider only outcomes satisfying  $X_{k} \in A$  in the first branch and the complement in the second branch and then sum both possibilities. Sampling from distributions  $X_{k} \sim D$  is a bit more complicated. Note that  $\mu(S) = \int[\mathbf{X} \in S] \mathrm{d}\mu(\mathbf{X})$ . To obtain the measure after sampling  $X_{k} \sim D$ , we care about the new state  $\mathbf{X}[k \mapsto Y]$  being in  $S$ , where  $Y$  has distribution  $D$ . This is exactly expressed by the double integral. The rule for compound distributions  $X_{k} \sim D(X_{j})$  is essentially the same as the one for distributions with constant parameters  $X_{k} \sim D$ , except the dependency on  $X_{j}$  needs to be respected.
The two new rules (not derivable from Kozen's work) are for fail and the implicit normalization normalize  $(\mu)$  as a final step at the end of a program. The rule for fail departs from Kozen's semantics in the sense that it yields a subprobability distribution (i.e. the total mass can be less than 1). Intuitively, fail "cuts off" or assigns the probability 0 to certain branches in the program. This is called hard conditioning in the PPL literature. For example, if  $X_{1} = 5$  {skip} else {fail} has the effect of conditioning on the event that  $X_{1} = 5$  because all other scenarios are assigned probability 0.
The normalization construct corresponds to the division by the evidence in Bayes' rule and scales the subprobability distribution of a subprogram  $P$  back to a probability distribution. The semantics is  $\frac{\mu}{\int \mathrm{d}\mu(X)}$ , so the distribution  $\mu$  is scaled by the inverse of the total probability mass of  $\mu$ , thus normalizing it.
B Details on Generating Functions
An exhaustive list of the supported distributions and their generating functions can be found in Tables 4a and 4b.
(a) GFs for common distributions
Distribution D	gf(X ~ D)(x)
Dirac(a)	xa
Bernoulli(p)	px + 1 - p
Categorical(p0, ..., pn)	∑i=0npi xi
Binomial(n, p)	(px + 1 - p)n
Uniform{a..b}	xa - xb+1/(b - a + 1)(1 - x)
NegBinomial(r, p)	\(\left(\frac{p}{1-(1-p)x}\right)^r\)
Geometric(p)	\(\frac{p}{1-(1-p)x}\)
Poisson(λ)	e^λ(x-1)
Exponential(λ)	λ/λ-log x
Gamma(α, β)	\(\left(\frac{\beta}{\beta-log x}\right)^α\)
Uniform[a, b]	{x^b - x^a / (b - a) log x x ≠ 1
1 x = 1
(b) GFs for common compound distributions
Distribution D(Y)	gf(X ~ D(Y))(x)
Binomial(Y,p)	gf(Y)(1-p+px)
NegBinomial(Y,p)	gf(Y)\(\left(\frac{p}{1-(1-p)x}\right)\)
Poisson(λ·Y)	gf(Y)\(e^{\lambda(x-1)}\)
Bernoulli(Y)	1+(x-1)\(gf(Y))'(1)\)
$$
\llbracket \operatorname {s k i p} \rrbracket_ {\mathrm {g f}} (G) = G
$$
$$
\llbracket P _ {1}; P _ {2} \rrbracket_ {\mathbf {g f}} (G) = \llbracket P _ {2} \rrbracket_ {\mathbf {g f}} (\llbracket P _ {1} \rrbracket_ {\mathbf {g f}} (G))
$$
$$
\llbracket X _ {k} := \mathbf {a} ^ {\top} \boldsymbol {X} + c \rrbracket_ {\mathbf {g f}} (G) (\mathbf {x}) = x _ {k} ^ {c} \cdot G \left(\boldsymbol {x} ^ {\prime}\right)
$$
where  $x_{k}^{\prime}\coloneqq x_{k}^{a_{k}}$  and  $x_{i}^{\prime}\coloneqq x_{i}x_{k}^{a_{i}}$  for  $i\neq k$
$$
[   [ \text {i f} X _ {k} \in A \left\{P _ {1} \right\} \text {e l s e} \left\{P _ {2} \right\} ]   ] _ {\mathbf {g f}} (G) = [   [ P _ {1} ]   ] _ {\mathbf {g f}} \left(G _ {X _ {k} \in A}\right) + [   [ P _ {2} ]   ] _ {\mathbf {g f}} \left(G - G _ {X _ {k} \in A}\right)
$$
where  $G_{X_k\in A}(\mathbf{x}) = \sum_{i\in A}\frac{\partial_k^iG(\mathbf{x}[k\mapsto 0])}{i!} x_k^i$  and  $A\subset \mathbb{N}$  finite
if  $X_{k}$  has support in  $\mathbb{N}$
$$
\llbracket X _ {k} \sim D \rrbracket_ {\mathbf {g f}} (G) (\mathbf {x}) = G (\mathbf {x} [ k \mapsto 1 ]) \mathbf {g f} (D) \left(x _ {k}\right)
$$
$$
\llbracket X _ {k} \sim D (X _ {j}) \rrbracket_ {\mathbf {g f}} (G) (\mathbf {x}) = G (\mathbf {x} [ k \mapsto 1, j \mapsto x _ {j} \cdot \mathbf {g f} (D (1)) (x _ {k}) ])
$$
for  $D\in \{\mathrm{Binomial}(-,p),\mathrm{NegBinomial}(-,p),\mathrm{Poisson}(\lambda \cdot -)\}$
$$
\llbracket X _ {k} \sim \operatorname {B e r n o u l l i} (X _ {j}) \rrbracket_ {\mathbf {g f}} (G) (\mathbf {x}) = G (\boldsymbol {x} [ k \mapsto 1 ]) + x _ {j} (x _ {k} - 1) \cdot \partial_ {j} G (\boldsymbol {x} [ k \mapsto 1 ])
$$
if  $X_{j}$  has support in [0, 1]
$$
[   [ \text {f a i l} ]   ] _ {\mathrm {g f}} (G) = \mathbf {0}
$$
$\mathrm{norm}\mathit{aize}(G) = \frac{G}{G(\mathbf{1}_n)}$
Figure 5: The generating function (GF) semantics  $\llbracket -\rrbracket_{\mathrm{gf}}$
B.1 Generating function semantics
The full generating function semantics can be found in Fig. 5. The first five rules in Fig. 5 were presented in [17, 4]. The GF for sampling from  $\operatorname{Binomial}(X_j, p)$  was first given in [26]. Chen et al. [4] also implicitly describe the GF semantics of sampling from  $\operatorname{Binomial}(X_j, p)$  and  $\operatorname{NegBinomial}(X_j, p)$  by expressing it as summing  $X_j$  iid samples from Bernoulli  $(p)$  and Geometric  $(p)$ , respectively.
Let us discuss each rule in detail. The skip command leaves the distribution and hence the generating function unchanged. Chaining commands  $P_{1};P_{2}$  is again achieved by composing the individual transformers for each command. To obtain some intuition for the affine transformation case  $X_{k}\coloneqq \mathbf{a}^{\top}\mathbf{X} + c$ , suppose we only have 2 variables:  $X_{1}\coloneqq 2X_{1} + 3X_{2} + 5$ . The idea is that the transformed generating function is given by  $[[X_1\coloneqq 2X_1 + 3X_2 + 5]]_{\mathrm{gf}}(G)(x) = \mathbb{E}[x_1^{2X_1 + 3X_2 + 5}x_2^{X_2}] = x_1^5\mathbb{E}[(x_1^2)^{X_1}(x_1^3 x_2)^{X_2}] = x_1^5\cdot G(x_1^2,x_1^3 x_2)$ .
The semantics of if  $X_{k} \in A\{P_{1}\}$  else  $\{P_{2}\}$  uses  $G_{X_{k} \in A}$ . If  $G$  is the generating function for a measure  $\mu$  then  $G_{X_{k} \in A}$  is the generating function for the measure  $\mu_{X_{k} \in A}(S) := \mu(\{x \in S \mid x_{k} \in A\})$ . To get intuition for this, remember that for discrete variables  $X_{1}, X_{2}$ , we have  $G(x_{1}, x_{2}) = \sum_{X_{1}, X_{2}} \mu(\{(X_{1}, X_{2})\}) \cdot x_{1}^{X_{1}} x_{2}^{X_{2}}$ . If we want to obtain the generating function for  $X_{1} = 5$ , we want to keep only terms where  $x_{1}$  has exponent 5. In other words, we need the Taylor coefficient  $\frac{\partial_1^5 G(0, x_2)}{5!}$ . This should give some intuition for  $G_{X_{k} \in A}$ . The semantics of if  $X_{k} \in A\{P_{1}\}$  else  $\{P_{2}\}$  transforms  $G_{X_{k} \in A}$  in the then-branch and the complement  $G - G_{X_{k} \in A}$  in the else branch, summing the two possibilities.
For sampling, say  $X_{1} \sim D$ , we first marginalize out the old  $X_{1}$  by substituting 1 for  $x_{1}$  in  $G$ :  $\mathbb{E}[x_2^{X_2}] = \mathbb{E}[1^{X_1}x_2^{X_2}] = G(1,x_2)$ . Then we multiply with the GF of the new distribution  $D$ . The rules for compound distributions  $D(X_{j})$  are more involved, so we offer the following intuition.
For the compound distributions  $D(n) = \mathrm{Binomial}(n,p)$ ,  $\mathrm{NegBinomial}(n,p)$ , Poisson  $(\lambda \cdot n)$ , we make essential use of their property that  $\mathsf{gf}(D(n))(x) = (\mathsf{gf}(D(1))(x))^n$ . So if  $G(x) := \mathbb{E}[x^X]$  is the GF of  $X$  and  $Y \sim D(X)$ , then  $\mathbb{E}_{X,Y}[x^Xy^Y] = \mathbb{E}_X[x^X\mathbb{E}_{Y \sim D(X)}[y^Y]] = \mathbb{E}_X[x^X\mathsf{gf}(D(X))(y)] = \mathbb{E}_X[x^X(\mathsf{gf}(D(1))(y))^X] = \mathbb{E}[(x \cdot \mathsf{gf}(D(1))(y))^X] = G(x \cdot \mathsf{gf}(D(1))(y))$ . This is not a fully formal argument, but it should give some intuition for the rule.
For  $D(p) = \operatorname{Bernoulli}(p)$ , the intuition is that if  $G(x) \coloneqq \mathbb{E}[x^X]$  is the GF of  $X$  and if  $Y \sim \operatorname{Bernoulli}(X)$ , then  $H(x, y) = \mathbb{E}[x^Xy^Y] = \mathbb{E}[x^X((1 - X)y^0 + Xy^1)] = \mathbb{E}[x^X - (y - 1)Xx^X] = G(x) - (y - 1)xG'(x)$ .
The GF semantics of fail is the zero function because its distribution is zero everywhere. For normalization, note that substituting  $\mathbf{1}_n$  in the generating function computes the marginal probabilities, similarly to the substitution of 1 in the semantics of sampling. It scales the GF of the subprogram  $P$  by the inverse of the normalizing constant obtained by that substitution.
We hope these explanations give some intuition for the GF semantics, but of course, proof is required.
B.2 Correctness proof
For the correctness proof, we need to describe the set of  $\pmb{x} \in \mathbb{R}^n$  where the generating function is defined. For  $R > 0$  and a measure  $\mu$  on  $\mathbb{R}_{\geq 0}^n$ , define it to be the product of open intervals
$$
T (R, \mu) := \left(Q _ {1}, R\right) \times \dots \times \left(Q _ {n}, R\right)
$$
where  $Q_{i} = -R$  if the  $i$ -th component of  $\mu$  is supported on  $\mathbb{N}$  (i.e.  $\mu(S) = \mu(\{X \in S \mid X_{i} \in \mathbb{N}\})$  for any measurable  $S$ ) and  $Q_{i} = 0$  otherwise. The case split is necessary because the generating functions are better-behaved, and thus have a larger domain, if the measure is supported on  $\mathbb{N}$ . In fact, it is important that  $x_{i} = 0$  is possible in the  $\mathbb{N}$ -case because otherwise we would not be able to extract probability masses from the GF.
Theorem B.1 (Correctness). The  $GF$  semantics is correct w.r.t. the standard semantics  $\llbracket -\rrbracket_{\mathrm{std}}$ : for any SGCL program  $P$  and subprobability distribution  $\mu$  on  $\mathbb{R}_{\geq 0}^n$ , we have  $\llbracket P\rrbracket_{\mathrm{gf}}(\mathsf{gf}(\mu)) = \mathsf{gf}(\llbracket P\rrbracket_{\mathrm{std}}(\mu))$ . In particular, it correctly computes the  $GF$  of the posterior distribution of  $P$  as normalize  $(\llbracket P\rrbracket_{\mathrm{gf}}(\mathbf{1}))$ .
If  $\mathsf{gf}(\mu)$  is defined on  $T(R,\mu)$  for some  $R > 1$  then  $\llbracket P\rrbracket_{\mathsf{gf}}(\mathsf{gf}(\mu))$  is defined on  $T(R^{\prime},[[P]]_{\mathsf{std}}(\mu))$  for some  $R^{\prime} > 1$ . In particular, the GFs  $\llbracket P\rrbracket_{\mathsf{gf}}(\mathbf{1})$  and thus normalize  $(\llbracket P\rrbracket_{\mathsf{gf}}(\mathbf{1}))$  are defined on  $T(R,[[P]]_{\mathsf{std}}(\mu))$  for some  $R > 1$ .
Proof. As usual for such statements, we prove this by induction on the structure of the program. The program  $P$  can take one of the following forms:
**Skip.** For the trivial program, we have  $\llbracket P\rrbracket_{\mathrm{gf}}(G) = G$  and  $\llbracket P\rrbracket_{\mathrm{std}}(\mu) = \mu$ , so the claim is trivial.
**Chaining.** If  $P$  is  $P_1; P_2$ , we know by the inductive hypothesis that  $[[P_1]]_{\mathrm{gf}}(\mathsf{gf}(\mu)) = \mathsf{gf}([P_1]_{\mathrm{std}}(\mu))$  and similarly for  $P_2$ . Taking this together, we find by the definitions of the semantics:
$$
\begin{array}{l} [   [ P _ {1}; P _ {2} ]   ] _ {\mathrm {g f}} (\mathbf {g f} (\mu)) = [   [ P _ {2} ]   ] _ {\mathrm {g f}} ([   [ P _ {1} ]   ] _ {\mathrm {g f}} (\mathbf {g f} (\mu))) = [   [ P _ {2} ]   ] _ {\mathrm {g f}} (\mathbf {g f} ([   [ P _ {1} ]   ] _ {\mathrm {s t d}} (\mu))) \\ = \operatorname {g f} \left(\llbracket P _ {2} \rrbracket_ {\mathrm {s t d}} \left(\llbracket P _ {1} \rrbracket_ {\mathrm {s t d}} (\mu)\right)\right) = \operatorname {g f} \left(\llbracket P _ {1}; P _ {2} \rrbracket_ {\mathrm {s t d}} (\mu)\right). \\ \end{array}
$$