SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Soft conditioning: another supportable construct could be score  $a^{X_k}$  for  $a \in [0,1]$ , which multiplies the likelihood of the current path by  $a^{X_k}$ . We think score  $q(X_k)$  where  $q(t) \coloneqq \sum_{i=0}^{m} q_i t^i$  is a polynomial with coefficients  $q_i \in [0,1]$  and with  $q \leq 1$  on the domain of  $X_k$  could also be supportable using similar techniques as for observations from Bernoulli( $X_k$ ).
None of these extensions seemed very useful for real-world models, in particular, they are not needed for our benchmarks. For this reason we did not formalize, implement, or prove them.
Support for loops is another big question. Bounded loops are easy enough to deal with by fully unrolling them. However exact inference for programs with unbounded loops seems extremely difficult, unless the inference algorithm is given additional information, e.g. a probabilistic loop invariant as in [4] or at least a loop invariant template with a few holes to be filled with real constants [18]. However, finding such a loop invariant for nontrivial programs seems exceedingly difficult. Furthermore, the variables of looping programs may have infinite expected values, so the generating function may not be definable at the point 1. But in our semantics, evaluating at 1 is very important for marginalization. For these reasons, we consider exact inference in the presence of loops an interesting but very hard research problem.
C Details on implementation and optimizations
Overview Our tool takes an SGCL program  $P$  with  $n$  variables as input, and a variable  $X_{i}$  whose posterior distribution is to be computed. It translates it to the GF  $[[P]]_{\mathrm{gf}}(\mathbf{1})$  according to the GF semantics and normalizes it:  $G := \text{normalize}([\mathbb{P}]_{\mathrm{gf}}(\mathbf{1}))$ . This  $G$  is the GF of the posterior distribution of the program, from which it extracts the posterior moments of  $X_{i}$  and its probability masses (if  $X_{i}$  is discrete).
Computation of moments First, it computes the first four (raw) posterior moments from the factorial moments, which are obtained by differentiation of  $G$ :
$$
M _ {1} := \mathbb {E} [ X _ {i} ] = \partial_ {i} G (\mathbf {1} _ {n})
$$
$$
M _ {2} := \mathbb {E} \left[ X _ {i} ^ {2} \right] = \mathbb {E} \left[ \left(X _ {i}\right) \left(X _ {i} - 1\right) \right] + \mathbb {E} \left[ X _ {i} \right] = \partial_ {i} ^ {2} G \left(\mathbf {1} _ {n}\right) + \partial_ {i} G \left(\mathbf {1} _ {n}\right)
$$
$$
\begin{array}{l} M _ {3} := \mathbb {E} \left[ X _ {i} ^ {3} \right] = \mathbb {E} \left[ X _ {i} \left(X _ {i} - 1\right) \left(X _ {i} - 2\right) \right] + 3 \mathbb {E} \left[ X _ {i} \left(X _ {i} - 1\right) \right] + \mathbb {E} \left[ X _ {i} \right] \\ = \partial_ {i} ^ {3} G (\mathbf {1} _ {n}) + 3 \partial_ {i} ^ {2} G (\mathbf {1} _ {n}) + \partial_ {i} G (\mathbf {1} _ {n}) \\ \end{array}
$$
$$
\begin{array}{l} M _ {4} := \mathbb {E} \left[ X _ {i} ^ {4} \right] = \mathbb {E} \left[ X _ {i} \left(X _ {i} - 1\right) \left(X _ {i} - 2\right) \left(X _ {i} - 3\right) \right] + 6 \mathbb {E} \left[ X _ {i} \left(X _ {i} - 1\right) \left(X _ {i} - 2\right) \right] \\ + 7 \mathbb {E} \left[ X _ {i} \left(X _ {i} - 1\right) \right] + \mathbb {E} \left[ X _ {i} \right] \\ = \partial_ {i} ^ {4} G (\mathbf {1} _ {n}) + 6 \partial_ {i} ^ {3} G (\mathbf {1} _ {n}) + 7 \partial_ {i} ^ {2} G (\mathbf {1} _ {n}) + \partial_ {i} G (\mathbf {1} _ {n}) \\ \end{array}
$$
From the raw moments, it computes the first four (centered/standardized) moments, i.e. the expected value  $(\mu \coloneqq \mathbb{E}[X_i])$ , the variance  $(\sigma^2 \coloneqq \mathbb{V}[X_i])$ , the skewness  $(\mathbb{E}[(X_i - \mu)^3] / \sigma^3)$  and the kurtosis  $(\mathbb{E}[(X_i - \mu)^4] / \sigma^4)$ :
$$
\mu := \mathbb {E} [ X _ {i} ] = M _ {1}
$$
$$
\sigma^ {2} := \mathbb {V} [ X _ {i} ] = \mathbb {E} [ (X _ {i} - \mu) ^ {2} ] = \mathbb {E} [ X _ {i} ^ {2} ] - 2 \mu \mathbb {E} [ X _ {i} ] + \mu^ {2} = M _ {2} - \mu^ {2}
$$
$$
\begin{array}{l} \operatorname {S k e w} \left[ X _ {i} \right] = \frac {\mathbb {E} \left[ \left(X _ {i} - \mu\right) ^ {3} \right]}{\sigma^ {3}} = \frac {\mathbb {E} \left[ X _ {i} ^ {3} \right] - 3 \mu \mathbb {E} \left[ X _ {i} ^ {2} \right] + 3 \mu^ {2} \mathbb {E} \left[ X _ {i} \right] - \mu^ {3}}{\sigma^ {3}} \\ = \frac {M _ {3} - 3 \mu M _ {2} + 2 \mu^ {3}}{\sigma^ {3}} \\ \end{array}
$$
$$
\begin{array}{l} \mathsf {K u r t} [ X _ {i} ] = \frac {\mathbb {E} [ (X _ {i} - \mu) ^ {4} ]}{\sigma^ {4}} = \frac {\mathbb {E} [ X _ {i} ^ {4} ] - 4 \mu \mathbb {E} [ X _ {i} ^ {3} ] + 6 \mu^ {2} \mathbb {E} [ X _ {i} ^ {2} ] - 4 \mu^ {3} \mathbb {E} [ X _ {i} ] + \mu^ {4}}{\sigma^ {4}} \\ = \frac {M _ {4} - 4 \mu M _ {3} + 6 \mu^ {2} M _ {2} - 3 \mu^ {4}}{\sigma^ {4}} \\ \end{array}
$$
Computation of probability masses If  $X_{i}$  is a discrete variable, the tool computes all the probability masses  $\mathbb{P}[X_i = m]$  until the value of  $m$  such that the tail probabilities are guaranteed to be below a certain threshold, which is set to  $\mathbb{P}[X_i \geq m] \leq \frac{1}{256}$ . This is achieved by setting
$m := \mu + 4\sqrt[4]{\mathbb{E}[(X_i - \mu)^4]}$  where  $\mu := \mathbb{E}[X_i]$ . Then we find
$$
\mathbb {P} [ X _ {i} \geq m ] \leq \mathbb {P} \left[ | X _ {i} - \mu | \geq 4 \sqrt [ 4 ]{\mathbb {E} [ (X _ {i} - \mu) ^ {4} ]} \right] = \mathbb {P} \left[ (X - \mu) ^ {4} \geq 2 5 6 \mathbb {E} [ (X _ {i} - \mu) ^ {4} ] \right] \leq \frac {1}{2 5 6}
$$
by Markov's inequality. Note that in practice, the tail probabilities are typically much smaller than  $\frac{1}{256}$ , usually in the order of  $10^{-5}$ . For all  $k \in \{0, \dots, m\}$ , the tool computes the posterior probability  $\mathbb{P}[X_i = k]$  by marginalizing out all the other variables (substituting 1 for them) and computing the Taylor coefficient at  $x_i = 0$ :
$$
\mathbb {P} [ X _ {i} = k ] = \frac {1}{k !} \partial_ {i} ^ {k} G (\mathbf {1} _ {n} [ i \mapsto 0 ]).
$$
It would be desirable to compute posterior densities for continuous distributions as well. In fact, there are mathematical ways of recovering the probability density function from a generating function via an inverse Laplace transform. However, this cannot be automated in practice because it requires solving integrals, which is intractable.
Implementation details Our tool Genfer is implemented in Rust [20], a safe systems programming language. The main reasons were low-level control and performance: the operations on the Taylor polynomials to evaluate derivatives of the generating function need to be fast and are optimized to exploit the structure of GFs arising from probabilistic programs (see Appendix C.3). C or  $\mathrm{C + + }$  would have satisfied the performance criterion as well, but Rust's language features like memory safety, enums (tagged unions), and pattern matching made the implementation a lot more pleasant, robust, and easier to maintain. The first author's experience with Rust was another contributing factor.
The coefficients of the Taylor polynomials are stored in a multidimensional array provided by the ndarray library<sup>4</sup>. Arbitrary-precision floating-point numbers and unbounded rational numbers are provided by the rug library<sup>5</sup>, which is an interface to the GNU libraries GMP (for rationals) and MPFR (for floats). Our implementation is available on GitHub: github.com/fzaiser/genfer
C.1 Automatic differentiation and Taylor polynomials
The main difficulty in implementing the GF semantics is the computation of the (partial) derivatives. This seems to be a unique situation where we want to compute  $d$ -th derivatives where  $d$  is in the order of hundreds. The reason is that the observe  $X_{k} = d$  construct is translated into a  $d$ -th (partial) derivative and if  $d$  is a real-world observation, it can be large. We have not come across another application of automatic differentiation that required derivatives of a total order of more than 10.
As a consequence, when we tried PyTorch, a mature machine learning library implementing automatic differentiation, the performance for derivatives of high order was very poor. Therefore, we decided to implement our own automatic differentiation framework. Our approach is to compute the Taylor expansion in all variables up to some order  $d$  of the generating function instead of a symbolic expression of the  $d$ -th derivative. This has the advantage of growing polynomially in  $d$ , not exponentially. Contrary to [27], it is advantageous to use Taylor coefficients instead of the partial derivatives because the additional factorial factors are can easily lead to overflows.
Taylor polynomials More formally, we define the Taylor polynomial  $\text{Taylor}_w^d(G)$  of a function  $G: \mathbb{R}^n \to \mathbb{R}$  at  $w \in \mathbb{R}^n$  of order  $d$  as the polynomial
$$
\operatorname {T a y l o r} _ {\boldsymbol {w}} ^ {d} (G) := \sum_ {\boldsymbol {\alpha} \in \mathbb {N} ^ {n}: | \boldsymbol {\alpha} | \leq d} \frac {1}{\boldsymbol {\alpha} !} \partial^ {\boldsymbol {\alpha}} G (\boldsymbol {w}) \cdot (\boldsymbol {x} - \boldsymbol {w}) ^ {\boldsymbol {\alpha}}
$$
where we used multi-index notation:  $|\alpha|$  stands for  $\alpha_1 + \dots + \alpha_n$ ;  $\alpha!$  for  $\alpha_1! \cdots \alpha_n!$ ; and  $\partial^\alpha$  for  $\partial_1^{\alpha_1} \ldots \partial_n^{\alpha_n}$ . The coefficients  $c_\alpha := \frac{1}{\alpha!} \partial^\alpha G(\boldsymbol{w})$  are the Taylor coefficients of  $G$  at  $\boldsymbol{w}$  and are stored in a multidimensional array in our implementation.
4https://crates.io/crates/ndarray
5https://crates.io/crates/rug
Operations The operations on generating functions (Appendix B) are implemented in terms of their effect on the Taylor expansions. Let  $G, H: \mathbb{R}^n \to \mathbb{R}$  be two functions with Taylor expansions  $\text{Taylor}_w^d(G) = \sum_{|\alpha| \leq d} g_\alpha(\boldsymbol{x} - \boldsymbol{w})^\alpha$  and  $\text{Taylor}_w^d(H) = \sum_{|\alpha| \leq d} h_\alpha(\boldsymbol{x} - \boldsymbol{w})^\alpha$ . Addition  $F = G + H$  of two generating functions is implemented by adding the coefficients:  $\text{Taylor}_w^d(G + H) = \sum_{|\alpha| \leq d} (g_\alpha + h_\alpha)(\boldsymbol{x} - \boldsymbol{w})^\alpha$ . Scalar multiplication  $F = c \cdot G$  is implemented by multiplying the coefficients:  $\text{Taylor}_w^d(c \cdot G) = \sum_{|\alpha| \leq d} (c \cdot g_\alpha)(\boldsymbol{x} - \boldsymbol{w})^\alpha$ . Multiplication  $F = G \cdot H$  of two generating functions is implemented by the Cauchy product:
$$
\operatorname {T a y l o r} _ {\boldsymbol {w}} ^ {d} (G \cdot H) = \sum_ {| \boldsymbol {\alpha} | \leq d} \left(\sum_ {\boldsymbol {\alpha} _ {1} + \boldsymbol {\alpha} _ {2} = \boldsymbol {\alpha}} g _ {\boldsymbol {\alpha} _ {1}} h _ {\boldsymbol {\alpha} _ {2}}\right) (\boldsymbol {x} - \boldsymbol {w}) ^ {\boldsymbol {\alpha}}.
$$
Division, exponentiation, logarithms, and powers can be implemented as well (see [13, Chapter 13] for details). Partial derivatives essentially correspond to shifting the index and a multiplication:
$$
\operatorname {T a y l o r} _ {\boldsymbol {w}} ^ {d - 1} \left(\partial_ {k} G\right) = \sum_ {| \boldsymbol {\alpha} | \leq d - 1} \left(\alpha_ {k} + 1\right) g _ {\boldsymbol {\alpha} \left[ k \mapsto \alpha_ {k} + 1 \right]} \boldsymbol {x} ^ {\boldsymbol {\alpha}}
$$
The most complicated case is composition/substitution, i.e.  $F(\pmb{x}) = G(\pmb{x}[k \mapsto H(\pmb{x})])$ . For this, we let  $\pmb{w}' := w[k \mapsto h_{\pmb{0}}]$  where  $\mathsf{Taylor}_{\pmb{w}}^{d}(H) = \sum_{|\beta| \leq d} h_{\beta} (\pmb{x} - \pmb{w})^{\beta}$  and we let  $\mathsf{Taylor}_{\pmb{w}'}^{d}(G) = \sum_{|\alpha| \leq d} g_{\alpha} (\pmb{x} - \pmb{w}')^{\alpha}$ . Then we have (where "h.o.t." stands for "higher order terms")
$$
\begin{array}{l} G (\boldsymbol {x} [ k \mapsto H (\boldsymbol {x}) ]) = G \left(\boldsymbol {x} \left[ k \mapsto \sum_ {| \beta | \leq d} h _ {\beta} (\boldsymbol {x} - \boldsymbol {w}) ^ {\beta} + \text {h . o . t .} \right]\right) \\ = \sum_ {| \boldsymbol {\alpha} | \leq d} g _ {\boldsymbol {\alpha}} \left(\boldsymbol {x} \left[ k \mapsto h _ {\mathbf {0}} + \sum_ {1 \leq | \boldsymbol {\beta} | \leq d} h _ {\boldsymbol {\alpha}} (\boldsymbol {x} - \boldsymbol {w}) ^ {\boldsymbol {\beta}} + \text {h . o . t .} \right] - \boldsymbol {w} ^ {\prime}\right) ^ {\boldsymbol {\alpha}} + \text {h . o . t .} \\ = \sum_ {| \boldsymbol {\alpha} | \leq d} g _ {\boldsymbol {\alpha}} \left(\left(\boldsymbol {x} - \boldsymbol {w}\right) \left[ k \mapsto \sum_ {1 \leq | \beta | \leq d} h _ {\boldsymbol {\alpha}} (\boldsymbol {x} - \boldsymbol {w}) ^ {\beta} \right]\right) ^ {\boldsymbol {\alpha}} + \mathrm {h . o . t .} \\ \end{array}
$$
because by definition  $\pmb{x}[k\mapsto h_0] - w' = (\pmb{x} - \pmb{w})[k\mapsto 0]$ . This means that we can obtain  $\mathsf{Taylor}_\pmb{w}^d (F)$  by substituting  $\sum_{1\leq |\beta |\leq d}h_{\pmb{\alpha}}(\pmb {x} - \pmb {w})^{\beta}$  for  $x_{k} - w_{k}^{\prime}$  in  $\mathsf{Taylor}_{\pmb{w}'}^d (G)$ .
Example C.1 (Calculations with Taylor polynomials). Here we show how the expectation for Example 2.1 can be computed using Taylor polynomials. We use the same naming as in Example 3.1. Recall that to compute  $\mathbb{E}[X]$ , we need to evaluate  $\partial_1 E(1,1)$ , so we compute the Taylor expansion of  $E$  at  $(1,1)$  of order 1, i.e.  $\mathsf{Taylor}_{(1,1)}^{1}(E)$ . To compute  $E(x,y) = \frac{D(x,y)}{D(1,1)}$ , we need  $D(x,y)$ , so we need  $\mathsf{Taylor}_{(1,1)}^{1}(D)$ . For  $D(x,y) = \frac{1}{2!} y^2 \partial_y^2 C(x,0)$ , we need the Taylor expansion of  $C$  at  $(1,0)$  of order  $1 + 2 = 3$ . For  $C(x,y) = B(x(0.1y + 0.9),1)$ , we need the Taylor expansion of  $B$  at  $(1(0.1 \cdot 0 + 0.9),1) = (0.9,1)$  of order 3. Since  $B(x,y) = \exp(20(x - 1)) = \exp(20((x - 0.9) - 0.1)) = \exp(20(x - 0.9) - 2)$ , this Taylor expansion is
$$
\operatorname {T a y l o r} _ {(0. 9, 1)} ^ {3} (B) = e ^ {- 2} + 2 0 e ^ {- 2} \cdot (x - 0. 9) + 2 0 0 e ^ {- 2} \cdot (x - 0. 9) ^ {2} + \frac {4 0 0 0}{3} e ^ {- 2} (x - 0. 9) ^ {3}.
$$
Since  $C(1 + (x - 1), 0 + y) = B((1 + (x - 1))(0.1y + 0.9), 1) = B(0.9 + 0.9(x - 1) + 0.1y + 0.1(x - 1)y, 1)$ , the Taylor expansion of  $C$  at  $(1, 0)$  is given by replacing  $(x - 0.9)$  with  $0.9(x - 1) + 0.1y + 0.1(x - 1)y$  and we get
$$
\begin{array}{l} \mathsf {T a y l o r} _ {(1, 0)} ^ {3} (C) = e ^ {- 2} \big (1 + 2 y + 2 y ^ {2} + \frac {4}{3} y ^ {3} + 1 8 (x - 1) + 3 8 (x - 1) y + 4 0 (x - 1) y ^ {2} \\ + 1 6 2 (x - 1) ^ {2} + 3 6 0 (x - 1) ^ {2} y + 9 7 2 (x - 1) ^ {3}) \\ \end{array}
$$
For  $D(x,y) = \frac{1}{2!} y^2\partial_y^2 C(x,0)$ , we compute the second partial derivative wrt.  $y$ , substitute  $y = 0$  and then multiply by  $y^{2} = (1 + (y - 1))^{2}$ , yielding
$$
\begin{array}{l} D (x, y) = \frac {1}{2 !} y ^ {2} e ^ {- 2} (4 + 8 y + 8 0 (x - 1) + \text {h i g h e r o r d e r t e r m s}) | _ {y = 0} \\ = e ^ {- 2} (4 + 8 0 (x - 1) + \text {h i g h e r o r d e r t e r m s}) (1 + (y - 1)) ^ {2} \\ = e ^ {- 2} (4 + 8 0 (x - 1) + 8 (y - 1) + \text {h i g h e r o r d e r t e r m s}) \\ \end{array}
$$
As a consequence, we find that  $D(1,1) = 4e^{-2}$ , so  $\mathsf{Taylor}_{(1,1)}^{1}(E) = 1 + 20(x - 1) + 2(y - 1)$ . So  $\mathbb{E}[X] = \partial_x E(1,1) = 20$ , as desired.
Memoizing intermediate results When computing the Taylor expansion of a function  $G$ , it is important to memoize the intermediate Taylor expansions of subexpressions of  $G$  if they occur more than once. For example, conditionals are translated to a generating function that uses the previous generating function twice. Evaluating it repeatedly would double the running time for each conditional. With memoization, it is possible to handle programs with lots of branching (e.g.  $> 2^{100}$  paths in the mixture model, cf. Table 5) in a reasonable amount of time.
C.2 Optimizing Observations from Compound Distributions
Observations from compound distributions are generally the bottleneck for the running time because the construct observe  $d \sim D(X_k)$  is syntactic sugar for  $X_{n+1} \sim D(X_k)$ ; observe  $X_{n+1} = d$ , which introduces a new variable, which is immediately discarded after the observation. This expansion has the semantics
$$
\begin{array}{l} [   [ \text {o b s e r v} d \sim D (X _ {k}) ]   ] _ {\mathbf {g f}} (G) (\boldsymbol {x}) = [   [ \text {o b s e r v} X _ {n + 1} = d ]   ] _ {\mathbf {g f}} ([   [ X _ {n + 1} \sim D (X _ {k}) ]   ] _ {\mathbf {g f}} (G)) (x _ {1}, \dots , x _ {n + 1}) \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} [   [ X _ {n + 1} \sim D (X _ {k}) ]   ] _ {\mathbf {g f}} (G) (x _ {1}, \dots , x _ {n + 1}) \Bigg | _ {x _ {n + 1} = 0} \\ \end{array}
$$
containing an extra variable  $x_{n + 1}$ , which worsens the running time of the Taylor approach significantly. We can find more efficient ways of computing this semantics.
Theorem C.1. Observing from compound binomial distributions can be implemented without introducing a new variable:
$$
\llbracket \operatorname {o b s e r v e} d \sim \operatorname {B i n o m i a l} (X _ {k}, p) \rrbracket_ {\mathbf {g f}} (G) = \frac {1}{d !} (p x _ {k}) ^ {d} \cdot \partial_ {k} ^ {d} G (\boldsymbol {x} [ k \mapsto (1 - p) x _ {k} ])
$$
Proof. A proof for the discrete setting was given in [27]. In our setting, which allows continuous distributions, we cannot use the usual argument about power series anymore. Instead, we reason as follows:
$$
\begin{array}{l} [   [ \text {o b s e r v} d \sim \operatorname {B i n o m i a l} (X _ {k}, p) ]   ] _ {\mathbf {g f}} (G) \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} [   [ X _ {n + 1} \sim \operatorname {B i n o m i a l} (X _ {k}) ]   ] _ {\mathbf {g f}} (G) (x _ {1}, \dots , x _ {n + 1}) \Big | _ {x _ {n + 1} = 0} \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} G (\boldsymbol {x} [ k \mapsto x _ {k} \cdot (p x _ {n + 1} + 1 - p) ]) \Bigg | _ {x _ {n + 1} = 0} \\ \stackrel{*}{=}\frac{1}{d!}\partial_{k}^{d}G(\boldsymbol {x}[k\mapsto x_{k}\cdot (px_{n + 1} + 1 - p)])\cdot (px_{k})^{d}\bigg|_{x_{n + 1} = 0} \\ = \frac {1}{d !} \left(p x _ {k}\right) ^ {d} \partial_ {k} ^ {d} G \left(\boldsymbol {x} [ k \mapsto (1 - p) x _ {k} ]\right) \\ \end{array}
$$
where  $*$  follows by iterating the following argument:
$$
\begin{array}{l} \frac {\partial}{\partial x _ {n + 1}} G (\boldsymbol {x} [ k \mapsto x _ {k} \cdot (p x _ {n + 1} + 1 - p) ]) \\ = \partial_ {k} G (\boldsymbol {x} [ k \mapsto x _ {k} \cdot (p x _ {n + 1} + 1 - p) ]) \cdot \frac {\partial (x _ {k} \cdot (p x _ {n + 1} + 1 - p))}{\partial x _ {n + 1}} \\ = \partial_ {k} G (\boldsymbol {x} [ k \mapsto x _ {k} \cdot (p x _ {n + 1} + 1 - p) ]) \cdot p x _ {k} \\ \end{array}
$$
which holds by the chain rule.
Theorem C.2. Observing from compound Poisson distributions can be implemented without introducing a new variable:
$$
[   [ \text {o b s e r v e} d \sim \text {P o i s s o n} (\lambda X _ {k}) ]   ] _ {\mathbf {g f}} (G) = \frac {1}{d !} D _ {k, \lambda} ^ {d} (G) (\boldsymbol {x} [ k \mapsto e ^ {- \lambda} x _ {k} ])
$$
$$
D _ {k, \lambda} (G) (\boldsymbol {x}) := \lambda x _ {k} \cdot \partial_ {k} G (\boldsymbol {x}).
$$
Proof. Let  $F:\mathbb{R}^n\to \mathbb{R}$  be a smooth function. Then
$$
\begin{array}{l} \frac {\partial}{\partial x _ {n + 1}} F \left(\boldsymbol {x} \left[ k \mapsto x _ {k} \cdot e ^ {\lambda \left(x _ {n + 1} - 1\right)} \right]\right) \\ = \partial_ {k} F (\boldsymbol {x} [ k \mapsto x _ {k} \cdot e ^ {\lambda (x _ {n + 1} - 1)} ]) \cdot x _ {k} e ^ {\lambda (x _ {n + 1} - 1)} \cdot \lambda \\ = D _ {k, \lambda} (F) (\boldsymbol {x} [ k \mapsto x _ {k} \cdot e ^ {\lambda (x _ {n + 1} - 1)} ]). \\ \end{array}
$$
Inductively, we get
$$
\frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} G (\boldsymbol {x} [ k \mapsto x _ {k} \cdot e ^ {\lambda (x _ {n + 1} - 1)} ]) = D _ {k, \lambda} ^ {d} (G) (\boldsymbol {x} [ k \mapsto x _ {k} \cdot e ^ {\lambda (x _ {n + 1} - 1)} ]).
$$
Substituting  $x_{n + 1}\mapsto 0$  in the final expression and dividing by  $d!$  yields:
$$
\begin{array}{l} [   [ \text {o b s e r v e} d \sim \text {P o i s s o n} (\lambda X _ {k}) ]   ] _ {\mathbf {g f}} (G) \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} \llbracket X _ {n + 1} \sim \operatorname {P o i s s o n} (\lambda X _ {k}) \rrbracket_ {\mathbf {g f}} (G) (x _ {1}, \dots , x _ {n + 1}) \Bigg | _ {x _ {n + 1} = 0} \\ = \frac {1}{d !} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} G (\boldsymbol {x} [ k \mapsto x _ {k} \cdot e ^ {\lambda (x _ {n + 1} - 1)} ]) \Bigg | _ {x _ {n + 1} = 0} \\ = \frac {1}{d !} D _ {k, \lambda} ^ {d} (G) \left(\boldsymbol {x} [ k \mapsto e ^ {- \lambda} x _ {k} ]\right) \\ \end{array}
$$
Theorem C.3. Observing from compound negative binomial distributions can be implemented without introducing a new variable:
$$
\llbracket \operatorname {o b s e r v e} d \sim \operatorname {N e g B i n o m i a l} (X _ {k}, p) \rrbracket_ {\mathbf {g f}} (G) = \frac {1}{d !} \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto p \cdot x _ {k} ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d} L _ {d, i}
$$
where the numbers  $L_{d,i}$  are known as Lah numbers and defined recursively as follows:
$$
L _ {0, 0} := 1
$$
$$
L _ {d, i} := 0
$$
$$
f o r i <   0 o r d <   i
$$
$$
L _ {d + 1, i} := (d + i) L _ {d, i} + L _ {d, i - 1}
$$
$$
f o r 0 \leq i \leq d + 1
$$
Proof. Let  $F: \mathbb{R}^n \to \mathbb{R}$  be a smooth function and  $y: \mathbb{R} \to \mathbb{R}$  given by  $y(z) := \frac{1}{1 - (1 - p)z}$ . Then  $y'(x) = -\frac{1}{(1 - (1 - p)x)^2} \cdot (-(1 - p)) = (1 - p)y(x)^2$  and
$$
\begin{array}{l} \frac {\partial}{\partial x _ {n + 1}} F \left(\boldsymbol {x} [ k \mapsto \frac {x _ {k} \cdot p}{1 - (1 - p) x _ {n + 1}} ]\right) \\ = \partial_ {k} F (\boldsymbol {x} [ k \mapsto x _ {k} p \cdot y (x _ {n + 1}) ]) \cdot p x _ {k} \cdot (1 - p) y (x _ {n + 1}) ^ {2} \\ \end{array}
$$
We claim that we can write
$$
\begin{array}{l} \frac {\partial^ {d}}{\partial x _ {n + 1} ^ {d}} F (\boldsymbol {x} [ k \mapsto \frac {p x _ {k}}{1 - (1 - p) x _ {n + 1}} ]) \\ = \sum_ {i = 0} ^ {d} \partial_ {k} ^ {i} F (\boldsymbol {x} [ k \mapsto x _ {k} \cdot y (x _ {n + 1}) ]) \cdot p ^ {i} x _ {k} ^ {i} \cdot (1 - p) ^ {d} L _ {d, i} \cdot y (x _ {n + 1}) ^ {d + i}. \\ \end{array}
$$
The claim is proved by induction. First the case  $d = 0$ :
$$
F (\boldsymbol {x} [ k \mapsto \frac {p x _ {k}}{1 - (1 - p) x _ {n + 1}} ]) = F (\boldsymbol {x} [ k \mapsto \frac {p x _ {k}}{1 - (1 - p) x _ {n + 1}} ]) \cdot (1 - p) ^ {0} x _ {k} ^ {0} L _ {0, 0} \cdot y (x _ {n + 1}) ^ {0}
$$
Next, the induction step:
$$
\frac {\partial^ {d + 1}}{\partial x _ {n + 1} ^ {d + 1}} F (\boldsymbol {x} [ k \mapsto \frac {p x _ {k}}{1 - (1 - p) x _ {n + 1}} ])
$$