SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The claim about the domain of  $\llbracket P\rrbracket_{\mathrm{gf}}(G)$  follows directly from the inductive hypotheses.
Affine assignments. If  $P$  is  $X_{k} \coloneqq \mathbf{a}^{\top}\mathbf{X} + c$ , we have  $[[P]]_{\mathrm{std}}(\mu)(S) = \mu (\{\mathbf{X} \in \mathbb{R}^{n} \mid \mathbf{X}[k \mapsto \mathbf{a}^{\top}\mathbf{X} + c] \in S\})$  and thus
$$
\begin{array}{l} \mathbf {g f} ([   [ P ]   ] _ {\mathrm {s t d}} (\mu)) (\boldsymbol {x}) = \int \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} ([   [ P ]   ] _ {\mathrm {s t d}} (\mu)) (\boldsymbol {X}) \\ = \int x _ {1} ^ {X _ {1}} \dots x _ {k - 1} ^ {X _ {k - 1}} x _ {k} ^ {a _ {1} X _ {1} + \dots + a _ {n} X _ {n} + c} x _ {k + 1} ^ {X _ {k + 1}} \dots x _ {n} ^ {X _ {n}} d \mu (X _ {1}, \dots , X _ {n}) \\ = x _ {k} ^ {c} \cdot \int \left(x _ {1} x _ {k} ^ {a _ {1}}\right) ^ {X _ {1}} \dots \left(x _ {k - 1} x _ {k} ^ {a _ {k - 1}}\right) ^ {X _ {k - 1}} \left(x _ {k} ^ {a _ {k}}\right) ^ {X _ {k}}. \\ \left(x _ {k + 1} x _ {k} ^ {a _ {k + 1}}\right) ^ {X _ {k + 1}} \dots \left(x _ {n} x _ {k} ^ {a _ {n}}\right) ^ {X _ {n}} d \mu (\boldsymbol {X}) \\ = x _ {k} ^ {c} \cdot \operatorname {g f} (\mu) \left(x _ {1} x _ {k} ^ {a _ {1}}, \dots , x _ {k - 1} x _ {k} ^ {a _ {k - 1}}, x _ {k} ^ {a _ {k}}, x _ {k + 1} x _ {k} ^ {a _ {k + 1}}, \dots , x _ {n} x _ {k} ^ {a _ {n}}\right) \\ = \llbracket P \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu)) (\boldsymbol {x}) \\ \end{array}
$$
The claim that it is defined on some  $T(R', \llbracket P \rrbracket_{\mathrm{std}}(\mu))$  holds if we choose  $R' = \min(\sqrt{R}, \min_{i=1}^{n} 2^{a_i} \sqrt{R}) > 1$  because then  $|x_i x_k^{a_i}| < \sqrt{R} (\sqrt[2a_i]{R})^{a_i} \leq R$  for  $i \neq k$  and  $|x_k^{a_k}| < (\sqrt[2a_k]{R})^{a_k} \leq \sqrt{R} \leq R$ .
Conditionals. If  $P$  is if  $X_{k} \in A\{P_{1}\}$  else  $\{P_2\}$ , the GF semantics defines  $(\mathrm{gf}(\mu))_{X_k \in A}$ , which is the same as  $\mathrm{gf}(\mu_{X_k \in A})$  by Lemma B.2. Using this fact, the linearity of  $\mathrm{gf}(-)$ , and the induction hypothesis, we find
$$
\begin{array}{l} \operatorname {g f} \left(\llbracket P \rrbracket_ {\operatorname {s t d}} (\mu)\right) = \operatorname {g f} \left(\llbracket P _ {1} \rrbracket_ {\operatorname {s t d}} \left(\mu_ {X _ {k} \in A}\right)\right) + \operatorname {g f} \left(\llbracket P _ {2} \rrbracket_ {\operatorname {s t d}} \left(\mu - \mu_ {X _ {k} \in A}\right)\right) \\ = \llbracket P _ {1} \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu_ {X _ {k} \in A})) + \llbracket P _ {2} \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu) - \mathbf {g f} (\mu_ {X _ {k} \in A})) \\ = \llbracket P _ {1} \rrbracket_ {\mathbf {g f}} ((\mathbf {g f} (\mu)) _ {X _ {k} \in A}) + \llbracket P _ {2} \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu) - (\mathbf {g f} (\mu)) _ {X _ {k} \in A}) \\ = \llbracket P \rrbracket_ {\mathrm {g f}} (\operatorname {g f} (\mu)) \\ \end{array}
$$
Also by Lemma B.2, we know that  $(\mathsf{gf}(\mu))_{X_k\in A}$  (and thus  $(\mathsf{gf}(\mu) - (\mathsf{gf}(\mu))_{X_k\in A}))$  is defined on  $T(R,\mu)$ , so by the induction hypothesis, both  $[P_1]_{\mathsf{gf}}((\mathsf{gf}(\mu))_{X_k\in A})$  and  $[P_2]_{\mathsf{gf}}(\mathsf{gf}(\mu) - \mathsf{gf}(\mu_{X_k\in A}))$  are defined on some  $T(R^{\prime})$ . Hence the same holds for  $[P]_{\mathsf{gf}}(\mathsf{gf}(\mu))$ .
Sampling. If  $P$  is  $X_{k}\sim D$ , we find:
$$
\begin{array}{l} \operatorname {g f} \left(\llbracket P \rrbracket_ {\mathrm {s t d}} (\mu)\right) (\boldsymbol {x}) = \int \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} \left(\llbracket P \rrbracket_ {\mathrm {s t d}} (\mu)\right) (\boldsymbol {X}) \\ = \iint \boldsymbol {x} ^ {\boldsymbol {X} [ k \mapsto Y ]} \mathrm {d} D (Y) \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \int x _ {k} ^ {Y} \mathrm {d} D (Y) \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \cdot \int x _ {k} ^ {Y} \mathrm {d} D (Y) \\ = \operatorname {g f} (\mu) (\boldsymbol {x} [ k \mapsto 1 ]) \operatorname {g f} (D) \left(x _ {k}\right) \\ = \llbracket P \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu)) (\boldsymbol {x}) \\ \end{array}
$$
Regarding the domain of  $\llbracket P\rrbracket_{\mathsf{gf}}(\mathsf{gf}(\mu))$ , note that  $\mathsf{gf}(D)$  is defined on all of  $\mathbb{R}$  for any finite distribution and for the Poisson distribution (cf. Table 4a). For  $D\in \{\mathrm{Geometric}(p),\mathrm{NegBinomial}(n,p)\}$ , the GF  $\mathsf{gf}(D)$  is defined on  $(-\frac{1}{1 - p},\frac{1}{1 - p})$ , so we can pick  $R^{\prime}\coloneqq \min (R,\frac{1}{1 - p}) > 1$ . The GF of
Exponential  $(\lambda)$  is defined for  $\log x < \lambda$ , i.e.  $x < \exp(\lambda)$ , so we can pick  $R' := \min(R, e^{\lambda}) > 1$ . The GF of  $\operatorname{Gamma}(\alpha, \beta)$  is defined for  $\log x < \beta$ , i.e.  $x < \exp(\beta)$ , so we can pick  $R' := \min(R, e^{\beta}) > 1$ . The GF of  $\operatorname{Uniform}[a, b]$  is defined on  $\mathbb{R}$ , so  $R' := R$  works.
Compound distributions. If  $P$  is  $X_{k} \sim D(X_{j})$ , we use the fact that  $\mathrm{gf}(D(n))(\pmb{x}) = (\mathrm{gf}(D(1))(\pmb{x}))^{n}$  for  $D(n) \in \{\text{Binomial}(n,p), \text{NegBinomial}(n,p), \text{Poisson}(\lambda \cdot n)\}$ , which can easily be checked by looking at their generating functions (cf. Table 4b).
$$
\begin{array}{l} \mathbf {g f} ([ P ] _ {\mathrm {s t d}} (\mu)) (\boldsymbol {x}) = \int \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} ([ P ] _ {\mathrm {s t d}} (\mu)) (\boldsymbol {X}) \\ = \int \int \boldsymbol {x} ^ {\boldsymbol {X} [ k \mapsto Y ]} \mathrm {d} D (X _ {j}) (Y) \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \int x _ {k} ^ {Y} \mathrm {d} D (X _ {j}) (Y) \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \operatorname {g f} (D (X _ {j})) (x _ {k}) \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} (\operatorname {g f} (D (1)) (x _ {k})) ^ {X _ {j}} \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1, j \mapsto x _ {j} \cdot \operatorname {g f} (D (1)) (x _ {k}) ]) ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \\ = \operatorname {g f} (\mu) \left(\boldsymbol {x} [ k \mapsto 1, j \mapsto x _ {j} \cdot \operatorname {g f} (D (1)) (x _ {k}) ]\right) \\ = \llbracket P \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu)) (\boldsymbol {x}) \\ \end{array}
$$
Regarding the domain of  $\llbracket P\rrbracket_{\mathrm{gf}}(\mathrm{gf}(\mu))$ , we have to ensure that  $|x_j\cdot \mathrm{gf}(D(1))(x_k)| < R$ . For the Binomial  $(1,p)$  distribution, we choose  $R^{\prime} = \sqrt{R} > 1$  such that  $|x_j||1 - p + px_k| < \sqrt{R}(1 - p + p\sqrt{R}) \leq \sqrt{R}((1 - p)\sqrt{R} + p\sqrt{R}) < R$ . For the NegBinomial  $(1)$  distribution, we choose  $R^{\prime} = \min(\sqrt{R}, \frac{\sqrt{R} - p}{\sqrt{R}(1 - p)}) = \min(\sqrt{R}, 1 + \frac{p(\sqrt{R} - 1)}{\sqrt{R} - p\sqrt{R}}) > 1$  because for any  $x \in (-R^{\prime}, R)$ , we have
$$
\left| \frac {p}{1 - (1 - p) x _ {j}} \right| <   \frac {p}{1 - (1 - p) R ^ {\prime}} \leq \frac {p}{1 - \frac {\sqrt {R} - p}{\sqrt {R}}} = \frac {p \sqrt {R}}{p} = \sqrt {R}
$$
and thus  $\left|x_{j}\frac{p}{1 - (1 - p)x_{k}}\right| < R$ . For the Poisson(λ) distribution, we choose  $R' = \min(\sqrt{R}, 1 + \frac{\log(R)}{2\lambda}) > 1$  because then  $|x_{j} \cdot \exp(\lambda(x_{k} - 1))| < \sqrt{R} \exp(\lambda\frac{\log(R)}{2\lambda}) = R$ .
For the Bernoulli distribution  $D(X_{j}) = \mathrm{Bernoulli}(X_{j})$ , we reason as follows:
$$
\begin{array}{l} \operatorname {g f} \left(\llbracket P \rrbracket_ {\operatorname {s t d}} (\mu)\right) (\boldsymbol {x}) = \dots = \int \left(\boldsymbol {x} [ k \mapsto 1 ]\right) ^ {\boldsymbol {X}} \operatorname {g f} \left(D \left(X _ {j}\right)\right) \left(x _ {k}\right) \mathrm {d} \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} (1 + X _ {j} (x _ {k} - 1)) d \mu (\boldsymbol {X}) \\ = \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) + (x _ {k} - 1) \cdot \int X _ {j} (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \\ = \mathbf {g f} (\mu) (\boldsymbol {x} [ k \mapsto 1 ]) + (x _ {k} - 1) \cdot \int x _ {j} \cdot \partial_ {j} (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \\ = \mathbf {g f} (\mu) (\boldsymbol {x} [ k \mapsto 1 ]) + x _ {j} (x _ {k} - 1) \cdot \partial_ {j} \int (\boldsymbol {x} [ k \mapsto 1 ]) ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \\ = \mathbf {g} \mathbf {f} (\mu) \left(\boldsymbol {x} [ k \mapsto 1 ]\right) + x _ {j} \left(x _ {k} - 1\right) \cdot \partial_ {j} \mathbf {g f} (\mu) \left(\boldsymbol {x} [ k \mapsto 1 ]\right) \\ = \llbracket P \rrbracket_ {\mathbf {g f}} (\mathbf {g f} (\mu)) (\boldsymbol {x}) \\ \end{array}
$$
Note that the interchange of integral and differentiation is allowed by Lemma B.3. It also implies that  $\llbracket P\rrbracket_{\mathbf{gf}}(\mathbf{gf}(\mu))(\pmb {x})$  is defined for  $x_{j}\in (0,R)$  by Lemma B.3.
Fail. If  $P$  is fail, then  $\llbracket P\rrbracket_{\mathrm{std}}(\mu)$  is the zero measure and  $\llbracket P\rrbracket_{\mathrm{gf}}(\mathsf{gf}(\mu))$  is the zero function, so the claim holds trivially. This GF is clearly defined everywhere.
Normalization. For normalization, we make use of the linearity of  $\mathrm{gf}(-)$ .
$$
\begin{array}{l} \operatorname {g f} (\text {n o r m a l i z e} (\mu)) = \operatorname {g f} \left(\frac {\mu}{\int \mathrm {d} \mu (\boldsymbol {X})}\right) \\ = \frac {\operatorname {g f} (\mu)}{\int \mathrm {d} \mu (\boldsymbol {X})} \\ = \frac {\operatorname {g f} (\mu)}{\int 1 ^ {X _ {1}} \dots 1 ^ {X _ {n}} \mathrm {d} \mu (\boldsymbol {X})} \\ = \frac {\operatorname {g f} (\mu)}{\operatorname {g f} (\mu) \left(\mathbf {1} _ {n}\right)} \\ = \operatorname {n o r m a l i z e} (\mathrm {g f} (\mu)) \\ \end{array}
$$
Furthermore, if  $\mathbf{gf}(\mu)$  is defined on  $T(R,\mu)$ , then so is  $\mathrm{normalize}(\mathbf{gf}(\mu))$  on  $T(R,\mathrm{normaize}(\mu)) = T(R,\mu)$ .
This finishes the induction and proves the claim.
Lemma B.2. Let  $A \subset \mathbb{N}$  be a finite set. Let  $\mu$  be a measure on  $\mathbb{R}^n$  such that its  $k$ -th component is supported on  $\mathbb{N}$ , i.e.  $\mu(S) = \mu(\{X \in S \mid X_k \in \mathbb{N}\})$  for all measurable  $S \subseteq \mathbb{R}_{\geq 0}$ . Define  $\mu_{X_k \in A}(S) \coloneqq \mu(\{X \in S \mid X_k \in A\})$ . Let  $G \coloneqq \operatorname{gf}(\mu)$ . Then
$$
\operatorname {g f} \left(\mu_ {X _ {k} \in A}\right) (\boldsymbol {x}) = G _ {X _ {k} \in A} (\boldsymbol {x}) := \sum_ {i \in A} \frac {\partial_ {k} ^ {i} G (\boldsymbol {x} [ k \mapsto 0 ])}{i !} \cdot x _ {k} ^ {i}
$$
Furthermore, if  $G$  is defined on  $T(R,\mu)$  then so is  $G_{X_k\in A}$  on  $T(R,\mu_{X_k\in A}) = T(R,\mu)$ .
Proof. Since  $\mu_{X_k\in A} = \sum_{i\in A}\mu_{X_k = i}$  where we write  $\mu_{X_k = i}\coloneqq \mu_{X_k\in \{i\}}$  , we have:
$$
\begin{array}{l} \operatorname {g f} \left(\mu_ {X _ {k} = i}\right) (\boldsymbol {x}) = \int \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} \mu_ {X _ {k} = i} (\boldsymbol {X}) \\ = \int x _ {1} ^ {X _ {1}} \dots x _ {k - 1} ^ {X _ {k - 1}} \cdot x _ {k} ^ {X _ {k}} \cdot x _ {k + 1} ^ {X _ {k + 1}} \dots x _ {n} ^ {X _ {n}} \cdot [ X _ {k} = i ] d \mu (\boldsymbol {X}) \\ = x _ {k} ^ {i} \int x _ {1} ^ {X _ {1}} \dots x _ {k - 1} ^ {X _ {k - 1}} \cdot x _ {k + 1} ^ {X _ {k + 1}} \dots x _ {n} ^ {X _ {n}} \cdot \left(\frac {1}{i !} \left. \frac {\partial^ {i} x _ {k} ^ {X _ {k}}}{\partial x _ {k} ^ {i}} \right| _ {x _ {k} = 0}\right) d \mu (\boldsymbol {X}) \\ \end{array}
$$
The reason for the last step is that for the function  $f(x) = x^l$ , we have
$$
f ^ {(i)} (x) = \left\{ \begin{array}{l l} 0 & \text {i f} l <   i \\ x ^ {l - i} \cdot \prod_ {j = l - i + 1} ^ {l} j & \text {i f} l \geq i \end{array} \right.
$$
Since  $0^{l - i}$  is 0 for  $l > i$  and 1 for  $l = i$ , we find that  $f^{(i)}(0) = [i = l]\cdot i!$ . Above, we used this fact with  $l = X_{k}$  and  $x = x_{k}$ . We continue:
$$
\begin{array}{l} \cdots = x _ {k} ^ {i} \int \frac {1}{i !} \left(x _ {1} ^ {X _ {1}} \dots x _ {k - 1} ^ {X _ {k - 1}} \cdot \frac {\partial^ {i} x _ {k} ^ {X _ {k}}}{\partial x _ {k} ^ {i}} \cdot x _ {k + 1} ^ {X _ {k + 1}} \dots x _ {n} ^ {X _ {n}}\right) \Bigg | _ {x _ {k} = 0} \mathrm {d} \mu (\boldsymbol {X}) \\ = \int \frac {\partial^ {i}}{\partial x _ {k} ^ {i}} \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \Bigg | _ {x _ {k} = 0} \cdot \frac {x _ {k} ^ {i}}{i !} \\ = \left. \frac {\partial^ {i}}{\partial x _ {k} ^ {i}} \int \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) \right| _ {x _ {k} = 0} \cdot \frac {x _ {k} ^ {i}}{i !} \\ = \partial_ {k} ^ {i} G (\boldsymbol {x} [ k \mapsto 0 ]) \cdot \frac {x _ {k} ^ {i}}{i !} \\ \end{array}
$$
as desired. Note that the integral and differentiation operators can be interchanged by Lemma B.3.
Lemma B.3. If the integral  $\int \pmb{x}^{\pmb{X}}\mathrm{d}\mu (\pmb {X})$  for a measure  $\mu$  on  $\mathbb{R}_{\geq 0}^{n}$  is defined for all  $\pmb {x}\in T(R,\mu)$ , we have
$$
\partial_ {i _ {1}} \dots \partial_ {i _ {m}} \int \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X}) = \int \partial_ {i _ {1}} \dots \partial_ {i _ {m}} \boldsymbol {x} ^ {\boldsymbol {X}} \mathrm {d} \mu (\boldsymbol {X})
$$
and both sides are defined for all  $\pmb{x} \in T(R, \mu)$ .
Furthermore, the right-hand side has the form  $\int p(\pmb{X})\pmb{x}^{\pmb{X} - \pmb{w}}\mathrm{d}\mu (\pmb{X})$  for a polynomial  $p$  in  $n$  variables and  $\pmb {w}\in \mathbb{N}^n$ , with the property that  $p(\pmb {X}) = 0$  whenever  $X_{j} < w_{j}$  for some  $j$ , in which case we define  $p(\pmb {X})\pmb{x}^{\pmb{X} - \pmb{w}}\coloneqq 0$ , even if  $\pmb{x}^{\pmb{X} - \pmb{w}}$  is undefined.
Proof. The proof is by induction on  $m$ . If  $m = 0$  then the statement is trivial. For the induction step  $m > 0$ , we may assume that
$$
\partial_ {i _ {2}} \dots \partial_ {i _ {m}} \int {\pmb x} ^ {\pmb X} \mathrm {d} \mu ({\pmb X}) = \int \partial_ {i _ {2}} \dots \partial_ {i _ {m}} {\pmb x} ^ {\pmb X} \mathrm {d} \mu ({\pmb X}) = \int p ({\pmb X}) {\pmb x} ^ {{\pmb X} - {\pmb w}} \mathrm {d} \mu ({\pmb X})
$$
By Lemma B.4, we reason
$$
\begin{array}{l} \partial_ {i _ {1}} \dots \partial_ {i _ {m}} \int \pmb {x} ^ {\pmb {X}} \mathrm {d} \mu (\pmb {X}) = \partial_ {i _ {1}} \int p (\pmb {X}) \pmb {x} ^ {\pmb {X} - \pmb {w}} \mathrm {d} \mu (\pmb {X}) \\ = \int \left(X _ {i _ {1}} - w _ {i _ {1}}\right) p (\boldsymbol {X}) \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w} \left[ i _ {1} \mapsto w _ {i _ {1}} + 1 \right]} d \mu (\boldsymbol {X}) \\ = \int \partial_ {i _ {1}} \dots \partial_ {i _ {m}} \boldsymbol {x} ^ {\boldsymbol {X}} d \mu (\boldsymbol {X}) \\ \end{array}
$$
which establishes the induction goal with the polynomial  $\tilde{p}(\mathbf{X}) \coloneqq (X_{i_1} - w_{i_1})p(\mathbf{X})$  and  $\tilde{w} \coloneqq w[i_1 \mapsto w_{i_1} + 1]$ . The property  $\tilde{p}(\mathbf{X}) = 0$  whenever  $X_j < \tilde{w}_j$  for some  $j$  still holds due to the added factor  $(X_{i_1} - w_{i_1})$  taking care of the case  $X_{i_1} = w_{i_1} = \tilde{w}_{i_1} - 1$ .
Lemma B.4. Let  $\pmb{w} \in \mathbb{N}^n$ ,  $p$  be a polynomial in  $n$  variables, and  $\mu$  a measure on  $\mathbb{R}_{\geq 0}^n$ . We define  $p(\pmb{X})\pmb{x}^{\pmb{X} - \pmb{w}} := 0$  whenever  $p(\pmb{X}) = 0$ , even if  $x_j \leq 0$  and  $X_j < w_j$  for some  $j$ . Suppose  $p(\pmb{X})\pmb{x}^{\pmb{X} - \pmb{w}}$  is defined  $\mu$ -almost everywhere and  $\mu$ -integrable for all  $\pmb{x} \in T(R, \mu)$ . Then
$$
\begin{array}{l} \frac {\partial}{\partial x _ {i}} \int p (\boldsymbol {X}) \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w}} \mathrm {d} \mu (\boldsymbol {X}) = \int \frac {\partial}{\partial x _ {i}} p (\boldsymbol {X}) \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w}} \mathrm {d} \mu (\boldsymbol {X}) \\ = \int \left(X _ {i} - w _ {i}\right) p (\boldsymbol {X}) \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w} [ i \mapsto w _ {i} + 1 ]} \mathrm {d} \mu (\boldsymbol {X}) \\ \end{array}
$$
holds and is defined for all  $\pmb{x} \in T(R, \mu)$ .
Proof. The proof is about verifying the conditions of the Leibniz integral rule. The nontrivial condition is the boundedness of the derivative by an integrable function. The proof splits into two parts, depending on whether the  $i$ -th component of  $\mu$  is supported on  $\mathbb{N}$ . If it is, then  $x_{i} \in (-R,R)$  by the definition of  $T(R,\mu)$  and can be nonpositive, but  $X_{i} - w_{i} < 0$  and  $p(\mathbf{X}) \neq 0$  happens  $\mu$ -almost never. Otherwise, we have  $x \in (0,R)$  and it is thus guaranteed to be positive, but  $X_{i} - w_{i}$  may be negative. Hence the two cases need slightly different treatment.
We first deal with the case that the  $i$ -th component is not supported on  $\mathbb{N}$ . Let  $0 < \epsilon < R' < R'' < R$ . We first prove  $\frac{\partial}{\partial x_i} \int p(\pmb{X}) \pmb{x}^{\pmb{X}} \, \mathrm{d}\mu(\pmb{X}) = \int \frac{\partial}{\partial x_i} p(\pmb{X}) \pmb{x}^{\pmb{X}} \, \mathrm{d}\mu(\pmb{X})$  for any  $x_i \in (\epsilon, R')$ , using the Leibniz integral rule. For this purpose, we need to bound the derivative:
$$
\begin{array}{l} \left| \frac {\partial}{\partial x _ {i}} p (\boldsymbol {X}) \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w}} \right| \leq | (X _ {i} - w _ {i}) p (\boldsymbol {X}) | \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w} [ i \mapsto w _ {i} + 1 ]} \leq \frac {1}{x _ {i}} | X _ {i} - w _ {i} | | p (\boldsymbol {X}) | \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w}} \\ \leq \frac {1}{\epsilon} | X _ {i} - w _ {i} | | p (\boldsymbol {X}) | \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w}} \\ \end{array}
$$
If  $X_{i} > w_{i}$ , we can choose  $M > 1$  sufficiently large such that  $|X_{i} - w_{i}|x_{i}^{X_{i} - w_{i}} \leq (X_{i} - w_{i})R^{\prime}X_{i} - w_{i} \leq R^{\prime \prime}X_{i} - w_{i}$  whenever  $|X_{i} - w_{i}| > M$ . (Such an  $M$  can be found because the exponential function  $y \mapsto \left(\frac{R^{\prime\prime}}{R^{\prime}}\right)^{y}$  grows more quickly than the linear function  $y \mapsto y$ .) If  $X_{i} - w_{i} \leq M$
then we also have  $|X_i - w_i| x_i^{X_i - w_i} \leq M \cdot R'' X_i - w_i$ . So the derivative is thus bounded on the set  $\{X \mid X_i > w_i\}$  by  $\frac{M}{\epsilon} \cdot |p(X)| (x[i \mapsto R''])^{X - w}$ , which is integrable by assumption.
On the complement set  $\{\pmb{X} \mid 0 \leq X_i \leq w_i\}$ , we find  $|X_i - w_i| x_i^{X_i - w_i} \leq |X_i - w_i| \epsilon^{X_i - w_i} \leq w_i \epsilon^{X_i - w_i}$ . Hence the derivative is bounded by  $\frac{w_i}{\epsilon} \cdot |p(\pmb{X})| (\pmb{x}[i \mapsto \epsilon])^{\pmb{X} - \pmb{w}}$ , which is again integrable by assumption. So the derivative is bounded by an integrable function and thus, by the Leibniz integral rule, interchanging differentiation and the integral is valid for all  $x_i \in (\epsilon, R')$  and thus for all  $x_i \in (0, R)$ .
Next, consider the case that the  $i$ -th component of  $\mu$  is supported on  $\mathbb{N}$ . Let  $0 < R' < R'' < R$ . Note that the set  $\{X \mid X_i < w_i \land p(X) \neq 0\}$  has measure zero because otherwise the integrand  $p(X)\pmb{x}^{X - w}$  would not be defined for  $x_i \leq 0$  due to the negative exponent  $X_i - w_i$ . Hence, for  $\mu$ -almost all  $X$ , we have  $p(X) = [X \geq w]p(X)$  and can bound the partial derivative for  $|x_i| < R'$ :
$$
\begin{array}{l} \left| \frac {\partial}{\partial x _ {i}} p (\boldsymbol {X}) \boldsymbol {x} ^ {\boldsymbol {X} - \boldsymbol {w}} \right| \leq | (X _ {i} - w _ {i}) p (\boldsymbol {X}) | | \boldsymbol {x} | ^ {\boldsymbol {X} - \boldsymbol {w} [ i \rightarrow w _ {i} + 1 ]} \\ \leq [ \boldsymbol {X} \geq \boldsymbol {w} ] | X _ {i} - w _ {i} | | p (\boldsymbol {X}) | | \boldsymbol {x} | ^ {\boldsymbol {X} - \boldsymbol {w} [ i \mapsto w _ {i} + 1 ]} \\ \leq [ \boldsymbol {X} \geq \boldsymbol {w} [ i \mapsto w _ {i} + 1 ] ] | X _ {i} - w _ {i} | | p (\boldsymbol {X}) | | \boldsymbol {x} | ^ {\boldsymbol {X} - \boldsymbol {w} [ i \mapsto w _ {i} + 1 ]} \\ \leq [ \boldsymbol {X} \geq \boldsymbol {w} [ i \mapsto w _ {i} + 1 ] ] | X _ {i} - w _ {i} | | p (\boldsymbol {X}) | | \boldsymbol {x} [ i \mapsto R ^ {\prime} ] | ^ {\boldsymbol {X} - \boldsymbol {w} [ i \mapsto w _ {i} + 1 ]} \\ \leq [ \boldsymbol {X} \geq \boldsymbol {w} ] | X _ {i} - w _ {i} | | p (\boldsymbol {X}) | | \boldsymbol {x} [ i \mapsto R ^ {\prime} ] | ^ {\boldsymbol {X} - \boldsymbol {w}} \\ \end{array}
$$
because the factor  $|X_i - w_i|$  vanishes for  $X_i = w_i$  and thus ensures that negative values for the exponent  $\pmb{X} - \pmb{w}[i \mapsto w_i + 1]$  don't matter, so that we can use the monotonicity of  $|\pmb{x}|^{X - \pmb{w}[i \mapsto w_i + 1]}$  in  $|\pmb{x}|$ .
Similarly to the first case, for a sufficiently large  $M > \max(1, w_i)$ , we have  $|X_i - w_i| R'^{X_i - w_i} \leq R''X_i - w_i$  whenever  $X_i - w_i > M$ . If  $X_i - w_i \leq M$  then  $[X \geq \pmb{w}] |X_i - w_i| \leq M$  and we also have  $[X \geq \pmb{w}] |X_i - w_i| R'^{X_i - w_i} \leq M \cdot R''X_i - w_i$ . The derivative is thus bounded by  $M \cdot |p(\pmb{X})| |x[i \mapsto R'']|^{X - w}$ , which is integrable by assumption. By the Leibniz integral rule, interchanging differentiation and the integral is valid for all  $x_i \in (-R', R')$  and thus for all  $x_i \in (-R, R)$ .
Remark B.1. As an example of what can go wrong with derivatives at zero if the measure is not supported on  $\mathbb{N}$ , consider the Dirac measure  $\mu = \operatorname{Dirac}(\frac{1}{2})$ . It has the generating function  $G(x) := \mathrm{gf}(\mu)(x) = \sqrt{x}$  defined for  $x \in \mathbb{R}_{\geq 0}$ . Its derivative is  $G'(x) := \frac{1}{2\sqrt{x}}$ , which is not defined for  $x = 0$ .
B.3 Possible extensions to the probabilistic programming language
The syntax of our language guarantees that the generating function of the distribution of any probabilistic program admits a closed form. But are the syntactic restrictions necessary or are there other useful programming constructs that can be supported? We believe the following constructs preserve closed forms of the generating function and could thus be supported:
- Additional distributions with constant parameters: any such distribution could be supported as long as its GF is defined on  $[0,1 + \epsilon)$  for some  $\epsilon > 0$  and its derivatives can be evaluated.
- Additional compound distributions: we spent a lot of time trying to find additional compound distributions with a closed-form GF since this would be the most useful for probabilistic models, but we were largely unsuccessful. The only such distributions we found are  $\operatorname{Binomial}(m, X_k)$ , which could just be written as a sum of  $m$  iid Bernoulli  $(X_k)$ -variables, and  $\operatorname{Gamma}(X_k, \beta)$  with shape parameter  $X_k$ , which could be translated to a GF with the same idea as for Binomial  $(X_k, p)$ , NegBinomial  $(X_k, p)$ , and Poisson  $(\lambda \cdot X_k)$ .
- Modulo, subtraction, iid sums: The event  $X_{k} \mod 2 = 0$ , and the statements  $X_{k} \coloneqq \max(X_{k} - m, 0)$  (a form of subtraction that ensures nonnegativity), and  $X_{k} \coloneqq \sum_{i} \operatorname{id}(D, X_{l})$  (i.e.  $X_{k}$  is assigned the sum of  $X_{l}$  independent random variables with distribution  $D$ ) can also be supported as shown in [4, 18].
- Nonlinear functions on variables with finite support: if a variable has finite support, arbitrary functions on it could be performed by exhaustively testing all values in its domain.