SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} [ p (C = c | \boldsymbol {w}) ] = p (c) \int_ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} p (\boldsymbol {w} | c) d \boldsymbol {w} (12) \\ = p (c) \int_ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} \sum_ {m = 1} ^ {| \mathcal {M} |} p (\boldsymbol {w} | m, c) p (m | c) d \boldsymbol {w} (13) \\ = p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \int_ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} p (\boldsymbol {w} | m, c) d \boldsymbol {w} (14) \\ = p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \int_ {a _ {1}} ^ {b _ {1}} \dots \int_ {a _ {D}} ^ {b _ {D}} \prod_ {f = 1} ^ {D} p \left(w _ {f} \mid m, c\right) d w _ {D} \dots d w _ {1} (15) \\ = p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \prod_ {f = 1} ^ {D} \int_ {a _ {f}} ^ {b _ {f}} p \left(w _ {f} \mid m, c\right) d w _ {f} (16) \\ = p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right), (17) \\ \end{array}
$$
where  $Z_{cmf}\sim \mathcal{N}(\cdot ;\mu_{cmf},\sigma_{cmf}^2)$
and the last equality follows from the known fact  $p(a \leq X \leq b) = F(b) - F(a)$ .
We introduce the following lemmas that will be useful in Theorem 3.5 and Theorem 3.8, which are about addition, linear transformation, and marginalization involving mixtures of distributions.
Lemma A.1. Let  $X, Y$  be two independent random variables with corresponding mixture distributions  $\phi_{X}(\pmb{w}) = \sum_{i=1}^{I} \alpha_{i} f_{i}(\pmb{w})$  and  $\phi_{Y}(\pmb{w}) = \sum_{j=1}^{J} \beta_{j} d_{j}(\pmb{w})$ . Then  $Z = X + Y$  still follows a mixture distribution.
Proof. By linearity of expectation the moment generating function of  $X$  (and analogously  $Y$ ) has the following form
$$
\begin{array}{l} M _ {X} (\boldsymbol {t}) = \mathbb {E} \left[ e ^ {\boldsymbol {t} ^ {T} \boldsymbol {w}} \right] = \int e ^ {\boldsymbol {t} ^ {T} \boldsymbol {w}} \phi_ {X} (\boldsymbol {w}) d \boldsymbol {w} (18) \\ = \int e ^ {\boldsymbol {t} ^ {T} \boldsymbol {w}} \sum_ {i = 1} ^ {I} \alpha_ {i} f _ {i} (\boldsymbol {w}) d \boldsymbol {w} (19) \\ = \sum_ {i = 1} ^ {I} \alpha_ {i} M _ {X _ {i}} (\boldsymbol {t}) (20) \\ \end{array}
$$
where  $X_{i}$  is the r.v. corresponding to a component of the distribution. Using the fact that the moment generating function of  $Z = X + Y$  is given by  $M_Z(\pmb {t}) = M_X(\pmb {t})M_Y(\pmb {t})$ , we have that
$$
M _ {Z} (\boldsymbol {t}) = \left(\sum_ {i = 1} ^ {I} \alpha_ {i} M _ {X _ {i}} (\boldsymbol {t})\right) \left(\sum_ {j = 1} ^ {J} \beta_ {j} M _ {Y _ {j}} (\boldsymbol {t})\right) = \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {J} \alpha_ {i} \beta_ {j} \underbrace {M _ {X _ {i}} (\boldsymbol {t}) M _ {Y _ {j}} (\boldsymbol {t})} _ {Z _ {i j} = X _ {i} + Y _ {j}}. \tag {21}
$$
Therefore,  $Z$  follows a mixture model with  $IJ$  components where each component follows the distribution associated with the random variable  $Z_{ij} = X_i + Y_j$ .
Lemma A.2. Let  $X$  be a r.v. with multivariate Gaussian mixture distribution  $\phi_X(\boldsymbol{w}) = \sum_{i=1}^{I} \alpha_i \mathcal{N}(\boldsymbol{w}; \boldsymbol{\mu}_i, \boldsymbol{\Sigma}_i), \boldsymbol{w} \in \mathbb{R}^D$ . Then  $Z = \Lambda X, \Lambda \in \mathbb{R}^{D \times D}$  still follows a mixture distribution.
Proof. Using the change of variable  $z = \Lambda w$  we have
$$
\phi_ {Z} (\boldsymbol {w}) = \phi_ {X} \left(\boldsymbol {\Lambda} ^ {- 1} \boldsymbol {w}\right) | \det \left(\boldsymbol {\Lambda} ^ {- 1}\right) | = \sum_ {i = 1} ^ {I} \alpha_ {i} \mathcal {N} \left(\boldsymbol {\Lambda} ^ {- 1} \boldsymbol {w}; \boldsymbol {\mu} _ {i}, \boldsymbol {\Sigma} _ {i}\right) | \det \left(\boldsymbol {\Lambda} ^ {- 1}\right). \tag {22}
$$
By expanding the terms, we see that the distribution of  $Z$  is still a mixture of distributions of the following form
$$
\phi_ {Z} (\boldsymbol {w}) = \sum_ {i = 1} ^ {I} \alpha_ {i} \mathcal {N} (\boldsymbol {w}; \boldsymbol {\Lambda} \boldsymbol {\mu} _ {i}, \boldsymbol {\Lambda} \boldsymbol {\Sigma} _ {i} \boldsymbol {\Lambda} ^ {T}). \tag {23}
$$
Lemma A.3. Let  $X, Y$  be two independent random variables with corresponding Gaussian mixtures of distributions  $\phi_{X}(w) = \sum_{i=1}^{I} \alpha_{i} \mathcal{N}(w; \mu_{X_{i}}, \sigma_{X_{i}}^{2})$  and  $\phi_{Y}(w) = \sum_{j=1}^{J} \beta_{j} \mathcal{N}(w; \mu_{Y_{j}}, \sigma_{Y_{j}}^{2})$ . Then
$$
\mathbb {E} _ {X} \left[ F _ {Y} (w) \right] = \int F _ {Y} (w) \phi_ {X} (w) d w = \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {J} \alpha_ {i} \beta_ {j} \Phi \left(\frac {\mu_ {X _ {i}} - \mu_ {Y _ {j}}}{\sqrt {\sigma_ {Y _ {j}} ^ {2} + \sigma_ {X _ {i}} ^ {2}}}\right) \tag {24}
$$
Proof. It is useful to look at this integral from a probabilistic point of view. In particular, we know that
$$
p (Y \leq X | X = w) = p (Y \leq w) = F _ {Y} (w) \tag {25}
$$
and that, by marginalizing over all possible values of  $X$ ,
$$
p (Y \leq X) = \int p (Y \leq X | X = w) p (X = w) d w = \underbrace {\int F _ {Y} (w) \phi_ {X} (w) d w} _ {\mathbb {E} _ {X} [ F _ {Y} (w) ]}. \tag {26}
$$
Therefore, finding the solution corresponds to computing  $p(Y - X \leq 0)$ . Because  $X, Y$  are independent, the resulting variable  $Z = Y - X$  is distributed as (using Lemma A.1 and Lemma A.2)
$$
p _ {Z} (w) = \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {J} \alpha_ {i} \beta_ {j} \mathcal {N} \left(w; \mu_ {Y _ {j}} - \mu_ {X _ {i}}; \sigma_ {Y _ {j}} ^ {2} + \sigma_ {X _ {i}} ^ {2}\right) \tag {27}
$$
and hence, using the fact that the  $c.d.f.$  of a mixture of Gaussians is the weighted sum of the individual components'  $c.d.f.s$ :
$$
p (Z \leq 0) = \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {J} \alpha_ {i} \beta_ {j} \Phi \left(\frac {\mu_ {X _ {i}} - \mu_ {Y _ {j}}}{\sqrt {\sigma_ {Y _ {j}} ^ {2} + \sigma_ {X _ {i}} ^ {2}}}\right). \tag {28}
$$
Theorem 3.5. Under assumptions 1,2,  $M_{c^{\prime}}(c,\varepsilon)$  has the following analytical form
$$
p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p \left(m ^ {\prime} \mid c ^ {\prime}\right) p \left(m \mid c\right) \prod_ {f = 1} ^ {D} \left(\Phi \left(\frac {\mu_ {c ^ {\prime} m ^ {\prime} f} + \frac {\varepsilon}{2} - \mu_ {c m f}}{\sqrt {\sigma_ {c m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}}\right) - \Phi \left(\frac {\mu_ {c ^ {\prime} m ^ {\prime} f} - \frac {\varepsilon}{2} - \mu_ {c m f}}{\sqrt {\sigma_ {c m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}}\right)\right). \tag{29}
$$
Proof. The proof relies on the previous Lemma A.3. We begin by expanding the formula and using the result of Proposition 3.3. We define the r.v.  $Z_{imf} \sim \mathcal{N}(w; \mu_{imf}, \sigma_{imf}^2)$  to write
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} \sim p (\boldsymbol {x} | c ^ {\prime})} [ M _ {\boldsymbol {x}} (c) ] (30) \\ = \int p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right) p (\boldsymbol {x} | c ^ {\prime}) d \boldsymbol {x} (31) \\ = p (c) \int \left(\sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right)\right) \left(\sum_ {m ^ {\prime} = 1} ^ {| \mathcal {M} |} p (\boldsymbol {x} | m ^ {\prime}, c ^ {\prime}) p \left(m ^ {\prime} \mid c ^ {\prime}\right)\right) d \boldsymbol {x} (32) \\ \end{array}
$$
$$
\begin{array}{l} p (c) \int \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p \left(m ^ {\prime} \mid c ^ {\prime}\right) p \left(m \mid c\right) \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right) p (\boldsymbol {x} \mid m ^ {\prime}, c ^ {\prime}) d \boldsymbol {x} (33) \\ = p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p \left(m ^ {\prime} \mid c ^ {\prime}\right) p \left(m \mid c\right) \int \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right) \prod_ {f = 1} ^ {D} p \left(x _ {f} \mid m ^ {\prime}, c ^ {\prime}\right) d \boldsymbol {x} (34) \\ = p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p \left(m ^ {\prime} \mid c ^ {\prime}\right) p \left(m \mid c\right) \int \dots \int \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right) p \left(x _ {f} \mid m ^ {\prime}, c ^ {\prime}\right) d x (35) \\ = p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p \left(m ^ {\prime} \mid c ^ {\prime}\right) p \left(m \mid c\right) \prod_ {f = 1} ^ {D} \int \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right) p \left(x _ {f} \mid m ^ {\prime}, c ^ {\prime}\right) d x _ {f} (36) \\ = p(c)\sum_{\substack{m = 1\\ m^{\prime} = 1}}^{|\mathcal{M}|}p(m^{\prime}|c^{\prime})p(m|c)\times \\ \times \prod_ {f = 1} ^ {D} \left(\underbrace {\int F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) p \left(x _ {f} \mid m ^ {\prime} , c ^ {\prime}\right) d x _ {f}} _ {\mathbb {E} _ {Z _ {c ^ {\prime} f}} \left[ F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) \right]} - \underbrace {\int F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right) p \left(x _ {f} \mid m ^ {\prime} , c ^ {\prime}\right) d x _ {f}} _ {\mathbb {E} _ {Z _ {c ^ {\prime} f}} \left[ F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right) \right]}\right). (37) \\ \end{array}
$$
We first note that  $F_{Z_{cmf}}\left(x_f + \frac{\varepsilon}{2}\right) = F_Y(x_f)$  where  $Y$  follows distribution  $\mathcal{N}(w; \mu_{cmf} - \frac{\varepsilon}{2}, \sigma_{cmf}^2)$  (and symmetrically for  $F_{Z_{cmf}}\left(x_f - \frac{\varepsilon}{2}\right)$ ), so we can apply Lemma A.3 and obtain
$$
p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p \left(m ^ {\prime} \mid c ^ {\prime}\right) p \left(m \mid c\right) \prod_ {f = 1} ^ {D} \left(\Phi \left(\frac {\mu_ {c ^ {\prime} m ^ {\prime} f} + \frac {\varepsilon}{2} - \mu_ {c m f}}{\sqrt {\sigma_ {c m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}}\right) - \Phi \left(\frac {\mu_ {c ^ {\prime} m ^ {\prime} f} - \frac {\varepsilon}{2} - \mu_ {c m f}}{\sqrt {\sigma_ {c m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}}\right)\right).
$$
Proposition 3.7. Let the first  $D$  derivatives of  $\sum_{i=1}^{|C|} M_{c'}(i)$  be different from 0 in an open interval  $I$  around  $\varepsilon = 0$  and not tending to infinity for  $\varepsilon \to 0$ . Then, under assumptions 1,2, Equation 7 has the following limits (the first of which requires the assumptions)
$$
\lim  _ {\varepsilon \rightarrow 0} p _ {c ^ {\prime}} (c, \varepsilon) = \frac {p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \phi_ {Z _ {c m m ^ {\prime} f}} (0)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {\substack {m = 1\\m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \phi_ {Z _ {i m m ^ {\prime} f}} (0)} \quad \lim  _ {\varepsilon \rightarrow \infty} p _ {c ^ {\prime}} (c, \varepsilon) = p (c) \tag{39}
$$
where  $Z_{imm'f}$  has distribution  $\mathcal{N}(\cdot; -a_{imm'f}, b_{imm'f}^2)$ ,
$$
a _ {i m m ^ {\prime} f} = 2 \left(\mu_ {c ^ {\prime} m ^ {\prime} f} - \mu_ {i m f}\right) \text {a n d} b _ {i m m ^ {\prime} f} = 2 \sqrt {\sigma_ {i m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}.
$$
Proof. The second limit follows from  $\lim_{x\to +\infty}\Phi (x) = 1$  and  $\lim_{x\to -\infty}\Phi (x) = 0$ . As regards the first limit, we first expand the terms
$$
\lim  _ {\varepsilon \rightarrow 0} p _ {c ^ {\prime}} (c, \varepsilon) = \lim  _ {\varepsilon \rightarrow 0} \frac {p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \left(\Phi \left(\frac {\varepsilon + a _ {c m m ^ {\prime} f}}{b _ {c m m ^ {\prime} f}}\right) - \Phi \left(\frac {- \varepsilon + a _ {c m m ^ {\prime} f}}{b _ {c m m ^ {\prime} f}}\right)\right)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | i) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \left(\Phi \left(\frac {\varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right) - \Phi \left(\frac {- \varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right)\right)}, \tag {40}
$$
where  $a_{imm^{\prime}f} = 2(\mu_{c^{\prime}m^{\prime}f} - \mu_{imf})$  and  $b_{imm^{\prime}f} = 2\sqrt{\sigma_{imf}^{2} + \sigma_{c^{\prime}m^{\prime}f}^{2}} \forall i \in \mathcal{C}$  to simplify the notation. By defining  $Z_{imm^{\prime}f} \sim \mathcal{N}(\cdot; -a_{imm^{\prime}f}, b_{imm^{\prime}f}^{2})$ , we can rewrite the limit as
$$
\begin{array}{l} \lim  _ {\varepsilon \rightarrow 0} \frac {p (c) \sum_ {\substack {m = 1\\m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} F _ {Z _ {c m m ^ {\prime} f}} (\varepsilon) - F _ {Z _ {c m m ^ {\prime} f}} (- \varepsilon)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {\substack {m = 1\\m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | i) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} F _ {Z _ {i m m ^ {\prime} f}} (\varepsilon) - F _ {Z _ {i m m ^ {\prime} f}} (- \varepsilon)} (41) \\ = \lim  _ {\varepsilon \rightarrow 0} \frac {p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} d _ {c m m ^ {\prime} f} (\varepsilon)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | i) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} (\varepsilon)}, (42) \\ \end{array}
$$
Both the numerator and denominator tend to zero in the limit, so we try to apply L'Hôpital's rule, potentially multiple times.
For reasons that will become clear soon, we show that the limit of the first derivative of each term in the product is not zero nor infinity:
$$
\lim  _ {\varepsilon \rightarrow 0} d _ {i m m ^ {\prime} f} ^ {(1)} (\varepsilon) = \lim  _ {\varepsilon \rightarrow 0} \left(\phi_ {Z _ {i m m ^ {\prime} f}} (\varepsilon) + \phi_ {Z _ {i m m ^ {\prime} f}} (- \varepsilon)\right) = 2 \phi_ {Z _ {i m m ^ {\prime} f}} (0), \tag {43}
$$
In addition, let us consider the  $n$ -th derivative of the product of functions (by applying the generalized product rule):
$$
\left(\prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} (\varepsilon)\right) ^ {(n)} = \sum_ {j _ {1} + \dots + j _ {D} = n} \binom {n} {j _ {1}, \dots , j _ {D}} \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(j _ {f})} (\varepsilon) \tag {44}
$$
and we note that, as long as  $n < D$  there will exist an assignment to  $j_{f} = 0 \forall f$  in the summation, and thus the limit of each product will always tend to 0 when  $\varepsilon$  goes to 0 since the individual terms cannot tend to infinity for our assumptions. However, when we take the  $D$ -th derivative, there exists one term in the summation that does not go to 0 in the limit, which is
$$
\binom {D} {1, \dots , 1} \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(1)} (\varepsilon) = D! \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(1)} (\varepsilon) \tag {45}
$$
$$
\lim  _ {\varepsilon \rightarrow 0} \binom {D} {1, \dots , 1} \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(1)} (\varepsilon) = 2 ^ {D} D! \prod_ {f = 1} ^ {D} \phi_ {Z _ {i m m ^ {\prime} f}} (0) (\varepsilon) \tag {46}
$$
Therefore, by applying the L'Hôpital's rule  $D$  times, we obtain
$$
\begin{array}{l} \lim  _ {\varepsilon \rightarrow 0} \frac {p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} d _ {c m m ^ {\prime} f} (\varepsilon)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {\substack {m = 1\\m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | i) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} (\varepsilon)} (47) \\ = \frac {p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \phi_ {Z _ {c m m ^ {\prime} f}} (0)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | i) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \phi_ {Z _ {i m m ^ {\prime} f}} (0)}. (48) \\ \end{array}
$$
However, to be valid, L'Hôpital's rule requires that the derivative of the denominator never goes to 0 for points different from  $\varepsilon = 0$ . This is part of our assumptions, hence we conclude the proof.
For  $n = 1$  we can show that this last requirement holds as  $d_{imm'f}^{(1)}(\varepsilon) > 0 \forall \varepsilon$ ,  $d_{imm'f}(\varepsilon) \geq 0$ , and  $d_{imm'f}(\varepsilon) = 0$  if and only if  $\varepsilon = 0$ . In fact,
$$
\left(\prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} (\varepsilon)\right) ^ {(1)} = \sum_ {j _ {1} + \dots + j _ {D} = 1} \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(j _ {f})} (\varepsilon) \tag {49}
$$
is a sum over terms that are all greater than 0 for  $\varepsilon \neq 0$ .
Analytical form of the  $n$ -th derivative of  $\sum_{i=1}^{|C|} M_{c'}(i)$  To verify if the hypotheses of Proposition 3.7 is true given the parameters of all features' distributions, one could compute the  $n$ -th derivative of the denominator w.r.t.  $\varepsilon$  and check that it is not zero around 0. We start again from
$$
\left(\prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} (\varepsilon)\right) ^ {(n)} = \sum_ {j _ {1} + \dots + j _ {D} = n} \binom {n} {j _ {1}, \dots , j _ {D}} \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(j _ {f})} (\varepsilon) \tag {50}
$$
and we proceed using the fact that the  $n$ -th derivative of the standard normal distribution  $S \sim \mathcal{N}(0,1)$  has a well-known form in terms of the  $n$ -th (probabilist) Hermite polynomial  $H e_{n}(x)$ . We note the following equivalence:
$$
\phi_ {Z _ {i m m ^ {\prime} f}} (w) = \frac {1}{b _ {i m m ^ {\prime} f}} \phi_ {S} \left(\frac {w + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right), \tag {51}
$$
hence we can write
$$
\begin{array}{l} d _ {i m m ^ {\prime} f} ^ {(n)} (\varepsilon) = \left(d _ {i m m ^ {\prime} f} ^ {(1)} (\varepsilon)\right) ^ {(n - 1)} = \left(\phi_ {Z _ {i m m ^ {\prime} f}} (\varepsilon) + \phi_ {Z _ {i m m ^ {\prime} f}} (- \varepsilon)\right) ^ {(n - 1)} (52) \\ = \left(\phi_ {Z _ {i m m ^ {\prime} f}} (\varepsilon)\right) ^ {(n - 1)} + \left(\phi_ {Z _ {i m m ^ {\prime} f}} (- \varepsilon)\right) ^ {(n - 1)} (53) \\ = \left(\frac {1}{b _ {i m m ^ {\prime} f}} \phi_ {S} \left(\frac {\varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right)\right) ^ {(n - 1)} (54) \\ + \left(\frac {1}{b _ {i m m ^ {\prime} f}} \phi_ {S} \left(\frac {- \varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right)\right) ^ {(n - 1)} (55) \\ = \frac {1}{b _ {i m m ^ {\prime} f}} \left(\left(\phi_ {S} \left(\frac {\varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right)\right) ^ {(n - 1)} + \left(\phi_ {S} \left(\frac {- \varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right)\right) ^ {(n - 1)}\right) (56) \\ = \frac {1}{b _ {i m m ^ {\prime} f}} \left(\left(\mathcal {N} \left(\frac {\varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}; 0, 1\right)\right) ^ {(n - 1)} + \left(\mathcal {N} \left(\frac {- \varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}; 0, 1\right)\right) ^ {(n - 1)}\right) (57) \\ = \frac {1}{b _ {i m m ^ {\prime} f}} \left((- 1) ^ {n - 1} H e _ {n - 1} \left(\frac {\varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right) \phi_ {S} \left(\frac {\varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right)\right) + \\ + (- 1) ^ {n - 1} H e _ {n - 1} \left(\frac {- \varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right) \phi_ {S} \left(\frac {- \varepsilon + a _ {i m m ^ {\prime} f}}{b _ {i m m ^ {\prime} f}}\right). (58) \\ \end{array}
$$
We use this result in the expansion of Equation 50
$$
\begin{array}{l} \sum_ {j _ {1} + \dots + j _ {D} = n} \binom {n} {j _ {1}, \dots , j _ {D}} \prod_ {f = 1} ^ {D} d _ {i m m ^ {\prime} f} ^ {(j _ {f})} (\varepsilon) (59) \\ = \sum_ {j _ {1} + \dots + j _ {D} = n} \binom {n} {j _ {1}, \dots , j _ {D}} \prod_ {\substack {j _ {f} > 0 \\ f = 1 \dots D}} d _ {i m m ^ {\prime} f} ^ {(j _ {f})} (\varepsilon) \prod_ {\substack {j _ {f} = 0 \\ f = 1 \dots D}} d _ {i m m ^ {\prime} f} (\varepsilon). (60) \\ \end{array}
$$
However, we can readily see that computing the derivative for a single  $\varepsilon$  has combinatorial complexity, which makes the application of the above formulas practical only for small values of  $D$ .
Class Separability Analysis of Section 3.2 We now present a result that will help us compute the SED divergence between two Gaussian mixtures of distributions.
Lemma A.4. Let  $X, Y$  be two independent random variables with corresponding Gaussian mixture distributions  $\phi_X(\pmb{w}) = \sum_{i=1}^{I} \alpha_i f_{X_i}(\pmb{w}) = \sum_{i=1}^{I} \alpha_i \mathcal{N}(\pmb{w}; \pmb{\mu}_i^X, \pmb{\Sigma}_i^X)$  and  $\phi_Y(\pmb{w}) = \sum_{j=1}^{J} \beta_j f_{Y_j}(\pmb{w}) = \sum_{j=1}^{J} \beta_j \mathcal{N}(\pmb{w}; \pmb{\mu}_j^Y, \pmb{\Sigma}_j^Y), \pmb{w} \in \mathbb{R}^D$ . Then the SED divergence between  $\phi_X(\pmb{w})$  and  $\phi_Y(\pmb{w})$  can be computed as
$$
S E D \left(\phi_ {X}, \phi_ {Y}\right) = \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {I} \alpha_ {i} \alpha_ {j} A _ {i, j} + \sum_ {i = 1} ^ {J} \sum_ {j = 1} ^ {J} \beta_ {i} \beta_ {j} B _ {i, j} - 2 \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {J} \alpha_ {i} \beta_ {j} C _ {i, j}, \tag {61}
$$
where (62)
$$
A _ {i, j} = \mathcal {N} \left(\boldsymbol {\mu} _ {i} ^ {X}; \boldsymbol {\mu} _ {j} ^ {X}, \left(\boldsymbol {\Sigma} _ {i} ^ {X} + \boldsymbol {\Sigma} _ {j} ^ {X}\right)\right) \tag {63}
$$
$$
B _ {i, j} = \mathcal {N} \left(\boldsymbol {\mu} _ {i} ^ {Y}; \boldsymbol {\mu} _ {j} ^ {Y}, \left(\boldsymbol {\Sigma} _ {i} ^ {Y} + \boldsymbol {\Sigma} _ {j} ^ {Y}\right)\right) \tag {64}
$$
$$
C _ {i, j} = \mathcal {N} \left(\boldsymbol {\mu} _ {i} ^ {X}; \boldsymbol {\mu} _ {j} ^ {Y}, \left(\boldsymbol {\Sigma} _ {i} ^ {X} + \boldsymbol {\Sigma} _ {j} ^ {Y}\right)\right). \tag {65}
$$
$$
\begin{array}{l} S E D \left(\phi_ {X}, \phi_ {Y}\right) = \int \left(\phi_ {X} (\boldsymbol {w}) - \phi_ {Y} (\boldsymbol {w})\right) ^ {2} d \boldsymbol {w} (66) \\ S E D \left(\phi_ {X}, \phi_ {Y}\right) = \int \left(\sum_ {i = 1} ^ {I} \alpha_ {i} f _ {X _ {i}} (\boldsymbol {w}) - \sum_ {j = 1} ^ {J} \beta_ {j} f _ {Y _ {j}} (\boldsymbol {w})\right) ^ {2} d \boldsymbol {w} (67) \\ = \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {I} \alpha_ {i} \alpha_ {j} \int f _ {X _ {i}} (\boldsymbol {w}) f _ {X _ {j}} (\boldsymbol {w}) d \boldsymbol {w} + \sum_ {i = 1} ^ {J} \sum_ {j = 1} ^ {J} \beta_ {i} \beta_ {j} \int f _ {Y _ {i}} (\boldsymbol {w}) f _ {Y _ {j}} (\boldsymbol {w}) d \boldsymbol {w} (68) \\ - 2 \sum_ {i = 1} ^ {I} \sum_ {j = 1} ^ {J} \alpha_ {i} \beta_ {j} \int f _ {X _ {i}} (\boldsymbol {w}) f _ {Y _ {j}} (\boldsymbol {w}) d \boldsymbol {w}. (69) \\ \end{array}
$$
Finally, we can compute the integral of the product of two Gaussians as follows (Section 8.1.8 of [53])
$$
\int \mathcal {N} (\boldsymbol {w}; \boldsymbol {\mu_ {1}}, \boldsymbol {\Sigma_ {1}}) \mathcal {N} (\boldsymbol {w}; \boldsymbol {\mu_ {2}}, \boldsymbol {\Sigma_ {2}}) d \boldsymbol {w} = \mathcal {N} (\boldsymbol {\mu_ {1}}; \boldsymbol {\mu_ {2}}, (\boldsymbol {\Sigma_ {1}} + \boldsymbol {\Sigma_ {2}})) \tag {70}
$$
to obtain the desired value.