SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem F.14. Let  $(W, f), (V, g) \in \mathcal{WL}_r$ , and suppose that  $\| \Phi \|_{\infty}, \| \xi_r^k \|_{\infty}, \| \xi_t^k \|_{\infty} \leq \rho$ , all message functions  $\xi$  are non-negative valued, and  $L_{\xi_t^k}, L_{\xi_t^k} < L$ , for every  $k = 1, \ldots, K$ . Then,
$$
\left\| \operatorname {A g g} \left(W, \Phi_ {f}\right) - \operatorname {A g g} \left(V, \Phi_ {g}\right) \right\| _ {\square} \leq 4 K L \rho \| f - g \| _ {\square} + K \rho \| W - V \| _ {\square}.
$$
The proof follows the steps of Theorem F.8.
Corollary F.15. Suppose that for every  $y \in \{r, t\}$  and  $k = 1, \ldots, K$
$$
\left| \Phi (0, 0) \right|, \left| \xi_ {\mathrm {y}} ^ {k} (0) \right| \leq B, \quad L _ {\phi}, L _ {\xi_ {\mathrm {y}} ^ {k}} <   L,
$$
and  $\xi, \Phi$  are all non-negatively valued. Then, for every  $(W, f), (V, g) \in \mathcal{WL}_r$ ,
$$
\| \operatorname {A g g} (W, \Phi_ {f}) - \operatorname {A g g} (V, \Phi_ {g}) \| _ {\square} \leq 4 K \left(L ^ {2} r + L B\right) \| f - g \| _ {\square} + K (L r + B) \| W - V \| _ {\square}.
$$
The proof follows the steps of Corollary F.11.
F.3 Lipschitz continuity theorems for MPNNs
The following recurrence sequence will govern the propagation of the Lipschitz constant of the MPNN and the bound of signal along the layers.
Lemma F.16. Let  $\mathbf{a} = (a_{1}, a_{2}, \ldots)$  and  $\mathbf{b} = (b_{1}, b_{2}, \ldots)$ . The solution to  $e_{t+1} = a_{t} e_{t} + b_{t}$ , with initialization  $e_{0}$ , is
$$
e _ {t} = Z _ {t} (\mathbf {a}, \mathbf {b}, e _ {0}) := \prod_ {j = 0} ^ {t - 1} a _ {j} e _ {0} + \sum_ {j = 1} ^ {t - 1} \prod_ {i = 1} ^ {j - 1} a _ {t - i} b _ {t - j}, \tag {26}
$$
where, by convention,
$$
\prod_ {i = 1} ^ {0} a _ {t - i} := 1.
$$
In case there exist  $a, b \in \mathbb{R}$  such that  $a_i = a$  and  $b_i = b$  for every  $i$ ,
$$
e _ {t} = a ^ {t} e _ {0} + \sum_ {j = 0} ^ {t - 1} a ^ {j} b.
$$
Setting 1.
Theorem F.17. Let  $\Theta$  be a MPNN with  $T$  layers. Suppose that for every layer and every y and  $k$ ,
$$
\left\| ^ {t} \xi_ {\mathrm {y}} ^ {k} \right\| _ {\infty}, \left\| \eta^ {t} \right\| _ {\infty} \leq \rho , \quad L _ {\eta^ {t}}, L _ {t \xi_ {\mathrm {y}} ^ {k}} <   L.
$$
Let  $(W,f),(V,g)\in \mathcal{WL}_r$ . Then, for MPNN with no update function
$$
\| \Theta_ {t} (W, f) - \Theta_ {t} (V, g) \| _ {\square} \leq (4 K L \rho) ^ {t} \| f - g \| _ {\square} + \sum_ {j = 0} ^ {t - 1} (4 K L \rho) ^ {j} 4 K \rho^ {2} \| W - V \| _ {\square},
$$
and for MPNN with update function
$$
\| \Theta_ {t} (W, f) - \Theta_ {t} (V, g) \| _ {\square} \leq (4 K L ^ {2} \rho) ^ {t} \| f - g \| _ {\square} + \sum_ {j = 0} ^ {t - 1} (4 K L ^ {2} \rho) ^ {j} 4 K \rho^ {2} L \| W - V \| _ {\square}.
$$
Proof. We prove for MPNNs with update function, where the proof without update function is similar. We can write a recurrence sequence for a bound  $\| \Theta_t(W,f) - \Theta_t(V,g)\|_{\square} \leq e_t$ , by Theorem F.8 and Lemma F.9, as
$$
e _ {t + 1} = 4 K L ^ {2} \rho e _ {t} + 4 K \rho^ {2} L \| W - V \| _ {\square}.
$$
The proof now follows by applying Lemma F.16 with  $a = 4KL^2\rho$  and  $b = 4K\rho^2 L$ .
Setting 2.
Lemma F.18. Let  $\Theta$  be a MPNN with  $T$  layers. Suppose that for every layer  $t$  and every  $\mathbf{y} \in \{\mathbf{r}, \mathbf{t}\}$  and  $k \in [K]$ ,
$$
\left| \eta^ {t} (0) \right|, \left| ^ {t} \xi_ {\mathrm {y}} ^ {k} (0) \right| \leq B, L _ {\eta^ {t}}, L _ {t \xi_ {\mathrm {y}} ^ {k}} <   L
$$
with  $L, B > 1$ . Let  $(W, f) \in \mathcal{WL}_r$ . Then, for MPNN without update function, for every layer  $t$ ,
$$
\left\| \Theta_ {t} (W, f) \right\| _ {\infty} \leq \left(2 K L ^ {2} B ^ {2}\right) ^ {2 ^ {t}} \| f \| _ {\infty} ^ {2 ^ {t}},
$$
and for MPNN with update function, for every layer  $t$ ,
$$
\| \Theta_ {t} (W, f) \| _ {\infty} \leq (2 K L ^ {3} B ^ {2}) ^ {2 ^ {t}} \| f \| _ {\infty} ^ {2 ^ {t}},
$$
Proof. We first prove for MPNNs without update functions. Denote by  $C_t$  a bound on  $\|^t f\|_\infty$ , and let  $C_0$  be a bound on  $\|f\|_\infty$ . By Lemma F.10, we may choose bounds such that
$$
C _ {t + 1} \leq K \left(L C _ {t} + B\right) ^ {2} = K L ^ {2} C _ {t} ^ {2} + 2 K L B C _ {t} + K B ^ {2}.
$$
We can always choose  $C_t, K, L > 1$ , and therefore,
$$
C _ {t + 1} \leq K L ^ {2} C _ {t} ^ {2} + 2 K L B C _ {t} + K B ^ {2} \leq 2 K L ^ {2} B ^ {2} C _ {t} ^ {2}.
$$
Denote  $a = 2KL^2 B^2$ . We have
$$
\begin{array}{l} C _ {t + 1} = a (C _ {t}) ^ {2} = a (a C _ {t - 1} ^ {2}) ^ {2} = a ^ {1 + 2} C _ {t - 1} ^ {4} = a ^ {1 + 2} (a (C _ {t - 2}) ^ {2}) ^ {4} \\ = a ^ {1 + 2 + 4} (C _ {t - 2}) ^ {8} = a ^ {1 + 2 + 4 + 8} (C _ {t - 3}) ^ {1 6} \leq a ^ {2 ^ {t}} C _ {0} ^ {2 ^ {t}}. \\ \end{array}
$$
Now, for MPNNs with update function, we have
$$
\begin{array}{l} C _ {t + 1} \leq L K (L C _ {t} + B) ^ {2} + B \\ = K L ^ {3} C _ {t} ^ {2} + 2 K L ^ {2} B C _ {t} + K B ^ {2} L + B \\ \leq 2 K L ^ {3} B ^ {2} C _ {t} ^ {2}, \\ \end{array}
$$
and we proceed similarly.
Theorem F.19. Let  $\Theta$  be a MPNN with  $T$  layers. Suppose that for every layer  $t$  and every  $\mathbf{y} \in \{\mathbf{r}, \mathbf{t}\}$  and  $k \in [K]$ ,
$$
\left| \eta^ {t} (0) \right|, \left| ^ {t} \xi_ {\mathrm {y}} ^ {k} (0) \right| \leq B, L _ {\eta^ {t}}, L _ {t \xi_ {\mathrm {y}} ^ {k}} <   L,
$$
with  $L, B > 1$ . Let  $(W,g), (V,g) \in \mathcal{WL}_r$ . Then, for MPNNs without update functions
$$
\begin{array}{l} \| \Theta_ {t} (W, f) - \Theta_ {t} (V, g) \| _ {\square} \leq \prod_ {j = 0} ^ {t - 1} 4 K \left(L ^ {2} r _ {j} + L B\right) \| f - g \| _ {\square} \\ + \sum_ {j = 1} ^ {t - 1} \prod_ {i = 1} ^ {j - 1} 4 K \left(L ^ {2} r _ {t - i} + L B\right) 4 K \left(L r _ {t - j} + B\right) ^ {2} \| W - V \| _ {\square}, \\ \end{array}
$$
$$
r _ {i} = \left(2 K L ^ {2} B ^ {2}\right) ^ {2 ^ {i}} \| f \| _ {\infty} ^ {2 ^ {i}},
$$
and for MPNNs with update functions
$$
\begin{array}{l} \| \Theta_ {t} (W, f) - \Theta_ {t} (V, g) \| _ {\square} \leq \prod_ {j = 0} ^ {t - 1} 4 K \left(L ^ {3} r _ {j} + L ^ {2} B\right) \| f - g \| _ {\square} \\ + \sum_ {j = 1} ^ {t - 1} \prod_ {i = 1} ^ {j - 1} 4 K \left(L ^ {3} r _ {t - i} + L ^ {2} B\right) 4 K L \left(L r _ {t - j} + B\right) ^ {2} \| W - V \| _ {\square}, \\ \end{array}
$$
$$
r _ {i} = (2 K L ^ {3} B ^ {2}) ^ {2 ^ {i}} \| f \| _ {\infty} ^ {2 ^ {i}}.
$$
Proof. We prove for MPNNs without update functions. The proof for the other case is similar. By Corollary F.11, since the signals at layer  $t$  are bounded by
$$
r _ {t} = \left(2 K L ^ {2} B ^ {2}\right) ^ {2 ^ {t}} \| f \| _ {\infty} ^ {2 ^ {t}},
$$
$$
\begin{array}{l} \left\| \Theta_ {t + 1} (W, f) - \Theta_ {t + 1} (V, g) \right\| _ {\square} \\ \leq 4 K \left(L ^ {2} r _ {t} + L B\right) \| \Theta_ {t} (W, f) - \Theta_ {t} (V, g) \| _ {\square} + 4 K \left(L r _ {t} + B\right) ^ {2} \| W - V \| _ {\square}. \\ \end{array}
$$
We hence derive a recurrence sequence for a bound  $\| \Theta_t(W, f) - \Theta_t(V, g) \|_{\square} \leq e_t$ , as
$$
e _ {t + 1} = 4 K \left(L ^ {2} r _ {t} + L B\right) e _ {t} + 4 K \left(L r _ {t} + B\right) ^ {2} \| W - V \| _ {\square}.
$$
We now apply Lemma F.16.
Setting 3.
Lemma F.20. Suppose that for every layer  $t$  and every  $\mathrm{y}\in \{\mathrm{r},\mathrm{t}\}$  and  $k = 1,\dots ,K$
$$
\left| \eta^ {t} (0) \right|, \left| \Phi^ {t} (0, 0) \right|, \left| ^ {t} \xi_ {\mathrm {y}} ^ {k} (0) \right| \leq B, L _ {\eta^ {t}}, L _ {\Phi^ {t}}, L _ {t \xi_ {\mathrm {y}} ^ {k}} <   L,
$$
and  $\xi, \Phi$  are all non-negatively valued. Then, for MPNNs without update function
$$
\| \Theta^ {t} (W, f) \| _ {\infty} \leq L ^ {t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {j} B,
$$
and for MPNNs with update function
$$
\| \Theta^ {t} (W, f) \| _ {\infty} \leq L ^ {2 t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {2 j} (L B + B),
$$
Proof. We first prove for MPNNs without update functions. By Lemma F.10, there is a bound  $e_t$  of  $\| \Theta^t(W, f) \|_{\infty}$  that satisfies
$$
e _ {t} = L e _ {t - 1} + B.
$$
Solving this recurrence sequence via Lemma F.16 concludes the proof.
Lastly, for MPNN with update functions, we have a bound that satisfies
$$
e _ {t} = L ^ {2} e _ {t - 1} + L B + B,
$$
and we proceed as before.
Lemma F.21. Suppose that for every  $\mathrm{y} \in \{\mathrm{r}, \mathrm{t}\}$  and  $k = 1, \dots, K$
$$
\left| \eta^ {t} (0) \right|, \left| \Phi (0, 0) \right|, \left| \xi_ {\mathrm {y}} ^ {k} (0) \right| \leq B, L _ {\Phi}, L _ {\xi_ {\mathrm {y}} ^ {k}} <   L,
$$
and  $\xi, \Phi$  are all non-negatively valued. Let  $(W, g), (V, g) \in \mathcal{WL}_r$ . Then, for MPNNs without update functions
$$
\| \Theta^ {t} (W, \Phi_ {f}) - \Theta^ {t} (V, \Phi_ {g}) \| _ {\square} = O (K ^ {t} L ^ {2 t + t ^ {2}} r ^ {t} B ^ {t}) \Big (\| W - V \| _ {\square} + \| f - g \| _ {\square} \Big),
$$
$$
\| \Theta^ {t} (W, \Phi_ {f}) - \Theta^ {t} (V, \Phi_ {g}) \| _ {\square} = O (K ^ {t} L ^ {3 t + 2 t ^ {2}} r ^ {t} B ^ {t}) \Big (\| W - V \| _ {\square} + \| f - g \| _ {\square} \Big)
$$
Proof. We start with MPNNs without update functions. By Corollary F.15 and Lemma F.20, there is a bound  $e_t$  on the error  $\| \Theta^t (W,\Phi_f) - \Theta^t (V,\Phi_g)\| _\square$  at step  $t$  that satisfies
$$
\begin{array}{l} e _ {t} = 4 K \left(L ^ {2} r _ {t - 1} + L B\right) e _ {t - 1} + K (L r + B) \| W - V \| _ {\square} \\ = 4 K \Big (L ^ {2} \big (L ^ {t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {j} B \big) + L B \Big) e _ {t - 1} + K \Big (L \big (L ^ {t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {j} B \big) + B \Big) \| W - V \| _ {\square}. \\ \end{array}
$$
Hence, by Lemma F.16, and  $Z$  defined by (26),
$$
e _ {t} = Z _ {t} (\mathbf {a}, \mathbf {b}, \| f - g \| _ {\square}) = O (K ^ {t} L ^ {2 t + t ^ {2}} r ^ {t} B ^ {t}) \big (\| f - g \| _ {\square} + \| W - V \| _ {\square} \big),
$$
where in the notations of Lemma F.16,
$$
a _ {t} = 4 K \left(L ^ {2} \left(L ^ {t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {j} B\right) + L B\right)
$$
$$
b _ {t} = K \left(L \left(L ^ {t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {j} B\right) + B\right) \| W - V \| _ {\square}.
$$
Next, for MPNNs with update functions, there is a bound that satisfies
$$
\begin{array}{l} e _ {t} = 4 K (L ^ {3} r _ {t - 1} + L ^ {2} B) e _ {t - 1} + K (L ^ {2} r + L B) \| W - V \| _ {\square} \\ = 4 K \left(L ^ {3} \left(L ^ {2 t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {2 j} (L B + B)\right) + L ^ {2} B\right) e _ {t - 1} \\ + K \left(L ^ {2} \left(L ^ {2 t} \| f \| _ {\infty} + \sum_ {j = 1} ^ {t - 1} L ^ {2 j} (L B + B)\right) + L B\right) \| W - V \| _ {\square}. \\ \end{array}
$$
$$
e _ {t} = O \left(K ^ {t} L ^ {3 t + 2 t ^ {2}} r ^ {t} B ^ {t}\right) \left(\| f - g \| _ {\square} + \| W - V \| _ {\square}\right).
$$
G Generalization bound for MPNNs
In this appendix we prove Theorem 4.2.
G.1 Statistical learning and generalization analysis
In the statistical setting of learning, we suppose that the dataset comprises independent random samples from a probability space that describes all possible data  $\mathcal{P}$ . We suppose that for each  $x\in \mathcal{P}$  there is a ground truth value  $y_{x}\in \mathcal{V}$ , e.g., the ground truth class or value of  $x$ , where  $\mathcal{V}$  is, in general, some measure space. The loss is a measurable function  $\mathcal{L}:\mathcal{V}^2\to \mathbb{R}_+$  that defines
similarity in  $\mathcal{V}$ . Given a measurable function  $\Theta : \mathcal{P} \to \mathcal{V}$ , that we call the model or network, its accuracy on all potential inputs is defined as the statistical risk  $R_{\mathrm{stat}}(\Theta) = \mathbb{E}_{x \sim \mathcal{P}}\bigl(\mathcal{L}(\Theta(x), y_x)\bigr)$ . The goal in learning is to find a network  $\Theta$ , from some hypothesis space  $\mathcal{T}$ , that has a low statistical risk. In practice, the statistical risk cannot be computed analytically. Instead, we suppose that a dataset  $\mathcal{X} = \{x_m\}_{m=1}^M \subset \mathcal{P}$  of  $M \in \mathbb{N}$  random independent samples with corresponding values  $\{y_m\}_{m=1}^M \subset \mathcal{V}$  is given. We estimate the statistical risk via a "Monte Carlo approximation," called the empirical risk  $R_{\mathrm{emp}}(\Theta) = \frac{1}{M} \sum_{m=1}^{M} \mathcal{L}(\Theta(x_m), y_m)$ . The network  $\Theta$  is chosen in practice by optimizing the empirical risk. The goal in generalization analysis is to show that if a learned  $\Theta$  attains a low empirical risk, then it is also guaranteed to have a low statistical risk.
One technique for bounding the statistical risk in terms of the empirical risk is to use the bound  $R_{\mathrm{stat}}(\Theta) \leq R_{\mathrm{emp}}(\Theta) + E$ , where  $E$  is the generalization error  $E = \sup_{\Theta \in \mathcal{T}} |R_{\mathrm{stat}}(\Theta) - R_{\mathrm{emp}}(\Theta)|$ , and to find a bound for  $E$ . Since the trained network  $\Theta = \Theta_{\mathcal{X}}$  depends on the data  $\mathcal{X}$ , the network is not a constant when varying the dataset, and hence the empirical risk is not really a Monte Carlo approximation of the statistical risk in the learning setting. If the network  $\Theta$  was fixed, then Monte Carlo theory would have given us a bound of  $E^2$  of order  $O\big(\kappa(p)/M\big)$  in an event of probability  $1 - p$ , where, for example, in Hoeffding's inequality Theorem G.2,  $\kappa(p) = \log(2/p)$ . Let us call such an event a good sampling event. Since the good sampling event depends on  $\Theta$ , computing a naive bound to the generalization error would require intersecting all good sampling events for all  $\Theta \in \mathcal{T}$ . Uniform convergence bounds are approaches for intersecting adequate sampling events that allow bounding the generalization error more efficiently. This intersection of events leads to a term in the generalization bound, called the complexity/capacity, that describes the richness of the hypothesis space  $\mathcal{T}$ . This is the philosophy behind approaches such as VC-dimension, Rademacher dimension, fat-shattering dimension, pseudo-dimension, and uniform covering number (see, e.g., [34]).
G.2 Classification setting
We define a ground truth classifier into  $C$  classes as follows. Let  $\mathcal{C}:\widetilde{\mathcal{W}\mathcal{L}_r}\to \mathbb{R}^C$  be a measurable piecewise constant function of the following form. There is a partition of  $\mathcal{W}\mathcal{L}_r$  into disjoint measurable sets  $B_{1},\ldots ,B_{C}\subset \widetilde{\mathcal{W}\mathcal{L}_{r}}$  such that  $\bigcup_{i = 1}^{C}B_{i} = \widetilde{\mathcal{W}\mathcal{L}_{r}}$ , and for every  $i\in [C]$  and every  $x\in B_i$ ,
$$
\mathcal {C} (x) = e _ {i},
$$
where  $e_i \in \mathbb{R}^C$  is the standard basis element with entries  $(e_i)_j = \delta_{i,j}$ , where  $\delta_{i,j}$  is the Kronecker delta.
We define an arbitrary data distribution as follows. Let  $\mathcal{B}$  be the Borel  $\sigma$ -algebra of  $\widetilde{\mathcal{WL}_r}$ , and  $\nu$  be any probability measure on the measurable space  $(\widetilde{\mathcal{WL}_r},\mathcal{B})$ . We may assume that we complete  $\mathcal{B}$  with respect to  $\nu$ , obtaining the  $\sigma$ -algebra  $\Sigma$ . If we do not complete the measure, we just denote  $\Sigma = \mathcal{B}$ . Defining  $(\widetilde{\mathcal{WL}_r},\Sigma,\nu)$  as a complete measure space or not will not affect our construction.
Let  $S$  be a metric space. Let  $\operatorname{Lip}(S, L)$  be the space of Lipschitz continuous mappings  $\Upsilon: S \to \mathbb{R}^C$  with Lipschitz constant  $L$ . Note that by Theorem 4.1, for every  $i \in [C]$ , the space of MPNN with Lipschitz continuous input and output message functions and Lipschitz update functions, restricted to  $B_i$ , is a subset of  $\operatorname{Lip}(B_i, L_1)$  which is the restriction of  $\operatorname{Lip}(\widetilde{\mathcal{W}\mathcal{L}_r}, L_1)$  to  $B_i \subset \widetilde{\mathcal{W}\mathcal{L}_r}$  for some  $L_1 > 0$ . Moreover,  $B_i$  has finite covering  $\kappa(\epsilon)$  given in (23). Let  $\mathcal{E}$  be a Lipschitz continuous loss function with Lipschitz constant  $L_2$ . Therefore, since  $\mathcal{C}|_{B_i}$  is in  $\operatorname{Lip}(B_i, 0)$ , for any  $\Upsilon \in \operatorname{Lip}(\widetilde{\mathcal{W}\mathcal{L}_r}, L_1)$ , the function  $\mathcal{E}(\Upsilon|_{B_i}, \mathcal{C}|_{B_i})$  is in  $\operatorname{Lip}(B_i, L)$  with  $L = L_1L_2$ .
G.3 Uniform Monte Carlo approximation of Lipschitz continuous functions
The proof of Theorem 4.2 is based on the following Theorem G.3, which studies uniform Monte Carlo approximations of Lipschitz continuous functions over metric spaces with finite covering.
Definition G.1. A metric space  $\mathcal{M}$  is said to have covering number  $\kappa : (0, \infty) \to \mathbb{N}$ , if for every  $\epsilon > 0$ , the space  $\mathcal{M}$  can be covered by  $\kappa(\epsilon)$  ball of radius  $\epsilon$ .
Theorem G.2 (Hoeffding's Inequality). Let  $Y_{1}, \ldots, Y_{N}$  be independent random variables such that  $a \leq Y_{i} \leq b$  almost surely. Then, for every  $k > 0$ ,
$$
\mathbb {P} \Big (\left| \frac {1}{N} \sum_ {i = 1} ^ {N} \left(Y _ {i} - \mathbb {E} [ Y _ {i} ]\right) \right| \geq k \Big) \leq 2 \exp \Big (- \frac {2 k ^ {2} N}{(b - a) ^ {2}} \Big).
$$
The following theorem is an extended version of [26, Lemma B.3], where the difference is that we use a general covering number  $\kappa(\epsilon)$ , where in [26, Lemma B.3] the covering number is exponential in  $\epsilon$ . For completion, we repeat here the proof, with the required modification.
Theorem G.3 (Uniform Monte Carlo approximation for Lipschitz continuous functions). Let  $\mathcal{X}$  be a probability metric space, with probability measure  $\mu$ , and covering number  $\kappa(\epsilon)$ . Let  $X_1, \ldots, X_N$  be drawn i.i.d. from  $\mathcal{X}$ . Then, for every  $p > 0$ , there exists an event  $\mathcal{E}_{\mathrm{Lip}}^p \subset \mathcal{X}^N$  (regarding the choice of  $(X_1, \ldots, X_N)$ ), with probability
$$
\mu^ {N} \left(\mathcal {E} _ {\mathrm {L i p}} ^ {p}\right) \geq 1 - p,
$$
such that for every  $(X_{1},\ldots ,X_{N})\in \mathcal{E}_{\mathrm{Lip}}^{p}$ , for every bounded Lipschitz continuous function  $F:\mathcal{X}\to \mathbb{R}^d$  with Lipschitz constant  $L_{F}$ , we have
$$
\left\| \int F (x) d \mu (x) - \frac {1}{N} \sum_ {i = 1} ^ {N} F \left(X _ {i}\right) \right\| _ {\infty} \leq 2 \xi^ {- 1} (N) L _ {f} + \frac {1}{\sqrt {2}} \xi^ {- 1} (N) \| F \| _ {\infty} (1 + \sqrt {\log (2 / p)}), \tag {27}
$$
where  $\xi (r) = \frac{\kappa(r)^2\log(\kappa(r))}{r^2}$  and  $\xi^{-1}$  is the inverse function of  $\xi$
Proof. Let  $r > 0$ . There exists a covering of  $\mathcal{X}$  by a set of balls  $\{B_j\}_{j \in [J]}$  of radius  $r$ , where  $J = \kappa(r)$ . For  $j = 2, \ldots, J$ , we define  $I_j := B_j \setminus \cup_{i < j} B_i$ , and define  $I_1 = B_1$ . Hence,  $\{I_j\}_{j \in [J]}$  is a family of measurable sets such that  $I_j \cap I_i = \emptyset$  for all  $i \neq j \in [J]$ ,  $\bigcup_{j \in [J]} I_j = \chi$ , and  $\mathrm{diam}(I_j) \leq 2r$  for all  $j \in [J]$ , where by convention  $\mathrm{diam}(\emptyset) = 0$ . For each  $j \in [J]$ , let  $z_j$  be the center of the ball  $B_j$ .
Next, we compute a concentration of error bound on the difference between the measure of  $I_{j}$  and its Monte Carlo approximation, which is uniform in  $j \in [J]$ . Let  $j \in [J]$  and  $q \in (0,1)$ . By Hoeffding's inequality Theorem G.2, there is an event  $\mathcal{E}_j^q$  with probability  $\mu(\mathcal{E}_j^q) \geq 1 - q$ , in which
$$
\left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} _ {I _ {j}} \left(X _ {i}\right) - \mu \left(I _ {k}\right) \right\| _ {\infty} \leq \frac {1}{\sqrt {2}} \frac {\sqrt {\log (2 / q)}}{\sqrt {N}}. \tag {28}
$$
Consider the event
$$
\mathcal {E} _ {\mathrm {L i p}} ^ {J q} = \bigcap_ {j = 1} ^ {J} \mathcal {E} _ {j} ^ {q},
$$
with probability  $\mu^{N}(\mathcal{E}_{\mathrm{Lip}}^{Jq}) \geq 1 - Jq$ . In this event, (28) holds for all  $j \in \mathcal{J}$ . We change the failure probability variable  $p = Jq$ , and denote  $\mathcal{E}_{\mathrm{Lip}}^{p} = \mathcal{E}_{\mathrm{Lip}}^{Jq}$ .
Next we bound uniformly the Monte Carlo approximation error of the integral of bounded Lipschitz continuous functions  $F: \chi \to \mathbb{R}^F$ . Let  $F: \chi \to \mathbb{R}^F$  be a bounded Lipschitz continuous function with Lipschitz constant  $L_F$ . We define the step function
$$
F ^ {r} (y) = \sum_ {j \in [ J ]} F (z _ {j}) \mathbb {1} _ {I _ {j}} (y).
$$
A metric space with a probability Borel measure, where we either take the completion of the measure space with respect to  $\mu$  (adding all subsets of null-sets to the  $\sigma$ -algebra) or not.
$$
\begin{array}{l} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F (X _ {i}) - \int_ {\chi} F (y) d \mu (y) \right\| _ {\infty} \leq \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F (X _ {i}) - \frac {1}{N} \sum_ {i = 1} ^ {N} F ^ {r} (X _ {i}) \right\| _ {\infty} \\ + \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F ^ {r} \left(X _ {i}\right) - \int_ {\chi} F ^ {r} (y) d \mu (y) \right\| _ {\infty} \tag {29} \\ + \left\| \int_ {\chi} F ^ {r} (y) d \mu (y) - \int_ {\chi} F (y) d \mu (y) \right\| _ {\infty} \\ =: (1) + (2) + (3). \\ \end{array}
$$
To bound (1), we define for each  $X_{i}$  the unique index  $j_{i} \in [J]$  s.t.  $X_{i} \in I_{j_{i}}$ . We calculate,
$$
\begin{array}{l} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F (X _ {i}) - \frac {1}{N} \sum_ {i = 1} ^ {N} F ^ {r} (X _ {i}) \right\| _ {\infty} \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| F (X _ {i}) - \sum_ {j \in \mathcal {J}} F (z _ {j}) \mathbb {1} _ {I _ {j}} (X _ {i}) \right\| _ {\infty} \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \| F (X _ {i}) - F (z _ {j _ {i}}) \| _ {\infty} \\ \leq r L _ {F}. \\ \end{array}
$$
We proceed by bounding (2). In the event of  $\mathcal{E}_{\mathrm{Lip}}^p$ , which holds with probability at least  $1 - p$ , equation (28) holds for all  $j \in \mathcal{J}$ . In this event, we get
$$
\begin{array}{l} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F ^ {r} \left(X _ {i}\right) - \int_ {\chi} F ^ {r} (y) d \mu (y) \right\| _ {\infty} = \left\| \sum_ {j \in [ J ]} \left(\frac {1}{N} \sum_ {i = 1} ^ {N} F \left(z _ {j}\right) \mathbb {1} _ {I _ {j}} \left(X _ {i}\right) - \int_ {I _ {j}} F \left(z _ {j}\right) d y\right) \right\| _ {\infty} \\ \leq \sum_ {j \in [ J ]} \| F \| _ {\infty} \left| \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} _ {I _ {j}} \left(X _ {i}\right) - \mu \left(I _ {j}\right) \right| \\ \leq J \| F \| _ {\infty} \frac {1}{\sqrt {2}} \frac {\sqrt {\log (2 J / p)}}{\sqrt {N}}. \\ \end{array}
$$
Recall that  $J = \kappa(r)$ . Then, with probability at least  $1 - p$
$$
\begin{array}{l} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F ^ {r} \left(X _ {i}\right) - \int_ {\chi} F ^ {r} (y) d \mu (y) \right\| _ {\infty} \\ \leq \kappa (r) \| F \| _ {\infty} \frac {1}{\sqrt {2}} \frac {\sqrt {\log (\kappa (r)) + \log (2 / p)}}{\sqrt {N}}. \\ \end{array}
$$
To bound (3), we calculate
$$
\begin{array}{l} \left\| \int_ {\mathcal {X}} F ^ {r} (y) d \mu (y) - \int_ {\mathcal {X}} F (y) d \mu (y) \right\| _ {\infty} = \left\| \int_ {\mathcal {X}} \sum_ {j \in [ J ]} F \left(z _ {j}\right) \mathbb {1} _ {I _ {j}} d \mu (y) - \int_ {\mathcal {X}} F (y) d \mu (y) \right\| _ {\infty} \\ \leq \sum_ {j \in [ J ]} \int_ {I _ {j}} \| F (z _ {j}) - F (y) \| _ {\infty} d \mu (y) \\ \leq r L _ {F}. \\ \end{array}
$$
By plugging the bounds of (1), (2) and (3) into (29), we get
$$
\begin{array}{l} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} F (X _ {i}) - \int_ {\chi} F (y) d \mu (y) \right\| _ {\infty} \leq 2 r L _ {F} + \kappa (r) \| F \| _ {\infty} \frac {1}{\sqrt {2}} \frac {\sqrt {\log (\kappa (r)) + \log (2 / p)}}{\sqrt {N}} \\ \leq 2 r L _ {F} + \frac {1}{\sqrt {2}} \kappa (r) \| F \| _ {\infty} \frac {\sqrt {\log (\kappa (r))} + \sqrt {\log (2 / p)}}{\sqrt {N}} \\ \leq 2 r L _ {F} + \frac {1}{\sqrt {2}} \kappa (r) \| F \| _ {\infty} \frac {\sqrt {\log (\kappa (r))}}{\sqrt {N}} (1 + \sqrt {\log (2 / p)}). \\ \end{array}
$$
Lastly, choosing  $r = \xi^{-1}(N)$  for  $\xi(r) = \frac{\kappa(r)^2 \log(\kappa(r))}{r^2}$ , gives  $\frac{\kappa(r) \sqrt{\log(\kappa(r))}}{\sqrt{N}} = r$ , so