SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also place the following condition on the weights of the model with respect to the mean of  $\mathbb{D}(d)$  and the edge-probability  $r$ .
Definition 4.9. Let  $\mathcal{M}$  be any  $\mathrm{SUMGNN}^+$  for binary graph classification with a non-linearity  $\sigma$  which is eventually constant in both directions. Let  $\mathbb{D}(d)$  be any distribution with mean  $\mu$ , and let  $r \in [0,1]$ . Then, the model  $\mathcal{M}$  is synchronously saturating for  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$  if the following conditions hold:
1. For each  $1 \leq i \leq d(1)$ :
$$
\left[ \left(r \boldsymbol {W} _ {n} ^ {(1)} + \boldsymbol {W} _ {g} ^ {(1)}\right) \boldsymbol {\mu} \right] _ {i} \neq 0
$$
2. For every layer  $1 < t \leq T$ , for each  $1 \leq i \leq d(t)$  and for each  $z \in \{\sigma_{-\infty}, \sigma_{\infty}\}^{d(t-1)}$ :
$$
\left[ \left(r \boldsymbol {W} _ {n} ^ {(t)} + \boldsymbol {W} _ {g} ^ {(t)}\right) \boldsymbol {z} \right] _ {i} \neq 0
$$
Analysis of our proof of the zero-one law for  $\mathrm{SUMGNN}^+$  models (Theorem 4.10 below) reveals that the asymptotic behaviour is determined by the matrices  $Q_{t}\coloneqq rW_{n}^{(t)} + W_{g}^{(t)}$ , where the asymptotic final layer embeddings are  $\sigma (Q_T(\sigma (Q_{T - 1}\dots \sigma (Q_0\pmb {\mu})\dots)))$ . To be synchronously saturating is to avoid the boundary case where one of the intermediate steps in the asymptotic final layer embedding computation has a zero component.
Similarly to the case of a non-splitting classifier, the class of synchronously saturating models is very wide. Indeed, the space of models which are not synchronously saturating is the union of solution space of each equality (i.e. the negation of an inequality in Definition 4.9). Thus, assuming that  $\mu$ ,  $\sigma_{-\infty}$  and  $\sigma_{\infty}$  are non-zero, the space of non-synchronously-saturating models has lower dimension than the space of all models, and thus has measure zero.
With these definitions in place we can now lay out the main result:
Theorem 4.10. Let  $\mathcal{M}$  be a SUMGNN $^+$  model used for binary graph classification and take  $r \in [0,1]$ . Then,  $\mathcal{M}$  satisfies a zero-one law with respect to graph distribution  $\mathbb{G}(n,r)$  and feature distribution  $\mathbb{D}(d)$  assuming the following conditions hold: (i) the distribution  $\mathbb{D}(d)$  is sub-Gaussian, (ii) the non-linearity  $\sigma$  is eventually constant in both directions, (iii) the graph-level representation uses either average or component-wise maximum pooling, and (iv)  $\mathcal{M}$  is synchronously saturating for  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$ .
The proof works differently than the GCN and MEANGNN $^+$  cases, but still rests on a probabilistic analysis of the preactivations in each layer. Assuming that  $\mathcal{M}$  is synchronously saturating for  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$ , we can show that the expected absolute value of each preactivation tends to infinity as the number of nodes increases, and that moreover the probability that it lies below any fixed value tends to 0 exponentially. Hence, the probability that all node embeddings after the first layer are the same and have components which are all  $\sigma_{-\infty}$  or  $\sigma_{\infty}$  tends to 1. We then extend this analysis to further layers, using the fact that  $\mathcal{M}$  is synchronously saturating, which yields inductively that all node embeddings are the same with probability tending to 1, resulting in the following key lemma.
Lemma 4.11. Let  $\mathcal{M}$ ,  $\mathbb{D}(d)$  and  $r$  be as in Theorem 4.10. Let  $\sigma_{-\infty}$  and  $\sigma_{\infty}$  be the extremal values taken by the non-linearity. Then, for every layer  $t$ , there is  $\boldsymbol{z}_t \in \{\sigma_{-\infty}, \sigma_{\infty}\}^{d(t)}$  such that when we sample graphs with node features from  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$  the probability that  $\mathbf{x}_v^{(t)} = \boldsymbol{z}_t$  for every node  $u$  tends to 1 as  $n$  tends to infinity.
The final classification output must therefore be the same asymptotically, since its input consists of node embeddings which always take the same value.
5 Graph neural networks with random node features
Up to this point we have been considering the graph plus node features as the (random) input to GNNs. In this section, we make a change in perspective and regard the initial node features as part of the model, so that its input consists solely of the graph without features. We focus in this section on  $\mathrm{SUMGNN}^+$ . Adding random initial features to GNNs is known to increase their power [32].
Note that Theorem 4.10 immediately yields a zero-one law for these models. This places restrictions on what can be expressed by  $\mathrm{SUMGNN}^+$  models with random features subject to the conditions of Theorem 4.10. For example, it is not possible to express that the number of graph nodes is even, since the property of being even does not satisfy a zero-one law with respect to any  $r\in [0,1]$ .
It is natural to wonder how tight these restrictions are: what precisely is the class of functions which can be approximated by these models? Let us first formalize the notion of approximation.
Definition 5.1. Let  $f$  be a Boolean function on graphs, and let  $\zeta$  be a random function on graphs. Take  $\delta > 0$ . Then  $\zeta$  uniformly  $\delta$ -approximates  $f$  if:
$$
\forall n \in \mathbb {N}: \mathbb {P} (\zeta (G) = f (G) \mid | G | = n) \geq 1 - \delta
$$
when we sample  $G\sim \mathbb{G}(n,1 / 2)$
The reason for sampling graphs from  $\mathbb{G}(n,1 / 2)$  is that under this distribution all graphs on  $n$  nodes are equally likely. Therefore, the requirement is the same as that for every  $n\in \mathbb{N}$  the proportion of  $n$  -node graphs on which  $\zeta (G) = f(G)$  is at least  $1 - \delta$
Building on results due to Abboud et al. [1], we show a partial converse to Theorem 4.10: if a graph invariant satisfies a zero-one law for  $\mathbb{G}(n,1 / 2)$  then it can be universally approximated by a  $\mathrm{SUMGNN}^+$  with random node features.
Theorem 5.2. Let  $\xi$  be any graph invariant which satisfies a zero-one law with respect to  $\mathbb{G}(n,1 / 2)$ . Then, for every  $\delta >0$  there is a  $\mathrm{SUMGNN}^+$  with random node features  $\mathcal{M}$  which uniformly  $\delta$ -approximates  $\xi$ .
The basis of the proof is a result due to Abboud et al. [1] which states that a  $\mathrm{SUMGNN}^+$  with random node features can approximate any graph invariant on graphs of bounded size. When the graph invariant satisfies a zero-one law, we can use the global readout to count the number of nodes. Below a certain threshold, we use the techniques of Abboud et al. [1] to approximate the invariant, and above the threshold we follow its asymptotic behavior. We emphasise that the combination of these techniques yields a model which provides an approximation which is uniform across all graph sizes.
We empirically verify our theoretical findings on a carefully designed synthetic experiment using ER graphs with random features. The goal of these experiments is to answer the following questions for each model under consideration:
Q1. Do we empirically observe a zero-one law?
Q2. What is the rate of convergence like empirically?
Q3. What is the impact of the number of layers on the convergence?
We report experiments for GCN, MEANGNN, and SUMGNN. The following setup is carefully designed to eliminate confounding factors:
- We consider 10 GNN models of the same architecture each with randomly initialized weights, where each weight is sampled independently from  $U(-1,1)$ . The non-linearity is eventually constant in both directions: identity between  $[-1,1]$ , and truncated to  $-1$  if the input is smaller than  $-1$ , and 1 if the input is greater than 1. In the appendix we include additional experiments in which test other choices of non-linearity (see Appendix E.2). We apply mean pooling to yield a final representation  $\mathbf{z}_G \in \mathbb{R}^d$  of the input graph.
- For every model, we apply a final classifier  $\sigma(f): \mathbb{R}^d \to \mathbb{B}$  where  $f$  is a 2-layer MLP with random weights and with tanh activation, which outputs a real value, and  $\sigma$  is the sigmoid function. Graphs are classified as 1 if the output of the sigmoid is greater than 0.5, and 0 otherwise.
- The input graphs are drawn from  $\mathbb{G}(n, 1/2)$  with corresponding node features independently drawn from  $U(0, 1)$ .
- We conduct these experiments with three choices of layers: 10 models with  $T = 1$  layer, 10 models with  $T = 2$  layers, and 10 models with  $T = 3$  layers.
The goal of these experiments is to understand the behavior of the respective GNN graph classifiers with mean-pooling, as we draw larger and larger ER graphs. Specifically, each model classifies graphs of varying sizes, and we are interested in knowing how the proportion of the graphs which are classified as 1 evolves, as we increase the graph sizes.
We independently sample 10 models to ensure this is not a model-specific behavior, aiming to observe the same phenomenon across the models. If there is a zero-one law, then for each model, we should only see two types of curves: either tending to 0 or tending to 1, as graph sizes increase. Whether it will tend to 0 or 1 depends on the final classifier: since each of these are independent MLPs with random weights the specific outcome is essentially random.
We consider models with up to 3 layers to ensure that the node features do not become alike because of the orthogonal over-smoothing issue [24], which surfaces with increasing number of layers. A key feature of our theoretical results is that they do not depend on the number of layers, and this is an aspect which we wish to validate empirically. Using models with random weights is a neutral setup, and random GNNs are widely used in the literature as baseline models [36], as they define valid graph convolutions and tend to perform reasonably well.
6.2 Empirical results
We report all results in Figure 1 for all models considered and discuss them below. Each plot in this figure depicts the curves corresponding to the behavior of independent models with random weights.
GCN. For this experiment, we use an embedding dimensionality of 128 for each GCN model and draw graphs of sizes up 5000, where we take 32 samples of each size. The key insight of Theorem 4.6 is that the final mean-pooled embedding vector  $\mathbf{z}_G$  tends to a constant vector as we draw larger graphs. Applying an MLP followed by a sigmoid function will therefore map  $\mathbf{z}_G$  to either 0 or 1, showing a zero-one law. It is evident from Figure 1 (top row) that all curves tend to either 0 or 1, confirming our expectation regarding the outcome of these experiments for GCNs. Moreover, this holds regardless of the number of layers considered. Since the convergence occurs quickly, already around graphs of size of 1000, we did not experiment with larger graph sizes in this experiment.
MEANGNN. Given that the key insight behind this result is essentially similar to that of Theorem 4.6, we follow the exact same configuration for these models as for GCNs. The proof structure is the same in both cases: we show that the preactivations and activations become closer and closer to
some fixed values as the number of nodes increases. Moreover, comparing the summations in the definitions of GCN and MEANGNN, on a typical ER graph drawn from  $\mathbb{G}(n,1 / 2)$  we would expect each corresponding summand to have a similar value, since  $\sqrt{|\mathcal{N}(u)||\mathcal{N}(v)|}$  should be close to  $|\mathcal{N}^{+}(v)|$ . Figure 1(mid row) illustrates the results for MEANGNN and the trends are reassuringly similar to those of GCNs: all models converge quickly to either 0 and 1 with all choices of layers. Interestingly, the plots for GCN and MEANGNN models are almost identical. We used the same seed when drawing each of the model weights, and the number of parameters is the same between the two. Hence, the GCN models were parameterized with the same values as the MEANGNN models. The fact that each pair of models preforms near identically confirms the expectation that the two architectures work in similar ways on ER graphs.
SUMGNN. Theorem 4.10 shows that, as the number of nodes grow, the embedding vector  $\mathbf{z}_v$  of each node  $v$  will converge to a constant vector with high probability. Hence, when we do mean-pooling at the end, we expect to get the same vector for different graphs of the same size. The mechanism by which a zero-one law is arrived at is quite different compared with the GCN and MEANGNN case. In particular, in order for the embedding vectors to begin to converge, there must be sufficiently many nodes so that the preactivations surpass the thresholds of the non-linearity. For this experiment, we use a smaller embedding dimensionality of 64 for each SUMGNN model and draw graphs of sizes up to 100000, where we take 32 samples of each size. Figure 1 shows the results for SUMGNN. Note that we observe a slower convergence than with GCN or MEANGNN.
7 Limitations, discussion, and outlook
The principal limitations of our work come from the assumptions placed on the main theorems. In our formal analysis, we focus on graphs drawn from the ER distribution. From the perspective of characterizing the expressiveness of GNNs this is unproblematic, and accords with the classical analysis of first-order properties of graphs. However, when considering the extrapolation capacity of GNNs, other choices of distributions may be more realistic. In Appendices E.4 and E.5 we report experiments in which zero-one laws are observed empirically for sparse ER graphs and Barabási-Albert graphs [3], suggesting that formal results may be obtainable. While we empirically observe that GNNs converge to their asymptotic behaviour very quickly, we leave it as future work to rigorously examine the rate at which this convergence occurs.
In this work we show that GNNs with random features can at most capture properties which follow a zero-one law. We complement this with an almost matching lower bound: Theorem 5.2 currently requires a graph invariant  $\xi$  which obeys a zero-one law with respect to a specific value of  $r$  (i.e.,  $1/2$ ), and if this assumption could be relaxed, it would yield a complete characterization of the expressive power of these models.
We would like to thank the anonymous reviewers for their feedback, which lead to several improvements in the presentation of the paper. The first author was supported by an EPSRC studentship with project reference 2271793.
[1] Ralph Abboud, Ismail Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI, pages 2112-2118, 2021.
[2] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In Proceedings of the Ninth International Conference on Learning Representations, ICLR, 2021.
[3] Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. Science, 286(5439):509-512, 1999.
[4] Pablo Barceló, Egor V. Kostylev, Mikaël Monet, Jorge Pérez, Juan L. Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. In Proceedings of the Eighth International Conference on Learning Representations, ICLR, 2020.
[5] Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks, 2018.
[6] Béla Bollobás. *Random Graphs*. Cambridge Studies in Advanced Mathematics. Cambridge University Press, second edition, 2001.
[7] David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gomez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In Proceedings of the Twenty-Eighth Annual Conference on Advances in Neural Information Processing Systems, NIPS, pages 2224-2232, 2015.
[8] Ronald Fagin. Probabilities on finite models. The Journal of Symbolic Logic, JSL, 41(1):50-58, 1976. ISSN 00224812.
[9] Peter A. Flach. First-order logic. In Claude Sammut and Geoffrey I. Webb, editors, Encyclopedia of Machine Learning, pages 410-415. Springer US, Boston, MA, 2010. ISBN 978-0-387-30164-8. doi: 10.1007/978-0-387-30164-8_311.
A Proof of the zero-one law for GCN
The proof of Lemma 4.7 and subsequently Theorem 4.6 relies on an asymptotic analysis of the distributions of the node embeddings at each layer. The following famous concentration inequality for sub-Gaussian random variables allows us to put bounds on the deviation of a sum of random variables from its expected value.
Theorem A.1 (Hoeffding Inequality for sub-Gaussian random variables). There is a universal constant  $c$  such that the following holds. Let  $z_1, \ldots, z_N$  be independent sub-Gaussian scalar random variables with mean 0. Assume that the constants  $C$  from Definition 4.3 for each  $z_i$  can be bounded by  $K$ . Then for all  $t > 0$ :
$$
\mathbb {P} \left(\left| \sum_ {i = 1} ^ {N} \mathbf {z} _ {i} \right| \geq t\right) \leq \exp \left(- \frac {c t ^ {2}}{K ^ {2} N}\right)
$$
Proof. See Theorem 2.6.2 in [38].
We also make use of the following three basic facts about sub-Gaussian random variables.
Lemma A.2. If  $\mathbf{z}$  is a sub-Gaussian random vector and  $\mathbf{q}$  is any vector of the same dimension then  $\mathbf{q} \cdot \mathbf{z}$  is sub-Gaussian.
Proof. This follows directly from Definition 4.3.  $\square$
Lemma A.3. If  $z$  is a sub-Gaussian scalar random variable then so is  $z - \mathbb{E}[z]$ .
Proof. See Lemma 2.6.8 in [38].
Lemma A.4. If  $z$  is a sub-Gaussian scalar random variable and  $a$  is an independent Bernoulli random variable then  $za$  is sub-Gaussian.
Proof. Let  $C$  be the constant given by Definition 4.3 for  $z$ . Let a take values  $\alpha$  and  $\beta$ . Using the Law of Total Probability:
$$
\begin{array}{l} \mathbb {P} (| \mathbf {z a} | \geq t) = \mathbb {P} (| \mathbf {z a} | \geq t \mid \mathbf {a} = \alpha) \mathbb {P} (\mathbf {a} = \alpha) + \mathbb {P} (| \mathbf {z a} | \geq t \mid \mathbf {a} = \beta) \mathbb {P} (\mathbf {a} = \beta) \\ = \mathbb {P} (| \mathbf {z} | \geq t / | \alpha |) \mathbb {P} (\mathbf {a} = \alpha) + \mathbb {P} (| \mathbf {z} | \geq t / | \beta |) \mathbb {P} (\mathbf {a} = \beta) \\ \leq 2 \exp \left(- \frac {t ^ {2}}{| \alpha | ^ {2} C ^ {2}}\right) \mathbb {P} (\mathbf {a} = \alpha) + 2 \exp \left(- \frac {t ^ {2}}{| \beta | ^ {2} C ^ {2}}\right) \mathbb {P} (\mathbf {a} = \beta) \\ \leq 2 \exp \left(- \frac {t ^ {2}}{\operatorname* {m a x} \{| \alpha | , | \beta | \} ^ {2} C ^ {2}}\right) \\ \end{array}
$$
Therefore  $\text{za}$  is sub-Gaussian.
We first prove the key lemma regarding the node embeddings.
Proof of Lemma 4.7. Let  $C$  be the Lipschitz constant for  $\sigma$ . Start by considering the first layer preactivations  $\mathbf{y}_v^{(1)}$  and drop superscript (1)'s for notational clarity. We have that:
$$
\mathbf {y} _ {v} = \sum_ {v \in \mathcal {N} ^ {+} (v)} \frac {1}{\sqrt {| \mathcal {N} (v) | | \mathcal {N} (u) |}} \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} + \boldsymbol {b}
$$
Fix  $i \in \{1, \dots, d(1)\}$ . The deviation from the expected value in the  $i$ th component is as follows:
$$
| [ \mathbf {y} _ {v} - \mathbb {E} [ \mathbf {y} _ {v} ] ] _ {i} | = \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \frac {1}{\sqrt {| \mathcal {N} (v) | | \mathcal {N} (u) |}} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right|
$$
Now every  $|\mathcal{N}(u)|$  is a sum of  $n$  independent 0-1 Bernoulli random variables with success probability  $r$  (since the graph is sampled from an Erdős-Rényi distribution). Since Bernoulli random variables are sub-Gaussian (Lemma A.4) we can use Hoeffding's Inequality to bound the deviation of  $|\mathcal{N}^{+}(u)| =$
$|\mathcal{N}(u)| + 1$  from its expected value  $nr + 1$ . By Theorem A.1 there is a constant  $K$  such that for every  $\gamma \in (0,1)$  and node  $u$ :
$$
\begin{array}{l} \mathbb {P} \left(| \mathcal {N} ^ {+} (u) | \leq \gamma n r\right) \leq \mathbb {P} \left(\left| \left| \mathcal {N} ^ {+} (u) \right| - n r \right| \geq (1 - \gamma) n r\right) \\ \leq \mathbb {P} (| | \mathcal {N} (u) | - n r | \geq (1 - \gamma) n r - 1) \\ \leq 2 \exp \left(- \frac {K ((1 - \gamma) n r - 1) ^ {2}}{n}\right) \\ \end{array}
$$
This means that, taking a union bound:
$$
\mathbb {P} (\forall u \in V \colon | \mathcal {N} ^ {+} (u) | \geq \gamma n r) \geq 1 - 2 n \exp \left(- \frac {K _ {0} ((1 - \gamma) n r - 1) ^ {2}}{n}\right)
$$
Fix  $i\in \{1,\ldots ,d(1)\}$  . In the case where  $\forall u\in V\colon |\mathcal{N}^{+}(u)|\geq \gamma nr$  we have that:
$$
\left. \right.\left|\left[ \mathbf {y} _ {v} - \mathbb {E} \left[ \mathbf {y} _ {v} \right]\right] _ {i} \right| \leq \frac {1}{\sqrt {\left| \mathcal {N} (v) \right| \gamma n r}} \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right]\right] _ {i} \right|
$$
Now, by Lemma A.2 and Lemma A.3 each  $\mathbf{W}_n\mathbf{x}_u^{(0)} - \mathbb{E}\left[\mathbf{W}_n\mathbf{x}_u^{(0)}\right]$  is sub-Gaussian. We can thus apply Hoeffding's Inequality (Theorem A.1) to obtain a constant  $K$  such that for every  $t > 0$  we have:
$$
\begin{array}{l} \mathbb {P} \left(\left| \left[ \mathbf {y} _ {v} - \mathbb {E} [ \mathbf {y} _ {v} ] \right] _ {i} \right| \geq t\right) \leq \mathbb {P} \left(\left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \geq t \sqrt {\left| \mathcal {N} (v) \right| \gamma n r}\right) \\ \leq 2 \exp \left(- \frac {K t ^ {2} | \mathcal {N} (v) | \gamma n r}{| \mathcal {N} ^ {+} (v) |}\right) \\ \leq 2 \exp \left(- \frac {K t ^ {2} \gamma n r}{2}\right) \\ \end{array}
$$
Now using the Law of Total Probability, partitioning depending on whether  $\forall u\in V\colon |\mathcal{N}^{+}(u)|\geq \gamma nr$  we get a bound as follows:
$$
\mathbb {P} (| [ \mathbf {y} _ {v} - \mathbb {E} [ \mathbf {y} _ {v} ] ] _ {i} | \geq t) \leq 2 \exp \left(- \frac {K t ^ {2} \gamma n r}{2}\right) + 2 n \exp \left(- \frac {K _ {0} ((1 - \gamma) n r - 1) ^ {2}}{n}\right)
$$
From now on fix any  $\gamma \in (0,1)$ .
Let  $\mathbf{z}_1 \coloneqq \sigma(\mathbb{E}[\mathbf{y}_v])$  for any  $v$  (this is the same for every  $v$ ). Applying the bound with  $t = C\epsilon$  we can bound the deviation of  $\mathbf{x}_v$  from  $\mathbf{z}_1$  as follows, using the Lipschitz constant  $C$ .
$$
\begin{array}{l} \mathbb {P} \left(\left| \left[ \mathbf {x} _ {v} - \boldsymbol {z} _ {1} \right] _ {i} \right| \geq \epsilon\right) = \mathbb {P} \left(\left| \left[ \sigma \left(\mathbf {y} _ {v}\right) - \sigma \left(\mathbb {E} \left[ \mathbf {y} _ {v} \right]\right) \right] _ {i} \right| \geq \epsilon\right) \\ \leq \mathbb {P} \left(\left| \left[ \mathbf {y} _ {v} - \mathbb {E} [ \mathbf {y} _ {v} ] \right] _ {i} \right| \geq C \epsilon\right) \\ \leq 2 \exp \left(- \frac {K C ^ {2} \epsilon^ {2} \gamma n r}{2}\right) + 2 n \exp \left(- \frac {K _ {0} ((1 - \gamma) n r - 1) ^ {2}}{n}\right) \\ \end{array}
$$
Taking a union bound, the probability that  $|[{\bf x}_v - z_1]_i| < \epsilon$  for every node  $v$  and every  $i\in \{1,\ldots ,d(1)\}$  is at least:
$$
1 - n d (i) \mathbb {P} \left(\left| \left[ \mathbf {x} _ {v} - \boldsymbol {z} _ {1} \right] _ {i} \right| \geq \epsilon\right)
$$