SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This tends to 1 as  $n$  tends to infinity, which yields the result for the first layer.
Now consider the preactivations for the second layer:
$$
\mathbf {y} _ {v} ^ {(2)} = \sum_ {u \in \mathcal {N} ^ {+} (v)} \frac {1}{\sqrt {| \mathcal {N} (v) | | \mathcal {N} (u) |}} W _ {n} ^ {(2)} \mathbf {x} _ {u} ^ {(1)} + \boldsymbol {b} ^ {(2)}
$$
As in the single layer case, we can bound the probability that any  $|\mathcal{N}(u)|$  is less than some  $\gamma nr$ . Condition on the event that  $\forall u \in V \colon |\mathcal{N}^{+}(u)| \geq \gamma nr$ .
By applying the result for the first layer to  $\epsilon' = \epsilon \sqrt{\gamma r} / (2C \| \mathbf{W}_n^{(2)} \|_{\infty})$ , we have that for each  $i \in \{1, \dots, d(2)\}$ :
$$
\mathbb {P} \left(\forall v: \left|\left[ \mathbf {x} _ {v} ^ {(1)} - \mathbf {z} _ {1} \right] _ {i} \right| <   \epsilon^ {\prime}\right)\rightarrow 1 \quad \text {a s} n \rightarrow \infty
$$
Condition additionally on the event that  $|[{\bf x}_v^{(1)} - z_1]_i| < \epsilon '$  for every node  $v$  and every  $i\in$ $\{1,\ldots ,d(1)\}$
$$
\boldsymbol {a} _ {2} := \sum_ {v \in \mathcal {N} ^ {+} (v)} \frac {1}{\sqrt {| \mathcal {N} (v) | | \mathcal {N} (u) |}} \boldsymbol {W} _ {n} ^ {(2)} \boldsymbol {z} _ {1} + \boldsymbol {b} ^ {(2)}
$$
Then we have that for every  $i\in \{1,\ldots ,d(1)\}$
$$
\begin{array}{l} | [ \mathbf {y} _ {v} ^ {(2)} - \boldsymbol {a} _ {2} ] _ {i} | \leq \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \frac {1}{\sqrt {| \mathcal {N} (u) | | \mathcal {N} (v) |}} \left[ \boldsymbol {W} _ {n} ^ {(2)} \left(\boldsymbol {x} _ {u} ^ {(1)} - \boldsymbol {z} _ {1}\right) \right] _ {i} \right| \\ \leq \frac {1}{\sqrt {| \mathcal {N} (v) | \gamma n r}} \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \pmb {W} _ {n} ^ {(2)} (\pmb {x} _ {u} ^ {(1)} - \pmb {z} _ {1}) \right] _ {i} \right| \\ \leq \frac {1}{\sqrt {| \mathcal {N} (v) | \gamma n r}} \left\| \boldsymbol {W} _ {n} ^ {(2)} \right\| _ {\infty} \sum_ {u \in \mathcal {N} ^ {+} (v)} \left\| \mathbf {x} _ {u} ^ {(1)} - \boldsymbol {z} _ {1} \right\| _ {\infty} \\ \leq \frac {\epsilon | \mathcal {N} ^ {+} (v) |}{2 C \sqrt {| \mathcal {N} (v) | n}} \\ \leq \frac {\epsilon (n + 1)}{2 C n} \\ \leq \frac {\epsilon}{C} \\ \end{array}
$$
Now let  $\pmb{z}_2 \coloneqq \sigma(\pmb{a}_2)$ . As in the single-layer case we can use the bound on  $|[\mathbf{y}_v^{(2)} - \pmb{a}_2]_i|$  and the fact that  $\sigma$  is Lipschitz to find that, for every node  $v$  and  $i \in \{1, \dots, d(2)\}$ :
$$
\left| \left[ \mathbf {x} _ {v} ^ {(2)} - \boldsymbol {z} _ {2} \right] _ {i} \right| <   \epsilon
$$
Since the probability that the two events on which we conditioned tends to 1, the result follows for the second layer.
Finally, we apply the argument inductively through the layers to obtain the ultimate result.
With the key lemma established we can prove the main result.
Proof of Theorem 4.6. By Lemma 4.7 the final node embeddings  $\mathbf{x}_v^{(T)}$  deviate less and less from  $z_{T}$  as the number of nodes  $n$  increases. Therefore, the average-pooled graph-level representation also deviates less and less from  $z_{T}$ . By inspecting the proof, we can see that this  $z_{T}$  is exactly the vector  $\pmb{\mu}_{T}$  in the definition of non-splitting (Definition 4.5). This means that  $z_{T}$  cannot lie on a decision boundary for the classifier  $\mathfrak{C}$ . Hence, there is  $\epsilon >0$  such that  $\mathfrak{C}$  is constant on:
$$
\{\boldsymbol {x} \in \mathbb {R} ^ {d (T)} \mid \forall i \in \{1, \dots , d (T) \}: [ \boldsymbol {z} _ {T} - \boldsymbol {x} ] _ {i} <   \epsilon \}
$$
Therefore, the probability that the output of  $\mathcal{M}$  is  $\mathfrak{C}(z_T)$  tends to 1 as  $n$  tends to infinity.
B Proof of the zero-one law for MEANGN $^{+}$
Let us turn now to establishing a zero-one law for GNNs using mean aggregation. We place the same conditions as with Theorem 4.6. This time the notion of 'non-splitting' is as follows.
Definition B.1. Consider a distribution  $\mathbb{D}(d)$  with mean  $\pmb{\mu}$ . Let  $\mathcal{M}$  be a MEANGNN $^+$  used for binary graph classification. Define the sequence  $\pmb{\mu}_0, \dots, \pmb{\mu}_T$  of vectors inductively by  $\pmb{\mu}_0 \coloneqq \pmb{\mu}$  and  $\pmb{\mu}_t \coloneqq \sigma((\pmb{W}_n^{(t)} + \pmb{W}_r^{(t)})\pmb{\mu}_{t-1} + \pmb{b}^{(t)})$ . The classifier  $\mathfrak{C}: \mathbb{R}^{d(T)} \to \mathbb{B}$  is non-splitting for  $\mathcal{M}$  if the vector  $\pmb{\mu}_T$  does not lie on a decision boundary for  $\mathfrak{C}$ .
Again, in practice essentially all classifiers are non-splitting.
Theorem B.2. Let  $\mathcal{M}$  be a MEANGNN $^+$  used for binary graph classification and take  $r \in [0,1]$ . Then,  $\mathcal{M}$  satisfies a zero-one law with respect to graph distribution  $\mathbb{G}(n,r)$  and feature distribution  $\mathbb{D}(d)$  assuming the following conditions hold: (i) the distribution  $\mathbb{D}(d)$  is sub-Gaussian, (ii) the non-linearity  $\sigma$  is Lipschitz continuous, (iii) the graph-level representation uses average pooling, (iv) the classifier  $\mathfrak{C}$  is non-splitting.
Note that the result immediately applies to the MEANGNN architecture, since it is a special case of MEANGNN+.
The overall structure of the proof is the same as for GCN. In particular, we prove the following key lemma stating that all node embeddings tend to fixed values.
Lemma B.3. Let  $\mathcal{M}$  and  $\mathbb{D}(d)$  satisfy the conditions in Theorem B.2. Then, for every layer  $t$ , there is  $\pmb{z}_t \in \mathbb{R}^{d(t)}$  such when sampling a graph with node features from  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$ , for every  $i \in \{1,\dots,d(t)\}$  and for every  $\epsilon > 0$  we have that:
$$
\mathbb {P} \left(\forall v \colon \left| \left[ \mathbf {x} _ {v} ^ {(t)} - \mathbf {z} _ {t} \right] _ {i} \right| <   \epsilon\right) \to 1 a s n \to \infty
$$
We are ready to present the proofs of the statements. The proof of the key lemma works in a similar way to the GCN case.
Proof of Lemma B.3. Let  $C$  be the Lipschitz constant for  $\sigma$ . Start by considering the first layer preactivations  $\mathbf{y}_v^{(1)}$  and drop superscript (1)'s for notational for clarity. We have that:
$$
\mathbf {y} _ {v} = \frac {1}{| \mathcal {N} ^ {+} (v) |} W _ {n} \sum_ {v \in \mathcal {N} ^ {+} (u)} \mathbf {x} _ {v} ^ {(0)} + \frac {1}{n} W _ {r} \sum_ {u \in V} \mathbf {x} _ {u} ^ {(0)} + \boldsymbol {b}
$$
Fix  $i \in \{1, \dots, d(1)\}$ . We can bound the deviation from the expected value as follows:
$$
| [ \mathbf {y} _ {v} - \mathbb {E} [ \mathbf {y} _ {v} ] ] _ {i} | \leq \frac {1}{| \mathcal {N} ^ {+} (v) |} \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| + \frac {1}{n} \left| \sum_ {u \in V} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right|
$$
By Lemma A.2 and Lemma A.3 both each  $\left[\boldsymbol{W}_n\mathbf{x}_u^{(0)} - \mathbb{E}\left[\boldsymbol{W}_n\mathbf{x}_u^{(0)}\right]\right]_i$  and each  $\left[\boldsymbol{W}_n\mathbf{x}_u^{(0)} - \mathbb{E}\left[\boldsymbol{W}_n\mathbf{x}_u^{(0)}\right]\right]_i$  are sub-Gaussian. We can therefore apply Hoeffding's Inequality to their sums. First, by Theorem A.1 there is a constant  $K_{\mathrm{g}}$  such that for any  $t > 0$ :
$$
\begin{array}{l} \mathbb {P} \left(\frac {1}{n} \left| \sum_ {u \in V} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \leq t\right) = \mathbb {P} \left(\left| \sum_ {u \in V} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \leq t n\right) \\ \leq 2 \exp \left(- \frac {K _ {\mathrm {g}} t ^ {2} n ^ {2}}{n}\right) \\ = 2 \exp \left(- K _ {\mathrm {g}} t ^ {2} n\right) \\ \end{array}
$$
Second, applying Theorem A.1 again there is a constant  $K_{\mathrm{n}}$  such that for any  $t > 0$ :
$$
\begin{array}{l} \mathbb {P} \left(\frac {1}{| \mathcal {N} ^ {+} (v) |} \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \leq t\right) \\ = \mathbb {P} \left(\left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \leq t | \mathcal {N} ^ {+} (v) |\right) \\ \leq 2 \exp \left(- \frac {K _ {\mathrm {n}} t ^ {2} | \mathcal {N} ^ {+} (v) | ^ {2}}{n}\right) \\ \end{array}
$$
Now  $|\mathcal{N}(v)|$  is a sum of  $n$  independent 0-1 Bernoulli random variables with success probability  $r$ . Hence, as in the proof of Lemma 4.7, by Hoeffding's Inequality (Theorem A.1) there is a constant  $K_{0}$  such that for every  $\gamma \in (0,1)$ :
$$
\mathbb {P} \left(\left| \mathcal {N} ^ {+} (v) \right| \geq \gamma n r\right) \leq 2 \exp \left(- \frac {K _ {0} \left((1 - \gamma) n r - 1\right) ^ {2}}{n}\right)
$$
We can then use the Law of Total Probability, partitioning on whether  $|\mathcal{N}^{+}(v)| \geq \gamma nr$ , to get a bound as follows:
$$
\begin{array}{l} \mathbb {P} \left(\frac {1}{| \mathcal {N} ^ {+} (v) |} \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left(\boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right]\right) \right| \leq t\right) \\ \leq 2 \exp \left(- \frac {K _ {\mathrm {n}} t ^ {2} (\gamma n r) ^ {2}}{n}\right) + 2 \exp \left(- \frac {K _ {0} ((1 - \gamma) n r - 1) ^ {2}}{n}\right) \\ \end{array}
$$
Finally let  $z_{1} \coloneqq \sigma(\mathbb{E}[\mathbf{y}_{v}])$  for any  $v$  (this is the same for every  $v$ ). Applying the two bounds with  $t = C\epsilon / 2$  we can bound the deviation of  $\mathbf{x}_{v}$  from  $z_{1}$  as follows, using the Lipschitz constant  $C$ .
$$
\begin{array}{l} \mathbb {P} \left(\left| \left[ \mathbf {x} _ {v} - \boldsymbol {z} _ {1} \right] _ {i} \right| \geq \epsilon\right) = \mathbb {P} \left(\left| \left[ \sigma (\mathbf {y} _ {v}) - \sigma \left(\mathbb {E} [ \mathbf {y} _ {v} ]\right) \right] _ {i} \right| \geq \epsilon\right) \\ \leq \mathbb {P} \left(\left| \left[ \mathbf {y} _ {v} - \mathbb {E} [ \mathbf {y} _ {v} ] \right] _ {i} \right| \geq C \epsilon\right) \\ \leq \left( \begin{array}{l} \mathbb {P} \left(\frac {1}{| \mathcal {N} ^ {+} (v) |} \left| \sum_ {u \in \mathcal {N} ^ {+} (v)} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \leq \frac {C \epsilon}{2}\right) \\ + \mathbb {P} \left(\frac {1}{n} \left| \sum_ {u \in V} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} - \mathbb {E} \left[ \boldsymbol {W} _ {n} \mathbf {x} _ {u} ^ {(0)} \right] \right] _ {i} \right| \leq \frac {C \epsilon}{2}\right) \end{array} \right) \\ \leq \left( \begin{array}{l} 2 \exp \left(- \frac {K _ {\mathrm {n}} (C \epsilon \gamma r) ^ {2} n}{4}\right) \\ + 2 \exp \left(- \frac {K _ {0} ((1 - \gamma) n r - 1) ^ {2}}{n}\right) \\ + 2 \exp \left(- K _ {\mathrm {g}} \frac {(C \epsilon) ^ {2} n}{4}\right) \end{array} \right) \\ \end{array}
$$
Taking a union bound, the probability that  $|[{\bf x}_v - {\bf z}_1]_i| < \epsilon$  for every node  $v$  and every  $i\in \{1,\ldots ,d(1)\}$  is at least:
This tends to 1 as  $n$  tends to infinity.
Let us turn now to the second layer. By applying the above result for the first layer to  $\epsilon' = \epsilon / (C \max \{\| \pmb{W}_n^{(2)} \|_{\infty}, \| \pmb{W}_r^{(2)} \|_{\infty}\}$ , we have that for each  $i \in \{1, \ldots, d(2)\}$ :
Condition on the event that  $|[{\bf x}_v^{(1)} - {\bf z}_1]_i| <   \epsilon '$  for every node  $v$  and every  $i\in \{1,\ldots ,d(1)\}$
Fix  $v$  and consider the second-layer preactivations:
$$
\mathbf {y} _ {v} ^ {(2)} = \frac {1}{| \mathcal {N} ^ {+} (v) |} W _ {n} ^ {(2)} \sum_ {u \in \mathcal {N} ^ {+} (v)} \mathbf {x} _ {u} ^ {(1)} + \frac {1}{n} W _ {r} ^ {(2)} \sum_ {u \in V} \mathbf {x} _ {u} ^ {(1)} + \boldsymbol {b} ^ {(2)}
$$
$$
\boldsymbol {a} _ {2} := \frac {1}{| \mathcal {N} ^ {+} (v) |} \boldsymbol {W} _ {n} ^ {(2)} \sum_ {u \in \mathcal {N} ^ {+} (v)} \boldsymbol {z} _ {1} + \frac {1}{n} \boldsymbol {W} _ {r} ^ {(2)} \sum_ {u \in V} \boldsymbol {z} _ {1} + \boldsymbol {b} ^ {(2)}
$$
Fix  $i\in \{1,\ldots ,d(2)\}$  . Then:
$$
\begin{array}{l} \left| \left[ \mathbf {y} _ {v} ^ {(2)} - \boldsymbol {a} _ {2} \right] _ {i} \right| = \left| \frac {1}{| \mathcal {N} ^ {+} (v) |} \boldsymbol {W} _ {n} ^ {(2)} \sum_ {u \in \mathcal {N} ^ {+} (v)} \left(\mathbf {x} _ {u} ^ {(1)} - \boldsymbol {z} _ {1}\right) + \frac {1}{n} \boldsymbol {W} _ {r} ^ {(2)} \sum_ {u \in V} \left(\mathbf {x} _ {u} ^ {(1)} - \boldsymbol {z} _ {1}\right) \right| \\ \leq \frac {1}{| \mathcal {N} ^ {+} (v) |} \left\| \boldsymbol {W} _ {n} ^ {(2)} \right\| _ {\infty} \sum_ {u \in \mathcal {N} ^ {+} (v)} \left\| \mathbf {x} _ {u} ^ {(1)} - \boldsymbol {z} _ {1} \right\| _ {\infty} + \frac {1}{n} \left\| \boldsymbol {W} _ {r} ^ {(2)} \right\| _ {\infty} \sum_ {u \in V} \left\| \mathbf {x} _ {u} ^ {(1)} - \boldsymbol {z} _ {1} \right\| _ {\infty} \\ \leq \frac {\epsilon}{C} \\ \end{array}
$$
Let  $z_{2} \coloneqq \sigma(a_{2})$ . Then we can use the Lipschitz continuity of  $\sigma$  to bound the deviation of the activation from  $z_{2}$  as follows.
$$
\left| \left[ \mathbf {x} _ {v} ^ {(2)} - \mathbf {z} _ {2} \right] _ {i} \right| = \left| \left[ \sigma \left(\mathbf {y} _ {v} ^ {(2)}\right) - \sigma (\mathbf {a} _ {2}) \right] _ {i} \right| \leq C \left| \left[ \mathbf {y} _ {v} ^ {(2)} - \mathbf {a} _ {2} \right] _ {i} \right| \leq \epsilon
$$
Since the probability that  $|[\mathbf{x}_v^{(1)} - \mathbf{z}_1]_i| < \epsilon'$  for every node  $v$  and every  $i \in \{1, \dots, d(1)\}$  tends to 1, we get that the probability that  $|[\mathbf{x}_v^{(2)} - \mathbf{z}_2]_i| < \epsilon$  for every node  $v$  and every  $i \in \{1, \dots, d(2)\}$  also tends to 1.
Finally we apply the above argument inductively through all layers to get the desired result.
The proof of the main result now proceeds as in the proof of Theorem 4.6.
Proof of Theorem B.2. By Lemma B.3 the final node embeddings  $\mathbf{x}_v^{(T)}$  deviate less and less from  $z_{T}$  as the number of nodes  $n$  increases. Therefore, the average-pooled graph-level representation also deviates less than less from  $z_{T}$ . By inspecting the proof, we can see that this  $z_{T}$  is exactly the vector  $\pmb{\mu}_{T}$  in the definition of non-splitting (Definition B.1). This means that  $\pmb{z}_{T}$  cannot lie on a decision boundary for the classifier  $\mathfrak{C}$ . Hence, there is  $\epsilon > 0$  such that  $\mathfrak{C}$  is constant on:
$$
\left\{\boldsymbol {x} \in \mathbb {R} ^ {d (T)} \mid \forall i \in \{1, \dots , d (T) \}: [ \boldsymbol {z} _ {T} - \boldsymbol {x} ] _ {i} <   \epsilon \right\}
$$
C Proof of the zero-one law for  $\mathrm{SUMGNN^{+}}$
The proof of the key lemma works rather differently to the GCN and MEANGNN $^{+}$  case, but we still make important use of Hoeffding's Inequality.
Proof of Lemma 4.11. Consider the first layer preactivations  $\mathbf{y}_v^{(1)}$  and drop superscript (1)'s for notational clarity. We can rearrange the expression as follows:
$$
\mathbf {y} _ {v} = \left(\boldsymbol {W} _ {s} + \boldsymbol {W} _ {g}\right) \mathbf {x} _ {v} ^ {(0)} + \left(\boldsymbol {W} _ {n} + \boldsymbol {W} _ {g}\right) \sum_ {u \in \mathcal {N} (v)} \mathbf {x} _ {u} ^ {(0)} + \boldsymbol {W} _ {g} \sum_ {u \in V \backslash \mathcal {N} ^ {+} (v)} \mathbf {x} _ {u} ^ {(0)} + \boldsymbol {b}
$$
For  $u, v \leq n$  define:
$$
\mathbf {w} _ {u, v} = (\mathbf {A} _ {u v} \boldsymbol {W} _ {n} + \boldsymbol {W} _ {g}) \mathbf {x} _ {u} ^ {(0)} 1 _ {u \neq v} + (\boldsymbol {W} _ {s} + \boldsymbol {W} _ {g}) \mathbf {x} _ {u} ^ {(0)} 1 _ {u = v}
$$
Using this, we can rewrite:
$$
\mathbf {y} _ {v} = \sum_ {u = 1} ^ {n} \mathbf {w} _ {u, v} + \boldsymbol {b}
$$
By assumption on the distribution from which we draw graphs with node features, the  $\mathbf{w}_{u,v}$ 's are independent for any fixed  $v$ .
Now fix  $i \in \{1, \dots, d(1)\}$ . By Lemma A.2 and Lemma A.4 we have that each  $[\mathbf{w}_{u,v}]_i$  is sub-Gaussian. We therefore apply Hoeffding's Inequality to the sum. Note that  $\mathbf{w}_{u,v}$  can have one of two (sub-Gaussian) distributions, depending on whether  $u = v$ . Therefore, by Theorem A.1 and Lemma A.3, there are constants  $c$  and  $K$  such that, no matter how many nodes  $n$  there are, we have that:
$$
\mathbb {P} (| [ \mathbf {y} _ {v} ] _ {i} - \mathbb {E} [ \mathbf {y} _ {v} ] _ {i} | \geq t) = \mathbb {P} \left(\left| \sum_ {u = 1} ^ {n} ([ \mathbf {w} _ {u, v} ] _ {i} - \mathbb {E} [ \mathbf {w} _ {u, v} ] _ {i}) \right| \geq t\right) \leq 2 \exp \left(- \frac {c t ^ {2}}{K ^ {2} n}\right)
$$
Let's now compute  $\mathbb{E}[\mathbf{y}_v]$ , by first computing  $\mathbb{E}[\mathbf{w}_{u,v}]$ . When  $u = v$  we have that:
$$
\begin{array}{l} \mathbb {E} [ \mathbf {w} _ {v, v} ] = \mathbb {E} [ (\boldsymbol {W} _ {s} + \boldsymbol {W} _ {g}) \mathbf {x} _ {v} ^ {(0)} ] \\ = \left(\boldsymbol {W} _ {s} + \boldsymbol {W} _ {g}\right) \mathbb {E} \left[ \mathbf {x} _ {v} ^ {(0)} \right] \\ = \left(\boldsymbol {W} _ {s} + \boldsymbol {W} _ {g}\right) \boldsymbol {\mu} \\ \end{array}
$$
When  $u \neq v$  we have, using the independence of  $\mathbf{A}_{uv}$  and  $\mathbf{x}_v$ :
$$
\begin{array}{l} \mathbb {E} \left[ \mathbf {w} _ {u, v} \right] = \mathbb {E} \left(\left[ \mathbf {A} _ {u v} \boldsymbol {W} _ {n} + \boldsymbol {W} _ {g}\right) \mathbf {x} _ {u} ^ {(0)} \right] \\ = \left(\mathbb {E} \left[ \mathbf {A} _ {u v} \right] \boldsymbol {W} _ {n} + \boldsymbol {W} _ {g}\right) \mathbb {E} \left[ \boldsymbol {x} _ {v} ^ {(0)} \right] \\ = \left(r \boldsymbol {W} _ {n} + \boldsymbol {W} _ {g}\right) \boldsymbol {\mu} \\ \end{array}
$$
Therefore (separating  $\mathbf{w}_{v,v}$  from  $\mathbf{w}_{u,v}$  for  $u\neq v$ ):
$$
\mathbb {E} [ \mathbf {y} _ {v} ] = \sum_ {u = 1} ^ {n} \mathbb {E} [ \mathbf {w} _ {u, v} ] + \boldsymbol {b} = (n - 1) (r \boldsymbol {W} _ {n} + \boldsymbol {W} _ {g}) \boldsymbol {\mu} + (\boldsymbol {W} _ {s} + \boldsymbol {W} _ {g}) \boldsymbol {\mu} + \boldsymbol {b}
$$
Since  $\mathcal{M}$  is synchronously saturating for  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$ , we know that  $[(r\pmb {W}_n + \pmb {W}_g)\pmb {\mu}]_i\neq 0$ . Assume without loss of generality that  $[(r\pmb {W}_n + \pmb {W}_g)\pmb {\mu}]_i > 0$ . Then the expected value of  $[\mathbf{y}_v]_i$  increases as  $n$  tends to infinity; moreover we have a bound on how much  $[\mathbf{y}_v]_i$  can vary around its expected value.
Recall that the non-linearity  $\sigma$  is eventually constant in both directions. In particular, it is constant with value  $\sigma_{\infty}$  above some  $x_{\infty}$ . When  $\mathbb{E}[\mathbf{y}_v]_i > x_{\infty}$  the probability that  $[\mathbf{y}_v]_i$  doesn't surpass this threshold is:
$$
\begin{array}{l} \mathbb {P} \left(\left[ \mathbf {y} _ {v} \right] _ {i} <   x _ {\infty}\right) \leq \mathbb {P} \left(\left| \left[ \mathbf {y} _ {v} \right] _ {i} - \mathbb {E} \left[ \mathbf {y} _ {v} \right] _ {i} \right| > \left| x _ {\infty} - \mathbb {E} \left[ \mathbf {y} _ {v} \right] _ {i} \right|\right) \\ \leq 2 \exp \left(- \frac {c \left| x _ {\infty} - \mathbb {E} \left[ \mathbf {y} _ {v} \right] _ {i} \right| ^ {2}}{K ^ {2} n}\right) \\ \end{array}
$$
There is a constant  $\rho$  such that  $|x_{\infty} - \mathbb{E}[\mathbf{y}_v]_i|\geq \rho n$ . Hence for sufficiently large  $n$  (i.e. such that  $\mathbb{E}[\mathbf{y}_v]_i > x_\infty$ ):
$$
\mathbb {P} ([ \mathbf {y} _ {v} ] _ {i} <   x _ {\infty}) \leq 2 \exp \left(- \frac {c \rho^ {2} n ^ {2}}{K ^ {2} n}\right) = 2 \exp \left(- \frac {c \rho^ {2} n}{K ^ {2}}\right)
$$
Since the activation  $[\mathbf{x}_v]_i = \sigma ([\mathbf{y}_v]_i)$ , the probability that  $[\mathbf{x}_v]_i$  takes value  $\sigma_{\infty}$  is at least  $1 - 2\exp \left(-c\rho^{2}n / K^{2}\right)$ . Now, for each node  $v$  and each  $i\in \{1,\dots ,d(1)\}$ , the activation  $[\mathbf{x}_v]_i$  is either  $\sigma_{\infty}$  with high probability or  $\sigma_{-\infty}$  with high probability. By taking a union bound, for sufficiently large  $n$  the probability that every  $[\mathbf{x}_v]_i$  takes its corresponding value is at least:
$$
1 - 2 n d (1) \exp \left(- \frac {c \rho^ {2} n}{K ^ {2}}\right)
$$
This tends to 1 as  $n$  tends to infinity. In other words, there is  $z_{1} \in \{\sigma_{-\infty}, \sigma_{\infty}\}^{d(1)}$  such that  $\mathbf{x}_v^{(1)} = z_1$  for every  $v$  asymptotically.
We now proceed to the second layer, and condition on the event that  $\mathbf{x}_v^{(1)} = z_1$  for every  $v$ . In this case, we have that the second layer preactivation for node  $v$  is as follows.
$$
\mathbf {y} _ {v} ^ {(2)} = (\pmb {W} _ {s} ^ {(2)} + | \mathcal {N} (v) | \pmb {W} _ {n} ^ {(2)} + n \pmb {W} _ {g} ^ {(2)}) \pmb {z} _ {1} + \pmb {b} ^ {(2)}
$$
Since we're in the situation where every  $\mathbf{x}_u^{(1)} = z_1$ , the degree  $|\mathcal{N}(v)|$  is simply binomially distributed  $\mathrm{Bin}(n,r)$ . The preactivation  $\mathbf{y}_v^{(2)}$  then has expected value:
$$
\mathbb {E} \left[ \mathbf {y} _ {v} ^ {(2)} \right] = n \left(r \boldsymbol {W} _ {n} ^ {(2)} + \boldsymbol {W} _ {g} ^ {(2)}\right) \boldsymbol {z} _ {1} + \boldsymbol {W} _ {s} ^ {(2)} \boldsymbol {z} _ {1} + \boldsymbol {b} ^ {(2)}
$$
Fix  $i \in \{1, \ldots, d(2)\}$ . Since  $\mathcal{M}$  is synchronously saturating for  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$ , we have that  $[(r\pmb{W}_n^{(2)} + \pmb{W}_g^{(2)})\pmb{z}_1]_i \neq 0$ . Assume without loss of generality that  $[(r\pmb{W}_n^{(2)} + \pmb{W}_g^{(2)})\pmb{z}_1]_i > 0$ . Then  $\mathbb{E}[\mathbf{y}_v^{(2)}]_i$  tends to infinity as  $n$  increases.
Furthermore, we can view  $[n(rW_n^{(2)} + W_g^{(2)})z_1]_i$  as the sum of  $n$  Bernoulli random variables (which take values  $[(W_n^{(2)} + W_g^{(2)})z_1]_i$  and  $[W_g^{(2)}z_1]_i$ ). Since by Lemma A.4 Bernoulli random variables are sub-Gaussian, as in the first-layer case we can apply Hoeffding's Inequality to bound the probability that  $[\mathbf{y}_v^{(2)}]_i$  is less than  $x_{\infty}$ . We get that, for sufficiently large  $n$ , there is a constant  $K$  such that this probability is bounded by  $2\exp (-Kn)$ .
Then, as before, we find  $z_2 \in \{\sigma_{-\infty}, \sigma_{\infty}\}^{d(2)}$  such that, for sufficiently large  $n$ , every  $\mathbf{x}_v^{(2)} = z_2$  with probability at least  $1 - 2nd\exp(-Kn)$ .
Finally, this argument is applied inductively through all layers. As the number of layers remains constant (since  $\mathcal{M}$  is fixed), we find that the node embeddings throughout the model are asymptotically constant.
With the key lemma in place, we can now prove the main theorem.
Proof of Theorem 4.10. Applying Lemma 4.11 to the final layer, we find  $\mathbf{z}_T \in \{\sigma_{-\infty}, \sigma_{\infty}\}^{d(T)}$  such that every  $\mathbf{x}_v^{(T)} = \mathbf{z}_T$  with probability tending to 1. Since we use either average or component-wise maximum pooling, then means that the final graph-level representation is asymptotically constant, and thus the output of the classifier must be asymptotically constant.
D Proof of the uniform expressive power of  $\mathbf{SUMGNN}^+$  with random features
We make use of a result due to Abboud et al. [1] which shows that  $\mathrm{SUMGNN^{+}}$  models with random features can approximate any graph invariant on graphs with a fixed number of nodes.
Definition D.1. Let  $f$  be a function on graphs, and let  $\zeta$  be a random function on graphs. Take  $\delta > 0$  and  $N \in \mathbb{N}$ . Then  $\zeta$ $\delta$ -approximates  $f$  up to  $N$  if:
$$
\forall n \leq N: \mathbb {P} (\zeta (G) = f (G) \mid | G | = n) \geq 1 - \delta
$$
For completeness, we state the definition of the linearized sigmoid here.