SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition D.2. The linearized sigmoid is the function  $\mathbb{R}\to \mathbb{R}$  defined as follows:
$$
x \mapsto \left\{ \begin{array}{l l} - 1 & \text {i f} x \in (- \infty , - 1), \\ x & \text {i f} x \in [ - 1, 1), \\ 1 & \text {o t h e r w i s e}. \end{array} \right.
$$
Theorem D.3. Let  $\xi$  be any graph invariant. For every  $N\in \mathbb{N}$  and  $\delta >0$  there is a  $\mathrm{SUMGNN}^+$  with random features  $\mathcal{M}$  which  $\delta$ -approximates  $\xi$  up to  $N$ . Moreover,  $\mathcal{M}$  uses the linearized sigmoid as the non-linearity and the distribution of the initial node embeddings consists of  $d$  iid  $U[0,1]$  random variables.
Proof. See [1, Theorem 1].
With this result we can now prove the uniform expressivity result.
Proof of Theorem 5.2. First,  $\xi$  satisfies a zero-one law for  $\mathbb{G}(n,1 / 2)$ . Without loss of generality assume that  $\xi$  is asymptotically 1. There is  $N\in \mathbb{N}$  such that for every  $n > N$  we have:
$$
\mathbb {P} (\xi (G) = 1 \mid G \sim \mathbb {G} (n, 1 / 2)) \geq 1 - \delta
$$
Note that this  $N$  depends on both  $\xi$  and  $\delta$ .
Second, by Theorem D.3 there is a  $\mathrm{SUMGNN}^+$  with random features  $\mathcal{M}$  which  $\delta$ -approximates  $\xi$  up to  $N$ . Moreover,  $\mathcal{M}'$  uses the linearized sigmoid as the non-linearity and the distribution of the initial node embeddings consists of  $d$  iid  $U[0,1]$  random variables.
Using the global readout and the linearized sigmoid, we can condition the model behavior on the number of nodes. We give a rough description of the model as follows. Define a  $\mathrm{SUMGNN}^+$  with random features  $\mathcal{M}$  by extending  $\mathcal{M}'$  as follows.
- Increase the number of layers to at least three.
- Increase each embedding dimension by 1. For convenience call this the 0th component of each embedding.
- Use the bias term in the first layer to ensure that the 0th component of the activation  $\mathbf{x}_v^{(1)}$  for each node  $v$  is 1.
- Use the global readout to threshold the number of nodes on  $N$ . The 0th row of the matrix  $W_{g}^{(2)}$  should have a 2 in the 0th position and 0's elsewhere. The 0th component of the bias vector  $b^{(2)}$  should be  $2N - 1$ . This ensures that the 0th component of every activation  $\mathbf{x}_v^{(2)}$  is 1 if  $n > N$  and -1 otherwise.
- Propagate this value through the 0th component of each layer embedding.
- In the final layer, use this value to decide whether to output what  $\mathcal{M}'$  would output, or simply to output 1.
For any  $n \leq N$  the model  $\mathcal{M}$  behaves like  $\mathcal{M}'$ . Therefore:
$$
\mathbb {P} (\xi (G) = \mathcal {M} (G) \mid | G | = n) \geq 1 - \delta
$$
On the other hand, for  $n > N$  the model  $\mathcal{M}$  simply outputs 1 and so:
$$
\mathbb {P} (\xi (G) = \mathcal {M} (G) \mid | G | = n) = \mathbb {P} (\xi (G) = 1 \mid | G | = n) \geq 1 - \delta
$$
Thus  $\mathcal{M}$  uniformly  $\delta$ -approximates  $\xi$ .
In this section, we focus on GCNs and provide further experiments regarding our results. In particular, we pose the following questions:
1. Our theoretical results entail a zero-one law for a large class of distributions: do we empirically observe a zero-one law when node features are instead drawn from a normal distribution (Appendix E.1)?
2. Our theoretical results state a zero-one law for a large class of non-linearities: do we empirically observe a zero-one law when considering other common non-linearities (Appendix E.2)?
3. Does a zero-one law also manifest itself empirically for GAT models (Appendix E.3)?
4. Do we empirically observe a zero-one law if we were to consider sparse Erdős-Rényi graphs (Appendix E.4)?
5. Is there empirical evidence for our results to apply to other random graph models, such as the Barabási-Albert model (Appendix E.5)?
E.1 Experiments with initial node features drawn from a normal distribution
Here we consider using a normal distribution to draw our initial node features. Note that normal distributions are sub-Gaussian, and hence our theoretical findings (Theorem 4.6) confer a zero-one law in this case. Figure 2 demonstrates the results for GCN models. We observe the expected asymptotic behavior in most cases, however in the two and three layer cases a few models have not converged by the end of the experiment.
E.2 Experiments with other non-linearities
In this subsection we test the effect of using different non-linearities in the layers of our GCN models. Theorem 4.6 applies in all of these cases, so we do expect to see a zero-one law. Figures 3 to 5 present the results for ReLU, tanh and sigmoid, respectively. We see the expected behavior in all cases. Note however that, in contrast with other non-linearities, when we use sigmoid we observe that the rate of convergence actually increases as the number of layers increases. This suggests a complex relationship between the rate of convergence, the non-linearity and the number of layers.
E.3 Experiments with GAT
Here we investigate the asymptotic behaviour of a GNN architecture not considered in the main body: the Graph Attention Network [37]. Cast as an MPNN, this architecture uses an attention mechanism as the aggregate function  $\phi$  in the message passing step. The techniques used in this paper to establish a zero-one law for other GNN architectures do not easily extend to GAT. However, our experiments demonstrate a very quick convergence to 0 or 1.
E.4 Experiments on sparse Erdős-Rényi graphs
In these experiments, we consider GCN models on a variation of the Erdős-Rényi distribution in which the edge probability  $r$  is allowed to vary as a function of  $n$ . Specifically, we set  $r = \log (n) / n$ , which yields sparser graphs than in the standard distribution. Our experiments provide evidence for a zero-one law also in this case (Figure 7).
E.5 Experiments on the Barabási-Albert random graph model
In this subsection, we consider another alternative graph distribution: the Barabási-Albert model [3]. This model aims to better capture the degree distributions commonly found in real-world networks. We can again observe a zero-one law for GCN models under this distribution (Figure 8).