SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sam Adam-Day*
sam.adam-day@cs.ox.ac.uk
Theodor-Mihai Iliant
theodor-mihai.iliant@lmh.ox.ac.uk
Graph neural networks (GNNs) are the de facto standard deep learning architectures for machine learning on graphs. This has led to a large body of work analyzing the capabilities and limitations of these models, particularly pertaining to their representation and extrapolation capacity. We offer a novel theoretical perspective on the representation and extrapolation capacity of GNNs, by answering the question: how do GNNs behave as the number of graph nodes become very large? Under mild assumptions, we show that when we draw graphs of increasing size from the Erdős-Rényi model, the probability that such graphs are mapped to a particular output by a class of GNN classifiers tends to either zero or to one. This class includes the popular graph convolutional network architecture. The result establishes 'zero-one laws' for these GNNs, and analogously to other convergence laws, entails theoretical limitations on their capacity. We empirically verify our results, observing that the theoretical asymptotic limits are evident already on relatively small graphs.
Graphs are common structures for representing relational data in a wide range of domains, including physical [35], chemical [7, 18], and biological [42, 10] systems, which sparked interest in machine learning over graphs. Graph neural networks (GNNs) [33, 14] have become prominent models for graph machine learning for a wide range of tasks, owing to their capacity to explicitly encode desirable relational inductive biases [5]. One important virtue of these architectures is that every GNN model can be applied to arbitrarily large graphs, since in principle the model parameters are independent of the graph size. This raises the question: how do GNNs behave as the number of nodes becomes very large? When acting as binary classifiers, GNNs can be thought of as parametrizing Boolean properties of (labelled) graphs. A classical method of specifying such properties is through first-order formulas, which allow for precise definitions using a formal language [9]. The celebrated 'zero-one law' for first-order logic [13, 8] provides a crisp answer to the question of the asymptotic behaviour of such properties: as graphs of increasing size are drawn from the Erdős-Rényi distribution, the probability that a first-order property holds either tends to zero or to one.
*Alternative email address: me@samadamday.com
In this paper, we show an analogous result for binary classification GNNs: under mild assumptions on the model architecture, several GNN architectures including graph convolutional networks [21] satisfy a zero-one law over Erdős-Rényi graphs with random node features. The principal import of this result is that it establishes a novel upper-bound on the expressive power of GNNs: any property of graphs which can be uniformly expressed by a GNN must obey a zero-one law. An example of a simple property which does not asymptotically tend to zero or one is that of having an even number of nodes. Note however that our result, combined with the manifest success of GNNs in practice, suggests that zero-one laws must be abundant in nature: if a property we cared about did not satisfy a zero-one law, none of the GNN architectures we consider would be able to express it.
Our main results flexibly apply both to the case where we consider the GNN as a classifier applied to randomly sampled graphs and node features, and where we consider the random node features as part of the model. For the latter, it is known that incorporating randomized features into the model significantly increases its expressive power on graphs with bounded number of nodes [32, 1]. Our results yield the first upper bound on the expressive power of these architectures which is uniform in the number of nodes. We complement this with a corresponding lower bound, showing that these architectures can universally approximate any property which satisfies a certain zero-one law.
A key strength of the results is that they apply equally well to randomly initialized networks, trained networks, and anything in between. In this sense, our asymptotic analysis is orthogonal to the question of optimization, and holds regardless of the choice of training method. Another interesting aspect of these results is that they unite analysis of expressive power with extrapolation capacity. Our zero-one laws simultaneously provide limits on the ability of GNNs to extrapolate from smaller Erdős-Rényi graphs to larger ones: eventually any GNN must classify all large graphs the same way.
To validate our theoretical findings, we conduct a series of experiments: since zero-one laws are of asymptotic nature, we may need to consider very large graphs to observe clear empirical evidence for the phenomenon. Surprisingly however, GNNs already exhibit clear evidence of a zero-one law even on small graphs. Importantly, this is true for networks with very few layers (even a single-layer), which is reassuring, as it precludes confounding factors, such as the effect of over-smoothing due to increased number of layers [23]. We provide further experimental results in the appendix of this paper, where all proofs of technical statements can also be found. We make the code for our experiments available online at https://github.com/SamAdamDay/Zero-One-Laws-of-Graph-Neural-Networks.
Random graphs and matrices. The focus of our study is on classes of random graphs with random features, for which we introduce some notation. We write  $\mathbf{x} \in \mathbb{R}^d$  to represent a vector, and  $\mathbf{X} \in \mathbb{R}^{d \times n}$  to represent a matrix. Analogously, we write  $\mathbf{x}$  to denote a random vector, and  $\mathbf{X}$  to denote a random matrix, whose entries are (real) random variables. We write  $\mathbb{G}(n,r)$  to denote the class of simple, undirected Erdős-Rényi (ER) graphs with  $n$  nodes and edge probability  $r$  and let  $\mathbb{D}(d)$  denote some distribution of feature vectors over  $\mathbb{R}^d$ . We define an Erdős-Rényi graph equipped with random node features as a pair  $\mathcal{G} = (\mathbf{A},\mathbf{X})$ , where  $\mathbf{A} \sim \mathbb{G}(n,r)$  is the random graph adjacency matrix of the graph  $G = (V,E)$  and  $\mathbf{X} \in \mathbb{R}^{d \times n}$  is a corresponding random feature matrix, independent of  $G$ , which contains, for each node  $v \in V$ , an initial random node feature  $\mathbf{x}_v \sim \mathbb{D}(d)$  as the corresponding columns of  $\mathbf{X}$ .<sup>2</sup>
Message passing neural networks. The focus of this work is on message-passing neural networks (MPNNs) [12, 17] which encapsulate the vast majority of GNNs. The fundamental idea in MPNNs is to update the initial (random) state vector  $\mathbf{x}_v^{(0)} = \mathbf{x}_v$  of each node  $v$  for  $T \in \mathbb{N}$  iterations, based on its own state and the state of its neighbors  $\mathcal{N}(v)$  as:
$$
\mathbf {x} _ {v} ^ {(t + 1)} = \phi^ {(t)} \left(\mathbf {x} _ {v} ^ {(t)}, \psi^ {(t)} \left(\mathbf {x} _ {v} ^ {(t)}, \{\{\mathbf {x} _ {u} ^ {(t)} | u \in \mathcal {N} (v) \}\}\right)\right),
$$
where  $\{\cdot\}$  denotes a multiset, and  $\phi^{(t)}$  and  $\psi^{(t)}$  are differentiable combination, and aggregation functions, respectively. Each layer's node representations can have different dimensions: we denote by  $d(t)$  the dimension of the node embeddings at iteration  $t$  and typically write  $d$  in place of  $d(0)$ .
2We define a  $d\times |V|$  dimensional (random) feature matrix as opposed to the more common  $|V|\times d$ . This is for ease of presentation, since we aim to work on the (random) column vectors of such matrices.
The final node representations can then be used for node-level predictions. For graph-level predictions, the final node embeddings are pooled to form a graph embedding vector to predict properties of entire graphs. The pooling often takes the form of simple averaging, summing or component-wise maximum. For Boolean node (resp., graph) classification, we further assume a classifier  $\mathfrak{C}:\mathbb{R}^{d(T)}\to \mathbb{B}$  which acts on the final node representations (resp., on the final graph representation).
There exist more general message passing paradigms [5] such as MPNNs with global readout which additionally aggregate over all node features at every layer, and are known to be more expressive [4]. Some model architectures considered in this paper include a global readout component and we consider different choices for the combine  $(\phi^{(t)})$  and aggregate  $(\psi^{(t)})$  functions, as we introduce next.
GCN. The primary GNN architecture we consider is graph convolutional networks (GCN) [21]. These are instances of MPNNs with self-loops, which aggregate over the extended neighborhood of a node  $\mathcal{N}^{+}(v)\coloneqq \mathcal{N}(v)\cup \{v\}$ . GCNs iteratively update the node representations as  $\mathbf{x}_v^{(t)} = \sigma \left(\mathbf{y}_v^{(t)}\right)$ , where the preactivations are given by:
$$
\mathbf {y} _ {v} ^ {(t)} = \boldsymbol {W} _ {n} ^ {(t)} \sum_ {u \in \mathcal {N} ^ {+} (v)} \frac {1}{\sqrt {| \mathcal {N} (u) | | \mathcal {N} (v) |}} \mathbf {x} _ {u} ^ {(t - 1)} + \boldsymbol {b} ^ {(t)}
$$
We apply the linear transformation  $\mathbf{W}_n^{(t)} \in \mathbb{R}^{d(t) \times d(t - 1)}$  to a normalized sum of the activations for the previous layers of the neighbors of the node under consideration, together with its own activation. Adding a bias term  $\mathbf{b}^{(t)}$  yields the preactivation  $\mathbf{y}_v^{(t)}$ , to which we apply the non-linearity  $\sigma$ .
MEANGNN. We also consider the MEANGNN $^+$  architecture which is a self-loop GNN with mean aggregation and global readout [17], and updates the node representation as  $\mathbf{x}_v^{(t)} = \sigma (\mathbf{y}_v^{(t)})$ , where:
$$
\mathbf {y} _ {v} ^ {(t)} = \frac {1}{| \mathcal {N} ^ {+} (v) |} W _ {n} ^ {(t)} \sum_ {u \in \mathcal {N} ^ {+} (v)} \mathbf {x} _ {u} ^ {(t - 1)} + \frac {1}{n} W _ {r} ^ {(t)} \sum_ {u \in V} \mathbf {x} _ {u} ^ {(t - 1)} + \boldsymbol {b} ^ {(t)}
$$
$\mathbf{MEANGNN}^+$  models additionally apply a linear transformation  $\pmb{W}_r^{(t)} \in \mathbb{R}^{d(t) \times d(t-1)}$  to the mean of all previous node representations. We refer to MEANGNN as the special case of this architecture which does not include a global readout term (obtained by dropping the second term in the equation).
SUMGNN. Finally, we consider the  $\mathrm{SUMGNN}^+$  architecture which is a GNN with sum aggregation and global readout [12], and updates the node representations as  $\mathbf{x}_u^{(t)} = \sigma (\mathbf{y}_u^{(t)})$ , where:
$$
\mathbf {y} _ {v} ^ {(t)} = \boldsymbol {W} _ {s} ^ {(t)} \mathbf {x} _ {v} ^ {(t - 1)} + \boldsymbol {W} _ {n} ^ {(t)} \sum_ {u \in \mathcal {N} (v)} \mathbf {x} _ {u} ^ {(t - 1)} + \boldsymbol {W} _ {r} ^ {(t)} \sum_ {u \in V} \mathbf {x} _ {u} ^ {(t - 1)} + \boldsymbol {b} ^ {(t)}
$$
This time, we separate out the contribution from the preactivation of the previous activation for the node itself. This yields three linear transformations  $\mathbf{W}_s^{(t)}, \mathbf{W}_n^{(t)}, \mathbf{W}_r^{(t)} \in \mathbb{R}^{d(t) \times d(t-1)}$ . The corresponding architecture without the global readout term is called SUMGNN.
Graph neural networks are flexible models which can be applied to graphs of any size following training. This makes an asymptotic analysis in the size of the input graphs very appealing, since such a study could lead to a better understanding of the extrapolation capabilities of GNNs, which is widely studied in the literature [41, 40]. Previous studies of the asymptotic behaviour of GNNs have focused on convergence to theoretical limit networks [20, 31] and their stability under the perturbation of large graphs [11, 22].
Zero-one laws have a rich history in first-order logic and random graph theory [13, 8, 25, 34, 6]. Being the first of its kind in the graph machine learning literature, our study establishes new links between graph representation learning, probability theory, and logic, while also presenting a new and interesting way to analyze the expressive power of GNNs. It is well-known that the expressive power of MPNNs is upper bounded by the  $l$ -dimensional Weisfeiler Leman graph isomorphism test (1-WL) [39, 29] and architectures such as  $\mathrm{SUMGNN}^+$  [29] can match this. Barceló et al. [4] further gives a logical characterization for a class of MPNNs showing  $\mathrm{SUMGNN}^+$  can capture any
function which can be expressed in the logic  $C^2$ , which is an extension of the two-variable fragment of first-order logic with counting quantifiers. Several works study the expressive power of these models under the assumption that there are unique node identifiers [26], or define higher-order GNN models [29, 27, 28, 19] to obtain more expressive architectures.
Our work has direct implications on GNNs using random node features [32, 1], which are universal in the bounded graph domain. Specifically, we derive a zero-one law for GNNs using random node features which puts an upper bound on the expressive power of such models in a uniform sense: what class of functions on graphs can be captured by a single GNN with random node features? Abboud et al. [1] prove a universality result for these models, but it is not uniform, since the construction depends on the graph sizes, and yields a different model parametrization depending on the choice of the graph sizes. Moreover, the construction of Abboud et al. [1] is of size exponential in the worst case. Grohe [15] recently improved upon this result, by proving that the functions that can be computed by a polynomial-size bounded-depth family of GNNs using random node features are exactly the functions computed by bounded depth Boolean circuits with threshold gates. This establishes an upper bound on the power of GNNs with random node features, by requiring the class of models to be of bounded depth (fixed layers) and of size polynomial. However, this result is still not uniform, since it allows the target function to be captured by different model parametrizations. There is no known upper bound for the expressive power of GNNs with random node features in the uniform setting, and our result establishes this.
Other limitations of MPNNs include over-smoothing [23, 30] and over-squashing [2] which are related to information propagation, and are linked to using more message passing layers. The problem of over-smoothing has also been studied from an asymptotic perspective [23, 30], where the idea is to see how the node features evolve as we increase the number of layers in the network. Our study can be seen as orthogonal to this work: we conduct an asymptotic analysis in the size of the graphs rather than in the number of layers.
4 Zero-one laws of graph neural networks
We first define graph invariants following Grohe [16].
Definition 4.1. A graph invariant  $\xi$  is a function over graphs, such that for any pair of graphs  $G_{1}$ ,  $G_{2}$ , and, for any isomorphism  $f$  from  $G_{1}$  to  $G_{2}$  it holds that  $\xi(G_{1}) = \xi(f(G_{2}))$ . Graph invariants for graphs with node features are defined analogously.
Consider any GNN model  $\mathcal{M}$  used for binary graph classification. It is immediate from the definition that  $\mathcal{M}$  is invariant under isomorphisms of the graphs on which it acts. Hence  $\mathcal{M}$ , considered as a function from graphs to  $\mathbb{B} = \{0,1\}$ , is a graph invariant. In this paper, we study the asymptotic behavior of  $\mathcal{M}$  as the number of nodes increases.
One remarkable and influential result from finite model theory is the 'zero-one law' for first-order logic. A (Boolean) graph invariant  $\xi$  satisfies a zero-one law if when we draw graphs  $G$  from the ER distribution  $\mathbb{G}(n,r)$ , as  $n$  tends to infinity the probability that  $\xi(G) = 1$  either tends to 0 or tends to 1. The result, due to Glebskii et al. [13] and Fagin [8], states that any graph invariant which can be expressed by a first-order formula satisfies a zero-one law. Inspired by this asymptotic analysis of first-order properties, we ask whether GNNs satisfy a zero-one law. As the input of a GNN is a graph with node features, we need to reformulate the statement of the law to incorporate these features.
Definition 4.2. Let  $\mathcal{G} = (\mathbf{A},\mathbf{X})$  be a graph with node features, where  $\mathbf{A}\sim \mathbb{G}(n,r)$  is a graph adjacency matrix and, independently,  $\mathbf{X}$  is a matrix of node embeddings, where  $\mathbf{x}_v\sim \mathbb{D}(d)$  for every node  $v$ . A graph invariant  $\xi$  for graphs with node features satisfies a zero-one law with respect to  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$  if, as  $n$  tends to infinity, the probability that  $\xi (\mathcal{G}) = 1$  tends to either 0 or 1.
Studying the asymptotic behavior of GNNs helps to shed light on their capabilities and limitations. A zero-one law establishes a limit on the ability of such models to extrapolate to larger graphs: any GNN fitted to a finite set of datapoints will tend towards outputting a constant value on larger and larger graphs drawn from the distribution described above. A zero-one law in this setting also transfers to a corresponding zero-one law for GNNs with random features. This establishes an upper-bound on the uniform expressive power of such models.
4.2 Graph convolutional networks obey a zero-one law
Our main result in this subsection is that (Boolean) GCN classifiers obey a zero-one law. To achieve our result, we place some mild conditions on the model and initial node embeddings.
First, our study covers sub-Gaussian random vectors, which in particular include all bounded random vectors, and all multivariate normal random vectors. We note that in every practical setup all node features have bounded values (determined by the bit length of the storage medium), and are thus sub-Gaussian.
Definition 4.3. A random vector  $\mathbf{x} \in \mathbb{R}^d$  is sub-Gaussian if there is  $C > 0$  such that for every unit vector  $\mathbf{y} \in \mathbb{R}^d$  the random variable  $\mathbf{x} \cdot \mathbf{y}$  satisfies the sub-Gaussian property; that is, for every  $t > 0$ :
$$
\mathbb {P} (| \mathbf {x} \cdot \boldsymbol {y} | \geq t) \leq 2 \exp \left(- \frac {t ^ {2}}{C ^ {2}}\right)
$$
Second, we require that the non-linearity  $\sigma$  be Lipschitz continuous. This is again a mild assumption, because all non-linearities used in practice are Lipschitz continuous, including ReLU, clipped ReLU, sigmoid, linearized sigmoid and tanh.
Definition 4.4. A function  $f\colon \mathbb{R}\to \mathbb{R}$  is Lipschitz continuous if there is  $C > 0$  such that for any  $x,y\in \mathbb{R}$  it holds that  $|f(x) - f(y)|\leq C|x - y|$ .
Third, we place a condition on the GCN weights with respect to the classifier function  $\mathfrak{C}\colon \mathbb{R}^{d(T)}\to \mathbb{B}$  which intuitively excludes a specific weight configuration.
Definition 4.5. Consider a distribution  $\mathbb{D}(d)$  with mean  $\pmb{\mu}$ . Let  $\mathcal{M}$  be a GCN used for binary graph classification. Define the sequence  $\pmb{\mu}_0, \dots, \pmb{\mu}_T$  of vectors inductively by  $\pmb{\mu}_0 \coloneqq \pmb{\mu}$  and  $\pmb{\mu}_t \coloneqq \sigma(\pmb{W}_n^{(t)}\pmb{\mu}_{t-1} + \pmb{b}^{(t)})$ . The classifier classifier  $\mathfrak{C}: \mathbb{R}^{d(T)} \to \mathbb{B}$  is non-splitting for  $\mathcal{M}$  if the vector  $\pmb{\mu}_T$  does not lie on a decision boundary for  $\mathfrak{C}$ .
For all reasonable choices of  $\mathfrak{C}$ , the decision boundary has dimension lower than the  $d(T)$ , and is therefore a set of zero-measure. This means that in practice essentially all classifiers are non-splitting.
Given these conditions, we are ready to state our main theorem:
Theorem 4.6. Let  $\mathcal{M}$  be a GCN used for binary graph classification and take  $r\in [0,1]$ . Then,  $\mathcal{M}$  satisfies a zero-one law with respect to graph distribution  $\mathbb{G}(n,r)$  and feature distribution  $\mathbb{D}(d)$  assuming the following conditions hold: (i) the distribution  $\mathbb{D}(d)$  is sub-Gaussian, (ii) the nonlinearity  $\sigma$  is Lipschitz continuous, (iii) the graph-level representation uses average pooling, and (iv) the classifier  $\mathfrak{C}$  is non-splitting.
The proof hinges on a probabilistic analysis of the preactivations in each layer. We use a sub-Gaussian concentration inequality to show that the deviation of each of the first-layer preactivations  $\mathbf{y}_v^{(1)}$  from its expected value becomes less and less as the number of node  $n$  tends to infinity. Using this and the fact that  $\sigma$  is Lipschitz continuous we show then that each activation  $\mathbf{x}_v^{(1)}$  tends towards a fixed value. Iterating this analysis through all the layers of the network yields the following key lemma, which is the heart of the argument.
Lemma 4.7. Let  $\mathcal{M}$  and  $\mathbb{D}(d)$  satisfy the conditions in Theorem 4.6. Then, for every layer  $t$ , there is  $\pmb{z}_t \in \mathbb{R}^{d(t)}$  such that when sampling a graph with node features from  $\mathbb{G}(n,r)$  and  $\mathbb{D}(d)$ , for every  $i \in \{1,\dots,d(t)\}$  and for every  $\epsilon > 0$  we have that:
$$
\mathbb {P} \left(\forall v: \left|\left[ \mathbf {x} _ {v} ^ {(t)} - \boldsymbol {z} _ {t} \right] _ {i} \right| <   \epsilon\right)\rightarrow 1 a s n \rightarrow \infty
$$
With the lemma established, the proof of Theorem 4.6 follows straightforwardly from the last two assumptions. Since the final node embeddings  $\mathbf{x}_v^{(T)}$  tend to a fixed value  $z_{T}$ , the average-pooled graph-level representations also tend to this. Since we assume that the classifier is non-splitting, this value cannot lie on a decision boundary, and thus the final output is asymptotically stable at  $\mathfrak{C}(z_T)$ .
We expect that the rate of converge will depend in a complex way on the number of layers, the embedding dimensionality, and the choice of non-linearity, which makes a rigorous analysis very challenging. However, considering the manner in which Lemma 4.7 is proved, we can arrive at the
following intuitive argument for why the rate of convergence should decrease as the embedding dimensionality increases: if we fix a node  $v$  and a layer  $t$  then each of the components of its preactivation can be viewed as the weighted sum of  $d(t - 1)$  random variables, each of which is the aggregation of activations in the previous layer. Intuitively, as  $d(t - 1)$  increases, the variance of this sum also increases. This increased variance propagates through the network, resulting in a higher variance for the final node representations and thus a slower convergence.
Using analogous assumptions and techniques to those presented in this section, we also establish a zero-one law for MEANGNN $^{+}$ , which we report in detail in Appendix B. Abstracting away from technicalities, the overall structure of the proofs for MEANGNN $^{+}$  and GCN is very similar, except that for the former case we additionally need to take care of the global readout component.
4.3 Graph neural networks with sum aggregation obey a zero-one law
The other variant of GNNs we consider are those with sum aggregation. The proof in the case works rather differently, and we place different conditions on the model.
Definition 4.8. A function  $\sigma \colon \mathbb{R} \to \mathbb{R}$  is eventually constant in both directions if there are  $x_{-\infty}, x_{\infty} \in \mathbb{R}$  such that  $\sigma(y)$  is constant for  $y < x_{-\infty}$  and  $\sigma(y)$  is constant for  $y > x_{\infty}$ . We write  $\sigma_{-\infty}$  to denote the minimum and  $\sigma_{\infty}$  to denote the maximum value of an eventually constant function  $\sigma$ .
This means that there is a threshold  $(x_{-\infty})$  below which sigma is constant, and another threshold  $(x_{\infty})$  above which sigma is constant. Both the linearized sigmoid and clipped ReLU are eventually constant in both directions. Moreover, when working with finite precision any function with vanishing gradient in both directions (such as sigmoid) can be regarded as eventually constant in both directions.