SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thien Le
thienle@mit.edu
TU Munich and MIT
stefje@csail.mit.edu
Can graph neural networks generalize to graphs that are different from the graphs they were trained on, e.g., in size? In this work, we study this question from a theoretical perspective. While recent work established such transferability and approximation results via graph limits, e.g., via graphons, these only apply nontrivially to dense graphs. To include frequently encountered sparse graphs such as bounded-degree or power law graphs, we take a perspective of taking limits of operators derived from graphs, such as the aggregation operation that makes up GNNs. This leads to the recently introduced limit notion of graphops (Backhausz and Szegedy, 2022). We demonstrate how the operator perspective allows us to develop quantitative bounds on the distance between a finite GNN and its limit on an infinite graph, as well as the distance between the GNN on graphs of different sizes that share structural properties, under a regularity assumption verified for various graph sequences. Our results hold for dense and sparse graphs, and various notions of graph limits.
Since the advent of graph neural networks (GNNs), deep learning has become one of the most promising tools to address graph-based tasks (Gilmer et al., 2017; Scarselli et al., 2009; Kipf and Welling, 2017; Bronstein et al., 2017). Following the mounting success of applied GNN research, theoretical analyses follow with many works studying GNNs' representational power (Azizian and Lelarge, 2021; Morris et al., 2019; Xu et al., 2019, 2020; Garg et al., 2020; Chen et al., 2020; Maron et al., 2019; Loukas, 2020a,b; Abboud et al., 2021).
A hitherto less addressed question of practical importance is the possibility of size generalization, i.e., transferring a learned GNN to graphs of different sizes (Ruiz et al., 2023a; Levie et al., 2022; Xu et al., 2021; Yehudai et al., 2021; Bevilacqua et al., 2021; Chuang and Jegelka, 2022; Roddenberry et al., 2022; Maskey et al., 2022, 2023), especially for sparse graphs. For instance, it would be computationally desirable to train a GNN on small graphs and apply it to large graphs. This question is also important to judge the reliability of the learned model on different test graphs. To answer the size generalization question, we need to understand under which conditions such transferability is possible - since it may not always be possible (Xu et al., 2021; Yehudai et al., 2021; Jegelka, 2022) - and what output perturbations we may expect. For a formal analysis of perturbations and conditions, we need a suitable graph representation that captures inductive biases and allows us to compare models for graphs of different sizes. Graph limits can help to formalize this, as they help understand biases as the graph size tends to infinity.
Formally, approximation theory asks for bounds between a GNN on a finite graph and its infinite counterpart, while transferability compares model outputs on graphs of different sizes.
The quality of the bounds depends on how the two GNNs (and corresponding graphs) are intrinsically linked, in particular, to what extent they share relevant structure. This yields conditions for size generalization. For example, the graphs could be sampled from the same graph limit (Ruiz et al., 2023a) or from the same random graph model (Keriven et al., 2020).
In particular, Ruiz et al. (2023a) study approximation and transferability via the lens of graphons (Lovasz, 2012; Lovasz and Szegedy, 2006), which characterize the limits of dense graphs. Yet, many real-world graphs are not dense, for instance, planar traffic networks, power law graphs, polymer graphs, Hamming graphs (including hypercubes for error-correcting code), or grid-like graphs e.g., for images. For sparser graphs, the correct notion of limit suitable for deep learning is still an open problem, as typical bounded-degree graph limits such as the Benjamini-Schramm limit of random rooted graphs (Benjamini and Schramm, 2001), or graphings (Lovasz, 2012) are less well understood and often exhibit pathological behaviors (see Section 2.1). Limits of intermediate graphs, such as the hypercubes, are even more obscure. Hence, understanding limits, inductive biases and transferability of GNNs for sparse graphs remains an open problem in understanding graph representation learning.
This question is the focus of this work. To obtain suitable graph limits for sparse graphs and to be able to compare GNNs on graphs of different sizes while circumventing challenges of sparse graph limits, we view a graph as an operator derived from it. This viewpoint is naturally compatible with GNNs, as they are built from convolution/aggregation operations. We show how the operator perspective allows us to define limits of GNNs of infinite sequences of graphs. We achieve this by exploiting the recently defined notion of graphop, which generalizes graph shift operators, and the action convergence defined in the space of graphops (Backhausz and Szegedy, 2022). Our definition of GNN limits enables us to prove rigorous bounds for approximation and transferability of GNNs for sparse graphs. Since graphops encompass both graphons and graphings, we generalize similar bounds for graphon neural networks (Ruiz et al., 2023a; Maskey et al., 2023) to a much wider set of graphs.
Yet, using graphops requires technical work. For instance, we need to introduce an appropriate discretization of a graphop to obtain its corresponding finite graph shift operators. We use these operators to define a generalized graphop neural network that acts as a limit object, with discretizations that become finite GNNs. Then we prove approximation and transferability results for both the operators (graphops and their discretizations) and GNNs.
Contributions. To the best of our knowledge, this is the first paper to provide approximation and transferability theorems specifically for sparse graph limits. Our main tool, graphops, has not been used to study GNNs before, although viewing graphs as operators is a classic theme in the literature. Our specific contributions are as follows:
1. We define a graphop convolution, i.e., an operator that includes both finite graph convolutions and a limit version that allows us to define a limit object for GNNs applied to graphs of size  $n \to \infty$ .
2. We rigorously prove an approximation theorem (Theorem 2) that bounds a distance between a graph  $A$  (acting on infinite-dimensional space) and and appropriate discretization  $A_{n}$  (acting on  $\mathbb{R}^n$ ), in the  $d_M$  metric introduced by Backhausz and Szegedy (2022). Our result applies to a more general set of nonlinear operators, and implies a transferability bound between finite graphs (discretizations) of different sizes.
3. For neural networks, we present a quantitative approximation and transferability bound that guarantees outputs of graphop neural networks are close to those of the corresponding GNNs (obtained from discretization).
A summary of comparisons between our framework and related papers is in Table 1.
In structure, the closest related work is (Ruiz et al., 2023a), which derives approximation and transferability theorems for graphon neural networks, i.e., dense graphs. For graphons, the convolution kernel has a nice spectral decomposition, which is exploited by Ruiz et al.
$^1$ Rate of convergence to a small positive constant. For convergence to 0, Ruiz et al. (2020) showed the rate of  $O(n^{-1/2})$ .
Sparse	Dense
Bounded-degree	Relatively-sparse
Number of edges	Θ(n)	Θ(n log n)	Θ(n2)
Examples covered under our assumptions	infinite grids, polymer graphs	hypercubes, Hamming graphs	graphons
Graphons (Ruiz et al., 2023a)	O(n-1)1
Unbounded graphons (Maskey et al., 2023)	inexplicit
Random graph model (Keriven et al., 2020)	O((log n)-1/2)	O(n-1/2)
Spectral methods (1 layer) (Levie et al., 2022)	inexplicit	inexplicit
Graphings (1 layer) (Roddenberry et al., 2022)	inexplicit
P-operators and graphops (ours)	O(n-1/2)	O(n-1/2)	O(n-1/2)
Table 1: Summary of our results compared to related work. Quantitative results (e.g.  $O(n^{-1/2})$ ) upper-bound the distance between GNNs on sampled graphs of size  $n$  and the limiting object in term of  $n$  (in an appropriate metric and limit notion). Empty cells are graph models where the approaches in the corresponding papers do not apply to or give trivial bounds (e.g. bounds that compare to a constant-0 graphon). "Inexplicit" refers to asymptotic results where rates of convergence is not explicit.
(2023a). In contrast, sparse graph limits are not known to enjoy nice convergence of the spectrum (Backhausz and Szegedy, 2022; Aldous and Lyons, 2007), also Appendix C.1, so we need to use different techniques. Since the notion of graphop generalizes both dense graph limits and certain sparse graph limits, our results apply to dense graphs as well. Our assumptions and settings are slightly different from Ruiz et al. (2023a). For instance, they allow the convolution degree  $K \to \infty$  and perform the analysis in the spectral domain, whereas our  $K$  is assumed to be a fixed finite constant. As a result, their bound has better dependence of  $O(1/n)$  on  $n$ -the resolution of discretization, but does not go to 0 as  $n \to \infty$ . Ours have extra dependence on  $K$  and a slower rate of  $O(n^{-1/2})$  but our bounds go to 0 as  $n \to \infty$ . Maskey et al. (2023) obtains further results for graphons on unbounded domain. Ruiz et al. (2023b) studies the spectrum of sparser (but still  $\Theta(n^2)$  edges) graphons.
Other works use other notions than graph limits to obtain structural coherence. Levie et al. (2022) obtain a transferability result for spectral graph convolution networks via analysis in frequency domains. They sample finite graphs from general topologies as opposed to a graph limit. Their graph signals are assumed to have finite bandwidth while ours is only assumed to be in  $L^2$ . Their signal discretization scheme is assumed to be close to the continuous signals, while ours is proven to be so. Roddenberry et al. (2022) address sparse graphs and give a transferability bound between the loss functions of two random rooted graphs. However, the metric under which they derive their result is rather simple: if the two graphs are not isomorphic then their distance is constant, otherwise, they use the Euclidean metric between the two graph signals. This metric hence does not capture combinatorial, structural differences of functions on non-isomorphic graphs. To study transferability, Keriven et al. (2020) sample from standard random graph models, resulting in a bound of order  $O(n^{-1/2})$  for dense graph and  $O((\log n)^{-1/2})$  for relatively-sparse graphs. They do not cover bounded-degree graphs and their bounds hold with high probability. In general, having access to deterministic graph limit is considered a weaker assumption than having access to truly random graphs since large graphs satisfy some notion of 'almost randomness' (e.g. via Szemerédi regularity lemma). There are fascinating, tangible separations between graph limits and random graphs, especially among sparse graphs, that are outside the scope of this paper.
Adjacent to transferability studies, Bevilacqua et al. (2021) (also inspired by Lovász and Szegedy (2006)) uses induced homomorphism density to construct a graph representation that works across graph sizes and demonstrates empirical gains in using this larger representation. Maskey et al. (2022) uses graph limits to deduce generalization bounds for GNNs.
Notation Let  $\mathbb{N}$  be  $\{1,2,\ldots\}$  and write  $[n] = \{1,\dots,n\}$  for any  $n \in \mathbb{N}$ . For a scalar  $\alpha \in \mathbb{R}$  and a set  $S \subset \mathbb{R}$ , let  $\alpha S = \{\alpha s : s \in S\}$ . 'A.e.' stands for 'almost everywhere'.
For a measure space  $(\Omega, \mathcal{B}, \mu)$  and  $p \in [1, \infty]$ , denote by  $L^p(\Omega)$  the corresponding  $L^p$  function spaces with norm  $\| \cdot \|_p : f \mapsto (\int_{\Omega} |f|^p d\mu)^{1/p}$ . For any  $p, q \in [1, \infty]$ , define the operator norms  $\| \cdot \|_{p \to q} : A \mapsto \sup_{v \in L^\infty} \| vA \|_q / \| v \|_p$ .
For function spaces, we use  $\mathcal{F} = L^2([0,1])$  and  $\mathcal{F}_n = L^2([n]/n)$ , for any  $n \in \mathbb{N}$ . For any  $L^p$  space  $\mathcal{H}$ , denote by  $\mathcal{H}_{[-1,1]}$  the restriction to functions with range in  $[-1,1]$  a.e. and  $\mathcal{H}_{\mathrm{Lip}(L)}$  the restriction to functions that are  $L$ -Lipschitz a.e. and  $\mathcal{H}_{\mathrm{reg}(L)} = \mathcal{H}_{[-1,1]} \cap \mathcal{H}_{\mathrm{Lip}(L)}$ .
Graph neural networks (GNNs)  $\mathrm{GNNs}^2$  are functions that use graph convolutions to incorporate graph structure into neural network architectures. Given a finite graph  $G = (V,E)$  and a function  $X:V\to \mathbb{R}$  (called graph signal or node features), a GNN  $\Phi_F$  ( $F$  for 'finite') with  $L$  layers,  $n_i$  neurons at the  $i$ -th layer, nonlinearity  $\rho$  and learnable parameters  $h$ , is:
$$
\Phi_ {F} (h, G, X) = X _ {L} (h, G, X), \tag {1}
$$
$$
\left[ X _ {l} (h, G, X) \right] _ {f} = \rho \left(\sum_ {g = 1} ^ {n _ {l - 1}} A _ {l, f, g} (h, G) \left[ X _ {l - 1} \right] _ {g}\right), \quad l \in [ L ], f \in [ n _ {l} ] \tag {2}
$$
$$
X _ {0} (h, G, X) = X, \tag {3}
$$
where  $[X_l]_f$  is the output of the  $f$ -th neuron in the  $l$ -th layer, which is another graph signal. The input graph information is captured through order  $K$  graph convolutions  $A_{l,f,g}(h,G) \coloneqq \sum_{k=0}^{K} h_{l,f,g,k} GSO(G)^k$ , where  $GSO(G)$  is a graph shift operator corresponding to  $G$  — popular examples include the adjacency matrix or the Laplacian (Kipf and Welling, 2017; Levie et al., 2022). The power notation is the usual matrix power, while the notation  $h_{l,f,g,k}$  highlights that there is a learnable parameter for each convolution order  $k$ , between each neuron  $f$  and  $g$  from layer  $l-1$  to layer  $l$  of the neural network. Thus, the number of learnable parameters in a GNN does not depend on the number of vertices of the graph.
2.1 Graph limits
Graph limit theory involves embedding discrete graphs into rich topological or geometric spaces and studying the behavior of convergent (e.g. in size) graph sequences.
Dense graphs A popular example of graph limits are graphons - symmetric  $L^1([0,1]^2)$  (Lebesgue-measurable) functions whose value at  $(x,y)$  can be thought of (intuitively) as the weight of the  $xy$ -edge in a graph with vertices in  $[0,1]$ . Convergence in this space, under the cut metric (see Appendix A for the exact definition), is dubbed dense graph convergence because for any  $W \in L^1([0,1]^2)$ ,  $\|W\|_{\square} = 0$  iff  $W = 0$  outside a set of Lebesgue measure 0. This implies that graphs with a subquadratic number of edges, such as grids or hypercubes, are identified with the empty graph in the cut norm. Dense graph convergence is very well understood theoretically and is the basis for recent work on GNN limits (Ruiz et al., 2023a).
Sparse graphs Graphing (Lovasz, 2012), is a direct counterpart of a graphon for sparse graphs. Recall that graphons are not suitable for sparse graphs because the Lebesgue measure on  $L^2([0,1]^2)$  is not fine enough to detect edges of bounded-degree graphs. Therefore, one solution is to consider other measure spaces. Graphings are quadruples  $(V, \mathcal{A}, \lambda, E)$  where  $V$  and  $E$  are interpreted as the usual vertex and edge sets and  $(V, \mathcal{A}, \lambda)$  together form a Borel measure such that  $E$  is in  $\mathcal{A} \times \mathcal{A}$  satisfying a symmetry condition. While Lebesgue measures are constructed from a specific topology of open sets on  $\mathbb{R}$ , for graphings, we are allowed the freedom to choose a different topological structure (for instance a local topology) on  $V$ . The definition of graphings is theoretically elegant but harder to work with since the topological structures are stored in the  $\sigma$ -algebra.
Furthermore, a famous open conjecture by Aldous and Lyons (2007) asks whether all graphings are weak local limits of some sequence of bounded-degree graphs. The unresolved conjecture of Aldous and Lyons means that one cannot simply take an arbitrary graphing and be guaranteed a finite bounded-degree graph sequence converging to said graphing, which is the main approach in Ruiz et al. (2023a) for dense graphs. As a result, we expect
2The architecture is known as graph convolutional networks, but we call them GNNs for brevity.
some regularity assumptions on the graph sequence for any work that handles sparse graph limits, unless the authors try to tackle this conjecture itself. A self-contained summary of graphings within the scope of this paper is provided in Appendix C. Infinite paths and cycles also have nice descriptions in terms of graphings (also in Appendix C), which we will use in our constructions for Lemma 2.
2.2 Graphops and comparing across graph sizes
More recently, Backhausz and Szegedy (2022) approach graph limits from the viewpoint of limits of operators, called graphops. This viewpoint is straightforward for finite graphs: both the adjacency matrix and Laplacian, each defining a unique graph, are linear operators on  $\mathbb{R}^{\# \text{vertices}}$ . Moreover, viewing graphs as operators is exactly what we do with GSOs and graph convolutions. Hence, graphop seems to be an appropriate tool to study GNN approximation and transferability. On the other hand, there are challenges with this approach: being related to graphings, they inherit some of graphings' limitations, such as the conjecture of Aldous and Lyons (2007) and discontinuity of eigenvalues at the limit (Appendix C.1). Moreover, to understand GNN transferability from size  $m$  to  $n$ , one needs to compare an  $m \times m$  matrix with an  $n \times n$  matrix, which is nontrivial. This is done by comparing their actions on  $\mathbb{R}^m$  versus  $\mathbb{R}^n$ . It turns out that these actions, under an appropriate metric, define a special mode of operator convergence called action convergence. The resulting limit objects are well-defined and nontrivial for sparse graphs and intermediate graphs, while also generalizing dense graphs limits. We will describe this mode of convergence, the corresponding metric, and our own relaxation of it later in this section.
We now describe how graphs of different sizes can be compared through the actions of their corresponding operators on some function spaces.
Nonlinear (not necessarily linear)  $P$ -operators For an  $n$ -vertex graph, its adjacency matrix, Laplacian, or random walks kernels are examples of operators on  $L^p([n]/n)$ . To formally generalize to the infinite-vertex case, Backhausz and Szegedy (2022) use  $P$ -operators, which are linear operators from  $L^\infty(\Omega)$  to  $L^1(\Omega)$  with finite  $\|A\|_{\infty \to 1}$ . In this paper, we further assume they have finite  $\| \cdot \|_{2 \to 2}$  norm but are not necessarily linear. This allows us to consider nonlinear GNN layers and the whole GNN itself in the same framework.
Graphops  $P$ -operators lead to a notion of graph limit that applies to both dense and sparse graphs. Graphops (Backhausz and Szegedy, 2022) are positivity-preserving<sup>3</sup>, self-adjoint  $P$ -operators. Adjacency matrices of finite graphs, graphons (Lovász and Szegedy, 2006), and graphings (Lovász, 2012) are all examples of graphops.
$(k,L)$ -profile of a nonlinear  $P$ -operator Actions of graphops are formally captured through their  $(k,L)$ -profiles, and these will be useful to compare different graphops. Pick  $k \in \mathbb{N}$ ,  $L \in [0,\infty]$  and  $A$  a  $P$ -operator on  $(\Omega, \mathcal{B}, \mu)$ . Intuitively, we will take  $k$  samples from the space our operators act on, apply our operator to get  $k$  images, and concatenate samples and images into a joint distribution on  $\mathbb{R}^{2k}$ , which gives us one element of the profile. For instance, for  $n$ -vertex graphs, the concatenation results in a matrix  $M \in \mathbb{R}^{n \times 2k}$ , so each joint distribution is a sum (over rows of  $M$ ) of  $n$  Dirac distributions. In the limit, the number of atoms in each element of the profile increases, and the measure converges (weakly) to one with density. More formally, denote by  $\mathcal{D}(v_1, \ldots, v_k)$  the pushforward of  $\mu$  via  $x \mapsto (v_1(x), \ldots, v_k(x))$  for any tuple  $(v_i)_{i \in [k]} \in L^2(\Omega)$ . The  $(k,L)$ -profile of  $A$  is:
$$
\mathcal {S} _ {k, L} (A) := \left\{\mathcal {D} \left(v _ {1}, \dots , v _ {k}, A v _ {1}, \dots , A v _ {k}\right): v _ {i} \in L _ {\operatorname {r e g} (L)} ^ {\infty} (\Omega), i = 1 \dots k \right\}. \tag {4}
$$
Formally, denote by  $\mathcal{P}(\mathbb{R}^k)$  the set of Borel probability distributions over  $\mathbb{R}^{2k}$ . Regardless of the initial graph size, or the space on which the operators act,  $(k,L)$ -profiles of  $A$  are always some subsets of  $\mathcal{P}(\mathbb{R}^{2k})$  which allow us to compare operators acting on different spaces.
Convergence of  $P$ -operators We compare two profiles (closed subsets  $X, Y \subset \mathcal{P}(\mathbb{R}^{2k})$ ) via a Hausdorff metric  $d_H(X, Y) \coloneqq \max(\sup_{x \in X} \inf_{y \in Y} d_{LP}(x, y), \sup_{y \in Y} \inf_{x \in X} d_{LP}(x, y))$ .
3action on positive functions results in positive functions. This condition can be swapped out for 'positiveness'  $(\langle Ax,x\rangle >0,\forall x\in \mathrm{Dom}(A)\backslash \{0\})$  to allow for Laplacians.
Here,  $d_{LP}$  is the Lévy-Prokhorov metric on  $\mathcal{P}(\mathbb{R}^{2k})$  (see exact definition in Appendix A), which metrizes weak convergence of Borel probability measures, and translates action convergence to weak convergence of measures. Finally, given any two  $P$ -operators  $A, B$ , we can compare their profiles across all different  $k$  at the same time as
$$
d _ {M} (A, B) := \sum_ {k = 1} ^ {\infty} 2 ^ {- k} d _ {H} \left(\mathcal {S} _ {k, L} (A), \mathcal {S} _ {k, L} (B)\right). \tag {5}
$$
Intuitively, we allow  $d_H$  to grow subexponentially in  $k$  by the scaling  $2^{-k}$ . Our definition of profile slightly differs from that of Backhausz and Szegedy (2022), using  $L_{\mathrm{reg}(L)}^{\infty}$  instead of their  $L_{[-1,1]}^{\infty}$ . However, we will justify this deviation in Section 4.4, Theorem 4: by letting  $L$  grow slowly in  $n$ , we recover the original limits in Backhausz and Szegedy (2022).
This action convergence turns out to be one of the 'right' notions of convergence that capture both sparse and dense graph limits, as well as some intermediate density graphs:
Theorem 1 (Theorem 1.1 Backhausz and Szegedy (2022)). Convergence under  $d_M$  is equivalent (results in the same limit) to dense graph convergence when restricted to graphons and equivalent to local-global convergence when restricted to graphings.
3 Graphop neural networks
Graph limits allow us to lift finite graphs onto the richer space of graphops to discuss convergent graph sequences  $G_{i} \to G$ . For finite GNNs (Eqn (2)), fixing the graph input  $G_{i}$  and learnable parameter  $h$  results in a function  $\Phi_F(h,G_i,\cdot)$  that transforms the input graph signal (node features) into an output graph signal. The transferability question asks how similar  $\Phi_F(h,G_i,\cdot)$  is to  $\Phi_F(h,G_j,\cdot)$  for some  $i \neq j$ . In our approach using approximation theory, we will compare both functions to the limiting function on  $G$ . This is done by an appropriate lift of the GNN onto a larger space that we call graphop neural networks.
We then introduce a discretization scheme of graphop neural networks to obtain finite GNNs, similar to graphon sampling (Ruiz et al., 2023a) and sampling from topological spaces (Levie et al., 2022). Finally, Lemma 1 asserts that, restricted to self-adjoint  $P$ -operators, discretizations of graphops are indeed graph shift operators (GSOs).
3.1 Convolution and graphop neural networks