Title: Limits, approximation and size transferability for GNNs on sparse graphs via graphops
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural networks

convolution

graph limits

size transferability
Abstract: Can graph neural networks generalize to graphs that are different from the graphs they were trained on, e.g., in size? In this work, we study this question from a theoretical perspective. While recent work established such transferability and approximation results via graph limits, e.g., via graphons, these only apply nontrivially to dense graphs. To include frequently encountered sparse graphs such as bounded-degree or power law graphs, we take a perspective of taking limits of operators derived from graphs, such as the aggregation operation that makes up GNNs. This leads to the recently introduced limit notion of graphops (Backhausz and Szegedy, 2022). We demonstrate how the operator perspective allows us to develop quantitative bounds on the distance between a finite GNN and its limit on an infinite graph, as well as the distance between the GNN on graphs of different sizes that share structural properties, under a regularity assumption verified for various graph sequences. Our results hold for dense and sparse graphs, and various notions of graph limits.

Limits, approximation and size transferability for GNNs on sparse graphs via graphops

Abstract

Can graph neural networks generalize to graphs that are different from the graphs they were trained on, e.g., in size? In this work, we study this question from a theoretical perspective. While recent work established such transferability and approximation results via graph limits, e.g., via graphons, these only apply nontrivially to dense graphs. To include frequently encountered sparse graphs such as bounded-degree or power law graphs, we take a perspective of taking limits of operators derived from graphs, such as the aggregation operation that makes up GNNs. This leads to the recently introduced limit notion of graphops (Backhausz and Szegedy, 2022). We demonstrate how the operator perspective allows us to develop quantitative bounds on the distance between a finite GNN and its limit on an infinite graph, as well as the distance between the GNN on graphs of different sizes that share structural properties, under a regularity assumption verified for various graph sequences. Our results hold for dense and sparse graphs, and various notions of graph limits.

1 Introduction

Since the advent of graph neural networks (GNNs), deep learning has become one of the most promising tools to address graph-based tasks (Gilmer et al., 2017; Scarselli et al., 2009; Kipf and Welling, 2017; Bronstein et al., 2017). Following the mounting success of applied GNN research, theoretical analyses follow with many works studying GNNs' representational power (Azizian and Lelarge, 2021; Morris et al., 2019; Xu et al., 2019, 2020; Garg et al., 2020; Chen et al., 2020; Maron et al., 2019; Loukas, 2020a,b; Abboud et al., 2021).

A hitherto less addressed question of practical importance is the possibility of size generalization, i.e., transferring a learned GNN to graphs of different sizes (Ruiz et al., 2023a; Levie et al., 2022; Xu et al., 2021; Yehudai et al., 2021; Bevilacqua et al., 2021; Chuang and Jegelka, 2022; Roddenberry et al., 2022; Maskey et al., 2022, 2023), especially for sparse graphs. For instance, it would be computationally desirable to train a GNN on small graphs and apply it to large graphs. This question is also important to judge the reliability of the learned model on different test graphs. To answer the size generalization question, we need to understand under which conditions such transferability is possible - since it may not always be possible (Xu et al., 2021; Yehudai et al., 2021; Jegelka, 2022) - and what output perturbations we may expect. For a formal analysis of perturbations and conditions, we need a suitable graph representation that captures inductive biases and allows us to compare models for graphs of different sizes. Graph limits can help to formalize this, as they help understand biases as the graph size tends to infinity.

Formally, approximation theory asks for bounds between a GNN on a finite graph and its infinite counterpart, while transferability compares model outputs on graphs of different sizes.

The quality of the bounds depends on how the two GNNs (and corresponding graphs) are intrinsically linked, in particular, to what extent they share relevant structure. This yields conditions for size generalization. For example, the graphs could be sampled from the same graph limit (Ruiz et al., 2023a) or from the same random graph model (Keriven et al., 2020).

In particular, Ruiz et al. (2023a) study approximation and transferability via the lens of graphons (Lovász, 2012; Lovász and Szegedy, 2006), which characterize the limits of dense graphs. Yet, many real-world graphs are not dense, for instance, planar traffic networks, power law graphs, polymer graphs, Hamming graphs (including hypercubes for error-correcting code), or grid-like graphs e.g., for images. For sparser graphs, the correct notion of limit suitable for deep learning is still an open problem, as typical bounded-degree graph limits such as the Benjamini-Schramm limit of random rooted graphs (Benjamini and Schramm, 2001), or graphings (Lovász, 2012) are less well understood and often exhibit pathological behaviors (see Section 2.1). Limits of intermediate graphs, such as the hypercubes, are even more obscure. Hence, understanding limits, inductive biases and transferability of GNNs for sparse graphs remains an open problem in understanding graph representation learning.

This question is the focus of this work. To obtain suitable graph limits for sparse graphs and to be able to compare GNNs on graphs of different sizes while circumventing challenges of sparse graph limits, we view a graph as an operator derived from it. This viewpoint is naturally compatible with GNNs, as they are built from convolution/aggregation operations. We show how the operator perspective allows us to define limits of GNNs of infinite sequences of graphs. We achieve this by exploiting the recently defined notion of graphop , which generalizes graph shift operators, and the action convergence defined in the space of graphops (Backhausz and Szegedy, 2022). Our definition of GNN limits enables us to prove rigorous bounds for approximation and transferability of GNNs for sparse graphs. Since graphops encompass both graphons and graphings, we generalize similar bounds for graphon neural networks (Ruiz et al., 2023a; Maskey et al., 2023) to a much wider set of graphs.

Yet, using graphops requires technical work. For instance, we need to introduce an appropriate discretization of a graphop to obtain its corresponding finite graph shift operators. We use these operators to define a generalized graphop neural network that acts as a limit object, with discretizations that become finite GNNs. Then we prove approximation and transferability results for both the operators (graphops and their discretizations) and GNNs.

Contributions. To the best of our knowledge, this is the first paper to provide approximation and transferability theorems specifically for sparse graph limits. Our main tool, graphops, has not been used to study GNNs before, although viewing graphs as operators is a classic theme in the literature. Our specific contributions are as follows:

We define a graphop convolution , i.e., an operator that includes both finite graph convolutions and a limit version that allows us to define a limit object for GNNs applied to graphs of size n →∞ .

We rigorously prove an approximation theorem (Theorem 2) that bounds a distance between a graphop A (acting on infinite-dimensional space) and and appropriate discretization A n (acting on R n ), in the d M metric introduced by Backhausz and Szegedy (2022). Our result applies to a more general set of nonlinear operators, and implies a transferability bound between finite graphs (discretizations) of different sizes.

For neural networks, we present a quantitative approximation and transferability bound that guarantees outputs of graphop neural networks are close to those of the corresponding GNNs (obtained from discretization).

1.1 Related work

A summary of comparisons between our framework and related papers is in Table 1.

In structure, the closest related work is (Ruiz et al., 2023a), which derives approximation and transferability theorems for graphon neural networks, i.e., dense graphs. For graphons, the convolution kernel has a nice spectral decomposition, which is exploited by Ruiz et al.

(2023a). In contrast, sparse graph limits are not known to enjoy nice convergence of the spectrum (Backhausz and Szegedy, 2022; Aldous and Lyons, 2007), also Appendix C.1, so we need to use different techniques. Since the notion of graphop generalizes both dense graph limits and certain sparse graph limits, our results apply to dense graphs as well. Our assumptions and settings are slightly different from Ruiz et al. (2023a). For instance, they allow the convolution degree K → ∞ and perform the analysis in the spectral domain, whereas our K is assumed to be a fixed finite constant. As a result, their bound has better dependence of O (1 /n ) on n -the resolution of discretization, but does not go to 0 as n →∞ . Ours have extra dependence on K and a slower rate of O ( n -1 / 2 ) but our bounds go to 0 as n →∞ . Maskey et al. (2023) obtains further results for graphons on unbounded domain. Ruiz et al. (2023b) studies the spectrum of sparser (but still Θ( n 2 ) edges) graphons.

Other works use other notions than graph limits to obtain structural coherence. Levie et al. (2022) obtain a transferability result for spectral graph convolution networks via analysis in frequency domains. They sample finite graphs from general topologies as opposed to a graph limit. Their graph signals are assumed to have finite bandwidth while ours is only assumed to be in L 2 . Their signal discretization scheme is assumed to be close to the continuous signals, while ours is proven to be so. Roddenberry et al. (2022) address sparse graphs and give a transferability bound between the loss functions of two random rooted graphs. However, the metric under which they derive their result is rather simple: if the two graphs are not isomorphic then their distance is constant, otherwise, they use the Euclidean metric between the two graph signals. This metric hence does not capture combinatorial, structural differences of functions on non-isomorphic graphs. To study transferability, Keriven et al. (2020) sample from standard random graph models, resulting in a bound of order O ( n -1 / 2 ) for dense graph and O ((log n ) -1 / 2 ) for relatively-sparse graphs. They do not cover boundeddegree graphs and their bounds hold with high probability. In general, having access to deterministic graph limit is considered a weaker assumption than having access to truly random graphs since large graphs satisfy some notion of 'almost randomness' (e.g. via Szemerédi regularity lemma). There are fascinating, tangile separations between graph limits and random graphs, especially among sparse graphs, that are outside the scope of this paper.

Adjacent to transferability studies, Bevilacqua et al. (2021) (also inspired by Lovász and Szegedy (2006)) uses induced homomorphism density to construct a graph representation that works across graph sizes and demonstrates empirical gains in using this larger representation. Maskey et al. (2022) uses graph limits to deduce generalization bounds for GNNs.

2 Background

Notation Let N be { 1 , 2 , . . . } and write [ n ] = { 1 , . . . , n } for any n ∈ N . For a scalar α ∈ R and a set S ⊂ R , let αS = { αs : s ∈ S } . 'A.e. ' stands for 'almost everywhere'.

For a measure space (Ω , B , µ ) and p ∈ [1 , ∞ ], denote by L p (Ω) the corresponding L p function spaces with norm ‖·‖ p : f ↦→ ( ∫ Ω | f | p dµ ) 1 /p . For any p, q ∈ [1 , ∞ ], define the operator norms ‖ · ‖ p → q : A ↦→ sup v ∈ L ∞ ‖ vA ‖ q / ‖ v ‖ p .

For function spaces, we use F = L 2 ([0 , 1]) and F n = L 2 ([ n ] /n ), for any n ∈ N . For any L p space H , denote by H [ -1 , 1] the restriction to functions with range in [ -1 , 1] a.e. and H Lip( L ) the restriction to functions that are L -Lipschitz a.e. and H reg( L ) = H [ -1 , 1] ∩ H Lip( L ) .

Graph neural networks (GNNs) GNNs 2 are functions that use graph convolutions to incorporate graph structure into neural network architectures. Given a finite graph G = ( V, E ) and a function X : V → R (called graph signal or node features ), a GNN Φ F ( F for 'finite') with L layers, n i neurons at the i -th layer, nonlinearity ρ and learnable parameters h , is:

where [ X l ] f is the output of the f -th neuron in the l -th layer, which is another graph signal. The input graph information is captured through order K graph convolutions A l,f,g ( h, G ) := ∑ K k =0 h l,f,g,k GSO ( G ) k , where GSO ( G ) is a graph shift operator corresponding to G - popular examples include the adjacency matrix or the Laplacian (Kipf and Welling, 2017; Levie et al., 2022). The power notation is the usual matrix power, while the notation h l,f,g,k highlights that there is a learnable parameter for each convolution order k , between each neuron f and g from layer l -1 to layer l of the neural network. Thus, the number of learnable parameters in a GNN does not depend on the number of vertices of the graph.

2.1 Graph limits

Graph limit theory involves embedding discrete graphs into rich topological or geometric spaces and studying the behavior of convergent (e.g. in size) graph sequences.

Dense graphs A popular example of graph limits are graphons - symmetric L 1 ([0 , 1] 2 ) (Lebesgue-measurable) functions whose value at ( x, y ) can be thought of (intuitively) as the weight of the xy -edge in a graph with vertices in [0 , 1]. Convergence in this space, under the cut metric (see Appendix A for the exact definition), is dubbed dense graph convergence because for any W ∈ L 1 ([0 , 1] 2 ) , ‖ W ‖ □ = 0 iff W = 0 outside a set of Lebesgue measure 0. This implies that graphs with a subquadratic number of edges, such as grids or hypercubes, are identified with the empty graph in the cut norm. Dense graph convergence is very well understood theoretically and is the basis for recent work on GNN limits (Ruiz et al., 2023a).

Sparse graphs Graphing (Lovász, 2012), is a direct counterpart of a graphon for sparse graphs. Recall that graphons are not suitable for sparse graphs because the Lebesgue measure on L 2 ([0 , 1] 2 ) is not fine enough to detect edges of bounded-degree graphs. Therefore, one solution is to consider other measure spaces. Graphings are quadruples ( V, A , λ, E ) where V and E are interpreted as the usual vertex and edge sets and ( V, A , λ ) together form a Borel measure such that E is in A×A satisfying a symmetry condition. While Lebesgue measures are constructed from a specific topology of open sets on R , for graphings, we are allowed the freedom to choose a different topological structure (for instance a local topology ) on V . The definition of graphings is theoretically elegant but harder to work with since the topological structures are stored in the σ -algebra.

Furthermore, a famous open conjecture by Aldous and Lyons (2007) asks whether all graphings are weak local limits of some sequence of bounded-degree graphs. The unresolved conjecture of Aldous and Lyons means that one cannot simply take an arbitrary graphing and be guaranteed a finite bounded-degree graph sequence converging to said graphing, which is the main approach in Ruiz et al. (2023a) for dense graphs. As a result, we expect

some regularity assumptions on the graph sequence for any work that handles sparse graph limits, unless the authors try to tackle this conjecture itself. A self-contained summary of graphings within the scope of this paper is provided in Appendix C. Infinite paths and cycles also have nice descriptions in terms of graphings (also in Appendix C), which we will use in our constructions for Lemma 2.

2.2 Graphops and comparing across graph sizes

More recently, Backhausz and Szegedy (2022) approach graph limits from the viewpoint of limits of operators, called graphops . This viewpoint is straightforward for finite graphs: both the adjacency matrix and Laplacian, each defining a unique graph, are linear operators on R #vertices . Moreover, viewing graphs as operators is exactly what we do with GSOs and graph convolutions. Hence, graphop seems to be an appropriate tool to study GNN approximation and transferability. On the other hand, there are challenges with this approach: being related to graphings, they inherit some of graphings' limitations, such as the conjecture of Aldous and Lyons (2007) and discontinuity of eigenvalues at the limit (Appendix C.1). Moreover, to understand GNN transferability from size m to n , one needs to compare an m × m matrix with an n × n matrix, which is nontrivial. This is done by comparing their actions on R m versus R n . It turns out that these actions, under an appropriate metric, define a special mode of operator convergence called action convergence . The resulting limit objects are well-defined and nontrivial for sparse graphs and intermediate graphs, while also generalizing dense graphs limits. We will describe this mode of convergence, the corresponding metric, and our own relaxation of it later in this section.

We now describe how graphs of different sizes can be compared through the actions of their corresponding operators on some function spaces.

Nonlinear (not necessarily linear) P -operators For an n -vertex graph, its adjacency matrix, Laplacian, or random walks kernels are examples of operators on L p ([ n ] /n ). To formally generalize to the infinite-vertex case, Backhausz and Szegedy (2022) use P -operators , which are linear operators from L ∞ (Ω) to L 1 (Ω) with finite ‖ A ‖ ∞→ 1 . In this paper, we further assume they have finite ‖ · ‖ 2 → 2 norm but are not necessarily linear. This allows us to consider nonlinear GNN layers and the whole GNN itself in the same framework.

Graphops P -operators lead to a notion of graph limit that applies to both dense and sparse graphs. Graphops (Backhausz and Szegedy, 2022) are positivity-preserving 3 , self-adjoint P -operators. Adjacency matrices of finite graphs, graphons (Lovász and Szegedy, 2006), and graphings (Lovász, 2012) are all examples of graphops.

( k, L ) -profile of a nonlinear P -operator Actions of graphops are formally captured through their ( k, L )-profiles, and these will be useful to compare different graphops. Pick k ∈ N , L ∈ [0 , ∞ ] and A a P -operator on (Ω , B , µ ). Intuitively, we will take k samples from the space our operators act on, apply our operator to get k images, and concatenate samples and images into a joint distribution on R 2 k , which gives us one element of the profile. For instance, for n -vertex graphs, the concatenation results in a matrix M ∈ R n × 2 k , so each joint distribution is a sum (over rows of M ) of n Dirac distributions. In the limit, the number of atoms in each element of the profile increases, and the measure converges (weakly) to one with density. More formally, denote by D ( v 1 , . . . , v k ) the pushforward of µ via x ↦→ ( v 1 ( x ) , . . . , v k ( x )) for any tuple ( v i ) i ∈ [ k ] ∈ L 2 (Ω). The ( k, L ) -profile of A is:

Formally, denote by P ( R k ) the set of Borel probability distributions over R 2 k . Regardless of the initial graph size, or the space on which the operators act, ( k, L )-profiles of A are always some subsets of P ( R 2 k ) which allow us to compare operators acting on different spaces.

Convergence of P -operators We compare two profiles (closed subsets X,Y ⊂ P ( R 2 k )) via a Hausdorff metric d H ( X,Y ) := max(sup x ∈ X inf y ∈ Y d LP ( x, y ) , sup y ∈ Y inf x ∈ X d LP ( x, y )) .

Here, d LP is the Lévy-Prokhorov metric on P ( R 2 k ) (see exact definition in Appendix A), which metrizes weak convergence of Borel probability measures, and translates action convergence to weak convergence of measures. Finally, given any two P -operators A,B , we can compare their profiles across all different k at the same time as

Intuitively, we allow d H to grow subexponentially in k by the scaling 2 -k . Our definition of profile slightly differs from that of Backhausz and Szegedy (2022), using L ∞ reg( L ) instead of their L ∞ [ -1 , 1] . However, we will justify this deviation in Section 4.4, Theorem 4: by letting L grow slowly in n , we recover the original limits in Backhausz and Szegedy (2022).

This action convergence turns out to be one of the 'right' notions of convergence that capture both sparse and dense graph limits, as well as some intermediate density graphs:

Theorem 1 (Theorem 1.1 Backhausz and Szegedy (2022)) . Convergence under d M is equivalent (results in the same limit) to dense graph convergence when restricted to graphons and equivalent to local-global convergence when restricted to graphings.

3 Graphop neural networks

Graph limits allow us to lift finite graphs onto the richer space of graphops to discuss convergent graph sequences G i → G . For finite GNNs (Eqn (2)), fixing the graph input G i and learnable parameter h results in a function Φ F ( h, G i , · ) that transforms the input graph signal (node features) into an output graph signal. The transferability question asks how similar Φ F ( h, G i , · ) is to Φ F ( h, G j , · ) for some i = j . In our approach using approximation theory, we will compare both functions to the limiting function on G . This is done by an appropriate lift of the GNN onto a larger space that we call graphop neural networks .

/negationslash

We then introduce a discretization scheme of graphop neural networks to obtain finite GNNs, similar to graphon sampling (Ruiz et al., 2023a) and sampling from topological spaces (Levie et al., 2022). Finally, Lemma 1 asserts that, restricted to self-adjoint P -operators, discretizations of graphops are indeed graph shift operators (GSOs).

3.1 Convolution and graphop neural networks

Similar to how GSOs in a GNN act on graph signals, graphops act on some L 2 signals (called graphop signals ). The generalization is straightforward: replacing GSOs in the construction of the GNN in Eqn. (2) with graphops results in graphop convolution and replacing graph convolution with graphop convolution gives graphop neural networks .

Formally, fix a maximum order K ∈ N . For some measure space (Ω , B , µ ), select a graphop A : L 2 (Ω) → L 2 (Ω) and a graphop signal X ∈ L 2 (Ω). We define a graphop convolution operator as a weighted sum of at most K -1 applications of A : H ( h, A )[ X ] := ∑ K -1 k =0 ( h k A k )[ X ], where h ∈ R K are (learnable) filter parameters and A k is the composition of k duplicates of A . The square bracket [ v ] i indicates the i -th entry of a tuple v .

For some number of layers L ∈ N , { n i } i ∈ [ L ] ∈ N , n 0 := 1, define a graphop neural network Φ with L layers and n i features in layer i as:

with filter parameter tuple h = ( h 1 , . . . , h L ), h l ∈ ( R K ) n l × n l -1 for any l ∈ [ L ], and graphop signal tuple X l ∈ ( L 2 (Ω)) n l for any l ∈ [ L ] ∪ { 0 } . Eqn (7) and Eqn (2) are almost identical, with the only difference being the input/output space: graphops replacing finite graphs, and graphop signals replacing graph signals.

3.2 From graphop neural networks to finite graph neural networks

We are specifically interested in finite GNNs that are discretizations of a graphop (for instance finite grids as discretizations of infinite grids), so as to obtain a quantitative bound that depends on the resolution of discretization. To sample a GNN from a given graphop A : F → F , we first sample a GSO and plug it into Eqn (2). Choose a resolution m ∈ N and define the GSO A m , for any graph signal X ∈ F m (defined in Section 2) as:

where graphop signal ˜ X ∈ F is an extension of graph signal X ∈ F m defined as

Shared structural properties of sampled graphs The discretization scheme introduced for graphop can be intuitively understood as partitioning the vertex set into finitely many sets and merging nodes and their connections in these sets, with an appropriate scaling of the edge weights. Imagine blurring an n × n matrix into an n/ 2 × n/ 2 matrix by average-pooling over each disjoint 2 × 2 square. As n →∞ (and even for uncountable [0 , 1]), this makes rigorous the notion of making a high-resolution graph on a huge number of vertices more 'blurry' by merging nodes in a way that still maintains some smoothness conditions, which is reminiscent of the real-world procedures of training with low-resolution images before fine-tuning with higher-resolution ones, or sampling from low-frequency graph Fourier transform domain.

Note that if A is linear then A m is necessarily linear, but our definition of graphop does not require linearity. Therefore, A m is strictly more general than the matrix representation of graph shift operators. We have the following well-definedness result:

Lemma 1. If a graphop A : F → F is self-adjoint, then for each resolution m ∈ N , the discretization A m : F m →F m defined above is also self-adjoint.

The proof can be found in Appendix B. Compared to previous works, our discretization scheme in Eqn (9) looks slightly different. In Ruiz et al. (2023a), given a graphon W : [0 , 1] 2 → R , the discretization at resolution n was defined by forming the matrix S ∈ R n × n : S i,j = W ( i/n, j/n ). A related discretization scheme involving picking the interval endpoints at random was also used, but the resulting matrix still takes values at discrete points in W . These two sampling schemes rely crucially on their everywhere continuous assumptions for the graphon W . Indeed, but for continuity requirements, two functions that differ only at finite discrete points ( i/n, j/n ) , i, j ∈ [ n ] are in the same L 2 class of functions, but will give rise to completely different samples. Furthermore, not every L 2 class of functions has a continuous representative. This means that our discretization scheme is strictly more general than that used by Ruiz et al. (2023a) even when restricted to graphons. This difference comes from the fact that we are discretizing an operator and not the graph itself. For our purpose, taking values at discrete points for some limiting object of sparse graphs will likely not work, since sparsity ensures that most discrete points are trivial.

4 Main result: Approximation and transferability

4.1 Results for P -operators

Our first set of theorems address approximation and transferability of P -operators: under certain regularity assumptions to be discussed later, P -operators are well approximated by their discretizations:

Theorem 2 (Approximation theorem) . Let A : F → F be a P -operator satisfying Assumption 2 with constant C A ; Assumption 3.A or 3.B with resolutions in N . Fix n ∈ N and consider ( k, C v ) -profiles. Let A n : F n →F n be a discretization of A as defined in Eqn (9) . Then:

Compared to theorems in Ruiz et al. (2023a), our explicit dependence on n has an extra n -1 / 2 term that stems from techniques used to bound the Lévy-Prokhorov distance between two entry distributions obtained from functions that differ by at most O ( n -1 ) in L 2 norm.

As an immediate corollary, invoking the triangle inequality yields a transferability bound.

Corollary 1 (Transferability) . Let A : F → F be a P -operator satisfying assumptions of Theorem 2 with constant C A and resolutions N . For any n, m ∈ N , let A n : F n →F n and A m : F m →F m be discretizations as defined in Eqn (9) . Then:

We emphasize that these theorems work for general nonlinear P -operators and not only the linear graphops defined in (Backhausz and Szegedy, 2022).

Proof sketch The full proof of Theorem 2 is in Appendix D. To bound the distance in d M between two operators, for each sample size k ∈ N , we give a bound on the Hausdorff metric d H between the two ( k, C v )-profiles. As long as the dependence on k of these bounds is polynomial, the infinite sum in the definition of d M converges. We do this by picking an arbitrary distribution η from S k,C v ( A ), which by definition is given by a k -tuple F of functions in L ∞ reg( C v ) . Discretize each element of F and consider its entry distribution results in η n ∈ S k,C v ( A n ). We show that we can give an upper bound of d LP ( η, η n ) that is independent of the choice of η and thus same upper bound holds for sup η ∈S k,Cv ( A ) inf η n ∈S k,Cv ( A n ) d LP ( η, η n ). By also selecting an arbitrary element of S k,C v ( A n ) and extending it to an element of S k,C v ( A ), we obtain another upper bound for sup η n ∈S k,Cv ( A n ) inf η ∈S k,Cv ( A ) d LP ( η, η n ) and thus for d H . The different assumptions come in via different techniques used to bound d LP by a high probability bound on the L 2 norm of the functions in F and their discretization/extension.

4.2 Results for graphop neural networks

Not only are graphops and their discretizations close in d M , but, as we show next, neural networks built from a graphop are also close to those built from graphop discretizations in d M . We iterate that here we are comparing nonlinear operators (graphop neural networks) that are acting on different spaces ( L 2 ([ n ] /n ) for some finite n versus L 2 ([0 , 1])).

Before stating theoretical guarantees for graphop neural networks, let us introduce some assumptions on the neural network activation function and parameters:

Assumption 1. Let the activation function ρ : R → R in the definition of graphop neural networks be 1 -Lipschitz and the convolution parameters h be such that | h | ≤ 1 element-wise.

Theorem 3 (Graphop neural network discretization) . Let A : F → F . Assume that A satisfies Assumption 2 with constant C A and Assumption 3.A or 3.B with resolutions in N . Fix n ∈ N and consider ( k, C v ) -profiles. Under Assumption 1, we have:

where C A := ( n max ∑ K i =1 C i A ) L , n max = max l ∈ [ L ] n l , and P 1 is a constant depending on K,L . Furthermore, we can invoke the triangle inequality to compare outputs of graphop neural networks built from two different discretizations of A . For any m,n ∈ N ,

Compared to the main theorems of Ruiz et al. (2023a), there are two main differences in our results. First, our rate of O ( n -1 / 2 ) is slower than the rate of O ( n -1 ) in Ruiz et al. (2023a) as a function of n . Yet, second, their bounds contain a small term that is independent of n and does not go to 0 as n goes to infinity. This small term depends on the variability of small eigenvalues in the spectral decomposition of the convolution operator associated with

0

-1

工

-2

d

0

-3

-4

log(1/resolution)

-5

log(1/sqrt(resolution))

1.0

1.5

2.0

2.5

3.0

3.5

logresolution

a graphon. The bound in Theorem 3, in contrast, goes to zero. We tested this bound in Figure 1 for GNNs on polymer graphs, which suggest that O ( n -1 ) rate may be possible.

The proof for this theorem is in Appendix D.3 for a more general Theorem 6. Note that it does not suffice to simply use the fact that the assumptions play well with composition with Lipschitz function ρ , which would result in a bound involving Φ( h, A, · ) and its discretization (Φ( h, A, · )) n as a nonlinear operator, as opposed to a bound between Φ( h, A, · ) and Φ( h, A n , · ).Our proof shares the same structure as that of Theorem 2 while making sure that the mismatch from discretizing/extending operators does not blow up with composition.

4.3 Assumptions

We discuss the main assumptions of our P -operators.

Assumption 2 (Lipschitz mapping) . An operator A : F → F is C A -Lipschitz if ‖ Af -Ag ‖ 2 ≤ C A ‖ f -g ‖ 2 for any f, g ∈ F .

We have already had a finite bound on the operator norm in the definition of P -operators. For linear operators, Assumption 2 is equivalent to a bounded operator norm and is thus automatically satisfied by linear P -operators.

The next few assumptions are alternatives; only one needs be satisfied by our P -operators. Intuitively, they ensure that the images of our operator are not too discontinuous:

Assumption 3.A (Maps constant pieces to constant pieces) . We say that an operator A : F → F maps constant pieces to constant pieces at resolutions in N ⊂ N if for any n ∈ N , and for any f ∈ F [ -1 , 1] that is a.e. constant on each interval ( u -1 /n, u ] for u ∈ [ n ] /n , Af is also constant on ( u -1 /n, u ] for each u .

Assumption 3.B (Maps Lipschitz functions to Lipschitz functions) . We say that an operator A : F → F maps Lipschitz functions to Lipschitz functions at resolutions in N ⊂ N if for any n ∈ N , and for any f ∈ F reg ( C v ) , Af is C v -Lipschitz.

This is the most restrictive assumption. However, the next lemma (proof in Appendix D) describes some dense, sparse and intermediate graphs that satisfy these assumptions.

Lemma 2 (Well-behaved operators) . The following examples satisy our assumptions:

Bounded-degree graphings: Let G be a graphing corresponding k -D grids or polymer graphs (not necessarily regular graphs) with fixed monomers (Appendix A). For each N ∈ N , there exists a locally equivalent graphing G ′ N such that its adjacency operator satisfies Assumption 3.A with some resolution set (see Lemma 5 and Lemma 7).

Lipschitz graphons: Let W be a C v -Lipschitz graphon on F reg ( C v ) . Then the HilbertSchmidt operator f ↦→ ∫ 1 0 W ( · , y ) g ( y )d y satisfies Assumption 3.B with resolution in N .

General graphs: Let G be a (potentially infinite) graph with a vertex coloring of N colors such that if two vertices have the same color, then the multisets of their neighbors' colors are the same. Then G 's adjacency operator satisfies Assumption 3.A with resolution N . An N -d hypercube (more generally, Hamming graphs) which is neither bounded-degree nor dense, satisfies the above condition with resolutions in { 2 n } n ∈ [ N ] .

All our results also hold with a less restrictive assumption that allows for a failure of Assumption 3.A and 3.B in a small set (see Assumption 4.A and 4.B in the Appendix). The most general results are proven in Appendix D and hold in even slightly more relaxed conditions which require the operators to map constant pieces to Lipschitz pieces (Assumption 5.A, 5.B in Appendix D).

4.4 Deviations and Justifications

All our theorems hold in slightly modified settings than those by Backhausz and Szegedy (2022). Namely, we allowed for nonlinear P -operators, assumed that they have finite ‖ · ‖ 2 → 2 norm, and used ( k, L )-profiles where we focus on Lipschitz functions (while Backhausz and Szegedy (2022) consider all measurable functions in their profiles). Therefore, we need to ensure that our changes still give us a useful mode of convergence that generalizes dense and sparse graph convergences.

First, without the linearity assumption, the convergence proof by Backhausz and Szegedy (2022) does not hold: we do not know if all limits of nonlinear graphops are still graphops. However, our approximation results (Theorem 2) show special convergent sequences of nonlinear operators, which go beyond the settings in (Backhausz and Szegedy, 2022). Studying special nonlinear operator sequences is interesting since graphop NNs themselves are nonlinear operators. We also assert that our restriction to operators acting on L 2 spaces does not affect convergence guarantees (Theorem 2.14 in (Backhausz and Szegedy, 2022)).

Next, we show that restriction to Lipschitz profiles, which is necessary for our proof technique, does not affect the original convergence either, if we allow our Lipschitz constant to grow:

Theorem 4 (Growing profiles) . Let L : N → R be a strictly increasing sequence such that L ( n ) n →∞ - - - - →∞ . Consider a sequence of P -operators ( A n : F n →F n ) n ∈ N that is Cauchy in the sense that d ′ M ( A n , A m ) := ∑ ∞ k =1 2 -k d H ( S k,L ( n ) ( A n ) , S k,L ( m ) ( A m )) → 0 as m,n →∞ . If A n → A under action convergence (Backhausz and Szegedy, 2022), then ( A n ) n ∈ N converges to the same limit under d ′ M .

This theorem allows us to replace the C v constant in our bound with a slowly growing function in n and get back 'action convergence' as described in Backhausz and Szegedy (2022) so that we can inherit its useful properties while still be able to draw on Lipschitz assumptions in the profiles, without any realistic slowdown in the bound.

Proof sketch For some k ∈ N , by the completeness of the Hausdorff metric over the closed subsets of the space of probability measures supported on R 2 k , the statement is equivalent to showing d H ( S k ( A ) , S k,L ( n ) ( A n )) → 0 as n →∞ . The proof uses a Lipschitz mollification argument to smooth out arbitrary measurable functions f 1 , . . . , f k that witness a measure in the k -profile of A . By selecting a Lipschitz mollifier φ , we ensure that convolving f j with φ /epsilon1 : x ↦→ /epsilon1 -1 φ ( x/epsilon1 -1 ) results in a Lipschitz function that converges to f in L 2 as /epsilon1 goes to 0.

5 Discussion and Future directions

In this paper, we study size transferability of finite GNNs on graphs that are discretizations of graphop, a recent notion of graph limit introduced by Backhausz and Szegedy (2022). We achieve this by viewing GNNs as operators that transform one graph signal into another. Under regularity assumptions, we proved that two GNNs, using two different-resolution GSOs discretized from the same graphop, are close in an operator metric built from weak convergence of measures.

For future direction, a principled study of spectral properties of graphops and graphop neural networks would open doors for techniques from Fourier analysis as used in (Ruiz et al., 2023a; Levie et al., 2022). This leads to distinct challenges, e.g., the spectral gap is not continuous with respect to local-global limits and thus action convergence, but many more properties of spectral measures of bounded-degree graphs are recently studied (Virag, 2018).

Acknowledgments and Disclosure of Funding

This work was supported by Office of Naval Research grant N00014-20-1-2023 (MURI ML-SCOPE), NSF award CCF-2112665 (TILOS AI Institute), NSF award 2134108.

References

Abboud, I. I. Ceylan, M. Grohe, and T. Lukasiewicz. The surprising power of graph neural networks with random node initialization. In International Joint Conference on Artificial Intelligence (IJCAI) , 2021.

David Aldous and Russell Lyons. Processes on Unimodular Random Networks. Electronic Journal of Probability , 12(none):1454 - 1508, 2007. doi: 10.1214/EJP.v12-463. URL https://doi.org/10.1214/EJP.v12-463 .

Azizian and M. Lelarge. Expressive power of invariant and equivariant graph neural networks. In Int. Conf. on Learning Representations (ICLR) , 2021.

Ágnes Backhausz and Balázs Szegedy. Action convergence of operators and graphs. Canadian Journal of Mathematics , 74(1):72-121, 2022. doi: 10.4153/S0008414X2000070X.

Itai Benjamini and Oded Schramm. Recurrence of Distributional Limits of Finite Planar Graphs. Electronic Journal of Probability , 6(none):1 - 13, 2001. doi: 10.1214/EJP.v6-96. URL https://doi.org/10.1214/EJP.v6-96 .

Beatrice Bevilacqua, Yangze Zhou, and Bruno Ribeiro. Size-invariant graph representations for graph classification extrapolations. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 837-851. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/bevilacqua21a.html .

Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine , 34(4):18-42, 2017. doi: 10.1109/MSP.2017.2693418.

Chen, L. Chen, S. Villar, and J. Bruna. Can graph neural networks count substructures? In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Chuang and S. Jegelka. Tree mover's distance: Bridging graph metrics and stability of graph neural networks. In Neural Information Processing Systems (NeurIPS) , 2022.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research , 22(78):1-8, 2021. URL http://jmlr.org/papers/v22/20-451.html .

K. Garg, S. Jegelka, and T. Jaakkola. Generalization and representational limits of graph neural networks. In Int. Conference on Machine Learning (ICML) , 2020.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 1263-1272. JMLR.org, 2017.

Jegelka. Theory of graph neural networks: Representation and learning. In Proc. of the International Congress of Mathematicians (ICM) , 2022.

Nicolas Keriven, Alberto Bietti, and Samuel Vaiter. Convergence and stability of graph convolutional networks on large random graphs. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017. URL https: //openreview.net/forum?id=SJU4ayYgl .

Ron Levie, Wei Huang, Lorenzo Bucci, Michael Bronstein, and Gitta Kutyniok. Transferability of spectral graph convolutional neural networks. J. Mach. Learn. Res. , 22(1), jul 2022. ISSN 1532-4435.

Loukas. What graph neural networks cannot learn: depth vs width. In Int. Conf. on Learning Representations (ICLR) , 2020a.

Loukas. How hard is to distinguish graphs with graph neural networks? In Advances in Neural Information Processing Systems (NeurIPS) , 2020b.

László Lovász. Large Networks and Graph Limits. , volume 60 of Colloquium Publications . American Mathematical Society, 2012. ISBN 978-0-8218-9085-1.

László Lovász and Balázs Szegedy. Limits of dense graph sequences. Journal of Combinatorial Theory, Series B , 96(6):933-957, 2006. ISSN 0095-8956. doi: https://doi.org/ 10.1016/j.jctb.2006.05.002. URL https://www.sciencedirect.com/science/article/ pii/S0095895606000517 .

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , Red Hook, NY, USA, 2019. Curran Associates Inc.

Sohir Maskey, Ron Levie, Yunseok Lee, and Gitta Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems (NeurIPS) , 2022. URL https://openreview.net/forum?id=p9lC_i9WeFE .

Sohir Maskey, Ron Levie, and Gitta Kutyniok. Transferability of graph neural networks: An extended graphon approach. Applied and Computational Harmonic Analysis , 63: 48-83, 2023. ISSN 1063-5203. doi: https://doi.org/10.1016/j.acha.2022.11.008. URL https://www.sciencedirect.com/science/article/pii/S1063520322000987 .

Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proc. AAAI Conference on Artificial Intelligence (AAAI) , 2019.

Mitchell Roddenberry, Fernando Gama, Richard G. Baraniuk, and Santiago Segarra. On local distributions in graph signal processing. IEEE Transactions on Signal Processing , 70:5564-5577, 2022. doi: 10.1109/TSP.2022.3223217.

Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems (NeurIPS) , volume 33, pages 1702-1712. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper_files/paper/2020/file/12bcd658ef0a540cabc36cdf2b1046fd-Paper.pdf .

Luana Ruiz, Luiz F. O. Chamon, and Alejandro Ribeiro. Transferability properties of graph neural networks. 2023a.

Luana Ruiz, Ningyuan Huang, and Soledad Villar. A spectral analysis of graph neural networks on dense and sparse graphs, 2023b.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1): 61-80, 2009. doi: 10.1109/TNN.2008.2005605.

Balint Virag. Operator limits of random matrices. 2018. doi: 10.48550/ARXIV.1804.06953. URL https://arxiv.org/abs/1804.06953 .

Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In Int. Conf. on Learning Representations (ICLR) , 2019.

Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. What can neural networks reason about? In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=rJxbJeHFPS .

Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. In International Conference on Learning Representations , 2021. URL https: //openreview.net/forum?id=UH-cmocLJC .

Yehudai, E. Fetaya, E. Meirom, G. Chechik, and H. Maron. From local structures to size generalization in graph neural networks. In Int. Conference on Machine Learning (ICML) , 2021.

A Additional details

Some highly repetitive parts in the proofs are presented informally (e.g. "use the same techniques as...") to highlight the main ideas in the NeurIPS 2023 version of the paper. For the complete formal proof, readers are advised to study a later version on arXiv.

A.1 Notations

We will use the following extra notations in the proof:

For some P -operator A ∈ L 2 (Ω) → L 2 (Ω), k ∈ N and { f 1 , . . . , f k } =: F ⊂ L 2 (Ω), let F A be the ordered 2 k -tuple ( f 1 , . . . , f k , Af 1 , . . . , Af k ) and denote

For some P -operator A ∈ L 2 (Ω) → L 2 (Ω), k ∈ N and { f 1 , . . . , f k } =: F ⊂ L 2 (Ω), let D A ( F ) be the entry distribution D ( f 1 , . . . , f k , Af 1 , . . . , Af k ) .

A.2 Definitions

Cut norm and cut metric Here we define concretely the cut metric over the space of graphons. Recall that graphons are L 1 ([0 , 1] , F , λ ) Lebesgue-integrable functions. The space of graphons is equipped with a norm known as the cut norm :

and a metric known as the cut metric :

where φ is taken over all measure-preserving bijections from [0 , 1] to [0 , 1]. Intuitively, taking the inf over all measure-preserving bijections allows the cut metric to identify graphons that are just a rearrangement away from another. This generalizes symmetries in graphs, where permuting the vertices (and the corresponding edges) do not change the graph itself. Lovász (2012) shows that graphon convergence under the cut metric is well-behaved: every graphon is a limit of a convergent sequence of graphons; and every Cauchy sequences converges to a graphon. This mode of convergence is known as dense graph convergence.

Lévy-Prokhorov metric The definition of Lévy-Prokhorov metric on P ( R 2 k ) is:

where B 2 k is the Borel σ -algebra generated from open subsets of R 2 k and

Specific graphs We will also define different graphs that we mentioned in the main paper. Unless otherwise stated, we consider undirected graphs. A path on n -vertices for some n ≥ 2 is the graph G = ([ n ] , { ( i, i +1) : i ∈ [ n -1] } ). Its high dimensional generalization are k-D grids on n k vertices are the graphs G k = ([ n ] k , { ( u, v ) : ‖ u -v ‖ 1 = 1 } ).

A Hamming graph H ( d, q ) with parameters d, q ≥ 1 ∈ N is the graph with vertex set [ q ] d and edge set contains all pairs u, v that differs at exactly 1 coordinate (Hamming distance 1). For instance, H (1 , q ) is the complete graph on q vertices and H (2 , q ) is the hypercube on 2 q vertices.

An polymer graph P k with k ≥ 2 and finite n -vertex monomer M = ( V M , E M ) is the graph on kn vertices that contains k copies M 1 , . . . , M k of M . Since M is finite, we can choose a start s 1 and end e 1 vertices in V M 1 . M i 's being copies of M also means that there is an isomorphism φ i from V M i -1 to V M i for each i ∈ [ k -1]. Let s i and e i be inductively defined as φ i ( s i -1 ) and φ i ( e i -1 ) for i from 2 to k . Beside edges in the copies of M , we also add all edges of the form ( e i -1 , s i ) for each i from 2 to k . See Figure 2 for an illustration.

0

1

x

x

+

a

mod 1

x

-

a

mod 1

(a) Cayley graph of

Z

0

1

x

((

x

+

a

))

((

x

+

b

))

((

x

-

b

))

((

x

-

a

))

(b) Cayley graph of

Z

2

. (()) is computation mod 1

...

...

0

1

x

x

+

.

2

x

+

.

4

x

+

.

6

x

+

.

8

((

x

-

a

))

(c) Polymer graph whose unit is a graph with 5 vertices. The pattern extends ad infinitum on both sides.

(d) Graphing of (c). (()) is computation mod 0

.

2.

Edges drawn are from a single unit in the polymer.

A.3 Illustrations

Figure 2 shows examples of graphings for a Cayley graph of Z , Z 2 and a polynomer graph.

A.4 Experimental details

To empirically study the rate of convergence of the d M metric, we study an approximation of Hausdorff metric between 1-profiles of GNNs build on small finite polymers to that built on a large one. The result is in Figure 1 and we now detail the set up.

Architecture The GNN has 2 hidden layers, uses ReLU activation at each layer, including the output layer. We fix a particular polynomial GSO A 2 + A where A is the normalized (by edge vertex degree) adjacency matrix of the input graph of the GNN. Monomer for each polymer has size 5 and is the same monomer seen in Figure 2. The small polymers consist of 2 , 4 , 8 , 16 , 32 monomers. While the large monomers that take the place of the limit object has 128 monomers.

1 -profile estimation Recall that the 1-profile of an operator P consists of entry distributions of ( f, P f ) where f runs through all measurable functions. It is therefore, not tractable to construct the 1-profile exactly. Instead, we draw random F = { f 1 , f 2 , . . . , f 10 } 's to construct a set of 10 probability distributions. Repetitions of this random drawing make up the different solid lines in Figure 1. We construct each f i : [0 , 1] → [ -1 , 1] as piecewise linear function with pieces (( u -1) / 1000 , u/ 1000] for u from 1 to 1000. We set f i (0) = 0 for each i and recursively set f i ( u/ 1000) = f i (( u -1) / 1000) + q/ 100 where q is a random draw from {-1 , 1 } uniformly. Finally, we linearly interpolate between two consecutive endpoints. Numpy random seed is set at 1234567.

After getting this set F , for each operator P that are either GNN built on small polymers or GNN built on the large one, we compute the set {D ( f i , P f i ) : f i ∈ F } to get the corresponding estimation of the 1-profile.

Hausdorff metric computation Given a pair of 1-profile estimations to compute d H over, we use optimal transport code from (Flamary et al., 2021) to compute the earth mover distance between each pairs of distributions. This gives an approximation of d LP between elements of the two profiles. Using the definition of Hausdorff distance, we obtain the result in Figure 1.

A.5 Milder assumptions

As mentioned in the main text, we will work with the following slightly less restrictive set of Assumptions.

Assumption 4.A (Maps constant pieces to constant pieces with high probability) . We say that an operator A : F → F maps constant pieces to constant pieces whp at resolutions in N ⊂ N if there exists a set E ⊂ [0 , 1] with Lebesgue measure λ ( E ) < inf n ∈N 1 /n , such that for any n ∈ N , and for any f ∈ F [ -1 , 1] that is a.e. constant on each interval ( u -1 /n, u ] for u ∈ [ n ] /n , Af is constant on ( u -1 /n, u ] \ E and ‖ Af E ‖ 1 < inf n ∈N 1 n .

✶ Assumption 5.A (Maps constant pieces to Lipschitz pieces) . We say that an operator A : F → F maps constant pieces to Lipschitz pieces at resolutions in N ⊂ N and constant C if for any n ∈ N , and for any f ∈ F [ -1 , 1] that is a.e. constant on each interval ( u -1 /n, u ] for u ∈ [ n ] /n , we have that Af is C -Lipschitz on each ( u -1 /n, u ] , for all u ∈ [ n ] /n .

✶ Assumption 4.B (Maps Lipschitz functions to Lipschitz functions with high probability) . We say that an operator A : F → F maps Lipschitz functions to Lipschitz functions whp at resolutions in N ⊂ N if there exists a set E ⊂ [0 , 1] with λ ( E ) < inf n ∈N 1 /n , such that for any n ∈ N , and for any f ∈ F [ -1 , 1] , Af is C v Lipschitz on [0 , 1] \ E and ‖ Af E ‖ 1 < inf n ∈N 1 n .

Assumption 5.B (Maps constant pieces to Lipschitz pieces with high probability) . We say that an operator A : F → F maps constant pieces to Lipschitz pieces whp at resolutions in N ⊂ N and constant C if for any n ∈ N , there exists a set E ⊂ [0 , 1] with λ ( E ) < 1 n such that for any f ∈ F [ -1 , 1] that is a.e. constant on each interval ( u -1 /n, u ] for u ∈ [ n ] /n , it holds that Af is C -Lipschitz on each ( u -1 /n, u ] \ E , for all u ∈ [ n ] /n and ‖ Af ✶ E ‖ 1 < inf n ∈N 1 n . B Omitted proofs from Section 3

Proof of Lemma 1. Fix m ∈ N and f, g ∈ F m . Since P -operators are bounded, to show that they are self-adjoint, it suffices to show that 〈 A m f, g 〉 = 〈 f, A m g 〉 where 〈· , ·〉 is the usual inner product in the Hilbert space F m . We have:

where the first line is the definition of the inner product in F m , the second line is the definition of the discretization A m (recall that for f ∈ F m , f ′ ∈ F is the extension of f defined as f ′ ( x ) = f ( /ceilingleft xm /ceilingright /m )), the third line is because g ′ is constant on fixed [ u -1 /m,u ] intervals for each u and the fourth line is because A is self-adjoint.

C Theory of graphings

In this subsection, we highlight definitions and key characteristics of graphings so that the paper is self-contained. A much more in-depth discussion can be found in Lovász (2012).

Definition 1 (Borel graphs) . Let Ω be a topological space and (Ω , F ) be the corresponding Borel space. A Borel graph is a graph (Ω , E ) such that E ∈ F × F .

The following proposition asserts that bounded-degree graphs without automorphisms are always Borel:

Lemma 3 (Proposition 18.6 from Lovász (2012)) . If G is a bounded-degree graph without automorphisms then there exists a topology τ on V ( G ) (called the local topology ) such that G is Borel with respect to the Borel space built from τ .

When the graph does have automorphisms, one can break the symmetries by coloring the nodes with some set of colors (the fact that the graph has bounded degree means that one only needs finitely many colors).

We next introduce the main object of interest:

Definition 2 (Graphings) . A graphing is a quadruple G = (Ω , F , λ, E ) such that F is a Borel σ -algebra that makes (Ω , E ) a Borel graph, and λ is a probability measure on (Ω , F ) satisfying: for any A,B ∈ F ,

where deg A ( x ) counts the number of neighbors in A of x .

As an example, we now describe paths and cycles in terms of graphings: let V be [0 , 1], F the Borel σ -algebra generated by open intervals with rational endpoints and for each x ∈ [0 , 1], put ( x, x ± a ) in E if x ± a ∈ [0 , 1] for some real number a < 1. For a < 1 2 , each connected component of the graphing is a finite path. If we consider the edge set ( x, x ± a mod 1) for rational a , then it is not hard to see that connected components of the graphing are finite cycles. If a is irrational, then the resulting graphing is a two-way infinite path, i.e., a path with no 'beginning' and no 'end'. A formal argument, with an appropriate metric on the space of graphings, can be made to show that the limit of cycles and paths coincides to be the two-way infinite path.

Graphings are not unique in representing certain graphs. There are weak equivalences between pairs of graphings that are formalized through the notion of local isomorphisms.

Definition 3 (Local isomorphisms of graphings) . Let G 1 , G 2 be two graphings. A measurepreserving map ϕ : V ( G 1 ) → V ( G 2 ) is a local isomorphism if its restriction to almost every connected component of G 1 (outside of a set of connected component of measure 0 ) is a graph isomorphism with one of the connect components of G 2 . More formally, where ≡ is rooted graph isomorphism and ( G 1 ) x is the connected component of G 1 rooted at x .

Note that local isomorphism of graphings are not symmetric and the map ϕ needs not be invertible. A stronger notion of equivalence, which is symmetric and transitive is local equivalence:

Definition 4 (Local equivalence (informal)) . G 1 and G 2 are locally equivalent if they have the same subgraph densities t ∗ ( F, G 1 ) = t ∗ ( F, G 2 ) for every connected simple graph F .

The above definition is informal since we have not defined subgraph densities (which is done via the Benjamini-Schramm interpretation of graphings). We state Definition 4 for readers who are familiar with dense graph convergence of graphons since this is how such convergences are defined. In fact, we can conveniently bypass formally defining local equivalence by the following characterization:

Lemma 4 (Bi-local isomorphism (Theorem 18.59 in Lovász (2012))) . Two graphings are locally equivalent iff there is a third graphing with a local isomorphism to each of them - a property called bi-local isomorphism .

C.1 Degeneracy of eigenvectors

We give an heuristical argument why directed graphing limits can fail to have eigenvalues and eigenvectors/eigenfunctions. Consider the sequence of one sided directed path: G n = ( V = [ n ] , E = { ( i, i +1) : i ∈ [ n -1] } ), for n ≥ 1. Note that this may not be a Cayley graph in certain authors' definition, which requires Caley graphs to be undirected. However, it is a realistic example in machine learning since convolutional layers in CNNs are exactly the adjacency operators of these directed graphs. The limiting adjacency operator for this sequence, for all intents and purposes, is intuitively the shift operator (say, on /lscript 2 ) A : ( x 1 , x 2 , x 3 , . . . ) ↦→ (0 , x 1 , x 2 , . . . ). An easy calculation then shows that this shift operator does not have eigenvalues.

For a more formal argument and construction on failure cases of existence of eigenvectors in undirected graphings (which is what we defined in this section), refer to Remark 1.6 of Backhausz and Szegedy (2022).

D Omitted proofs from Section 4

D.1 Proof of Lemma 2

We prove each bullet point in Lemma 2 separately, in the next three lemmas.

In the following lemma, we show that graphings corresponding to infinite paths and high dimensional grids satisfy the assumptions in the main results. Similar to how isomorphic graphs represent the same graph, we only need to specify a locally isomorphic graphing that represent the equivalence class of graphings containing the infinite path and high dimensional grids.

Lemma 5 (Well-behaved GSOs - Graphings) . Let G be a graphing corresponding to the Cayley graph of Z (two-way infinite paths) or high-dimensional generalizations (infinite 2D and 3D grids). For a graphing H , let A ( H ) be its adjacency operator. If H is regular, let deg ( H ) be the degree of any of its vertices. For each N ∈ N , there exists locally equivalent graphings:

G ′ N such that A ( G ′ N ) satisfies Assumption 3.A with resolution set d ( N ) = { x ∈ N : xα = N for some α ∈ N } .

G ′′ N such that A ( G ′′ N ) satisfies Assumption 4.A with resolution set [ N ] .

G ′′′ N such that A ( G ′′′ N ) / deg ( G ) satisfies Assumption 4.B with resolution set [ N ] .

Proof of Lemma 5. We will first show the results for two-way infinite paths. Higher dimensional versions follow almost verbatim. Fix a ∈ R \ Q irrational. Recall from Lovász (2012) that the graphing G = ([0 , 1] , F , λ, E ) where F is the Borel σ -algebra generated by open intervals with rational endpoints, λ is some probability measure on ([0 , 1] , F ) and E ∈ F ×F is defined as:

That G is a graphing and each of its connected components is a copy of the Cayley graph of Z generated by {-1 , 1 } is asserted in Lovász (2012).

Fix N ∈ N , the goal is to define a graphing G ′ N that is locally equivalent to G such that A ( G ′ N ) satisfies Assumption 3.A with resolution set d ( N ) - the divisor set of N .

Defining G ′ N . Let G ′ N := ([0 , 1] , F , λ, E ′ N ) where,

Intuitively, G ′ N consists of N disjoint copies of G shrunk to the space [0 , 1 /N ). Since E ⊂ [0 , 1] × [0 , 1], G ′ N is a graphing in the same Borel space as G .

G ′ N is locally equivalent to G . From Lemma 4 it suffices to display a local isomorphism from a third graphing to each of them. Let G ′ 2 N , defined similarly as G ′ N , be the third graphing. We claim that ϕ 1 : x ↦→ 2 x mod 1 is a local isomorphism from G ′ 2 N to G ′ N and ϕ 2 : x ↦→ 2 Nx mod 1 is a local isomorphism from G ′ 2 N to G . Intuitively, G ′ 2 N contains 2 N copies of G while G ′ N contains N copies of G . Thus, our local isomorphisms only need to make sure that a connected component in one copy of G in G ′ N is mapped bijectively to a connected component in another copy of G in G ′ 2 N . We give a rigorous argument below.

For x picked randomly according to λ , let j ∈ [2 N ] / (2 N ) be such that x ∈ [ j -1 / (2 N ) , j ). By definition of G ′ 2 N , the connected component ( G ′ 2 N ) x consists of vertices of the form

for some k ∈ Z and there is an edge from v ( k ) to v ( k ± 1). Consider ϕ 1 ( x ) = 2 x mod 1 that sends v ( k ) to

Now consider the connected component of ϕ 1 ( x ) = 2 x mod 1 in G ′ N . Since x ∈ [ j -1 2 N , j ) , ϕ 1 ( x ) ∈ [ j ′ -1 /N,j ′ ) where [ N ] /N /owner j ′ := 2 j mod 1 N . Thus, the connected component of ( G ′ N ) ϕ 1 ( x ) consists of vertices of the form:

With some modulo arithmetic manipulation, it is not hard to see that ϕ 1 ( v ( k )) = v ′ ( k ) for all k ∈ Z . By definition of G ′ N , there is an edge ( v ′ ( k ) , v ′ ( k ± 1)). Therefore, there is an edge ( ϕ 1 ( v ( k )) , ϕ 1 ( v ( k ± 1))). If λ is the uniform measure then we can ignore vertices at the endpoints of our intervals ( points in [ N ] /N ) and conclude that ( G ′ 2 N ) x ≡ ( G ′ N ) ϕ 1 ( x ) with probability 1 when x ∼ λ .

Now consider ϕ 2 ( x ) = 2 Nx mod 1 that sends v ( k ) to:

The connected component of φ 2 ( x ) in G is in [0 , 1] and consists of vertices of the form:

Thus v ′′ ( k ) = ϕ 2 ( v ( k )) for each k ∈ Z and the definition of G implies that ( G ′ 2 N ) x ≡ G ϕ 2 ( x ) since both are isomorphic to the two-way infinite path. If λ is the uniform measure then the above hold for a.e. x ∈ [0 , 1].

G ′ N satisfies Assumption 3.A with resolution set d ( N ) . Let D be a divisor of N , then Dα = N for some α ∈ N . The intuition is rather straightforward, since D is a divisor of N , the partition into N equal intervals of [0 , 1] is simply a finer partition into D equal intervals. By construction, a vertex x of G ′ N only has neighbors in the 1 /N interval containing it. Therefore, if f is constant on each of the D pieces, it is also constant on each of the N pieces and f ( y ) = f ( x ) = f ( x ′ ) = f ( y ′ ) for any ( x, y ) , ( x ′ , y ′ ) neighbors such that x and x ′ come from the same 1 /D pieces.

Here is a more formal argument. Fix f ∈ F [ -1 , 1] that is a.e. constant on each interval ( u -1 /d, u ] for u ∈ [ D ] /D . We need to show that A ( G ′ N ) f is a.e. constant on the same pieces where A ( H ) is the adjacency operator of a regular graphing H . Fix d ∈ [ D ], fix x, x ′ ∈ [ d -1 /D,d ). Let j be such that x ∈ [ j -1 /N,j ) and j ′ be

such that x ′ ∈ [ j ′ -1 /N,j ′ ). Since Dα = N , we have [ j -1 /N,j ) , [ j ′ -1 /N,j ′ ) ⊆ [ d -1 /D,d ). Furthermore, by definition of G ′ N , all neighbors of x and x ′ are in [ j -1 /N,j ) and [ j ′ -1 /N,j ′ ) respectively, and thus in [ d -1 /D,d ). Therefore, A ( G ′ N ) f ( x ) = 2 f ( x ) = 2 f ( x ′ ) = A ( G ′ N ) f ( x ′ ) for all x, x ′ ∈ [ d -1 /D,d ), which means that A ( G ′ N ) f is constant on each of the pieces [ d -1 /D,d ) for each d ∈ [ D ].

Fix N ∈ N , the goal is to define a graphing G ′′ N that is locally equivalent to G such that A ( G ′′ N ) satisfies Assumption 4.A with resolution set [ N ].

Since Q is dense in R , there exists a number δ ( N ) < 1 4 N 2 such that 1 4 N 2 + δ ( N ) is an irrational number. Let G ′′ N = ([0 , 1] , F , λ, E ′′ N ) where,

That G ′′ N is locally equivalence to G is easily seen via the ambiguity of selecting a when defining G . Now we show that A ( G ′′ N ) satisfies Assumption 4.A with resolution set [ N ]. Pick M ≤ N and f ∈ F [ -1 , 1] that is a.e. constant on each pieces ( m -1 /M,m ] for each m ∈ [ M ] /M . Let

then

Pick x, x ′ ∈ ( m -1 /M,m ] \ E then x, x ′ ∈ ( m -1 /M + /epsilon1,m -/epsilon1 ] where /epsilon1 = 1 / (4 N 2 ) + δ ( N ). Thus we have both x ± /epsilon1 and x ′ ± /epsilon1 are in ( m -1 /M,m ]. By definition of G ′′ N , all neighbors of x and x ′ are in the same 1 /M piece as x and x ′ . Therefore, if f ∈ F [ -1 , 1] are constant on these pieces, so is Af .

Fix N ∈ N , the goal is define a graphing G ′′′ N that is locally equivalent to G such that A ( G ′′′ N ) satisfies Assumption 4.B with resolution set [ N ].

Since Q is dense in R , there exists a number δ ( N ) < 1 / (4 N ) such that 1 / (4 N )+ δ ( N ) is an irrational number. Let G ′′′ N = ([0 , 1] , F , λ, E ′′′ N ) where

That G ′′′ N is locally equivalent to G is easily seen via the ambiguity of selecting a when defining G . Now we show that A ( G ′′′ N ) satisfies Assumption 4.B with resolution set [ N ]. Pick M ≤ N and f ∈ F reg( C v ) that is a.e. C v -Lipschitz on [0 , 1]. Set /epsilon1 = 1 4 N + δ ( N ) and let:

Then λ ( E ) = 2 /epsilon1 < 1 N . Pick x, x ′ ∈ [0 , 1] \ E , then x ± /epsilon1 and x ′ ± /epsilon1 do not 'loop over' in the interval [0 , 1] ( y mod 1 = y , for y ∈ { x, x ′ } + {± /epsilon1 } ). Thus we have:

where in the last line we use Lipschitz property of f . This finishes the proof.

Lemma 6 (Well-behaved operators - General graphs) . Let G be a (potentially countably infinite) graph with a coloring C : V ( G ) → [ N ] for some N such that for each vertex u, v with the same color, the multisets of their neighbors' colors { C ( u ′ ) : ( u ′ , u ) ∈ E } are the same. Additionally, assume that the cardinality of vertices of each color is the same: that there is a bijection from { v : C ( v ) = c } to { v : C ( v ) = c ; } for any colors c, c ′ . Then its adjacency operator satisfies Assumption 3.A with resolution N .

Proof. Given that the cardinality of vertices of each color is the same, it is straightforward to map (via a Lebsesgue-measure preserving bijection) vertices of V into equipartition of [0 , 1] into N pieces I 1 , ..., I N such that each partition contains vertices of the same color and vertices from different partitions will have different colors. Let A be the normalized adjacency operator of G and f be a function with finite L 2 norm such that f is constant on each I j for each j from 1 to N . The goal is to show that Af is also constant on these pieces.

We show this by direct computation. Pick a vertex x and another vertex y from the same piece, say I k for some particular k . By our construction, x and y have the same color since they come from the same piece. By our assumption, the multisets of their neighbors' colors is the same. However, since f is constant on each pieces, we have f ( z ) = f ( u ) for u, v of the same color. Therefore, the multisets { f ( z ) : ( z, x ) ∈ E } and { f ( z ) : ( z, y ) ∈ E } is exactly the same. Taking the appropriate countable sum over each multiset thus result in the same number, i.e. Af ( x ) = Af ( y ), which finishes the proof.

Note that we can drop the countable requirement of the previous proof by invoking an appropriate integral definition. Now we show that the hypercubes - an intermediate graph that is neither dense nor bounded degree, satisfies the requirements of Lemma 6.

Lemma 7 (Well-behaved operators - Polymer graphs) . Let P be a polymer graphing on finite monomer n -vertex graph M = ( V M , E M ) . Let A be then normalized (by each vertex degree) adjacency operator of a graphing. For each N ∈ N , there exists locally equivalent graphings:

P ′ N such that A ( P ′ n ) satisfies Assumption 3.A with resolution set nd ( N ) = { nx ∈ N : xα = N for some α ∈ N } .

P ′′ N such that A ( P ′′ N ) satisfies Assumption 4.A with resolution set n [ N ] .

P ′′′ N such that A ( P ′′′ N ) satisfies Assumption 4.B with resolution set n [ N ] .

Proof. The exact same technique as Lemma 5 (make copies of 'shrinked down' graphings to get the result with Assumption 3.A and find a small enough δ ( N ) so that the wrap around has small measure to get the result with Assumption 4.A and Assumption 4.B) works. Intuitively, this is because we are replacing each node in the path graphing with a monomer graph.

To handle coloring at the monomers details, use Lemma 6 with the coloring that assigns the same color to the same node in each monomers (since each monomer is a copy of each other, there is an isomorphism between the nodes and by 'the same node' we meant under this isomorphism).

Lemma 8 (Well-behaved operators - Graphons) . Let W : [0 , 1] 2 → [0 , 1] be a Lipschitz graphon, with Lipschitz constant C . In other words, W has finite L 2 norm and is Lipschitz in both variables. Then the Hilbert-Schmidt integral operator H that defines the adjacency operator of W satisfies Assumption 3.B at any resolution when applied to graph(on) signal f with L 1 norm 1 .

Proof. The proof follows from definition. Take f ∈ L 2 ([0 , 1]) such that ‖ f ‖ L 2 = 1, then:

where the first line is definition of the Hilbert-Schmidt operator H , the second line is triangle inequality and the last line is due to Lipschitzness of W and L 1 norm of f

Lemma 9 (Well-behaved operators - Hypercubes) . Let C be a hypercube of dimension N . Then the normalized adjacency matrix of C satisfies assumption 3.A with resolutions 2 [ n ] for each n < N .

Proof. We first display a mapping of the vertices in the hypercube over the interval [0 , 1] such that Assumption 3.A will be shown to hold. Recall that a hypercube vertices can be represented by a binary string of N numbers. Here, similarly, we will associate a vertex v of a hypercube with a number in [0 , 1] by adding a 0 . in front of its binary representation. For example, 0 . 110101 (as a binary number) is a vertex correspond to the string 110101 when N = 6. We call this representation mapping f : V → [0 , 1]. Two vertices u, v are connected by an edge iff f ( v ) and f ( v ) differs by exactly one digit in their binary representation.

To get a hypercube representation over the interval [0 , 1], we simply take disconnected copies of (uncountably) infinitely many hypercubes described above. To be more precise, for a number x ∈ [0 , 1], let A ( x ) be the N -letter binary string such that 0 .A (as a string) is the trunction of x to the N -th digit after the binary point. Then, x is connected to y ∈ [0 , 1] iff A ( x ) and A ( y ) differs in exactly one digit and x -A ( x ) = y -A ( y ). For example, for N = 4, we connect 0 . 110101 with 0 . 010101 , 0 . 100101, 0 . 111101 and 0 . 110001. Notice how the first four digits after the dots differ from the original number at exactly one letter, and the last few digits always stay the same. In this example, A ( x ) = 0 . 1101 for x = 0 . 110101.

With this vertex representation in mind, we verify the conditions of Lemma 6. Let V . Fix n < N and divide [0 , 1] into 2 n equipartitions and let vertices in the same partition enjoy the same color. We will also number the partition/color by the binary representation of their left hand endpoint. For example, the first interval is labelled 00 .. 0 ( n digits), the second interval is labelled 00 .. 01 ( n digits). Select a vertex x from a partition with label a 1 a 2 . . . a n . Then, by the definition of our hypercube representation, the neighbors of x have colors:

Exactly 1 neighbor with color ( a 1 +1 mod 1) a 2 . . . a n .

Exactly 1 neighbor with color a 1 ( a 2 +1 mod 1) a 3 . . . a n .

.. .

Exactly 1 neighbor with color a 1 a 2 . . . a n -1 ( a n +1 mod 1).

Remaining N -n neighbors all have color a 1 a 2 . . . a n .

Therefore, the multiset of neighbors' colors of x only depends on the fact that x comes from a partition with label a 1 . . . a n and thus, two vertices of the same color has the same multiset of neighbor's colors. Since the condition of Lemma 6 holds, we have the conclusion for this particular resolution n . Since n was chosen arbitrarily, the result holds for every n < N .

In future work, we formalizes the way in which the above proof holds for limiting object of hypercube.

D.2 Proof of Theorem 2

In this section, we prove a slightly more general version of Theorem 2.

Theorem 5 (General approximation theorem) . Let A : F → F be a P -operator satisfying Assumption 2 with constant C A ; Assumption 5.A with constant C c and resolutions in N . Fix n ∈ N and consider ( k, C v ) -profiles. Let A n : F n →F n be a discretization of A as defined in Equation (9) . Then:

If instead of Assumption 5.A, A satisfies Assumption 5.B with constant C c and resolution set N , then:

If instead of Assumption 5.A and 5.B, A satisfies Assumption 4.B with resolution set N then:

Proof of Theorem 2. Fix k ∈ N . In order to derive an upper bound for d M , we find an upper bound for the Hausdorff distance d H between the two k -profiles of A and A n .

Bounding sup η ∈S k,Cv ( A ) inf η n ∈S k,Cv ( A n ) d LP ( η, η n ) . To bound the supinf quantity, we first select an arbitrary η ∈ S k,C v ( A ). From this measure, we will construct a measure η n ∈ S k,C v ( A n ). If we can upper bound d LP ( η, η n ) < M then we have:

If further M does not depend on the choice of η , then we have

We now proceed with this plan. Fix an arbitrary η ∈ S k ( A ), by definition of k -profiles, there is a corresponding tuple F = ( f 1 , . . . , f k ) with elements in F reg( C v ) such that D A ( F ) = η . Form:

That f ′ j ∈ F n, reg( C v ) is asserted in Lemma 10.

Bounding d LP ( η, η n ) For some /epsilon1 > 0 to be we want to show that d LP ( η, η n ) ≤ /epsilon1 , which is equivalent to showing, for any U ∈ B 2 k ,

Recall that U /epsilon1 was defined in Eqn (18).

Fix any U ∈ B 2 k , we have and

Subtracting both sides yield:

Similarly, we have:

Let y x = /ceilingleft xn /ceilingright /n . Recall that E ⊂ [0 , 1] defined in Assumption 5.B and 4.B is the set of x where Lipschitzness of the image under A may fail. Let E = ∅ if we are using Assumption 5.A alternatively. Define the events:

Using this notation, one also has

It is straightforward to see E l U ( /epsilon1 ) ⊆ E ′ ( /epsilon1 ) , l = 1 , 2 for any U by definition of U /epsilon1 . Furthermore,

since if all 2 k dimensions are bounded in absolute value by /epsilon1/ √ 2 k then the Euclidean distance of the vector is bounded by /epsilon1 .

Therefore it suffices to bound λ ( E j ( /epsilon1 )) + λ ( E j,A ( /epsilon1 )) for each j ∈ [ k ].

Bounding λ ( E j ( /epsilon1 )) . Since f j is C v -Lipschitz for all j , we have:

where the first line is definition of f ′ j , the second line is because λ (( u -1 /n, u ]) = 1 n , the third line is by triangle inequality and the last line is because of Lipschitzness of f j .

Thus, choosing /epsilon1 > √ 2 kC v /n means that λ ( E j ( /epsilon1 )) = 0. (We can tighten this bound by only assuming that f j is C v -Lipschitz outside a set of small measure.)

Bounding λ ( E j,A ) . Let F [ -1 , 1] /owner ˜ f be the extension of f ′ defined as ˜ f ( x ) = f ′ ( /ceilingleft xn /ceilingright /n ) for all x ∈ [0 , 1]. Note that ˜ f is not continuous in general and hence not Lipschitz. We have

for any x ∈ [0 , 1]:

We have:

Thus selecting /epsilon1 > 2 √ ( √ 2 k kC A C v ) /n = 2 5 4 k 3 4 ( C A C v /n ) 1 2 gives λ ( E 1 j,A ( /epsilon1 )) ≤ /epsilon1 2 k .

where we used uniformity of λ and triangle inequality. From here, we proceed slightly differently depending on the specific assumptions.

Proof via Assumption 5.A or 5.B. If A satisfies Assumption 5.A or Assumption 5.B, then we have for each x :

Define the following events:

Then it is clear that E j,A ⊆ E 1 j,A ∪ E 2 j,A ∪ E and thus λ ( E j,A ) ≤ λ ( E 1 j,A ) + λ ( E 2 j,A ) + λ ( E ).

Bounding λ ( E 1 j,A ( /epsilon1 )) via Assumption 5.A or 5.B. Because of Assumption 2, we have: ‖ A ˜ f j -Af j ‖ 2 ≤ C A ‖ ˜ f j -f j ‖ 2 . By L p norms inequality, we have

where the last inequality is due to

Bounding λ ( E 2 j,A ( /epsilon1 )) via Assumption 5.A or 5.B. Notice that ˜ f j is constant in each ( u -1 /n, u ] and thus by Assumption 5.A or 5.B, A ˜ f j is C c -Lipschitz in each ( u -1 /n, u ] \ E . Therefore we have, for x ∈ [0 , 1] \ E and z ∈ ( y x -1 /n, y x ] \ E :

When z ∈ E , we use the second condition in Assumption 5.B to get ‖ A ˜ f j ✶ E ‖ 1 ≤ 1 n and thus for any x /negationslash∈ E :

where E x = ( y x -1 /n, y x ] ∩ E .

For x ∈ E 2 j,A ( /epsilon1 ), we have:

or equivalently,

Since 1 /n ≥ ‖ A ˜ f j E ‖ 1 , we have:

Thus choosing /epsilon1 > 4 √ 2 k ( C c +1) /n means that:

or

Finally, choosing /epsilon1 > √ 16 k √ 2 k n = 2 9 4 k 3 4 n 1 2 makes λ ( E 2 j,A ) ≤ /epsilon1 4 k ; and choosing /epsilon1 > 4 k n makes λ ( E ) < 1 n < /epsilon1 4 k .

Putting everything together via Assumption 5.A or 5.B. Thus, we can choose /epsilon1 = 8 k (√ C A C v +1 n + C v + C c +1 n ) to get:

which allows us to conclude:

Since η was chosen arbitrarily, we have for all η ∈ S k ( A ),

Thus we also have:

Proof via Assumption 4.B. Here, we use the other triangle inequality to get for each x :

Define the following events:

Then it is clear that E j,A ⊆ E 1 j,A ∪ E 2 j,A ∪ E and thus λ ( E j,A ) ≤ λ ( E 1 j,A ) + λ ( E 2 j,A ) + λ ( E ).

Bounding λ ( E 1 j,A ( /epsilon1 )) via Assumption 4.B. Because of Assumption 2, we have: ‖ A ˜ f j -Af j ‖ 2 ≤ C A ‖ ˜ f j -f j ‖ 2 . By L p norms inequality, we have

where the last inequality is due to

Consider:

Thus selecting /epsilon1 > √ C A C v 4 k √ 2 k n = 2 5 4 k 3 4 ( C A C v /n ) 1 2 gives λ ( E j,A ( /epsilon1 )) ≤ /epsilon1 2 k .

Bounding λ ( E 2 j,A ( /epsilon1 )) via Assumption 4.B. Notice that f j is Lipschitz in [0 , 1] and thus by Assumption 4.B, Af j is C v -Lipschitz in [0 , 1] \ E . Therefore we have, for x ∈ [0 , 1] \ E and z ∈ ( y x -1 /n, y x ] \ E and:

When z ∈ E , we use the second condition in Assumption 4.B to get ‖ Af j ✶ E ‖ 1 ≤ 1 n and thus for any x /negationslash∈ E :

where E x = ( y x -1 /n, y x ] ∩ E .

For x ∈ E 2 j,A ( /epsilon1 ), we have:

or equivalently,

Since 1 /n ≥ ‖ Af j E ‖ 1 , we have:

Thus choosing /epsilon1 > 4 √ 2 k ( C v +1) /n means that:

or

Finally, choosing /epsilon1 > √ 16 k √ 2 k n = 2 9 4 k 3 4 n 1 2 makes λ ( E 2 j,A ) ≤ /epsilon1 4 k ; and choosing /epsilon1 > 4 k n makes λ ( E ) < 1 n < /epsilon1 4 k .

Putting everything together via Assumption 4.B. Thus, we can choose /epsilon1 = 8 k (√ C A C v +1 n + C v +1 n ) to get:

which allows us to conclude:

Since η was chosen arbitrarily, we have for all η ∈ S k ( A ),

Thus we also have:

Bounding sup η n ∈S k,Cv ( A n ) inf η ∈S k,Cv ( A ) d LP ( η, η n ) . In this direction, we proceed identically, but now choose η n arbitrarily in S k,C v ( A n ). By definition of ( k, C v )-profiles, there exists a tuple F = ( f 1 , . . . , f k ) each in F n, reg( C v ) such that η n = D A n ( F ).

Construct

where y x = /ceilingleft xn /ceilingright /n for each j ∈ [ k ]. That f ′ j ∈ F reg( C v ) is asserted in Lemma 11. Intuitively, f ′ j is the continuous piecewise linear function that interpolates f j , created by joining f j ( u -1 /n ) and f j ( u ) with a line segment for each u ∈ [ n ] /n . Note also that f ′ j ( u ) = f j ( u ) for all u ∈ [ n ] /n .

Bounding d LP ( η, η n ) . As with the previous direction, we start with an arbitrary U ∈ B k and write down the differences:

Define the events:

Using this notation, one also has

It is straightforward to see E l U ( /epsilon1 ) ⊆ E ′ ( /epsilon1 ) , l = 1 , 2 for any U by definition of U /epsilon1 . Furthermore,

since if all 2 k dimensions are bounded in absolute value by /epsilon1/ √ 2 k then the Euclidean distance of the vector is bounded by /epsilon1 .

Therefore it suffices to bound λ ( E j ( /epsilon1 )) + λ ( E j,A ( /epsilon1 )) for each j ∈ [ k ].

Bounding λ ( E j ( /epsilon1 )) . Since f ′ j is C v -Lipschitz for all j (Lemma 11), we have:

Thus, choosing /epsilon1 > √ 2 kC v /n means that λ ( E j ( /epsilon1 )) = 0. (We can tighten this bound by only assuming that f j is C v -Lipschitz outside a set of small measure.)

Bounding λ ( E j,A ) . Let F [ -1 , 1] /owner ˜ f be the extension of f defined as ˜ f ( x ) = f ( /ceilingleft xn /ceilingright /n ) for all x ∈ [0 , 1]. Note that ˜ f is not continuous in general and hence not Lipschitz. We have for any x ∈ [0 , 1]:

where we used uniformity of λ and triangle inequality. The last thing that we need to show is:

From here, by a word-for-word argument, we can show that the same choice of /epsilon1 does the trick to make λ ( E ′ ( /epsilon1 )) ≤ /epsilon1 . This works since we only use the fact that f ′ j ∈ F reg( C v ) as well as assumption conditions in the previous proof.

Bounding d M . We have:

Therefore,

Lemma 10. Fix n ∈ N . Fix f ∈ F reg ( C v ) . Define the restriction f ′ : 1 n [ n ] → [ -1 , 1] : u ↦→ n ∫ u u -1 /n f ( z )d λ ( z ) . Then f ′ ∈ F n, reg ( C v ) .

Proof. Firstly, we have for any u ∈ [ n ] /n :

Therefore ‖ f ′ ‖ L 2 ([ n ] /n ) ≤ n/n = 1 and thus f ′ is measurable. Finally, for any u < u ′ ∈ [ n ] /n :

where we use Lipschitz property of f in the last inequality. This shows that f ′ is also C v -Lipschitz.

Remark 1. The restriction to F [ -1 , 1] is necessary for this lemma to work because L 2 ([0 , 1]) functions can blow up near 0 .

Lemma 11. Fix n ∈ N . Fix f ∈ F n, reg ( C v ) . Let y x := /ceilingleft xn /ceilingright /n for any x ∈ [0 , 1] . Define the extension f ′ : x ↦→ (1 -y x n + xn ) f ( y x ) + ( xn -y x n ) f ( y x -1 /n ) . Then f ′ ∈ F reg ( C v ) .

Proof. Firstly, since f ′ linearly interpolates between points of f , its range cannot exceed that of f . The restricted range immediately implies that the L 2 norm is bounded by 1 since the support is also in [0 , 1]. Finally, for any x < x ′ ∈ [0 , 1], if there is a u such that x, x ′ ∈ [ u -1 /n, u ) then the fact that the interpolation is linear means that the line segment from f ′ ( x ) to f ′ ( x ′ ) shares the same slope as that from f ( u ) to f ( u -1 /n ), which is at most C v since f ∈ F n, reg( C v ) . Otherwise, there exists a u < u ′ ∈ [ n ] /n such that x ∈ [ u -1 /n, u ) and x ′ ∈ [ u ′ -1 /n, u ′ ). We have:

D.3 Proofs for Section 4.2

The following result characterizes the behaviors of Assumptions 2, 5.A, 5.B and 4.B under addition, multiplication by scalars, power, and element-wise composition with a 1-Lipschitz map:

Lemma 12. Fix k ∈ N and α ∈ R . Recall that ρ : R → R is a 1 -Lipschitz map. Let A 1 , A 2 : F → F satisfy Assumption 2 with constant C 1 A and C 2 A respectively; and Assumption 5.A or 5.B or 4.B with constant C 1 c and C 2 c respectively with common resolution set N . Then:

A 1 + A 2 satisfy Assumptions 2 and 5.A or 5.B or 4.B with constant ( C 1 A + C 2 A ) and ( C 1 c + C 2 c ) respectively.

αA 1 satisfies Assumption 2 and 5.A or 5.B or 4.B with constant | α | C 1 A and | α | C 1 c respectively.

ρA 1 (where the composition is done element-wise) satisfies Assumption 2 and 5.A or 5.B or 4.B with constant C 1 A and C 1 c respectively.

Furthermore, if C 1 c = 0 then A 2 ◦ A 1 satisfies Assumption 2 and 5.A 5.B or 4.B with constant C 1 A C 2 A and C 2 c respectively.

Proof of Lemma 12. To recall, A 1 , A 2 : F → F are P -operators that satisfies Assumption 2 with constant C 1 A and C 2 A respectively and Assumption 5.A or 5.B or 4.B with constant C 1 c and C 2 c respectively with common resolution set N . We now show each part of the Lemma.

We have, for any f, g ∈ F :

where the first line is triangle inequality and the second line is Assumption 2.

Let f ∈ F reg( C v ) piecewise constant on intervals [ n ] /n and x, y ∈ ( u -1 /n, u ] for some n ∈ N and u ∈ [ n ], we have:

where the first line is triangle inequality and the second line is Assumption 5.A.

We have, for any f, g ∈ F :

where the first line is property of norm and the second line is Assumption 2.

Let f ∈ F reg( C v ) piecewise constant on intervals [ n ] /n , and x, y ∈ ( u -1 /n, u ] for some n ∈ N and u ∈ [ n ], we have:

where the second line is Assumption 5.A.

We have, for any f, g ∈ F :

where the first line is Lipschitz property of ρ and the second line is Assumption 2. Let f ∈ F reg( C v ) piecewise constant on intervals [ n ] /n and x, y ∈ ( u -1 /n, u ] for some n ∈ N and u ∈ [ n ], we have:

where the second line is Lipschitz property of ρ and Assumption 5.A.

We have, for any f, g ∈ F :

where the first line is Lipschitz property of A 2 and the second line is that of A 1 .

Let f ∈ F reg( C v ) piecewise constant on intervals [ n ] /n . Since A 1 send constant pieces to constant pieces, the final implication follows direction from Assumption 5.A of A 2 .

Lemma 13. Fix n ∈ N , k ∈ [ K ] ∪ { 0 } , A : F → F satisfies Assumption 2 with constant C A and Assumption 5.A with constant C c and resolution set N . Let n ∈ N , f 1 ∈ F n , f 2 ∈ F . Recall that ˜ f ∈ F denotes the extension of f ∈ F n to [0 , 1] as ˜ f ( x ) = f ( /ceilingleft xn /ceilingright ) /n . If ‖ ˜ f 1 -f 2 ‖ 2 ≤ M for some positive constant M then

If A satisfies Assumption 5.B with constant C c instead then the bound becomes:

If A satisfies Assumption 4.B instead then the bound becomes:

Proof. When k = 0, the bound is vacuously true. We will hide the measure in integrals when it is clear from context in this proof.

Assume that the bound is correct up to some k -1. We have:

Here we can use Jensen's inequality to get:

(168)

We proceed slightly differently based on the exact assumptions that we have. If A satisfies Assumption 5.A or 5.B then:

Here, we give a heuristic argument while the formal argument is exactly similar to that in Theorem 2. The term 3 ∫ 1 0 ( A ˜ [ A k -1 n f 1 ]( z ) nλ ( E z )) 2 d z can be made as close to 0 as possible simply by changing the requirement on λ ( E ) to be smaller and smaller (see the discussion

after Assumption 4.A). At the same time, our assumptions ensure that since ˜ A k -1 n f 1 is a piecewise constant function, the action of A on it cannot become too wild (Lipschitz outside of E and bounded L 1 norm inside E ). This heuristic argument, when made rigorous can help us conclude that:

Plug in the inductive hypothesis and solve the recurrent to get the bound.

Similarly, if we are instead using Assumption 4.B then we use the other triangle inequality to conclude:

Again, we will make use of a heuristic argument to argue that the middle term can be controlled by controlling λ ( E ) since A sends Lipschitz functions to Lipschitz functions outside of E and has bounded L 1 norm inside of E . Therefore:

Solve the recurrent to get the result in the statement of the Lemma.

Lemma 14. In the same setting as Lemma 13, recall that a tilde over a function in F n denotes its extension to F . If A satisfies Assumption 2 with constant C A and Assumption 5.A with constant C c and resolution set N . Let Φ( h, A, · ) = ρ ( ∑ g ∈ n l ∑ K -1 k =0 A k Φ g ( h, A, · ) ) for some Φ g graphop neural network such that ‖ ˜ Φ g ( h, A n , f ) -Φ g ( h, A, ˜ f ) ‖ 2 < M for all g ∈ [ n l ] . Then:

If instead A satisfies Assumption 5.B with constant C c then the bound becomes:

If instead A satisfies Assumption 4.B then the bound becomes:

Proof. We have:

where in the first line, we use the fact that continuous extension commutes with finite element-wise sum and element-wise application of ρ , while expanding L 2 norm; the second line uses Lipschitz property of ρ ; the third line uses triangle inequality; the fourth line uses equivalence of p -norms in finite dimensional vectors; the fifth line is again L 2 norm definition; the sixth line applies Lemma 13 and the rest is algebra.

To get the rest of the cases, applies different versions of Lemma 13.

Lemma 15. Let Φ be an L -layer graphop neural network in the same setting as Lemma 13. If A satisfies Assumption 5.A with constant C c then:

If instead A satisfies Assumption 5.B with constant C c then the bound becomes:

If instead A satisfies Assumption 4.B then the bound becomes:

Proof. Solve the recurrent in Lemma 14.

We now prove a more general version of Theorem 3

Theorem 6. Let A ∈ F satisfying Assumption 2 with constant C A and Assumption 5.A with constant C c and resolution set N ⊆ N . Let n ∈ N and form the discretization A n as per Theorem 2. Let h be normalized such that | h | ≤ 1 element-wise and form the graphop neural network Φ( h, A, · ) : F → F and Φ( h, A n , · ) : F n →F n . We have the following approximation bound:

If instead A satisfies Assumption 5.B with constant C c then the bound becomes:

If instead A satisfies Assumption 4.B then the bound becomes:

Proof. The proof structure is similar to the proof of Theorem 2. For brevity of exposition, we will write h as the (shared) set of parameters in the graphop neural networks and shorten Φ := Φ( h, A, · ) and Φ n := Φ( h, A n , · ). Fix k ∈ N , we will bound sup η ∈S k,Cv (Φ) inf η n ∈S k,Cv (Φ n ) d LP ( η, η n ).

Fix arbitrary η ∈ S k,C v (Φ). By definition of a ( k, C v )-profile, there exists f 1 , . . . , f k ∈ L ∞ reg( C v ) ([0 , 1]) such that η = D Φ ( f 1 , . . . , f k ). For all j ∈ [ k ], let F n /owner f ′ j : [ n ] /n /owner u ↦→ ∫ u u -1 /n f j ( z )d λ ( z ). That f ′ j ∈ L ∞ reg( C v ) ([ n ] /n ) is shown in Lemma 10. Set η n = D Φ n ( f ′ 1 , . . . , f ′ k ).

Bounding d LP ( η, η n ) . Fix some /epsilon1 > 0 to be specified later, by definition of d LP , we need to bound, for each U ∈ B k :

Notice that since λ is the Lebesgue measure, one can denote y x = /ceilingleft xn /ceilingright /n and write:

Since the integrand is only positive when ( f 1 ( x ) , . . . , Φ f k ( x )) ∈ U and ( f ′ 1 ( y x ) , . . . , Φ n f ′ k ( y x )) /negationslash∈ U /epsilon1 and this conjuction only happens when ‖ ( f 1 ( x ) , . . . , Φ f k ( x )) -( f ′ 1 ( y x ) , . . . , Φ n f ′ k ( y x )) ‖ 2 > /epsilon1 , we have:

where

where B 0 is the identity operator for any operator B in the appropriate space.

Bounding λ ( E 1 j ( /epsilon1 )) . Fix j ∈ [ k ], we have:

where the last line uses Lemma 15 under Assumption 5.A with constant C c . Similar results are obtained for the other assumptions.

We also have:

Thus selecting

Bounding λ ( E 0 j ( /epsilon1 )) . This step simply uses Lipschitzness of the graph signal to bound its discretization and is thus identical to that in Theorem 2. In short, choosing /epsilon1 > √ 2 kC v /n gives λ ( E 0 j ( /epsilon1 )) = 0.

Bounding sup η n ∈S k,Cv (Φ n ) inf η ∈S k,Cv (Φ) d LP ( η, η n ) . This direction uses the same technique as in Theorem 2.

Putting everything together Defining C A = ( n max KC K A ) L gives a choice of:

where P 1 = 3 KL and P 2 = 3 K K 2 .

For other choices of assumptions, we obtain similar bounds.

E Additional results

E.1 Proof of Theorem 4

Because of completeness of d H in the space of closed subsets of P ( R 2 k ) for every k ∈ N , the statement in Theorem 4 is equivalent to showing that for each k ∈ N , we have S k,L ( n ) ( A n ) converges to S k ( A ) in d H . We do this via a mollification argument.

Definition 5 (Lipschitz mollifier) . A Lipschitz mollifier in R is a smooth (infinitely differentiable) function φ : R → R satisfying:

∫ R φ ( x )d λ ( x ) = 1 .

lim /epsilon1 → 0 φ /epsilon1 ( x ) := lim /epsilon1 /epsilon1 -1 φ ( x//epsilon1 ) = δ ( x ) - the Dirac function.

Although not standard, we require φ to be 1 -Lipschitz and symmetric around 0 ( φ ( x ) = φ ( -x ) ).

Given a measureable function f ∈ L ∞ [ -1 , 1] ( R ) , defines the convolution operation:

The next result shows the existence of such a function:

Lemma 16. Let φ : R → R be:

where Z is a normalization constant to make sure that ∫ R φ d λ = 1 . Then φ is a Lipschitz mollifier.

Proof. The first property is built into the definition. The second property is obvious since the support of φ ( x//epsilon1 ) is [ -/epsilon1, /epsilon1 ] and thus it converges to the Dirac function as /epsilon1 goes to 0. Lipschitz-ness can be seen by computing the first derivative (since the function is smooth) over ( -1 , 1) and see that it is bounded in [ -1 , 1]. Symmetry is also obvious since the function depends only on the absolute value of its argument.

Consequences of a Lipschitz mollifier include:

Lemma 17. Let φ be a Lipschitz mollifier and /epsilon1 > 0 , then for any measurable f ∈ F ∞ [ -1 , 1] ( R ) , f ∗ φ /epsilon1 is max(1 , /epsilon1 -2 ) -Lipschitz and lim /epsilon1 → 0 ‖ f -f ∗ φ /epsilon1 ‖ 2 = 0 .

Proof. For the first part of the statement, consider:

For the second part, we have:

where we use two changes of variables.

Square both sides and take integral over x and use Jensen inequality to get:

Apply Fubini-Tonelli theorem to factorize the mollifier, we get:

Apply a final change of variable:

Therefore, as /epsilon1 goes to 0, the inner integrand goes to 0. Since f and f ∗ φ /epsilon1 are all bounded (as f ∈ [ -1 , 1]) for small enough /epsilon1 , we can apply dominated convergence to conclude that the integral itself goes to 0 and thus the 2-norm on the left hand side also goes to 0.

We are now ready to proceed with the proof of Theorem 4. Given an element η ∈ S k ( A ), there exists a set of functions F = { f 1 , . . . , f k } ⊆ L ∞ [ -1 , 1] ([0 , 1]). Each of these functions can be extended to R by setting f ( x ) = f (0) if x ≤ 0 and f ( x ) = f (1) if x > 1. Call the extended function f ′ . Now we can apply mollification convolution to each of them to get a family of functions { f j,/epsilon1 := f ′ j ∗ φ /epsilon1 } j ∈ [ k ] ,/epsilon1> 0 . Recall that we have shown f j,n := f j, 1 / √ L ( n ) to be L ( n )-Lipschitz for each n ∈ N . Let f ′′ j,n be the restriction of f j,n to F n . Then f ′′ j,n is still L ( n )-Lipschitz (in the metric on 1 /n [ n ] induced by metric on [0 , 1]) and thus we can find a profile for F ′′ n := { f ′′ 1 ,n , . . . , f ′′ k,n } in S k,L ( n ) ( A n ).

Furthermore, by property of mollifier and the fact that L ( n ) → ∞ as n goes to infinity, we have ‖ f j,n -f ′ j ‖ 2 → 0 with n → ∞ . Since f ′ j is constant outside [0 , 1], we also have ‖ f j,n, | [0 , 1] -f j ‖ F → 0 with n . Using the same proof technique as Theorem 2, we can conclude that:

and thus S k ( A ) ⊆ lim n S k,L ( n ) ( A n ). For the other direction, recall that S k ( A ) = lim n S k ( A n ) and that S k,L ( n ) ( A n ) ⊆ S k ( A n ). Together with completeness of d H , we conclude that

Conjecture 1 (Action convergence of graphop neural networks) . Let ( A n ) n ∈ N be an action convergent sequence of graphops. Then (Φ( h, A n , · )) n ∈ N is an action convergent sequence of P -operators.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the transferability and approximation qualities of Graph Neural Networks (GNNs) when applied to graphs or Graph Signal Operators (GSOs) sampled from graph limit objects, known as graphops. The authors demonstrate that when a sequence of graphs is sampled from a graphop, the sequence converges according to a well-known metric. The paper also introduces the concept of graphop neural networks, which are utilized as a tool for analysis. It is shown that when a GNN is applied to a sequence of graphs sampled from a graphop, the output converges to the corresponding output from the graphop neural network applied to the graphop itself.\n\nSTRENGTHS:\n- Presents a novel integration of known techniques, specifically graphops and Graph Neural Networks (GNNs). The authors notably generalized the results for graphs converging to a graphon, which has been proven in [1,2,3].\n- This research could potentially pave the way for further exploration into the transferability of GNNs in relation to sparse graphs. However, the authors miss to mention some work from Keriven et al. and Ruiz et al. which already consider non-dense graphs.\n\nWEAKNESSES:\n- The quality of writing and presenting is poor leading to bad comprehensibility. None of the theorems are self-contained. While the proofs seem correct, it is difficult to follow them due to notational issues and gaps in the argumentation (see below for a non-complete list). Note that I did not check all proofs thoroughly. \n- The authors make some, at least, weird assumptions such as graphops being self-adjoint, but allowing them to be non-linear. The set of such operators is empty. Also, the choices of the convergence notion are not clear, even though a subsection is dedicated to it. \n- Some assumptions appear restrictive, e.g.: The graphops are assumed to be sampled from $[0,1]$, in contrast to many other transferability papers (e.g. [2-4]). Assumption 3.A  also appears restrictive.\n- The paper lacks clear organization and writing, with numerous notational errors scattered throughout. The order of presenting assumptions before other information also appears confusing. The appendix also contains many, at least, notational issues.\n- The practical and theoretical implications of the results remain unclear. For instance, it would be beneficial to understand:\na) The potential impact of this work on theorists and practitioners, b) Examples of graphs for which the results are applicable, c) How can the convergence stated in Theorem 2 and 3 be experimentally demonstrated?, and d) The restrictiveness of the assumptions made in the study.\n\nQUESTIONS:\n- Could you clarify how your convergence definition aligns with the one proposed by Backhaus and Szegedy? In particular, how can we assure that convergence results still hold if we consider a sequence of Lipschitz constant $(C_{v_n})_n$ that grows sublinearly in n? Why does $d_M$ from Backhaus and Szegedy also converge to zero in this setting?\nIf this is sufficient, why do the authors change the notion of convergence?\nWhat motivates the chosen notion of convergence? If not for mathematical convenience, what other reasons justify this choice?\n- What implications do these results have for practitioners? Maybe the authors can provide some (simple) experiments, where Theorem 2 and 3 are verified? This would help for clarification and realizing an example (other than dense graph convergence which was already covered in previous works)\n- What implications may these results have for developing new theories? For example, do you expect to achieve generalization bounds for sparse graphs such as in [4]?\n- On line 73, could you explain why nonlinear operators are significant? My understanding is that all Graph Signal Operators (GSOs) are linear operators.\n- You sample nodes deterministically, can you compare the loss of generality with works that do not assume this? E.g., [2,3,4]\n- You mention that all graphops are self-adjoint. In light of this, why are nonlinear operators brought up? Note that every self-adjoint operator is linear.\n- Regarding Theorem 4, could you clarify its utility? Also, the proof seems peculiar and certain terms such as $A$ and $A_n$ lack clear definitions.\nFor example, why does $S_{k,L(n)} (A_n) \\subset S_k(A)$ hold? These profiles consist of different distributions, right? More details are provided below.\n\n### Minor Issues and Questions:\n- Line 12: Could you clarify why graphs sampled from the same graphop share structural properties? Especially, for graphops that are not induced by graphons.\n- Line 24: Please consider adding [2,3].\n- Line 43: Please refer to Keriven's work on \"relatively sparse\" graphs in the context of random graph models.\n- Line 53: Do you mean \"limits of GNNs of infinite sequences of graphs\"?\n- Line 58: [3] generalized the results to unbounded graphon operators and more diverse GNNs.\n- Line 89: Ruiz's work also indicates a $\\sqrt(n)$ dependence which decays to zero, this statement seems incorrect. Also, there are other works by Ruiz considering sparse graphs.\n- Line 100: Random Graph Models (RGMs) essentially consist of graphons/kernels which are graph limits, and they are not opposed to these.\n- Line 102: Could you define \"closeness\"?\n- Line 119: Please mention that the entire work considers only spectral (polynomial) GNNs.\n- Line 130: The space also contains limits of sparse graphs.\n- Line 185: What does \"positiveness\" mean?\n- Line 235: Do the results generalize to higher dimensional features?\n- Line 246: What is $F_m$? \n- Line 310: How do you define d_M for these non-linear operators that can neither be self-adjoint nor positive?\n- Line 334: What do you mean with \"wild\"?\n- Line 335: Could you elaborate on what \"specifically designed for our discretization scheme\" means?\n- Line 341: Why \"high probability\"?\n- Line 379: Could you clarify what this sentence means: \"Then, A_n converges to the same limit as action convergence\"?\n\n#### Proof of Theorem 2:\n- In bounding d_H, why is the maximum over certain quantities not considered?\n- Line 731: Could you define the conditions for epsilon in the phrase \"for some epsilon\"?\nThere seem to be numerous typos throughout, such as in equation 56 and line 727, $S_k,C_v$.\n- The definition of $U_\\varepsilon$ is difficult to locate and appears incorrect.\nEquality 60 should be an inequality.\n- Line 763: What does $\\bar{\\mathcal{E}}$ denote? \n\n#### Proof of Theorem 4:\n\n- Line 1007: Could you clarify what it means for $f''$ to be Lipschitz in $l^2(n)$?\n- Line 1011: What does the notation $\\lim_\\bar{S_{k,L(n)}(A_n)}$ signify?\n\n[1] Ruiz, L., Chamon, L., & Ribeiro, A. (2020). Graphon neural networks and the transferability of graph neural networks. Advances in Neural Information Processing Systems, 33, 1702-1712.\n\n[2] Keriven, N., Bietti, A., & Vaiter, S. (2020). Convergence and stability of graph convolutional networks on large random graphs. Advances in Neural Information Processing Systems, 33, 21512-21523.\n\n[3] Maskey, S., Levie, R., & Kutyniok, G. (2023). Transferability of graph neural networks: an extended graphon approach. Applied and Computational Harmonic Analysis, 63, 48-83.\n\n[4] Maskey, S., Levie, R., Lee, Y., & Kutyniok, G. (2022). Generalization analysis of message passing neural networks on large random graphs. In Advances in Neural Information Processing Systems.\n\nLIMITATIONS:\nThe authors do not address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the transferability of Graph Neural Networks (GNNs) in terms of size. It proposes Graphop Neural Networks that operate on graphop signals and discretization of the network. It proves that the discretization of graphop and the graph are close, and consequently, two discretizations of different resolutions are close. It also shows that the corresponding Graphop Neural Network outputs are close which implies the transferability of GNN in size.\n\nSTRENGTHS:\nMy background does not support me in evaluating the paper fairly. However, I believe the tackled problem of GNN size transferability is critical, and the author's analysis using graph limit and its discretization is promising. The assumption seems mild, which could make the theory applicable to most existing GNNs and real-world sparse graphs.\n\nWEAKNESSES:\nThe related work session is scarce, which makes it more difficult for me to understand the background. The paper is purely theoretical, which is fine. Still, it will be more sensible for readers like me if the authors can provide some form of experiment on synthetic graphs of variable size to better understand the implication of the theorem.\n\nQUESTIONS:\n- What is the key difference between this work and the graphon neural network which also adopt a graph limit perspective?\n- The bounds require the number of vertices in both graphs to be sufficiently large. Then how to interpret the bound when one graph is large and the other is smaller, which is more important for graph size transferability, and also two small graphs.\n\nLIMITATIONS:\nReview Summary:\nDue to my lack of background knowledge, I could not appropriately relate it to previous research and assess the proof fairly. However, the paper addresses a crucial problem in graph learning and seems to provide a sound theoretical analysis, which leaves it on the borderline.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the theoretical perspective of whether graph neural networks (GNNs) can generalize to graphs that are different from the ones they were trained on. The authors study the transferability and approximation results via graph limits, including sparse graphs such as bounded-degree or power law graphs. The paper presents various notions of graph limits and develops quantitative bounds on the distance between a finite GNN and its limit on an infinite graph. The authors also verify the regularity assumption for various graph sequences in this study. Overall, the paper's contributions include a theoretical framework for studying the generalization of GNNs to different graphs, as well as insights into the regularity of graph sequences and the use of graph limits for approximation and transferability analysis.\n\nSTRENGTHS:\nS1. The paper takes a unique perspective of studying the transferability and approximation results of GNNs via graph limits. The proposed approach is novel and provides a theoretical framework for studying the generalization of GNNs to different graphs.  \nS2. The authors provide rigorous mathematical proofs and analysis to support their claims, which potentially have important implications for the practical use of GNNs in real-world applications.  \nS3. The paper's results hold for both dense and sparse graphs, and various notions of graph limits, which makes the paper's contributions applicable to a wide range of graph-based learning tasks.\n\nWEAKNESSES:\nW1. Lack of real-world application examples: The paper focuses primarily on the theoretical aspects of graph limits and their implications for GNN generalization. However, it would be valuable to include real-world application examples where the proposed approach can be applied and demonstrate its practical utility. This would provide concrete evidence of the significance and relevance of the work in real-world scenarios.  \nW2. Limited experimental evaluation: While the paper includes some experimental results, the evaluation could be more extensive and diverse. It would be beneficial to include experiments on a wider range of datasets and graph structures to demonstrate the effectiveness and generalizability of the proposed approach. Additionally, providing a detailed analysis of the experimental results, including statistical significance tests and comparisons with baseline methods, would further strengthen the empirical findings.  \nW3. Lack of comparison with existing methods: The paper could benefit from a more comprehensive comparison with existing methods that address the generalization of GNNs to different graphs. This would provide a clearer understanding of the novelty and effectiveness of the proposed approach. Including a comparison with state-of-the-art methods and discussing the advantages and limitations of the proposed method in relation to existing approaches would strengthen the paper.\n\nQUESTIONS:\nQ1. Could you expand the experimental evaluation by including experiments on a wider range of datasets and graph structures? This would provide a more comprehensive understanding of the effectiveness and generalizability of your proposed approach. Additionally, providing a detailed analysis of the experimental results, including statistical significance tests and comparisons with baseline methods, would strengthen the empirical findings.  \nQ2. Can you provide real-world application examples where your proposed approach can be applied? Demonstrating the practical utility of your approach in real-world scenarios would further highlight the significance and relevance of your work.\nCould you discuss the limitations of your proposed approach in more detail? Specifically, addressing the scalability of the approach to larger graphs, considering different types of GNN architectures, and exploring the impact of different graph properties on generalization performance would be interesting.   \nQ3. Have you considered the computational complexity of your proposed approach? It would be helpful to discuss the computational requirements and scalability of your method, especially when applied to larger graphs.  \nQ4. Can you provide more insights into the regularity assumption verified for various graph sequences? How does the regularity assumption impact the generalization performance of GNNs, and are there any specific conditions or properties that need to be satisfied for effective generalization?  \nQ5. Can you discuss potential future directions for your research? Specifically, addressing the scalability of the approach, exploring different types of graph structures, and considering the impact of different graph properties on generalization performance would be valuable areas for further investigation.\n\nLIMITATIONS:\nThe paper does not explicitly address the potential negative social impact of the proposed approach. While the paper focuses primarily on the theoretical aspects of graph limits and their implications for GNN generalization, it would be beneficial to include a discussion on the potential ethical implications of the proposed approach. Specifically, the authors could consider the potential impact of their work on issues such as privacy, fairness, and bias in machine learning.\n\nAdditionally, the paper could benefit from a more comprehensive discussion of the limitations of the proposed approach. While the authors acknowledge some limitations, such as the need for regularity assumptions and the limited experimental evaluation, a more detailed discussion on the potential limitations and their impact on the generalization performance of GNNs would be valuable.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work is essentially extending the existing theoretical study of GNNs through graphons by replacing them with graphops, an operator view on graphs that allows for results of sparse graphs. Then, standard results on size generalization for instance, are presented.

STRENGTHS:
The paper is technically sound as far as I could check. The theoretical contributions are non-trivial and present enough mathematical rigor for this venue. It is a first step towards moving being graphon-restricted analysis of GNNs.

WEAKNESSES:
The issue I have with this paper is its usefulness for machine learning applications. We understand the space L^1([0,1]^2) pretty well for graphons, but the newly introduced operator view from Backhausz and Szegedy isn't clear to me yet. Could the authors give the readers more intuitions on how P-operators can generate real-world graphs? Remember that in the end of the day the GNN is applied to real-world graphs. Graphons can be seen as latent factor models, which is somehow interpretable as a graph generating process. How can P-operators be seen by the readers? This might be a flaw in my understanding that the authors can help me with, so I'm willing to raise my score. In particular how can  (k,L)-profiles be interpreted in real-world? Graphs as operators make sense in a lot of fields, but I'm a little concerned that graph data isn't fit to this view.

QUESTIONS:
Could the authors give the readers more intuitions on how P-operators can generate real-world graphs? 
how can  (k,L)-profiles be interpreted in real-world?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10