Title: Compositional PAC-Bayes: Generalization of GNNs with persistence and beyond
Venue: NeurIPS 2024
Accepted: yes
Keywords: generalization

tda

GNN

PAC-Bayes
Abstract: Heterogeneity, e.g., due to different types of layers or multiple sub-models, poses key challenges in analyzing the generalization behavior of several modern architectures. For instance, descriptors based on Persistent Homology (PH) are being increasingly integrated into Graph Neural Networks (GNNs) to augment them with rich topological features; however, the generalization of such PH schemes remains unexplored. We introduce a novel _compositional_ PAC-Bayes framework that provides a general recipe to analyze a broad spectrum of models including those with heterogeneous layers. Specifically, we provide the first data-dependent generalization bounds for a widely adopted PH vectorization scheme (that subsumes persistence landscapes, images, and silhouettes) as well as PH-augmented GNNs. Using our framework, we also obtain bounds for GNNs and neural nets with ease. Our bounds also inform the design of novel regularizers. Empirical evaluations on several standard real-world datasets demonstrate that our theoretical bounds highly correlate with empirical generalization performance, leading to improved classifier design via our regularizers. Overall, this work bridges a crucial gap in the theoretical understanding of PH methods and general heterogeneous models, paving the way for the design of better models for (graph) representation learning. 
Our code is available at https://github.com/Aalto-QuML/Compositional-PAC-Bayes.

Compositional PAC-Bayes: Generalization of GNNs with persistence and beyond

Abstract

Heterogeneity, e.g., due to different types of layers or multiple sub-models, poses key challenges in analyzing the generalization behavior of several modern architectures. For instance, descriptors based on Persistent Homology (PH) are being increasingly integrated into Graph Neural Networks (GNNs) to augment them with rich topological features; however, the generalization of such PH schemes remains unexplored. We introduce a novel compositional PAC-Bayes framework that provides a general recipe to analyze a broad spectrum of models including those with heterogeneous layers. Specifically, we provide the first data-dependent generalization bounds for a widely adopted PH vectorization scheme (that subsumes persistence landscapes, images, and silhouettes) as well as PH-augmented GNNs. Using our framework, we also obtain bounds for GNNs and neural nets with ease. Our bounds also inform the design of novel regularizers. Empirical evaluations on several standard real-world datasets demonstrate that our theoretical bounds highly correlate with empirical generalization performance, leading to improved classifier design via our regularizers. Overall, this work bridges a crucial gap in the theoretical understanding of PH methods and general heterogeneous models, paving the way for the design of better models for (graph) representation learning. Our code is available at https://github.com/Aalto-QuML/Compositional-PAC-Bayes .

1 Introduction

Topological data analysis (TDA) harnesses tools from algebraic topology to unveil the underlying shape and structure of data. TDA has recently gained significant traction within machine learning mainly due to its flagship method: persistent homology (PH) [11], which allows for capturing topological invariants (like connected components and loops) of the input domain at multiple scales. In particular, PH has recently been leveraged as a tool to augment the representational capabilities of graph neural networks (GNNs) [20, 53, 58], with expressivity gains formally established [22, 44]. Intuitively, PH furnishes global structural signatures that complement the local nature of GNNs [5, 18, 20, 57].

Understanding the generalization behavior of these models is crucial as it plays a pivotal role in ensuring their reliability and applicability [41]. In this context, there are two fundamental approaches to achieving generalization bounds: data-independent and data-dependent [47], each offering unique insights into the generalization problem. Both these approaches have been investigated to analyze the generalization ability of GNNs [12, 14, 25, 33, 36, 46, 48, 52, 59]. Data-dependent generalization bounds evoke particular interest since they are typically much tighter than the agnostic bounds afforded by, e.g., VC dimension. However, no such bounds have been unearthed for PH methods (i.e., learnable vectorization schemes) and, consequently, for GNNs enhanced with PH-based descriptors.

We bridge this gap with the first data-dependent generalization bound for classifiers based on a versatile and widely used vectorization framework for persistence diagrams, namely, PersLay [5].

Vikas Garg

PersLay leverages extended persistence to effectively represent detailed topological features, and subsumes commonly used methods such as persistence landscapes [4], images [1], and silhouettes [6].

Central to our analysis is a novel PAC-Bayes framework (Lemma 2) that provides a general recipe to analyze the generalization of a broad spectrum of models, including those with heterogeneous layers and those comprising multiple sub-models. To achieve this, we introduce general conditions (Equations 6-9) that are satisfied by commonly used learning architectures and, surprisingly, their compositions (Section 4). Leveraging Lemma 2, we show how to obtain bounds for heterogeneous MLPs and GNNs in a straightforward manner (Table 2). Notably, we also establish the first generalization bounds for GNNs augmented with persistence layers (PersLay).

Our exposition focuses on graphs; however, i) our Lemma 2 can be used in any domain and ii) our bound for PersLay considers persistence diagrams obtained from any non-learnable filtration function, and therefore extends more generally to input domains beyond graphs. From a technical perspective, our approach hinges on contrasting previous analyses within the PAC-Bayes framework [9, 37, 38] to extract the common structure encoded in the general conditions of Lemma 2. This

allows us to overcome challenges arising from the heterogeneity of the models we consider.

Our experiments on several standard real-world datasets confirm strong correlation between the empirical performance and our theoretical bounds. We reinforce the merits of our analysis via regularized PH-based models informed by our bounds with demonstrable empirical benefits.

Our main contributions are:

(Theoretical, see Table 1) We develop a general recipe for obtaining PAC-Bayes bounds for a broad class of (possibly heterogeneous) models and their compositions. We also provide the first data-dependent bounds for PH-based classifiers and combinations of GNNs and PersLay;

(Empirical) We show that the dependence on parameters depicted in our bounds strongly correlates with the observed performance. We also show that novel regularization schemes based on our bounds can reduce the generalization gap of PH-augmented GNNs on multiple datasets.

2 Preliminaries

This section overviews GNNs, persistent homology on graphs, and their combination. We also provide basic notions and results in PAC-Bayes learning, which serve as a background for this work.

̸

Graph neural networks (GNNs). Message-passing GNNs [15, 55] employ a sequence of messagepassing steps: each node v aggregates messages from its neighbors N ( v ) = { u : ( v, u ) ∈ E } , using the resulting vector to update its own embedding. Starting from z (0) v = z v , GNNs recursively apply

Notation. We consider attributed graphs denoted as a tuple G = ( V, E, z ) , where V = { 1 , 2 , ..., n } is the vertex set, E ⊆ V × V is the edge set, and z : V → R d z assigns to each vertex v ∈ V an attribute (or color) z ( v ) . For convenience, hereafter, we denote the feature vector of v by z v . We consider classification tasks with input and label spaces X and Y = { 1 , . . . , K } ( K is the number of classes) and the γ -margin loss l γ : Y × R K →{ 0 , 1 } where l γ ( y, ˆ y ) = 1 (ˆ y y ≤ γ +max j = y ˆ y j ) and γ ≥ 0 is the margin parameter. Let S = { ( x i , y i ) } i m =1 ⊆ X × Y denote a collection of m input/label pairs sampled i.i.d. from some unknown distribution D . Then, the empirical error of a hypothesis g w : X → R K with parameters w is defined as ˆ L S ,γ ( g w ) = 1 / m ∑ m i =1 l γ ( y i , g w ( x i )) . Accordingly, we can define the generalization error as L D ,γ ( g w ) = E ( x,y ) ∼D [ l γ ( y, g w ( x ))] . We use ∥ · ∥ 2 to refer the ℓ 2 norm (vectors) and the spectral norm (matrices), and ∥ · ∥ F to refer to the Frobenius norm. Also, we denote the set { 1 , ..., n } by [ n ] . We provide a notation table in the Appendix (Table 5).

where { {·} } denotes a multiset, Agg ℓ is an order-invariant function and Upd ℓ is an arbitrary update function - often a multilayer perceptron (MLP).

Persistence homology (PH) on graphs aims to extract detailed (multiscale) topological features from graphs. A filtration of a graph G is a finite nested sequence of subgraphs of G , i.e., ∅ = G 0 ⊂ G 1 ⊂ ... ⊂ G -alternatively, clique complexes can also be built at each step (see [2]). While filtrations can be obtained in different ways [2, 18], a typical choice consists of leveraging a real-valued filtering function f on the vertices of G (or their features) to compute the vertex level set V α = { v : f ( v ) ≤ α } at scale α ∈ R . Let G α be the subgraph of G induced by V α . By increasing α from -∞ to ∞ , we obtain a nested sequence of subgraphs called the sub-level filtration of G induced by f . The idea of PH is to keep track of the appearance and disappearance of topological features (e.g., connected components, loops) in a filtration. If a topological feature first appears in G α b and disappears in G α d , then we encode its persistence as a pair ( α b , α d ) ; if a feature does not disappear, then its persistence is ( α b , ∞ ) . The collection of all pairs forms a multiset that we call persistence diagram . We use Dg i ( G ) to denote the persistence diagram for i -dim topological features of graph G . For details on PH, we refer to Edelsbrunner and Harer [10], Hensel et al. [17], and Hofer et al. [19].

Persistence layers (PersLay). Carrière et al. [5] introduced a general way to vectorize persistence diagrams . Given a persistence diagram Dg( G ) for an arbitrary dimension, PERSLAY computes

where Agg is any permutation invariant operation (e.g., minimum, maximum, sum, or kth largest value), ω : R 2 ↦→ R is a weight function for the elements in Dg( G ) , and φ : R 2 ↦→ R h is the so-called point transformation . More specifically, given a persistence pair p = [ p 1 , p 2 ] ⊤ ∈ R 2 , PersLay introduces the triangle point transformation ( Λ ):

the Gaussian point transformation ( Γ ):

and the line point transformation ( Ψ ):

In all of these transformations t 1 , . . . , t h are learnable parameters. Notably, the architectural design of PERSLAY is quite versatile and accommodates a wide range of traditional persistence diagram vectorizations, extending DeepSets [56] and including persistence landscapes [4], persistence silhouette [6], persistence images [1], and other Gaussian-based kernel approaches [23, 29, 31].

To obtain class predictions, the output of PersLay is typically fed to an MLP with Lipschitz activations. We refer to this joint model (PersLay followed by MLP) as PersLay Classifier (PC).

GNNs with persistence. Recently, PH has been used to boost the expressive power of GNNs. For instance, Horn et al. [20] and Immonen et al. [22] leverage node embeddings at each layer of a GNN to obtain persistence descriptors. This topological information can be added to GNN's node embeddings (as in [20]) or concatenated with GNN's graph-level representation (as in [22]) - which we refer to the

parallel mode of integrating PH into GNNs. Figure 1 illustrates the latter (parallel mode) with persistence diagrams vectorized using PersLay.

PAC-Bayesian analysis adopts a Bayesian approach to the PAC learning framework [30, 37, 38, 50]. The idea consists of placing a prior distribution P over our hypothesis class and then use the training data to obtain a posterior Q , i.e., the learning process induces a posterior distribution over the hypothesis class. Importantly, we can leverage the Kullback-Leibler (KL) divergence between Q and P to bound the difference between the generalization and empirical errors [37]. To compute PACBayes bounds for models like neural networks, we can i) choose a prior, ii) apply a learning algorithm; and iii) add random perturbations (from some known distribution) to the learned parameters such that we ensure tractability of the KL divergence. Following this recipe, Neyshabur et al. [40] established the following generalization bound (Lemma 1) when the change in the output of the model due to perturbations over the learned parameters is small with high probability.

Lemma 1 (Neyshabur et al. [40]) . Let g w ( x ) : X → R K be any model with parameters w , and let P be any distribution on the parameters that is independent of the training data. For any w , we construct a posterior Q ( w + u ) by adding any random perturbation u to w , s.t., P u (max x ∈X | g w + u ( x ) -g w ( x ) | ∞ < γ 4 ) > 1 2 . Then, for any γ, δ > 0 , with probability at least 1 -δ over an i.i.d. sizem training set S drawn according to D , for any w , we have:

3 Generalized PAC-Bayes

This section first presents a general procedure for obtaining generalization bounds for heterogeneous models, i.e., going beyond spectrally-normalized layers and architecture-specific models (as in [33, 40]). Then, we show how to leverage such a procedure to extend existing bounds in the literature and to obtain the first generalization bound for PersLay.

Our next result (Lemma 2) applied perturbation-based PAC-Bayes bounds to arbitrary models with (possibly) non-homogeneous layers. To achieve this generality, we carefully contrasted results in [33, 40, 48] to identify the conditions (Equations 6, 7, 8, and 9) that are sufficient to subsume the considered models as well as to extend to a broader class of models. In Section 4, we will also exploit Lemma 2 in the analysis of different combinations of neural models (e.g., GNNs and PersLay).

General recipe for PAC-Bayesian bounds for heterogeneous models

Lemma 2. Let f w : X ↦→ R K be a model with parameters w = vec { W 1 , ..., W n } . If there exists T ∈ R + and a sequence ( S i ) i ∈ [ n ] with S i ∈ R + both of which may depend on w , and parameter-independent C 1 , C 2 ∈ R + and sequence ( η i ) i ∈ [ n ] with η i ∈ (0 , 1] such that:

the output change can be bounded under a small perturbation of the parameters, i.e., for all u = vec { U 1 , ..., U n } with ∥ U i ∥ 2 ≤ η i S i :

the following auxiliary conditions hold:

Then, for any γ, δ > 0 with probability at least 1 -δ over the choice of training sets S with m i.i.d. samples drawn according to some distribution D , we have:

where h is the maximum dimension across the matrices ( W i ) i ∈ [ n ] .

the output is bounded by C 1 T :

Proof sketch. We build on the main result in [40] by extending it to a broader context. The general idea involves employing Lemma 1. Following [40], we define the prior distribution P as an isotropic Gaussian with variance σ 2 and the posterior distribution Q as a shifted isotropic Gaussian with the same variance. To achieve a tighter bound, it is essential to maximize σ (since the KL-divergence scales as O ( 1 / σ 2 ) ); consequently, σ should be determined based on the parameter β . However, since P must remain independent of the learned weights, we set σ according to an approximation of the learned weights. Specifically, we define β = T ∑ i 1 / S i and at first consider only w such that β fall within the range | β -˜ β | ≤ 1 / 2 β for some arbitrary ˜ β , an approximation. We then select σ based on this approximation, ˜ β . At this point we can apply Lemma 1 for all w such that β falls into the defined earlier interval. To account for other values of β , we establish a finite grid across the relevant β values and choose an appropriate ˜ β for each interval on the grid. Finally, a union-bound argument across all ˜ β values provides the final result. Although Equation 7 and Lemma 1 have their own constraints on the random perturbation, the above steps outline a method to set the variance σ that satisfies these constraints and maintains independence from the learned weights.

Discussion. We note that Lemma 2 requires choosing values for the variables ( S i ) i ∈ [ n ] and T . In this regard, one might set S i as the spectral norm of the weight matrix W i , i.e., S i = ∥ W i ∥ 2 , and make T equal to ∏ i S i . In this case, the condition in Equation 8 is satisfied - the geometric mean is always smaller than or equal to the arithmetic mean. Regarding the variables ( η i ) i , a typical choice is to set η i = O ( 1 / n ) . By doing so, our bound implicitly depends on n also through η i .

The role of Equation 6 is to constraint w to non-trivial parameter spaces. In particular, if T is too small, the magnitude of the model output might not be sufficient to distinguish different inputs up to a margin γ . In this case, the model would have large empirical loss. In turn, Equation 7 is directly associated with the condition in Lemma 1, enabling us to use it.

As discussed, Neyshabur et al. [40] assume spectrally-normalized weight matrices. To avoid this assumption, we introduce the conditions in Equation 8 and Equation 9, which allow us to pick perturbations that meet the condition in Equation 7, again justifying the application of Lemma 1.

The bound also includes a somewhat unconventional term, max { 1 , ∥ w ∥ 2 2 } . While this technical term allows for a more concise proof, we note that it does not impose suboptimality. More specifically, in most real-world cases, the squared norm of w is greater than 1. See Appendix I for a discussion.

Applying Lemma 2 to MLPs and GNNs. As previously mentioned, using Lemma 2 involves defining the variables T, ( S i ) i , ( η i ) i , C 1 , C 2 to meet all conditions in Lemma 2's statement. Typically, this definition comes naturally from the perturbation analysis of the model. To illustrate the power of Lemma 2, Table 2 shows how we can apply it using the perturbation analysis for MLPs and GNNs provided in [33, 40, 48]. Detailed proofs are given in Appendix E. Importantly, we note that we do not make any additional assumption beyond those in the original papers - we state all assumptions before each proof in the Appendix for clarity.

Notably, our approach generalizes results by Neyshabur et al. [40] and Liao et al. [33] to MLPs/GCNs with different activation functions - note that the original works only consider ReLU activations. For inherently non-homogeneous models like MPGNNs, our method leads to tighter bounds in several settings. We provide a comparison between our bounds and previous ones in Appendices F and G.

Applying Lemma 2 to PersLay. Next, Corollary 1 provides a PAC-Bayes bound for PersLay under fixed filtration functions - see Appendix H for a discussion about filtration functions. To the best of our knowledge, this is the first generalization result for vectorization schemes of persistence diagrams. Again, the proof consists of verifying if the requirements of Lemma 2 are met. For readability, we omit the proof and provide only the constant weighting function case in the main text, proof as well as the arbitrary weighting function case can be found in Appendix E. The perturbation analysis of PersLay is also given in the supplementary material (Lemma 10).

Corollary 1 (PersLay with constant weighting function) . Let f w : G ↦→ R k with w = vec { W ( φ ) } be a PersLay where W ( φ ) denotes the parameters of the point-transformation function, φ . Let Dg be a mapping from graphs to (extended) persistence diagrams with a fixed filtration function and B such that max G ∈G max p ∈ Dg( G ) ∥ p ∥ 2 ≤ B , then f w satisfies the requirements of Lemma 2 with:

where and

PersLay's special cases. Carrière et al. [5] designed PersLay with flexibility in mind to subsume commonly used persistence vectorization schemes in the literature. Consequently, we can obtain bounds for these schemes - we provide the values of C 2 (divided by 2) which is enough to compare the schemes since everything else is the same in the constant weighting function case:

where card is the max G ∈G Dg( G ) . If "images" and "silhouettes" use "sum" as an aggregating function, then our generalization analysis suggests that " k -landscapes" would have stronger guarantees. If these schemes use "mean" as an aggregating function, then C 2 for " k -landscapes" would be at most C 2 for "silhouettes", and the result of comparison of " k -landscapes" and "images" could be in favor of both "landscapes" and "images" depending on chosen parameters τ and B .

4 Compositional PAC-Bayes

In this section, we present two lemmas (Lemmas 3 & 4) that allow us to compose models satisfying Lemma 2 requirements. At the end of the section (Corollary 2), we showcase our framework by

getting generalization bounds for combinations of GNNs and PersLay. For readability, here we provide informal statements and defer the formal ones to the Appendix (Lemma 5, Lemma 6).

In particular, Lemma 3 establishes that the composition of MLPs with models that satisfy Lemma 2 also satisfy it. As a result, we can derive PAC-Bayes bounds for heterogeneous models that leverage MLPs using our framework in a straightforward way. This is particularly relevant since deep learning models often employ learnable feature extractors followed by MLPs as classification heads.

Lemma3 (Informal; Composition with MLP) . Let f be an MLP and g be a model satisfying Lemma 2 requirements, then f ◦ g also satisfies Lemma 2 requirements.

In addition, we show in Lemma 5 (Appendix) that this result also extends to an arbitrary number of models beyond MLPs. In particular, the result holds whenever we can upper bound output deviations due to perturbations on parameters and inputs, i.e., sup x ∈X ∥ f w + u ( x +∆ x ) -f w ( x ) ∥ 2 is bounded.

Our next lemma suggests that models satisfying Lemma 2 requirements are closed under parallel concatenation. We note that combining two (or more) models in parallel is also a common design choice in deep learning. For instance, this encompasses persistence-augmented GNNs [22] and ensemble methods [13].

Lemma 4 (Informal; Models in parallel) . Let f 1 , f 2 be two models satisfying Lemma 2 requirements and g be an aggregating Lipschtiz function. Then, g ( f 1 ( · ) , f 2 ( · )) also satisfies Lemma 2 requirements.

We also provide a generalization of this lemma in the Appendix (Lemma 6) for n > 2 models in parallel. It leads to tighter bounds than a naive 2-by-2 sequential application of Lemma 4.

Corollary 2 (Informal) . By combining the results for MLPs, GCNs, MPGNNs (Table 2) and that for PersLay (Corollary 1) with Lemma 3 and Lemma 4, we can get generalization bounds on various compositions of these models. In particular, for GNNs with persistence (see Figure 1), we have

T for the overall model scales with the product of PersLay's and GNN's T variables;

C 1 and C 2 of the overall model scale linearly with C 1 , C 2 of PersLay and GNN.

Despite the generality of our results, Corollary 2 demonstrates the benefits of our framework in the domain of graph representational learning. To the best of our knowledge, this the first result providing generalization guarantees for graph neural networks combined with persistence vectorization schemes. Furthermore, our findings can aid practitioners in making informed architectural decisions to enhance the generalizability of their models. Specifically, in the case of combining PersLay with GNNs, a tighter bound can be achieved by selecting a PersLay dimension that is considerably smaller than the GNN dimension. Failing to do so may result in a bound dependency on the width of the form O ( h √ ln h ) rather than O ( √ h ln h ) . Additionally, we recommend using aggregation functions such as "mean" or " k -max" instead of "sum" as the latter introduces a term max G ∈G card ( G ) to the bound, which may be large in practical scenarios.

5 Related Works

Expressivity and generalization of GNNs. GNNs have achieved state-of-the-art performance across various applications [16, 28, 45, 51, 54], and have garnered significant attention. Maron et al. [35], Xu et al. [55] analyzed the representational power of GNNs in terms of the 1-WL test, revealing theoretical limits on their expressivity. This has motivated a surge of works aiming to go beyond 1-WL with GNNs [e.g., 32]. Regarding generalization, Scarselli et al. [46] provided upper bounds on the order of growth of VC-dimension for GNNs. Garg et al. [14] presented the first data-dependent generalization bounds for GNNs via Rademacher complexity. Recently, Morris et al. [39] employed the WL test alongside VC-dimension to gain insights about the generalization performance of GNNs. For details about the expressivity and learning of GNNs, we refer to Jegelka [24].

Learning theory and PH. Birdal et al. [3], Dupuis et al. [8] and Chen et al. [7] investigate connections between learning theory and topological data analysis. In particular, Birdal et al. [3], Dupuis et al. [8] explored the concept of PH dimension as a complexity measure to analyze generalization. Chen et al. [7] proposed a topological regularizer to simplify decision boundaries by penalizing non-essential topological features. In contrast, we apply learning theory to derive data-dependent generalization bounds for arbitrary heterogeneous layers, specifically targetting persistence-aware GNNs, and introduce a regularizer informed by these bounds to guide the design of robust and generalizable models.

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

Generalization

ρ

: 0.78

±

0.08

DHFR

Generalization gap

Spectral norm

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

ρ

: 0.95

±

0.02

MUTAG

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

ρ

: 0.89

±

0.06

PROTEINS

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

ρ

: 0.94

±

0.07

NCI1

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

ρ

: 0.79

±

0.04

IMDB-BINARY

2

6

2

8

Width,

h

0

.

0

0

.

5

1

.

0

Generalization gap

ρ

: 0.91

±

0.07

DHFR

Empirical gap

Our bound

2

6

2

8

Width,

h

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ρ

: 0.77

±

0.12

MUTAG

2

6

2

8

Width,

h

0

.

0

0

.

5

1

.

0

ρ

: 0.95

±

0.03

PROTEINS

2

6

2

8

Width,

h

0

.

0

0

.

5

1

.

0

ρ

: 0.98

±

0.02

NCI1

2

6

2

8

Width,

h

0

.

0

0

.

5

1

.

0

ρ

: 0.95

±

0.04

IMDB-BINARY

PAC-Bayes. The PAC-Bayes framework [37, 38] allows us to leverage knowledge about learning algorithms and distributions over the hypothesis set to achieve tighter generalization bounds. Remarkably, Neyshabur et al. [40] presented a generalization bound for feedforward networks in terms of the product of the spectral norm of weights using a PAC-Bayes analysis. Liao et al. [33] provided PAC-Bayes bounds for GNNs, and Sun and Lin [48] enhanced their analysis considering the adversarial case as well. In a seminal work, Dziugaite and Roy [9] optimized PAC-Bayes bounds directly to obtain non-vacuous generalization bounds for deep stochastic neural networks.

6 Experiments

To illustrate the practical relevance of our analysis, we now consider the generalization of persistenceaware models on real-world datasets, and report results for regularized models based on our bounds. In particular, we conduct two main experiments. The first one aims to analyze how well our bounds capture generalization gaps as a function of model variables. The second assesses to which extent a structural risk minimization algorithm that uses our bound on the weights spectral norm improve generalization compared to empirical risk minimizers. We implemented all experiments using PyTorch [42], and implementation details are given in the Appendix J. Our code is available at https://github.com/Aalto-QuML/Compositional-PAC-Bayes .

Datasets and evaluation setup. We use six popular benchmarks for graph classification: DHFR, MUTAG, PROTEINS, NCI1, IMDB-BINARY, MOLHIV, which are available as part of TUDatasets [26] and OGB [21]. We use a 80/10/10% (train/val/test) split for all datasets when we perform model selection. Here, we consider both PersLay Classifiers and GNNs with persistence models with constant weight functions and Gaussian point transformations. For the experiments with GNNs, we kept only the larger datasets (and added results for the NCI109 dataset). Regarding filtration functions, we closely follow [5] and use Heat kernels with parameter values equal to 0 . 1 and 10 .

Dependence on model components . Figure 2 and Figure 4 show the generalization gap (measured as L D , 0 -ˆ L S ,γ =1 ) and the bound on the weights spectral norm ( T from Lemma 2) over the training epochs for PersLay Classifier and GNNs with persistence, respectively. To evaluate how well our bound captures the trend observed in the empirical gap, we compute correlation coefficients between the two sequences across different seeds and report their mean and standard deviation for each dataset. Overall, the coefficients are greater than 0.7 in 7 out of 9 experiments, indicating a good correlation.

Figure 3 shows the empirical gap and our estimated bound as a function of the model's width for the PersLay classifier. Again, we compute correlation coefficients between the two curves and find they are highly correlated (with an average correlation above 0 . 91 on 4 out of 5 datasets). Also, we note that these curves are obtained at the final training epoch. We report additional results across different epochs and hyper-parameters in the supplementary material. Again, these results validate that our theoretical bounds can capture the trend observed in the empirical generalization gap.

1

1000

2000

Epoch

0

.

0

0

.

5

1

.

0

Generalization

ρ

: 0.77

±

0.10

NCI1

Generalization gap

Our bound

1

1000

2000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

ρ

: 0.68

±

0.15

NCI109

1

1000

2000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

ρ

: 0.80

±

0.13

PROTEINS

1

1000

2000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

ρ

: 0.73

±

0.16

IMDB-BINARY

Regularizing PersLay. We compare variants of PersLay trained via ERM (empirical risk minimization) and a regularized version with loss given by ˆ L S , 1 + α r √ n 2 ln n ∥ w ∥ 2 2 β 2 ; α r is a hyper-parameter that balances the influence of the two terms and β = T ∑ i 1 / S i -see proof sketch of Lemma 2 in Section 3. This is similar to a weight-decay regularization approach, with the spectral norm of weights appearing in β . Here, we consider models with n = 1 or 2 , selected via hold-out validation. We note that PersLay classifier does not use node features, it only exploits graph structures.

±

±

±

±

±

±

Table 3 reports accuracy results (mean and standard deviations) computed over five runs. Overall, the regularized approach significantly outperforms the ERM variant despite the use of small-sized networks. On 5/6 datasets, PersLay with spectral norm regularization is the best-performing model.

Regularizing GNNs with persistence. We now evaluate the impact of using our bound to regularize different GNNs combined with persistent homology (PersLay) in parallel mode. We consider GCN [28], GraphSage [16], and GIN [55] architectures. Table 4 reports the test classification error and the generalization gap on the NCI, NCI109, and PROTEINS datasets - mean and standard deviation obtained over five independent runs. For our regularizer, we select the optimal penalization factor α r ∈ { 1e-5, 1e-6, 1e-7, 1e-8 } using the validation set. Overall, the results show that the regularized methods achieve smaller generalization gaps and slightly lower classification errors. In particular, our spectral regularizer leads to a significant drop in generalization gap in all experiments.

±

±

±

±

7 Conclusion, Broader Implications, and Limitations

We derive the first generalization bounds for neural networks that appeal to persistent homology for graph learning. Notably, the analyzed framework (PersLay) offers a flexible and general way to extract vector representations from persistence diagrams. Due to this generality, our analysis covers several methods available in the literature. The developed framework also allows to analyze composite models like, GNNs combined with PersLay. Our constructions involve a perturbation and generalization behavior analysis of non -homogeneous networks in rather general setting, which poses specific technical challenges.

While we provide valuable insights and methodologies, we would like to underscore the need for future investigations to delve into PH schemes that encompass parametrized filtration functions. Nonetheless, while some works showed gains using learnable filtrations [20], others have reported no benefits and advocated fixed functions instead [5, 34]. Moreover, the tightness of our bounds can further be improved since there is still considerable gap between empirical results and the theoretical one. By shedding new light on the generalization of machine learning models based on persistent homology, we hope to contribute to the community by providing key insights about the limits and power of these methods, paving the path to further theoretical developments on PH-based neural networks for graph representation learning.

Acknowledgments

VG acknowledges support from the Research Council of Finland (grant decision 342077) for the project 'Human-steered next-generation machine learning for reviving drug design', and the Jane and Aatos Erkko Foundation (grant 7001703) for 'Biodesign: Use of artificial intelligence in enzyme design for synthetic biology'. VG also thanks the Finnish Ministry for Education and Culture for their support via the 'MEC Global Programme pilot USA' initiative. AS acknowledges the support from the Conselho Nacional de Desenvolvimento Científico e Tecnológico CNPq (404336/2023-0) and the Silicon Valley Community Foundation through the University Blockchain Research Initiative (Grant #2022-199610). KB thanks Aalto University for the support during the internship.

References

Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence Images: AStable Vector Representation of Persistent Homology. Journal of Machine Learning Research , 18(8):1-35, 2016.

M.E. Aktas, E. Akbas, and A.E Fatmaoui. Persistence homology of networks: methods and applications. Applied Network Science , 2019.

Tolga Birdal, Aaron Lou, Leonidas Guibas, and Umut ¸ Sim¸ sekli. Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

P. Bubenik. Statistical topological data analysis using persistence landscapes. Journal of Machine Learning Research , 16:77-102, 2015.

Mathieu Carrière, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda. PersLay: A Neural Network Layer for Persistence Diagrams and New Graph Topological Signatures. In Artificial Intelligence and Statistics (AISTATS) , 2020.

Frédéric Chazal, Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, and Larry Wasserman. Stochastic Convergence of Persistence Landscapes and Silhouettes. In Proceedings of the Thirtieth Annual Symposium on Computational Geometry , 2014.

Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. A Topological Regularizer for Classifiers via Persistent Homology, October 2018.

Benjamin Dupuis, George Deligiannidis, and Umut ¸ Sim¸ sekli. Generalization Bounds with Data-dependent Fractal Dimensions. arXiv e-prints , 2023.

Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Conference on Uncertainty in Artificial Intelligence (UAI) , 2017.

H. Edelsbrunner and J. Harer. Computational Topology - an Introduction. American Mathematical Society, 2010.

Herbert Edelsbrunner and John Harer. Persistent homology-a survey. In Jacob E. Goodman, János Pach, and Richard Pollack, editors, Contemporary Mathematics , volume 453, pages 257-282. American Mathematical Society, 2008.

Pascal Esser, Leena Chennuru Vankadara, and Debarghya Ghoshdastidar. Learning theory can (sometimes) explain generalisation in graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

M.A. Ganaie, Minghui Hu, A.K. Malik, M. Tanveer, and P.N. Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence , 115, 2022.

Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and Representational Limits of Graph Neural Networks. In International Conference on Machine Learning (ICML) , 2020.

J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML) , 2017.

William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Conference on Neural Information Processing Systems (NeurIPS) , 2017.

Felix Hensel, Michael Moor, and Bastian Rieck. A Survey of Topological Machine Learning Methods. Frontiers in Artificial Intelligence , 4, 2021. ISSN 2624-8212.

Christoph Hofer, Roland Kwitt, Marc Niethammer, and Andreas Uhl. Deep Learning with Topological Signatures. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Christoph Hofer, Florian Graf, Bastian Rieck, Marc Niethammer, and Roland Kwitt. Graph filtration learning. In International Conference on Machine Learning (ICML) , 2020.

M. Horn, E. De Brouwer, M. Moor, Y. Moreau, B. Rieck, and K. Borgwardt. Topological graph neural networks. In International Conference on Learning Representations (ICLR) , 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Johanna Emilia Immonen, Amauri H. Souza, and Vikas Garg. Going beyond persistent homology using persistent homology. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Jan Reininghaus, Stefan Huber, Ulrich Bauer, and Roland Kwitt. A stable multi-scale kernel for topological machine learning. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015.

Stefanie Jegelka. Theory of graph neural networks: Representation and learning. ArXiv , abs/2204.07697, 2022.

Haotian Ju, Dongyue Li, Aneesh Sharma, and Hongyang R. Zhang. Generalization in graph neural networks: Improved pac-bayesian bounds on graph diffusion. In International Conference on Artificial Intelligence and Statistics , 2023.

Kristian Kersting, Nils M. Kriege, Christopher Morris, Petra Mutzel, and Marion Neumann. Benchmark data sets for graph kernels, 2016. URL http://graphkernels.cs. tu-dortmund.de .

D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR) , 2017.

Genki Kusano, Yasuaki Hiraoka, and Kenji Fukumizu. Persistence weighted Gaussian kernel for topological data analysis. In International Conference on Machine Learning (ICML) , pages 2004-2013, 2016.

John Langford and John Shawe-Taylor. PAC-Bayes & Margins. In Advances in Neural Information Processing Systems (NeurIPS) , volume 15. MIT Press, 2002.

Tam Le and Makoto Yamada. Persistence Fisher Kernel: A Riemannian Manifold Kernel for Persistence Diagrams. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

P. Li, Y . Wang, H. Wang, and J. Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Renjie Liao, Raquel Urtasun, and Richard Zemel. A PAC-Bayesian Approach to Generalization Bounds for Graph Neural Networks. In International Conference on Learning Representations (ICLR) , 2020.

Yuankai Luo, Lei Shi, and Veronika Thost. Improving self-supervised molecular representation learning using persistent homology. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

H. Maron, H. Ben-Hamu, H. Serviansky, and Y. Lipman. Provably powerful graph networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Sohir Maskey, Ron Levie, Yunseok Lee, and Gitta Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

David McAllester. Simplified PAC-Bayesian Margin Bounds. In Bernhard Schölkopf and Manfred K. Warmuth, editors, Learning Theory and Kernel Machines , Lecture Notes in Computer Science, pages 203-215, Berlin, Heidelberg, 2003. Springer.

David A. McAllester. PAC-Bayesian model averaging. In Proceedings of the Twelfth Annual Conference on Computational Learning Theory , COLT '99, pages 164-170. Association for Computing Machinery, July 1999.

Christopher Morris, Floris Geerts, Jan Tönshoff, and Martin Grohe. WL meet VC. In International Conference on Machine Learning (ICML) , 2023.

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks. In International Conference on Representation Learning (ICLR) , 2018.

Vahid Partovi Nia, Guojun Zhang, Ivan Kobyzev, Michael R. Metel, Xinlin Li, Ke Sun, Sobhan Hemati, Masoud Asgharian, Linglong Kong, Wulong Liu, and Boxing Chen. Mathematical Challenges in Deep Learning. ArXiv e-prints: 2303.15464 , 2023.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems (NeurIPS - Workshop) , 2017.

Giovanni Petri, Martina Scolamiero, Irene Donato, Francesco Vaccarino, Thomas Gilbert, Markus Kirkilionis, and Gregoire Nicolis. Networks and Cycles: A Persistent Homology Approach to Complex Networks . Proceedings of the European Conference on Complex Systems 2012. Springer International Publishing, 2013. ISBN 978-3-319-00394-8.

Bastian Rieck. On the Expressivity of Persistent Homology in Graph Learning. arXiv e-prints , (arXiv:2302.09826), 2023.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1):61-80, 2009.

Franco Scarselli, Ah Chung Tsoi, and Markus Hagenbuchner. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks , 108:248-259, 2018.

Milad Sefidgaran and Abdellatif Zaidi. Data-dependent Generalization Bounds via Variable-Size Compressibility. arXiv e-prints , 2023.

Tan Sun and Junhong Lin. Pac-bayesian adversarially robust generalization bounds for graph neural network. arXiv e-prints , 2024.

Joel A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics , 12(4):389-434, August 2012. ISSN 1615-3375, 1615-3383.

L. G. Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, November 1984. ISSN 0001-0782. doi: 10.1145/1968.1972.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR) , 2018.

S. Verma and Z.-L. Zhang. Stability and generalization of graph convolutional neural networks. In International Conference on Knowledge Discovery & Data Mining (KDD) , 2019.

Yogesh Verma, Amauri H Souza, and Vikas Garg. Topological neural networks go persistent, equivariant, and continuous. In International Conference on Machine Learning (ICML) , volume 235, 2024.

Feng Xia, Ke Sun, Shuo Yu, Abdul Aziz, Liangtian Wan, Shirui Pan, and Huan Liu. Graph Learning: A Survey. IEEE Transactions on Artificial Intelligence , 2(2):109-127, April 2021. ISSN 2691-4581. doi: 10.1109/TAI.2021.3076021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR) , 2019.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets, 2018.

Qi Zhao and Yusu Wang. Learning metrics for persistence-based summaries and applications for graph classification. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Qi Zhao, Ze Ye, Chao Chen, and Yusu Wang. Persistence Enhanced Graph Neural Network. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2020.

Yangze Zhou, Gitta Kutyniok, and Bruno Ribeiro. OOD link prediction generalization capabilities of message-passing GNNs in larger test graphs. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

A Notation

Table 5 summarizes the main mathematical symbols and abbreviations used in this work.

B Model descriptions

MLP.

where H k ∈ R h k are the vectors computed after layer k , and W k ∈ R h k -1 × h k , k ∈ [ n ] are the parameters in the k -th layer, and we let H 0 = x . ψ k , k ∈ [ n -1] are some element-wise non-linear Lip k -Lipschitz functions. Note that we change d (number of layers in Neyshabur et al. [40] notation) to n to be consistent with the number of parameters notation.

GCN.

where H k ∈ R | V |× h k , k ∈ [ n -1] are the node representations in each layer, H n ∈ R 1 × K is the readout, and W k ∈ R h k -1 × h k , k ∈ [ n ] are the parameters in the k -th layer. And we let H 0 = Z , where Z is the matrix constructed from node features, z v for v ∈ V . The matrix P G ∈ R | V |×| V | is related to the graph structure and ψ k ( · ) , k ∈ [ n -1] are some element-wise non-linear Lip k -Lipschitz mappings. Practically, for GCN, we take P G as the Laplacian of the graph, defined as ˜ D -1 / 2 ˜ A ˜ D -1 / 2 , where ˜ A is the adjacency matrix with +1 on the diagonal and ˜ D = diag ( ∑ | V | j =1 ˜ A ij , i ∈ [ | V | ] ) is the degree matrix of ˜ A . Note that we sightly changed (comparing to Liao et al. [33], Sun and Lin [48]) the notation to be consistent: instead of l , number of layers, we have n , instead of n , number of nodes, we have | V | and instead of σ i we have ψ i as activation functions.

MPGNN.

where k ∈ [ n -1] , H k ∈ R | V |× h k are node representations/states and H n ∈ R 1 × K is the output representation. Here we initialize H 0 = 0 . WLOG, we assume ∀ k ∈ [ n -1] , H k ∈ R | V |× h and M k ∈ R | V |× h since h is the maximum hidden dimension. C in ∈ R | V |×| E | and C out ∈ R | V |×| E | are the incidence matrices corresponding to incoming and outgoing nodes 1 respectively. Specifically, rows and columns of C in and C out correspond to nodes and edges respectively. C in [ i, j ] = 1 indicates that the incoming node of the j -th edge is the i -th node. Similarly, C out [ i, j ] = 1 indicates that the outgoing node of the j -th edge is the i -th node. g, ϕ, ρ are nonlinear mappings, e.g., ReLU and Tanh. Technically speaking, g : R h → R h , ϕ : R h → R h , and ρ : R h → R h operate on vector-states of individual node/edge. However, since we share these functions across nodes/edges, we can naturally generalize them to matrix-states, e.g., ˜ ϕ : R | V |× h → R | V |× h where ˜ ϕ ( X )[ i, :] = ϕ ( X [ i, :]) . By doing so, the same function could be applied to matrices with varying size of the first dimension. For simplicity, we use g, ϕ, ρ to denote such generalization to matrices. We denote the Lipschitz constants of g, ϕ, ρ under the vector 2 -norm as C g , C ϕ , C ρ respectively. We also assume g ( 0 ) = 0 , ϕ ( 0 ) = 0 , and ρ ( 0 ) = 0 and define the percolation complexity as C = C g C ϕ C ρ ∥ W 2 ∥ 2 following Garg et al. [14]. Note that we sightly changed (comparing to Liao et al. [33], Sun and Lin [48]) the notation to be consistent: instead of l , number of layers, we have n , instead of n , number of nodes, we have | V | and instead of W l , parameter matrix, we have W 3 .

C Proofs of main lemma and composition lemmas

Lemma 2. Let f w : X ↦→ R K be a model with parameters w = vec { W 1 , ..., W n } . If there exists T ∈ R and S i ∈ R for i ∈ [ n ] depending on W 1 , ..., W n ; C 1 , C 2 > 0; η i ∈ (0 , 1] for i ∈ [ n ] such that:

maximum norm of the output is bounded by T and C 1 :

maximum change of the output with perturbed weights, u = vec { U 1 , ..., U n } , is bounded if the perturbation is small, ∥ U i ∥ 2 ≤ η i S i :

arithmetic mean of S i inverses is at least the n th root of T inverse:

minimum η i is at most the ratio of C 1 and 2 C 2 :

then for any γ, δ > 0 with probability at least 1 -δ over the choice of training set S of size m sampled accordingly to some data distribution D we have the bound on the generalization gap:

where h is the upper bound for dimensions of W i .

Proof. First, we note that if w such that C 1 T ≤ γ 2 , then this imply for any i ∈ [ K ]

̸

where we used Equation 6 and the fact that every coordinate of the vector is at most ℓ 2 -norm of the vector. In this case the empirical loss would be equal to one, since no index i can satisfy f w ( x ) i ≥ γ +max i = j f w ( x ) and the inequality in Equation 10 becomes trivial since generalization error can not be greater than 1.

So from now on we consider w such that C 1 T ≥ γ 2 . Moreover we note that:

We denote

Following Neyshabur et al. [40] and Liao et al. [33], we consider the prior P = N ( 0 , σ 2 I ) and random perturbation u ∼ N ( 0 , σ 2 I ) . Note that the σ of the prior and the perturbation are the same

and will be set according to β . More precisely, we will set the σ based on some approximation ˜ β of β since the prior cannot depend on any learned weights directly. The approximation ˜ β is chosen to be a cover set which covers the meaningful range of β . For now, let us assume that we have a fixed ˜ β and consider β which satisfies | β -˜ β | ≤ εβ for some ε > 0 . Note that this also implies:

This setup is very like the setup in Neyshabur et al. [40] and Liao et al. [33]

Choosing σ . From Tropp [49] we know that if U ∈ R h × h , U ij ∼ N (0 , σ 2 ) , then

So applying it to U 1 , ..., U n (all of them are smaller than h × h ) and t = σ √ 2 h ln 4 nh we have:

For simplicity of notation we denote C t = √ 2 h ln 4 nh .

Using defined perturbation u = vec { U 1 , ..., U n } and Equation 23 combined with Equation 7 we get with probability at least 1 2 :

If and that

then the perturbation is bounded by γ 4 . However, we need to satisfy the condition of Equation 7, i.e. ∀ i ∈ [ n ] : ∥ U i ∥ ≤ η i S i . Let us find C σ ≥ 1 such that:

̸

̸

In this case C σ ≥ 1 because of Equation 9 and continuing Equation 26:

where (i) comes from the fact that ¯ η is the minimum among η i s, gives us that ∀ i ∈ [ n ] : ∥ U i ∥ 2 ≤ η i S i . Plugging this into expression for σ , Equation 25, we get:

At this point, we selected a prior, P = N ( 0 , σ 2 I ) , and perturbation distribution, Q = N ( w , σ 2 I ) , such that the conditions of Lemma 1 can be satisfied.

Getting the bound for weights | β -˜ β | ≤ εβ . Now, we can compute the KL-divergence and get the bound using Lemma 1 for case when β is around ˜ β , i.e. w such that | β -˜ β | ≤ εβ .

where (i) comes from σ definition, Equation 28, (ii) comes from the upper-bound on ˜ β , Equation 22. We are ready to put this into Lemma 1:

where we upper-bounded ∥ w ∥ 2 2 as max { 1 , ∥ w ∥ 2 2 } to simplify further derivations.

(i) comes from Equation 21. (ii) we just multiplied and divided on η i S i . (iii) we plugged in the expression for β , Equation 20. (iv) 1 + ∑ k = i S i S k ≥ 1 . (v) comes from Equation 19

We choose

Union bound. Finally, we need to consider multiple choices of ˜ β so that for any β , we can bound the generalization error like Equation 30. In order to do this we (i) find interval of reasonable β s, (ii) define a covering and upper-bound the number of balls in the covering, (iii) combine bounds for every ˜ β together and get the final result.

If β is too large, the KL-divergence would be too large and generalization gap would be greater than 1 which thereby trivialize the bound since the generalization error cannot be greater than 1.

where (i) holds since we remove multipliers that are greater than 1 and ln m δ which is ≥ 0 and appears in sum, (ii) comes from the fact that ¯ η ≤ 1 .

On the other hand, if the β is too small, then the empirical loss would be too large since the model would not be able to classify with margin , γ , as it was shown in Equation 19. Using this equation and Arithmetic versus Geometric mean we get the following inequality:

So, if β ≥ γ √ m C 1 , then the bound becomes trivial.

where (i) comes from Equation 8 and (ii) comes from Equation 19.

Combining lower (Equation 32) and upper (Equation 31) bound on β we get:

To satisfy the condition | β -˜ β | ≤ εβ we can take εn ( γ 2 C 1 ) n -1 n as the radius of the covering, C . In this case let us derive the upper bound for the size of the covering, | C | :

Let us consider cases:

First inequality holds because function a 1 /x x is monotonously decreasing on the interval (0 , + ∞ ) . So, we can combine:

It is left to apply the union bound argument for the events of Equation 30 happening with ˜ β taking value from i th ball. Let us denote such event E i :

Hence, if we set δ ′ = δ | C | , then with probability at least 1 -δ for all w we have:

Note that we substituted ε with 1 2 and now max { 1 , 1 C 1 } appearing under logarithm comparing to Equation 30.

Lemma 5 (Generalization of Lemma 3) . Let f (1) w 1 : X 1 ↦→ R h 1 , ..., f ( k ) w k : X k ↦→ R h k for k ∈ N with w i = vec { W ( i ) 1 , ..., W ( i ) n i } and h 1 = K be some models that we can compose , i.e. f = f (1) ( f (2) ( ... ( f ( k ) ) ... )) . If there exists for j ∈ [ k ] : T ( j ) depending on W ( j ) 1 , ..., W ( j ) n j ; for j ∈ [ j ] : S ( j ) 1 , ..., S (1) n j ∈ R depending on W ( j ) 1 , ..., W ( j ) n j ; η (1) 1 , ..., η (1) n 1 , ..., η ( k ) 1 , ..., η ( k ) n k ∈ (0 , 1] and C (1) 1 , ..., C ( k ) 1 ∈ R , C (1) 2 , ..., C ( k ) 2 ∈ R such that:

maximum output for every model is bounded as follows:

and for j = k we can have weaker condition - the same as in Lemma 2, maximum perturbation of the output during small perturbation of weights, ∥ U ( j ) i ∥ 2 ≤ η ( j ) i S ( j ) i , for every model is bounded as follows:

and for j = k we can have weaker condition - the same as in Lemma 2, arithmetic mean of inverses of S ( j ) i is greater than n j -root of T ( j ) :

and ¯ η ( j ) := min i ∈ [ n j ] η ( j ) i is upper-bounded by the ration C ( j ) 1 and C ( j ) 2 :

we denote n = k ∑ j =1 n j , then f meets requirements of Lemma 2 with

Proof. We denote as f k : j composition of models from k to j First we test Equation 6 by plugging in Equation 37 for every model:

Next, we test Equation 7 by plugging in Equation 38 and Equation 37 for every model:

To test Equation 8 we use arithmetic versus geometric mean inequality:

̸

̸

It is left to verify Equation 9. As one can notice C 1 2 C 2 ≥ C (ind) 1 C (ind) 2 ≥ ¯ η .

Lemma 6 (Generalization of Lemma 4) . Let f (1) w 1 : X ↦→ R h 1 , ..., f ( k ) w k : X ↦→ R h k with w 1 = vec { W (1) 1 , ..., W (1) n 1 } , ..., w k = vec { W ( k ) 1 , ..., W ( k ) n k } be some models satisfying Lemma 2 conditions. If Agg : R h 1 × ... × R h k ↦→ R K such that:

for some A > 0 and Agg( 0 , ..., 0 ) = 0 . Also we denote n = k ∑ j =1 n j , then Agg( f (1) w 1 ( · ) , ..., f ( k ) w k ( · )) satisfies Lemma 2 conditions with either:

or with

where ind = arg min j ∈ [ k ] ¯ η ( j ) .

Proof. First, we test Equation 6 for composite model by applying assumption about Agg and its Equation 6 for every f ( j ) :

To test Equation 7 for composite model we again apply assumption about aggregation function and its Equation 7 version for every f ( j ) :

To test Equation 8 for composite model we employ arithmetic vs geometric mean together with Equation 8 for every model:

where the last inequality comes from the fact that k ∏ j =1 T ( j ) ≤ T

It is left to test Equation 9. As one can notice in either case C 1 2 C 2 ≥ C (ind) 1 2 C (ind) 2 ≥ ¯ η

D Perturbation analysis

Lemma 7. Let w = vec { W 1 , ..., W n } be a vector of weight matrices of an n -layer MLP, f w : X ↦→ R K . Let ψ i be a Lip i -Lipschitz activation function after layer i , for i ∈ [ n -1] . Then for any input and input perturbation, x , ∆ x ∈ X , weight perturbation, u = vec { U 1 , ..., U n } , and constants, η 1 , ..., η n , such that for i ∈ [ n ] : ∥ U i ∥ 2 ≤ η i ∥ W i ∥ 2 , we have two inequalities:

where we denote Lip n = 1

Proof. The proof follows the Neyshabur et al. [40] with some modifications concerning the ∆ x and Lip i and η i .

We denote truncated f after i +1 th layer as f ( i +1) . First, we provide the bound on the norm of the output after i +1 th layer:

where (i) is the definition of MLP, (ii) comes from the definition of operator norm, (iii) comes from Lipschitzness. Unrolling the recursion of Equation 49 will get us (we denote f (0) w ( x ) to be x ):

Now let us provide the bound for the perturbation after i +1 th layer. We denote this perturbation as ∆ i +1 :

where (i) comes from the definition of ∆ i +1 , (ii) comes from the defintion of an MLP, (iii) add and substract U i +1 ψ i ( f ( i ) w ( x )) , (iv) comes from triangle inequality combined with operator norm definition, (v) comes from the definition of η i +1 , Lipschitzness and Equation 50, (vi) comes from multiplying and dividing by ∥ W i +1 ∥ 2 the second term. Now we unroll the recursion of Equation 51

where (i) comes from Equation 51, (ii) comes from unrolling the recursion ∆ i , (iii) comes from the fact that 1 + η k ≥ 1 .

Lemma 8 (GCN perturbation analysis) . Let f w : X ↦→ R K with w = vec { W 1 , ..., W n } be a n -layer GCN model. Let ψ i be Lip i activation functions after layer i ∈ [ n -1] . If u = vec { U 1 , ..., U n } , B ∈ R are such that: ∀ i ∈ [ n ] : ∥ U i ∥ 2 ≤ 1 n ∥ W i ∥ 2 and ∀ G ∈ X , ∀ v ∈ G : | z v | 2 ≤ B , and ∀ G ∈ X , G is simple and have maximum degree of d , then for any G = ( V, E, z ) ∈ X

where Z ∈ R | V |× d z is the matrix consisting of z v for v ∈ V , and P G is a Laplacian of the graph G which is equal to ˜ D -1 / 2 ˜ A ˜ D -1 / 2 where ˜ A is an adjacency matrix with +1 on the diagonal and

.

Proof. The proof follows [48] except for the fact that we add Lip i to the bound. This change is straightforward, however, for the completeness of the picture we provide this proof here. The detailed description of the architectures can be found in Appendix B.

First we prove two helpful propositions

Proposition 1. For any matrix A ∈ R n × m , B ∈ R m × p , we have,

Proof. Let x ⊤ i , a ⊤ i be i th row of AB and A respectively, then we have:

Proposition 2. For any undirected graph G = ( V, E ) , let A ∈ R | V |×| V | be the adjacency matrix, D = diag( D 1 , . . . , D | V | ) be the degree matrix, and d = max i ∈ [ | V | ] { D i } be the maximum degree,

where D i = ∑ | V | j =1 A ij , i ∈ [ | V | ] . Then we have,

( i ) ∥ A ∥ 2 ≤ d ;

Proof. For (i), by the definition of spectral norm, we have

For (ii), let ˜ E = E ∪ { ( i, i ) | i ∈ V } be the edge set associated with the adjacency matrix ˜ A . By the definition of spectral norm, we have

Before proving the first inequality we note that 1 √ | V | ∥ X ∥ F ≤ B and ∥ P G ∥ n -1 2 ≤ 1 , so the second inequality is rather straightforward. The last inequality comes from the fact that d ≥ 1 .

We denote the node representation in the j -th ( j ≤ l ) layer as

Adding perturbation u to the parameter w , that is, for the j -th ( j ≤ n ) layer, the perturbed parameters are W j + U j and denote H ′ j = f j w + u ( G ) , j ∈ [ n ] .

Upper Bound on the Node Representation. For any j < n ,

where the first inequality holds since ψ j is a Lipschitz and ψ j (0) = 0 , and the second and the last ones hold by Proposition 1. Then, unrolling the recursion and setting H 0 = X , we have

Upper Bound on the Change of Node Representation. For any j < | V | ,

Using the triangle inequality, we have

The first term can be bounded as

and the second term can be bounded as

Therefore,

Unrolling the recursion while simplifying notation as: ∥ H j -H ′ j ∥ F ≤ a j -1 ∥ H ′ j -1 -H j -1 ∥ F + b j -1 we get:

Plugging in Equation 54, we have:

Final Bound on the Readout Layer.

where in a last inequality we first apply that ∥ A ∥ 2 ≤ ∥ A ∥ F for any matrix A and then use Proposition 1.

Using Equation 54 and Equation 57, we have:

where we set Lip n = 1 for simplicity of notation and last inequality holds since 1 ≤ (1 + 1 n ) n ≤ e .

Our next lemma helps us to upper bound the PERSLAY's perturbation, when Agg = k -max . Lemma 9. Let X be an arbitrary finite set and f, g : X ↦→ R . Then we can say that:

Proof. Denote n : N ↦→ X by a function that maps natural number k to an element of X that would be on k th position in order sorted by f . Denote m as an analogous function but for g . Then, we are interested in the following expression: | f ( n ( k )) -g ( m ( k )) | . Let us rewrite it:

Now, the task is to prove that | g ( n ( k )) -g ( m ( k )) | ≤ 2 max x ∈ X | f ( x ) -g ( x ) | Let us consider four cases:

g ( n ( k )) > g ( m ( k )) and f ( n ( k )) ≥ f ( m ( k )) (Fig. 5). In this case ∃ i ∈ N such that f ( n ( i )) > f ( n ( k )) and g ( n ( i )) < g ( m ( k )) . Indeed, if none of the elements "to the right"

of n ( k ) moved "to the left" of m ( k ) , then "to the right" of m ( k ) , there are at least n -k +1 elements; however, there are must be exactly n -k elements.

g ( n ( k )) > g ( m ( k )) and f ( n ( k )) ≤ f ( m ( k )) (Fig. 6)

The rest of the cases can be handled analogously.

Lemma 10 (Perturbation analysis of PersLay) . Let f w : G ↦→ R k with w = { W ( ω ) , W ( φ ) } be a PersLay where W ( ω ) is a parameter vector (matrix) of weight function, ω , and W ( φ ) is a parameter vector (matrix) of point-transformation function, φ . Let Dg be a mapping from graphs to (extended) persistence diagrams with a fixed filtration function and B such that max G ∈G max p ∈ Dg( G ) ∥ p ∥ 2 ≤ B , then:

and for η ( ω ) and u = { U ( ω ) , U ( φ ) } such that ∥ U ( ω ) ∥ 2 ≤ η ( ω ) T ( ω ) , we have:

where A 1 = max G ∈G card (Dg( G )) if Agg is sum and A 1 = 1 if Agg is k -max or mean; A 2 = max G ∈G card (Dg( G )) if Agg is sum or A 2 = 3 if Agg is k -max or mean;

T ( ω ) = max { 1 , ∥ W ( ω ) ∥ 2 } ; and C ( ω ) , Lip ( ω ) are such that:

max G ∈G max p ∈ Dg( G ) | ω w ( p ) | 2 ≤ C ( ω ) T ( ω ) and max G ∈G max p ∈ Dg( G ) | ω w + u ( p ) -ω w ( p ) | ≤ Lip ( ω ) ∥ U ( ω ) ∥ 2

for any w and u .

Proof. First we prove the inequaliry about maximum output norm.

Maximum output norm.

where A 1 is max G ∈G card (Dg( G )) if Agg is sum and A 1 = 1 if Agg is k -max or mean.

From the Lemma statement we can upper bound ω w ( p ) :

Maximum norm of φ :

φ = Λ .

φ = Γ .

φ = Ψ .

Hence:

Hence:

Hence:

Combining all together we get the Equation 58.

Maximum perturbation of the PersLay output.

where A 2 is max G ∈G card (Dg( G )) if Agg is sum, A 2 is 3 if Agg is k -max by Lemma 9 and A 2 is 1 if Agg is mean.

By the Lemma statement we have:

and

Moreover, from Paragraph about max norm of φ :

Maximum perturbation of φ :

φ = Λ . Since g ( x ) = | x | is 1 -Lipschitz we have that:

Hence,

φ = Γ . Suppose g ( x, y ) = exp ( -x 2 + y 2 2 τ 2 ) . Then

by mean-value theorem for some x ′ , y ′ between x and x +∆ x and y and y +∆ y .

Let us find maximum of the gradient by every coordinate. Since the function is symmetric we need to do it only for one of the coordinates.

The maximum of the norm of the first coordinate of the gradient is achieving at the point t i, 1 = p 1 ± τ , and the gradient value at these points is at most 1 τe 1 / 2 . So we have:

φ = Ψ . In this case g ( x, y, z ) = p 1 x + p 2 y + z is max { B, 1 } -Lipschitz, so

Combining all together we have:

E Proofs of corollaries

Corollary 3. Let f w : X ↦→ R K with w = { W 1 , ..., W n } be an n -layer MLP with Lip i -Lipschitz activation functions ψ i , for i ∈ [ n -1] . Let B ∈ R be such that ∀ x ∈ X : ∥ x ∥ 2 ≤ B . Then f w satisfy requirements of Lemma 2 with:

Proof. To test Equation 6 and Equation 7 we use Lemma 7.

To check Equation 8 we apply arithmetic-geometric mean inequality:

̸

Corollary 4. Let f w : X ↦→ R k with w = { W 1 , ..., W n } be a n -layer GCN network with readout layer. Let ψ i for i ∈ [ n -1] be a Lip i -Lipschitz activation function. Let node feature of any graph be contained in ℓ 2 -ball of radius B , i.e. ∀ G ∈ X : ∥ z v ∥ 2 ≤ B for every node v and ∀ G ∈ X , G is simple and has maximum degree at most d -1 . Then f w satisfy requirements of Lemma 2 with:

if using perturbation analysis from Liao et al. [33]

or if using perturbation analysis from Sun and Lin [48]

Proof. To test Equation 6, Equation 7 we use Lemma 8. We can apply Lemma 8 because ¯ η = 1 6 n ≤ 1 n . To test Equation 8 we use arithmetic vs geometric mean inequality:

̸

It is left to test Equation 9

̸

It is left to show that Equation 9 holds:

̸

if d C ̸ = 1 :

if d C = 1 :

Proof. From [33] (Lemma 3.3) we know that:

where η = max { ∥ U 1 ∥ 2 ∥ W 1 ∥ 2 , ∥ U 2 ∥ 2 ∥ W 2 ∥ 2 , ∥ U 3 ∥ 2 ∥ W 3 ∥ 2 } ≤ 1 n . In our case ¯ η ≤ 1 6 n , so we can apply these inequalities. Note that in our notation we have n -layer MPGNN and instead of W l we have W 3 (and instead of U l we have U 3 ).

And

Eq. (76) and Eq. (68) respectively.

Equation 6 follows from Equation 64.

To test Equation 7 we note that:

Now Equation 7 follows from Equation 63 and Equation 65.

To test Equation 8 we employ arithmetic vs geometric mean inequality:

where in case d C = 1 , S 2 ≤ 1 and in case d C ̸ = 1 , S 2 ≤ d C and ( d C ) n -1 -1 d C1 ≥ d C for n > 2 .

,

To test Equation 9 we provide the following derivation:

Corollary 1. Let f w : G ↦→ R k with w = { W ( ω ) , W ( φ ) } be a PersLay where W ( ω ) is a parameter vector (matrix) of weight function, ω , and W ( φ ) is a parameter vector (matrix) of point-transformation function, φ . Let Dg be a mapping from graphs to (extended) persistence diagrams with a fixed filtration function and B such that max G ∈G max p ∈ Dg( G ) ∥ p ∥ 2 ≤ B , then f w satisfy requirements of Lemma 2

with:

where A 1 = max G ∈G card (Dg( G )) if Agg is sum and A 1 = 1 if Agg is k -max or mean; A 2 = max G ∈G card (Dg( G )) if Agg is sum or A 2 = 3 if Agg is k -max or mean;

T ( ω ) = max { 1 , ∥ W ( ω ) ∥ 2 } ; and C ( ω ) , Lip ( ω ) are such that:

for any w and u .

Proof. To test Equation 6 and Equation 7 we apply Lemma 10. η ( ω ) in Lemma 10 is arbitrary, so we can use this lemma with η ( ω ) = 1 and in this case 1 + η ( ω ) = 2 , so our choice of C 2 works.

To test Equation 8 we use arithmetic vs geometric mean inequality:

It is left to test Equation 9

F Comparing bounds with prior works

MLP & GCN The result is not as tight as in [40] because of the homogenity assumption that they do. Specificaly, matching our result to theirs, we have B 2 ( h ln nh )¯ η -2 the same as B 2 n 2 ( h ln nh ) (instead of n they have d in their notation). We have log ( m δ max { 1 , 1 B } ) which is in most cases better than log m · n δ . However, the suboptimality of our bound comes from:

by Cauchy-Schwarz inequality. The most left expression is the term that is in Neyshabur et al. [40] and the most right expression is the one we have in our bound.

However, the advantage of our result is that it does not depend on the assumption of all activation functions being equal to ReLU and that we can compose it with results for other networks to get bounds for compositions of networks.

MPGNNs As we discussed in the main text the difference between our bound and the one in Liao et al. [33] comes from the dependency on the weights.

Their dependency: max { ζ -( l +1) , ( λξ ) l +1 l } , where ζ = min {∥ W 1 ∥ 2 , ∥ W 2 ∥ 2 , ∥ W l ∥ 2 } , λ = ∥ W 1 ∥ 2 ∥ W l ∥ 2 , ξ = C ϕ ( d C ) l -1 -1 d C1 . Let us show, how our bound depends on λ, ζ and ξ :

and

Note that λξ = T . Now let us analyze two cases:

So, in the case when d Lip ϕ Lip g Lip ρ is lower bounded by some constant we can conclude that asymptotically our bound is not inferior, than the one in Liao et al. [33].

G Dependency on Model Parameters and Hyperparameters

In the Table 6 we present the dependency of existing results and our result on the model parameters and hyperparameters.

H Learnable Filtration Functions

Fixed filtration functions dominate the PH/ML literature. The widespread use of learnable functions is a relatively recent phenomenon in PH-based ML, and usually runs orders of magnitude slower compared to non-learnable ones. Arguably, applying non-learnable functions still represents the mainstream approach in TDA.

Some works have explicitly advocated for fixed filtration functions (with learnable vectorizations) over learnable filtrations. Filtration functions can come in different flavors; for instance, they can rely on node degree [18], cliques [43], or node attributes [22]. Some of the popular options are parameter-free. Also, while some works showed gains using learnable filtrations [20], others have reported no benefits and adopted fixed functions instead [5, 34]. There is still no consensus about the significance of the gains associated with learnable filtration in many applications.

Perslay [5] uses fixed filtration functions. Despite the generality of our results, we provide specific bounds for PersLay, which employs fixed filtration functions.

Our work lays a strong foundation for analyzing learnable filtrations. One way to analyze PH with learnable filtration schemes could be to get upper bounds on perturbation of outputs in terms of the

filtration function parameters. This would additionally require an analysis of Wasserstein distances between persistence diagrams obtained with different parameters. We believe that for a specific class of graphs we can get modified upper bounds for perturbation with respect to filtration function parameters that would depend on Wasserstein distance of the same order. This additional analysis could be readily integrated into our framework to get generalization bounds for learnable filtrations.

I Discussion on the max over 1 and parameters norm.

From the first glance it may seem that introducing max { 1 , ∥ w ∥ 2 2 } is suboptimal; however, ∥ w ∥ 2 2 is greater than 1 in many real-world scenarios. For instance, suppose we have parameters within the range of O ( ε ) ; then the squared norm would be greater than 1 if the number of parameters exceeds 1 ε 2 . Considering a typical choice in the Deep Learning literature, ε = 0 . 01 , we need more than 10 , 000 parameters. For example, a two-layer MLP with a hidden dimension of 64 (also a typical choice) has more than 8 , 000 parameters, so in practice, ∥ w ∥ 2 2 ≥ 1 is usually true.

J Implementation details

J.1 Datasets

Table 7 reports summary statistics of the datasets used in this paper.

J.2 Models

For the experiments with PersLay Classifier, we closely follow the filtration functions used in [5]. In particular, we use Kernel heat functions with parameters t = 0 . 1 and t = 10 for the remaining datasets. Instead of processing each diagram type using separate models, we combine ordinary and extended diagrams for 0- and 1-dimensional features and apply a single model. We use mean aggregation function in all experiments, and Gaussian point transformations. For the feedforward part of PersLay, we apply ReLU activation functions. All models are trained with Adam [27] and learning rate of 10 -3 for 3000 epochs.

For the experiments with GNNs with persistence, we use graph isomorphism networks (GINs) with 2 layers and 64 hidden units. The models were trained for 2000 epochs using the Adam optimizer.

Dependence on model paramaters. Regarding the dependence on the spectral norm of weights, we reported results for a model with a final MLP (multilayer perceptron) of 2 hidden layers (3 layers in total) and width of 128 for all layers. The number of parameters of the point transformation was h = 100 . For the experiments on width vs. generalization gap, we used h = 100 , and 1 hidden layer with a varying width in { 32 , 64 , 128 , 256 , 512 } .

Regularizing PersLay. For the experiments regarding ERM and spectral norm regularizers, we perform model selection for (number of layers) l ∈ { 2 , 3 } , and α r ∈ { 10 -3 , 10 -4 , 10 -5 , 10 -6 } . Again, we use Gaussian point transformation, h = 100 , and width equals to 128. Our goal was to see if we could observe gains from the regularized version even for shallow neural networks.

Regularizing GNNs with persistence. Here, we consider GNNs with 64 hidden units of 64 (width) and 2 layers. We set the dimensionality of PersLay parameters equal to 100, Gaussian point transformation, and mean aggregation function. We apply hold-out model selection with penalty term α r ∈ { 10 -5 , 10 -6 , 10 -7 , 10 -8 } using the validation set.

Hardware. For all experiments, we use Tesla V100 GPU cards and consider a memory budget of 32GB of RAM.

2

6

2

8

Width,

h

-

0

.

5

0

.

0

0

.

5

1

.

0

Generalization gap

ρ

: 0.91

±

0.04

PROTEINS

Empirical gap

Our bound

2

6

2

8

Width,

h

0

.

0

0

.

5

1

.

0

ρ

: 0.93

±

0.04

NCI1

2

6

2

8

Width,

h

0

.

0

0

.

5

1

.

0

ρ

: 0.72

±

0.06

IMDB-BINARY

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

Generalization

ρ

: 0.72

±

0.07

PROTEINS

Generalization gap

Spectral norm

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

ρ

: 0.84

±

0.07

NCI1

1

1000

2000

3000

Epoch

-

0

.

5

0

.

0

0

.

5

1

.

0

ρ

: 0.69

±

0.04

IMDB-BINARY

K Additional visualizations

Figure 7 and Figure 8 report additional results for the triangle point transformation on the three largest datasets: PROTEINS, NCI1, and IMDB-BINARY. In particular, Figure 7 shows the dependence of the generalization on width, while Figure 8 shows the dependence on the spectral norm. Overall, our bound can capture the trend in the empirical gap and produces high correlation values for all datasets.

NeurIPS Paper Checklist

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Table 1 maps the results supporting our claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The discussion about the Table 2 in the Section 3 contains comparison with prior works. The discussion after Corollary 1 contains description of the limitation of our PersLay analysis. Moreover, the conclusion summarizes some of the important limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are clearly stated in the statements, and in the Appendix E and Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details are provided in the Experiment Section as well as in the Appendix J. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to our code in Section 6.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in the Experiment Section as well as in the Appendix J Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Plots count on error bars and tables count on standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our submission follow the NeurIPS ethical guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide a perspective on broader impacts in Section 7, but do not foresee any direct negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not foresee any direct risk stemming from our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: All code was made by the authors

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No experiments with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper derives novel generalization bounds for a special class of GNNs augmented with persistent homology descriptors, specifically PersLay. They empirically test the bound and use the bound to propose a regularization, which works well in practice.

STRENGTHS:
1. The paper provides a clear background introduction to persistence homology and its integration with GNNs.
2. The derived bound has positive correlation with the empirical generalization gap of GNNs.
3. The paper is well positioned within related works on perturbation based generalization bounds of MLPs and GNNs.

WEAKNESSES:
1. It is hard to parse the significance of this work as the bound applies only to a restricted family of models augmented with PersLay. This issue could be mitigated by demonstrating that a model derived from this analysis is comparable with more advanced GNN models (not necessarily with PH), which the paper fails to do.
2. The paper would benefit from listing all assumptions used in the derivation.
3. The experiments could be strengthened by considering different GNN models augmented by different PersLay variants.

QUESTIONS:
How does the bound for GNNs with PH compare with their counterparts without PH? Does the result provide any insights into why PH is beneficial for enhancing GNN performance?

LIMITATIONS:
Please refer to the weaknesses section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-29

comments: SUMMARY:
The paper introduces a novel Compositional PAC-Bayes framework that addresses challenges related to the heterogeneity of Graph Neural Network (GNN) layers and persistent vectorization components. It provides data-dependent generalization bounds for PH vectorization schemes and persistence-augmented GNNs, offering insights into improved classifier design and generalization performance predictions.

STRENGTHS:
1. The introduction of the Compositional PAC-Bayes framework is a significant contribution to the field, especially in handling heterogeneous GNN layers.
2.  The provision of data-dependent generalization bounds adds a valuable dimension to the analysis of GNNs and persistence-augmented models.
3.  The empirical evaluations on real-world datasets demonstrate the practical applicability and effectiveness of the proposed framework.

WEAKNESSES:
Clarity: Some sections of the paper may require further clarification to enhance readability and understanding for a broader audience. From experiments, we can observe that there exists a considerable gap between the theoretical results and empirical results.

QUESTIONS:
1. This paper mentioned the expressivity of PH and GNNs for many times. Does the expressivity have anything to do with the generalization?
2. What is the influence of using different filtration functions in the proposed framework?
3. Does the proposed framework have any concrete applications? Can you provide a case study?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel compositional PAC-Bayes framework for analyzing the generalization of heterogeneous machine learning models, with a particular focus on graph neural networks (GNNs) augmented with persistent homology (PH) features. The work develops a general PAC-Bayes lemma for heterogeneous models that not only recovers existing bounds for neural networks and GNNs but also extends them to more complex architectures. Notably, it provides the first data-dependent generalization bounds for PH-based models.\nThe key innovation lies in the compositional approach, presenting lemmas that allow for combining bounds from different model components. This enables the analysis of complex architectures like GNNs augmented with PH features, bridging an important gap in the theoretical understanding of topology-based graph representation learning methods. The framework's versatility is demonstrated by recovering existing bounds for various models and deriving new bounds for PersLay and its variants. The empirical evaluations across multiple datasets validate the theoretical results, showing correlation between the derived bounds and observed generalization performance.\n\nSTRENGTHS:\n- Novel theoretical contributions that advance the state-of-the-art in generalization analysis for GNNs and PH-based methods.\n- Flexible framework that recovers existing bounds and enables analysis of new model compositions\n\nWEAKNESSES:\n- Empirical evaluation focuses mostly on graph classification tasks; additional experiments on node classification or link prediction could strengthen the results\n\nQUESTIONS:\n- The theoretical analysis assumes fixed filtration functions for PH. How limiting is this in practice, and could the framework be extended to learnable filtrations?\n\nLIMITATIONS:\n- While the experiments cover several datasets, they focus primarily on graph classification tasks. The paper could benefit from a broader range of experiments, including node classification or link prediction tasks, to demonstrate the generality of the approach.\n- While the paper derives a regularization scheme from the bounds, it doesn't fully explore how the theoretical results could guide the design of better GNN+PH architectures in general. Some discussion on how the bounds suggest optimal ways to combine GNNs and PersLay could enhance the practical impact of the work. \n\nThese points did not diminish the overall contribution of the paper but addressing them could significantly strengthen its impact and applicability.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a compositional PAC learning framework for bounding the generalization gap in deep graph networks that are augmented by PersLay-vectorization of persistent homology features. Topological features can be complementary to deep features. Empirically, this combination can boost the empirical test performance. This paper investigates whether this observation reflects to the theory, and concludes that it does, up to some assumptions, for example in filtrations.\n\nSTRENGTHS:\n- To the best of my knowledge, this is the first generalization bound of its kind, presenting bounds for a heterogeneous network composed of PH vectorization and GNNs.\n- I really like the presentation and exposition in this paper, that makes it informative and easy to follow.\n- The idea of using a compositional pac-bayes framework for bounding generalization in PH-augmented GNNs really makes sense to me.\n- Although I have not been able to thoroughly check all the proofs, the theoretical parts I investigated made sense.\n- Given the nicheness of the topic, the models (GNN + PersLay) are intuitive for integrating topological features in deep graph learning. I have seen such architectures a few times.\n\nWEAKNESSES:\n- One obvious drawback is that the architectures of concern in this paper are rather limited and not the ones used in practice. While this limits the applicability, the paper's theoretical contributions make it less of a concern. According to me, speaking of generalization for PH-augmented GNNs through the lens of compositional pac-bayes is interesting in its own right.\n\n- Most importantly, does the proposed bound, similar to the KL-terms in Neyshabur's bound, depend on the number of parameters in the network? (I was suspecting this due to the use of the norm.) If so, it is definitely worth discussing, as the recent theory of deep learning states that the intrinsic dimension matters rather than the ambient dimension. And if not, please clarify.\n\n- I'm curious about the limitations of the compositional framework. Can we leverage the nature of PH and the integration to do something more specific to tighten the bound? The compositional framework seems to adapt a rather late fusion. \n\n-  Ln. 81: The generalization error is defined to be the population risk and not as the difference between them. I believe some sort of a difference between test/train would be more appropriate. Why is this chosen? Is this common convention? \n\n- I see PH as a hand-crafted way of extracting topological features, which goes a bit orthogonal to the current trends. Can the same analysis be extended to hybrid classical & topological deep learning (like the ones presented in [*,**]) which also operates on complexes? If so, do the authors see a straightforward way? \n\n- There are multiple ways to combine PH(-vectorization) and GNNs. A two-branch architecture + PersLay is definitely a good way. Yet, I would expect that the paper investigates other forms of combination. For example, what about learning a filtration?\n\n- I'm noticing that only a single family of GNNs is used in the paper. Is it possible to show results with more modern architectures at least like GraphSage?\n\n- There is also a rather recent literature of leveraging PH to bound generalization [***] (other way around), which this paper does not seem to discuss. It would be good to have both sides of this picture to stress on the impact of PH in the generalization theory.\n\n[*] Hajij, Mustafa et al. \"Topological deep learning: Going beyond graph data.\" arXiv preprint arXiv:2206.00606 (2022).\n\n[**] Papamarkou, Theodore, Tolga Birdal, Michael M. Bronstein, Gunnar E. Carlsson, Justin Curry, Yue Gao, Mustafa Hajij et al. \"Position: Topological Deep Learning is the New Frontier for Relational Learning.\" In Forty-first International Conference on Machine Learning. 2024.\n\n[***] Birdal, Tolga, Aaron Lou, Leonidas J. Guibas, and Umut Simsekli. \"Intrinsic dimension, persistent homology and generalization in neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 6776-6789.\n\nQUESTIONS:\nI have appended my questions after each relevant weakness. I would be happy if the authors could address them. In addition: \n- Why would the bounds worsen with the increase number of epochs (where I believe training gets better)?\n- There is also growth in the width disproportionately to the empirical gap. Why would this happen?\n\nIn general, some more light on the empirical findings would be useful.\n\nLIMITATIONS:\nThe paper appropriately discusses the limitations and remains to be largely theoretical in a very niche domain of machine learning. As such, I don't see any issues with broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}