Title: Graph Classification via Reference Distribution Learning: Theory and Practice
Venue: NeurIPS 2024
Accepted: yes
Keywords: graph neural networks

graph classification
Abstract: Graph classification is a challenging problem owing to the difficulty in quantifying the similarity between graphs or representing graphs as vectors, though there have been a few methods using graph kernels or graph neural networks (GNNs). Graph kernels often suffer from computational costs and manual feature engineering, while GNNs commonly utilize global pooling operations, risking the loss of structural or semantic information. This work introduces Graph Reference Distribution Learning (GRDL), an efficient and accurate graph classification method. GRDL treats each graph's latent node embeddings given by GNN layers as a discrete distribution, enabling direct classification without global pooling, based on maximum mean discrepancy to adaptively learned reference distributions. To fully understand this new model (the existing theories do not apply) and guide its configuration (e.g., network architecture, references' sizes, number, and regularization) for practical use, we derive generalization error bounds for GRDL and verify them numerically. More importantly, our theoretical and numerical results both show that GRDL has a stronger generalization ability than GNNs with global pooling operations. Experiments on moderate-scale and large-scale graph datasets show the superiority of GRDL over the state-of-the-art, emphasizing its remarkable efficiency, being at least 10 times faster than leading competitors in both training and inference stages.

Graph Classification via Reference Distribution Learning: Theory and Practice

Abstract

Graph classification is a challenging problem owing to the difficulty in quantifying the similarity between graphs or representing graphs as vectors, though there have been a few methods using graph kernels or graph neural networks (GNNs). Graph kernels often suffer from computational costs and manual feature engineering, while GNNs commonly utilize global pooling operations, risking the loss of structural or semantic information. This work introduces Graph Reference Distribution Learning (GRDL), an efficient and accurate graph classification method. GRDL treats each graph's latent node embeddings given by GNN layers as a discrete distribution, enabling direct classification without global pooling, based on maximum mean discrepancy to adaptively learned reference distributions. To fully understand this new model (the existing theories do not apply) and guide its configuration (e.g., network architecture, references' sizes, number, and regularization) for practical use, we derive generalization error bounds for GRDL and verify them numerically. More importantly, our theoretical and numerical results both show that GRDL has a stronger generalization ability than GNNs with global pooling operations. Experiments on moderate-scale and large-scale graph datasets show the superiority of GRDL over the state-of-the-art, emphasizing its remarkable efficiency, being at least 10 times faster than leading competitors in both training and inference stages. The source code of GRDL is available at https://github.com/jicongfan/GRDL-Graph-Classification .

1 Introduction

Graphs serve as versatile models across diverse domains, such as social networks [Wang et al., 2018], biological compounds [Jumper et al., 2021], and the brain [Ktena et al., 2017]. There has been considerable interest in developing learning algorithms for graphs, such as graph kernels [Gärtner et al., 2003, Shervashidze et al., 2011, Chen et al., 2022b] and graph neural networks (GNNs) [Kipf and Welling, 2016, Defferrard et al., 2016, Gilmer et al., 2017]. GNNs have emerged as powerful tools, showcasing state-of-the-art performance in various graph prediction tasks [Veliˇ ckovi´ c et al., 2017, Gilmer et al., 2017, Hamilton et al., 2017, Xu et al., 2018, Sun et al., 2019, You et al., 2021, Ying et al., 2021, Liu et al., 2022b, Chen et al., 2022a, Xiao et al., 2022, Sun et al., 2023, Sun and Fan, 2024, Sun et al., 2024]. Despite the evident success of GNNs in numerous graph-related applications, their potential remains underutilized, particularly in the domain of graph-level classification.

Current GNNs designed for graph classification commonly consist of two components: the embedding of node features through message passing [Gilmer et al., 2017] and subsequent aggregation by some permutation invariant global pooling (also called readout) operations [Xu et al., 2018]. The primary purpose of pooling is to transform a graph's node embeddings, a matrix, into a single vector.

Empirically, pooling operations play a crucial role in classification [Ying et al., 2018]. However, these pooling operations tend to be naive, often employing methods such as simple summation or averaging. These functions collect only first-order statistics, leading to a loss of structural or semantic information. In addition to the conventional sum or average pooling, more sophisticated pooling operations have shown improvements in graph classification [Li et al., 2015, Ying et al., 2018, Lee et al., 2019, 2021, Buterez et al., 2022, Yu et al., 2024], but they still carry the inherent risk of information loss.

Different from graph kernel methods and existing GNN methods, we propose a novel GNN method that classifies the nodes' embeddings themselves directly, thus avoiding the global pooling step. In our method, we treat the nodes' latent representations of each graph, learned by a neural network, as a discrete distribution and classify these distributions into K different classes. The classification is conducted via measuring the similarity between the latent graph's distributions and K discriminative reference discrete distributions. The reference distributions can be understood as nodes' embeddings of representative virtual graphs from K different classes, and they are jointly learned with the parameters of the neural network in an end-to-end manner. To evaluate our method, we analyze the generalization ability of our model both theoretically and empirically. Our contributions are two-fold.

We propose a novel graph classification method GRDL that is efficient and accurate.

-GRDL does not require any global pooling operation and hence effectively preserves the information of node embeddings.

-Besides its high classification accuracy, GRDL is scalable to large graph datasets and is at least ten times faster than leading competitors in both training and inference stages.

We provide theoretical guarantees, e.g. generalization error bounds, for GRDL.

-The result offers valuable insights into how the model performance scales with the properties of graphs, neural network structure, and reference distributions, guiding the model design.

-For instance, the generalization bounds reveal that the references' norms and numbers have tiny impacts on the generalization, which is also verified by the experiments.

-More importantly, we theoretically prove that GRDL has a stronger generalization ability than GNNs with global pooling operations.

The rest of this paper is organized as follows. We introduce our model in Section 2 and analyze the generalization ability in Section 3. Related works are discussed in Section 4. Section 5 presents the numerical results on 11 benchmarks in comparison to 12 competitors.

2 Proposed Approach

2.1 Model Framework

Following convention, we denote a graph with index i by G i = ( V i , E i ) , where V i and E i are the vertex (node) set and edge set respectively. Given a graph dataset G = { ( G 1 , y 1 ) , ( G 2 , y 2 ) , . . . , ( G N , y N ) } , where y i ∈ { 1 , 2 , . . . , K } is the associated label of G i and y i = k means G i belongs to class k , the goal is to learn a classifier f from G that generalizes well to unseen graphs. Since in many scenarios, each node of a graph has a feature vector x and the graph is often represented by an adjacency matrix A , we also write G i = ( A i , X i ) for convenience, where A i ∈ R n i × n i , X i ∈ R n i × d 0 , n i = | V i | is the number of nodes of graph i , and d 0 denotes the number of features. We may alternatively denote the graph dataset as G = { (( A 1 , X 1 ) , y 1 ) , (( A 2 , X 2 ) , y 2 ) , . . . , (( A N , X N ) , y N ) } .

Our approach is illustrated in Figure 1. For graph classification, we first use a GNN, denoted as f G , to transform each graph to a node embedding matrix H i ∈ R n i × d that encodes its properties, i.e.,

where f G ∈ F G and F G denotes a hypothesis space. The remaining task is to classify H i without global pooling. Direct classification of node embeddings is difficult due to two reasons:

̸

(i) Different graphs have different numbers of nodes, i.e. in general, n i = n j if i = j .

̸

(ii) The node embeddings of each graph are permutation invariant, namely, PH i and H i represent the same graph for any permutation matrix P .

However, the two properties are naturally satisfied if we treat the node embeddings of each graph as a discrete distribution. Specifically, each H i is a discrete distribution and each row of H i is an

outcome of the distribution. There is no order between the outcomes in each distribution. Also, different distributions may have different numbers of outcomes. Before introducing our method in detail, we first give a toy example where the commonly used mean and max pooling operations fail.

Example 2.1. Suppose two graphs G 1 and G 2 have self-looped adjacency matrices ˜ A 1 = [1 1 1 0; 1 1 0 1; 1 0 1 1; 0 1 1 1] and ˜ A 2 = [1 0 0 0; 0 1 1 0; 0 1 1 0; 0 0 0 1] respectively, and have one-dimensional node features X 1 = [3 6 9 12] ⊤ and X 2 = [6 6 9 9] ⊤ respectively. Let ˆ A i be the normalized adjacency matrices, i.e., ˆ A i = diag ( ˜ A i 1 ) -1 / 2 ˜ A i diag ( ˜ A i 1 ) -1 / 2 . Performing the neighbor aggregation H i = ˆ A i X i , i = 1 , 2 , we obtain H 1 = [6 7 8 9] ⊤ and H 2 = [6 7 . 5 7 . 5 9] ⊤ . We see that mean( H 1 ) = mean( H 2 ) = 7 . 5 and max( H 1 ) = max( H 2 ) = 9 . This means the simple mean and max pooling operations failed to distinguish the two graphs. In contrast, our method treats H 1 and H 2 as two different discrete distributions and hence is able to distinguish the two graphs. Note that incorporating a learnable parameter W , i.e., H i = ˆ A i X i W , or performing multiple times of neighbor aggregation does not change the conclusion.

We propose to classify the discrete distributions { H 1 , H 2 , . . . , H N } ≜ H by a reference layer f D . The classification involves measuring the similarity between H i and K reference discrete distributions { D 1 , D 2 , . . . , D K } ≜ D that are discriminative. Each D k ∈ R m k × d can be understood as node embeddings of a virtual graph from the k -th class, k ∈ [ K ] . We make m 1 = · · · = m K = m for convenience. Letting ξ be a similarity measure between two discrete distributions, then

This forms a matrix S = [ s 1 , s 2 , . . . , s N ] ⊤ where

f D ∈ F D and F D denotes a hypothesis space induced by the reference layer. References in D are parameters of the reference layer and are jointly learned with node-embedding network parameters in an end-to-end manner. Now combining Equation (1) and Equation (3), we arrive at

f D ◦ f G calculates s i , representing similarities between G i and all references. We get G i 's label by

To train the model, we first use the softmax function to convert s i to a label vector ˆ y i = [ˆ y i 1 , . . . , ˆ y iK ] ⊤ , where

Using the cross-entropy loss, we minimize

Intuitively, the reference distributions in D should be different from each other to ensure discriminativeness. Therefore, we also consider the following discrimination loss:

̸

Then we solve the following problem:

where λ ≥ 0 is a hyperparameter. We call (9) Graph Classification via Reference Distribution Learning (GRDL). Specific designs of F G and F D are detained in the next section.

2.2 Design of F G and F D

We get GRDL's network F by concatenating the node embedding module and the reference module:

Design of F G We use an L -layer message passing network as our node embedding module F G :

F l is the l -th message passing layer (e.g. a GIN layer [Xu et al., 2018]) that updates the representation of a node by aggregating representations of its neighbors, meaning

where h ( l ) v is the feature vector of node v produced by the l -th layer F l . Different GNNs have different choices of COMBINE ( l ) ( · ) and AGGREGATE ( l ) ( · ) .

Design of F D Based on (3), the hypothesis space defined by the reference layer is

In our work, we choose ξ ( · , · ) to be the negative squared Maximum Mean Discrepancy (MMD). Initially used for two-sample tests, MMD is now widely used to measure the dissimilarity between distributions [Gretton et al., 2012a]. For an embedding H ∈ R n × d and a reference D ∈ R m × d ,

where ϕ is some feature map, h ⊤ i is the i -th row of H , and d ⊤ j is the j -th row of D . The MMD in (14) is known as biased MMD [Gretton et al., 2012a] and its performance is almost the same as the unbiased one, in our experiments. Therefore we only present (14) here. Using kernel trick k ( x , x ′ ) = ϕ ( x ) ⊤ ϕ ( x ′ ) , we obtain from (14) that

In this work, we employ the Gaussian kernel, i.e.,

where θ > 0 is a hyperparameter. The Gaussian kernel defines an infinite-order polynomial feature map ϕ , covering all orders of statistics of the input variable. Consequently, MMD with the Gaussian kernel characterizes the difference between two distributions across all moments. Actually, we found that, in GRDL, the Gaussian kernel often outperformed other kernels such as the polynomial kernel.

Several other statistical distances are available for measuring the difference between distributions, including Wasserstein distance and Sinkhorn divergence [Peyré and Cuturi, 2020]. However, their computational complexity is prohibitively high, making the model impractical for large-scale graph datasets. We also find, through experiments, that in our method, the classification performance of MMDis better than that of Wasserstein distance and Sinkhorn divergence as shown later in Table 1. These explain why we prefer MMD.

2.3 Algorithm Implementation

The θ in the Gaussian kernel (15) plays a crucial role in determining the statistical efficiency of MMD. Optimally setting of θ remains an open problem and many heuristics are available [Gretton et al., 2012b]. To simplify the process, we make θ learnable in our GRDL and rewrite ξ as ξ θ . Our empirical results in Appendix D.5 show that GRDL with learnable θ performs better. For convenience, we denote all the parameters of f G as w and let f w , D ,θ = f D ◦ f G . Then we rewrite problem (9) as

̸

The (mini-batch) training of GRDL model is detailed in Algorithm 1 (see Appendix C).

3 Theoretical Analysis

In this section, we provide theoretical guarantees for GRDL, due to the following motivations:

As the proposed approach is novel, it is necessary to understand it thoroughly using theoretical analysis, e.g., understand the influences of data and model properties on the classification.

It is also necessary to provide guidance for the model design to guarantee high accuracy in inference stages.

3.1 Preliminaries

Matrix constructions We construct big matrices X , A and D , where X = [ X ⊤ 1 , X ⊤ 2 , . . . , X ⊤ N ] ⊤ ∈ R ( ∑ i n i ) × d ; A = diag ( A 1 , A 2 , . . . , A N ) ∈ R ( ∑ i n i ) × ( ∑ i n i ) is a block diagonal matrix, D = [ D ⊤ 1 , D ⊤ 2 , . . . , D ⊤ K ] ⊤ ∈ R Km × d . The adjacency matrix with self-connectivity is ˜ A = A + I . The huge constructed graph is denoted by G = ( ˜ A , X ) . This construction allows us to treat all graphs in dataset G as a whole and it is crucial for our derivation.

Neural network Previously, for a deterministic network f ∈ F , its output after feeding forward a single graph is f ( G i ) . However, we mainly deal with the huge constructed graph G in this section, and notation will be overloaded to f ( G ) = S ∈ R N × K , a matrix whose i -th row is f ( G i ) ⊤ .

We instantiate the message passing network as Graph Isomorphism Network (GIN) [Xu et al., 2018]. We choose to focus on GIN for two reasons. Firstly, the analysis on GIN is currently limited, most of the current bounds for GNNs don't apply for GIN [Garg et al., 2020, Liao et al., 2021, Tang and Liu, 2023]. The other reason is that GIN is used as the message-passing network in our numerical experiments. Notably, our proof can be easily adapted to other message-passing GNNs (e.g. GCN [Kipf and Welling, 2016]). GIN updates node representations as

where h ( l ) v denotes the node features generated by l -th GIN message passing layer. Let ε ( l ) = 0 for all layers and suppose all MLPs have r layers, the node updates can be written in matrix form as

where W ( l ) i ∈ R d ( l ) i -1 × d ( l ) i is the weight matrix, and H ( l ) is the matrix of node features with H (0) = X . σ ( · ) is the non-linear activation function. Let F l be the function space induced by the l -th message passing layer, meaning

where B ( l ) i is some constraint set on the weight matrix W ( l ) i and H ( l ) is given by (18). The L -layer GIN function space F G is the composition of F l for l ∈ [ L ] , i.e.,

Letting s ik = -MMD 2 ( H ( L ) i , D k ) , the reference layer defines the following function space

Our proposed network (GRDL) is essentially F := F D ◦ F G .

Loss Function Instead of the cross entropy loss (7), we consider a general loss function l γ ( · , · ) satisfying 0 ≤ l γ ≤ γ to quantify the model performance. Importantly, this loss function is not restricted to the training loss because our generalization bound is optimization-independent. For instance, the loss function can be the ramp loss that is commonly used for classification tasks [Bartlett et al., 2017, Mohri et al., 2018]. Given a neural network f ∈ F , we want to upper bound the model population risk of graphs and labels from an unknown distribution X × Y

Given the observed graph dataset G sampled from X × Y , the empirical risk is

of which (7) is just a special case. Appendix E provides more details about the setup and our idea.

3.2 Main Results

For convenience, similar to [Bartlett et al., 2017, Ju et al., 2023], we make the following assumptions.

Assumption 3.1. The following conditions hold for F γ := { ( G,y ) ↦→ l γ ( f ( G ) , y ) : f ∈ F} :

(i) The activation function σ ( · ) is 1-Lipschitz (e.g. Sigmoid, ReLU).

(ii) The weight matrices satisfy W ( l ) i ∈ B ( l ) i := { W ( l ) i : ∥ W ( l ) i ∥ σ ≤ κ ( l ) i , ∥ W ( l ) i ∥ 2 , 1 ≤ b ( l ) i } .

(iii) The constructed reference matrix satisfy ∥ D ∥ 2 ≤ b D .

(iv) The Gaussian kernel parameter θ is fixed.

(v) The loss function l γ ( · , y ) : R K → R is µ -Lipschitz w.r.t ∥ · ∥ 2 and 0 ≤ l γ ≤ γ .

˜ ¯ ( l ) .

Theorem 3.2 (Generalization bound of GRDL) . Let n = min i n i , c = ∥ A ∥ σ , and d = max i,l d i Denote R G := c 2 L ∥ X ∥ 2 2 ln(2 ¯ d 2 ) ( ∏ L l =1 ( ∏ r i =1 κ ( l ) i ) 2 )( ∑ L l =1 ∑ r i =1 ( b ( l ) i κ ( l ) i ) 2 / 3 ) 3 . For graphs G = { ( G i , y i ) } N i =1 drawn i.i.d from any probability distribution over X × { 1 , . . . , K } and references { D k } K k =1 , D k ∈ R m × d , with probability at least 1 -δ , every loss function l γ and network f ∈ F under Assumption 3.1 satisfy

The bound shows how the properties of the neural network, graphs, reference distributions, etc, influence the gap between training error and testing error. A detailed discussion will be presented in Section 3.3. Some interesting corollaries of Theorem 3.2, e.g., misclassification rate bound, can be found in Appendix F.7. Besides small generalization error L γ ( f ) -ˆ L γ ( f ) , a good model should have small empirical risk ˆ L γ ( f ) . The empirical risk ˆ L γ ( f ) is typically a surrogate loss of misclassification

rate of training data and a lower misclassification rate implies a smaller ˆ L γ ( f ) . We now provide a guarantee for the correct classification of training data, namely small ˆ L γ ( f ) .

̸

Notably, the node embeddings H i from the k -th class as well as the reference distributions D k are essentially some finite samples from an underlying continuous distribution P k . One potential risk is that, although the continuous distributions P 1 , P 2 , . . . , P K are distinct, we can only observe their finite samples and may fail to distinguish them from each other with MMD. Specifically, suppose a node embedding H i is from the k -th class, although 0 = MMD( P k , P k ) < MMD( P k , P j ) for any j = k , it is likely that MMD( H i , D k ) > MMD( H i , D j ) for some j = k . The following theorem provides the correctness guarantee for the training dataset G :

̸

Theorem 3.3. All graphs in the training set G are classified correctly with probability at least 1 -δ if

̸

̸

Theorem 3.3 implies that a larger reference distribution size m benefits the classification accuracy of training data, resulting in a lower ˆ L γ ( f ) . Moreover, a larger min i = j MMD( P i , P j ) also makes correct classification easier according to the theorem, justifying our usage of discriminative loss (8).

3.3 Bound Discussion and Numerical Verification

Let ¯ κ = max i,l κ ( l ) i , ¯ b = max i,l b ( l ) i κ ( l ) i and suppose δ is large enough, we simplify Theorem 3.2 as

I. Dependence on graph property One distinctive feature of our bound is its dependence on the spectral norm of graphs' adjacency matrix. The large adjacency matrix ˜ A is a block-diagonal matrix, so its spectral norm c = ∥ ˜ A ∥ σ = max i ∈ [ N ] ∥ ˜ A i ∥ σ . By Lemma F.8, incorporating c L is sufficient for any L -step GIN message passing. This result aligns with Ju et al. [2023], who achieved this conclusion via PAC-Bayesian analysis. Our derivation, based on the Rademacher complexity, provides an alternative perspective supporting this result. Notably, Liao et al. [2021] and Garg et al. [2020] proposed bounds scaling with graphs' maximum node degree, which is larger than the spectral norm of the graphs' adjacency matrix (Lemma F.18). Consequently, our bound is tighter.

II. Use moderate-size message passing GIN The bound scales with the size of the message passing GIN, following ˜ O ( c L ( Lr ) 3 2 ¯ κ Lr ) . Empirical observations reveal ¯ κ > 1 , and we prove that c > 1 (refer to Lemma F.20). Therefore, when the message-passing GNN has sufficient expressive power (resulting in a small ˆ L γ ( f ) ), a network with a smaller L and r may guarantee a tighter bound on the population risk compared to a larger one. Therefore, a promising strategy is to use a moderate-size message passing GNN. This is empirically supported by Figure 5 of Appendix D.7.

III. Use moderate-size references The bound scales with the size of reference distributions m as ˜ O ( √ m ) . When m is smaller, the bound tends to be tighter. However, if m is too small, the model's expressive capacity is limited, potentially resulting in a large empirical risk ˆ L γ ( f ) , and consequently, a large population risk. Therefore, using moderate-size references is a promising choice, as supported by our empirical validation results in Appendix D.3 (see Figure 6).

IV. Regularization on references norm barely helps Regularizing the norm of references ∥ D ∥ 2 , i.e., reducing b D , might be considered to enhance the model's generalization. However, it is important to note that b D only influences the term v 3 (in logarithm) in Theorem 3.2 and has a tiny influence on the overall bound. Conversely, such regularization constrains the model's expressive capacity, potentially leading to a large ˆ L γ ( f ) and increasing the population risk. This observation is empirically supported by experiments in Appendix D.7 (see Table 10).

V. GRDL has a tighter bound than GIN with global pooling In Appendix A, we provide the generalization error bound, i.e., Theorem A.1, for GIN with global pooling and compare it with Theorem 3.2. The result shows that our GRDL has a stronger generalization ability than GIN, which is further supported by the numerical results in Table 4.

Remark 3.4 . Currently, we use K reference distributions for classification (one for each class). One natural approach to enhancing the model's expressive power is increasing the number of references for each class. However, counterintuitively, our empirical observations, supported by Theorem B.1, suggest that having only one reference per class is optimal. We discuss this further in Appendix B.

4 Related Work

Various sophisticated pooling operations have been designed to preserve the structural information of graphs [Bianchi et al., 2020, Ranjan et al., 2020, Baek et al., 2021, Chen and Gel, 2023, Yu et al., 2024]. For instance, DIFFPOOL, designed by Ying et al. [2018], learns a differentiable soft cluster assignment for nodes and maps nodes to a set of clusters to output a coarsened graph. Another method by Lee et al. [2019] utilizes a self-attention mechanism to distinguish nodes for retention or removal, and both node features and graph topology are considered with the self-attention mechanism.

A recent research direction focuses on preserving structural information by leveraging the optimal transport (OT) [Peyré and Cuturi, 2020]. OT-GNN, proposed by Chen et al. [2021], embeds a graph to a vector by computing Wasserstein distances between node embeddings and some 'learned point clouds". TFGW, introduced by Vincent-Cuaz et al. [2022], embeds a graph to a vector of Fused Gromov-Wasserstein (FGW) distance [Vayer et al., 2018] to a set of 'template graphs". OT distances have also been combined with dictionary learning to learn graph vector embedding in an unsupervised way (GDL) [Liu et al., 2022a, Vincent-Cuaz et al., 2021, Zeng et al., 2023].

Similar to the 'learned point clouds" in OT-GNN, 'template graphs" in TFGW, and dictionaries in GDL, our GRDL preserves information in node embeddings using reference distributions. To the best of the authors' knowledge, we are the first to model a graph's node embeddings as a discrete distribution and propose to classify it directly without aggregating it into a vector, marking our novel contribution. Additionally, our work stands out as the first to analyze the generalization bounds for this type of model, adding a theoretically grounded dimension to the research. By the way, our method is much more efficient than OT-GNN and TFGW. Please see Figure 2 and Table 8.

5 Numerical Experiments

5.1 Graph Classification Benchmark

Datasets We leverage eight popular graph classification benchmarks [Morris et al., 2020], comprising five bioinformatics datasets (MUTAG, PROTEINS, NCI1, PTC-MR, BZR) and three social network datasets (IMDB-B, IMDB-M, COLLAB). We also use three large-scale imbalanced datasets (PC-3, MCF-7, and ogbg-molhiv [Hu et al., 2020]). A summary of data statistics is in Table 6.

Baselines Our approach is benchmarked against four groups of state-of-the-art baselines: 1) GNN models with global or sophisticated pooling operations, including PATCHY-SAN [Niepert et al., 2016], DIFFPOOL [Ying et al., 2018], GIN [Xu et al., 2018], DropGIN [Papp et al., 2021], SEP [Wu et al., 2022], GMT [Baek et al., 2021], MinCutPool [Bianchi et al., 2020], ASAP [Ranjan et al., 2020], and Wit-TopoPool

[Chen and Gel, 2023]; 2) Optimal transport based models such as WEGL [Kolouri et al., 2020] and OT-GNN [Chen et al., 2021]; 3) Kernel-based approaches including FGW [Titouan et al., 2019] operating on adjacency (ADJ) and shortest path (SP) matrices, the WL subtree kernel [Shervashidze et al., 2011], and the Wasserstein WL kernel [Togninalli et al., 2019]; 4) Graph transformers including Graphormer [Ying et al., 2021] and SAT [Chen et al., 2022a]. We also show the results of two variations of our GRDL: GRDL using Sinkhorn divergence (GRDL-S) and GRDL using Wasserstein distance (GRDL-W). For large imbalanced datasets, we only benchmark our GRDL against PATCHYSAN, GIN, and DIFFPOOL because other methods are too costly. Details about the initialization and hyper-parameters setting can be found in Appendix D.3.

Experiment Settings Due to the page limitation, please refer to Appendix D.2.

Classification Results Table 1 shows the classification results. The AUC-ROC scores of experiments results on the three large imbalanced datasets are reported in Table 2. Our method has top 3 classification performance over baselines in almost all datasets. Our GRDL, GRDL-W and GRDL-S have close performance. However, as shown later in Figure 2, our original GRDL has significantly lower time costs and thus is preferable for practical use. Graph transformers also have competitive performance, but they have significantly larger amount parameters and much higher time costs than our model, as shown by Table 13 in Appendix D.8 .

5.2 Time Cost Comparison

We compare the time cost of our GRDL with two models that leverage optimal transport distances discussed in Section 4: OT-GNN [Chen et al., 2021] and TFGW [Vincent-Cuaz et al., 2022]. Compared with them, our model has significantly lower time costs. We present empirical average training time per epoch in Figure 2 and average prediction time per graph in Table 9 in Appendix D.4. Experiments were conducted on CPUs (Apple M1) using identical batch sizes, ensuring a fair comparison. It's noteworthy that the OT solver employed in TFGW and OT-GNN is currently confined to CPU, influencing the choice of hardware for this evaluation. We analyzed the theoretical time complexity in Appendix D.4 (see Table 8).

We also compare training time with two latest pooling methods including Wit-TopoPool [Chen and Gel, 2023] and MSGNN [Lv et al., 2023] on eight real datasets and three synthetic datasets. The three synthetic datasets have 2000 graphs with 100(SYN-100), 300(SYN-300), and 500(SYN-500) nodes per graph, respectively. The edge number is 0 . 1 n 2 where n is the number of nodes. The empirical training time per epoch is shown in Table 3, where empty of MSGNN means it takes more than 200 seconds to train a single epoch, which is too costly. As can be seen, our method is the most efficient among these three methods.

5.3 Graph Visualization

We use t-SNE [Van der Maaten and Hinton, 2008] to visualize the distributions of graphs' node embeddings given by our GRDL model, which is equivalent to visualizing each graph in a 3-D coordinate system. Firstly we use MMD to calculate a distance matrix C ∈ R ( N + K ) × ( N + K ) between

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

2

4

6

8

Time Cost (second)

0.4

1.8

2.5

3.6

9.3

MUTAG

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

5

10

15

20

25

30

35

Time Cost (second)

3.4

13.2

17.8

34.7

34.4

PROTEINS

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

25

50

75

100

125

150

175

Time Cost (second)

12.6

51.3

65.1

129.4

186.3

NCI1

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

10

20

30

40

50

60

70

Time Cost (second)

2.4

12.1

17.7

34.7

67.6

IMDB-B

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

10

20

30

40

50

60

70

80

Time Cost (second)

3.5

13.7

23.1

46.0

78.3

IMDB-M

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Time Cost (second)

0.8

4.4

6.0

9.8

17.2

PTC-MR

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

2

4

6

8

10

12

14

Time Cost (second)

1.2

7.9

10.2

14.4

8.1

BZR

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

50

100

150

200

Time Cost (second)

16.3

93.3

85.6

189.2

226.1

COLLAB

GRDL

GRDL-W

GRDL-S

OT-GNN

TFGW

0

10

20

30

40

50

60

70

80

Time Cost (second)

5.1

24.7

28.5

57.5

78.4

Average

the node embeddings { H i } N i =1 and the reference distributions { D k } K k =1 . The 3-D visualization given by t-SNE using C is presented in Figure 3. The graphs are located around the references. It means that the learned references can represent realistic graphs' latent node embeddings from the data.

5.4 More Numerical Results

The ablation study, influence of θ , generalization comparison with GIN are in Appendices D.5, D.6, and A, respectively.

6 Conclusions

We proposed GRDL, a novel framework for graph classification without global pooling operations and hence effectively preserve the information of node embeddings. What's more, we theoretically analyzed the generalization ability of GRDL, which provided valuable insights into how the generalization ability scales with the properties of the graph data and network structure. Extensive experiments on moderatescale and large-scale benchmark datasets verify the effectiveness and efficiency of GRDL in comparison to baselines. However, on some benchmark datasets (e.g. NCI1), our model does not outperform the baseline, which may be a limitation of our work and requires further investigation in the future.

Class 0

。

Class 1

Reference0

Reference 1

1.5

1.0

0.5

0.0

-0.5

-1.0

-1.5

4

2

-4

0

-2

2

0

2

-4

4

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant No.62376236, the Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001), Shenzhen Science and Technology Program ZDSYS20230626091302006, Shenzhen Stability Science Program 2023, and Hetao ShenzhenHong Kong Science and Technology Innovation Cooperation Zone Project (No.HZQSWS-KCCYB2024016). The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

The authors appreciate the reviewers and area chair of the paper.

References

Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations with graph multiset pooling. arXiv preprint arXiv:2102.11533 , 2021.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Yoshua Bengio and Yves Grandvalet. No unbiased estimator of the variance of k-fold cross-validation. Advances in Neural Information Processing Systems , 16, 2003.

Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning , pages 874-883. PMLR, 2020.

David Buterez, Jon Paul Janet, Steven J Kiddle, Dino Oglic, and Pietro Liò. Graph neural networks with adaptive readouts. Advances in Neural Information Processing Systems , 35:19746-19758, 2022.

Benson Chen, Gary Bécigneul, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Optimal transport graph neural networks, 2021.

Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In Proceedings of the 39th International Conference on Machine Learning (ICML) , Proceedings of Machine Learning Research, 2022a.

Samantha Chen, Sunhyuk Lim, Facundo Mémoli, Zhengchao Wan, and Yusu Wang. Weisfeilerlehman meets gromov-wasserstein. In International Conference on Machine Learning , pages 3371-3416. PMLR, 2022b.

Yuzhou Chen and Yulia R. Gel. Topological pooling on graphs. In Proceedings of the ThirtySeventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence , AAAI'23/IAAI'23/EAAI'23, 2023.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems , 29, 2016.

Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning , pages 3419-3430. PMLR, 2020.

Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings , pages 129-143. Springer, 2003.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012a.

Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. Advances in neural information processing systems , 25, 2012b.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Haotian Ju, Dongyue Li, Aneesh Sharma, and Hongyang R. Zhang. Generalization in graph neural networks: Improved pac-bayesian bounds on graph diffusion. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 6314-6341. PMLR, 25-27 Apr 2023.

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Soheil Kolouri, Navid Naderializadeh, Gustavo K Rohde, and Heiko Hoffmann. Wasserstein embedding for graph learning. arXiv preprint arXiv:2006.09430 , 2020.

Sofia Ira Ktena, Sarah Parisot, Enzo Ferrante, Martin Rajchl, Matthew Lee, Ben Glocker, and Daniel Rueckert. Distance metric learning using graph convolutional networks: Application to functional brain networks. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part I 20 , pages 469-477. Springer, 2017.

Dongha Lee, Su Kim, Seonghyeon Lee, Chanyoung Park, and Hwanjo Yu. Learnable structural semantic readout for graph classification. In 2021 IEEE International Conference on Data Mining (ICDM) , pages 1180-1185. IEEE, 2021.

Junhyun Lee, Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling. In International conference on machine learning , pages 3734-3743. PMLR, 2019.

Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 , 2015.

Renjie Liao, Raquel Urtasun, and Richard Zemel. A PAC-Bayesian approach to generalization bounds for graph neural networks. In International Conference on Learning Representations , 2021.

Weijie Liu, Jiahao Xie, Chao Zhang, Makoto Yamada, Nenggan Zheng, and Hui Qian. Robust graph dictionary learning. In The Eleventh International Conference on Learning Representations , 2022a.

Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and S Yu Philip. Graph selfsupervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering , 35(6): 5879-5900, 2022b.

Yiqin Lv, Zhiliang Tian, Zheng Xie, and Yiping Song. Multi-scale graph pooling approach with adaptive key subgraph for graph representations. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , CIKM '23. Association for Computing Machinery, 2023.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing systems , 32, 2019.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) , 2020. URL www.graphlearning.io .

Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International conference on machine learning , pages 2014-2023. PMLR, 2016.

Pál András Papp, Karolis Martinkus, Lukas Faber, and Roger Wattenhofer. Dropgnn: Random dropouts increase the expressiveness of graph neural networks. Advances in Neural Information Processing Systems , 34:21997-22009, 2021.

Gabriel Peyré and Marco Cuturi. Computational optimal transport, 2020.

Ekagra Ranjan, Soumya Sanyal, and Partha Talukdar. Asap: Adaptive structure aware pooling for learning hierarchical graph representations. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 5470-5477, 2020.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(9), 2011.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Yan Sun and Jicong Fan. MMD graph kernel: Effective metric learning for graphs via maximum mean discrepancy. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=GZ6AcZwA8r .

Ziheng Sun, Chris Ding, and Jicong Fan. Lovász principle for unsupervised graph representation learning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Ziheng Sun, Xudong Wang, Chris Ding, and Jicong Fan. Learning graph representation via graph entropy maximization. In Forty-first International Conference on Machine Learning , 2024.

Huayi Tang and Yong Liu. Towards understanding generalization of graph neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 33674-33719. PMLR, 23-29 Jul 2023.

Vayer Titouan, Nicolas Courty, Romain Tavenard, Chapel Laetitia, and Rémi Flamary. Optimal transport for structured data with application on graphs. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 6275-6284. PMLR, 09-15 Jun 2019.

Matteo Togninalli, Elisabetta Ghisu, Felipe Llinares-López, Bastian Rieck, and Karsten Borgwardt. Wasserstein weisfeiler-lehman graph kernels. Advances in neural information processing systems , 32, 2019.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Titouan Vayer, Laetita Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Fused gromovwasserstein distance for structured objects: theoretical foundations and mathematical properties, 2018.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Cédric Vincent-Cuaz, Titouan Vayer, Rémi Flamary, Marco Corneli, and Nicolas Courty. Online graph dictionary learning. In International conference on machine learning , pages 10564-10574. PMLR, 2021.

Cédric Vincent-Cuaz, Rémi Flamary, Marco Corneli, Titouan Vayer, and Nicolas Courty. Template based graph neural network with optimal transport distances. Advances in Neural Information Processing Systems , 35:11800-11814, 2022.

Zhouxia Wang, Tianshui Chen, Jimmy Ren, Weihao Yu, Hui Cheng, and Liang Lin. Deep reasoning with knowledge graph for social relationship understanding. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 , pages 1021-1028. International Joint Conferences on Artificial Intelligence Organization, 7 2018. doi: 10.24963/ijcai.2018/142. URL https://doi.org/10.24963/ijcai.2018/142 .

Junran Wu, Xueyuan Chen, Ke Xu, and Shangzhe Li. Structural entropy guided graph hierarchical pooling. In International conference on machine learning , pages 24017-24030. PMLR, 2022.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled selfsupervised learning for graphs. Advances in Neural Information Processing Systems , 35:620-634, 2022.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems , 34:28877-28888, 2021.

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems , 31, 2018.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132. PMLR, 2021.

Jiajun Yu, Zhihao Wu, Jinyu Cai, Adele Lu Jia, and Jicong Fan. Kernel readout for graph neural networks. In Kate Larson, editor, Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 , pages 2505-2514. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: 10.24963/ijcai.2024/277. URL https://doi.org/10. 24963/ijcai.2024/277 . Main Track.

Zhichen Zeng, Ruike Zhu, Yinglong Xia, Hanqing Zeng, and Hanghang Tong. Generative graph dictionary learning. In International Conference on Machine Learning , pages 40749-40769. PMLR, 2023.

Appendix

Table of Contents

A Generalization Comparison to GIN with Global Pooling

To see the advantage of our GRDL, we compare it with GIN. The only difference between GRDL and GIN is that GRDL uses a reference layer while GIN uses readout. We add an r ′ -layer MLP as the classifier after message-passing modules in the GIN. The following theorem gives an upper bound of GIN's generalization error:

Theorem A.1 (Generalization bound of GIN) . Let n = min i n i , c = ∥ ˜ A ∥ σ , and ¯ d = max i,l d ( l ) i . Denote R G := c 2 L ∥ X ∥ 2 2 ln(2 ¯ d 2 ) ( ∏ L l =1 ( ∏ r i =1 κ ( l ) i ) 2 )( ∑ L l =1 ∑ r i =1 ( b ( l ) i κ ( l ) i ) 2 / 3 ) 3 . For graphs G = { ( G i , y i ) } N i =1 drawn i.i.d from any probability distribution over X × { 1 , . . . , K } , with probability at least 1 -δ , GIN network with mean readout satisfies

This bound is derived using the same techniques as our GRDL bound in Theorem 3.2. To compare these two bounds, we only need to compare the following two terms:

where v 1 = 64 θKµ 2 n R G , v 2 = Km ¯ d , and v 3 = 24 √ θNb D µ √ m . Our observations are as follows.

The θ in v 1 can be absorbed into W ( L ) r and { D k } K k =1 . Since n ≫ K , we conclude that 64 θK n is smaller than 1 in practice. Therefore, v 1 ≤ µ 2 R G . √

In Theorem A.1, ∏ r ′ i =1 κ ( L +1) i is typically larger than 3 for r ′ > 1 based on empirical observations. We also observe that ∏ r ′ i =1 κ ( L +1) i may be smaller than 1 for r ′ = 1, but the linear classifier's expressive capacity is very limited and thus has large training error. Therefore, we focus on the case where r ′ > 1 .

v 2 = Km ¯ d and v 3 = 24 θNb D µ √ m are much smaller than R ′ G as well as R G , i.e., v 2 ≪ R ′ G and v 3 ≪ R ′ G . The reason is that R ′ G and R G involve the multiplication of terms related to c , ∥ X ∥ 2 2 , and κ ( l ) i .

Now we can conclude that Q GRDL < Q GIN. Therefore, the generalization error upper bound of GIN is larger than of GRDL, meaning our GRDL generalizes better on unseen data than GIN in the worst case. It is worth noting that these results apply to other GNNs such as GCN.

We now use numerical experiments on real datasets to support our claim. The training and testing accuracy of GRDL and GIN are shown in Table 4. We see that the training accuracy of our GRDL is close to that of GIN, but the testing accuracy of our GRDL is much higher than that of GIN. This means that GRDL and GIN have similar training errors but the former has a stronger generalization ability.

B Theory and Experiments of GRDL with Multiple Reference Distributions

Currently, we use K reference distributions for classification (one for each class). One natural approach to enhance the model's expressive power is to increase the number of reference distributions for each class. However, counterintuitively, our empirical observations suggest that having only one reference per class is optimal. In this section, we will explore and provide insights into this phenomenon.

Suppose we have P reference distributions for each class, i.e. D ≜ { D ( p ) k } p ∈ [ P ] k ∈ [ K ] , where D ( p ) k is the p -th reference in the k -th class. The prediction in Equation (5) is changed to

The training algorithm is nearly the same as our GRDL with one reference per class (Algorithm 1) except for the mini-batch training loss because of the multiple references, i.e.,

̸

We compare the model with P = 2 (GRDL-2) and P = 3 (GRDL-3) with our GRDL ( P = 1 ). Table 5 shows the classification accuracy of the models on the benchmark datasets.

To explain why GRDL performs better than the models with more references, we first introduce the following theorem

Theorem B.1. Let n be the minimum number of nodes for graphs { G i } N i =1 , θ be the hyper-parameter in the Gaussian kernel (Equation (15) ), c = ∥ ˜ A ∥ σ . For graphs G = { ( G i , y i ) } N i =1 drawn i.i.d from any probability distribution over X ×{ 1 , . . . , K } and references { D ( p ) k } p ∈ [ P ] k ∈ [ K ] , D ( p ) k ∈ R m × d , with probability at least 1 -δ , every loss function l γ and network f ∈ F under Assumption 3.1 satisfy

where

This is essentially a more general version of Theorem 3.2. The following is a brief proof of this theorem.

Proof. The only difference between multiple reference distributions and a single reference distribution comes from the calculation of s ij .

Then with minor modifications of proof of Lemma F.14, the covering number of F is given by

.

Then the theorem can be proved using the same process as the proof of Theorem 3.2.

Let ¯ κ = max i,l κ ( l ) i , ¯ b = max i,l b ( l ) i κ ( l ) i and suppose δ is sufficiently large. The bound in Theorem B.1 can be simplified to

Empirically, we observe that the training loss ˆ L γ and the misclassification rate are nearly the same for small P and large P as shown in Figure 4. Therefore, smaller P implies tighter generalization bound in (26). This means that one reference distribution for each class ( P = 1 ) may be the optimal choice.

Another explanation is that, the nodes embeddings of graphs in the same class can be regarded as samples drawn from a single discrete distribution, thus learning a single reference distribution is sufficient to provide high classification accuracy. On the other hand, the union of multiple distributions can be regarded as a single distribution.

C Detailed Training Algorithm of GRDL

In practice, since the scale of θ is different from the scale of other parameters in the model, a different learning rate is used to update it. Suppose Adam [Kingma and Ba, 2014] is used to optimize the parameters, then the training of GRDL model is presented in Algorithm 1.

D More Experiments

D.1 Dataset Statistics

The statistics of the datasets are reported in Table 6. PC-3, MCF-7, and ogbg-molhiv are three large graph datasets.

MUTAG

IMDB-BINARY

P =1

P=1

0.55

0.6

P=2

P=2

P=3

0.50

P=3

Mis-classfication Rate

0.5

Mis-classfication Rate

0.45

0.4

0.40

0.3

0.35

0.30

0.2

0.25

0.1

0.20

0

10

20

30405060708090100

0

10

20

030405060708

90100

Epoch

Epoch

Algorithm 1 GRDL Training

1: Input: Graphs { G i } N i =1 , α 1 the learning rate of w and D , α 2 the learning rate of θ , batch size B .

2: Initialize w , D , and θ .

3: repeat

4: Sample a minibatch { G i : i ∈ B , |B| = B }

5: s i ← f w , D ,θ ( G i ) , i ∈ B

6: ˆ y i ← softmax( s i ) , i ∈ B

8: ( g w , g D , g θ ) ←∇ w , D ,θ L

9: ( w , D ) ← ( w , D ) + α 1 · Adam(( w , D ) , ( g w , g D ))

10: θ ← θ + α 2 · Adam( θ, g θ )

11: until convergence conditions are met

12: Output: f w , D ,θ

D.2 Experiment Settings

In the GNN literature, researchers typically perform 10-fold cross-validation and report the best average accuracy along with standard deviation [Xu et al., 2018, Papp et al., 2021, Maron et al., 2019]. But here, we adopt a different strategy used in [Vincent-Cuaz et al., 2022]. We quantify the generalization capacities of models by performing a 10-fold cross-validation with a holdout test set which is never seen during training. The validation accuracy is tracked every 5 epochs, and the model that maximizes the validation accuracy is retained for testing. This setting is more realistic than a simple 10-fold CV and can better quantify models' generalization performances [Bengio and Grandvalet, 2003]. However, the test sets for MUTAG and PTC-MR contain only 18 and 34 graphs respectively, making them too small for assessing generalization ability. Therefore, for MUTAG and PTC-MR, we use 10-fold cross-validation following [Xu et al., 2018]. Notice that kernel-based methods do not require a stopping criterion dependent on a validation set, so we report results of 10-fold nested cross-validation repeated 10 times following [Titouan et al., 2019].

D.3 Hyper-parameter Settings and Parameter Initializations

Model For all the baselines, we adopt the hyper-parameters suggested in the original papers. For our methods, we use GIN [Xu et al., 2018] layers as the embedding network. Every GIN layer is an

̸

MLP of 2 layers ( r = 2 ) with batch normalization, whose number of units is validated in { 32 , 64 } for all datasets. The parameter λ is validated in { 0 . 1 , 1 } . We validate the number of GIN layers ( L ) in { 3 , 4 , 5 , 6 , 7 , 8 , 9 } . For each dataset, the references' dimension ( m ) is validated in the minimum number of nodes (G1), average of the minimum and median number of nodes (G2), median number of nodes (G3), average of the median and maximum number of nodes (G4), and the maximum number of nodes (G5) of graphs in the dataset. The reference for each class is initialized as follows: 1) If there is a m -node graph in the dataset that belongs to the corresponding class, then the reference is initialized as node embeddings of the graph. 2) If no graph in the class has m nodes, then we perform K-Means clustering on the graphs of the class, and the m clustering center is chosen to be the initial reference.

Optimization The models are trained with Adam optimizer with an initial learning rate α 1 = 10 -3 for network weights and references. The learning rate α 1 decays exponentially with a factor 0.95. Since the Gaussian kernel parameter θ is small in practice (around 10 -3 in our model), it is hard to choose a learning rate for it. Therefore, we consider π = 1 θ instead. π is initialized to 500 and its initial learning rate α 2 is validated in { 0 . 1 , 1 } . The batch size for all datasets is fixed to 32.

Detailed hyper-parameters setting can be found in Table 7.

.

D.4 Time Complexity

To provide a comprehensive understanding, we first show the forward propagation time complexity of the three models for a single graph G = ( A , X ) , A ∈ R n × n , X ∈ R n × d . Given that all three models employ a Graph Neural Network (GNN) for obtaining node embeddings, we denote the complexity and the number of parameters of the GNN embedding part as C 1 and N 1 , respectively. Additionally, both OT-GNN and TFGW are augmented with an MLP for classification, introducing

an extra complexity C 2 and additional parameters N 2 . In the case of our GRDL, the number of references aligns with the number of classes K .

Let's assume the number of references for OT-GNN is K 1 , and for TFGW, it is K 2 . Notably, TFGW and OT-GNN usually choose K 1 = 2 K and K 2 = 2 K . Additionally, assuming that all references D i ∈ R m × d . Since the GW distance is iteratively computed in Vincent-Cuaz et al. [2022], we denote the number of iterations for convergence as T . The time complexity and the number of parameters are outlined in Table 8.

Notice that Wasserstein distance can be approximated by sinkhorn iterations with complexity O (( m + n ) 2 ) per iteration [Cuturi, 2013]. But in practice, the exact calculation with O (( m + n ) 3 log ( m + n )) complexity empirically gives better performance in terms of both precision and speed, so it is implemented in the original paper of OT-GNN [Chen et al., 2021]. Theoretically, our GRDL has lower prediction time complexity and a reduced parameter count. Table 9 shows the empirical prediction time per graph

D.5 Ablation Study

We consider two variants of GRDL. The first one is GRDL with a fixed θ = 10 -2 in the Gaussian kernel. The other is GRDL with λ = 0 in (9), which does not have the discriminativeness constraint on the references. We also include another baseline where we first use sum pooling over node embeddings and get the graph embedding vectors. The graph vectors are then used to compare with references (vectors in this case) with discrimination loss. The classification results of benchmark datasets are given in Table 11. The original model with learnable θ and discriminativeness constraints consistently outperforms the other two.

D.6 Influence of θ

In our model, we initialized the Gaussian kernel hyper-parameter θ to 2 × 10 -3 and it was adaptively learned during training. Actually, all values between 1 × 10 -4 and 1 × 10 -1 give similar performance, as it is adaptively adjusted in the training. The initialization of the neural network parameters and the reference distributions cannot guarantee that x is close to x ′ . If θ is too large, the Gaussian kernel k ( x , x ′ ) = exp( -θ ∥ x -x ′ ∥ 2 2 ) will be too sharp, which will lead to almost zero values. Hence, MMD will fail to effectively quantify the distance between the embeddings and reference distributions, as shown in Table 12.

D.7 Experiments on The Generalization Error Bound

Use moderate-size message passing network We choose the training loss to be the cross-entropy loss ( λ = 0 ) and l γ is chosen the same as the training loss. We validate the number of GIN layers L ∈ { 3 , 4 , 5 , 6 } and the number of MLP layers for each GIN layer r ∈ { 2 , 3 , 4 } . L γ is set to be the validation loss and ˆ L γ is set to be the training loss. From Figure 5, we have the following observations

For any fixed r , if L is increasing, the empirical risk ˆ L γ increases, and the population risk L γ either increases ( r = 3 and r = 4 ) or first decreases then increases ( r = 2 ).

For any fixed L , if r is increasing, the population risk L γ first decreases then increases in most of the cases, and the empirical risk ˆ L γ decreases in most of the cases.

These observations align with our bound and support our claim that moderate-size GNN should be used.

Use moderate-size references We validate the reference size ( m ) in our experiments on real datasets, as detailed in Appendix D.3. The results in Appendix D.3 show that moderate-size references (G2, G3, G4) provide better generalization results. Here, we present the classification results for MUTAG and PROTEINS by choosing m from more fine-grained sets. For MUTAG, m is chosen from 1 , 2 , . . . , 30 . For PROTEINS, m is chosen from 1 , 3 , 5 , . . . , 61 . The figures in Figure 6 show that our model performs the best when a moderate m is used.

Regularization on references norm barely helps Consider a model regularizing the norm of references ∥ D ∥ 2 (GRDL-R)

The hyper-parameter λ 2 is set as 0 . 01 . Table 10 compares the empirical results of GRDL model and the results of GRDL-R model. From the table, we can see that GRDL performs better than GRDL-R on all datasets, which verifies our discussion in Section 3.3. Therefore, the regularization of references barely helps in our model.

D.8 Parameter Number and Time Cost Comparison with Graph Transformers

Table 13 compares the number of parameters and training time per epoch of our GRDL with two graph transformers method. We can see that our GRDL has significantly fewer parameters and training time, making it preferable. All experiments are conducted on one RTX3080.

r = 2, L =3

r = 2, L = 4

r = 2, L = 5

r = 2, L = 6

0.695

train

0.695

train

0.695

train

0.695

train

0.690

valid

0.690

valid

0.690

valid

0.690

valid

0.685

0.685

0.685

0.685

0.680

0.680

0.680

SSOI

0.675

0.675

0.670

0.670

0.670

0.670

0.665

0.665

0.665

0.665

0.660

0.660

0.660

0.655

0102030405060708090100

0102030405060708090100

0102030405060708090100

0102030405060708090100

r=3, L=3

r = 3, L = 4

r=3, L =5

r=3,L =6

0.70

0.695

train

1

train

train

0.70

train

0.690

valid

0.69

valid

0.69

valid

0.69

valid

0.685

0.68

0.68

0.68

0.680

SSOl

loss

0.67

0.67

0.670

0.66

0.665

0.66

0.66

0.660

0.65

0.655

0.65

0.65

0102030405060708090100

0102030405060708090100

0102030405060708090100

0102030405060708090100

r = 4, L = 3

r = 4, L = 4

r = 4, L = 5

r = 4, L = 6

0.695

train

train

train

train

0.690

valid

0.69

valid

0.69

valid

0.69

valid

0.685

0.680

0.68

0.68

0.68

SSOl

loss

SSol

0.670

0.67

0.67

0.67

0.665

0.66

0.66

0.66

0.660

0.65

0102030405060708090100

0102030405060708090100

0102030405060708090100

0102030405060708090100

.

E Proof Setups

Vector and matrix norms The ℓ 2 -norm ∥ · ∥ 2 is always computed entry-wise; thus, for a matrix, it corresponds to the Frobenius norm. The metric ρ of function spaces is defined as the ℓ 2 -norm of the difference between the outputs of functions given some input X , i.e.,

Finally, let ∥ · ∥ σ be the spectral norm and ∥ · ∥ p,q be the ( p, q ) matrix norm defined by ∥ A ∥ p,q := ∥ ( ∥ A : , 1 ∥ p , . . . , ∥ A : ,m ∥ p ) ∥ q for A ∈ R d × m .

0.13

0.12

0.11

0.10

Mis-classficationRate

0.09

0.08

1357911131517192123252729

m

(a) MUTAG

(b) MUTAG

0.190

Mis-classfication Rate

0.185

0.180

0.175

159 13172125293337414549535761

m

Rademacher complexity Rademacher complexity is a standard complexity measure of hypothesis function space. Given dataset G and hypothesis function space F , the Rademacher complexity is defined as

where σ 1 , . . . , σ N are independent Rademacher variables.

Then the bound can be derived with the help of the following lemma:

Lemma E.1. Given hypothesis function space F that maps a graph G ∈ X to R K and any γ > 0 , define F γ := { ( G,y ) ↦→ l γ ( f ( G ) , y ) : f ∈ F} . Then, with probability at least 1 -δ over a sample

This Lemma is a standard tool in Rademacher complexity [Mohri et al., 2018]. The only problem left is to calculate the Rademacher complexity R G ( F γ ) .

Covering number complexity bounds Direct calculation of the Rademacher complexity is often hard and the covering number is typically used to upper bound it. V is an ϵ -cover of U with respect to some metric ϱ if for all v ∈ U , there exists v ′ ∈ V such that ϱ ( v, v ′ ) ≤ ϵ , meaning

The covering number N ( ϵ, U, ϱ ) is defined as the least cardinality of the subset V . With covering number, the Rademacher complexity is upper bounded by the following Dudley entropy integral:

Lemma E.2 (Lemma A.5 of Bartlett et al. [2017], reformulated) . Let F γ be a real-valued function class taking values in [0 , γ ] , and assume that 0 ∈ F γ . Then

Now the only thing left is to bound N ( ϵ, F γ , ρ ) .

F Proof for Theorems, Corollaries, and Lemmas

F.1 Correctness Analysis

We first provide the formal definition of correct classification:

̸

Definition F.1 (Correctness of Classification) . For a graph G i with node embedding H i belonging to the k -th class, it is correctly classified if MMD( H i , D k ) < min j = k MMD( H i , D j ) .

Lemma F.2 (Correctness of Classification) . The classification of a graph G i belonging to the k -th class (with latent node embedding H i ) is correct with probability at least 1 -δ if

̸

Lemma F.2 suggests that a larger reference distributions size ( m ) and graphs with more nodes ( n ) induce a smaller ( 1 √ m + 1 √ n )( 4 + 4 √ log 2 δ ) , making correct classification easier. Proof of this theorem is provided in Appendix F.10.

F.2 Lipschitz properties

This section proves some useful lemmas related to functions' lipschitz property.

Proof. First consider matrices X , Y ∈ R d × d where X , Y are positive semi-definite. Y is unitarily diagonalizable, means Q Λ Q -1 where Λ = diag ( λ 1 , . . . , λ d ) is the diagonal matrix of eigenvalus of Y . Then we have

Let P = Q -1 XQ , and λ 0 = max i λ i , we have

Take X = ( Z -Z ′ ) ⊤ ( Z -Z ′ ) and Y = WW ⊤ , easy to see that

Lemma F.4. If σ : R d → R d is κ -Lipschitz along ever coordinate, then it is κ -Lipschitz according to ∥ · ∥ p for any p ≥ 1 .

Proof. For any z, z ′ ∈ R d ,

LemmaF.5 (Lemma A.3 of Bartlett et al. [2017]) . For every j and every p ≥ 1 , M ( · , j ) is 2 -Lipschitz w.r.t ∥ · ∥ p .

̸

Proof. Let v, v ′ , j be given. Without loss of generality, suppose M ( v, j ) ≥ M ( v ′ , j ) . Choose coordinate i = j so that M ( v ′ , j ) = v ′ j -v ′ i . Then

̸

̸

Lemma F.6. For every p > 1 , r ζ ( -M ( · , y )) is 2 ζ -Lipschitz w.r.t ∥ · ∥ p .

Proof. Recall that,

̸

Then the proof is trivial be Lemma F.5.

Lemma F.7. Cross entropy loss l ( x , y ) = -∑ K k =1 y k log exp ( x k ) ∑ K j =1 exp ( x j ) is √ 2 -Lipschitz w.r.t ∥ · ∥ 2 .

Proof. Since l is differentiable, it is sufficient to find µ such that ∥∇ l ∥ 2 ≤ µ . Without loss of generality, suppose y i = 1 , then l ( x , y ) = -log exp ( x i ) ∑ K j =1 exp ( x j ) . Let s = ∑ k exp x k , we have

̸

̸

Therefore

̸

where ( a ) is because Cauchy-Schwarz inequality.

F.3 Covering number

The following lemma provides an upper bound of the covering number for the network F G .

Lemma F.8 (Covering number bound of F G ) . Let c = ∥ ˜ A ∥ σ and ¯ d = max i,l d ( l ) i . Given an L -layer GIN message passing network F G , for any ϵ > 0

where R G = c 2 L ∥ X ∥ 2 2 ln(2 ¯ d 2 ) ( ∏ L l =1 κ 2 l )( ∑ L l =1 ( τ l ) 2 3 ) 3 and κ l = ∏ r j =1 κ ( l ) j , τ l = ( ∑ r i =1 ( b ( l ) i κ ( l ) i ) 2 / 3 ) 3 / 2 .

Firstly, we introduce the core lemma used to find the covering number of compositions of multiple hypothesis function classes.

Lemma F.9. Given hypothesis function classes F 1 , F 2 , . . . F k that maps input from matrix space to output in matrix space and their covering radius ( ϵ 1 , ϵ 2 , . . . , ϵ k ) . Assume all functions in F i is κ i -Lipschitz w.r.t. ∥ · ∥ 2 , and ln N ( ϵ i , F i , ρ ) ≤ g ( η i ) for some function g with parameters η i ( η i can be multi-valued). Then there exists ϵ -cover C of F = F k ◦F k -1 ◦· · · F 1 with ϵ = ∑ k i =1 ( ϵ i ∏ k j = i +1 κ j )

such that

Proof. Inductively construct covers as follows.

Let C 1 be the ϵ 1 -cover of F 1 . By our assumption,

Let C j as a ϵ j -cover of F j ◦ · · · ◦ F 1 . Suppose ln |C j | ≤ ∑ j i =1 g ( η i ) . For every f ′ j ◦ . . . f ′ 1 ∈ C j , we construct C j +1 ,f ′ j ,...,f ′ 1 as an ϵ j +1 -cover of F j +1 ◦ f ′ j ◦ · · · ◦ f ′ 1 . Define

̸

It is clearly a cover of F j +1 ◦ F j ◦ · · · ◦ F 1 By our assumption, we know that

By the inductive arguments above, we can conclude that

Next, inductively find the cover radius ϵ for C .

It is trivial in the base case that the cover of C 1 is ϵ 1 .

Suppose for C h , the cover radius satisfies

For all f h +1 ◦ f h ◦ · · · ◦ f 1 ∈ F h +1 ◦ F h ◦ · · · ◦ F 1 , there exists f ′ h +1 ◦ f ′ h ◦ · · · ◦ f ′ 1 ∈ C h +1 such that

By the inductive arguments,

Then, it is trivial to see

The following matrix covering number is well-known and the detailed proof can be found in Bartlett et al. [2017].

Lemma F.10. Let conjugate exponents ( p, q ) and ( r, s ) be given with p ≤ 2 , as well as positive reals ( a, b, ϵ ) and positive integer m. Let matrix X ∈ R n × d be given with ∥ X ∥ p ≤ b . Then

For the composition of a hypothesis function class and a κ -Lipschitz function, we have the following lemma

Lemma F.11. Suppose ψ is a κ -Lipschitz function, then ln N ( ϵ, ψ ◦ F , ρ ) ≤ ln N ( ϵ/κ, F , ρ )

Proof. Let C denote ϵ κ -cover of F , then for any f ∈ F , there exists f ′ ∈ C such that ρ ( f, f ′ ) ≤ ϵ κ . Let Z denote the input, we have

Lemma F.12. Given H , H ′ ∈ R n × d and D , D ′ ∈ R m × d . Squared MMD distance with gaussian kernel k ( x, y ) = exp { -θ ∥ x -y ∥ 2 2 } satisfies

Proof. The matrices have the form

Then we have

In the above derivation, (a) holds due to ∣ ∣ exp( -x 2 ) -exp( -y 2 ) ∣ ∣ ≤ | x -y | for any x, y ≥ 0 , (b) holds due to the triangle inequality, and (c) holds by the Cauchy-Schwarz inequality.

The covering number for a single GIN message passing layer F l with the following Lemma:

Lemma F.13 (Covering number of F l ) . Let c = ∥ ˜ A ∥ σ . For any l ∈ [ L ] and ϵ > 0

where κ i = ∏ j ≤ r κ ( i ) j , τ l = ( ∑ r i =1 ( b ( l ) i κ ( l ) i ) 2 / 3 ) 3 / 2

Proof. With a little abuse of notation, remove the superscript in Equation (19) for now

where B i := { W i : ∥ W ⊤ i ∥ σ ≤ κ i , ∥ W i ∥ 2 , 1 ≤ b i } . Denote F i = { Z ↦→ σ ( ZW i ) : W i ∈ B i } for i ∈ [ r -1] , F r = { Z ↦→ ZW r : W r ∈ B r } , then

For any f i ∈ F i , i ∈ [ r -1] with arbitrary input Z , Z ′

Similarly, for any f r ∈ F ′ r , Lemma F.3 gives

Denoting Z i -1 as the input to F i ( Z 0 = Z = ˜ AH ) and using the Lipschitz conditions, we have

for any f i ◦ f i -1 ◦ · · · ◦ f 1 ∈ F i ◦ F i -1 ◦ · · · ◦ F 1 . So ∥ Z i -1 ∥ 2 ≤ c ( ∏ i -1 j =1 κ j ) ∥ H ∥ 2 ≜ c i -1 . By Lemma F.10 and Lemma F.11, we have

W is the maximum dimension of weight matrices (as previously defined in the main text). Thus by Lemma F.9, we have the covering number

with cover radius ϵ = ∑ r i =1 ( ϵ i ∏ r j = i +1 κ j ) . Next we need to choose ϵ i to minimize the right hand side of the above inequality. Holder's inequality states that when 1 p + 1 q = 1 ,

With the covering number of F G , we can calculate the covering number of F in Equation (10). Lemma F.14 (Covering number of F ) . Suppose θ in the kernel (Equation (15) ) is fixed. For any ϵ > 0 √

where R G is defined the same as Lemma F.8.

Proof. Denote S ∈ R N × K as the output of function F . Consider the entry ( i, j ) of S , by Lemma F.12, it has

Then for the whole matrix S ,

where (a) holds due to ( x + y ) 2 ≤ 2( x 2 + y 2 ) . If ∥ H -H ′ ∥ 2 ≤ ϵ 1 and ∥ D -D ′ ∥ 2 ≤ ϵ 2 , then

by choosing ϵ 1 = √ n 8 √ Kθ ϵ and ϵ 2 = √ m 8 √ Nθ ϵ . Let B D := { D ∈ R Km × d : ∥ D ∥ 2 ≤ b D } . It is well-known that there exists an ϵ 2 -cover obeying

Denote the output space of F G , F as H , Z respectively, we can bound the covering number as

It follows from Lemma F.8 and inequality (34) that

F.4 Proof of Lemma E.2

The Dudley entropy integral bound used by Bartlett et al. [2017] is

Lemma F.15 (Lemma A.5 of Bartlett et al. [2017]) . Let F be a real-valued function class taking values in [0 , 1] , and assume that 0 ∈ F . Then

Lemma E.2 can be proved with simple modifications.

Proof. Let F ′ = ψ ◦ F γ where ψ ( x ) = 1 γ x , then F ′ is a real-valued function class taking values in [0 , 1] . By Lemma F.15, it has

Multiplying both side by γ , it has

F.5 Proof of Lemma F.8

For any f l ∈ F l , given input Z , it has

which means all f l ∈ F l is cκ l -Lipschitz. Applying Lemma F.9, we have

with ϵ = ∑ L l =1 ( ϵ l ∏ r j = l +1 cκ j ) . The only thing left is to minimize ∑ L l =1 τ 2 l ( ∏ l j =1 cκ j ) 2 ϵ 2 l by controlling ϵ l 's. Choose α 2 l = τ 2 l ( ∏ l j =1 cκ j ) 2 , β l = ∏ L j = l +1 cκ j . Choose p = 1 3 , q = 2 3 , and apply

Holder's inequality in the same ways as Equation (33), this yields

Thus derives the conclusion

F.6 Proof of Theorem 3.2

Proof. Since l γ ( · , y ) is µ -Lipschitz, we can bound the covering number of F γ (defined in Lemma E.1)

Denote v 1 = 64 θKR G µ 2 n , v 2 = Km ¯ d, v 3 = 24 θNb D µ √ m , then by Lemma E.2, we can bound the Rademacher complexity

where (a) holds due to ln 1 x ≤ 1 x 2 , (b) holds due to √ x + y ≤ √ x + √ y . For (c), we have chosen α = 1 √ N .

It can be shown that the covering number bound of F D satisfies N ( ϵ, F D , ρ ) ≤ (3 b D /ϵ ) Kmd (Lemma F.14). Combining the bounds of N ( ϵ, F G , ρ ) and N ( ϵ, F D , ρ ) and Lemmas E.1 and E.2, we derive the generalization bound:

This finished the proof.

F.7 Corollary of Theorem 3.2

Corollary F.16 (Mis-classification rate upper bound of GRDL) . Let n be the minimum number of nodes for graphs { G i } N i =1 , θ be the hyper-parameter in gaussian kernel (Equation (15) ), c = ∥ ˜ A ∥ σ . For graphs G = { ( G i , y i ) } N i =1 drawn i.i.d from any probability distribution over X × { 1 , . . . , K } and references { D k } K k =1 , D k ∈ R m × d , with probability at least 1 -δ , every margin ζ > 0 and network f ∈ F under Assumption 3.1 satisfy

̸

where

̸

where M ( v, y ) := v y -max i = y v i is the margin operator and

is called the ramp loss. The population ramp risk is defined as L ζ ( f ) := E G ∼X [ r ζ ( -M ( f ( G ) , y )))] . Given the graph dataset G sampled from X , the empirical ramp risk is ˆ L ζ ( f ) := N -1 ∑ N i =1 r ζ ( -M ( f ( G i ) , y i )) . It is clear that 1 [arg max j f ( G ) j = y ] ≤ r ζ ( -M ( f ( G ) , y )) , so

̸

̸

̸

It is easy to see that γ = 1 in this case. Also by Lemma F.6, µ = 2 ζ . Then it is trivial to get the bound by Theorem 3.2 with simple substitution

Pr

G

∼X

j

f

(

G

)

j

̸

=

y

]

L

ζ

(

f

)

̸

where v 1 = 256 θKR G nζ 2 , v 2 = Km ¯ d, v 3 = 48 b D √ θN √ mζ , R G = c 2 L ∥ X ∥ 2 2 ln(2 ¯ d 2 ) ( ∏ L l =1 ( ∏ r i =1 κ ( l ) i ) 2 ) ( ∑ L l =1 ∑ r i =1 ( b ( l ) i κ ( l ) i ) 2 / 3 ) 3 . Also, r ζ ( -M ( f ( G i ) , y i )) ≤ 1 [ f ( G i ) y i ≤ ζ +argmax j = y i f ( G i ) j ] , so we have

̸

Corollary F.17 (Generalization bound with cross-entropy loss) . Suppose l γ ( · , y ) is the cross-entropy loss L CE (7) . Let n be the minimum number of nodes for graphs { G i } N i =1 , θ be the hyper-parameter in gaussian kernel (Equation (15) ), c = ∥ ˜ A ∥ σ . For graphs G = { ( G i , y i ) } N i =1 drawn i.i.d from and ˆ L ζ ( f ) ≤ N -1 ∑ i 1 [ f ( G i ) y i ≤ ζ +argmax j = y i f ( G i ) j ] .

Proof. Choose the loss l γ ( · , y ) as

̸

any probability distribution over X × { 1 , . . . , K } and references { D k } K k =1 , D k ∈ R m × d , with probability at least 1 -δ , every network f ∈ F under Assumption 3.1 satisfy

Proof. According to Lemma F.7, µ = √ 2 . Then the proof is trivial by substituting it into Theorem 3.2.

F.8 Adjacency matrix spectral norm

Lemma F.18. Let G = ( V, E ) be an undirected graph with adjacency matrix A ∈ R n × n , d G be the maximum degree of G. Then, the adjacency matrix satisfies

Proof. Based on the definition of the spectral norm, we have

where (a) is because A

Lemma F.19. For any matrix X ∈ R m × n , ∥ X ∥ σ ≤ ∥ X ∥ 2 ≤ √ r ∥ X ∥ σ where r = rank ( X )

Proof. By the definition of spectral norm,

where λ max denotes the largest eigenvalue. Since X ⊤ X is a positive semi-definite real symmetric matrix, it must has n real eigenvalues that can be ordered as

Then it has

Lemma F.20. Let G = ( V, E ) be a graph with adjacency matrix A ∈ R n × n . Assume | E | > 0 , then c = ∥ ˜ A ∥ σ = ∥ A + I ∥ σ > 1 .

where (a) is because | E | > 1

F.9 Generalization of MMD (to be used in Section F.10)

Before stating the lemma, we first give alternative definitions of MMD. Let P be a continuous probability distribution of some random variable Z taking values from space Z . Then, the kernel mean embedding of P associated with the continuous, bounded, and positive-definite kernel function k : Z × Z → R is

which is an element in the Reproducing Kernel Hilbert Space (RKHS) H associated with kernel k . In many practical situations, it is unrealistic to assume access to the true distribution P . Instead, we only have access to samples P = { z i } n i =1 from P . We can approximate (35) by the empirical kernel mean embedding

For another continuous distribution Q with samples Q = { z ′ i } i m =1 , the MMD between the two probability distribution is defined as

Denote d := MMD( P , Q ) and ˆ d := MMD( P, Q ) , we have the follow Lemma:

Lemma F.21. With probability at least 1 -δ we have

Proof. We use the following Lemma:

Lemma F.22 (Theorem 7 of Gretton et al. [2012a], reformulated) . Assume 0 ≤ k ( x, y ) ≤ K . Then with probability at least 1 -2 exp ( -ε 2 mn 2 K ( m + n ) )

Let δ = 2exp ( -ε 2 mn 2 K ( m + n ) ) , then it has

Therefore, with probability at least 1 -δ

and the empirical MMD is

F.10 Proof of Lemma F.2

Proof. For simplicity, let k ′ := arg min j = k MMD( H i , D j ) . Since H i and D k are finite samples from P k , and D k ′ is sampled from P k ′ , by Lemma F.21, we have

̸

Therefore, with probability at least 1 -2 δ ′ (union bound), we have

It follows that

By Definition F.1, to ensure correct classification, we can let

This means

Therefore, if (37) holds, the classification is correct with probability at least 1 -2 δ ′ . Letting δ = 2 δ ′ , we finish the proof.

F.11 Proof of Theorem A.1

Let H ∈ R ∑ i n i × d be the output of the message passing layers. Then the mean readout is a matrix multiplication

It is easy to see ∥ M ∥ σ ≤ 1 . Since a MLP is concatenated after readout, by the proof of Lemma F.8, the covering number of GIN F ′ is

Then by Lemma E.2, the generalization bound can be easily derived by taking α = 1 √ N

i i . .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides the full set of assumptions and a complete proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports information about the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: All the datasets used in the paper are open-source. There is no negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators and original owners of assets used in the paper are properly credited and the license and terms of use explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInstead of compressing node embedding matrix into a graph-level vector, this paper proposes a reference distribution learning method GRDL especially designed for graph classification task. GRDL achieves graph classification by measuring the dissimilarity between distributions of the input graph and the references. Theoretical analysis of the generalization error bound offers guidance of hyperparameter tuning.\n\nSTRENGTHS:\n- This paper is well-motivated to avoid information loss caused by node embedding downsampling.\n- The idea of considering graph classification as a distribution comparison problem is interesting.\n- The authors have solid mathematical skills.\n\nWEAKNESSES:\n- The idea is borrowed from domain transfer learning but lacks adaption to graph classification task. In graph classification task, graphs in each dataset “are drawn i.i.d” as the authors mentioned at line 205. This means that although graphs have different classes, they belong to an identical distribution, which is contradictory to the hypothesis of this paper that different classes belong to different reference distributions.\n- In my opinion, this paper didn’t avoid information compression compared to mean pooling and max pooling. MMD unifies the source and target inputs as a vertex in the same RKHS and then calculate the distance between them. I think the mapping from embedding matrix into a vertex can be regarded as information compression, and the calculation of mean discrepancy is similar to mean pooling.\n- The time complexity is too high, the datasets used for large-scale experiments are unreasonable, and the competitors used in efficiency experiment is unreasonable. The time complexity is $O(N^2)$, which is the same as the highly criticised time-consuming node clustering pooling methods (such as DiffPool[1]), while node dropping methods (such as SAGP[2]) only requires a time complexity of $O(E)$. The complexity restricts its scalability to larger-scale graphs. Three so-called large-scale datasets are not really large-scale. They are only large in the number of graphs but not in the number of nodes in each single graph. The authors should add experiments on synthetic large-scale datasets. Besides, in time cost comparison, the authors didn’t choose competitors with SOTA efficiency but used two time consuming methods, which is quite unreasonable.\n- The theoretical contribution of generalization error bound is limited. Firstly, the result offers fuzzy insights into the choose of hyperparameters by giving “moderate-size message passing GIN”, and “moderate-size references”. How to define “moderate”? At line 240, the authors analysis that “a network with a smaller $L4 and $r4 may guarantee a tighter bound on the population risk compared to a larger one. Therefore, a promising strategy is to use a moderate-size message passing GNN”. What’s the causality between “smaller” and “moderate-size”? That’s quite confusing. Experimental results in Appendix show that hyperparameters are still choosed on trial and error. Besides, the generalization ability comparison with GIN with mean readout is meaningless because the competitor is not the SOTA.\n- Important baselines are missing in the experiments. Please add comparison results with recent graph pooling methods such as ASAP[3], MinCutPool[4], StructPool[5], MuchPool[6], TAP[7], Wit-TopoPool[8], and MSGNN[9].\n\nReferences\n\n[1]Ying Z, You J, Morris C, et al. Hierarchical graph representation learning with differentiable pooling[J]. Advances in neural information processing systems, 2018, 31.\n\n[2]Lee J, Lee I, Kang J. Self-attention graph pooling[C]//International conference on machine learning. PMLR, 2019: 3734-3743.\n\n[3]Ranjan E, Sanyal S, Talukdar P. Asap: Adaptive structure aware pooling for learning hierarchical graph representations[C]//Proceedings of the AAAI conference on artificial intelligence. 2020, 34(04): 5470-5477.\n\n[4]Bianchi F M, Grattarola D, Alippi C. Spectral clustering with graph neural networks for graph pooling[C]//International conference on machine learning. PMLR, 2020: 874-883.\n\n[5]Yuan H, Ji S. Structpool: Structured graph pooling via conditional random fields[C]//Proceedings of the 8th International Conference on Learning Representations. 2020.\n\n[6]Du J, Wang S, Miao H, et al. Multi-Channel Pooling Graph Neural Networks[C]//IJCAI. 2021: 1442-1448.\n\n[7]Gao H, Liu Y, Ji S. Topology-aware graph pooling networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4512-4518.\n\n[8]Chen Y, Gel Y R. Topological pooling on graphs[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 7096-7103.\n\n[9]Lv Y, Tian Z, Xie Z, et al. Multi-scale Graph Pooling Approach with Adaptive Key Subgraph for Graph Representations[C]//Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2023: 1736-1745.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel algorithm called GRDL for graph classification. GRDL  treats each graph’s latent node embeddings given by GNN layers as a discrete distribution and directly classify distributions without global pooling. The authors derived generalization error bounds for GRDL and verified them numerically. The experiments on many graph datasets show the superiority of GRDL. GRDL is 10 times faster than leading competitors.\n\nSTRENGTHS:\n* Originality: The proposed algorithm GRDL and the theoretical results are novel.\n* Quality: The paper is well-organized and contains rich theoretical results and numerical comparisons. \n* Clarity: The motivation (e.g. Example 2.1), assumptions, optimization, implication of theorems, and the experimental setting have been clearly explained.\n* Significance: Graph classification is a challenging problem due to the difficulty in converting nodes’ embeddings to a vector as the global representation of each graph. The proposed method gets rid of readout operations and directly classifies the discrete distributions formed by nodes’ embeddings. It can outperform graph kernels, GIN, and graph transformers. Moreover, the paper proved that the proposed model has better generalization ability than the baseline GIN, which is a big contribution.\n\nWEAKNESSES:\nThere is no major weakness found. Please refer to my questions in the next section.\n\nQUESTIONS:\n* Besides GIN, there are other GNN models such as GCN and GAT. Why did the authors consider GIN only in the theoretical analysis and experiments?\n* I think the parameter $\\theta$ of the Gaussian kernel can be absorbed into the neural network parameter. I suggest the authors discuss the necessity of setting or optimizing $\\theta$ separately. \n* In Theorem 3.2, the bound is related to $K$, the number of classes. But, in many literature of generalization analysis, the bound is not related to the number of classes. Could the author explain the difference?\n\nLIMITATIONS:\nThe authors discussed the limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to make graph-level predictions via distribution comparison between node-level representations and discrete reference distributions. The authors claim that their proposed method avoids the requirements of graph pooling for graph-level tasks. and reduce the risk of information loss. Theoretical and empirical justification results are provided.\n\nSTRENGTHS:\n- The authors propose a novel and simple method for graph classification and provide theoretical analysis on the generalization bound. The discussion is clear and extensive.\n- The method shows advantage in time cost compared to related works.\n- The authors performed extensive ablation study on the proposed method.\n\nWEAKNESSES:\n- Equation in L138-139 (a missing equation index?) also requires summation over nodes. The description of \"avoid graph pooling operation\" seems to be overclaimed. I suggest that the authors reconsider it.\n- An ablation study on the discrimination loss and the usage of node-level representation is required. I think the main improvement in performance may be attributed to the discrimination loss. It helps the model learn distant representations for graphs of different labels in the feature space.  The performance of the proposed method should be compared with and without the discrimination loss. Besides, the authors should also implement a baseline with discrimination loss where node representations are first sum-pooling and then compared with the reference distribution.\n- Minor: Latest baseline models are required for a comprehensive empirical comparison, such as graph pooling including SEP[1], GMT[2], and CoCN[3] and graph transformers including Exphormer[4], GRIT[5], and MPNN-VN[6]. Considering the time limitation and the similarity of the datasets, a comparison on part of the datasets will be sufficient.\n\n[1] Structural entropy guided graph hierarchical pooling. ICML'22\n\n[2] Accurate Learning of Graph Representations with Graph Multiset Pooling. ICLR'22\n\n[3] All in a row: Compressed convolution networks for graphs. ICML'23\n\n[4] Exphormer: Sparse Transformers for Graphs. ICML'23\n\n[5] Graph Inductive Biases in Transformers without Message Passing. ICML'23\n\n[6] On the Connection Between MPNN and Graph Transformer. ICML'23\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nYes. The authors describe the limited performances of the proposed method on certain datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}