Title: Exploring Consistency in Graph Representations: from Graph Kernels to Graph Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Neural Networks

Representation Consistency

Graph Classification
Abstract: Graph Neural Networks (GNNs) have emerged as a dominant approach in graph representation learning, yet they often struggle to capture consistent similarity relationships among graphs.  To capture similarity relationships, while graph kernel methods like the Weisfeiler-Lehman subtree (WL-subtree) and Weisfeiler-Lehman optimal assignment (WLOA) perform effectively, they are heavily reliant on predefined kernels and lack sufficient non-linearities. Our work aims to bridge the gap between neural network methods and kernel approaches by enabling GNNs to consistently capture relational structures in their learned representations. Given the analogy between the message-passing process of GNNs and WL algorithms, we thoroughly compare and analyze the properties of WL-subtree and WLOA kernels. We find that the similarities captured by WLOA at different iterations are asymptotically consistent, ensuring that similar graphs remain similar in subsequent iterations, thereby leading to superior performance over the WL-subtree kernel. Inspired by these findings, we conjecture that the consistency in the similarities of graph representations across GNN layers is crucial in capturing relational structures and enhancing graph classification performance. Thus, we propose a loss to enforce the similarity of graph representations to be consistent across different layers. Our empirical analysis verifies our conjecture and shows that our proposed consistency loss can significantly enhance graph classification performance across several GNN backbones on various datasets.

Exploring Consistency in Graph Representations: from Graph Kernels to Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have emerged as a dominant approach in graph representation learning, yet they often struggle to capture consistent similarity relationships among graphs. While graph kernel methods such as the WeisfeilerLehman subtree (WL-subtree) and Weisfeiler-Lehman optimal assignment (WLOA) kernels are effective in capturing similarity relationships, they rely heavily on predefined kernels and lack sufficient non-linearity for more complex data patterns. Our work aims to bridge the gap between neural network methods and kernel approaches by enabling GNNs to consistently capture relational structures in their learned representations. Given the analogy between the message-passing process of GNNs and WL algorithms, we thoroughly compare and analyze the properties of WL-subtree and WLOA kernels. We find that the similarities captured by WLOA at different iterations are asymptotically consistent, ensuring that similar graphs remain similar in subsequent iterations, thereby leading to superior performance over the WL-subtree kernel. Inspired by these findings, we conjecture that maintaining consistency in the similarities of graph representations across GNN layers is crucial for capturing relational structures and improving graph classification performance. Thus, we propose a loss to enforce the similarity of graph representations to be consistent across different layers. Our empirical analysis verifies our conjecture and shows that our proposed consistency loss can significantly enhance graph classification performance across several GNN backbones on various datasets.

1 Introduction

Graph classification tasks are extensively applied across multiple domains, including chemistry [Liu et al., 2022, Xu et al., 2023], bioinformatics [Yan et al., 2019, Li et al., 2023a,b], and social network analysis [Ying et al., 2018, Wang et al., 2024]. Graph neural networks (GNNs) [Kipf and Welling, 2017, Xu et al., 2019, Velickovic et al., 2018, Huang et al., 2024] have emerged as the predominant approach for performing graph classification, owing to their ability to extract rich representations from various types of graph data. A typical GNN employs the message-passing mechanism [Gilmer et al., 2017], where node features are propagated and aggregated across connected nodes. This process effectively captures local tree structures, enabling the differentiation between various graphs. However, GNNs often struggle to preserve relational structures among graphs, resulting in inconsistent relative similarities across the layers. As shown in Figure 1, graphs with higher relative similarity in one layer may exhibit reduced similarity in the subsequent layer. This phenomenon arises from the limitations of cross-entropy loss, which fails to preserve relational structures, as it forces graphs within the same class into identical representations.

GNNLayer

Similarity(

Similarity(

1×』>←x

GNNLayer

Similarity(

Similarity(

hl-1

m

-1

GNNLayer

Similarity(

Similarity(

hi

hn

While effective in capturing relative graph similarity, kernel methods rely on predefined kernels and exhibit insufficient non-linearities, limiting their ability to capture complex patterns in highdimensional data. Additionally, kernel methods are computationally costly, making them unsuitable for handling large datasets and consequently limiting their overall applicability.

In this work, we aim to bridge the gap between kernel methods and GNN models. Given the iterative nature of GNNs, we study a class of kernels which are induced from graph representations obtained through an iterative process and name them iterative graph kernels (IGK). Within this framework, we define the consistency property, which ensures that similar graphs remain similar in subsequent iterations. Our analysis demonstrates that kernels with this property ensure better generalization ability, leading to improved classification performance.Furthermore, we find that this property sheds light on why the WLOA kernel outperforms the WL-subtree kernel. The WLOA kernel asymptotically demonstrates consistency as the iteration goes to infinity, whereas the WL-subtree kernel does not exhibit this behavior. Inspired by these findings and the analogy between message-passing GNNs and the WL-subtree kernel [Xu et al., 2019], we hypothesize that this principle is also applicable to GNNs. To explore this, we introduce a novel loss function designed to align the ranking of graph similarities across GNN layers. The aim is to ensure that the relational structures of the graphs are preserved and consistently reflected throughout the representation space of these layers. We validate this hypothesis by applying our loss function to different GNN backbones across various graph datasets. Extensive experiments demonstrate that our proposed model-agnostic consistency loss improves graph classification performance comprehensively.

In summary, the main contributions of this work are as follows:

Novel perspective: We present a novel perspective on understanding the graph classification performance of GNNs by analyzing the similarity relationships captured by different layers.

New insights: We are the first to introduce and formalize the consistency principle within both kernel-based and GNN methods for graph classification tasks. Additionally, we provide theoretical proofs explaining how this principle enhances the performance.

Simple yet effective method: Empirical results demonstrate that the proposed consistency loss universally enhances performance across a wide range of base models and datasets.

2 Preliminaries

In this section, we begin by introducing the notations and definitions used throughout the paper. Next, we provide an introduction to the fundamentals of Weisfeiler-Lehman isomorphism test, GNNs and graph kernels.

2.1 Notations and Definitions

Let G ( V , E , X ) be an undirected and unweighted graph with N nodes, where V denotes the node set, E denotes the edge set, and X denotes the feature matrix, where each row represents the features of a corresponding node. The neighborhood of a node v is defined as the set of all nodes that connected to v : N ( v ) = { u | ( v, u ) ∈ E} . In a graph dataset, each graph G i is associated with a label Y i , which is sampled from a label set L . In this paper, we focus on the graph classification task, where a model ϕ is trained to map each graph to its label.

2.2 Weisfeiler-Lehman Isomorphism Test

We first introduce the Weisfeiler-Lehman isomorphism test [Weisfeiler and Lehman, 1968], which can be used to distinguish different graphs and is closely related to the message-passing process of GNNs [Xu et al., 2019]. The WL algorithm operates by iteratively relabeling the colors of vertices in the graph. Initially, all vertices are assigned the same color C 0 . In iteration i , the color of a vertex v is updated based on its current color C v,i -1 and the colors of its neighbors { C u ∈N ( v ) ,i -1 } . The update is given as follows:

where f i c is an injective coloring function that maps the multisets to different colors at iteration i .This process continues for a predefined number of iterations or until the coloring stabilizes (i.e., the colors no longer change).

2.3 Graph Neural Network

Most GNNs adopt the message-passing framework [Gilmer et al., 2017], which can be viewed as a derivative of the Weisfeiler-Lehman coloring mechanism. Specifically, let h ( k -1) v represent the feature vector of node v at the ( k -1) -th iteration. A GNN computes the new feature for v by aggregating the representations of itself and its neighboring nodes u ∈ N ( v ) as follows:

The initial node representations h (0) v are set to the raw node features X v . At the k -th iteration, the aggregation function AGGR ( k ) ( · ) computes the messages m ( k ) v received from neighboring nodes. Subsequently, the update function UPDATE ( k ) ( · ) computes a new representation for each node by integrating the neighborhood messages m ( k ) v with its previous embedding h ( k -1) v . After T iterations, the final node representations are combined into a graph representation using a readout function:

The readout function, essentially a set function learned by the neural network, commonly employs AVERAGE or MAXPOOL .

2.4 Graph Kernel

A kernel is a function used to measure the similarity between pairs of objects. For a non-empty set χ and a function K : χ × χ → R , the function K qualifies as a kernel on χ if there exists a Hilbert space H k and a feature map function ϕ : χ → H k , such that K ( x, y ) = ⟨ ϕ ( x ) , ϕ ( y ) ⟩ for any x, y ∈ χ , where ⟨· , ·⟩ denotes the inner product in H k . Notably, such a feature map exists if and only if K is a positive semi-definite function. Let K be a kernel defined on χ , and let S = { x 1 , . . . , x n } be a finite set of n samples on χ . The Gram matrix for S is defined as G ∈ R n × n , with each element G ij = K ( x i , x j ) representing the kernel value between the i -th and j -th data points in S . The Gram matrix is always positive semi-definite.

Graph kernel methods apply the kernel approaches to graph data, typically defined using the R -convolution framework [Haussler et al., 1999, Bause and Kriege, 2022]. Consider two graphs, G and G ′ . The key idea is to decompose the graphs into substructures using a predefined feature map function ϕ , and then compute the kernel value by taking the inner product in the feature space: K ( G , G ′ ) = ⟨ ϕ ( G ) , ϕ ( G ′ ) ⟩ , based on these substructures. Weisfeiler-Lehman (WL) graph kernels stand out as one of the most widely used approaches. These methods employ the Weisfeiler-Lehman

coloring scheme to iteratively encode graphs, calculating kernel values based on these colorings. The final kernel values are derived through the aggregation of intermediate results. Next, we introduce the WL-subtree kernel and the WLOA kernel in more detail. Let f i be the coloring function at the i -th iteration, mapping the colored graph from the previous iteration to a new colored graph. Define ψ i as the function that captures the cumulative coloring effect up to the i -th iteration: ψ i = f i ◦ · · · ◦ f 1 . Specifically, the WL-subtree kernel computes the kernel value by directly using the label histogram as a feature to compute the dot product, which is expressed as:

The WLOA kernel applies a histogram intersection kernel to match substructures between different graphs, which is formulated as:

where ω ( i ) is a nonnegative, monotonically non-decreasing weight function, and ω ( i ) = 1 is commonly used in practice [Kriege et al., 2016, Siglidis et al., 2020]. The operator histmin denotes the histogram intersection kernel. It is computed by comparing and summing the smallest matching elements between two sets. For example, consider two sets S 1 : { a , a , b , b , c } and S 2 : { a , b , b , c , c } . The histmin ( S 1 , S 2 ) is calculated by taking the minimum frequency of each distinct element across the two sets, yielding min(2 , 1) + min(2 , 2) + min(1 , 2) = 4 . To ensure that the kernel values fall in the range of 0 to 1, normalization is often applied. The normalized kernel value ˜ K ( h ) is then expressed as follows:

3 Consistency Principles

To encode relational structures in GNN learning, we first examine how similarities are represented in graph kernels. In this section, we start by defining a class of graph kernels, i.e., the iterative graph kernels, which encompasses many widely used kernels. Then, we delve into a key property known as the consistency property, which may play an important role in enhancing classification performance. We support this assertion through theoretical analysis, elucidating how different kernels adhere to or deviate from this property, thereby explaining their performance differences.

3.1 Iterative Graph Kernels

In this paper, we are interested in a set of kernels defined as follows:

Definition 3.1 Given a colored graph set χ , a feature map function ϕ : χ → H k (where H k is a Hilbert space), and a set of coloring functions F c = { f 0 , f 1 , . . . , f i } on χ (with f i : χ → χ ), we define the set of iterative graph kernels (IGK) as:

where x, y ∈ χ and ψ i ( · ) represents a composite function given by: ψ i = f i ◦ · · · ◦ f 1 , Then the normalized kernel is given by:

Based on this definition, we can see that graph kernels utilizing the Weisfeiler-Lehman framework, including the WL-subtree kernel [Shervashidze et al., 2011], WLOA [Kriege et al., 2016], and the Wasserstein Weisfeiler-Lehman kernel [Togninalli et al., 2019], should be classified as iterative graph kernels. Conversely, the subgraph-pattern approaches, such as the graphlet kernel [Shervashidze et al., 2009] and the shortest-path kernel [Borgwardt and Kriegel, 2005], do not fall into this category.

3.2 Key Properties: Monotonic Decrease & Order Consistency

To effectively capture relational structures, we design the IGKs to progressively differentiate between graphs. With each iteration, additional structural features are considered, enabling the distinction of graphs that may have been indistinguishable in earlier iterations. This implies two properties: (1) the kernel values monotonically decrease with larger iterations, as the similarity between two graphs decreases with the consideration of more features; and (2) the similarity rankings across different iterations should remain consistent, meaning that graphs deemed dissimilar in early iterations should not be considered similar in later iterations. We then provide formal definitions for these two properties and prove how they can improve generalization in the binary classification task.

Definition 3.2 (Monotonic Decrease) The normalized iterative graph kernels ˜ K F c ,ϕ ( x, y, i ) are said to be monotonically decreasing if and only if:

Definition 3.3 (Order Consistency) The normalized iterative graph kernels ˜ K F c ,ϕ ( x, y, i ) are said to preserve order consistency if the similarity ranking remains consistent across different iterations for any pair of graphs, which is defined as:

Next, we show that these two properties can lead to a monotonically decreasing upper bound on the test error in the binary classification task, which suggests improved performance.

Consider a binary graph classification task and assume that the graph representations obtained at any iteration have a uniform norm. This can be achieved by simply normalizing the graph representations at the end of each iteration. That is, for any graph x : ∥ ϕ ( ψ i ( x )) ∥ = 1 . Then, for any IGK that is monotonically decreasing and preserves the order consistency, the following theorem holds:

Theorem 3.4 Let ˜ K F c ,ϕ ( x, y, i ) be a normalized iterative graph kernel. Its generalization ability on the test set is provably strong when it decreases monotonically and preserves order consistency.

Proof sketch. We consider a binary classification task where supervision is provided through labels and similarity orders. We demonstrate that the presence of these two properties enhances the generalizability of the learned iterative kernel. The proof is structured in two steps:

We first show that iterative kernels can learn the correct ranking given sufficient data during training, using the PAC-learning framework.

Next, we demonstrate that, once the correct ranking is learned, an iterative kernel learned with the properties of monotonic decrease and order consistency guarantees that the error rate of the graphs in the test set decreases as the number of iterations increases.

Details of the proof can be found in Appendix A.

3.3 Theoretical Verification with WL-based Kernels

As discussed in Section 3.1, WL-based kernels can be categorized as iterative graph kernels, as they are generated by the coloring functions in an iterative refinement process. Consequently, a natural question arises regarding how various kernels adhere to these properties and whether their adherence reflects their actual performance. We thus investigate two popular WL-based kernels: the WL-subtree kernel [Shervashidze et al., 2011] and the WLOA kernel [Kriege et al., 2016].

Theorem 3.5 The normalized WL-subtree kernel is neither monotonically decreasing nor does it preserve order consistency.

Theorem 3.6 The normalized WLOA kernel is monotonically decreasing and asymptotically preserves order consistency when ω ( i ) = 1 .

Proof sketch. For Theorem 3.5, we illustrate a counterexample, while for Theorem 3.6, we consider two graph pairs where the similarity condition holds:

The similarity at the next iteration h +1 is scaled by a factor dependent ω ( i ) , i = 1 , · · · , h +1 . The unnormalized kernel increases monotonically, though with diminishing increments over iterations. Given this, when ω ( i ) = 1 and h →∞ , we obtain:

We include the complete proof in Appendix B.1 and B.2.

These findings imply that the WLOA kernel better preserves relational structures compared to the WLsubtree kernel, leading to improved classification performance, as supported by the literature [Kriege et al., 2016].

4 Proposed Strategy

Given the analogy between WL-based kernels and GNNs [Shervashidze et al., 2011, Gilmer et al., 2017], and the observation that GNNs often fail to preserve relational structures, we hypothesize that the consistency principle is also beneficial to GNN models. Thus, we aim to explore how to effectively preserve this principle within the GNN architectures.

Probability Increase

Distance Increase

Pair-wise Cosine

Similarity

xk'sreference probabilitymatrix

Dh-1

m,n

k-throw

2(1 +sign(mth

nth

GNNLayer

Dist(x,xm)Dist(xkxn)

Pair-wise Cosine

Similarity

Cross

Dh

Entropy

GNNLayer

1

k-th row

nth

m,n

···

Dist(xk,xm)

Dist(xk,xn)

4.1 Consistency Loss

Our objective is to enhance graph representation consistency across GNN layers, which has significant potential to preserve the relational structure in the representation space. If we compare ϕ ( ψ i ( G )) to the graph representations obtained at the i -th layer, preserving the consistency principle is equivalent to preserving the ordering of cosine similarity among the graph representations. However, due to the non-differentiable nature of ranking operations, directly minimizing the ranking differences between consecutive layers is not feasible using gradient-based optimization techniques. Therefore, we aim to optimize pairwise ordering relations instead of the entire ranking list. In this work, our proposed loss employs a probabilistic approach inspired by [Burges et al., 2005]. The entire framework is illustrated in Figure 2.

Let H h ∈ R n × d denote the graph embedding matrix for n examples in a batch, each with a d -dimensional feature vector. We first compute the distance matrix D h for all the graphs in a batch at

h

the h -th layer. The entries D i,j of this matrix represent the distance between the representations of the i -th and j -th graphs, calculated as D h i,j = Dist ( H h x i , H h x j ) . Here, we use the cosine distance, the complement of cosine similarity in positive space, expressed as: 1 -H h x i · H h x j ∥ H h x i ∥ · ∥ ∥ ∥ H h x j ∥ ∥ ∥ . Considering the distance relationship to an arbitrary graph x k in the batch, the predicted probability ˆ P h n,m | k that x k is more similar to graph x n than to graph x m at layer h is defined as ˆ P h n,m | k ( ˆ D h k,n < ˆ D h k,m ) . This probability score, which ranges from 0 to 1, is formulated using the sigmoid function as follows:

Given the distance matrix from the previous layer D h -1 , the known probability that graph x k is more similar to graph x n than to graph x m , denoted as ˜ P h -1 n,m ( ˆ D h -1 k,n , ˆ D h -1 k,m ) , can be formulated as follows:

We can then minimize the discrepancy between the predicted and the known probability distributions to enhance representation consistency across the layers. Here,we employ the cross-entropy loss to effectively measure the divergence between these two distributions. Specifically, for a pair ( x n , x m ) centered on x k at layer h , the cross-entropy loss can be expressed as:

Then, the total loss function, which quantifies the consistency of pair-wise distance relations for graph x k at layer h , can be formulated as

The overall objective function of the proposed framework can then be formulated as the weighted sum of the original loss and the consistency loss.

Here, λ is a hyperparameter that controls the strength of the consistency constraint.

5 Experiment

In this section, we examine the effectiveness of the proposed consistency loss for the graph classification task. Specifically, we aim to address the following questions: Q1: Does the consistency loss effectively enhance the performance of various GNN backbones in the graph classification task? Q2: How does the consistency loss influence the rank correlation of graph similarities across GNN layers? Q3: How does the consistency loss influence dataset performance across varying levels of complexity, both in structural intricacy and task difficulty?

5.1 Experiment Setup

Dataset We conduct extensive experiments using the TU Dataset [Morris et al., 2020], the Open Graph Benchmark (OGB) [Hu et al., 2020] and Reddit Threads(Reddit-T) dataset [Bause and Kriege, 2022]. The TU Dataset consists of eight graph classification datasets, categorized into three main types: (1) Chem/Bioinformatics datasets, including D&D, NCI1, NCI109, and PROTEINS; (2) Social Network datasets, including IMDB-BINARY,IMDB-MULTI and COLLAB, where a constant feature value of 1 was assigned to all vertices due to the absence of vertex features; and (3) a Computer Vision dataset, COIL-RAG. The OGB datasets include ogbg-molhiv, a molecular property prediction dataset used to determine whether a molecule inhibits HIV replication. The Reddit-T dataset is

used for classifying threads from Reddit as either discussions or non-discussions, where users are represented as nodes and replies between them as links.

For the TU dataset and Reddit-T, consistent with prior work [Xinyi and Chen, 2019, Xu et al., 2019], we utilize an 8:1:1 ratio for training, validation, and testing sets. For the OGB datasets, we use the official splits provided. Training stops at the highest validation performance, and test accuracy is taken from the corresponding epoch in each fold. Final results are reported as the mean accuracy (except ogbg-molhiv) and standard deviation over 10 folds. For ogbg-molhiv, we follow the official evaluator and use ROC-AUC as the evaluation metric. Detailed information about these datasets can be found in Appendix C.

Model We use three widely adopted GNN models as baselines: namely, GCN [Kipf and Welling, 2017], GIN [Xu et al., 2019], and GraphSAGE [Hamilton et al., 2017]. We also include two recent GNN models, namely GTransformer [Shi et al., 2021] and GMT [Baek et al., 2021]. To ensure a fair comparison, we maintain the same number of layers and layer sizes for both the base models and the models with our proposed consistency loss, ensuring the sharing of the same network architecture. Detailed information about hyperparameter tuning is provided in Appendix D.

5.2 Effectiveness of Consistency Loss

To answer Q1 , we present the results for the TU, OGB, and Reddit-T datasets in Table 1. As shown in this table, GNN models with the consistency loss yield significant performance improvements over their base models on different datasets. These findings suggest that the consistency framework is a versatile and robust approach for enhancing the predictive capabilities of GNNs in real-world datasets, irrespective of the base model and dataset domain. Notably, the GIN method demonstrates the most significant improvements, achieving enhancements of up to 4.51% on the D&D dataset, 4.32% on the COLLAB dataset, and 3.70% on the IMDB-B dataset. This improvement can be linked to our empirical observation regarding the weak ability of GIN to preserve consistency across layers. In addition, our method demonstrates satisfactory improvements on datasets with numerous classes (e.g., COIL-RAG) and large-scale datasets (e.g., ogbg-molhiv and Reddit-T), indicating that our approach is both flexible and scalable for handling complex and extensive datasets.

Furthermore, we analyze the complexity and scalability of our method on the TU Dataset, with additional details provided in Appendices E.1 and E.2. Additionally, we demonstrate the method's potential to enhance performance significantly, even with a marginal increase in computational cost, as illustrated in Appendix F.

5.3 Effect of the Consistency Loss on Rank Correlation

To answer Q2 , we compare the consistency of graph representations across layers with and without the proposed consistency loss. Specifically, we use the Spearman's rank correlation coefficient, a widely accepted method for computing correlation between ranked variables, to quantitatively measure the consistency of graph similarities across layers. For a fair comparison, we construct a

distance matrix D h for all test data at each layer h , where each row D h x i ; represents the distances from graph x i to all other graphs. We then compute the rank correlation between D h x i , : and D h +1 x i ; for each graph x i .

We average the correlation values for all graphs to obtain the overall correlation for layer h . Then, we compute the mean of these values across layers, enabling a global comparison of relational consistency throughout the model and dataset. All results were averaged over 5 repeated experiments with the same training setting.

We present our results on a series of datasets from the TU Dataset in Table 2. As shown in the table, it is evident that the representation space becomes more consistent with our proposed consistency loss. For example, a significant enhancement is observed for the GIN model. Another notable point is the result for the GCN model on the NCI109 dataset and for the GMT model on the IMDB-B dataset. We find that the correlation is already fairly high even without the implementation of L consistency, resulting in mini-

mal correlation improvements with our method. This phenomenon provides a plausible explanation for why our method is not effective in these two cases.

5.4 Study on Task Complexity

To address Q3 and further evaluate the performance of our method across different scenarios, we extended our study by conducting experiments on graph datasets with increasing task and structural complexity.

Increasing Task Complexity We increase the task complexity by expanding the number of classes that the model needs to classify. To assess the effect of increased class complexity on our method's performance, we sampled subsets from the REDDIT-MULTI 5K dataset [Yanardag and Vishwanathan, 2015] with a progressively greater number of classes, which originally consists of five classes. Specifically, we randomly sampled between 2 and 4 classes to construct new datasets from the original dataset and conducted classification tasks using both GCN and GCN with L consistency on these newly constructed datasets. We report the mean test accuracy over five experiments for each subset, as presented in Table 3.

The results demonstrate that the effectiveness of our method remains robust, even as the number of classes increases. In fact, it may provide greater advantages when applied to multi-class classification tasks. This resilience likely stems from our method's focus on identifying relational structures in the intermediate representations of GNN models, rather than relying heavily on label information. This approach helps mitigate the impact of potential label noise in the original data. These findings align with the noticeable performance improvements observed in both binary and multi-class classification tasks, as shown in Table 1.

Increasing Structural Complexity We assessed the impact of structural complexity by partitioning the IMDB-B dataset into three subsets with progressively increasing graph densities. Graph density, denoted as d = 2 M N ( N -1) , where N is the number of nodes and M is the number of edges in graph

G , was used as the criterion for creating these subsets. The dataset was divided into three groups: (small) for graphs with densities below the 33rd percentile, (median) for densities between the 33rd and 67th percentiles, and (large) for graphs with densities above the 67th percentile. We applied both GCN and GCN+ L consistency models to these subsets, and the results are summarized in Table 4.

The above results show that the GCN model, enhanced with the L consistency loss function, consistently outperforms the original version across different structural complexity groups, demonstrating the robustness and effectiveness of the proposed method.

Furthermore, we evaluate the method's efficiency under various complexity scenarios, as detailed in the Appendix E.3.

6 Related Work

Graph Distance and Similarity Measuring distances or similarities between graphs is a fundamental problem in graph learning. Graph kernels, which define graph similarity, have gained significant attention. Most graph kernels use the R -Convolution framework [Haussler et al., 1999] to compare substructure similarities. A trailblazing kernel by [Kashima et al., 2003] used node and edge attributes to generate label sequences through a random walk. The WL-subtree kernel [Shervashidze et al., 2011] generates graph-level features by summing node representation contributions. Recent works align matching substructures between graphs. For instance, Kriege et al. [2016] proposed a discrete optimal assignment kernel based on vertex kernels from WL labels. Togninalli et al. [2019] extended this to include fractional assignments using the Wasserstein distance. Additionally, measuring graph distances is also a prevalent problem. Vayer et al. [2019] combined the Wasserstein and GromovWasserstein distances [Villani and Society, 2003, Mémoli, 2011]. Chen et al. [2022] proposed a polynomial-time WL distance for labeled Markov chains, treating labeled graphs as a special case.

Bridging Graph Kernels and GNNs Many studies have explored the connection between graph kernels and GNNs, attempting to integrate them into a unified framework. Certain approaches focus on leveraging GNN architecture to design novel kernels. For instance, Mairal et al. [2014] presents neural network architectures that learn graph representations within the Reproducing Kernel Hilbert Space (RKHS) of graph kernels. Similarly, Du et al. [2019] proposed a graph kernel equivalent to infinitely wide GNNs, which can be trained using gradient descent. Conversely, other studies have incorporated kernel methods directly into GNNs. For example, Nikolentzos and Vazirgiannis [2020] utilized graph kernels as convolutional filters within GNN architectures. Additionally, Lee et al. [2024] proposes a novel Kernel Convolution Network that employs the random walk kernel as the core mechanism for learning descriptive graph features. Instead of applying specific kernel patterns as mentioned in previous work, we introduce a general method for GNNs to capture consistent similarity relationships, thereby enhancing classification performance.

7 Conclusion

In this paper, we study a class of graph kernels and introduce the concept of the consistency property for graph classification tasks. We theoretically prove that this property leads to a more structure-aware representation space for classification using kernel methods. Based on this analysis, we extend this principle to enhance GNN models. We propose a novel, model-agnostic consistency learning framework for GNNs that enables them to capture relational structures in the graph representation space. Experiments show that our proposed method universally enhances the performance of backbone networks on graph classification benchmarks, providing new insights into bridging the gap between traditional kernel methods and GNN models.

References

Jinheon Baek, Minki Kang, and Sung Ju Hwang. Accurate learning of graph representations with graph multiset pooling. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net /forum?id=JHcqXGaqiGn .

Franka Bause and Nils Morten Kriege. Gradual weisfeiler-leman: Slow and steady wins the race. In Bastian Rieck and Razvan Pascanu, editors, Learning on Graphs Conference, LoG 2022, 9-12 December 2022, Virtual Event , volume 198 of Proceedings of Machine Learning Research , page 20. PMLR, 2022. URL https://proceedings.mlr.press/v198/bause22a.html .

Karsten M. Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Proceedings of the 5th IEEE International Conference on Data Mining (ICDM 2005), 27-30 November 2005, Houston, Texas, USA , pages 74-81. IEEE Computer Society, 2005. doi: 10.1109/ICDM.2005.132. URL https://doi.org/10.1109/ICDM.2005.132 .

Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning to rank using gradient descent. In Luc De Raedt and Stefan Wrobel, editors, Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005 , volume 119 of ACM International Conference Proceeding Series , pages 89-96. ACM, 2005. doi: 10.1145/1102351.1102363. URL https://doi.org/10.1145/1102351.1102363 .

Samantha Chen, Sunhyuk Lim, Facundo Mémoli, Zhengchao Wan, and Yusu Wang. Weisfeilerlehman meets gromov-wasserstein. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 3371-3416. PMLR, 2022. URL https://proceedings.mlr.pres s/v162/chen22o.html .

Simon S. Du, Kangcheng Hou, Ruslan Salakhutdinov, Barnabás Póczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 5724-5734, 2019. URL https://proceedings.neurips.cc/paper/201 9/hash/663fd3c5144fd10bd5ca6611a9a5b92d-Abstract.html .

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 1263-1272. PMLR, 2017. URL http://proceedings.mlr.press/v70/gilmer17a.html .

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1024-1034, 2017. URL https://proceedings.neurips. cc/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html .

David Haussler et al. Convolution kernels on discrete structures. Technical report, Citeseer, 1999.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/fb60d411a5c5b72b2e7d3527c fc84fd0-Abstract.html .

Zheng Huang, Qihui Yang, Dawei Zhou, and Yujun Yan. Enhancing size generalization in graph neural networks through disentangled representation learning. In Forty-first International Conference on Machine Learning , 2024.

Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. Marginalized kernels between labeled graphs. In Tom Fawcett and Nina Mishra, editors, Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA , pages 321-328. AAAI Press, 2003. URL http://www.aaai.org/Library/ICML/2003/icml03-044.php .

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openrevi ew.net/forum?id=SJU4ayYgl .

Nils M. Kriege, Pierre-Louis Giscard, and Richard C. Wilson. On valid optimal assignment kernels and applications to graph classification. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 1615-1623, 2016. URL https://proceedings.neurips.cc/paper /2016/hash/0efe32849d230d7f53049ddc4a4b0c60-Abstract.html .

Nils M. Kriege, Fredrik D. Johansson, and Christopher Morris. A survey on graph kernels. Appl. Netw. Sci. , 5(1):6, 2020. doi: 10.1007/S41109-019-0195-3. URL https://doi.org/10.1007/ s41109-019-0195-3 .

Meng-Chieh Lee, Lingxiao Zhao, and Leman Akoglu. Descriptive kernel convolution network with improved random walk kernel. In Tat-Seng Chua, Chong-Wah Ngo, Ravi Kumar, Hady W. Lauw, and Roy Ka-Wei Lee, editors, Proceedings of the ACM on Web Conference 2024, WWW 2024, Singapore, May 13-17, 2024 , pages 457-468. ACM, 2024. doi: 10.1145/3589334.3645405. URL https://doi.org/10.1145/3589334.3645405 .

Gaotang Li, Marlena Duda, Xiang Zhang, Danai Koutra, and Yujun Yan. Interpretable sparsification of brain graphs: Better practices and effective designs for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1223-1234, 2023a.

Gaotang Li, Danai Koutra, and Yujun Yan. Size generalization of graph neural networks on biological data: Insights and practices from the spectral perspective. arXiv preprint arXiv:2305.15611 , 2023b.

Lu Lin, Jinghui Chen, and Hongning Wang. Spectral augmentation for self-supervised learning on graphs. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id= DjzBCrMBJ_p .

Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pretraining molecular graph representation with 3d geometry. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=xQUe1pOKPam .

Julien Mairal, Piotr Koniusz, Zaïd Harchaoui, and Cordelia Schmid. Convolutional kernel networks. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada , pages 2627-2635, 2014. URL https://proceedings.neurips.cc/paper/2014/hash/81ca026 2c82e712e50c580c032d99b60-Abstract.html .

Facundo Mémoli. Gromov-wasserstein distances and the metric approach to object matching. Found. Comput. Math. , 11(4):417-487, 2011. doi: 10.1007/S10208-011-9093-5. URL https: //doi.org/10.1007/s10208-011-9093-5 .

Mehryar Mohri. Foundations of machine learning, 2018.

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. CoRR , abs/2007.08663, 2020. URL https://arxiv.org/abs/2007.08663 .

Giannis Nikolentzos and Michalis Vazirgiannis. Random walk graph neural networks. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ba95d78a7c942571185308775 a97a3a0-Abstract.html .

Giannis Nikolentzos, Giannis Siglidis, and Michalis Vazirgiannis. Graph kernels: A survey. J. Artif. Intell. Res. , 72:943-1027, 2021. doi: 10.1613/JAIR.1.13225. URL https://doi.org/10.161 3/jair.1.13225 .

Nino Shervashidze, S. V. N. Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten M. Borgwardt. Efficient graphlet kernels for large graph comparison. In David A. Van Dyk and Max Welling, editors, Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, AISTATS 2009, Clearwater Beach, Florida, USA, April 16-18, 2009 , volume 5 of JMLR Proceedings , pages 488-495. JMLR.org, 2009. URL http://proceedings.mlr.press/v5/s hervashidze09a.html .

Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. J. Mach. Learn. Res. , 12:2539-2561, 2011. doi: 10.5555/1953048.2078187. URL https://dl.acm.org/doi/10.5555/1953048.2078187 .

Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 1548-1554. ijcai.org, 2021. doi: 10.24963/IJCAI.2021/214. URL https://doi.org/10.24963/ijcai.2021/214 .

Giannis Siglidis, Giannis Nikolentzos, Stratis Limnios, Christos Giatsidis, Konstantinos Skianis, and Michalis Vazirgiannis. Grakel: A graph kernel library in python. J. Mach. Learn. Res. , 21: 54:1-54:5, 2020. URL https://www.jmlr.org/papers/v21/18-370.html .

Matteo Togninalli, M. Elisabetta Ghisu, Felipe Llinares-López, Bastian Rieck, and Karsten M. Borgwardt. Wasserstein weisfeiler-lehman graph kernels. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 6436-6446, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/73fed7fd472e502d8 908794430511f4d-Abstract.html .

Titouan Vayer, Nicolas Courty, Romain Tavenard, Laetitia Chapel, and Rémi Flamary. Optimal transport for structured data with application on graphs. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6275-6284. PMLR, 2019. URL http://proceedings.mlr.press/ v97/titouan19a.html .

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=rJXMpikCZ .

Villani and American Mathematical Society. Topics in Optimal Transportation . Graduate studies in mathematics. American Mathematical Society, 2003. ISBN 9781470418045. URL https://books.google.com/books?id=MyPjjgEACAAJ .

Haohui Wang, Yuzhen Mao, Yujun Yan, Yaoqing Yang, Jianhui Sun, Kevin Choi, Balaji Veeramani, Alison Hu, Edward Bowen, Tyler Cody, et al. Evolunet: Advancing dynamic non-iid transfer learning on graphs. In Forty-first International Conference on Machine Learning , 2024.

Boris Weisfeiler and AA Lehman. A reduction of a graph to a canonical form and an algebra arising during this reduction. Nauchno-Technicheskaya Informatsia , 2(9):12-16, 1968.

Zhang Xinyi and Lihui Chen. Capsule graph neural network. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Byl8BnRcYm .

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id= ryGs6iA5Km .

Minkai Xu, Alexander S. Powers, Ron O. Dror, Stefano Ermon, and Jure Leskovec. Geometric latent diffusion models for 3d molecule generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 38592-38610. PMLR, 2023. URL https: //proceedings.mlr.press/v202/xu23n.html .

Yujun Yan, Jiong Zhu, Marlena Duda, Eric Solarz, Chandra Sripada, and Danai Koutra. Groupinn: Grouping-based interpretable neural network for classification of limited, noisy brain data. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 772-782, 2019.

Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernels. In Longbing Cao, Chengqi Zhang, Thorsten Joachims, Geoffrey I. Webb, Dragos D. Margineantu, and Graham Williams, editors, Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015 , pages 1365-1374. ACM, 2015. doi: 10.1145/2783258.2783417. URL https://doi.org/10.1145/2783258.2783417 .

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Yike Guo and Faisal Farooq, editors, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018 , pages 974-983. ACM, 2018. doi: 10.1145/3219819.3219890. URL https://doi.org/10.1145/3219819.3219890 .

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/ hash/3fe230348e9a12c13120749e3f9fa4cd-Abstract.html .

Zhen Zhang, Mianzhi Wang, Yijian Xiang, Yan Huang, and Arye Nehorai. Retgk: Graph kernels based on return probabilities of random walks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 3968-3978, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/7f16109f1619fd7a733da f5a84c708c1-Abstract.html .

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph Contrastive Learning with Adaptive Augmentation. In Proceedings of The Web Conference 2021 , WWW'21, pages 2069-2080, New York, NY, USA, April 2021. Association for Computing Machinery. ISBN 9781450370233. doi: 10.1145/3442381.3449802. URL https://doi.org/10.1145/3442381. 3449802 .

A Proof of Theorem 3.4

A.1 Learning Correct Similarity Order with Iterative Kernels

In this section, we first prove that the iterative kernel with bounded feature mapping is capable of learning the pairwise similarity order given sufficient training data, as shown by the PAC-learning framework. We consider a dataset consisting of instances x ∈ χ , where χ is the input space. The dataset is denoted as χ = { x 1 , x 2 , . . . , x n } . We employ a kernel function K : χ × χ → R with an associated feature mapping ϕ : χ →H K such that K ( x, y ) = ⟨ ϕ ( x ) , ϕ ( y ) ⟩ H K . Here, H K is the Reproducing Kernel Hilbert Space (RKHS) associated with K , and ⟨· , ·⟩ H K denotes the inner product in H K . We assume that the feature mappings are bounded, i.e., ∥ ϕ ( x ) ∥ H K ≤ R, ∀ x ∈ χ for some R > 0 .

̸

Using this setup, we focus on an arbitrary fixed data point x i and compute its similarity with other data points x j ( j = i ) using the kernel function:

Thus, we define the function f as f ( x j ) = ⟨ w,ϕ ( x j ) ⟩ H K where w = ϕ ( x i ) . Since w = ϕ ( x i ) and ∥ ϕ ( x i ) ∥ H K ≤ R , it follows that ∥ w ∥ H K ≤ R . So the function f representing the similarity between x i and any other point x j can be formulated as:

̸

Since we are dealing with a ranking problem, we use the hinge loss function for pairs ( x j , x k ) ( j, k = i ) :

where s ∗ jk = sign ( f ∗ ( x j ) -f ∗ ( x k )) , and f ∗ represents the ground-truth ranking function.

Thus, the expected loss over all pairs is given by:

and the empirical loss over the sample S is given by:

̸

since there are ( n -1)( n -2) 2 pairs among the n -1 data points excluding x i . To assess how well the learned function f generalizes to test data, we derive a generalization bound using Rademacher complexity. We define the function class for pairs by:

which represents the similarity difference between x j and x k given the reference graph x i .

Given that f ( x ) = ⟨ w,ϕ ( x ) ⟩ with ∥ w ∥ ≤ R and ∥ ϕ ( x ) ∥ ≤ R , we can bound f ( x j , x k ) by:

since | f ( x ) | = |⟨ w,ϕ ( x ) ⟩| ≤ ∥ w ∥ · ∥ ϕ ( x ) ∥ ≤ R · R = R 2 .

Given f ( x j , x k ) takes values in the range [ -2 R 2 , 2 R 2 ] . Therefore, following the PAC-learning framework in [Mohri, 2018], we derive the following generalization bound. For any δ > 0 , with probability at least 1 -δ over a sample S of size m drawn, the following holds for all f ∈ F pair :

ˆ R m ( F pair ) is the empirical Rademacher complexity of F pair .

where:

M = 1 + 2 R 2 is an upper bound on the hinge loss function ℓ hinge .

m = ( n -1)( n -2) 2 is the number of pairs.

δ is the confidence level.

It is observed that as m increases, the term √ ln(1 /δ ) 2 m decreases at a rate proportional to √ 1 /m . Next, we proceed by proving that the Rademacher complexity is bounded and decreases as m increases.

Theorem A.1 Assuming ∥ ϕ ( x ) ∥ H K ≤ R for all x ∈ χ , the empirical Rademacher complexity of the function class F pair can be bounded as:

Proof 1 Based on the definition of Rademacher complexity, for function class F pair with a sample size m , the complexity ˆ R S ( F pair ) is defined as:

where σ = ( σ 1 , . . . , σ m ) are independent Rademacher variables (i.e., each σ k takes the value +1 or -1 with equal probability), and { ( x j i , x k i ) } m i =1 are pairs sampled from the data.

We can express ˆ R S ( F pair ) as:

Applying the Cauchy-Schwarz inequality:

we have:

Using Jensen's inequality in the form E [ ∥ X ∥ ] ≤ ( E [ ∥ X ∥ 2 ]) 1 / 2 , we obtain:

Since σ i are independent Rademacher variables, we have E σ [ σ 2 i ] = 1 . Therefore:

Given that ∥ ϕ ( y ) ∥ H K ≤ R , we have:

Therefore,

Thus

Therefore, the empirical Rademacher complexity decreases with m .

Thus, consider Equation 8:

The term √ ln(1 /δ ) 2 m decreases proportionally to 1 / √ m .

Empirical Rademacher complexity ˆ R S ( F pair ) ≤ 2 R 2 √ m also decreases proportionally to 1 / √ m .

Therefore, the generalization gap between L ( f ) and ˆ L ( f ) decreases as m increases, demonstrating that the kernel-based ranking function is PAC-learnable.

A.2 Linking Two Key Properties to Generalizability in Binary Classification

Building on the setup and conclusions from the previous section, we establish that the kernel can learn the correct ranking with sufficient training data. Here, we investigate how, once the correct ranking is achieved, an iterative kernel with monotonic decrease and order consistency properties ensures a decreasing error rate for test set graphs as iterations progress in binary classification tasks.

To formalize this, we define a probability space over the whole graph space where addition and scalar multiplication are defined. Let Ω be the set of all possible data points (graphs). Each data point g i ∈ Ω represents a possible outcome, and the event of observing g i is the singleton set { g j } ⊆ Ω . By considering all subsets of Ω , we construct an event space F . Assuming the existence of an underlying probability distribution over these outcomes, we obtain the probability space (Ω , F , P ) . Using this framework, let µ x and µ y represent the unknown mean graph in class 1 and class 2, respectively. Let K gt ( · , · ) ∈ [0 , 1] represent the ground-truth kernel function for pairs of graphs.

Without loss of generality, we analyze the error rate for graphs belonging to class 1; the error rate for class 2 can be derived in a similar manner. We define two random variables associated with the ground-truth kernel: s x = K gt ( µ x , x ) and s y = K gt ( µ y , x ) , where x is a randomly sampled instance from class 1. The terms s x ( x ) and s y ( x ) denote specific realizations of s x and s y , respectively, for a given realization of the graph x belonging to class 1. Since s x = 0 implies that there exists graph from class 1 that is completely different from µ x , which is counterintuitive, we assume that the probability density function p ( u ) , u ∈ [0 , 1] , of s x has the following asymptotic property on the right neighborhood of u = 0 :

where constant C satisfies that 0 ≤ C < + ∞ . Let ¯ s ( i ) x ( x ) = K ( i ) (¯ µ ( i ) x , x ) and ¯ s ( i ) y ( x ) = K ( i ) (¯ µ ( i ) y , x ) represent the estimated realizations of s x and s y at iteration i , respectively, for a given graph realization x . ¯ µ ( i ) x and ¯ µ ( i ) y are estimates for µ x and µ y , respectively. We assume that ¯ µ ( i ) x and ¯ µ ( i ) y are obtained through operations on the training graphs, where these operations are closed in the graph space. Due to the capability of the iterative kernels, we further assume that as the iteration i → ∞ , the sequences { ¯ µ ( i ) x ( · , · ) } + ∞ i =1 , { ¯ µ ( i ) y ( · , · ) } + ∞ i =1 and { K ( i ) ( · , · ) } + ∞ i =1 converge to µ x , µ y and K gt , respectively. We assume that the convergence rate of { K ( i ) ( · , · ) } + ∞ i =1 is much slower than the convergence rate of { ¯ µ ( i ) x ( · , · ) } + ∞ i =1 and { ¯ µ ( i ) y ( · , · ) } + ∞ i =1 . Furthermore, every function in the series of function { K ( i ) ( · , · ) } + ∞ i =1 is continuous.

Denote the training data for class 1 and class 2 as { x 1 , x 2 , . . . , x n } ⊂ Ω and { y 1 , y 2 , . . . , y m } ⊂ Ω , respectively. Since the learned kernels K ( i ) preserve the similarity order, we can assume, without loss of generality, the following empirical ordering of the training graphs for the kernel at the i -th iteration:

□

In this setting, for an arbitrary test graph g sampled from the graph space, the decision boundary of the kernel learned during the i -th iteration is defined as follows:

We assume that ¯ s ( i ) x ( g ) and ¯ s ( i ) y ( g ) , as well as s x ( g ) and s y ( g ) , are not equal for any of the training data points. Next, we analyze the misclassification rate for graphs in class 1. Let the integer k satisfies: ¯ s ( i ) x ( x k ) < ¯ s ( i ) y ( x k ) , while ¯ s ( i ) x ( x k +1 ) > ¯ s ( i ) y ( x k +1 ) or a given iteration i , which means x k is the misclassified training graph in class 1 closest to the decision boundary. If the training error is 0, we take x 0 = y 1 . We note that the integer k is independent of the iteration i due to the consistency principle. Next, we demonstrate this using the following lemma.

Lemma A.2 For any sufficiently large iteration number i , there exists a fixed index k such that ¯ s ( i ) x ( x k ) < ¯ s ( i ) y ( x k ) , while ¯ s ( i ) x ( x k +1 ) > ¯ s ( i ) y ( x k +1 ) .

Proof 2 Since the series { ¯ µ ( i ) x } + ∞ i =1 converges to µ x , it satisfies Cauchy's convergence criterion. Specifically, ∀ δ > 0 , ∃ N 1 > 0 , such that when i > N 1 ,

Since we have assumed that K ( i ) ( · , · ) are continuous: ∀ ϵ > 0 , ∃ δ > 0 , ∃ N 2 > 0 , such that for any x in class 1, ∀ i > N 2 and | ¯ µ ( i +1) x -¯ µ ( i ) x | < δ , we have:

Based on Equation 23, we have:

Since the consistency principle holds, K ( i +1) (¯ µ ( i ) x , x k ) -K ( i +1) (¯ µ ( i ) y , x k ) and ¯ s ( i ) x ( x k ) -¯ s ( i ) y ( x k ) have the same sign. If ¯ s ( i ) x ( x k ) -¯ s ( i ) y ( x k ) < 0 , then K ( i +1) (¯ µ ( i ) x , x k ) -K ( i +1) (¯ µ ( i ) y , x k ) < 0 . For | K ( i +1) (¯ µ ( i ) ,x k ) -K ( i +1) (¯ µ ( i ) ,x k ) | .

a sufficiently large N , we select an ϵ > 0 such that ϵ < min { i>N } x y 2 Given our assumption that no training data point x k lies precisely on the decision boundary, such an ϵ is guaranteed to exist. Consequently, Equation 24 simplifies to:

Thus, ¯ s ( i +1) x ( x k ) -¯ s ( i +1) y ( x k ) have the same sign as ¯ s ( i ) x ( x k ) -¯ s ( i ) y ( x k ) . If ¯ s ( i ) x ( x k ) -¯ s ( i ) y ( x k ) > 0 , the same conclusion can be derived in a similar manner. Since the number of training samples is finite, we can find a sufficiently large iteration such that Equation 25 holds for all x k after that iteration.

Thus, some point between ¯ s ( i ) x ( x k ) and ¯ s ( i ) x ( x k +1 ) is the decision boundary learned at the i -th iteration.

Based on our assumption that the convergence rate of { K ( i ) ( · , · ) } + ∞ i =1 is much slower than the convergence rate of { ¯ µ ( i ) x ( · , · ) } + ∞ i =1 and { ¯ µ ( i ) y ( · , · ) } + ∞ i =1 , ¯ s ( i ) x ( x k ) is asymptotically monotonically decreasing.

Thus,

Thus, ∀ x ∈ class 1, the probability of misclassifying x is given by: P { ¯ s ( i ) x ( x k ) < K gt ( µ x , x k ) } . Specifically, we have:

In the following theorem, we show that the probability of misclassifying any graph from class 1 can be bounded using Markov's inequality.

Theorem A.3 If the iterative kernel decreases monotonically and preserves the similarity order, the probability of misclassifying a graph x ∈ χ from class 1 has an upper bound that decreases monotonically as the number of iterations increases.

Proof 3 To upper bound the error rate, we begin by applying the transformation 1 s x -1 to the original random variable s x . The expectation of this transformed variable is:

Since u = 0 is singular point, we reformulate the improper integral as:

Here, since lim u → 0 + p ( u ) u = C , the limit exists and thus the given integral converges, ensuring the existence of the expectation. With this result, we can apply Markov's inequality to the transformed random variable 1 s x -1 . For any nonnegative a , we have:

Thus

Let a = 1 ¯ s ( i ) x ( x k ) -1 . Referring to Equation 28, we have:

Since ¯ s ( i ) x ( x k ) decreases monotonically and E ( 1 s x -1 ) is a constant, it follows that P { s x < K gt ( µ x , x k ) } has a monotonically decreasing upper bound.

B Theoretical Verification with WL-based Kernels

B.1 Proof of Theorem 3.5

Following Equation 2.4, the normalized WL-subtree kernel after h iterations can be expressed as:

Next, we show by counterexample that the WL-subtree kernel may not preserve relational structure.

For example, consider two graphs, G and G ′ , in the h -th iteration. The color (label) set of the nodes in these graphs is denoted as { C a , C b , C c , . . . } . Graph G contains 200 nodes labeled C a and 4 nodes labeled C b , whereas graph G ′ contains 4 nodes labeled C a and 200 nodes labeled C b . The similarity is computed using the vectors ϕ ( ψ i ( G ) ) = [200 , 4] and ϕ ( ψ i ( G ′ ) ) = [4 , 200] . Consequently, the resulting similarity ˜ K ( h ) wl _ subtree is 0.0400.

In the next iteration, for graph G , assume that half of the nodes currently labeled C a are relabeled to C c , while the other half are relabeled to C d . Additionally, all nodes labeled C b are relabeled to

C e . As a result, the updated histogram vector for G becomes ϕ ( ψ i +1 ( G ) ) = [100 , 100 , 4] . Similarly, in graph G ′ , half of the nodes labeled C a are relabeled to C c and the remaining half to C d , with all nodes labeled C b relabeled to C e , resulting in a histogram vector of ϕ ( ψ i +1 ( G ′ ) ) = [2 , 2 , 200] . Here, the normalized similarity ˜ K ( h ) wl _ subtree is 0.0404, which exceeds ˜ K ( h ) wl _ subtree .

This situation violates the principle of monotonic decrease, indicating that the WL-subtree kernel does not exhibit monotonicity-a phenomenon frequently observed in real-world datasets. □

B.2 Proof of Theorem 3.6

B.2.1 Monotonic Decrease in WLOA Kernel

The WLOA kernel after h iterations can be computed as follows:

where ω ( i ) is a monotonically increasing function. The normalized kernel is given by:

At any iteration i , the expression histmin { ϕ ( ψ i ( G )) , ϕ ( ψ i ( G )) } · ω ( i ) equals to ω ( i ) |V| . Consequently, for the h -th iteration, this implies:

Thus, the normalized kernel value for the graph pair ( G , G ′ ) can be expressed as:

Given this, the difference between the kernel value at the h -th iteration, ˜ K ( h ) WLOA , and the ( h +1) -th iteration, ˜ K ( h +1) WLOA , can be expressed as follows:

Considering the hierarchical structure of the refinement processes, one can deduce the following inequality:

histmin { ϕ ( ψ i ( G )) , ϕ ( ψ i ( G ′ )) } ≤ histmin { ϕ ( ψ i -1 ( G )) , ϕ ( ψ i -1 ( G ′ )) } ≤ · · · histmin { ϕ ( ψ 1 ( G )) , ϕ ( ψ 1 ( G ′ )) } Thus, for the kernel at the ( h +1) -th iteration, it follows that:

Given this inequality, we get

Therefore, we get D ≥ 0 . The value of Equation 33 is non-negative, indicating that WLOA is monotonically decreasing.

B.2.2 Order Consistency in WLOA Kernel

Assume two graph pairs satisfy the following relation:

The normalized WLOA kernel for a graph pair ( G , G ′ ) at iteration h +1 can be expressed as:

K ( h ) WLOA is monotonically increasing with the iteration number h , as it follows from: K ( h +1) WLOA = K ( h ) WLOA + histmin { ϕ ( ψ h +1 ( G )) , ϕ ( ψ h +1 ( G ′ )) } · ω ( h +1) . Thus, we have:

Given the monotonic decrease property of ˜ K ( h ) WLOA ( G , G ′′ ) ,we have:

Thus, we can conclude:

When ω ( i ) = 1 , lim h →∞ ω ( h +1) ∑ h +1 i =1 ω ( i ) = lim h →∞ h +1 ( h +2)( h +1) / 2 = lim h →∞ 2 h +2 = 0 . Therefore, ˜ K ( h +1) WLOA ( G , G ′ ) ≥ ˜ K ( h +1) WLOA ( G , G ′′ ) when h →∞ . □

C Dataset

In this section, we provide the statistics of the datasets used, as shown in Table 5.

D Detailed Set-Up

TU Dataset We restrict the hyperparameters and ensure the same architecture is used for both the base and enhanced models on the same dataset for a fair comparison. Specifically, we fixed the hidden size to 32, the number of layers to 3, and used a global mean pooling layer to generate the graph representations. The Adam optimizer was used for optimization. During the training process, we tuned both the base and enhanced models with the same search ranges: batch size {64, 128}, dropout rate {0, 0.3} and learning rate {0.0001, 0.001, 0.01}. The only additional parameter for the enhanced model is the regularization term, which ranges from {0.1, 0.5, 1, 10}.

OGB Considering the complexity of the OGB dataset, we slightly expand the hyperparameter search range. Initially, we train the model with consistency loss, exploring hidden sizes {32, 64, 128, 256}, batch sizes {64, 128, 256}, and the number of layers {3, 4}, while keeping other parameters consistent with our experiments on the TU dataset. Subsequently, we fix the hidden size and the number of layers to ensure an identical network structure. We then repeat the parameter search process and employ the optimal settings for testing on the base model. All experiments on the OGB dataset are repeated 5 times to calculate the mean and standard deviation.

Reddit-T Given that the Reddit-T and OGB datasets are of comparable size, we employ similar experimental settings for training models on these datasets. The architecture is kept fixed, and we search for optimal hyperparameters for both the base model and the models incorporating consistency loss. Each model is trained and evaluated over 5 independent runs, with the mean and standard deviation of the results recorded for performance comparison.

E Complexity& Scalability

E.1 Time Complexity

We present the time complexity analysis for our proposed consistency loss. The loss computation involves the computation of pairwise similarities of graphs in a batch, resulting in a computation complexity of O ( batchsize · batchsize -1 2 ) = O ( batchsize 2 ) . Given that there are datasetsize batchsize batches in each training epoch and that the similarities are computed between consecutive layers, the total

complexity is:

This analysis shows that the time required to compute consistency loss scales linearly with dataset size, batch size, and the number of layers. It is important to note that the training time for baseline models also scales linearly with dataset size.

Since batch size and the number of layers are generally small compared to dataset size, our experiments primarily focus on how dataset size affects training time. We evaluate the training time of several models-GCN, GIN, GraphSAGE, GTransformer, and GMT-each enhanced with our consistency loss. This evaluation is conducted on different subsets of the ogbg-molhiv dataset, with subset sizes adjusted by varying the sampling rates. The training times, measured in seconds, are presented in Figure 3 . As shown, our findings confirm that training time increases linearly with dataset size, indicating that our method maintains training efficiency comparable to baselines without adding significant time burdens.

5000

10000

15000

20000

25000

30000

35000

40000

Dataset Size

2

4

6

8

Training Time(s/epoch)

GTransformer

GMT

GCN

GIN

GraphSAGE

Furthermore, we empirically measure the training time for both the baseline models and our proposed methods. Each model comprises three layers and is trained on the ogbg-molhiv dataset (40,000+ graphs) for 100 epochs. We calculate the average training time per epoch in seconds and present the results in Table 6, showing that while the inclusion of the consistency loss slightly increases the training time, the impact is minimal.

E.2 Space Complexity

Next, we present the space complexity analysis for our consistency loss. At each iteration, the loss function requires storing two pairwise similarity matrices corresponding to two consecutive layers, which is given by:

Since we use stochastic gradient descent, similarity matrices are not retained for the next iteration. The consistency loss requires significantly less space than node embeddings, making the additional space requirement minimal. Table 7 shows the peak memory usage in megabytes (MB) for different models when training on the ogbg-molhiv dataset, illustrating that the space costs are negligible.

E.3 Efficiency on Different Task and Structural Complexities

Task Complexity We measured the runtime of the models on different subsets to evaluate how task complexity, in terms of the number of classes, influences the efficiency of the proposed method. The results are presented in Table 8. As demonstrated, the additional computational time remains minimal even with an increasing number of classes, suggesting that the method scales effectively with growing class complexity.

Structure Complexity We also conducted experiments to assess the training costs on datasets with varying structural complexities when introducing the L consistency . The results, summarized below in Table 9, show that the additional training cost remains minimal across datasets with different structures. This demonstrates the broad applicability of the proposed method, regardless of structural complexity.

F Efficient Consistent Learning

To further minimize the overhead of our proposed consistency loss, we examined a scenario where the consistency loss, denoted as L FL , was only applied to the first and last layers.

Building upon the experimental setup described in Section 5, we conducted experiments using various backbone models. The results are summarized in Table 10. The penultimate column of this table highlights the performance gains achieved by applying the consistency loss across all layers, while the final column demonstrates the improvements observed when the consistency loss is only applied to the first and last layers.

Notably, applying the consistency loss only to the first and last layers achieves performance comparable to that of applying it across all layers, with both configurations yielding substantial improvements over the original model. This finding suggests that our proposed approach can be accelerated with minimal additional computational cost while still enhancing performance, thereby validating the effectiveness of the consistent learning principle.

G Similarity/Difference with Contrastive learning

In this section, we discuss the similarities and differences between our method and graph contrastive learning. Graph Contrastive Learning (GCL) is a self-supervised technique for graph data that emphasizes instance discrimination [Lin et al., 2023, Zhu et al., 2021]. A typical GCL framework generates multiple graph views via augmentations and contrasts positive samples (similar instances) with negative samples (dissimilar instances). This approach facilitates effective representation learning by capturing relationships between views, ensuring positive pairs remain close in the embedding space while distinctly separating negative pairs.

While both GCL and our method leverage graph similarity, our approach focuses on maintaining consistency across layers, rather than solely capturing similarities as in contrastive learning. To demonstrate this, we integrated the GraphCL technique [You et al., 2020] into a GCN model (GCN+CL) and assessed its performance and layer consistency across various datasets. The results, detailed in Tables 11 and 12, use classification accuracy and Spearman rank correlation to measure performance and consistency, respectively.

As demonstrated by the results, our method consistently outperforms GCN+CL in both graph classification performance and in enhancing similarity consistency across layers. This underscores the significant differences between our approach and regular GCL methods.

H Boarder Impact

This paper aims to advance the field of graph learning by proposing a model-agnostic consistency learning framework. Our framework can be plugged into and improve current methods for graph classification tasks. This has potential benefits in sectors such as chemistry, bioinformatics, and social analysis, where graph classification is widely used. Furthermore, we do not foresee direct negative societal or ethical consequences arising from our work.

I Limitation

One limitation of our work is that the method involves additional computational costs, especially during large batch training processes. To extend our framework, sampling methodologies on data or layers can be applied during the consistency-preserving training process. By selectively sampling data points or specific layers, we can reduce the computational burden while still maintaining the effectiveness of the cross-layer consistency loss, making the framework more scalable and applicable to larger datasets.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The experiments and results to support the claims in the introduction and abstract are all in the main paper and the appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the main paper Sec. I Limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: This paper include theoretical result and proof involved in main paper and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The training schedule and hyper-parameters are listed in Appendix Sec. D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data are public, and code is provided in the anonymous github.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have listed all the settings in the appenix for paramtere settings and training schedules. The data splits are decribed under each case study section in the main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All of our results are listed with error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources of experiments are reported in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms, in every respect, to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impact is discussed in the appendix H.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:[NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: They are properly cited and credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: the new model introduced in the paper is well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work aims to bridge the gap between neural network methods and kernel methods by enabling GNNs to consistently capture relational structures in their learned representations. The authors propose a loss function that enforces the similarity of graph representations to remain consistent across different layers. Extensive experiments demonstrate that the proposed consistency loss can significantly enhance the graph classification performance of various GNN backbones on different datasets.

STRENGTHS:
This work explores the connection between kernels and GNNs, and proposes a consistency criterion. It has a strong theoretical foundation.

WEAKNESSES:
1.Does the consistency loss effectively enhance the performance of various GNN backbones in the graph clustering task?
2.This algorithm does not seem suitable for application to large-scale datasets.
3. There are several typos and grammatical/spelling errors that should be corrected.

QUESTIONS:
Please see ‘Weaknesses’.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

comments: SUMMARY:
The authors propose to improve the quality of graph embeddings in GNNs by encouraging a n motion of consistency in the representation obtained at the various layers.

STRENGTHS:
The work presented here offers interesting contributions: 1) a new perspective on understanding the graph classification performance of GNNs by examining the similarity relationships captured across different layers; 2) a novel consistency principle in both kernel and GNNs with theoretical proofs explaining how this principle enhances performance; 3) empirical evaluation across diverse GNN model types.

WEAKNESSES:
The paper could be improved by:
- devising an artificial experiments with graph datasets of increasing structural complexity and with an increasing label alphabet to show a link between the complexity of the task and the amount of improvement offered by the proposed technique

QUESTIONS:
Suggestions could include:
- reporting the run time information to gauge the tradeoff between the additional complexity and the performance improvement 
- reporting a critical difference diagram ( https://scikit-posthocs.readthedocs.io/en/latest/generated/scikit_posthocs.critical_difference_diagram.html ) to summarise the results of Table 1 and 2.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
In this paper, the authors introduce the shortcomings of graph neural networks (GNNs) in capturing consistency and similarity relationships between graphs, and proposes a new loss function, aiming to enhance graph representation at different levels. Through theoretical analysis and experimental verification, the author proved that consistency loss can significantly improve the performance of GNN in graph classification tasks.

STRENGTHS:
1. The structure of the paper is clear and easy to follow.

2. The proposed method seems reasonable and sound.

3. The method is effective in comparison to base models.

WEAKNESSES:
1. I have some concerns about the efficiency of the method. The authors give the limitation about additional computational costs, which is commendable. I suggest that the authors provide the complexity analysis, specific running times and memory usage to enable further evaluation of the paper.

2. The authors mention "across a wide range of base models and datasets" in the Introduction section, but the datasets used in experiments only have two classes. To further verify the effectiveness of the method on a wide range of datasets, I suggest that the author conduct experiments on datasets  with more classes.

QUESTIONS:
1. If only the consistency of the first and last layers of the graph representation is enhanced to replace the consistency constraints of all layers, how much will the experimental performance drop? Will there be an improvement compared to the base model?

2. Contrastive learning can also capture similarity between samples. Can the author give the differences and similarities between contrastive learning and the proposed method in the paper?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12