Geometry Awakening: Cross-Geometry Learning Exhibits Superiority over Individual Structures

Authors: yadong  sun
xiaofeng  cao
yu  wang
wei  ye
jingcai  guo
qing  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:21:56

Abstract: Recent research has underscored the efficacy of Graph Neural Networks (GNNs) in modeling diverse geometric structures within graph data. However, real-world graphs typically exhibit geometrically heterogeneous characteristics, rendering the confinement to a single geometric paradigm insufficient for capturing their intricate structural complexities. To address this limitation, we examine the performance of GNNs across various geometries through the lens of knowledge distillation (KD) and introduce a novel cross-geometric framework. This framework encodes graphs by integrating both Euclidean and hyperbolic geometries in a space-mixing fashion. Our approach employs multiple teacher models, each generating hint embeddings that encapsulate distinct geometric properties. We then implement a structure-wise knowledge transfer module that optimally leverages these embeddings within their respective geometric contexts, thereby enhancing the training efficacy of the student model. Additionally, our framework incorporates a geometric optimization network designed to bridge the distributional disparities among these embeddings. Experimental results demonstrate that our model-agnostic framework more effectively captures topological graph knowledge, resulting in superior performance of the student models when compared to traditional KD methodologies.

Main paper: 
 introduction: 
 Graph Neural Networks (GNNs) have emerged as indispensable tools for analyzing relational data in diverse domains, such as natural language processing [1, 2, 3] , computer vision [4, 5] , recommendation systems [6, 7] . Their conventional approach of operating within Euclidean space encounters limitations when confronted with datasets embodying non-Euclidean characteristics, such as powerlaw distribution and hierarchical structures, prevalent in real-world applications [8] . Recognizing this challenge, our community ventures into the realm of non-Euclidean Graph Neural Networks, seeking to harness alternative geometries, notably hyperbolic space, for more adeptly capturing the intricate topological features inherent in many real-world networks [9] . By synthesizing recent advancements and empirical findings, we endeavor to elucidate the potential of non-Euclidean GNNs in effectively modeling complex relational data structures, thereby paving the way for advancements in various application domains [10, 11, 12, 13, 14, 15] .Unlike the constant and flat Euclidean geometry, hyperbolic geometry offers greater flexibility by integrating curvature information, enabling better alignment with the characteristics of non-Euclidean input graphs. This endeavor has rendered hyperbolic GNNs more accessible and comparable to their Euclidean counterparts, resulting in promising performance and interpretability in graph representa-tion learning. Hyperbolic GNNs [16] extended the neighborhood aggregation operation by computing centroids in the hyperbolic geometry. This approach effectively fuses the node features and hierarchical structure, thereby learning superior node representations. Furthermore, a full manifold-preserving feature transformation operation has been developed in hyperbolic geometry [15] , eliminating the complicated transformations between hyperbolic and tangent spaces. With these essential operation, hyperbolic GNNs can achieve comparable or even superior performance than Euclidean GNNs.Question. Although there has been a surge of research on Euclidean and non-Euclidean GNNs in the community, it remains unclear which geometry offers greater advantages. Real-world graphs often exhibit geometrically structural heterogeneity, characterized by variations in clustering and density among nodes [17, 18] , as shown in Figure 1 . The structural heterogeneity pose a challenge when attempting to accurately model the graph structure using GNNs solely equipped with either Euclidean or non-Euclidean geometry.Motivation. According Local Subgraph Preservation Property [19] , the properties of a node largely depend on the properties of the local subgraph centered around it. Considering the hyperbolic property of local subgraphs, i,e., hyperbolicity 1 . Employing hyperbolic geometry modeling achieves higher precision and minimal information loss when hyperbolicity is low. Conversely, when hyperbolicity is elevated, opting for Euclidean geometry modeling results in lower complexity and slightly superior performance compared to hyperbolic geometry. Consequently, the primary limitation of existing graph neural networks is their inability to adaptively select the appropriate geometry for representing nodes with different local structures [21, 22] .Figure 1 : Visualization of the embedding of the same graph in hyperbolic space (left) and Euclidean space (right), with different colors representing different class labels. Tree-like subgraphs maintain significant inter-class margins in hyperbolic space, leading to improved classification boundaries, while intra-class nodes with Euclidean properties may be overstretched due to the utilization of hyperbolic metrics, hence embedding in Euclidean space is preferred.Our scheme. In this paper, we propose a crossgeometric graph knowledge distillation framework that encodes graphs utilizing both Euclidean and hyperbolic geometry in a locally space-mixing fashion. In contrast to traditional methods that compute hyperbolicity for the overall graph and roughly analyze its applicability to different geometries, our approach performs fine-grained analysis on the local subgraphs surrounding each node. This enables the selection of embeddings in the most appropriate geometry for local subgraphs, which is subsequently utilized to transfer knowledge to the student model. Additionally, we introduce a geometric embedding optimization module to optimize the distribution of embeddings produced by the teacher models. To evaluate the performance of our proposed approach, we conduct distillation experiments on node classification (NC) and link prediction (LP) tasks across various types of graph data. The experimental results demonstrate the superiority of our approach in teaching student models compared to other baseline methods. Our approach highlights enhanced effectiveness and generalization, ultimately achieving state-of-the-art performance in graph data distillation tasks. The salient aspects of our contributions are as follows:• Structured analysis reveals that both Euclidean and hyperbolic geometries demonstrate commendable performance in graph processing, despite their inherent disparities and potential geometric conflicts. This prompts scholarly inquiry into reconciling these divergent geometric spaces within GNNs, inspiring new research avenues in geometry awareness.• With heightened awareness, there's potential to represent graph structures across dimensions, transcending singular space limitations. We thus adapt and integrate teacher embeddings from diverse geometries, transferring them to more effective cross-geometric space.• Extensive experiments employing KD techniques on diverse graph datasets demonstrate that cross-geometric methods significantly outperform traditional approaches in the context of knowledge transfer. This is particularly evident in NC and LP tasks, thereby affirming the superior efficacy of these methods.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We have concisely written our motivation, contributions, and experimental results in the abstract.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Graph Neural Networks. GNNs are neural network models that capture interdependencies between nodes by propagating messages among them within a graph. The most representative model is Graph Nonvolutional Network (GCN) [23, 24, 25] , which can be regarded as a generalization of convolutional neural networks to graph data. The GraphSAGE [26] employs a neighbor sampling strategy to address graph data, enabling information aggregation based on the local neighborhood structure of nodes. The attention-based GNN [27, 28, 29] model employs masked self-attention, assigning diverse weights to node representations based on the varying features of neighboring nodes. Notably, constructing GNNs in the hyperbolic space [13, 15, 30] significantly reduces embedding distortions caused by the inability of Euclidean space to handle power-law distributions, particularly in the case of tree-like or highly hierarchical data.Knowledge Distillation. KD initially proposed by [31] , is a model compression technique that involves leveraging pre-trained teacher models to guide the training of a lightweight student model [32, 33] . After that, [34] aligns student and teacher model intermediate features using a regressor and a loss function to minimize feature differences. [35] employs attention mechanisms to extract features from teacher model's intermediate layers and transfer them to the student model. As GNNs have demonstrated breakthrough performance in various deep learning tasks, a number of graphbased KD frameworks have been proposed successively. [36] introduces a local structure preserving module to extract knowledge from intermediate layers of the GNN model, guiding the student model to optimize learned topological structures. [37] proposes a novel approach to effectively learn multi-scale topological semantics from multiple GNN teacher models to guide student model. [38] incorporates a VQ-VAE to learn a codebook that represents informative local structures, and uses these local structures as additional information for distillation. However, all these methods rely on the Euclidean geometry. Our proposed approach leverages both Euclidean and non-Euclidean geometries to learn representations of highly hierarchical local structures, ensuring that the knowledge transmitted to the student model is highly reliable.3 Problem Definition and PreliminariesQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In the final section, we have analyzed the drawbacks of our method and proposed the existing limitations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 For the graph KD, given a graph G = (V, E), where V denotes the node set and E denotes the edge set.Let N denotes the total number of nodes in the graph G, X denotes nodes' feature matrix with each row corresponding to the feature vector of a node, and A denotes graph's N × N adjacency matrix, where A ij signifies whether there is an edge between nodes i and j. If A ij = 1, an edge exists; otherwise, no edge is present. Let M T = {m T1 , m T2 , ..., m T R } denotes the teacher models pre-trained on G, R represents the number of geometries. Our fundamental objective is to extract information from G by M T , and employing it to boost the training process of the student model, denoted as m S , which in Euclidean space and has significantly smaller size. Let Z T = {Z T1 , Z T2 , ..., Z T R , } be the outputs of the teacher models and Z S be the outputs of the student model. The optimization goal is to minimize the disparity between predictions of M T and m S on G, i.e.,EQUATIONwhere β j denotes the weight of the j-th teacher model, F dis denotes disparity measurement function.In this paper, We focus on distillation performance of the teacher model individually in Euclidean, hyperbolic, and spherical geometries, as well as across geometries. We give a necessary introduction of hyperbolic geometry in this subsection, with other information available in Appendix A.Hyperbolic geometry studies the properties of curved space with negative curvature. Hyperbolic space can be modeled using five isometric models [39, 40] , and in this paper, we adopt Poincaré disk model. The Poincaré disk model evinces a distinctive property wherein distances from the geometric center to the periphery undergo a non-linear augmentation as a function of layer depth [9] . This phenomenon engenders a nonlinear and multi-branch composite structure within the model's geometric framework.Definition 3.1 (Poincaré Disk Model). A n-dimensional Poincaré disk model (B n c , g B) is a complete Riemannian manifold with a negative constant curvature c, which defined asEQUATIONwhere ∥ • ∥ denotes the Euclidean norm, g denotes the Riemannian metric, and The superscripts B and E indicate that the vector or matrix resides in hyperbolic space and Euclidean space, respectively. Definition 3.2 (Hyperbolic Operations). Given two points x, y ∈ B n c , the hyperbolic distance between them is defined asEQUATIONwhere ⊕ c denotes Möbius addition, i.e.,xEQUATION)Definition 3.3 (Tangent Space). The tangent space at a point x in hyperbolic space, denoted as T x B n c , serves as the first-order approximation of the original space, a n-dimensional tangent space is isomorphic to Euclidean space R n . Representations between hyperbolic and tangent space can be transformed via the exponential and logarithmic map as follows:EQUATIONwhere v ∈ T x B n c , y ∈ B n c and λ c x has same meaning in Eq. ( 2). Here we utilize the origin point o in hyperbolic space as a reference point x to equalize errors across various directions.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
cross-geometry learning with kd: 
 This section reveals three key aspects: why cross-geometry learning demonstrates superior performance, why it is feasible, and how this superiority is achieved by employing KD. Thus, we analyze our method from three perspectives: reasonableness, superiority, and trustworthiness.Reasonableness: According local subgraph perservation peoperty theorem [19] , nodes near the central node strongly affect its features, while distant nodes typically have negligible impact. The graph data in real-world often exhibits significant complexity, where diverse local subgraphs often entail distinct geometric properties [17, 18] , employing cross-geometry system can offer more effective embedding selections for local graphs, thereby achieving performance beyond that of single geometry methods. Definition 4.1 (Subgraphs of Centroid p): For a given node p belonging to graph G, its corresponding k-hops subgraph G p comprises all nodes q ∈ V\{p} within a distance no greater than k from p, along with their respective edges.We employ the k-hops neighbors method to generate subgraphs. Hence, we determine the optimal value of k through the statistical analysis of graph data in section 5.4. For each subgraph G i , we calculate their Gromov's δ-hyperbolicity (See the appendix A.1 for the calculating process) based on X , denoted as δ Gi , which serves as a geometric characterization metric for the central node i. denotes the hyperbolic teaching. GEO: Enhancing hint embeddings from the teacher models, reducing the negative effects of inconsistencies between different geometries.Superiority: We leverage KD technology, utilizing its ability to transfer knowledge between different model architectures, as a medium for interoperability between different geometries. Our proposed KD framework is model-agnostic, making it applicable to various geometric models.Herein, the framework will be explained using the GCN model [23] . To minimize disparities between the intermediate layers of the teacher and student models, our method uses pre-activation node embeddings z to guide the student model and constructs an embedding matrix Z for all nodes' z. During the forward propagation process of a GCN layer in Euclidean space, the intermediate embeddings of nodes in the l-th layer of GCN is given byEQUATIONwhereH (l-1),E T = activation(Z (l-1),E T) denotes the node representation matrix from the output of the (l -1)-th GCN layer. Â denotes the symmetrically normalized adjacency matrix, W (l) denotes the weight matrix.For the i-th node in layer l, its embedding in hyperbolic space is denoted as z (l),B T,i and its representation is denoted as h (l),B T,i . During the forward propagation process of a GCN layer in hyperbolic space, the transformed feature is given byEQUATIONBy performing neighborhood aggregation on these features, we obtain the hyperbolic intermediate embeddings of i-th node in the l-th layer as follows:EQUATIONwhere w ij is the weight coefficient computed by f(l),BT,i and f (l),B T,j .Trustworthiness: EQUATIONSimilarly, we can obtain representation set of i-th local structure in Euclidean geometry denoted as u (l),E T,i . SWKT generates induced k-hops subgraphs centered at each node, computes their δ Gi as the hierarchical level of the central node i, and obtains teacher models' middle layer representations of the i-th node in the l-th layer based on δ Gi as follows:EQUATIONwhere I denotes indicator function, the threshold λ is a hyperparameter that is typically set to a smaller value on graphs with higher δ-hyperbolicity values to achieve better performance.In our method, embeddings of l-th guided middle layers of student model isZ (l),E S, and applying Eq. ( 9) likewise yields the student model i-th local structure representation u (l)EQUATIONS,iN }. For node i, the similarity between the local structures of the teacher model and the student model is measured as:EQUATIONwhere D KL represents the Kullback-Leibler divergence.SWKT minimizes the local structure similarity P to transfer knowledge from hint embeddings in different geometry to student model. Thus, the loss function for the SWKT module isEQUATIONwhere L denotes the total number of intermediate layers used for distillation.Trustworthiness: Simply concatenating embeddings from Euclidean and hyperbolic teacher models to teach student model can lead to confusion due to geometric inconsistencies. This confusion may result in the student model performing worse than when learning from a single geometric teacher model, as shown in section 5.2. To mitigate the negative effects caused by this inconsistency, we propose a Geometric Embedding Optimization module (GEO) to optimize cross-geometric space.Specifically, for a given node i from layer l, we have its local geometric information δ Gi and teacher embeddings from different geometric spaces. We obtain an initial fused feature as follows:EQUATIONwhere λ has the same meaning as Eq. ( 10).Next, we use a single-layer GCN (which can be replaced by other sufficiently capable networks, such as a Multi-Layer Perceptron (MLP)) to optimize the initially fused features e(l)T,i . The optimization network should select loss functions based on the specific downstream task. In this study, we adopt the triplet loss function [41] , which enlarges the distance between different-class nodes and reduces the distance between same-class nodes, to enhance node classification and link prediction performance. To apply triplet loss to graph data, we organize triplets as follows: Given the hint embeddings from teacher, we sample extensive sets of nodes, where each set includes an anchor node, a positive node with the same label as the anchor, and a negative node with a different label.Assuming F e is corresponding function of a pre-trained GEO network with weight matrix W e , the elements of the local structure vector u (l) T,i for the i-th node of layer l can be represented asEQUATIONwhere F sub has the same meaning as Eq. ( 9).Subsequently, we can reference Eq. ( 11) to compute the structural similarity between the outputs of GEO and the guided embeddings of l-th layer of the student model as:EQUATION)In our proposed graph KD approach, the teacher model's early L -1 layers guide the student model using the SWKT module, while the L-th layer guides via the GEO module. Additionally, the student model also learns the logits distribution of teacher models, i.e., outputs of GEO in last layer. Given the logits y T from teacher models and the predicted logits y S from student model, our overall KD loss is as follows: EQUATION 
experiments: 
 In this section, we first give the experimental setup and baselines. Then we compare our graph KD framework with some baselines on NC and LP tasks. Hyperparameters and ablation analysis also be given. Further experimental results can be found in Appendix C.Setups. We preform NC and LP tasks on citation network datasets Cora [42] ,Citeseer [43] and Pubmed [44] , wikipedia-based article hyperlink network dataset Wiki-CS [45] , and Physics part of the Coauthor dataset Co-Physics [46] . The student and teacher models are both GCN composed of two hidden layers and one output layer. The hidden layer node dimensions are 8 for the student and 128 for teachers. The model parameters are uniformly initialized using the Xavier's uniform initialization [47] method The optimizer uses Adam [48] or Riemannian Adam [49] . We set the value of k for k-hops subgraphs to 4. To mitigate errors caused by randomness, each F1-score and ROC AUC is the average of 10 experiments with different random seed values.Baselines. To evaluate the performance of our method, we compare it with KD methods formulated in single geometry, including the following methods. FitNet [34] utilizes a regressor to align the intermediate features of the student model with those of the teacher model, quantifying the feature discrepancy using L2 distance. AT [35] averages attention maps from both teacher and student models' intermediate hidden layers, quantifying differences between them using a designed loss function. LSP [36] extracts local structures from both teacher and student models' intermediate feature maps and measures their difference using KL divergence. MSKD [37] Utilizes diverse teacher models with varying layers to guide the student model in capturing topology at different scales. VQG [38] incorporates a VQ-VAE to learn a codebook that represents informative local structures, and uses these local structures as additional information for distillation. To comprehensively demonstrate the superiority of cross-geometry over individual geometry, we conducted experiments for each method separately in Euclidean space E, hyperbolic space B, and spherical space S. Here, we adopt a spherical space S with curvature of 1, for further details, please refer to the Appendix A.4. We further conducted exploratory experiments on alternative geometric integration approaches based on that proposed in section 4, as illustrated by Cross in Table 1 .We use F1 scores as the evaluation metric for node classification task and present results in Table 1 .In comparing LSPs across different geometries, we found a counterintuitive outcome. In datasets with low δ-hyperbolicity, hyperbolic LSP was anticipated to outperform Euclidean LSP. However, it performed worse, dropping by 3.05% (Wiki-CS). This suggests that even if a teacher model excels in one geometry, its guidance may be less effective when the student model operates in a different geometry. This highlights a significant gap between hint embeddings in different geometries. Despite employing diverse geometries, our method obtains 1.11% average improvement over SOTA baselines and especially 2.66% and 1.37% on Cora and Wiki-CS, indicating that our method excels on datasets with lower δ-hyperbolicity. Besides, even in graph with high δ-hyperbolicity, where graph exhibit few hierarchical levels, our method consistently achieves higher F1-score compared to student models obtained by baseline KD methods in a single geometrc space.With the rapid growth of information, real-world graph data is expanding in scale. To demonstrate the effectiveness of our method on large-scale graphs, we evaluated the F1 scores of NC tasks using distilled student models on larger datasets, Ogbn-Arxiv (1,166,243 edges, 169,343 nodes) and Ogbn-Proteins (39,561,252 edges, 132,534 nodes) [50] . Results in Table 2 show that our method consistently achieves superior distillation performance on these larger datasets.We use ROC AUC as the evaluation metric for link prediction task and present results in Table 1 . The average ROC AUC showed an improvement of 1.58%, with particularly notable enhancements on datasets Wiki-CS and Cora, reaching 1.87% and 2.57%, respectively. Employing KD methods solely based on hyperbolic geometry outperformed those exclusively utilizing Euclidean geometry, particularly on the Citeseer. Our cross-geometry KD method outperformed SOTA baselines by 0.93% on average, underscoring the efficacy of geometry-specific methods in cross-geometry learning for overall performance enhancement.EQUATIONk of Subgraphs. A small radius k limits local hierarchical assessment, while a large k increases computational complexity. We explored δ i value distributions for local geometric properties across k values (1 to 7) on Wiki-CS and Cora datasets, shown in Figure 3 (left). Stability in distributions occurs at k ≥ 4, suggesting sufficient capture of geometric characteristics in subgraphs of this size. Thus, we set k = 4 for our experiments.We varied the hyperparameters on the Wiki-CS and Cora datasets to test the F1 scores for the NC task, with λ ∈ {0.0, 0.5, 1.0, 1.5, 2.0, 2.5} , β ∈ {1, 2, 3, 4, 5, 10}. Here, λ denotes the threshold value in Eq. (10) and Eq. (13) . A larger λ leads to more local subgraphs being embedded in hyperbolic space, while a smaller λ results in more subgraphs being embedded in Euclidean space. β denotes the weight of the GEO module. The results from Figure 3 (middle) indicate that the hyperparameters λ and β have a generally minimal impact on the outcomes. The combination of λ = 1.5 and β = 3.0 maintains optimal performance.To evaluate the convergence efficiency of our proposed KD method, we have recorded the F1-scores trends for student models guided by all KD methods during training epochs on the Cora and Wiki-CS datasets in Figure 3 (right). As illustrated, our KD method consistently outperform other methods within the same training epochs. Specifically, our KD method achieves state-of-the-art (SOTA) performance before reaching 500 epochs, while the others are still undergoing training. These results serve to validate the effectiveness and efficiency of our method. Due to their simpler architecture, student models generally have faster inference speeds compared to teacher models. The speedup achieved by student models relative to teacher models is also an indicator of the efficiency of distillation methods. The inference times (in milliseconds) of both teacher and student models, measured on our device, are shown in Table 3 . Results demonstrate that our method achieves an average speed-up of approximately 232x.In addition, we evaluated the training time for each method, the inference time for the corresponding student models, and calculated the compression ratio of student model size relative to the teacher model. The results can be found in Appendix C.To further validate the efficacy of cross-geometric learning and the two proposed modules, we conducted additional experiments by adapting our method to operate on a single Euclidean or hyperbolic geometry. Additionally, we selectively excluded the SWKT and GEO modules. These ablation experiments were conducted on the Wiki-CS dataset, and the results are presented in Table 4 .We have the following observations:• Using only a single geometry, even with the GEO module optimizing embeddings, the enhancement compared to the baseline is minimal. • The cross-geometric approach consistently outperforms the single-geometric methods.Whether excluding SWKT or GEO, the results are inferior to the comprehensive method, indicating their crucial roles in optimizing geometric embedding distribution.The overall results of ablation analysis further demonstrate the importance of cross-geometry learning and our proposed two modules.To demonstrate the model-agnostic nature of our framework, we alter the architecture and the number of layers L in the teacher model. Due to page limitations, we only present key results above. For more details on the dataset, experimental setup, and results, please refer to Appendix B and C. 
conclusion: 
 Hyperbolic geometry has shown expressive non-Euclidean modeling in the graph community. Noteworthy models, such as Poincaré and Lorentz models, facilitate vector projections between Euclidean and hyperbolic neurons. Our investigation reveals that tree-like or power-law distributed graphs exhibit multiple different hierarchical within locally connected structures. Consequently, training across Euclidean and hyperbolic geometry intuitively emerges as a more flexible approach to graph modeling, yielding significant enhancements in KD tasks. To this end, we introduce a novel KD framework that models the hint embeddings of the teacher models across diverse geometries. By leveraging δ-hyperbolicity, we transfer local subgraphs information to the student model. Our analysis and experiments provide positive support for this innovative perspective on geometry modeling.Limitations. Despite the performance improvements achieved by cross-geometry learning across various tasks, it presents some potential issues. For instance, integrating different geometric information introduces hyperparameters, making the task outcomes somewhat dependent on their selection, thus affecting the method's stability. Additionally, the distillation phase demands more complex pre-trained models, increasing resource and time requirements. These limitations are critical areas for future enhancement in cross-geometry learning.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our work can reduce resource consumption and have a positive impact on society, without any negative effects.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not have a high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have provided proper citations and acknowledgments for all the resources we have utilized.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Gromov Hyperbolicity Gromov's δ-hyperbolicity [20] measures a graph's tree-like structure, with lower δ values indicating higher hyperbolicity in a graph dataset, where δ = 0 represents a tree. In this paper, we compute the hyperbolicity of the k-hop subgraph G i for each node i as the geometric feature information of the local structure of that node. Here, we provide the detailed calculation process.First, four nodes a, b, c, d are randomly sampled from the subgraph G i . Let S 1 , S 2 and S 3 be defined as follows:EQUATIONwhere dist denotes shortest path length between two nodes.Let M 1 and M 2 be the two largest values among S 1 , S 2 and S 3 . We defineEQUATION)The hyperbolicity δ Gi of the graph G i is the maximum of hyp over all possible 4-tuples (a, b, c, d) divided by 2, i.e.,EQUATION)In our paper, we calculate the δ Gi for each k-hop subgraphs. For subgraphs with fewer than four nodes, we label their δ Gi value as N/A.Geometry is a branch of mathematics concerned with properties of space such as the distance, shape, size and relative position of figures. In this paper, we analyze the modeling capabilities of graph neural networks in Euclidean , hyperbolic and spherical geometries. Euclidean geometry studies the properties of flat space with zero curvature, where parallel lines never meet, and angles of a triangle sum to 180 degrees. In Euclidean geometry, the volume of space exhibits polynomial growth associated with the dimensionality of the space. The majority of neural network models perform inference operations in this space, where operations such as convolution, pooling, and activation are based on the basic arithmetic operations of addition, subtraction, multiplication, and division.Euclidean geometry is an axiomatic system, in which all theorems are derived from a small number of simple axioms.• To draw a straight line from any point to any point.• To produce (extend) a finite straight line continuously in a straight line.• To describe a circle with any centre and distance (radius).• That all right angles are equal to one another.• [The parallel postulate]: That, if a straight line falling on two straight lines make the interior angles on the same side less than two right angles, the two straight lines, if produced indefinitely, meet on that side on which the angles are less than two right angles.These axioms provide the fundamental mathematical framework for Euclidean space, allowing GNN models to incorporate information about the absolute positions of nodes, properties of lines, and spatial relationships.Hyperbolic geometry is non-Euclidean geometry, also called Lobachevsky-Bolyai-Gauss geometry. This geometry adheres to all of Euclid's postulates, with the exception of the parallel postulate, which has been substituted with:• If a straight line intersects two other straight lines, and so makes the two interior angles on one side of it together less than two right angles, then the other straight lines will meet at a point if extended far enough on the side on which the angles are less than two right angles.Hyperbolic space is a homogeneous space with constant negative curvature. In Euclidean space, the curvature is zero, while in hyperbolic space, the curvature is a negative constant. Moreover, smaller curvature leads to faster volume growth, as illustrated in Figure 4 . The hyperbolic space can be modelled using five isomorphic models which are the Lorentz model [51] , the Poincaré ball model and Poincaré half space model, and the Klein model. In this paper, we utilize a hyperbolic geometric teacher model based on the Poincaré model. The Poincaré model B is a manifold equipped with a Riemannian metric g B . This metric is conformal to the Euclidean metric g B . Formally, an n dimensional Poincaré unit ball (manifold) is defined asEQUATIONwhere ∥ • ∥ denotes the Euclidean norm. Formally, the distance between x, y ∈ B n is defined as:d(x, y) = arcosh(1 + 2 ∥x -y∥ 2 (1 -∥x∥ 2 )(1 -∥y∥ 2 )).The Möbius addition ⊕ for x and y in B n is defined asEQUATION)The Möbius scalar multiplication ⊗ is defined asEQUATIONwhere r is a scalar factor.The Möbius vector multiplication M ⊗ (x) is defined asEQUATIONSpherical geometry studies the properties of curved space with constant positive curvature, where the angles of a triangle add up to exceeds 180 degrees. All lines in spherical geometry intersect, as there are no parallel lines on a sphere. In the field of graph embedding, spherical geometry plays a significant role as it provides a more realistic model, particularly applicable in geographic information systems and computer graphics. Through spherical geometry, we can accurately describe features on the surface of the Earth and construct data representations with spherical topological structures in for l in {1, 2, ..., L} do 3:Update Z (l),E and Z (l),B , ZEQUATION,B T ← Eq. ( 6), Eq. ( 8); 4:Select appropriate geometry for middle representations, uT ← Eq. ( 10):5:Calculate structure similarity, P (l) ← Eq. ( 11); 6:Reduce geometries' discrepancy, Optimize embeddings by Eq. ( 14);7:L SW KT , L GEO ← Eq. ( 12), (end for 9:Calculate overall Loss by Eq.( 16); 11.0 three-dimensional space, which is crucial for applications such as map-making, virtual reality, and computer games. In graph embedding, concepts and algorithms of spherical geometry are utilized to process data with spherical topological structures, such as mapping the Earth's surface onto a twodimensional plane while preserving the correctness of geographic locations and spatial relationships. Therefore, spherical geometry is not only a theoretical discipline but also an indispensable tool in practical applications.Given the graph data, we initially train teacher models in Euclidean, hyperbolic, and spherical spaces, respectively. Subsequently, we train the MLP model of the GEO module, initializing the student model randomly. By inputting learning parameters alongside hyperparameters λ and β, we employ Algorithm 1 to obtain the distilled student model.Here, we present detailed information for each dataset in Our proposed framework is model-agnostic. To validate its universality and effectiveness, we conducted experiments by replacing the teacher model from GCN to GAT. These experiments were conducted across three geometries: Euclidean, hyperbolic, and spherical, for cross-geometry learning.The experimental results are presented in Table 8 . As depicted in the table, even when the teacher model is replaced with other models, our framework consistently maintains a strong distillation effect, with the combination of hyperbolic and Euclidean geometries still proving to be optimal. Moreover, as the performance of the teacher model improves, there is a corresponding enhancement in the performance of the student model.We also replaced the Euclidean teacher model with the Graph Transformer Network [52] . The results are shown in Table 9 . GTN teacher has 4 layers and a hidden dimension of 128. Specifically, during distillation, student model's l layers match the last l layers of teacher accordingly. The GTN and HGCN teacher output intermediate representations from each layer to the SWKT module for local subgraph structure extracting and selection. These distributions are then optimized by GEO module. Then, these features extracted from the optimized cross-geometric intermediate representations are transferred to students via the corresponding loss function. Additionally, traditional KD loss is computed from the logits output by both the teacher and student models.The student model can operate in other geometric spaces. At first, we chose a Euclidean student model to combine hyperbolic accuracy benefits with Euclidean efficiency and stability. Our framework is model-agnostic, allowing replacement of the student model with other neural networks.To validate student on various geometric spaces, we tested NC F1 scores (%) on the Cora dataset in Table 10 .Euclidean and hyperbolic teachers' F1 score is 86.98% and 90.90%.we conducted experiments using student models with the same architecture as the teacher models in our method. Following results are NC F1 scores (%) on the Cora dataset. Euclidean and hyperbolic teachers' F1 score is 86.98% and 90.90%. The results are shown in Table 11 Compared to the results presented in Table 1 of our paper, student models now even outperform some teacher models, but using the same architecture as the teacher makes student models larger and slower, limiting their suitability for resource-constrained devices. In addition to model-agnostic features, our framework demonstrates excellent scalability. All experiments in this study were conducted with both teacher and student models having a hidden layer depth of 2. To verify scalability, we configured four types of teacher models, varying their hidden layer depths from 1 to 4, while keeping all other settings constant. By applying the SWKT and GEO modules to each layer, we expanded our framework, as illustrated in Table 12 . As observed in the table, despite variations in the number of layers in the teacher models, our framework consistently achieves effective distillation results, with the combination of hyperbolic and Euclidean components remaining optimal. Furthermore, as the performance of the teacher models improves, there is a corresponding enhancement in the performance of the student models.We reduced the embeddings of student models to 2-dimensional space by t-SNE and visualized them in Figure 5 , our method yields a superior embedding distribution, which more suitable for NC task.We conducted a more comprehensive hyperparameter analysis on the Co-Physics, Pubmed, and CiteSeer datasets. By adjusting the hyperparameters, we evaluated the F1 scores for the NC task, with λ ∈ 0.0, 0.5, 1.0, 1.5, 2.0, 2.5 and β ∈ 1, 2, 3, 4, 5, 10. The results, as shown in Figure 6 , indicate that the hyperparameters λ and β have a minimal overall impact on the outcomes. The performance is generally optimal when β = 3, and λ shows better performance at intermediate values.Following the ablation experiment strategy outlined in Section 5.2, we conducted NC experiments on five other datasets. The results, presented in Table 13 , reveal that the performance of the comprehensive method consistently outperforms other conditions across all datasets. Figure 5 : t-SNE Visualization of embeddings obtained by student models.In contrast to baselines,our method achieves embeddings that fully utilize the entire space, ensuring substantial inter-class distances and thereby enhancing node classification performance.In dataset Wiki-CS, we assessed the training and inference time per epoch and the ratio of the total parameter count of the student model to that of the teacher model, as outlined in Table 15 . Our KD method achieves similar time efficiency in both training and inference stages compared to other methods. Notably, our method achieves superior results at the highest compression level, thereby further validating the efficacy of our KD method in generating compact yet high-performing student models.We provide an analysis of the time spent by each knowledge distillation (KD) method during the training of student models on the network classification (NC) task, recorded for every epoch across all datasets, as shown in Table 16 . We also present the time taken for inference of the student models on the NC task in Table 14 . From the results, it is clear that in various scenarios, the time required for our method is comparable to that of other methods, with no significant increase in time cost. This suggests that our approach effectively balances performance and computational efficiency, making it a practical choice for applications in this field. The failure of one or more teacher models could potentially impact the student model's performance, we have implemented several mechanisms in our method to mitigate this risk:• Ensemble Learning: Using multiple teacher models that capture different geometric properties provides redundancy and robustness. If one model fails, the others still contribute valuable insights, minimizing the impact on the student model.• Geometric Optimization Network:GEO dynamically adjusts the weight of information from each teacher model based on the loss function, reducing the influence of any underperforming model and ensuring the student model receives the most reliable information.We designed various experimental strategies to assess the impact of failing teachers on students:• S1: Train student models without KD.• S2: Train student models with the best-tuned teacher model.• S3: Train student models with an underperforming teacher model.• S4: Train student models with an untrained teacher model.• S5: Train student models with all untrained teacher models.Note: All methods except MSKD and ours use a single teacher model; thus, S5 is N/A. 17 . Except S5, which teachers have an average performance of only 30%, our method's distilled student models consistently maintain stable performance even when some teacher models fail.NeurIPS Paper Checklist38th Conference on Neural Information Processing Systems (NeurIPS 2024).Gromov's δ-hyperbolicity[20] (See the appendix A.1 for the calculating process) measures a graph's tree-like structure, with lower δ values indicating higher hyperbolicity in a graph dataset, where δ = 0 represents a tree. 


Decision: This paper proposes a graph knowledge distillation framework for transferring knowledge between different graph geometries. The framework distills knowledge from multiple teacher GNNs, each aware of a specific geometry, into a smaller student GNN. This approach not only improves the quality of graph representations but also accelerates inference speed. The authors successfully addressed the reviewers' concerns during the author-reviewer discussion phase, and at the end, all reviewers unanimously support acceptance. AC also finds that this paper has several strengths, including its novelty, extensive experimental results, and superior performance.

Therefore, AC recommends acceptance and highly encourages the authors to incorporate all experimental results and discussions from the discussion phase into the final manuscript, as this would strengthen the contributions of this paper.

Reviews: review: paper_summary: paper_summary: This paper presents a cross-geometric graph knowledge distillation method for graph neural networks. This method employs multiple teacher models, each generating different embeddings with distinct geometric properties, such as Euclidean, hyperbolic, and spherical spaces. The student model is based on Euclidean space. Two modules, Structure-Wise Knowledge Transfer (SWKT) and Geometric Embedding Optimization (GEO), are proposed to enhance performance. To evaluate the proposed approach, distillation experiments are conducted on node classification (NC) and link prediction (LP) tasks across various types of graph data.
strength_weakness: strength_weakness: 1. In the presented cross-geometric knowledge distillation framework, the student model learns from multiple teacher models with different geometric information. Properly training all these teacher models is challenging. It is unclear how to ensure the student model's reliability if some teacher models fail.
 
2. The student model operates only in Euclidean space, raising questions about whether other student models, such as those in hyperbolic space, can achieve similar results. 

3. All experiments are conducted solely on GCN, with no results provided for other graph architectures. 

4. It is unclear if a student model using the same architecture as the teacher models, within the cross-geometric knowledge distillation framework, can achieve better performance than the teacher models.
questions: questions: 1. If one or more teacher models fail, how does this impact the performance of the student model?
2. Can the student model operate in other geometric spaces, such as hyperbolic space?
3. Can the proposed framework work with other architectures, such as Graph Transformer Networks [1]?
4. If the student model uses the same architecture as the teacher models, how does it perform?

[1] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, Hyunwoo J. Kim:
Graph Transformer Networks. NeurIPS 2019
limitations: limitations: This work has discussed the potential limitations in their paper:
paper_summary: paper_summary: This paper proposes a novel methodology to distill graph neural network (GNN) integrating the various geometries (e.g. Euclidean and hyperbolic). The author first utilizes a structure-wise knowledge transfer module with multiple teacher models from distinct geometric properties.  Then, the authors demonstrate a geometric embedding optimization to guide a student model optimizing cross-geometric space. The authors show the efficacy of their approach through experiments on node classification and link prediction with various datasets compared with existing state-of-the-art methods.
strength_weakness: strength_weakness: 1. Some questions exist about the authors' empirical results. Please see the questions section.
questions: questions: 1. Are Table 1 results all reproduced by the authors? If so, the reported performance of other baselines seems significantly lower than the numbers reported in other papers. For example, MSKD (Zhang et al.) reports an 89.67 F1-Score with an 8.2 ratio on the CiteSeer node classification task. On the other hand, the MSKD NC F1-Score in Table 1 is only 71.42.
2. Following up with 1, GAT results in Table 7 on CiteSeer are significantly lower than the MSKD paper reported. For example, the teacher model trained on CiteSeer with Euclidean geometries only achieves an F1-score of 75.24, while the MSKD paper reports their student model achieves a 95.47 F1-Score with a 17.6 ratio! 
3. Can you do the hyperparameter tuning on the training such that the teacher's performance on Citeseer is close to what MSKD (Zhang et al.) has reported? In distillation literature, experimenting with the very well-trained teacher model is important. If not, the gain from the student model could have been illusory since the performance gap could have been larger if the teacher had trained with better hyperparameters. 
4. VQG authors (Yang et al.) claim that VQGraph has a significant advantage in the inference speed (828x compared to GNN), but the Table 2 inference speeds report all the same inference speeds on FitNet, AT, LSP, MSK, VQG, and the author's method on NC. I am skeptical whether these are the right numbers.  
5. Also, VQG may not be the right candidate to compare since VQG tackles a different problem, a GNN to MLP distillation.
limitations: limitations: Overall, the authors' experiments show their new contributions can improve the GNN models in an agnostic way. However, some numerical performance results from the authors are too low (including both teacher and student) compared to the other papers reported. After the authors match their teacher's performance to the other papers (e.g. CiteSeer from MSKD by Zhang et al.) and observe good student performance with a small gap to the teacher + state-of-the-art results, I am happy to raise the score.
paper_summary: paper_summary: This paper introduces a new graph distillation framework using special teacher networks to consider Euclidean and hyperbolic geometries when performing the distillation to the light-weight student GNN model.
Key techniques include Structure-Wise Knowledge Transfer (SWKT) for selecting appropriate geometric spaces and Geometric Embedding Optimization (GEO) for feature fusion across geometries.
strength_weakness: strength_weakness: 1. The paper lacks an efficiency study comparing the computational efficiency between the teacher and student models.

2. Larger dataset should be included to better evaluate the proposed distillation methods such as ogbn-arxiv and ogbn-product

3. The motivation of choosing GNN as the student network is not sufficiently justified. 
While previous works focus on focus on MLP-based students, more clarity on this choice would be beneficial.
questions: questions: 1. I am curious that the input of the student model is full graphs or sampled subgraphs? How does this differ from student networks using MLPs [1].

2. Are there any baseline methods that do not incorporate various manifolds to compare with to show the effectiveness of manifold-aware approaches?


[1] Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a novel method for graph knowledge distillation. The proposed method incorporates knowledge from both Euclidean and hyperbolic teacher models and transfers it to a student model in a way that leverages the appropriate geometry for different local subgraphs. A SWKT module is used  to select embeddings from the most suitable geometry based on the local subgraph structure. Furthermore, a GEO module refines the transferred knowledge by mitigating inconsistencies between Euclidean and hyperbolic spaces.
Experimental results demonstrate that the proposed method outperforms other KD methods.
strength_weakness: strength_weakness: W1. The paper would benefit from a thorough analysis of the method's time and space complexity. 

W2. The proposed approach uses two geometries (,i.e., Euclidean and hyperbolic). While this is a significant advancement, the paper could be strengthened by discussing its potential extension to incorporate additional non-Euclidean geometries. Exploring the necessary modifications for handling multiple geometries would broaden the applicability of the method.

W3. While providing background knowledge can be helpful, the paper should ensure all definitions in the preliminaries section are essential for understanding the core method. Definitions like the Poincaré Disk Model and Tangent Space, if not directly used, could be moved to an appendix or omitted for clarity.

W4. The paper primarily focuses on Euclidean and hyperbolic teacher models. Further exploration with teacher models employing different geometries, such as spherical space, would provide valuable insights into the method's adaptability. Analyzing the performance with these variations would strengthen the paper's contribution.
questions: questions: See weaknesses.
limitations: limitations: The authors have discussed the limitations.

Comments: title: General Response
Thank you!
Thank you!
Thank you!
Official comments by Reviewer GamE
comment: We sincerely thank the reviewers for their time and effort in reviewing our paper and discussing with us. We appreciate the AC, SAC, and PC for their diligent efforts in organizing and overseeing the paper review process. We are delighted to see that during the discussion phase, Reviewer ogwd, Z3B3, GamE stated that our responses and new experiments have successfully addressed most of their concerns, and that all reviewers ultimately have a positive attitude toward our paper. We will ensure that these analyses and experimental results are included in the revised version of the paper.

Best regards,

Authors
Thank you for your positive feedback and we are glad to know that our rebuttal and new experiments have addressed most of your concerns!

Graph data may exhibit geometric heterogeneity (e.g., grid or tree structures) across different local structures. Therefore, we are committed to matching appropriate embedding geometric spaces through meticulous analysis and extraction of local structural features to enhance graph representation quality. We validated the effectiveness of this paradigm through knowledge distillation tasks. 

We are pleased to learn that you found our paper is well-written and that the experimental results validate the effectiveness of our agnostic cross-geometric framework and modules. In our initial rebuttal, we analyzed the speedup achieved by our method from both theoretical and practical perspectives and compared our method with GNN-to-MLP methods by replacing the student model of our method with MLP. We will ensure that these important details are included in the revised version. Thank you again for your insightful comments and valuable suggestions!

Best regards,

Authors
We are glad that our initial rebuttal addressed your concerns well!

Graph data may exhibit geometric heterogeneity (e.g., grid or tree structures) across different local structures. Therefore, we are committed to matching appropriate embedding geometric spaces through meticulous analysis and extraction of local structural features to enhance graph representation quality. We validated the effectiveness of this paradigm through knowledge distillation tasks. 

We are delighted that you find our approach of consider various geometries in distillation intriguing, and that the experimental results demonstrate its effectiveness. We fully agree that conducting experiments on larger datasets and analyzing the selection motivations and computational efficiency of the student models will enhance the quality of our paper. In our initial rebuttal, we conducted experiments on larger datasets and provided a comparative analysis of the computational efficiency between the teacher and student models, as well as explained the rationale behind the selection of student model architectures. We will incorporate these additions into the revised version of our paper as your suggestions! Thank you again for your insightful comments and valuable suggestions!

Best regards,

Authors
We sincerely appreciate your valuable reviews and are glad to know that our rebuttal and new experiments have addressed most of your concerns. 

Graph data may exhibit geometric heterogeneity (e.g., grid or tree structures) across different local structures. Therefore, we are committed to matching appropriate embedding geometric spaces through meticulous analysis and extraction of local structural features to enhance graph representation quality. We validated the effectiveness of this paradigm through knowledge distillation tasks. 

We are delighted to see that you consider our method, which combines the advantages of teacher models from different geometric spaces and analyzes local subgraph features from a fine-grained perspective for knowledge distillation, to be an innovative distillation paradigm. We fully agree that further validating the robustness of our framework (e.g., cases where some teacher models fail) and its model-agnostic nature (e.g., changing the architecture of teacher and student models) will strengthen our work. We conducted some experiments in our initial response to provide support and will incorporate these results into the experimental section in the revised version as your suggestions! Thank you again for your insightful comments and valuable suggestions!

Best regards,

Authors
**Dear Reviewer rr98:**

Thank you for your response! We noticed that you did not mention whether our prior responses addressed your concerns. We cautiously infer that  our prior responses may not have fully alleviated your concerns regarding the performance of our framework when extended to other geometric spaces. Upon receiving your initial comments, we began conducting additional experiments to integrate spherical space into our framework. To date, we have obtained the following results (some data are from Table 1), the best results are in bold, and the suboptimal results are italic:


- F1 score (%) of the **Euclidean student model** on the dataset when incorporating the spherical teacher:

|                                                 | ogbn-arxiv | ogbn-proteins | Wiki-CS   | Co-Physics | Pubmed    | Citeseer  | Cora      |
| ----------------------------------------------- | ---------- | ------------- | --------- | ---------- | --------- | --------- | --------- |
| **Spherical Teacher**                           | 70.11      | 70.74         | 69.13     | 96.27      | 82.14     | 71.88     | 82.48     |
| **Spherical + Hyperbolic Teachers**             | **71.25**  | *70.89*       | 70.07     | 96.17      | *82.23*   | 71.90     | 82.74     |
| **Spherical + Euclidean Teachers**              | 70.87      | 70.56         | *70.85*   | 96.07      | 80.45     | *71.98*   | 82.89     |
| **Spherical + Hyperbolic + Euclidean Teachers** | 70.75      | 70.58         | 68.70     | *96.37*    | 81.50     | 71.77     | *83.19*   |
| **Hyperbolic + Euclidean Teachers**             | *70.89*    | **71.22**     | **74.17** | **96.87**  | **82.73** | **72.60** | **86.05** |


- F1 score (%) of the **spherical student model** on the dataset when incorporating the spherical teacher:

|                                                 | ogbn-arxiv | ogbn-proteins | Wiki-CS   | Co-Physics | Pubmed    | Citeseer  | Cora      |
| ----------------------------------------------- | ---------- | ------------- | --------- | ---------- | --------- | --------- | --------- |
| **Spherical Teacher**                           | 68.11      | 65.23         | 74.34     | 95.27      | 84.63     | 75.15     | 82.17     |
| **Spherical + Hyperbolic Teachers**             | *70.13*    | *70.71*       | *77.98*   | 96.10      | 85.00     | *77.97*   | 84.37     |
| **Spherical + Euclidean Teachers**              | 69.54      | 69.71         | 77.17     | **96.21**  | 85.08     | 77.96     | 84.13     |
| **Spherical + Hyperbolic + Euclidean Teachers** | 68.53      | 70.10         | 73.10     | 96.07      | *85.72*   | 77.63     | *84.52*   |
| **Hyperbolic + Euclidean Teachers**             | **70.59**  | **70.97**     | **79.24** | *96.13*    | **85.83** | **78.21** | **86.73** |



The experimental results indicate that, in some cases, the distillation performance of the teacher group including spherical space teachers outperform those of the hyperbolic and Euclidean combination. This suggests that seeking additional knowledge from other geometric spaces, including spherical space, is of significant importance for improving our framework.

We thank you again for your detailed review and valuable suggestions. We hope that these supplementary comments can address your concerns. We would be glad to further discuss these new experimental results with you!

Best regards,

Authors
Thanks for the authors' effort to address my concerns. I have raised my score.
Thank you for your response. After thoroughly reviewing it and considering the results of the additional experiments, most of my concerns have been addressed. I raised my score.
Thank you for the response. I have carefully reviewed it and the results of the new experiments have addressed most of my concerns. I am pleased to increase my score.
Thank the authors for their response. I will keep my score.
Dear Reviewer rr98, 

We thank again for your thorough review and constructive feedback. In our response to your comments, we included an analysis of our method's time and space complexity and noted that our original work have explored integrating other geometric models into the framework and presented the results. We genuinely hope you could check our responses, and kindly let us know your valuable feedback. We would be happy to provide any additional clarifications that you may need.

Best regards,

Authors
Dear Reviewer Z3B3,

We sincerely thank you for your time and effort in reviewing our paper. In responses to your comments, we reviewed relevant works and demonstrated that our pre-trained teacher models are sufficiently trained on the Citeseer dataset. We analyzed the acceleration ratio of the student model compared to the teacher model from both theoretical and practical perspectives. Additionally, we replaced our student model with an MLP model of the same architecture as used in the VQG method and conducted comparative experiments. We would be grateful for the opportunity to discuss whether your concerns have been fully addressed. Please let us know if you still have any unclear part of our work.

Best regards,

Authors
Dear Reviewer ogwd,

We sincerely thank you for your time and effort in reviewing our paper. In responses to your comments, we have carefully designed and conducted experiments and analyses, which we believe have covered your concerns. We would be grateful for the opportunity to discuss whether your concerns have been fully addressed. Please let us know if you still have any unclear part of our work.

Best regards,

Authors
Dear Reviewer GamE, 

We appreciate your thorough review and constructive feedback. We have provided an analysis of the computational efficiency of our method and further validated its performance on the ogbn-arxiv and ogbn-proteins datasets. We have analyzed the differences between GNN and MLP student models and clarified potential misunderstandings about the baseline methods. We would be grateful for the opportunity to discuss whether your concerns have been fully addressed. Please let us know if you still have any unclear part of our work.

Best regards,

Authors