Geometry Awakening: Cross-Geometry Learning Exhibits Superiority over Individual Structures
Authors: Yadong Sun
Xiaofeng Cao
Yu Wang
Wei Ye
Jingcai Guo
Qing Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Cross-geometry learning
Graph neural networks
Graph distillation
Primary area: graph_neural_networks

Abstract: Recent research has underscored the efficacy of Graph Neural Networks (GNNs) in modeling diverse geometric structures within graph data. However, real-world graphs typically exhibit geometrically heterogeneous characteristics, rendering the confinement to a single geometric paradigm insufficient for capturing their intricate structural complexities. To address this limitation, we examine the performance of GNNs across various geometries through the lens of knowledge distillation (KD) and introduce a novel cross-geometric framework. This framework encodes graphs by integrating both Euclidean and hyperbolic geometries in a space-mixing fashion. Our approach employs multiple teacher models, each generating hint embeddings that encapsulate distinct geometric properties. We then implement a structure-wise knowledge transfer module that optimally leverages these embeddings within their respective geometric contexts, thereby enhancing the training efficacy of the student model. Additionally, our framework incorporates a geometric optimization network designed to bridge the distributional disparities among these embeddings. Experimental results demonstrate that our model-agnostic framework more effectively captures topological graph knowledge, resulting in superior performance of the student models when compared to traditional KD methodologies.

Geometry Awakening: Cross-Geometry Learning Exhibits Superiority over Individual Structures
Yadong Sun! Xiaofeng Cao Yu Wang' Wei Ye² Jingeai Guo
Qing Guo
School of Artificial Intelligence, Jilin University. China 2College of Electronic and Information Engineering, Tongji University, China The Hong Kong Polytechnic University
CFAR and IHPC, Agency for Science, Technology and Research (A*STAR), Singapore
s sunyd220mails wwwjjllueeee. edu cn Xiaof Alaofengcao@jlu edu. cn yu www0mails....mmmmmmmmmmmm...m1 edu cn
yew@tongj1.co edu.
JC- jG-jingeal... guo@polyu edu. hk tsingoguo@leee.ccm org
Abstract
Recent research has underscored the efficacy of Graph Neural Networks (GNNs) in modeling diverse geometric structures within graph data. However, real-world graphs typically exhibit geometrically heterogeneous characteristics, rendering the confinement to a single geometric paradigm insufficient for capturing their intricate structural complexities. To address this limitation, examine the performance of GNNs across various geometries through the lens of knowledge distillation (KD) and introduce novel cross-geometric framework. This framework encodes graphs by integrating both Euclidean and hyperbolic geometries in a space-mixine fashion. Our approach employs multiple teacher models, each generating hint embeddings that encapsulate distinct geometric properties. We then implement structure- wise knowledge transfer module that optimally leverages these embeddings within their respective geometric contexts, thereby enhancing the training efficacy of the stu- dent model. Additionally, our framework incorporates geometric optimization network designed to bridge the distributional disparities among these embeddings. Experimental results demonstrate that our model-aggostic framework more effec- tively captures topological graph knowledge, resulting in superior performance of the student models when compared traditional KD methodologies.
Introduction
Graph Neural Networks (GNNs) have emerged indispensable tools for analyzing relational data in diverse domains, such as natural language processing 21 computer vision [4] 5), recommen- dation systems [6] 7 Their conventional approach of operating within Fuclidean space encounters limitations when confronted with datasets embodying non-Euclidean characteristics, such as power- law distribution and hierarchical structures, prevalent in real-world applications
[8] Recognizing this challenge, our community ventures into the realm of non-Euclidean Graph Neural Networks, seeking harness alternative geometries, notably hyperbolic space, for more adeptly capturing the intricate topological features inherent in many real-world networks [9]. By synthesizing recent advancements and empirical findings, endeavor elucidate the potential of non-Euclidean GNNs in effectively modeling complex relational data structures, thereby paving the way for advancements in various application domains
[10 12
13 14 [5).
Unlike the constant and flat Euclidean geometry, hyperbolic geometry offers greater flexibility by integrating curvature information, enabling better alignment with the characteristics of -unFuucide input graphs. This endeavor has rendered hyperbolic GNNs more accessible and comparable to their Euclidean counterparts, resulting in promising performance and interpretability in graph representa-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tion learning. Hyperbolic GNNs [6 extended the neighborhood aggregation operation by computing centroids in the hyperbolic geometry. This approach effectively fuses the node features and hierarchi- cal structure, thereby learning superior node representations. Furthermore, a full manifold-preservii feature transformation operation has been developed hyperbolic geometry [S], eliminating the complicated transformations between hyperbolic and tangent spaces. With these essential operation, hyperbolic GNNs can achieve comparable or even superior performance than Euclidean GNNs.
Question. Although there has been a surge of research on Euclidean and non-Euclidean GNNs in the community, remains unclear which geometry offers greater advantages. Real-world graphs often exhibit geometrically structural heterogeneity, characterized by variations in clustering and density among nodes (17 [8], as shown Figure(] The structural heterogeneity pose a challenge when attempting to accurately model the graph structure using GNNs solety equipped with either Fuclidean or Ennllidee geometry.
Motivation. According Local Subgraph Preservation Property 19, the properties of a node largely depend on the properties of the local subgraph centered around Considering the Employing hyperbolic geometry
hyperbolic property of local subgraphs, i.e., hyperbolicity | modeling achieves higher precision and minimal information loss when hyperbolicity is low. Conversely, when hyperbolicity is elevated, opting for Euclidean geometry modeling results in lower complexity and slightly superior performance compared to hyperbolic geometry. Conse- quently, the primary limitation of existing graph neural networks is their inability to adaptively select the appropriate geometry for representing nodes with different local structures 21 Our scheme. In this paper, we propose cross- geometric graph knowledge distillation frame- work that encodes graphs utilizing both Eu- clidean and hyperbolic geometry in locally space-mixing fashion. In contrast to traditional methods that compute hyperbolicity for the over- all graph and roughly analyze its applicability to different geometries, our approach performs fine-grat fine-grained analysis on the local subgraphs rounding each node. This enables the selection of embeddings in the most appropriate geometry for local subgraphs, which is subsequently utilized to transfer knowledge to the student model. Ad- ditionally, we introduce a geometric embedding optimization module to optimize the distribution of embeddings produced by the teacher models. To evaluate the performance of our proposed ap- proach, we conduct distillation experiments on node classification (NC) and link prediction (LP) tasks across various types of graph data. The experimental results demonstrate the superiority of our approach in teaching student models com- pared to other baseline methods. Our approach highlights enhanced effectiveness and generaliza- tion, ultimately achieving state-of-the-at perfor- mance in graph data distillation tasks. The salient aspects of our contributions are follows:
Intra-Class Distance
Inter-Class Distance
Figure 1: Visualization of the embedding of the same graph in hyperbolic space (left) and Eu- clidean space (right), with different colors rep- resenting different class labels. Tree-like sub- graphs maintain significant inter-class margins in hyperbolic space, leading to improved classi- fication boundaries. while intra-class nodes with Euclidean properties may overstretched due to the utilization of hyperbolic metrics, hence em- bedding in Euclidean space is preferred.
Structured analysis reveals that both Euclidean and hyperbolic geometries demonstrate commendable performance in graph processing, despite their inherent disparities and po- tential geometric conflicts. This prompts scholarly inquiry into reconciling these divergent geometric spaces within GNNs, inspiring new research avenues in geometry awareness.
With heightened awareness, there's potential to represent graph structures across dimensions, transcending singular space limitations. We thus adapt and integrate teacher embeddings from diverse geometries, transferring them to more effective cross- cross-geometric space.
'Gromov's 9-hyperbolicity 200 (See the appendix A.1 for the calculating process) measures a graph's tree- like structure, with lower values indicating higher hyperbolicity in a graph dataset, where â = 0 represents tree.

---

Extensive experiments employing KD techniques on diverse graph datasets demonstrate that cross-gcometric methods significantly outperform traditional approaches the context of knowledge transfer. This is particularly evident in NC and I.P tasks, thereby affirming the superior efficacy of these methods.
2
Related Work
Graph Neural Networks. GNNs are neural network models that capture interdependencies between nodes by propagating messages among them within a graph. The most representative model is Graph Nonvolutional Network (GCN) [23 24 25], which can be regarded as generalization of convolutional neural networks to graph data. The GraphSAGE (26) employs neighbor sampling strategy to address graph data, enabling information aggregation based on the local neighborhood structure of nodes. The attention- assigning diverse weights to node representations based on the varying features of neighboring nodes.
GNN [27] 28 (9 model employs masked self-atter attf-tntentoon
Notably, constructing GNNs in the hyperbolic space [33 [5 30) significantly reduces embedding distortions caused by the inability of Euclidean space to handle power-law distributions, particularly in the case of tree-like or highly hierarchical data.
Knowledge Distillation. KD initially proposed by [1, is a model compression technique that involves leveraging pre-trained teacher models to guide the training of a lightweight student model After that, 34 aligns student and teacher model intermediate features using a regressor and loss function to minimize feature differences. [35] employs attention mechanisms to extract features from teacher model's 's intermediate layers and transfer them to the student model. As GNNs have demonstrated breakthrough performance in various deep learning tasks, number of graph- based KD frameworks have been proposed successively. [36] introduces a local structure preserving module to extract knowledge from intermediate layers of the GNN model, guiding the student model to optimize learned topological structures. [37 proposes a novel approach to effectively learn multi-scale topological semantics from multiple GNN teacher models to guide student model. (38) incorporates a VQ-VAE to learn a codebook that represents informative local structures, and uses these local structures as additional information for distillation. However, all these methods rely on the Euclidean geometry. Our proposed approach leverages both Euclidean and non- oon-Euclidean geometries to learn representations of highly hierarchical local structures, ensuring that the knowledge transmitted to the student model is highly reliable.
32 33
3 Problem Definition and Preliminaries
3.1 Problem Definition
For the graph KD, given graph G = E), where V denotes the node set and denotes the edge set. Let V denotes the total number of nodes in the graph G, X denotes nodes' feature matrix with each row corresponding the feature vector of node, and A denotes graph's 's N x N adjacency matrix, where Aij signifies whether there is edge between nodes and j. If A,j = 1, an edge exists; otherwise, no edge is present. Let M1 = {mr,, M7,, m&w } denotes the teacher models pre-trained on G, R represents the number of geometries. Our fundamental objective is to extract information from G by Mr, and employing it to boost the training process the student model, denoted as m,, which in Euclidean space and has significantly smaller size. Let 27 (ZT, ZTz: ZTR be the outputs of the teacher models and Zş be the outputs of the student model. The optimization goal to minimize the disparity between predictions of Mr and mg on i.e.
1 N
min
j=1 33 Fais (ZT, 1=2.,),
where By denotes the weight of the j-th teacher model, Fate denotes disparity measurement function.
3.2 Preliminaries
In this paper, We focus on distillation performance of the teacher model individually in Euclidean, hyperbolic, and spherical geometries, as well as across geometries. We give necessary introduction of hyperbolic geometry in this subsection, with other information available in Appendix A
3

---

Hyperbolic geometry studies the properties of curved space with negative curvature. Hyperbolic space can be modeled using five isometric models 39 40 and in this paper, we adopt Poincaré disk model. The Poincaré disk model evinces distinctive property wherein distances from the geometric center to the periphery undergo non-linear augmentation as function of layer depth [ This phenomenon engenders nonlinear and multi-branch composite structure within the model's geometric framework.
Definition 3.1 (Poincaré Disk Model), A n-dimensional Poincaré disk model Riemannian manifold with a negative constant curvature c, which defined as
) is complete
2
B :
(= e e R" -c|=]
= (X
X
1 2
where || denotes the Euclidean norm, g denotes the Riemannian metric, and The superscripts' & and indicate that the vector matrix resides in hyperbolic space and Euclidean space, respectively.
Definition 3.2 (Hyperbolic Operations). Given two points x, y By, the hyperbolic distance between them is defined as
de( y) = Vc
tanh-1 (Vcll -1 De v1l)
where Be denotes Möbius addition, i.e.,
(1 +1+ x. 8) + clyyl x (1 (1- -| + 2c(x,
a Be y
(4)
Definition 3.3 (Tangent Space). The tangent space point in hyperbolic space, denoted as Te serves as the first-order approximation of the original space, a n-dimensional tangent space is isomorphic to Euclidean space R.. Representations between hyperbolic and tangent space can be transformed via the exponential and logarithmic map as follows:
Vc Na 1allls1
T + +B :xxx +c (tanh
2
x + y
B T. log (3) = do (x, y)
x x De yll
where v E ToIB B and X has same meaning in Eq. 2 Here we utilize the origin point o in hyperbolic space reference point equalize errors across various directions.
Cross-Geometry Learning with KD
This section reveals three key aspects: why cross-geometry learning demonstrates superior perfor- mance, why feasible, and how this superiority is achieved by employing KD. Thus, we analyze our method from three perspectives: reasonableness, superiority, and trustworthiness.
4.1 Geometric Features of Local Subgraphs
According local subgraph perservation peoperty theorem [19, nodes near the central node strongly affect its features, while distant nodes typically have negligible impact. The graph data in real-world often exhibits significant complexity, where diverse local subgraphs often entail distinct geometric properties [17 18), employing cross-geometry system can offer more effective embedding selections for local graphs, thereby achieving performance beyond that of single geometry methods.
Reasonableness:
Definition 4.1 (Subgraphs of Centroid p): For given node belonging graph G, its corresponding k-hops subgraph Gp comprises all nodes E VN within a distance no greater than from along with their respective edges.
employ the k-hops neighbors method to generate subgraphs. Hence, we determine the optimal value of & through the statistical analysis of graph data in section 5.4. For each subgraph G., we calculate their Gromov's 's d-hyperbolicity (See the appendix A. for calculating process) based on X. denoted as dg., which serves geometric characterization metrie for the central node i.

---

Input Graph
Teacher
Z
SWKT
max
transfer
Teacher
Student
g
X
soft max
B
x
lug
log
hodes
A
subgrophs SWKT
optimization
cross
transfer
tronsfer
geometry
seft
max
Student
GEO
Figure 2: Illustration of our proposed cross-geometry graph KD framework. Structure- Knowledge Transfer (SWKT); Choosing embeddings in appropriate geometric spaces using óG, of nodes, conveying local subgraph topological knowledge to the student model: z denotes Euclidean teaching, and ( denotes the hyperbolic teaching. GEO: Enhancing hint embeddings from the teacher models, reducing the negative effects of inconsistencies between different geometries.
4.2 Geometric Teacher Models
Superiority: We leverage KD technology, utilizing its ability to transfer knowledge between different model architectures, medium for interoperability between different geometries. Our proposed KD framework is model-agnostic, making applicable to various geometric models.
Herein, the framework will be explained using the GCN model [23]. To minimize disparities between intermediate layers the teacher and student models, our method uses pre-activation node embedding matrix Z for all nodes'
embeddings to guide the student model and constructs x. During the forward propagation process of a GCN layer in Euclidean space, the intermediate embeddings of nodes in the 1-th layer of
GCN is given by
(0.E ÁH 1) w!))
where H (l--)).E activation (
(1-1). denotes the node representation matrix from the output of the (1 1)-th GCN layer. À denotes the symmetrically normalized adjacency matrix, W(0 denotes the weight matrix.
(0). For the i-th node in layer I, its embedding in hyperbolic space is denoted as 27 and its representation [B).B denoted HT, During the forward propagation process of GCN layer in hyperbolic space, the transformed feature is given by
(D.B
expo w loge (hT,, (nT,)))3) C b
By performing neighborhood aggregation these features, obtain the hyperbolic intermediate embeddings of i-th node in the 1-th layer as follows:
.0) T
Σ Datisee Wig log (F(0)P)
= expe m...
(8)
where Why is the weight coefficient computed by (1).B and fTj
(1),
4.3 Structure-Wi Wise Knowledge Transfer
Trustworthiness: To achieve a more fine-grained selection of embeddings appropriate geometry, designed Structure- Knowledge Transfer (SWKT) module. This module determines suitable

---

geometric representations based on the geometric feature ég, of subgraphs and transfers them to the student, facilitating more effective information extraction and guidance.
Specifically, we obtain representations of local subgraphs centered around each node based on the 1-th intermediate layer hint embeddings of the teacher model in different geometries. We denote i-th
local structure representation in hyperbolic geometry as u T [ TT22
(0)3
(1). (0).B ().B
T 1. TAN ).
().. Element WT,ij
in hyperbolic geometry can be computed as follows:
(0.3 "T,3
=Fuub (2) log (27,))
exp [1066 I
(1.B lo (2T,)
Σ
(1.. exp (esp(110s6 lo 1ogo (=Tg
(9)
Similarly, can obtain representation set of i-th local structure in Euclidean geometry denoted as uT.
( SWKT generates induced k-hops subgraphs centered at each node, computes their óg, as the hierarchical level of the central node and obtains teacher models' middle layer representations of the i-th node in the (-th layer based on òg. as follows:
().B U1 L1i I(6g. < + UTS I(6g. 2 A)
(1)
(..E
(10)
where I denotes indicator function, the threshold À is hyperparameter that is typically set to smaller value on graphs with higher j-hyperbolicity values to achieve better performance.
In our method, embeddings of 1-th guided middle layers student model is z and ap-
plying Eq. (9) likewise yields the student model i-th local structure representation us (1) (1)
(13,1 " "S,12>-" ). For node i, the similarity between the local structures of the teacher model
(1)
and the student model is measured as:
(1) p! = DKL U. (IS.ITTs)) =
(1) log
UsSú
(11)
UT
where DKL represents the Kullback-Leible divergence.
SWKT minimizes the local structure similarity to transfer knowledge from hint embeddings in different geometry to student model. Thus, the loss function for the SWKT module is
LSw L -'_,
N (!))
(12)
where L denotes the total number of intermediate layers used for distillation.
4.4 Geometric Embedding Optimization
Simply concatenating embeddings from Euclidean and hyperbolic teacher models to teach student model can lead to confusion due to geometric inconsistencies. This confusion may result in the student model performing worse than when learning from single geometric teacher model, as shown in section 5.2 To mitigate the negative effects caused by this inconsistency, we propose Geometric Embedding Optimization module (GEO) optimize cross-geometric space.
Trustworthines::
Specifically, for a given node i from layer we have its local geometric information òg, and teacher embeddings from different geometric spaces. We obtain an initial fused feature follows:
( T 1 (xppp-(66, -))
1
(O).
exp(- (6g, - A)) 1 exp( - (8g. A))
(U)BB
TT
=
(13)
where À has the same meaning as Eq. (TO..
Next, we single-layer GCN (which can be replaced by other sufficiently capable networks, such as a Multi-Layer Perceptron (MLP)) to optimize the initially fused features e The optimization network should select loss functions based on the specific downstream task. In this study, we adopt the triplet loss function 41 which enlarges the distance between different-class nodes and reduces the

---

distance between same-class nodes, to enhance node classification and link prediction performance. To apply triplet loss to graph data, we organize triplets as follows: Given the hint embeddings from teacher, we sample extensive sets of nodes, where each set includes an anchor node, a positive node with the same label as the anchor, and a negative node with a different label.
Assuming F. is corresponding function of a pre-trained GEO network with weight matrix We, the (1) elements of the local structure vector UT for the i-th node of layer can be represented as
TJ Fubb FFu W.). W.)),
(14)
where Fubb has the same meaning Eq. 9
Subsequently, we can reference Eq. TT to compute the structural similarity between the outputs of GEO and the guided embeddings of 1-th layer of the student model as:
1 N LGEO N
Σ i=1
DKL (u !
(15)
4.5 Distillation Framework
In our proposed graph KD approach, the teacher model's early L - 1 layers guide the student model using the SWKT module, while the L-th layer guides via the GEO module. Additionally, the student model also learns the logits distribution of teacher models, i.e., outputs of GEO in last layer. Given the logits yy from teacher models and the predicted logits y from student model, our overall KD loss is as follows:
L= LsWKT (yr, ys) -BGGEO
(16)
where Lce denotes the cross-entropy loss function, / is weight coefficient.
The space complexity
RNH KN E)), where N is the number of nodes, D is the feature dimension, [E is the number of edges, R is the number of teacher models, and & is the k-pop parameter. For time complexity, forward propagation has complexity of O + E H), local subgraph generation is O(k:(EE) the Structured- Wiss Knowledge Transfer module is O(NN H), similarity computation is (kN), and the Geometric Embedding Optimization module contributes O(N Thus, the overall time complexity is O(N + E + KNN(E) H)).
5 Experiments
In this section, we first give the experimental setup and baselines. Then we compare our graph KD framework with some baselines on NC and LP tasks. Hyperparameters and ablation analysis also be given. Further experimental results can be found in Appendix
5.1 Experimental Settings
Setups. We preform NC and LP tasks on citation network datasets Cora
42
Citteseer
43 and
and Physics part of the Coauthor dataset Co- Co-Phyyics [6). The student and teacher models are both GCN composed of two hidden layers and one output layer. The hidden layer node dimensions are 8 for the student and 128 for teachers. The model parameters are uniformly initialized using the Xavier's uniform initialization [47 method The optimizer uses Adam (48] or Riemannian Adam 49). We set the value of & for k-hops subgraphs to 4. To mitigate errors caused by randomness, each F1-score and ROC AUC is the average of 10 experiments with different random seed values.
Pubmed
4. wikipedia-based article hyperlink network dataset Wiki-CS 45
To evaluate the performance of our method, we compare it with KD methods formulated in single geometry, including the following methods. FitNet [34] utilizes a regressor to align the intermediate features of the student model with those of the teacher model, quantifying the feature discrepancy using L2 distance. AT (35] averages attention maps from both teacher and student models' intermediate hidden layers, quantifying differences between them using a designed loss function. LSP (36) extracts local structures from both teacher and student models' intermediate feature maps and measures their difference using KL divergence. MSKD 37 Utilizes diverse teacher models with varying layers to guide the student model capturing topology at different scales. VQG [38] incorporates VQ-VAE to learn codebook that represents informative local structures, and uses
Baselines.
7

---

Table 1: Fl sc and ROC { of student model distilled by all KD methods for NC and LP tasks. E, B. S respectively denote the method being in Euclidean, hyperbolic, and spherical spaces, with multiple symbols representing cross geometric space. represents the global hyperbolicity.
Wiki-S
Co- Physiis
Pubmed
Cheseer
Cora 11.0
Method
M
NC
LP
NC
LP
NC
LP
NC
LP
NC
LP
8
79.94.906
93.77,911
26.75.018
95.27
82.56
94-91
73.97
98.27
86.981008
92.22,031
Teacher
B
81,83,000
95.11
97.02.019
98.14
86.24.005
94.67a619
81.83
94.34
90.90
91,98a02s
8
81.61
85. 85.30
96.989937
97.74
86.14
94.63
80.37am
94.43
89.43300
92.52g4 15
67.89-440
84.51
15
90.28
80,7
84.86
83.5 83.39
80.32+209
81,50=::4
FiLNet
12
84.31 es
96.49:
90. 10
81 11.94
84.47a. 11.1
34
13.39
83.00=244
8
72.73
62.652) 65
96.67
64
81. kn
78 78
71 19
77.00ga33
83. 83.2888025
72.19
22.80
s4.5
96.48
90.60
81, 20.20
70.24
69.44
82.49
80.49
61.06
AT
21
84.76=24)
96-58-40m
72
81.31
L1
71.08+125
23.25
82.164n02
-3.81=215
70.71 2000
62.72
25
96.07:00
89 25
81,564825
78.73
71.95
76.062231
83.08
72.72e2.10
69.52
84.71 10:0
96.52.00m
$1.73 90.79
87 s2:2
1142
83 83
13.34
83.962::41
LSP
69.52.0070
84.21 2a5s
96.50
90.44:01:
81.72s00s
86.48
71.44
84.17
62.70a1.0
82.21
69.13,,00
63.00
96.27
89.05
14.14
18.861141
11.88.mmm
82.48
12.89.030
X
20
84.81
16.48
82.04
86. 12
71.261006
83.77
12.48.112
84 84.38
MSKD
B
12.56:1115
84.63
16.58
89.70
81.961077
86.25j.m
71.77
85.04
12.16
83.86
8
72.13.20m
62.07.075
96.27.00
19.55g1
82.16
78.60
71.85
76.72
82.8610001
77
46.4000000
89 30
81 19.4
78
70.57.1.
76.07
83.02.1 an
72.72
VQG
+279
69.02
96 96.652015
89 89.05
81. 81,50
80.24
70.92+002
76.40=231
83.21
72. 22.30=2a
72.51
65.75
96 96.642911
88 33
81. 81,50
78. 10
69.62
74.72
83.15
24.84
E.S
20.85 an
61.89
96.07
a0.45
78.89.204
11.98
g2.08
82.89
71.52
Cross
3.5
70.07
62.75=23>
96.17
90.26
82.23
79.74
71.90
74.33
82.74
71.76=24:
5,
68. 68.70
62.51 =235
96.37amm
89 35
81,50an32
77.99
71.77
e7.33
83.194222
m2
Our
E
14.17.100
86.63 4431
96.87
91.88
82.73,0a2
S8L32
72.60
86.37
86.05
86.95
these local structures additional information for distillation. To comprehensively demonstrate the superiority of cross-geometry over individual geometry, we conducted experiments for each method separately in Euclidean space E, hyperbolic space B, and spherical space S. Here, adopt spherical space S with curvature of 1, for further details, please refer to the Appendix[A.4] We further conducted exploratory experiments on alternative geometric integration approaches based on that proposed in section 4, as illustrated by Cross in Table I
5.2 Node Classification
We use Fl scores as the evaluation metric for node classification task and present results in Table In comparing LSPs across different geometries, we found a counterintuitive outcome. In datasets with low d-hyperbolicity, hyperbolic LSP was anticipated to outperform Euclidean LSP. However, performed worse, dropping by 3.05% (Wiki-CS). This suggests that even if teacher model excels in one geometry, its guidance may be less effective when the student model operates in a different geometry. This highlights significant gap between hint embeddings in different geometries. Despite employing diverse geometries, our method obtains 1.11% average improvement over SOTA baselines and especially 2.66% and 1. 1.37% Cora and Wiki-CS, indicating that our method excels on datasets with lower ō 9-hyperbolicity. Besides, even in graph with high 5 d-hyperbolicity, where graph exhibit few hierarchical levels, our method consistently achieves higher Fi-score compared to student models obtained by baseline KD methods in a single geometre space.
With the rapid growth of information, real-world graph data is expanding in scale. To demonstrate the effectiveness of our method on large-scale graphs, we evaluated the Fl scores of NC tasks using distilled student models on larger datasets, Ogbn-Arxer Ogbn- Proteinn (39,561,252 edges, 132,534 nodes) [50]. Results in Table[[]shor show that our method consistently achieves superior distillation performance on these larger datasets.
(1,166,243 edges, 169,343 nodes) and
5.3 Link Prediction
We use ROC AUC as the evaluation metric for link prediction task and present results in Table I The average ROC AUC showed an improvement of 1.58%, with particularly notable enhancements on datasets Ciki-CS and Cora, reaching 1.87% and 2.57%, respectively. Employing KD methods solely based on hyperbolic geometry outperformed those exclusively utilizing Fuclidean geometry, particularly on the Citeseer. Our cross-geometry KD method outperformed SOTA basclines by
8

---

kn2
kM
km10
VGG
Nodes
8000
Student Scores Student Scores
Student F1 Scores Student F1 Scores
4000
0.6
72.5
2000
0.2
70.5
# Nodes
0.8
85.5
0.6
500
85.0
0.2
84.5
Hyperbolicity
Hyperparameter
Epochs
Figure 3: 0g, distribution of subgraphs (left), hyperparameters sensitivity analysis (middle), compari- son of convergence rates (right) the Cora (row I) and Wiki-CS (row 2)
0.93% on average, underscoring the efficacy of geometry-specific methods in cross-geometry learning for overall performance enhancement.
5.4 Hyperparameters Analysis
k of Subgraphs. A small radius l limits local hierarchical assessment, while large k increases computational complexity. We explored 6, value distributions for local geometric properties across l values (1 to 7) on Wiki-CS and Cora datasets, shown in Figure 3 (left). Stability in distributions occurs at A: 2 4, suggesting sufficient capture geometric characteristics in subgraphs this size. Thus, set l 4 for experiments.
varied the hyperparameters on the Wiki-CS and Cora datasets to test the Fl scores for the NC task, with À {0.0, 0.5, 1.0, 1.5, 2.0, 2.5} E 2,, 4, 5, 10). Here, À denotes the threshold value in Eq. (T0) and Eq. (T3). larger leads to more local subgraphs being embedded in hyperbolic space, while smaller À results in more subgraphs being embedded in Euclidean space. denotes the weight of the GEO module. The results from Figure (middle) indicate that the hyperparameters À and have generally minimal impact on the outcomes. The combination of À 1.5 and / 3.0 maintains optimal performance.
5.5 Distillation Efficiency
To evaluate the convergence efficiency of our proposed KD method, we have recorded the F1-scores trends for student models guided by all KD methods during training epochs on the Cora and Wiki CS datasets in Figure (right). As illustrated, our KD method consistently outperform other methods within the same training epochs. Specifically, our KD method achieves state-of-the-arr (SOTA) performance before reaching 500 epochs, while the others are still undergoing training. These results serve to validate the effectiveness and efficiency of our method. Due to their simpler architecture, student models generally have faster inference speeds compared to teacher models. The speed- up achieved by student models relative to teacher models is also an indicator of the efficiency of distillation methods. The inference times (in milliseconds) of both teacher and student models, measured our device, are shown in Table Results demonstrate that our method achieves an average speed-up of approximately 232x.
In addition, we evaluated the training time for each method, the inference time for the corresponding student models, and calculated the compression ratio of student model size relative to the teacher model. The results can be found in Appendix
5.6
Ablation Study
To further validate the efficacy of cross-geometric learning and the two proposed modules, we conducted additional experiments by adapting our method to operate single Euclidean or

---

Table 2: Fl ( student model distilled by all KD methods for NC on Arxiv and Proteins.
Teacher /E
Teacher RR
FitNet/
AT/E
LSP/
MSKD/
VQG /
Our / E,
Arxiv Proteins
71.91 1006
73.21 1019
67,56
11779
67 48 1025
69.53 =0.00
69 1021
68.59 1011
70.89 10.55
72.83 1009
1002
68.71 2181
20:35
-0.33
70.97 1097
69.54 10.36
71.22 2::1
Table Speed-up up comparison.
Table 4: Ablation study results.
Datasets
Teacher
Student
Speed-up
Method
scores (%)
ROC AUC (%)
Wiki-CS
906
3.98 ms
227x
Co-Physics
3410 ms
12.0 ms
284x
w/ Euclidean Teacher
72.84 1 1.66
84.86 1: 1,02
Pubmed
914 ms
4,46 ms
204x
w/ Hyperbolic Teacher
72.38 + 1.83
84.55 = 0.69
Citeseer
908 ms
4.01 ms
226x
w/o SWKT module
73.40 = 1.26
85.08 = 0.55
Cora
975 ms
4.43 ms
220x
w/o GEO module
73.39
+ 1.27
85.49 0.97
Average
1422 ms
4.22 ms
232x
Comprehensive Method
74.17 1 0.50
86,63 1 0.31
hyperbolic geometry. Additionally, we selectively excluded the SWKT and GEO modules. These ablation experiments were conducted on the Wiki-CS dataset, and the results are presented in Table We have the following observations:
Using only single geometry, even with the GEO module optimizing embeddings, the enhancement compared to the baseline is minimal.
The cross-geometric approach consistently outperforms the kingleggeomtric methods. Whether excluding SW or GEO, the results are inferior to the comprehensive method, indicating their crucial roles in optimizing geometric embedding distribution.
The overall results of ablation analysis further demonstrate the importance of cross-geometry learning and our proposed two modules.
To demonstrate the model-agnostic nature of our framework, we alter the architecture and the number of layers L in the teacher model. Due page limitations, only present key results above. For more details on the dataset, experimental setup, and results, please refer to Appendix B C
Conclusion
Hyperbolic geometry has shown expressive non-Euclidean modeling in the graph community. Note- worthy models, such as Poincaré and Lorentz models, facilitate vector projections between Euclidean and hyperbolic neurons. Our investigation reveals that tree-like power-law distributed graphs exhibit multiple different hierarchical within locally connected structures. Consequently, training across Euclidean and hyperbolic geometry intuitively emerges as more flexible approach to graph modeling, yielding significant enhancements in KD tasks. To this end, we introduce a novel KD framework that models the hint embeddings of the teacher models across diverse geometries. By leveraging 6- 6-hyperbolicity, we transfer local subgraphs information to the student model. Our analysis and experiments provide positive support for this innovative perspective on geometry modeling.
Limitations. Despite the performance improvements achieved by cross-geometry learning across various tasks, presents some potential issues. For instance, integrating different geometric informa- tion introduces hyperparameters. making the task outcomes somewhat dependent on their selection, thus affecting the method's stability. Additionally, the distillation phase demands more complex pre-trained models, increasing resource and time requirements. These limitations are critical areas for future enhancement cross-geometry learning.
Acknowledge
This work was supported by National Natural Science Foundation of China, Grant Number: 62476109, 62206108, 62176184, and the Natural Science Foundation of Jilin Province, Grant Number: 20240101373JC, and Jilin Province Budgetary Capital Construction Fund Plan, Grant Number: 2024C008-5, and Research Project of Jilin Provincial Education Department, Grant Number: 12852.
10

---

References
[1] Nikhil Mehta, María Leonor Pacheco, and Dan Goldwasser. Tackling fake news detection by continually improving social context representations using graph neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1363-1380, 2022.
[2] Hongcai Xu, Junpeng Bao, and Wenbo Liu. Double-branch multi-attention based graph neural network for knowledge graph completion. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15257-15271, 2023.
[3] Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classi- fication. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370-7377,22 2019.
[4] Feiyu Chen, Jie Shao, Shuyuan Zhu, and Heng Tao Shen. Multivariate, multi- mulli-ffrquency and multimodal: Rethinking graph neural networks for emotion recognition in conversation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10761-10770, 2023.
[5] Amit Aflalo, Shai Bagon, Tamar Kashti, and Yonina Eldar. Deepeut: Unsupervised segmenta- tion using graph neural networks clustering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 32 41, 2023.
[6] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pages 417-426, 2019.
[7] Mengru Chen, Chao Huang, Lianghao Xia, Wei Wei, Yong Xu, and Ronghua Luo. Heteroge- neous graph contrastive learning for recommendation. In Proceedings of the sixteenth ACM international conference on web search and data mining, pages 344-552, 2023.
[8] Lilapati Waikhom and Ripon Patgiri. survey of graph neural networks in various learning paradigms: methods, applications, and challenges. Artificial Intelligence Review, 56(7):6295- 6364, 2023.
[9] Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyper- bolic deep neural networks: A survey. IEEE Transactions on pattern analysis and machine intelligence, 44(12):10023-1004 2021.
[10] Chih Yao Chen, Tun Min Hung, Yi-Li Hsu, and Lun-Wei Ku. Label-aware hyperbolic embed- dings for fine-gres tne-grained emotion classification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 10947-10958. 2023.
[11] Liping Wang, Fenyu Hu, Shu Wu, and Liang Wang. Fully hyperbolic graph convolution network for recommendation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 3483-3487, 2021.
[12] Jianing Sun, Zhaoyue Cheng, Saba Zuberi, Felipe Pérez, and Maksims Volkovs. Hgef: Hy- perbolic graph convolution networks for collaborative filtering. In Proceedings of the Web Conference 2021, pages 593 601, 2021.
[13] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32, 2019.
[14] Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. Advances in neural information processing systems, 32, 2019.
[15] Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5672-5686, 2022.
[16) Yiding Zhang, Xiao Wang, Chuan Shi, Nian Liu, and Guojie Song. Lorentzian graph convolu- tional networks. In Proceedings of the Web Conference 2021, pages 1249-1261, 2021.
11

---

[17] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-44 422 2017.
[18] Erzsébet Ravasz and Albert- Barabási. Hierarchical organization in complex networks. Physical review E, 67(2):026111 12, 2003.
[19] Kexin Huang and Marinka Zitnik, Graph meta learning via local subgraphs. Advances in neural information processing systems, 33:5862-5874, 2020.
[20] Aaron B Adcock, Blair D Sullivan, and Michael W Mahoney. Tree-like structure in large social and information networks. In 2013 IEEE 13th international conference data mining, pages 1-10. IEEE, 2013.
[21] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
[22] Max Kochurov, Sergey Ivanov, and Eugeny Burnaev. Are hyperbolic representations in graphs created equal? arXiv preprint arXiv:2007.076908 2020.
[23] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2016.
[24] Franco Manessi, Alessandro Rozza, and Mario Manzo. Dynamic graph convolutional networks Pattern Recognition, 97:107000, 2020.
[25] Yiding Yang, Zunlei Feng, Mingli Song. and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems, 33:20286-20296, 2020.
[26] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
[27] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations. 2018.
[28] Huiting Hong, Hantao Guo, Yucheng Lin, Xiaoqing Yang, Zang Li, and Jieping Ye. An attention-bact based graph neural network for heterogencous structural learning. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 4132-4139, 2020.
[29] Yao Ding, Zhili Zhang, Xiaofeng Zhao, Danfeng Hong, Wei Cai, Nengjun Yang, and Bei Wang. Multi-scale receptive fields: Graph attention neural network for hyperspectral image classification. Expert Systems with Applications, 223:1 19858, 2023.
[30] Yiding Zhang, Xiao Wang, Chuan Shi, Xunqiang Jiang, and Yanfang Ye. Hyperbolic graph attention network. IEEE Transactions on Big Data, 166:1169- 2021.
[31] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: I 555.00033..... 2015.
[32] Xiaofeng Cao and Ivor W Tsang. Distribution matching for machine teaching. IEEE Transac- tions on Neural Networks and Learning Systems, 2023.
[33] Chen Zhang, Xiaofeng Cao, Weiyang Liu, Ivor Tsang, and James Kwok. Nonparametric iterative machine teaching. In International Conference on Machine Learning, pages 40851-40870 PMLR, 2023.
[34] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[35] Nikos Komodakis and Sergey Zagoruyko. Paying more attention attention: improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
12

---

[36] Yiding Yang, Jiayan Qiu. Mingli Song. Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 7074-7083, 2020.
[37] Chunhai Zhang, Jie Liu, Kai Dang, and Wenzheng Zhang. Multi-scale distillation from multiple graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4337-4344 2022.
[38] Ling Yang, Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu, Wentao Zhang, Bin CUI, Muhan Zhang, and Jure Leskovec. VQGraph: Rethinking graph representation space for bridg- ing GNNs and MLPs. In The Twelfth International Conference on Learning Representations, 2024.
[39] Eugenio Beltrami. fondamentl degli spazii di curvatura costante memoria. F. Zanetti, 1868.
[40] James W Cannon, William Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, 31(59-115):2, 1997.
[41] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[42] Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127-163,
2000.
[43] Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citeseer: An automatic citation indexing system. In Proceedings of the third ACM conference Digital libraries, pages 89-98, 1998.
[44] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008
[45] Péter Mernyei and Cătälina Cangea. Wiki-es: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901, 2020.
[46] Kuansan Wang. Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1):396-413, 2020.
[47] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 246-256. JMLR Workshop and Conference Proceedings, 2010.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[49] Gary Becigneul and Octavian Eugen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations, 2018.
[50] Weihua Hu, Matthias Fey, Marinka Zitikk, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118 1211332, 2020.
[51] Xiaofeng Cao and Ivor W Tsang. Distribution disagreement via lorentzian focal representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10);6872 6889, 2021.
[52] Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
13

---

(a) c= = 1.0
cb) 0
(c) -0.5
(d) C -1
Figure Spaces with different curvatures. (a) Spherical space with curvature c c= 1.0. (b) Euclidean
space. (c) Hyperbolic space with curvature = 0.5. (d) Hyperbolic space with curvature c = -1,
which have faster grow rate of volume.
A Additional Theoretical Support
A.1 Gromov Hyperbolicity
Gromov's 6-hyperbolicity 20 measures a graph's tree-like structure, with lower d values indicating higher hyperbolicity in graph dataset, where 8 = represents tree. In this paper, we compute the hyperbolicity of the k-hop subgraph Gi for each node as the geometric feature information of the local structure of that node. Here, we provide the detailed calculation process.
First, four nodes a, b, d are randomly sampled from the subgraph Gs. Let S1, S2 and S3 be defined as follows:
S = dist(a, b) + dist(c, d), S2 dist(a, c) + dist(b, d), S3 dist(a, d) + dist c),
(17)
where disl denotes shortest path length between two nodes. Let Mj and M2 be the two largest values among S1, S2 and Sz. We define
hyp(a, b, d) M1 M22-
(18)
The hyperbolicity 8g. of the graph G, is the maximum of hyp over all possible 4-tuples (a, b, c, d) divided by 2, i.e.,
hypla,b,c,d)
8(G)
max a.b.c,d
(19)
In paper, calculate the Jg. for each k-hop subgraphs. For subgraphs with fewer than four nodes, we label their 6g, value as N/A.
A.2 Euclidean Geometry
Geometry is branch of mathematics concerned with properties of space such as the distance, shape, size and relative position of figures. this paper, we analyze the modeling capabilities of graph neural networks in Euclidean. hyperbolic and spherical geometries. Euclidean geometry studies the properties of flat space with zero curvature, where parallel lines never meet, and angles a triangle sum to 180 degrees. In Euclidean geometry, the volume of space exhibits polynomial growth associated with the dimensionality of the space. The majority of neural network models perform inference operations this space, where operations such convolution, pooling, and activation based on the basic arithmetic operations of addition, subtraction, multiplication, and division.
Euclidean geometry is axiomatic system, in which all thcorems derived from small number of simple axioms.
To draw a straight line from any point any point.
To produce (extend) finite straight line continuously in straight line.
14

---

Spherical geometry studies the properties of curved space with constant positive curvature, where the angles of triangle add up to exceeds 180 degrees. All lines in spherical geometry intersect, as there are no parallel lines on a sphere. In the field of graph embedding, spherical geometry plays a significant role as it provides a more realistic model, particularly applicable in geographic information systems and computer graphics. Through spherical geometry, we can accurately describe features on the surface of the Earth and construct data representations with spherical topological structures in To describe circle with any centre and distance (radius).
That all right angles are equal to one another.
[The parallel postulate]: That, if straight line falling on two straight lines make the interior angles on the same side less than two right angles, the two straight lines, if produced indefinitely, meet on that side on which the angles are less than two right angles.
These axioms provide the fundamental mathematical framework for Euclidean space, allowing GNN models to incorporate information about the absolute positions of nodes, properties of lines, and spatial relationships.
A.3 Hyperbolic Geometry
Hyperbolic geometry is -on-Euclidean geometry, also called Jobachevsky-Bolyee eer--BoolyGaauui -aaass geometry. This geometry adheres to all of Euclid's postulates, with the exception of the parallel postulate, which has been substituted with:
If straight line intersects two other straight lines, and makes the two interior angles on one side of it together less than two right angles, then the other straight lines will meet at a point if extended far enough on the side which the angles are less than two right angles.
Hyperbolic space a homogeneous space with constant negative curvature. In Euclidean space, the curvature is zero, while in hyperbolic space, the curvature is negative constant. Moreover, smaller curvature leads to faster volume growth, as illustrated in Figure 4 The hyperbolic space can be modelled using five isomorphic models which are the Lorentz model [51], the Poincaré ball model and Poincaré half space model, and the Klein model. In this paper, we utilize a hyperbolic geometric teacher model based on the Poincaré model. The Poincaré model B manifold equipped with Riemannian metric gd This metric is conformal to the Euclidean metric g" Formally, an n dimensional Poincaré unit ball (manifold) is defined
B (x E R. |||||| < 1},
(20)
where ||
denotes the Euclidean norm. Formally, the distance between x, y E B defined as:
|| yl?
d(x, y) = arcosh( + (1 [1 |1
(21)
The Möbius addition @ for x and in B" is defined as
(1 + 2(x, y) + |1 + (1 || (?) y
I + y
(22)
1 + 2(x, v) + ||
The Möbius scalar multiplication is defined
8 tanh (r (r (rartanh(|=]])
B" x = 0,
(23)
0,
where r is a scalar factor.
The Möbius vector multiplication M (1:) is defined as
||ax|
Mr
M 0 (x)
tanh
actanh((><]) (x:11)
(24)
|
Max
A.4 Spherical Geometry
15

---

Algorithm Cross-Geometrii Graph KD
Input: Graph G {V Pre-trained teacher My and GEO model; Initialization parameters 0 of
student.
Parameter: Threshold X; Weight B.
Output: Distilled model's parameter 6'.
1: while student model not converged do for / {1, 2, ..., L} do
2:
Update ( and ( Z Z Eq. 6 Eq.
3:
4:
Select appropriate geometry for middle representations, thy +- Eq.
10
5:
Calculate structure similarity, p()) t Eq. (II
6:
Reduce geometries" discrepancy, Optimize embeddings by Eq. 14
7:
Lsw #SWKT, KT, LGEO + Eq. 15
8:
end for
9:
Calculate overall Loss by 16)
Update student model's parameter, 166)
11: end while
Table Statistics of datasets.
# Nodes
# Edges
# Features
# Classes
Global Hyperbolicity
Wiki-CS
11,701
431,726
300
10
1.0
Co-Phi Co-Physics
34,493
495,924
8,415
5
2.5
Pubmed 19,717
88,651
500
3.5
Citeseer
3,327
9,928
3,703
6
4.0
Cora 1,044
10,556
1,433
11.0
three-dimensional space, which crucial for applications such as map-making, virtual reality, and computer games. In graph embedding, concepts and algorithms of spherical geometry are utilized to process data with spherical topological structures, such as mapping the Earth's 's surface onto two- dimensional plane while preserving the correctness of geographic locations and spatial relationships. Therefore, spherical geometry is not only a thcoretical discipline but also an indispensable tool in practical applications.
B
Experiment Details
Algorithm
Given the graph data, initially train teacher models in Euclidean, hyperbolic, and spherical spaces, respectively. Subsequently, we train the MLP model of the GEO module, initializing the student model randomly. By inputting learning parameters alongside hyperparameters and 3, we employ Algorithm i to obtain the distilled student model.
B.2 Datasets
Here, we present detailed information for each dataset in Table Wiki-CS consists of 11,701 nodes with 431,726 edges, each node characterized by a 300-dimensional feature, and the node labels are categorized into 10 classes. Cora consists of 1,044 nodes with 10,556 edges, each node characterized by a 1.433-dimensional feature, and the node labels are categorized into 7 classes. Pubmed consists of 19,717 nodes with 88,651 edges, each node characterized by 500-dimmesio feature, and the node labels are categorized into 3 classes. Co-Physics consists of 34.493 nodes with 495, 195,924 edges, each node characterized by 1415-ddmensione feature, and the node labels are categorized into 5 classes. Citeseer consists of 3,327 nodes with 9,928 edges, each node characterized by 3,703-dimensional feature, and the node labels are categorized into 6 classes. To ensure fairness, we uniformly apply standard splits (70% 51155 15%) for node classification tasks and standard splits (85%/5%/10%) for link prediction tasks.
16

---

Table 6: Parameter Settings in NC task.
Parameters
Wiki-CS
Co-Physics
Pubmed
Citeseer
Cora
# layers
2
2
2
teacher hidden dim
128
128
128
128
128
student hidden dim
8
8
16
learning rate
0.01
0.01
0.01
0.05
0.01
weight decay
0.0000
0.0000
0.0005
0.0001
0.0000
dropout
0.00
0.00
0.00
0.00
0.00
A
1.5
1.5
1.5
1.5
1.5
B
3.0
3.0
3.0
3.0
3.0
Table 7: Parameter Settings in LP task.
Parameters
Wiki-CS
Co-Physics
Pubmed
Citeseer
Cora
# layers
2
2
2
teacher hidden dim
128
128
128
128
128
student hidden dim
8
8
8
learning rate
0.01
0.01
0.01
0.01
0.01
weight decay
0.0000
0.0000
0,0000
0.0000
0.0000
dropout
0.00
0.00
0.00
0.00
0.00
r (in fd decoder)
2.00
2.00
2.00
2.00
2.00
(in fd decoder)
1.00
1.00
1.00
1.00
1.00
A
1.5
1.5
1.5
1.5
1.5
B
3.0
3.0
3.0
3.0
3.0
B.3 Setups
For fair comparison, all methods employ identical teacher and student model architectures the same dataset. All methods use GCN as the Euclidean teacher model and HGCN as the hyperbolic teacher model. The teacher models consist of two hidden layers and one output layer, with a hidden feature dimension of 128. The student GCN model has two hidden layers and one output layer, with hidden dimension of During training, the optimizer uses Adam Riemannian Adam, and hyperparameters such as learning rate, weight decay, and hierarchy threshold are fine-tuned based on the performance of student models on validation sets of different datasets, maintaining consistent hyperparameters for different methods on the same dataset. The parameter configurations for NC are detailed in Table 6 while those for LP are delineated in Table 7 The model parameters are uniformly initialized using the Xavier's uniform initialization method, with random seed chosen 1000. The geo model is trained for 300 epochs, and random sampling during
from the range its optimization process involves extracting 100 sets of node pairs for each class.
Environments. The running environment includes an Intel Core Intel i7- 17-13700KF CPU with a clock speed of 3.40GHz, boasting 16 cores and 24 threads. A robust NVIDIA GeForce RTX 4070Ti GPU, featuring 12GB of VRAM, encompasses 7680 CUDA cores. The system is equipped with 16GB of RAM. The operating system is Windows 11, and Python 3. 3.10 serves as the programming language. For deep learning tasks, PyTorch version 11 is employed, while CUDA version 12.2 enhances GPU acceleration. Package management is facilitated through the use of Anaconda. For large datasets, Pubmed and CoauthorPhysics. experiments were conducted on a high-performance server with the following specifications: Intel Xeon Gold 5220 CPUs running at 2.20GHz, equipped with 72 cores and 144 threads. The system features 4 Quadro RTX 6000 GPUs, each boasting 24GB VRAM and 4608 CUDA cores. The system boasts 500GB of RAM and runs Ubuntu 18.04.6.
17

---

Table 8: Fl cores (%)f of student models distilled from GAT teacher models on the NC Task.
Wiki-CS
Co-Physics == 2.5
Pubmed
Citeseer
Cora
Method
M
8 = 1.0
3.5
6= 4.0
8= 11.0
80.52:t 0.23
96.86 0.17
84.78 0.11
75.24 0.18
90.07 0.09
Teacher
82,46 0.27
97.13 0.23
87.35 0.16
82,99 0.15
90.661 0.05
S
81.96 0.18
96.79 0.14
87.31 0.08
81.97 0.23
89.97 0.13
E,S
71.75 + 0.12
96.13 0.01
80.89 + 0.55
72.11 0.21
83.35 0.16
Cross
IB,3
71.27 + 0.51
96.27 + 0.07
82.25 + 0.36
72.13 + 0.15
83.74 + 0.37
E,B,
69.12 0.37
96.21 0.02
82.51 0.42
72.25 0.28
83.27 0.25
Our
E,B
74.52
0.79
97.01 0.04
83.46 = 0.56
72.89 0.08
86.75 0.48
Table 9: Fl Scores (%)t of student models distilled from GTN teacher models on the NC Task.
Teacher
GTN Student
GCN Student
E
B
Fl Scores
Inference Time
Fl Scores
Inference Time
Wiki-CS
82.74
81.83
82.48
15.34 ms
74.26
3.98 ms
Cora
87.87
90.90
86.37
17.67 ms
86.24
4,43 ms
C More Experiment Results and Analysis
C.1 Replacing Teacher Models
Our proposed framework is model-agnostic. To validate its universality and effectiveness, conducted experiments by replacing the teacher model from GCN to GAT. These experiments were conducted across three geometries: Euclidean, hyperbolic, and spherical, for cross-geometry learning. The experimental results are presented in Table As depicted in the table, even when the teacher model is replaced with other models, our framework consistently maintains strong distillation effect, with the combination of hyperbolic and Euclidean geometries still proving to be optimal. Moreover, the performance of the teacher model improves, there is a corresponding enhancement in the performance of the student model.
We also replaced the Euclidean teacher model with the Graph Transformer Network [52]. The results are shown Table 9 GTN teacher has 4 layers and hidden dimension of 128. Specifically, during distillation, student model's layers match the last layers of teacher accordingly. The GTN and HGCN teacher output intermediate representations from each layer to the SWKT module for local subgraph structure extracting and selection. These distributions then optimized by GEO module. Then, these features extracted from the optimized cross-geometric intermediate representations are transferred to students via the corresponding loss function. Additionally, traditional KD loss is computed from the logits output by both the teacher and student models.
C.2 Replacing Student Models
The student model can operate in other geometric spaces. At first, we chose a Fuclidean student model to combine hyperbolic accuracy benefits with Euclidean efficiency and stability. Our framework is model-agnostie, allowing replacement of the student model with other neural networks. To validate student on various geometric spaces, we tested NC Fl scores (%) on the Cora dataset Table I0 Euclidean and hyperbolic teachers' Fl score is 86.98% and 90.90%.
we conducted experiments using student models with the same architecture as the teacher models in our method. Following results are NC F1 scores (%) on the Cora dataset. Euclidean and hyperbolic teachers' Fl score 86.98% and 90.90%. The results are shown presented in Table 1 of our paper, student models now even outperform some teacher models, but using the same architecture as the teacher makes student models larger and slower, limiting their suitability for resource-constrainge devices.
[I
the results
18

---

Table 10: Fl cores (%)f of student models in different geometry on the NC Task.
FitNet
AT
LSP
MSKD
VQG
Our
Euclidean Studnet
80.32
80.49
83.34
82.48
83.02
86.05
Hyperbolic Student
86.73
86.00
87.96
88.21
88.45
90.42
Spherical Student
75.92
83.54
84.77
85.26
83.54
86.73
Average
80.99
83.34
85.35
85.31
85.00
87.73
Table 11: F1 Scores (%)f student models with the same architecture the teacher models on the NC Task.
FitNet
AT
LSP
MSKD
VQG
Our
Student in paper
80.32
80.49
83.34
82.48
83.02
86.02
Euclidean student
86.73
86.24
86.98
86.98
86.49
87.47
Hyperbolic student
N/A
N/A
N/A
N/A
N/A
90.42
C.3 Changing Teacher Layers
In addition to model-agnostic features, our framework demonstrates excellent scalability. All ex- periments in this study were conducted with both teacher and student models having a hidden layer depth of 2. To verify scalability, we configured four types of teacher models, varying their hidden layer depths from to 4, while keeping all other settings constant. By applying the SWKT and GEO modules to each layer, expanded our framework, as illustrated in Table T2] As observed in the table, despite variations the number layers achieves effective distillation results, with the combination of hyperbolic and Euclidean components remaining optimal. Furthermore, as the performance the teacher models improves, there is corresponding enhancement in the performance of the student models.
teacher models, our framework consistently
C.4 Embeddings Optimization Results
We reduced the embeddings of student models 2-dimensional space by t-SNE and visualized them in Figure 3 our method yields a superior embedding distribution, which more suitable for NC task.
C.5 Hyperparameters Analysis
We conducted more comprehensive hyperparameter analysis on the Co-Physics, Pubmed, and CiteSeer datasets. By adjusting the hyperparameters, we evaluated the Fl scores for the NC task, with À E 0.0, 0.5, 1.0, 1.5, 2.0, 2.5 and | E 1,2,3, 3, 4,5, 10. The results, as shown in Figure 6 indicate that the hyperparameters À and / have minimal overall impact the outcomes. The performance generally optimal when 3 3, and shows better performance at intermediate values.
C.6 Ablation Study
Following the ablation experiment strategy outlined in Section 5.2, conducted NC experiments on five other datasets. The results, presented in Table 13 reveal that the performance of the comprehen- sive method consistently outperforms other conditions across datasets.
Table 14: Training time spent per epoch (in ms) for NC task.
Datasets
AT
FitNet
LSP
MSKD
VQG
Our
Wiki-CS
5.25
5.50
5.18
5.27
5.26
5.16
Cora
5.22
6.38
6.09
6.03
6.01
5.81
Pubmed
5.43
6.16
5.94
6.01
5.21
6.31
Citeseer
5.39
6,89
5.78
5.78
5.56
5.95
Co Co-Physics
10.5
11.2
10.5
10.5
10.5
10.3
19

---

Table 12: F1 cores (%)} of student models distilled from teacher models with different layers on the
NC Task.
Method
M
X
x 2
x 3
X 4
E
68.42 + 0.12
79.94 + 0.16
81.22 + 0.38
78.12 + 0.35
Teacher
BB
70.04 0.35
81.83 + 0.09
82.73 + 0.42
80.45 0.18
$
70.15 + 0.28
81.61 + 0.60
82.51 + 0.37
79.86 + 0.33
E,
51.42 + 0.79
70.85 + 0.51
71.32 + 0.76
69.21 + 1.18
Cross
I8,8
52.43 + 0.52
70.07 = 0.67
70.72 + 1.35
69.89 + 1.21
E,B,
52.72 + 1.15
68.70 = 0.14
69.28
= 1.17
68.35
= 0.48
Our
IB,IB
58.12 + 2.38
74.17 + 0.50
74.22 + 1.24
73.75 + 1.18
(a) FitNet
(b) AT
(c) LSP
(d) MSKD
(e) VQG
(l) Our
Figure 5: t-SNE Visualization of embeddings obtained by student models. In contrast to baselines,our method achieves embeddings that fully utilize the entire space, ensuring substantial inter-class distances and thereby enhancing node classification performance.
C.7 Distillation Efficiency
In dataset Wiki-CS, we assessed the training and inference time per epoch and the ratio of the total parameter count the student model to that the teacher model, as outlined in Table [5] Our KD method achieves similar time efficiency in both training and inference stages compared to other methods. Notably, our method achieves superior results at the highest compression level, thereby further validating the efficacy of our KD method in generating compact yet fighh-erforming student models.
We provide an analysis of the time spent by each knowledge distillation (KD) method during the training of student models on the network classification (NC) task, recorded for every epoch across all datasets, as shown in Table [6] We also present the time taken for inference of the student models on the NC task in Table From the results, it clear that in various scenarios, time required for our method is comparable to that of other methods, with no significant increase in time cost. This suggests that our approach effectively balances performance and computational efficiency, making practical choice for applications in this field.
20

---

Student Fl Scores
Student F1 Scores
Student
12.5
$-22
24
Hyperparameter
Hyperparameter
Hyperparameter
Figure 6: Hyperparameters sensitivity analysis
Co Co-Physics (left), Pubmed (middle) and
Citeseer (right)
Table 13: Ablation experiments for NC task across all datasets, evaluated using FI score %
Method
Wiki-CS
Cora
Pubmed
Co-Physics
Citeseer
w/ Euclidean Teacher
72.84
84.55 0.73
81.85 + 0.26
96.50
71.16 1.13
w/ Hyperbolic Teacher
72.38
84.43
81.55 + 1.71
71.30 1.64
W/o SWKT module
73.40
84.16 0.89
82.12 + 0.41
96.68
70.72 2.62
w/o module
73.39 1.27
84.33 + 0.73
82.26 = 0.35
96.65
71.24 1.46
Comprchensive Method
74.17
0.50
84.84 = 0.60
82.61 + 0.23
96.87 0.22
72.60 = 0.84
C.8 Failed Teachers
The failure one or more teacher models could potentially impact the student model's performance, have implemented several mechanisms in our method to mitigate this risk:
Ensemble Learning: Using multiple teacher models that capture different geometric properties provides redundancy and robustness. If one model fails, the others still contribute valuable insights, minimizing the impact on the student model.
Geometric Optimization Network: dynamically adjusts the weight of information from each teacher model based on the loss function, reducing the influence of any underper- forming model and ensuring the student model receives the most reliable information.
We designed various experimental strategies assess the impact of failing teachers on students:
S1: Train student models without KD.
S2: Train student models with the best-tuned teacher model.
S3: Train student models with underperforming teacher model.
S4: Train student models with an untrained teacher model.
S5: Train student models with all untrained teacher models.
Note: All methods except MSKD and ours use a single teacher model; thus, is N/A.
Table 15:
Time spent per epoch and compression ratio.
Training (ms)
Inference (ms)
Method
Ratio (%)
NC
LP
NC
LP
FitNet
5.50
305.7
3.98
22.92
4.47
AT
5.25
314.9
3.98
22.62
4.56
LSP
5.18
318.8
3.98
22.92
4.56
MSKD
5.27
311.4
3.98
23.94
2.67
VQG
5.23
312.5
3.98
22.60
4.47
Our
5.16
305.1
3.98
23,03
2.28
21

---

Table 16: Inference time spent (in ms) for NC task.
Datasets
AT
FitNet
LSP
MSKD
VQG
Our
Wiki-CS
3.98
3.98
3.98
3.98
3.98
3.98
Cora
4.43
4.43
4.43
4.43
4.43
4.43
Pubmed
4.46
4.46
4.46
4.46
4.46
4.46
Citeseer
4.01
4.01
4.01
4.01
4.01
4.01
Co-Physics
12.0
12.0
12.0
12.0
12.0
12.0
Table 17: Fl sc)res(%)) of student model distilled by all KD methods for NC under failed teachers.
Methods
S1
S2
S3
S4
S5
Std(S2-5)
Te)cher(E))
N/A
86.98
66.09
36.61
25.55
24.21
Teacher(IB)
N/A
90.90
64.86
52.58
30.13
21.94
FitNet
82.06
80.32
56.27
53.81
N/A
11.95
AT
82.06
80.49
63.39
51.11
N/A
12.04
LSP
82.06
83.34
71.33
56.57
N/A
10.86
MSKD
82.06
82.48
81.82
78.62
21.62
1.68
VQG
82.06
83.02
70.02
56.02
N/A
11.02
Our
82.06
86.05
85.26
81.33
22.85
2.06
NC fl score (%) of student models on Cora under different experimental strategies are shown in T7 Except S5, which teachers have average performance of only 30%, method's 's distilled student models consistently maintain stable performance even when some teacher models fail.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have concisely written our motivation, contributions, and experimental results in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the final section, we have analyzed the drawbacks of our method and proposed the existing limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification:
Our paper does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided sufficient experimental details in the paper and appendix, and we believe that the experiments in the paper can be replicated.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We submitted our code and datasets, and provided replication guide.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ huapls..e/ppblli/guuudeeee/// CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed explanations of the experimental setup. dataset partitioning, and hyperparameter sensitivity analysis in the core of paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reduced the errors through repeated experiments, and used standard deviation to represent the error range.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the details of the experimental environment in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have confirmed that our code does not violate the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work can reduce resource consumption and have positive impact on society, without any negative effects.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have provided proper citations and acknowledgments for all the resources we have utilized.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work did not create any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not include any crowdsourcing experiments research involv- ing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "YnFiryr1f0", "reviewer": "Reviewer_ogwd", "summary": "This paper presents a cross-geometric graph knowledge distillation method for graph neural networks. This method employs multiple teacher models, each generating different embeddings with distinct geometric properties, such as Euclidean, hyperbolic, and spherical spaces. The student model is based on Euclidean space. Two modules, Structure-Wise Knowledge Transfer (SWKT) and Geometric Embedding Optimization (GEO), are proposed to enhance performance. To evaluate the proposed approach, distillation experiments are conducted on node classification (NC) and link prediction (LP) tasks across various types of graph data.", "questions": "1. If one or more teacher models fail, how does this impact the performance of the student model?\n2. Can the student model operate in other geometric spaces, such as hyperbolic space?\n3. Can the proposed framework work with other architectures, such as Graph Transformer Networks [1]?\n4. If the student model uses the same architecture as the teacher models, how does it perform?\n\n[1] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, Hyunwoo J. Kim:\nGraph Transformer Networks. NeurIPS 2019", "limitations": "This work has discussed the potential limitations in their paper:", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work presents a novel cross-geometric knowledge distillation framework. Distilling knowledge from Euclidean and hyperbolic geometries in a space-mixing fashion is a new approach for me. \n2. A fine-grained analysis of the subgraphs is provided. \n3. Experiments show better results compared to previous knowledge distillation methods.", "weaknesses": "1. In the presented cross-geometric knowledge distillation framework, the student model learns from multiple teacher models with different geometric information. Properly training all these teacher models is challenging. It is unclear how to ensure the student model's reliability if some teacher models fail.\n \n2. The student model operates only in Euclidean space, raising questions about whether other student models, such as those in hyperbolic space, can achieve similar results. \n\n3. All experiments are conducted solely on GCN, with no results provided for other graph architectures. \n\n4. It is unclear if a student model using the same architecture as the teacher models, within the cross-geometric knowledge distillation framework, can achieve better performance than the teacher models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort in reviewing our paper. We hope that our response can resolve your concerns. Please feel free to ask any follow-up questions.\n\n---\n# W1 & Q1:\n\nIt is true that the failure of one or more teacher models could potentially impact the student model’s performance, we have implemented several mechanisms in our method to mitigate this risk:\n- **Ensemble Learning:** Using multiple teacher models that capture different geometric properties provides redundancy and robustness. If one model fails, the others still contribute valuable insights, minimizing the impact on the student model.\n- **Geometric Optimization Network:** GEO dynamically adjusts the weight of information from each teacher model based on the loss function, reducing the influence of any underperforming model and ensuring the student model receives the most reliable information.\n\n**Additional experiments**：\nWe designed various experimental strategies to assess the impact of failing teachers on students:\n\nS1: Train student models without KD.  \nS2: Train student models with the best-tuned teacher model.  \nS3: Train student models with an underperforming teacher model.  \nS4: Train student models with an untrained teacher model.  \nS5: Train student models with all untrained teacher models.  \nNote: All methods except MSKD and ours use a single teacher model,so their S5 f1 scores are N/A.\n\nTeachers' F1 score (%) of different experimental strategies:\n\n|                    | S2    | S3    | S4    | S5    |\n| ------------------ | ----- | ----- | ----- | ----- |\n| Euclidean Teacher  | 86.98 | 66.09 | 36.61 | 25.55 |\n| Hyperbolic Teacher | 90.90 | 64.86 | 52.58 | 30.12 |\n\nNC f1 score (%) of student models on Cora under different experimental strategies:\n\n| Methods | S1↑   | S2↑   | S3↑   | S4↑   | S5↑   | Std(S2-S4) |\n| ------- | ----- | ----- | ----- | ----- | ----- | ---------- |\n| FitNet  | 82.06 | 80.32 | 56.27 | 53.81 | N/A   | 11.95      |\n| AT      | 82.06 | 80.49 | 63.39 | 51.11 | N/A   | 12.04      |\n| LSP     | 82.06 | 83.34 | 71.33 | 56.76 | N/A   | 10.86      |\n| MSKD    | 82.06 | 82.48 | 81.82 | 78.62 | 21.62 | 1.68       |\n| VQG     | 82.06 | 83.02 | 70.02 | 56.02 | N/A   | 11.02      |\n| Our     | 82.06 | 86.05 | 85.26 | 81.33 | 22.85 | 2.06       |\n\nExcept S5, which teachers have an average performance of only 30%, our method’s distilled student models consistently maintain stable performance even when some teacher models fail. \n\n---\n# W2 & Q2:\n\nYes, the student model can operate in other geometric spaces. At first, we chose a Euclidean student model to combine hyperbolic accuracy benefits with Euclidean efficiency and stability. Our framework is model-agnostic, allowing replacement of the student model with other neural networks. \n\nTo validate student on various geometric spaces, we tested NC F1 scores (%) on the Cora dataset. Euclidean and hyperbolic teachers' F1 score is 86.98% and 90.90%.\n\n|                    | FitNet | AT    | LSP   | MSKD  | VQG   | Our       |\n| ------------------ | ------ | ----- | ----- | ----- | ----- | --------- |\n| Euclidean Student  | 80.32  | 80.49 | 83.34 | 82.48 | 83.02 | 86.05     |\n| Hyperbolic Student | 86.73  | 86.00 | 87.96 | 88.21 | 88.45 | 90.42     |\n| Spherical Student  | 75.92  | 83.54 | 84.77 | 85.26 | 83.54 | 86.73     |\n| Average            | 80.99  | 83.34 | 85.35 | 85.31 | 85.00 | **87.73** |\n\n---\n# W3 & Q3:\n\nOur framework is model-agnostic and works with various model architectures. \nIn fact, we have conducted experiments with alternative architectures. We apologize if our results for replacing GCN with GAT, originally presented in Table 7 in the appendix due to space constraints, escaped the reviewer's attention.\n\nWe also followed your suggestion and replaced the Euclidean teacher model with the Graph Transformer Network (GTN)[1]. \n\n|        | GTN Teacher | Hyperbolic Teacher | GTN Student | GCN Student | GTN Student Inference time | GCN Student Inference time |\n| ------ | ----------- | ------------------ | ----------- | ----------- | -------------------------- | -------------------------- |\n| WikiCS | 82.74       | 81.83              | 82.48       | 74.26       | 15.34ms                    | 3.98ms                     |\n| Cora   | 87.87       | 90.90              | 86.37       | 86.24       | 17.67ms                    | 4.43ms                     |\n\nGTN teacher has 4 layers and a hidden dimension of 128. Specifically, during distillation, student model's $l$ layers match the last $l$ layers of teacher accordingly. The GTN and HGCN teacher output intermediate representations from each layer to the SWKT module for local subgraph structure extracting and selection. These distributions are then optimized by GEO module. Then, these features extracted from the optimized cross-geometric intermediate representations are transferred to students via the corresponding loss function. Additionally, traditional KD loss is computed from the logits output by both the teacher and student models.\n\n---\n# W4 & Q4:\n\nAs your suggestion, we conducted experiments using student models with the same architecture as the teacher models in our method. Following results are NC F1 scores (%) on the Cora dataset. Euclidean and hyperbolic teachers' F1 score is 86.98% and 90.90%.\n\n|                    | FitNet | AT    | LSP   | MSKD  | VQG   | Our   |\n| ------------------ | ------ | ----- | ----- | ----- | ----- | ----- |\n| Student in paper   | 80.32  | 80.49 | 83.34 | 82.48 | 83.02 | 86.05 |\n| Euclidean student  | 86.73  | 86.24 | 86.98 | 86.98 | 86.49 | 87.47 |\n| hyperbolic student | N/A    | N/A   | N/A   | N/A   | N/A   | 90.42 |\n\nCompared to the results presented in Table 1 of our paper, student models now even outperform some teacher models, but using the same architecture as the teacher makes student models larger and slower, limiting their suitability for resource-constrained devices.\n\n---\nReferences:\n\n[1] Graph Transformer Networks. NeurIPS'19", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1qto2dYVKS", "reviewer": "Reviewer_Z3B3", "summary": "This paper proposes a novel methodology to distill graph neural network (GNN) integrating the various geometries (e.g. Euclidean and hyperbolic). The author first utilizes a structure-wise knowledge transfer module with multiple teacher models from distinct geometric properties.  Then, the authors demonstrate a geometric embedding optimization to guide a student model optimizing cross-geometric space. The authors show the efficacy of their approach through experiments on node classification and link prediction with various datasets compared with existing state-of-the-art methods.", "questions": "1. Are Table 1 results all reproduced by the authors? If so, the reported performance of other baselines seems significantly lower than the numbers reported in other papers. For example, MSKD (Zhang et al.) reports an 89.67 F1-Score with an 8.2 ratio on the CiteSeer node classification task. On the other hand, the MSKD NC F1-Score in Table 1 is only 71.42.\n2. Following up with 1, GAT results in Table 7 on CiteSeer are significantly lower than the MSKD paper reported. For example, the teacher model trained on CiteSeer with Euclidean geometries only achieves an F1-score of 75.24, while the MSKD paper reports their student model achieves a 95.47 F1-Score with a 17.6 ratio! \n3. Can you do the hyperparameter tuning on the training such that the teacher's performance on Citeseer is close to what MSKD (Zhang et al.) has reported? In distillation literature, experimenting with the very well-trained teacher model is important. If not, the gain from the student model could have been illusory since the performance gap could have been larger if the teacher had trained with better hyperparameters. \n4. VQG authors (Yang et al.) claim that VQGraph has a significant advantage in the inference speed (828x compared to GNN), but the Table 2 inference speeds report all the same inference speeds on FitNet, AT, LSP, MSK, VQG, and the author's method on NC. I am skeptical whether these are the right numbers.  \n5. Also, VQG may not be the right candidate to compare since VQG tackles a different problem, a GNN to MLP distillation.", "limitations": "Overall, the authors' experiments show their new contributions can improve the GNN models in an agnostic way. However, some numerical performance results from the authors are too low (including both teacher and student) compared to the other papers reported. After the authors match their teacher's performance to the other papers (e.g. CiteSeer from MSKD by Zhang et al.) and observe good student performance with a small gap to the teacher + state-of-the-art results, I am happy to raise the score.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written.\n2. The authors have extensive experiments including the comparison to the state-of-the-art and ablation studies on their new contributing components.", "weaknesses": "1. Some questions exist about the authors' empirical results. Please see the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort in reviewing our paper. We take all comments seriously and try our best to address every raised concerns. Please feel free to ask any follow-up questions.\n\n---\n# Q1:\n\nYes, our experimental settings are strictly following the original papers. We used same pre-trained teachers, which were the best-performing models obtained by extensive hyperparameter tuning. \n\nMSKD[1] is an amazing baseline method that introduces multi-scale teachers, providing more comprehensive information to enhance distillation efficiency. Regarding the impressive performance of the teacher model in MSKD  (GCN 89.67%, GAT 95.47%) on Citeseer, despite extensive efforts, we were unable to make our GCN teacher has the similar performance. Since the authors did not release Citeseer dataset used, the significant performance differences may be due to variations in dataset preprocessing or dataset versions.\n\nWe believe that we have obtained a well-trained and effective GCN teacher model on Citeseer, since our GCN teacher have a comparable performance (73.97%) with some latest works which use Citeseer. VQG (one of our baseline method, ICLR'24) [2] shows that the GCN teacher model of performs at 70.49% on Citeseer. Additionally, reported F1 scores for node classification with GCN on Citeseer are 71.0% in [3] (NIPS’23) , 72.9% in [4] (NIPS’23), and 73.18% in [5] (WWW’23). \n\n---\n# Q2:\n\nWe also believe that we have obtained a well-trained and effective GAT teacher on Citeseer. Reported F1 scores with GAT on Citeseer are 73.00% in [4] (NIPS’23) ,76.63% in [5] (WWW’23) . These results are comparable to our GAT teacher's performance (75.24%)\n\n---\n# Q3：\n\nUnfortunately, due to a lack of detailed experimental information and specific dataset versions, we could not replicate MSKD teachers' performance on Citeseer. \n\nWe agree that using a well-trained teacher model is very important in KD. Therefore, we conducted extensive hyperparameter tuning to ensure our teacher models achieved near-optimal performance on each dataset, comparable to the best results reported in latest other graph-related studies [2][3][4][5].\n\nMoreover, all methods employed the same student architecture. Thus, the concern that insufficient training of the teacher model may result in illusory performance of the student model should not appear in our experiments.\n\n---\n# Q4:\n\nAll runtime data were recorded using Python's datetime library.\nIn our experiments, we replaced the student model of VQG with GCN to maintain consistency with other baseline methods  This adjustment is likely the primary reason why all students, including in VQG, exhibit similar inference times. \n\nWe analyzed the speed-up of the student compared to their teachers in our method:\n\n**Theoretically**：\n\nGCN layer's time complexity is: $O(\\sum^L_{l=1}(|E|H + NH^2))$\n\nwhere $N, |E|, L$ denote the number of nodes, edges, and layers, respectively, $H$ denote the hidden layer dimension. The teacher and student models have hidden dimension of 128 and 8, respectively.\n\nAssuming the graph is sparse, which means $|E| \\approx O(N)$. the student model is theoretically **229x** faster than the teacher model about.\n\n**Empirically**:\n\nThe inference times (ms) of the teacher and student models measured by  datetime library are shown below:\n\n|          | Wiki-CS | Co-Physics | Pubmed | Citeseer | Cora | Average |\n| -------- | ------- | ---------- | ------ | -------- | ---- | ------- |\n| Teacher  | 906     | 3410       | 914    | 908      | 975  | 1422    |\n| Student  | 3.98    | 12.0       | 4.46   | 4.01     | 4.43 | 4.22    |\n| Speed-up | 227x    | 284x       | 204x   | 226x     | 220x | 232x    |\n\nOur method achieves a notable acceleration with **an average speedup of 232×**. While the different student architectures result in lower speedup compared to VQG, our method's student  achieves a high compression rate (~2%, see Table 2), whereas VQG's MLP students and teachers are similar in size (see appendix in [2]).\nWe chose GNN as the student model due to considerations of the trade-off between performance and inference speed. Please refer our responses to **Reviewer GamE's W3 and Q1** for more details.\n\n---\n# Q5:\n\nYes, VQG[6] was originally proposed to address the GNN-to-MLP conversion. In our experiments, we replaced the student model of VQG with GCN to maintain consistency with other baseline method. \n\nWe use GCN as the teacher model and replaced the student model with MLP same as VQG's student for a comparison with VQG and another MLP-based graph KD method GLNN[7]. We keep same settings with [6], and the results shown in the following table demonstrate that our method achieves effective distillation even with MLP-based student networks.\n\n|          | Citeseer  | Pubmed    | Cora      |\n| -------- | --------- | --------- | --------- |\n| **GLNN** | 69.23     | 74.70     | 78.97     |\n| **VQG**  | 74.59     | 77.13     | **81.15** |\n| **Our**  | **76.00** | **77.49** | 81.08     |\n\n(part of results from [6])\n\n---\nIn the responses, we explained the reasons why our teacher models are being well-trained. Besides, all methods used same settings and pre-trained teacher models, with results averaged over 10 trials. Based on these solid results, our method achieved sota performance, we hope this can change your opinion.\n\n---\nReferences:\n\n[1] Multi-Scale Distillation from Multiple Graph Neural Networks , AAAI’22\n\n[2] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs, ICLR’24\n\n[3] On Class Distributions Induced by Nearest Neighbor Graphs for Node Classification of Tabular Data, NIPS’23\n\n[4] Re-Think and Re-Design Graph Neural Networks in Spaces of Continuous Graph Diffusion Functionals, NIPS’23\n\n[5] GIF: A General Graph Unlearning Strategy via Influence Function, WWW’23\n\n[6] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs, ICLR’24\n\n[7] Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation, ICLR’22", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rp4r5o0PLW", "reviewer": "Reviewer_GamE", "summary": "This paper introduces a new graph distillation framework using special teacher networks to consider Euclidean and hyperbolic geometries when performing the distillation to the light-weight student GNN model.\nKey techniques include Structure-Wise Knowledge Transfer (SWKT) for selecting appropriate geometric spaces and Geometric Embedding Optimization (GEO) for feature fusion across geometries.", "questions": "1. I am curious that the input of the student model is full graphs or sampled subgraphs? How does this differ from student networks using MLPs [1].\n\n2. Are there any baseline methods that do not incorporate various manifolds to compare with to show the effectiveness of manifold-aware approaches?\n\n\n[1] Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It is intriguing to consider various geometrics in distillation.\n\n2. Experimental results demonstrate effective performance on small graph node classification datasets.\n\n3. The sensitivity analysis of hyperparameters, including thresholds like $\\delta$ and $\\lambda$, is insightful. It will be great if the sensitivity analysis of the weight coefficient $\\beta$ could be further considered.", "weaknesses": "1. The paper lacks an efficiency study comparing the computational efficiency between the teacher and student models.\n\n2. Larger dataset should be included to better evaluate the proposed distillation methods such as ogbn-arxiv and ogbn-product\n\n3. The motivation of choosing GNN as the student network is not sufficiently justified. \nWhile previous works focus on focus on MLP-based students, more clarity on this choice would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort in reviewing our paper. We hope that our response can resolve your concerns. Please feel free to ask any follow-up questions.\n\n---\n# S3:\n\nIn fact, we provided sensitivity analysis of the weight coefficient $\\beta$ in Figure 3 (middle).  We performed a joint analysis of the hyperparameters $\\lambda$ and $\\beta$ to evaluate their combined effects on our method.  Lines with different markers denote different $\\beta$ amd the best value of $\\beta$ is around 3.\n\n---\n# W1: \n\n**Theoretically**：\n\nGCN's time complexity is: $O(\\sum^L_{l=1}(|E|H + NH^2))$, where $N, |E|, L$ denote the number of nodes, edges, and layers, respectively, $H$ denote hidden dimension. In our paper, teacher and student models' hidden dimension is 128 and 8, respectively. Assuming graphs is sparse, which means $|E| \\approx O(N)$, student is theoretically **229x** faster than their teacher models.\n\n**Empirically**:\n\nThe inference times (ms) of the teacher and student models measured on our device are shown in the table below:\n\n|          | Wiki-CS | Co-Physics | Pubmed | Citeseer | Cora | Average |\n| -------- | ------- | ---------- | ------ | -------- | ---- | ------- |\n| Teacher  | 906     | 3410       | 914    | 908      | 975  | 1422    |\n| Student  | 3.9     | 12.0       | 4.4    | 4.0      | 4.4  | 4.2     |\n| Speed-up | 227x    | 284x       | 204x   | 226x     | 220x | 232x    |\n\nOur method achieves an average acceleration of **232×**.\n\nAdditionally, we provided time and space complexity analysis of SWKT and GEO modules in the response to **W1 commented by reviewer rr98**. In our paper, we provide the parameter ratios (compression rate) between teacher and student for each KD method in Table 2, distillation training times and student inference time in Tables 2, 10, 11.\n\n---\n# W2: \n\nThe largest dataset used in our experiments was Coauthor Physics (495,924 edges and 34,493 nodes). We conduct further experiments on larger datasets ogbn-arxiv (1,166,243 edges, 169,343 nodes) and ogbn-proteins (39,561,252 edges, 132,534 nodes). Results show that our method consistently achieves best distillation performance on larger datasets.\n\n|               | Euclidean Teacher | Hyperbolic Teacher | FitNet | AT    | LSP   | MSKD  | VQG   | Our       |\n| ------------- | ----------------- | ------------------ | ------ | ----- | ----- | ----- | ----- | --------- |\n| Ogbn-arxiv    | 71.91             | 73.21              | 67.56  | 67.48 | 69.53 | 69.27 | 68.59 | **70.89** |\n| Ogbn-proteins | 72.83             | 69.23              | 68.71  | 68.53 | 69.45 | 70.97 | 69.54 | **71.22** |\n\n---\n# W3: \n\nYes, some remarkable pervious works focused on MLP-based students[1][4].\nMeanwhile, there are also significant studies in graph KD that use GNNs as the student networks, such as [2] and [3].\n\n**Our motivations for choosing GNNs as student**: We selected GNNs as the student model to explore **a new trade-off** among performance, inference speed, and compression rate. GNNs are designed for graph-based data, so GNN students can achieve performance close to their teachers even with a much small size (e.g., our method uses a GNN student model with **a hidden dimension of only 8 and ~2% of the teacher model's size**). Furthermore, GNN students demonstrate notable efficiency in inference acceleration, achieving an average **speed-up of 232x** compared to their teacher models in our method.\n\n---\n# Q1: \n\nOur student model takes full graphs as input. The sampled subgraphs you mentioned are used by the SWKT module to enable it to extract local structure features and corresponding manifolds.\n\n**Differences between MLPs and GNNs students**:\n- GNN students are smaller, with a ~2% size of their teachers (see Table 2). MLP students and their teachers have similar sizes due to having the same number of layers and hidden dimension (see appendixes in [1][4]).\n- GNN students achieve better performance on graph data since they leverage adjacency information.\n- MLP students have faster inference speeds due to their lack of graph dependencies, with [1] showing a speed increase of 273×, our GNN students is 232× faster than their teachers.\n\nIn summary, GNN students are suitable for scenarios with limited space resource, whereas MLP students are better suited for scenarios requiring higher inference speed.\n\n**Additional experiments with MLP-based student**: Our distillation method is model-agnostic. Using the same settings as [1] and [4], we test our method with GCN teachers and MLP students. Following results show that our KD method has better or similar performance with MLP students compared to GLNN [1] and VQG [4].\n\n|          | Citeseer  | Pubmed    | Cora      |\n| -------- | --------- | --------- | --------- |\n| **GLNN** | 69.23     | 74.70     | 78.97     |\n| **VQG**  | 74.59     | 77.13     | **81.15** |\n| **Our**  | **76.00** | **77.49** | 81.08     |\n\n(part of results from [4])\n\n---\n# Q2: \n\nIn fact, except our method, **all baseline methods do not incorporate various manifolds, as we are the first to introduce a KD framework using multiple geometric teachers**. In Table 1, rows marked as $\\mathbb{E}$ show results for baseline methods in their original versions without additional manifolds information, while rows marked as $\\mathbb{B}$ and $\\mathbb{S}$ show results after adapting to hyperbolic and spherical spaces, respectively. This adaptation ensures a fair comparison and evaluates the SWKT and GEO modules. We apologize for any confusion caused by unclear descriptions about Table 1. We will improve the presentation of the experimental section to enhance clarity.\n\n---\nReferences:\n\n[1] Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation, ICLR’22\n\n[2] Distilling Knowledge from Graph Convolutional Networks, CVPR’20\n\n[3] Boosting Graph Neural Networks via Adaptive Knowledge Distillation, AAAI’23\n\n[4] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs, ICLR’24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UEThTVyqEY", "reviewer": "Reviewer_rr98", "summary": "This paper proposes a novel method for graph knowledge distillation. The proposed method incorporates knowledge from both Euclidean and hyperbolic teacher models and transfers it to a student model in a way that leverages the appropriate geometry for different local subgraphs. A SWKT module is used  to select embeddings from the most suitable geometry based on the local subgraph structure. Furthermore, a GEO module refines the transferred knowledge by mitigating inconsistencies between Euclidean and hyperbolic spaces.\nExperimental results demonstrate that the proposed method outperforms other KD methods.", "questions": "See weaknesses.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The authors propose a method that leverages the strengths of both Euclidean and hyperbolic representations.\n\nS2. The experimental results demonstrate that the proposed method outperforms various graph data distillation baselines.\n\nS3. The proposed method has a similar running time as other baselines with a much higher compression performance.", "weaknesses": "W1. The paper would benefit from a thorough analysis of the method's time and space complexity. \n\nW2. The proposed approach uses two geometries (,i.e., Euclidean and hyperbolic). While this is a significant advancement, the paper could be strengthened by discussing its potential extension to incorporate additional non-Euclidean geometries. Exploring the necessary modifications for handling multiple geometries would broaden the applicability of the method.\n\nW3. While providing background knowledge can be helpful, the paper should ensure all definitions in the preliminaries section are essential for understanding the core method. Definitions like the Poincaré Disk Model and Tangent Space, if not directly used, could be moved to an appendix or omitted for clarity.\n\nW4. The paper primarily focuses on Euclidean and hyperbolic teacher models. Further exploration with teacher models employing different geometries, such as spherical space, would provide valuable insights into the method's adaptability. Analyzing the performance with these variations would strengthen the paper's contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments! We hope that our response can resolve your concerns. Please feel free to ask any follow-up questions.\n\n---\n# W1:\n\n**Notation Definitions:**\n- $N$: Number of nodes\n- $|E|$: Number of edges\n- $D$: Dimension of node features\n- $H$: Dimension of hidden layers\n- $R$: Number of teachers\n- $k$: Parameter for $k$-pop subgraphs\n\n**Space Complexity:**\n- Node feature matrix: $O(N \\cdot D)$\n- Adjacency matrix: $O(|E|)$ (assuming the graph is sparse)\n- Representations of hidden layers:  $O(R \\cdot N \\cdot H)$\n- Optimized distribution of local subgraph representations: $O(N \\cdot k \\cdot |E|)$\n\n**Overall space complexity: $O(N \\cdot D + |E| + R \\cdot N \\cdot H + N \\cdot k \\cdot |E|)$**\n\n**Time Complexity:**\n- Forward propagation: $O(N \\cdot H^2 + |E| \\cdot H)$\n- Local subgraph generation: $O(N \\cdot k \\cdot |E|)$\n- Structured-WiseKnowledge Transfer (SWKT) module: $O(N \\cdot k \\cdot H)$\n- Similarity measurement computation: $O(N \\cdot k)$\n- Geometric Embedding Optimization (GEO) module: $O(N \\cdot H^2)$\n\n**Overall time complexity: $O(N \\cdot H^2 + |E| \\cdot H + N \\cdot k \\cdot |E| + N \\cdot k \\cdot H + N \\cdot k)$**\n\nBesides, we also conducted a computational efficiency study comparing the teacher and student models from both theoretical and empirical perspectives in our response to **W1 commented by Reviewer GamE**. In Table 2 of our paper, we present the parameter ratios (compression rates) between the teacher and student models for each knowledge distillation (KD) method. Additionally, we provide information on distillation training times and student inference times in Tables 2, 10, and 11.\n\n---\n# W2:\n\nWe fully agree with your suggestion that exploring the necessary modifications for handling multiple geometries would broaden the applicability of our method. In our paper, **we have simply attempted experiments incorporating spherical geometry**.\n\nIn Table 1, the Cross method represent simplified variants of our framework. The rows marked $\\mathbb{E}$ and $\\mathbb{S}$ correspond to the use of both Euclidean and spherical geometries, while rows $\\mathbb{E}$, $\\mathbb{B}$, and $\\mathbb{S}$ represent the use of Euclidean, hyperbolic, and spherical geometries simultaneously. The results indicate that spherical geometry does not provide significant additional benefits within our framework. We apologize for any confusion caused by unclear descriptions about Table 1 of our paper. We will improve the presentation of the experimental section to enhance clarity.\n\nIn future work, we will explore feasible approaches to extend our framework to other potential geometries. For example, we may consider integrating more complex Riemannian geometries or other non-Euclidean geometries to further enhance the performance and applicability of the method. We appreciate your valuable suggestion and look forward to exploring these aspects in our future research.\n\n---\n# W3:\n\nYes, moving preliminary definitions that are not directly related to the core method to the appendix will enhance the clarity of the paper. \n\nAfter carefully review, we found that the expression for the Poincaré Disk Model in Definition 1 is not directly used in the presentation of our method, so we will move it to the appendix. The description of our method requires the projection formula between the tangent space and hyperbolic space in Definition 3, and this conversion involves hyperbolic operations described in Definition 2. Therefore, we will retain Definitions 2 and 3 in the main paper.\n\nThank you for your constructive suggestion. We will make the necessary adjustments based on your feedback to improve the clarity and readability of the paper.\n\n---\n# W4:\n\nIn fact, **we have conducted some experiments using spherical teacher model in our paper.** In Tables 1,7,8 of our paper, the rows labeled $\\mathbb{S}$ indicate results obtained using spherical models either alone or as one of the teacher models. Experimental results indicate that the use of spherical teacher models provides limited improvement in the results. This situation may be improved after further refining the framework for spherical space.\n\nWe will explore using teacher models from other geometric spaces to enhance the adaptability of our framework. We will also provide more detailed descriptions of the experimental setups in the paper to avoid similar misunderstandings and improve clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely thank the reviewers for the time and effort in reviewing our paper. We take all comments seriously and try our best to address every raised concerns. Please feel free to ask any follow-up questions.

It is encouraging to learn that the reviewers stated that our method, which leverages the advantages of different geometric spaces for graph knowledge distillation, is novel (ogwd, rr98) and intriguing (GamE). Reviewers also acknowledged that extensive (Z3B3) experiments conducted on various datasets (ogwd, Z3B3) show that our method outperforms baseline methods (ogwd, Z3B3, rr98). Reviewer Z3B3 kindly noted that the paper is well written.

Since the reviewers focused on empirical performance of our method, we would like to highlight our **theoretical contributions** here. To address the challenge that real-world graphs often exhibit geometrically heterogeneous characteristics, we have introduced **a novel model-agnostic cross-geometric knowledge distillation framework**. This framework, for the first time, leverages the advantages of various geometric spaces to offer high-quality guidance to the student model. Our dual-teacher setup and adaptive GEO module ensure that our method remains effective even under adverse conditions (such as partial teacher fail). Furthermore, we introduce **a new fine-grained distillation perspective** that evaluates embeddings, extracts knowledge, and transfers it as hints to the student model based on local subgraphs.

In the past week, we carefully improved the experiments (utilizing all available computational resources), clarified various aspects, and expanded our discussions to address the concerns, questions, and requests from all four reviewers. **In summary, we have implemented the following improvements**:

- We analyzed the computational efficiency of student models compared to teacher models in our method, examined the inference acceleration achieved by student models, and validated these findings through experiments. (in response to reviewer Z3B3’s Q4, Game’s W1)

- We provided a complete analysis of the time and space complexity of our method. (in response to reviewer rr98's W1)

- We conducted distillation experiments with non-Euclidean student models. (in response to reviewer ogwd’s Q2)

- We conducted distillation experiments by replacing the teacher model with graph transformer. (in response to reviewer ogwd’s Q3)

- We replaced the student model with MLPs, conducted distillation experiments, and compared the results with method of using MLPs as student. (in response to reviewer Z3B3’s Q5, Game’s Q1)

- We conducted distillation experiments on larger datasets, ogbn-arxiv and ogbn-proteins. (in response to reviewer Game’s W2)

- We analyzed the impact of fail teacher model on distillation results and designed experiments to validate this. (in response to reviewer ogwd’s Q1)

- We clarified the motivation for choosing GNN as the student model and its differences from MLP students. (in response to reviewer Game’s W3 and Q1)

- We conducted distillation experiments where the teacher and student models had identical architecture. (in response to reviewer ogwd’s Q4)

Meta-review: This paper proposes a graph knowledge distillation framework for transferring knowledge between different graph geometries. The framework distills knowledge from multiple teacher GNNs, each aware of a specific geometry, into a smaller student GNN. This approach not only improves the quality of graph representations but also accelerates inference speed. The authors successfully addressed the reviewers' concerns during the author-reviewer discussion phase, and at the end, all reviewers unanimously support acceptance. AC also finds that this paper has several strengths, including its novelty, extensive experimental results, and superior performance.

Therefore, AC recommends acceptance and highly encourages the authors to incorporate all experimental results and discussions from the discussion phase into the final manuscript, as this would strengthen the contributions of this paper.
Decision: Accept (poster)