Training Graph Transformers via Curriculum-Enhanced Attention Distillation
Authors: Yisong Huang
Jin Li
Xinlong Chen
Yang-Geng Fu
Venue: ICLR 2024 poster (2024)
Keywords: graph transformers
graph neural networks
knowledge distillation
curriculum learning
node classification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recent studies have shown that Graph Transformers (GTs) can be effective for specific graph-level tasks. However, when it comes to node classification, training GTs remains challenging, especially in semi-supervised settings with a severe scarcity of labeled data. Our paper aims to address this research gap by focusing on semi-supervised node classification. To accomplish this, we develop a curriculum-enhanced attention distillation method that involves utilizing a Local GT teacher and a Global GT student. Additionally, we introduce the concepts of in-class and out-of-class and then propose two improvements, out-of-class entropy and top-k pruning, to facilitate the student's out-of-class exploration under the teacher's in-class guidance. Taking inspiration from human learning, our method involves a curriculum mechanism for distillation that initially provides strict guidance to the student and gradually allows for more out-of-class exploration by a dynamic balance. Extensive experiments show that our method outperforms many state-of-the-art approaches on seven public graph benchmarks, proving its effectiveness.

Published as a conference paper at ICLR 2024
TRAINING GRAPH CURRICULUM-ENHAACEED ATTENTION DISTILLATION
TRANSFORMERS
VIA
Yisong Huang', , Jin Li Xinlong Chen' & Yang-Geng Ful- 'College of Computer and Data Science, Fuzhou University, Fuzhou, China 2 FI Thrust, Information Hub, HKUST (Guangzhou), Guangzhou, China huangyisong1 90 90 com jslijin201S@utloook f Ejxinlong@gmaill com fuê fulfzu.edde cn
ABSTRACT
Recent studies have shown that Graph Transformers (GTs) can be effective for specific graph-level tasks. However, when it comes to node classification, train- ing GT's remains challenging, especially in semi-supervised settings with a severe scarcity of labeled data. Our paper aims to address this research gap by focus- ing on semi-supervised node classification. To accomplish this, we develop a curriculum-enhaaneed attention distillation method that involves utilizing Local GT teacher and a Global GT student. Additionally, we introduce the concepts of in-class and out-of-class and then propose two improvements, out-of-class entropy and top-k pruning, to facilitate the student's 's out-of-class exploration under the teacher's in-class guidance. Taking inspiration from human learning, our method involves a curriculum mechanism for distillation that initially provides strict guid- ance to the student and gradually allows for more out-of-class exploration by a dynamic balance. Extensive experiments show that our method outperforms many state-of-thed -ft---rttth-ttttt methods on seven public graph benchmarks, proving its effective- ness.
INTRODUCTION
Graph-structured data is ubiquitous, and various real-world systems can modeled as graphs. In recent years, Graph Neural Networks (GNNs) (Wu et al. 2020: Chen et al. 2020) have shown
excellence in addressing graph-related tasks (e.g., node classification Kipf' & Welling 2017), link prediction (Yun et al.) 2021), and graph classification (Errica et al.) 2020 and play crucial role in applications across diverse domains, including recommender systems (Ying et al. 2018). traffic forecasting (Guo et al. 2019), and molecular property prediction (Kearnes et al. 2016).
GNNs follow the message passing paradigm (Gimeer et al. 2017). which updates the representation of node by iteratively aggregating messages from neighboring nodes and integrating them with its representation. However, GNNs have some limitations, such as a) difficulty in capturing information from distant nodes effectively and high requirements for the reliability of the input graph structure. These limitations restrict, to some extent, the application of GNNs.
Due to the powerful representation capacity of the Transformer architecture (Vaswani et al. 2017), it has found significant success in the fields of natural language processing (NLP) Kenton & Toutanova 2019) and computer vision (CV) (Dosovitskiy et al. 202T Liu et al. 202T.. Researchers have been exploring the generalization of Transformers into the field of graphs, commonly referred as Graph Transformers (GTs). GTs leverage the self-attention mechanism capture global de- pendencies among nodes in the graph. This enables each node to adaptively aggregate information not only from its neighbors but also from distant nodes. As a result, this alleviates the limitations of traditional GNNs mentioned above.
GTs can be classified into two main categories: Global Graph Transformers (GGTs) and Local Graph Transformers (LGTs). GGTs, such as Graphormer (Ying et al.) 2021) and SAT (Chen et al.) 2022 ignore the graph structure and treat each node as a token input to the Transformer module.
*Corresponding author.

---

Published as a conference paper at ICLR 2024
However, preserving the original graph structure is critical for graph-related tasks, leading to various studies exploring dedicated structural and positional encoding methods. On the other hand, LGTs, such GT (Dwivedi & Bresson 2020) and HGT (Hu et al. 2020), perform attention mechanism computations on the original graph structure, with a limited receptive field for each node to capture neighborhood information.
Although GT's have achieved state-of-the-ar performance in some graph-level tasks, their ef- fectiveness in node-level tasks, especially semi- supervised node classification, has received limited attention. This work aims to bridge this research gap by enhancing the performance of GT's in semi-supervised node classification tasks. One of the challenges training GT's is the scarcity of labeled nodes, which can lead to overfitting due to their increased number of learnable parameters. Limited labeled data can make GTss more vulnerable to noise or non-representative nodes, reducing model performance and generalization.
Table 1: Node classification accuracy of GCN, GT, and vanilla Transformer on Cora and Citeseer (mean = std %, the best results are bolded).
Method
Cora
Citeseer
GCN
82.06 0.69
71.6 = 0.32
GT
77.04 + 0.05
66.22 = 0.11
naive Transformer
54.24 + 3.16
50.60 + 3.83
To conduct an initial exploratory experiment, we select a set of representative models, including GCN (Kipf & Welling 2017) (a GNN model), GT (Dwivedi & Bresson (0200) (an LGT model), and the naive Transformer (Vaswani et al., 2017) (a GGT model), to investigate their node classification accuracy on the Cora and Citeseer datasets. The results are shown in Table [ It be observed that GCN outperforms GT and the naive Transformer. There exists a significant performance gap between GT and the naive Transformer. GT can be viewed as a variant of the naive Transformer since it restricts the receptive field of each node to the first-order neighbors, whereas the naive Transformer considers global nodes. Despite the potential of the naive Transformer, its performance falls significantly short. Unleashing the potential of the GT's particularly the GGTSs is the focus of our research.
To our knowledge, we the first to propose attention distillation as solution address challenges training GTs, particularly their susceptibility to overfitting. Specifically, we utilize the attention coefficients from each transformer layer in the LGT model (teacher model) to guide the learning of attention coefficients in each transformer layer of the GGT model (student model), enhancing the training process of the GGT model. The student model's learning process consists of key components: in-class learning, where benefits from the teacher's guidance, and out-of-class ex- ploration, where leverages acquired implicit knowledge seek more beneficial interactions. We propose two improvements, out-of-class entropy and top-k pruning, to encourage the student model actively explore more refined interactions during out-of-class exploration. However, it's impor- tant to note that the distillation process mentioned above has certain limitations. To address them, we introduce "curriculum distillation", an enhanced method incorporating curriculum learning to further unleash the potential of attention distillation.
The contributions of this paper are summarized as follows:
We propose utilizing attention distillation to enhance the training process of GT's This involves selecting an LGT as the teacher model and a GGT the student model for layer- to-layer attention distillation. Furthermore, we propose two significant improvements, out- of-class entropy and top-k pruning, to constrain the student's behavior during out-of-class exploration and enhance its generalization capability.
To address the limitations of the aforementioned method, we further enhance the proposed distillation method by integrating the concept of curriculum learning, which we refer to as "curriculum distillation".
We validate the effectiveness of our proposed method on seven graph benchmark datasets. Our method consistently outperforms existing GTs and GNNs, demonstrating enhanced performance and improved generalization capability.
Lastly, we provide an overview of the structure of this paper. We introduce the necessary background and preliminary concepts in Section 2 We present our proposed method in Section 3 Experimental
2

---

Published as a conference paper at ICLR 2024
results and discussions provided in Section 4 Finally, we conclude the paper in Section S For review of related work on Transformers, knowledge distillation, and curriculum learning in the field of graphs, please refer to Appendix
2
PRELIMINARIES
2.1 PROBLEM DEFINITION
Given a graph G = {V.E) where V and denote the sets of nodes and edges, respectively. The graph's s structure can be represented by adjacency matrix A RNXN, where N is the number of nodes. If there exists an edge between node v; and f/2 then A, = 1; otherwise, A, = 0. The nodes are associated with a feature matrix X E R Nxx, where dg denotes the dimension of the features. Given the labels of of nodes Vr., the goal of semi-supervised node classification is predict the labels of nodes in VLV..
2.2 SELF-ATTENTION
self-attention (MHA) is the de facto building block of Transformers. For simplicity, we will describe the single-head self-attention module. Given an input sequence H = [|1,..., hm! R. xds, where m denotes number of tokens, and d, denotes the hidden dimension. The self- attention module firstly multiplies the input H by the weight matrices W R& xdy, W Rdd xd,, WV E IRd,, xd, to obtain queries Q xmixd, keys E mxxd and values V e max.
HW? K HWK = HWW
The self-attention is then calculated as follows:
Attn (Q, K, K) Softmax QKT
VV,
(2)
Vdz
where v dd is the scaling factor that avoids gradient vanishing. The attention matrix Softmax captures the pairwise interactions among nodes in the graph.
QKK vdd
3 METHODOLOGY
In this section, we begin by detailing our motivation in Section We then provide an in-depth introduction to attention distillation in Section 3.2 Afterward, we present a specific training frame- work in Section 3.3 which involves selecting an LGT model as the teacher and GGT model as the student for attention distillation. We also propose two improvements to constrain the student's behavior during out-of-class exploration in Section 3.4 To address the limitations of the above distillation method, we introduce curriculum distillation in Section 3.5
3.1 MOTIVATION
We view the model training process from the perspective of knowledge comprehension. During pre- training, the teacher model learns knowledge from input data and generates downstream predictions based on the acquired knowledge. Considering these predictions there are inherent limitations even if they can provide guidance to the student model. When facing familiar problems, the student model produces results similar those of the teacher model. How- ever, when encountering novel complex problems, the student model often falls short because it merely memorizes the teacher's 's output without deeper understanding of the underlying knowledge. Furthermore, the teacher's ' predictions may contain errors and biases.
representation of knowledge,
Rather than guiding the student to imitate the teacher's predictions, is more beneficial to help leam the teacher's "higher-level knowledge understanding" and encourage innovation on this basis. However, extracting the "higher-level knowledge understanding" from the teacher model is a non- trivial task, and considering the model's unique characteristics is essential. The uniqueness of GT models lies in their inherent ability to capture the dependencies among interacting objects (as sug- gested by (Kim et al. 2022), GT's are good structural learning models). These dependencies support
3

---

Published as a conference paper at ICLR 2024
40
35:
22
2002
4720
1000
500
19300
2032
4003
0320
5000
.0002
(a) Cora
(b) Citeseer
Performance compari- son: GGT student's compliance to LGT teacher's guidance.
Figure 2:
Figure 1: The F-norm dynamics of attention coefficients be- tween GGT student and LGT teacher over time.
the powerful performance of GT models and establish the foundation for their interpretability. These dependencies are effectively represented through the attention coefficients at each layer of the GT models. We argue that these coefficients accurately represent the teacher's "higher-level knowledge understanding", encompassing its deep comprehension of the problem and underlying reasons for predictions. Transferring this knowledge to students enables them to engage more fundamental knowledge innovation, resulting in more reliable outcomes.
After discussing the core motivation, we establish the feasibility our method with three points of additional validation and correction:
a) Feasibility of teacher guidance: This refers whether the student will follow the teacher's guidance under certain conditions. In experiments, we conduct attention distillation (in- troduced in Section B.2). where the student accurately fit the teacher's attention efficients (as shown in Figure []. This indicates a strong ability to retain the teacher's experience and closely adhere the teacher's guidance.
b) Effectiveness of knowledge comprehension: After learning the knowledge mentioned above, whether the student will make predictions based the acquired knowledge and achieve comparable performance to the teacher model. Experimental results (as shown in Figure 2 show that the student achieves performance comparable to that of the teacher after following the guidance.
c) Secondary innovativeness: This refers to the student's potential for secondary innovation, surpassing the teacher's performance, which will be thoroughly discussed in Section 4
3.2 ATTENTION DISTILLATION
To enhance clarity, we introduce the teacher and student models as ( and Misl, respectively. Their attention coefficients for the layers E [1, E] are denoted as (TT(()) E [] and (S(())
[0, 1|_ respectively, where & is the number of layers, and N is the number of nodes. These coefficients satisfy Σ Oi.j model's attention coefficients to guide the student model's training, which can be formalized as follows:
(T)()) Σラ= Rij
[8]()) = 1, 1, E [1, i E N]. We use the teacher
g+(
arg min d( ((
(1 (A,x;)),,
(3)
where (---) denotes the loss function, such cross-entropy loss. MI91 E [0, N the probability distribution predicted by the student model, where C the number of classes. A N and X E R Nx denote the adjacency matrix and node feature matrix of the input graph, respectively, where d,r is the feature dimension. Y, denotes the ground truth labels for the labeled training nodes in Vz. (-[S] denotes the optimal parameters of the student model. d(:,)) denotes distance metric between two matrices, such Euclidean distance Kullback-Leibh divergence. D, 2 is predefined hyperparameter that indicates the level of constraint imposed by the teacher model on the student model.
The essence of Equation 3 is introducing the teacher's constraint into the student's 's optimization process, resulting in constrained optimization problem. Designing the constraint penalty term
4

---

Published as a conference paper at ICLR 2024
(or regularization term) is common implementation approach:
0-13| arg min ( (M, (A,
+y P(d(a ,a Do),
(4)
where -(- is penalty function that imposes penalty when d(als) a) exceeds Do. and is used to control the strength of the penalty. The penalty function P(.-)) can be designed as:
P(d(a a Do) max(d(a "1 Do, 0).
(5)
Considering that the roles of and Do are similar, for simplicity, we set Dn 0 directly, and the Equation 4 is simplified to:
g-1s) = arg min (l (M (A,X;;);, YT.) + 7 a
(6)
3.3 DISTILLATION-BASER LGT-GGT TRAINING FRAMEWORK
Our method involves employing GT teacher with limited representation capacity to guide a GT student with stronger representation capacity but challenging optimize. We select GGT model as the student for its considerable potential, but it tends to overfit without proper guidance. Selecting an LGT model the teacher stems from considerations: a) LGT's attention coefficients are more reliable and less prone to overfitting than GGT's; b) Utilizing an LGT teacher to guide a GGT student enhances interpretability, simulating a more effective teacher-student relationship.
Teachers often provide reliable guidance in specific fields in real life, and the "less is more" teach- ing approach is highly valued. Therefore, we prefer that the teacher model's attention coefficients consider only subset of nodes rather than all nodes, making LGT preferable choice. For spe- cific node vs in the GGT student, the teacher guides attention coefficients within its first-order neighborhood (including itself). We define this region as the "in-class" region, denoted c(in) In real-world datasets, this region generally relatively small. Node 0 also has substantial unguided region, defined as the "out-of-class" region, denoted as c. = Vicc.in)
For simplicity, we choose a naive LGT model the teacher and naive GGT model the student. Equation can be refined as:
0 (S:GGT) arg min ClS.GGT) (MIS:COT))
(A,X;0)
11 EN
d(a (a.S:GGTO (8:GGT(() a (TLGT(()) 1.c!
1,c
Furthermore, it's worth noting that framework be easily extended to handle multi-head attention.
3.4
OUT-OF-CLASS ENTROPY AND TOP-K K PRUNING
In this subsection, we propose two improvements to simulate better "teacher-student learning" be- havior and improve the student model's performance based on our earlier discussions.
3.4.1
OUT-OF-CLASS ENTROPY
We introduce out-of-class entropy to address the following two issues:
a) Lack of motivation for student innovation: The student model often shows reluctance to- wards out-of-class exploration. Increasing 7 in Equation 7 encourages the student focus on in-class learning, but a small doesn't motivate sufficient out-of-class exploration. In other words, when 7 small, the student still follows the teacher's guidance, it reduces both supervised and distillation losses. However, while reducing supervised loss, out-of- class exploration may increase distillation loss, resulting optimization difficulties. This is the fundamental reason for the lack of innovation drive.
b) Distraction of out-of-class attention: During out-of-class exploration, for any given node U; V. it interacts with all nodes in the out-of-class region. This results in relatively
5

---

Published as a conference paper at ICLR 2024
distribution of out-of-class attention across multiple nodes, which lead to potentially risky interactions. To mitigate this, we encourage the student model to focus its out-of-class attention specific and relevant nodes based on its acquired in- knowledge.
Specifically, the out-of-class entropy Eous E IR is designed as follows:
11 N
a [5:GGT1()
Eout
Entropy'
N
1.c%om
(8)
Σ
1 log (a (ais (
= E
1=1
jEC
where (S:GGTT)))) denotes the normalized out-of-class attention coefficient from node U; to in k...c
the 1-th layer of the student model, which is defined as:
[s.GTT(()
1(:GGTT))
E
[0, 1),
Σ
i
(SGGT]
+
jEC!
where e a sufficiently small positive constant.
From Equation X we can clearly observe that how the out-of-class entropy can alleviate the afore- mentioned two issues:
a) For any node 2 V, let 8;
[S:GGT(())
E
[S:GGT]() aij
+ (0, 1), which represents
SEC
the total attention allocated by node v, in the out-of-class exploration. If this value is smaller, Eout tends to be larger. Therefore, minimizing Eou: will drive the student to engage in out-of-class exploration.
[S:GGT]() is kept constant, minimizing Faut results in the sparsity of out-of-class at- (8:GGT(()) jecc The detailed proof is provided in Ap- tention coefficient distribution {ax
b) If 8
pendix B
We incorporate the out-of-class entropy into the loss function, and Equation is improved as:
(S:GGT)
arg min mig cls.GGT) (MIs:GGTT (A,X:6)L. YL)
ΣΣ
N
d(a [8:GGT(() i.
[T-LGT(0)) + 3 E
(10)
+ N
a (,c!!))
where / is hyperparameter that controls the degree of out-of-class drive.
3.4.2 TOP-K PRUNING
For each node v; E V, its out-of-class region c(out) generally large, containing numerous nodes due to the low average degree in real graph datasets. Using GT models for node classification tasks can risk overfitting when normalizing attention coefficients, as seen in (Wu et al. 2022), due to the large number of nodes involved in the normalization process.
To address the aforementioned issue, we propose top-k pruning, simple yet practical improvement. prunes the out-of-class attention coefficients before computing out-of-class entropy. Specifically, we first obtain {a: setting the rest
[S:GGT }jEcion, and then retain only the top k: largest attention coefficients while
(.GGT(() a i
j E Index (N)
|S:GGT
VI E
Index (t)
0.
(1I)
Index (k) = arg tops
(la!5)
(S:GGT](1)

---

Published as a conference paper at ICLR 2024
where the arg topp (-) operation selects the indices corresponding to the top k largest attention coefficients the out-of-class region for each node. is a hyperparameter used to control the number of nodes retained in the out-of-class region.
3.5 CURRICULUM DISTILLATION
Reviewing the previous method, the student model maintains a constant strategy, keeping y and fixed throughout the learning process. We aim for larger values of y and | ensure that the student receives effective guidance from the teacher while maintaining sufficient motivation for out-of-class exploration. However, there exists competition in the allocation of attention between in-class learning and out-of-class exploration. When both 7 and B are large, conflicts arise, leading to the following two issues:
Inadequate out-of-class exploration due to insufficient attention allocation.
Excessive out-of-class exploration without a proper understanding of in-class knowledge, hindering both in-class learning and out-of-class exploration.
Therefore, attention allocation should evolve dynamically during training to enhance the student model's performance. Initially, priority should be given in-class knowledge, and it consolidates, attention to out-of-class exploration should gradually increase, ultimately achieving a balanced state.
Based the above motivation, propose "curriculum distillation", which integrates the "curricu- lum" concept into the distillation process. It allows for a structured teaching approach where the teacher initially provides strict supervision and guidance the student, then gradually allows the student to engage in out-of-class exploration and innovation freely.
We present an abstract framework that dynamically manages the hyperparameter ry:
y00 Schedule,, (t), Vt E [1, Teputh)
(12)
where Topoch denotes the total number of epochs, 7(()) denotes the value of at the t-th epoch, and (chedule,, is a non-strictly decreasing schedule function.
We also need to manage the hyperparameter 3 dynamically for out-of-class exploration by designing a similar schedule function, except that this function non-strictly increasing:
g(() Schedule,(t), Vt E [1, Tqucctl
(13)
Equation 12 and Equation design to calculate (t);
13 can be implemented differently. Specifically, we adopt the following
y Schedule, (t, cp (begin) ,p cp(ond) (ond) (begin) ((cont) vt E [1, Tepuch)
(bagin)
5 1. < < (bogin))
(14)
(bogin)
t t-ep Ci-epp(pppp (leinn) n opical) - Uptlegin (1 + cos
(ool) - (lggn))
ep (bogin)
S 5. (ndd))
(end),
(p(d))
l ~ T spoch
Within the epoch interval [ep (hogii)) ep (ond)) y varies, while its values remain unchanged outside this interval. The hyperparameters (hegin) and (end) represent the initial and final values of y, respec-
tively, with 7. (begin) (end) indicating non-strictly decreasing. Similarly, B follows Equation 14 sharing the same epoch interval [ep (begin), (epggiiieep..emme (end)) with but having different initial and final values, ((end).
denoted as (hegiin) and (end), respectively, where (hegin)
For an intuitive understanding, we present the curves illustrating the variations of hyperparameters y and across epochs under different curriculum designs in Appendix C
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We briefly introduce the datasets and baselines in experiments. More details are in Appendix E

---

Published as a conference paper at ICLR 2024
Datasets. We evaluate our method on seven benchmark datasets, including citation network datasets Cora, Citeseer, and Pubmed (Sen et al.
2008); Actor co-occurrence network dataset (Chien et al.
202I]; and WebKB datasets (Pei et al. 2020) including Cornell, Texas, and Wisconsin. We apply the standard splits for the citation network datasets, as in the previous work (Kipf & Welling 2017 For the remaining datasets, we the train- iiin--aaiidatoo-eest split 48%/32%/200
Baselines. We compare our method against ten baseline methods, including five GNN methods, i.e. GCN (Kipf & Welling 2017), GAT (Veličković et al. 2018). GraphSAGE (Hamilton et al. 2017). SuperGAT SD (Kim & Oh, 20,T), and APPNP (Gasteiger et al. 2019), along with five Graph Transformers, i.e., GT (Dwivedi & Bresson 2020). SÄN (Kreuzer al. (02T). Graphormer (Ying
et al. 2021), ANS-GT (Zhang et al. 2022) and NodeFormer (Wu et al. 2022).
4.2
NODE CLASSIFICATION PERFORMANCE
We conduct five random runs and report the average and standard deviation of the experimental re- sults. The results are shown in 2 We have the following observations: a) The proposed method outperforms all models and shows sigmificant improvements on multiple datasets compared to the baselines. For example, it achieves relative improvement of 6.70% over GCN on Citeseer. Com- pared to Graphormer, it achieves relative improvements of 34.27% on Cora and 22.85% on Cornell. b) GTs perform less effectively two reasons: First, GTs have more parameters than GNNs, making them more challenging to train with limited labeled data. Second, GTs lack local inductive biases, limiting their ability to leverage the inherent structural information in the graph effectively.
node classification tasks than GNNs, which can be attributed to
Table 2: Node classification performance (mean + std%, the best results are bolded and the runner- ups are underlined).
Method
Cora
Citeseer
Pubmed
Actor
Comell
Texas
Wisconsin
GCN
82.06 + 0.69
71.60 + 0.32
79.58 + 0.26
27.88
0.07
53,51 + 1,21
69.19 + 1.48
$7.25 + 1.25
GAT
82.84 + 0.67
72.28
+ 0.72
78.52
: 0.32
28.71
= 0.25
55.14
0.48
68.65
:: 1.48
58.82 2.40
GraphSAGE:
81.40 + 0.52
71.68
+ 0.13
78.50
+ 0.44
36.24
= 0.39
63.78
2.42
75.14
1.21
76.08
0.88
SuperGAT sD
82.70 + 0.60
72.50
+: 0.80
81.30
: 0.50
30.18
= 0.25
54.59
+ 1.21
69.73
1 1.21
$8.04 1: 1.07
APPNP
84.10 + 0.43
72.14
- 0.22
80.02 = 0.19
33.47 0.38
61.08
1.48
71.35
+ 1.48
65.10 + 0.88
GT
77.58 +: 0.22
66.96 +
76.48 0.13
37.15 0.40
61.62 2.26
74.60
71.76 1.75
SAN
77.60 +: 0.23
68.64 +: 0.92
76.62 + 0.22
37.79 0.20
63.24 1.48
75.14 1:14::20 2.26
77.25
1.75
Graphormer
63.08 + 0.27
61.08 +: 0.04
00M
00M
62.70 1: 2.26
76.76 : 1.48
72.16 2 0.88
ANS-GT
77.68 0.81
38.29 0.61
74.92 1.86
76.22
2.26
76.47
1.96
Nodeformer
79.02 + 0.57
69.66 = 0.13
76.06 0.98
34,80 0.48
68.11 1 1.21
77.84
1.71
76.47
4.80
Ours
84,70 +: 0.56
76,40 = 0.70
83,10 = 0.42
39.40 0.09
77.03 + 1.90
85.58 1.56
81.95
0.87
Table 3: Distillation performance comparison.
Model
Cora
Citeseer
Pubmed
Actor
Comell
Texas
Wisconsin
Teacher (LGT))
77.40 + 0,56
68.05 1.06
79.05 + 0.63
36.87
0.14
68.46 1,56
76.57 + 1.56
75.48 1.38
Student (GGT)
54,24 + 3.16
50.60 = 3.83
72.50 + 0.87
35.97 1 0.29
65.76 + 5.62
77.03 + 2.70
77.55 +: 3.93
Ours
84.70 +
76.40 0.70
83.10 + 0.42
39.40 0.09
77.03 + 1.90
85,58 + 1.56
81.95 0.87
4.3 DISTILLATION COMPARATIVE STUDIES
Our method is meaningful only if the student model's performance improves with teacher guidance and surpasses the teacher model's after training. We summarize the performance comparison of our proposed method (Ours), the teacher model (L.GT), and the unguided student model (GGT) on all seven datasets, including average accuracy and standard deviation, as shown in Table 3
From Table 3 we observe the following: a) On average, our proposed method exhibits an absolute improvement 61% compared to the teacher model. Specifically, on the Cornell dataset, a relative performance of 12.51% % is achieved; b) On average, our method achieves absolute improvement of 13.50% compared to the unguided student model. Specifically, a relative performance improvement of 56.16% is observed training process of the GGT student model despite both the selected teacher and student models being naive GT models with notable limitations.
the Cora dataset. Therefore, the proposed method significantly aids the
8

---

Published as a conference paper at ICLR 2024
4.4 ABLATION STUDIES
We conduct ablation studies to verify and understand the roles of different internal modules in our method. The results are shown in Table
Module Ablation. "w.o. 7" and "w.o. 5 indicate the removal of the distillation loss and the out-of-class entropy by setting them O. respectively. "w.o. top-k" indicates the removal of the top-k pruning. Our proposed method exhibits an average absolute performance of 34.29%, 9.05%, and 6.29% across these three variants all datasets, demonstrating the significance and design rationality of each module.
Curriculum Ablation. "w.o. curriculum" indicates the removal of the curriculum learning frame- work. "w.o. epoch design" indicates the removal of the fine-graine grained interval range of the curriculum by setting Regin 0 and ep = 500. Compared to these two variants, our proposed method achieves an average absolute performance improvement of 10.01% % and 1.74%, respectively, indi- cating the importance and design rationality of the curriculum framework.
Distillation Ablation. "GGT GGT" indicates using GGT model as the teacher instead of an LGT to guide the GGT student. "Local neighborhood uniform distribution" indicates degrading the teacher's ' attention coefficients each layer to a fixed first-order neighborhood uniform distribution. "Global uniform distribution" indicates degrading the teacher's ' attention coefficients at each layer to fixed global uniform distribution. Compared to these three variants, our proposed method achieves average absolute performance improvements of 12.88%, 15.13% 13%, and 32.96%, respectively, on all datasets, demonstrating the rationality of the selected teacher model and the significant importance of the attention distillation compared to the degraded regularization.
Table The performance evaluation of variants of our method.
Categories
Mothod
Cora
Citescor
Pabmed
Actor
Cornell
Texas
Wisconsin
w.o. 'I
40.70
05
Module Ablation
w.o. /
81.20
72.70
81.00
36.
59.46
67.57
66.67
top-k
36.18
Curriculum Ablation
curriculum w.o. opoch design
82.10
72.10
79.90
34.74
64.86
67.57
56.80
83.90
74.90
81.30
38.75
72.97
83.78
80.39
GGT GGT
56.60
56.23
00M
0OM
69.37
77.70
81.37
Distillation Ablation
Local neighhorhond uniform distribution
81.70
72.50
79.70
26.11
$1.35
$4.05
56.86
Global uniform distribution
31,90
18.10
00M
25.46
54.05
64.86
52.94
Ours
84.70
76.40
83.10
39.40
77.03
85.58
81.95
4.5 ADDITIONAL ANALYSIS
We conduct a comparison between our method and other traditional knowledge distillation methods, along with hyperparameter studies, interpretability studies, and visualization. For detailed experi- mental results and analysis, please refer to Appendix F
Additionally, we demonstrate the extension of method to GNNs, such as GCN, by performing distillation experiment from GCN to GGT. For comprehensive examination of results and analysis, please refer Appendix H
5 CONCLUSION
In this paper, we propose a novel curriculum-enhanced attention distillation method to improve the training process of Graph Transformers, which is both simpler and more interpretable. By utilizing the attention coefficients within the Transformer layers as knowledge representations, we achieve a more efficient knowledge transfer from the LGT teacher to the GGT student. Furthermore, we introduce out-of-class entropy and top-k pruning as enhancements to promote active out-of-class exploration by the student under the teacher's guidance, enabling the student to surpass the teacher's performance and unleash its potential. Moreover, inspired by curriculum learning, we introduce curriculum distillation to capture the dynamic evolution of the attention distribution between in- class learning and out-of-class exploration during the student's 's learning process, further improving performance. Experimental results on multiple datasets demonstrate the effectiveness of our method in improving the training process of GTss and enhancing their generalization capability.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENTS
This research was supported by the University-nndustry Cooperation Project of Fujian Province, China (2023H6008) and the National Natural Science Foundation of China (12271098).
REFERENCES
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009,
Dexiong Chen, Leslie O' O'Bray, and Karsten Borgwardt. Structure-awar transformer for graph rep- resentation learning. In International Conference on Machine Learning, pp. 3469-3489. PMLR, 2022.
Zhiqian Chen, Fanglan Chen, Lei Zhang, Taoran Ji, Kaiqun Fu, Liang Zhao, Feng Chen, Lingfei Wu, Charu Aggarwal, and Chang- Chang-Tien Lu. Bridging the gap between spatial and spectral domains: A survey on graph neural networks. arXiv preprint arXiv:2002. 1867, 2020.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.
Guanyi Chu, Xiao Wang, Chuan Shi, and Xunqiang Jiang. Cuco: Graph representation with cur- riculum contrastive learning. In IJCAI, pp. 3300-2306, 2021.
Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recogni- fion, pp. 11898-11900 19988, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Vijay Prakash Dwivedi and Xavier Bresson. generalization of transformer networks to graphs. arXiv preprint axXiv:20/2.09699 2020.
Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neu- ral networks for graph classification. In International Conference on Learning Representatiions 2020.
Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Combining neural networks with personalized pagerank for classification on graphs. International Conference on Learning Representations, 2019.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 922-929, 2019.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs Advances in neural information processing systems, 30, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint urXiv:1503.02533 2015.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In Proceedings of the web conference 2020, pp. 2704-2710, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. International conference machine learning, pp. 2304-2313. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Chaitanya K Joshi, Fayao Liu, Xu Xun, Jie Lin, and Chuan Sheng Foo. On representation knowledge distillation for graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30:595- 608, 2016.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of naacL-HLT, volume 1, pp. 2019.
Dongkwan Kim and Alice Oh. How to find your friendly neighborhood: Graph attention design with supervisoon In International Conference on Learning Representations, 2021.
Jinwoo Kim, Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. Advances Neural Information Processing Systems, 35:14582- 14595, 2022.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations, 2017.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Re- thinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 14-21618-2162,, 2021.
Lujun Li. Self-regulated feature learning via teacher-free feature distillation. In European Confer- ence on Computer Vision, pp. 347-363. Springer, 2022.
Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge trans- fer. Advances in Neural Information Processing Systems, 35:635 35:633-649, 2022.
Zheng Li, Xiang Li, Lingfeng Yang. Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang- Curriculum temperature for knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504-1512, 2023.
Jing Liu, Tongya Zheng, and Qinfen Hao. Hire: Distilling high-order relational knowledge from heterogeneous graph neural networks. Neurocomputing, 507:67-83, 2022.
Xiaolong Liu, LUJUN LI, Chao Li, and Anbang Yao. NORM: Knowledge distillation via n-to-one representation matching. In The Eleventh International Conference on Learning Representations, 2023. https: http///ppppeeee/////o/pp/p. n for ummmm wtw.oonnnnannaaaaaaaaaaaaaa
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In International Conference on Learning Representations, 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints thin deep nets. arXiv preprint arXiv: 1412.650,,, 2014.
Mrinmaya Sachan and Eric Xing. Easy questions first? case study curriculum learning for ques- tion answering. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 453 463, 2016.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rai Collective classification in network data. Al magazine, 29(3):93-99 2008
Petru Soviany, Radu Tudor lonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, 1366)11526-1155, 2022.

---

Published as a conference paper at ICLR 2024
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task task-agnostic bert for resource-limites devices. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2158 2170, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- lion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In International Conference Learning Representations, 2018.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neu- ral Information Processing Systems, 33:5776-577 5788, 2020.
Deep self-
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555-455)) 2021a.
Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. Curgraph: Curriculum learning Prononnnnnnrreeeeeeeeeeee Proceedings of the Web Conference 2021, pp. 1238-1248, 2021b.
for graph classification. In
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference muchine learning, pp- 6861-6871. PMIR, 2019.
Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems, 35:27387-27401, 2022.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24, 2020.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Cur- riculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6095-6104, 2020.
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7074-7083, 2020.
Chengxuan Ying. Tianle Cai. Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William I. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 974 983, 2018.
Seongjun Yun, Seoyoon Kim, Junhyun Lee, Jaewoo Kang, and Hyunwoo J Kim. Neo-gnns: Neigh- borhood overlap-aware graph neural networks for link prediction. Advances in Neural Information Processing Systems, 34:13683-13694 2021.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint 1rXiv:6612.03926 2016.
Jiawei Zhang, Haopeng Zhang, Congying Xia, and Li Sun. Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140, 2020.
12

---

Published as a conference paper at ICLR 2024
Zaixi Zhang, Qi Liu, Qingyong Hu, and Chee- Lee. Hierarchical graph transformer with adaptive node sampling. Advances in Neural Information Processing Systems, 35:21171-21183, 2022.
Tianyi Zhou, Shengjie Wang, and Jeffrey Bilmes. Curriculum learning by dynamic instance hard- ness. Advances in Neural Information Processing Systems, 33:8602-8613, 2020.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORK
A.1 TRANSFORMERS FOR GRAPH
The Transformer architecture (Vaswani ct al. 2017) has achieved great success in many fields, such as NLP and CV, stimulating researchers' interest in generalizing it to graph-structure data. GT's are designed to solve graph-related problems such as node and graph classification. Some pioneering studies on GTs focus on node classification and have achieved competitive performance compared to GNNs. ANS-GT (Zhang et al. 2022) adaptively selects node sampling strategy from four strate- gies (I-hop neighbors, 2-hop neighbors, KNN, and PPR) through multi-armed bandit mechanism to select the most informative nodes imput to the Transformer module. Graph-Bert Bert (Zhang al., 2020) limits the receptive field of each target node to the top-k nodes in the intimacy matrix cal- culated based on the PageRank algorithm. NodeFormer (Wu et al.) (0022) approximates pairwise similarity through kernel functions, and it also uses Gumble-Softmax to enable differentiable op- timization of graph structures. Compared the above methods, our method is more concise and interpretable.
A.2 KNOWLEDGE DISTILLATION
Knowledge distillation was first proposed by Hinton (Hinton et al. 2015). It is a "teacher-student" network training method that transfers the knowledge learned by large model with complex archi- tecture and sufficient capacity (called the teacher model) to a small model with a simple architecture and insufficient capacity (called the student model) to improve its performance.
Traditional knowledge distillation typically involves two main stages. Initially, the teacher model undergoes pre-training, and its output intermediate layer representations guide the student model. For instance, KD (Hinton et al., (015)) utilizes the teacher's logits as soft supervision, while Fit- Nets (Romero et al. 2014) leverages the teacher model's intermediate layer features as hints for knowledge transfer. and AT (Zagoruyko & Komodakis 2016) employs spatial attention maps for guidance. Some current research focuses on enhancing the effectiveness of knowledge distillation. NORM (Liu et al.) 2023) focuses on many-to-one representation matching, introducing a Feature Transform module for this purpose. DisWOT (Dong et al., 2023) addresses model capacity gaps by using a neural network architecture without training to find the optimal student architecture. To bridge the performance gap between offline and online distillation methods, SHAKE (Li & Jin 2022) combines both approaches using proxy teacher model with shadow head and student model for bidirectional distillation. Furthermore, there are variants of knowledge distillation frame- works. For example, Tf-FD (Li (022) introduces teacher-free feature distillation method that utilizes salient features within the student model for distillation.
Beyond its widespread use in the fields of CV and NLP, knowledge distillation is also gradually being introduced into graph-related tasks. LSP (Yang et al.) 2020) is the first knowledge distillation framework tailored for homogeneous GNNs. It develops a local structure preserving module (LSP) in GNNs to capture structural information and transfer the structural knowledge from the teacher the student by minimizing the distance between their distributions. HIRE (Liu et al. 2022) aims to preserve relational knowledge in heterogeneous GNNs, which distills the correlation knowledge between a single node soft label and different node types into the student model. G-CRD (Jshh et al. 2022) employs a contrastive learning method to implicitly preserve the global topology by aligning student node embeddings with teacher node embeddings
shared representation space.
Differing from existing knowledge distillation methods that aim at model compression, our method is designed to address the challenging task of training Graph Transformers in semi-supervised sce- narios. The representation capacity of the teacher model we employ is weaker than that of the student model. In this situation, the teacher's guidance can be seen as a constraint on the student. However, the student model possesses significant potential that requires proper guidance for its full realization. We intend to utilize knowledge distillation to alleviate issues encountered by the student model during training, such as overfitting, and to improve the quality of learned representations.
14

---

Published as a conference paper at ICLR 2024
A.3 CURRICULUM LEARNING
As mature strategy for training machine learning models, curriculum learning Waag et al. 202 la Soviany al. 2022). has gained widespread adoption in recent years across various fields, including NLP(Sachhan & Xing 2016 Xu et al. (2020) and CV (Jiang et al. 2018 [hou et al. 2020). It involves organizing samples from easy difficult (Bengio al.) 2009), enabling the model to gradually learn from simple to complex knowledge, similar to human learning, and enhancing the learning effect. Curriculum learning for graph-structure data is an emerging technology that aims improve the performance of graph-related tasks. To solve graph classification tasks, CurGraph (Wang et al.) 2021b) uses neural density estimation to model the distribution of graph-level embeddings. Then, calculates the difficulty scores of graphs based on their embeddings' intra-class and inter-class distributions. CuCo (Chu et al. (021)) combines curriculum learning with graph contrastive learning. It uses similarity measure function to score the negative samples and then ranks the negative samples from easy to hard according to the scores. CuCo starts with dasier-oo-distinuiit negative samples during training and progressively introduces
challenging ones.
B PROOF: MINIMIZING OUT-OF-CLASS ENTROPY FOR SPARSE ATTENTION COEFFICIENTS
In the main text, we introduce the out-of-class entropy. Notably, the normalized out-of-class atten- tion coefficients o (s:CCT]()) approximately satisfies:
VI E [1,
Σ
a
(s:GGT10) i.j.c!!a) clu))
=
(15)
jEC(
Here, we prove that when the sum of out-of-class attention coefficients for node v; held constant, [S:CCT]() ie., 8
Σ 1,
(S:GGT(() + E = 80 E (0, 1). minimizing the out-of-class entropy E
jEC
[s:GG](() jEc((m)
leads to sparser distribution of out-of-class attention coefficients {a
Proof.
Entropy'
a (S.GGT]0 (.c/on)
(S:GGTT)) log ai (..CCT10)) c..c
jEC!
[S:GTTO
Σ
1S:GGT10 (
log
[S:GGZ1(0)
S.GGTJU
jEC(
($:GGT(()
Σ
(S.GGT]()
log
+ log
[SGGTT/
jEC(un)
[8:GT]0)
Σ
a (SS.GGTO
log
an
IS.GGTIO
log
(S:GTT())
S.GGTa
c.
(16)
jEC(u)
jec(
Σ
a (S.GGTTO0
log
a [SCGTT]() (..ffo))
+ log
s
(S.GGT]())
Σ
(SGGTT(
jEc(us)
/EC(on)
Σ jECfu)
1S:GGM10 t..c
log a (8:GGT]() (s.cfmm)
+ log
(s:GGT()
(a (ss.aaai (.
~ Entropy
+ log(so)
jec/on)
Entropy (a c.(c))
15

---

Published as a conference paper at ICLR 2024
Thus, based the principle that minimizing entropy corresponds to sparser probability distribu- tion, we conclude that this result is valid, thereby completing the proof.
C VARIATIONS OF HYPERPARAMETER UNDER CURRICULUM DESIGN
Here, we present the curves of hyperpaarameters and as they evolve over epochs (under different curriculum designs), as shown Figure
8.00
2.00
7.00
1.75
6.00
*ohi eolla
1.50
5.00
1.25
9a. 1.00
4.00
0.75
3.00
0.50
2.00
0.25
1.00
200
800
400
1000
Epoch
Epoch
(a) y
(b) 8
Figure 3: The variation curves of hyperparameters 7 and 3 under different curriculum designs.
D COMPLEXITY ANALYSIS
The time and space complexity of pre-training the teacher model can be neglected compared train- ing the student model, and curriculum learning does not inherently increase complexity. Therefore, only discuss the time and space complexity of the student model.
The time complexity of a single forward propagation of the attention module is O(nd² + n²dd) Onn2d), and the space complexity is Omd-+n2) The feed-forward network (FFN) requires (ndd) time and ((nd) space per layer. Assuming each attention module is followed by 4eE layers of FFN (typically byrN small), the overall time and space complexity of single forward propagation student model are O(Ex n d LINN x nd?) O x n²? and Olexn' n +l ex nd + l 8 LAN x nd) Ol n²), respectively.
the
E EXPERIMENTAL DETAILS
E.
DATASET STATISTICS
In Table we show the detailed statisties of seven datasets.
Table 5: The statistics of the datasets.
Dataset
#Nodes
#Edges
#Features
#Classes
Hom
Train/Val/Test
Cora
2708
5429
1433
7
0.81
140/500/1000
Citeseer
3327
4732
3703
0.74
120/500/1000
Pubmed
19717
44338
500
3
0.8
60/500/1000
Actor
7600
33544
931
0.22
3648/2432/1520
Cornell
183
295
1703
5
0.30
87/59/37
Texas
183
309
1703
0.11
87/59/37
Wisconsin
251
499
1703
5
0.21
120/50/81
16

---

Published as a conference paper at ICLR 2024
E.2 IMPLEMENTATION DETAILS
In our experiments, we use cross-yntropy as the distance metric. For the LGT model, we adopt the GT model (Dwivedi & Bresson 2020). which computes attention by considering only the first-order neighborhood (including the node itself). As for the GGT model, we employ the vanilla Transformer model Vaswani al. 2017). Specifically, the implementation of the Transformer layer in the GGT model is identical to that of the encoder's Transformer layer in the original Transformer.
We maintain fixed values for certain hyperparameters: the pre-training epochs of the teacher model are set to 200, the training epochs of the student model to 500, and the weight decay to 5e-4. We conduct hyperparameter tuning for other important hyperparameters on each dataset using grid search. The hyperparameter ranges are presented in Table We provide the specific configurations of hyperparameters on each dataset in Table 7 We implement our method with Python and Py Torch and use Adam as the optimizer. All experiments are conducted
GeForce RTX 4090 GPU.
Table 6: The searching space hyperparameters
Parameter Names
Hyperparameter Ranges
hidden size
64, 128, 256
head number
1, 2,
dropout ratio
0.3, 0.4, 0.5, 0.6
h
10, 15, 20, 25, 30, 35, 40, 45, 50
ep (hegin)
0, 50, 100, 150, 200, 250, 300
ep (end)
100, 200, 300, 500
500
y (hegin)
2, 4, 5, 6, 7,
7 (end)
1, 4
8 (begin)
0. 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1
3(end)
0.2, 0.4, 0.6, 0.8, 1, 1.5, 2
learning rate
0.01,0.005.0.001
F ADDITIONAL RESULTS AND ANALYSIS
F.1 COMPARATIVE ANALYSIS OF OUR METHOD AGAINST OTHER DISTILLATION METHODS
In this subsection, we conduct comprehensive comparison between our method and traditional knowledge distillation methods. Specifically, we select two well-established methods, KD (Hinton al. 2013) and FitNets (Romero et al. 2014), and perform a detailed evaluation against our method.
To ensure a fair and meaningful comparison, we consistently employ LGT as the teacher model, ensuring their performances are comparable. As for the student model, we employ GGT throughout the experiments. The results are shown in Table
The results lead to a clear conclusion: our method significantly outperforms KD and FitNets in terms of performance. On average, across all datasets, our method demonstrates an absolute performance improvement of 6.39% and 741 compared to KD and FitNets, respectively. More specifically, on the Cora dataset, our method achieves an absolute performance improvement of 7.36% and 7.51 % over KD and FitNets, respectively, further confirming the outstanding performance of our method in practical tasks.
F.2 HYPERPARAMETER STUDIES
In this subsection, we investigate the impact of byperparameters ry and 3 and their trade-off with the supervised loss the student model's performance across the Cora and Citeseer datasets. In total,
17

---

Published as a conference paper at ICLR 2024
Table Optimal hyperparameter configurations across seven graph datasets
Datasets
Type
Hyperparameters
hidden size 128, layers head number 2,
Teacher
dropout = 0.5, Ir = 0.001, weight decay 0.0005,
Cora
hidden size = 128. layers 2, head number = 2,
dropout 0.6, 0.005, weight decay 0.0005,
Student
25, ((hegin) 3, (<(nd) 1.5, (hegin) 0.1,
[ 0.4, (begin) 0, ep(s = 400
hidden size 128. layers 2, head number 2,
Teacher
dropout 0.4, 0.003, weight decay 0.0005,
Citeseer
hidden size = 128, layers = 2, head number 2, dropout 0.7, 1r 0.01, weight decay 0.0005, k: = 15, [(hegin) 8, a (end) 2.5, (hegin) 0.
Student
5 (end) 1.5, ep e5 hegin) = 50, ep(end) (end) 150
hidden size = 128, layers 2, head number = 1,
Teacher
dropout 0.3, 0.001, weight decay 0.0005,
Pubmed
hidden size = 256, layers head number = 1,
dropout = 0.3, Ir 0.005, weight decay 0.0005,
Student
h: = 20, ((egin) = 2.5, 7 (end) 0.5, 3 (hegin) = 0,
(end) 0.2, ep (hegin) O, ep (end) 50
(p(ed)
hidden size = 128, layers 2, head number = 2,
Teacher
dropout 0.5, Ir 0.001, weight decay 0.0005,
Actor
hidden size 256, layers head number 2,
dropout = 0.6, Ir = 0.005, weight decay = 0.0005,
Student
= 30, 7 (begin) 5, (end) 3.5, 8 ((bggn)) 0.2.
(end) 2.5, ep (hegin) 0, ep (nd)) 50
hidden size 128, layers head number 2,
Teacher
dropout = 0.5, Ir 0.001, weight decay 0.0005,
Coreell
hidden size = 256, layers 2, head number = 2,
dropout = 0.5, 0.01, weight decay = 0.0005,
Student
k 10, ((eegin) 1.5, (end) 1.5, (hegin) 0.2,
[(end) 0.8, ep (begin) 300, (end) 500
hidden size 128. layers 2, head number = 2,
Teacher
dropout 0.5, 0.005, weight decay 0.0005,
Texas
hidden size = 256, layers = 2, head number = 2,
dropout 0.6, = 0.01, weight decay 0.0005, k = 10. > (hhginn) 6, (end) 2.5, S (hegin) 1,
Student
8 (end) = 2.5, ep (hegin) 100, ep 400
hidden size = 128, layers 2, head number = 2,
Teacher
dropout 0.3, 0.001, weight decay 0.0005,
Wisconsin
hidden size = 128, layers head number = 2,
dropout = 0.5, Ir 0.01, weight decay = 0.0005.
Student
k = 10, 7 ((hegin)) = 1.5, (end) 0.5, 8 (hegin) = 0.3,
((ndd) 0.5, ep (hegin) 50, ep end) 100
18

---

Published as a conference paper at ICLR 2024
Performance comparison across knowledge distillation methods. Teacher model, S: Student model, Distilled: Distilled model.
Table 8:
KD
FitNets
Ours
type
T
S
Distilled
T
S
Distilled
T
S
Distilled
Cora
77.40
55.90
77.34
77.40
54.80
77.19
77.40
54.24
84.70
Citeseer
68.60
55.80
70.27
68.6
57.5
69.51
68.05
50.60
76.40
Pubmed
77.60
73.30
78.12
77.8
77.2
78.42
79.05
72.50
83.10
Actor
37.17
37.50
37.78
37.17
37.04
38.20
36.87
35.97
39.40
Cornell
70.27
59.46
63.78
70.27
67.57
62.16
68.46
65.76
77.03
Texas
78.38
75.68
75.14
78.38
75.68
77.57
76.57
77.03
85,58
Wisconsin
76.47
80.39
80.98
76.47
78.43
75.10
75.48
77.55
81.95
we conduct three sets of studies: a) We vary the values of 3 while keeping y fixed: b) We vary the values of 7 while keeping 3 fixed; c) We investigate the impact of r 0 by fixing the ratio and adjusting accordingly.
Figure iillustrates the variation in the student model's 's performance concerning the value while keeping fixed. From the results, we can observe that: a) The curves generally exhibit gradually decreasing trend or a slight increase followed by a decrease. This indicates that for a fixed y value, there exists an appropriate threshold for 3. The student's out-of-class exploration yields benefit only when 3 does not exceed this threshold. the out-of-class exploration exceeds the range supported by the in-class knowledge, the performance of the student model will decline; If y slowly in- creases, the downward trend of the curves is alleviated, suggesting that improving in-class guidance facilitates the student's ability to engage in more extensive out-of-class exploration.
Figure illustrates the variation in the performance of the student model concerning the value of y while keeping 3 fixed. It can be seen that increasing y while keeping 3 constant contributes to improving performance. As the teacher's in-class guidance exceeds a threshold, out-of-class exploration becomes increasingly effective. However, an excessively high proportion of in-class guidance may lead to performance degradation, as out-of-class exploration could be marginalized. Furthermore, comparing the curves shows that higher values of /3 may require a correspondingly higher y, indicating a more substantial level of teacher guidance to elevate the student's performance from a lower level.
The study of the relationship between between supervised loss and distillation loss, investigating the benefits of applying distillation naive supervised learning. From Figure 6 it can be observed that regardless of the value of r, when is set to 0, Y must also be O. In this case, the student model relies solely on the supervised loss for learning, resulting in suboptimal performance. When is small, the student receives weak in- class guidance during out-of-class exploration, applying distillation on top of supervised learning does not improve performance, and higher values of / lead to worse performance. As r is appro- priately increased, with an increasing 3, the curves undergo transition from a decreasing trend to an increasing-ther thenin---tne----aiiinnn trend. The initial performance improvement indicates that distillation becomes effective with enhanced in-class guidance. However, the subsequent performance decline is because the student primarily focuses out-of-class exploration, and distillation cannot yet place the supervised loss. As r further increases, the decreasing trend of the curves gradually slows down, indicating weakening effect of supervised loss. When is relatively large, the downward trend of the curves stabilizes, indicating that the teacher's 's in-class guidance can somewhat substitute for supervised loss, enabling the student to learn in unsupervised manner.
and performance essentially explores the balance
F.3 INTERPRETABILITY STUDIES
To investigate the impact of method enhancing the global homophily of the GGT student model during training, we plot the curves showing the average node homophily variation over time (epochs) and compare them with those of the unguided GGT student model (Figure
19

---

Published as a conference paper at ICLR 2024
0.8
0.7
0.6
Accuracy
Accuracy
0.6
0.5
=025
y=25
0.5
==0
0.4
==0
Y=0.5
-0775
--1.0
0.3
==1
0.2
y=15
0.3
0.5
0.75
0.5
0.75
B
(a) Cora
(b) Citeseer
Figure The impact of hyperparameter 3 on the performance of the student model.
0.7
0.7
0.6
Accuracy
=0.25
$ m022
0.6
000
0.5
6-05
8-00
0.4
8 -1.0
0.3
0.4
0.2
1
2.5
3.0
3.5
4.0
1.0
3.5
4.0
(a) Cora
Citeseer
Figure 5: The impact of hyperparameter on the performance of the student model.
0.8
0.7
0.6
r-10
r -1.0
r15
r155
Accuracy
220
Accuracy
0.5
=20
r=80
1100
0.4
0.5
0.3
0.4
0.2
0.6
1.25
1.5
1.75
1.25
1.5
1.75
B
(a) Cora
(b) Citeseer
Figure 6: The impact of hyperparameter ratio r on the performance of the student model.
20

---

Published as a conference paper at ICLR 2024
0.55
0.50
Homogenetty Score
0.6
0.45
0.5
GGT
Dxsiliation
0.4
0.35
Dxciliation Durtcalum
0.30
0.25
0.20
100
200
300
400
Epoch
Epoch
(a) Cora
(b) Citeseer
0.6
Homogeneey Score
0.5
Homogeenelty Score
0.6-
GCT
Distilation
0.5
Distilation
0.3
Distilation
0.2
Distilation Curricuum
100
200
300
400
500
Epoch
Epoch
(c) Actor
(d) Texas
Figure 7: Evolution of average global homophily for GGT student models: a comparison of un- guided, guided, and curriculum-enhanced.
From Figure 7 it can be observed that the unguided student struggles to discover beneficial interac- tions globally, leading to limited realization its potential and overfitting. However, with teacher guidance, the student's capacity for independent exploration significantly improves, resulting in notable performance enhancement. Furthermore, incorporating curriculum design further enhances the student's 's learning outcomes.
F.4 VISUALIZATION
In this subsection, we present scatter plots of the original features and embeddings produced by GCN and our method on the Cora and Citeseer datasets (Figure 8) The scatter plots show the following: a) The original node features appear messy and exhibit cross-class fusion. b) The embeddings generated by our method demonstrate more explicit classification boundaries and better separability than those produced by GCN.
G FURTHER DISCUSSIONS
In this paper, we propose attention distillation for Graph Transformers, presenting novel method to the field of graphs. While similar works exist in NLP, such as MINILM (Wang al. 20200,, which proposes distilling the attention coefficients of the last Transformer layer of the teacher model to the student model, and MobileBERT (Sun al. 2020). which performs attention distillation layer-to- layer, their motivations differ significantly from ours. We primarily focus GT models, aiming to improve their training, mitigate overfitting, and enhance generalization abilitics based on their unique characteristics. We address specific and specialized problem, whereas previous research mainly focuses on model compression for generic models.
Further, introduce "curriculum distillation" in work. Although the term has been used in previous works, our method differs significantly context, motivation, and methodology from those earlier studies. For example, this paper (Li et al. 2023) employs temperature control in the Softmax
21

---

Published as a conference paper at ICLR 2024
(a) Cora(Original)
(b) Cora(GCNN
(c) Cora(Ours)
(d) Crteseer(Original))
(c) Citescer(GCN)
(f) Citeseer(Ours)
Figure 8: Scatter plots of original node features and embedding produced by GCN and our method on the Cora and Citeseer datasets.
operator of the teacher's output to gradually transit from fuzzy to clear teacher guidance, focusing on model compression. provides a relatively simple student model with knowledge ranging from easy to complex. In contrast, we tackle the challenge of overfitting in complex student model. Our curriculum design captures a dynamic balance between strict guidance and independent innovation over time. In terms of methodology, previous works tend to "constrain"" the teacher initially and gradually "release" it, while our method adopts a different method by "strictly guiding" the student initially and allowing for "innovation" later. Therefore, these two methods differ significantly in context, motivation, and methodology.
H THE EXTENSION OF OUR METHOD TO GNNS
While initially designed for Transformers and not specifically tailored for GNNs, our method is read- ily adaptable and extendable to various GNN models, including GCN (Kipf & Welling (017)) and SGC (Wu et al. 2019). with minor adjustments. Specifically, we can generate similarity matrices for non-attention models using their node embeddings. These matrices serve rough approximations of the attention matrices within our knowledge distillation framework.
To illustrate this adaptability, present preliminary experiment distilling knowledge from GCN to GGT. The corresponding experimental results are summarized in Table
Table 9: Distillation performance comparison.
Midel
Cora
Citeseer
Pubmed
Actor
Comeel
Texas
Wisconsin
Teacher (GCN)
82.06 + 0.69
71.60 0.32
79.58 + 0.26
27.88 + 0.07
53.51 + 1.21
69.19 # 1.21
57.25 1.25
Student (GGT)
54.24 + 3.16
50.60
3.83
72.50 + 0.87
35.97 + 0.29
65.76
5.62
77.03 4 2.70
77.55
3,93
GCN GGT
82.90 0.76
72.46 0.95
81.73 0.92
38.08
72.97 2.20
81.08
2.20
79.90 1.87
The experimental results indicate that applying method to knowledge distillation from GCN to GGT can significantly improve the performance of the student model across all datasets, with an average absolute performance gain of 10.78%. Specifically, on the Cora dataset, the absolute perfor- mance gain reaches 28. 33.66% compared to the unguided student model, demonstrating the superiority of our method.
22

---

Published as a conference paper at ICLR 2024
However, it can be observed that the performance of "GCN - GGT" still exhibits certain per- formance gap compared to the "LGT - GGT" employed in the main text (refer to Table for performance details). We attribute this performance gap to the inherent advantages of using Trans- former models as both teacher and student models. In transformer models, the internal attention coefficients automatically and inherently learned, eliminating the need to generate additional relational structures based on embeddings, as required in GCN. This ensures the reliability of at- tention mechanisms, as the teacher model strictly follows these learned coefficients for aggregation. In contrast, considering other models (such GCN), the similarity matrix is derived from their node embeddings in a post-hoc manner, and thus, its reliability, interpretability. and quality heavily depend on and are restricted by the embeddings. Therefore, subsequent construction of relational structures (e-g., similarity matrix) cannot be guaranteed be equally reliable, interpretable, and high-quality. From this perspective, inherent attention-based models might be more suitable and can benefit more from our distillation framework.
23


Reviews:
[{"review_id": "hHtVKAkAXD", "reviewer": "Reviewer_pnsY", "summary": "This paper proposes a curriculum-enhanced attention distillation method that involves utilizing a Local Graph Transformer as the teacher model and a Global Graph Transformer as the student model. They also introduce the concepts of “in-class” and “out-of-class” regions and introduce out-of-class entropy and top-k pruning. Experiments are conducted on 7 node classification benchmarks, which show the performance benefits of the proposed method.", "questions": "Q1: The distillation losses are calculated by distances between the teacher and student attention coefficients, which are averaged over graph nodes and attention layers, as demonstrated by Equation (7). Does the averaging calculation smooth out the signal?\n\nQ2: What is the exact distance metric $d$ do you use?\n\nQ3: For the top-k pruning, what is the exact $k$ do you use? Does it vary significantly between datasets? For example Pubmed has 19,717 nodes while Cornell and Texas only have 183 nodes, does the k values different? \n\nQ4: What are the specific LGT and GGT models do you use in the experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper proposes an effective method of training Graph Transformers. The method is verified in multiple node classification benchmarks. The distillation comparative studies show that the method generally outperforms both the teacher and student models. The paper is well-written with justified motivations and reasonable solutions.", "weaknesses": "Some of the implementation details are unclear. Please see the questions listed below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W02s5xCJDY", "reviewer": "Reviewer_qbbX", "summary": "This paper proposes a novel curriculum-enhanced attention distillation method to improve the training of graph transformers for semi-supervised node classification. The key ideas and contributions are: \n\n- Proposes using attention coefficients from a Local Graph Transformer (LGT) teacher to guide an untrained Global Graph Transformer (GGT) student via layer-to-layer attention distillation. - Introduces the concepts of in-class and out-of-class attention. Proposes two techniques - out-of-class entropy and top-k pruning - to encourage active out-of-class exploration by the student. \n\n- Inspired by curriculum learning, proposes curriculum distillation where the teacher gradually allows more out-of-class exploration via dynamic scheduling of distillation loss weights. \n\n- Achieves significant performance gains over strong baselines on seven benchmark graph datasets, demonstrating the ability to improve training and generalization of graph transformers.\n\nOverall, the work makes multiple innovations in adapting knowledge distillation for graph transformers in a semi-supervised node classification setting. The curriculum-based attention distillation framework is intuitive and achieves strong empirical results.", "questions": "see Weaknesses\n\n\n-------------------------------------------\n\n\nThe author's response addressed my concerns well, so I'm improving my score to acceptance, thanks!", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  The work presents a novel perspective of applying attention-based knowledge distillation for graph transformers. Using the teacher's attention maps to guide the student is an original idea. The curriculum distillation framework that balances in-class and out-of-class attention over time is also a creative approach.\n\n-  The methodology is technically sound, with clear algorithm descriptions, reasonable design choices, and rigorous empirical evaluation. The curriculum scheduling strategies are grounded in educational learning principles. The improvements for out-of-class attention demonstrate thoughtful analysis.\n\n-  The paper is well-written and easy to follow. The motivations are clearly explained, and the methodology sections provide sufficient details. Figures aid understanding of the core concepts like curriculum scheduling.", "weaknesses": "- The proposed method relies on selecting the right teacher-student pair, but the criteria for these choices are not fully analyzed. How does performance vary for different teacher-student combinations?\n\n- While outperforming baselines, the absolute performance gaps are sometimes small (~1-2%). The gains may not justify the added complexity in some applications.\n\n- Curriculum scheduling adds hyperparameters like epoch boundaries. The impact of these hyperparameters could be studied more systematically. In addition, I highly recommend that authors use auto-distillation (KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs (NeurIPS-2023, Automated Knowledge Distillation via Monte Carlo Tree Search (ICCV2023)) to optimize different parameter options.\n\n\n- Attention distillation for other graph model families besides transformers could also be promising but is not explored. It is essential to incorporate a thorough discussion of relevant KD-related studies, including Self-Regulated Feature Learning via Teacher-free Feature Distillation (ECCV2022), NORM: Knowledge Distillation via N-to-One Representation Matching (ICLR2023), Shadow Knowledge Distillation: Bridging Offline and Online Knowledge Transfer (NIPS2022), DisWOT: Student Architecture Search for Distillation Without Training (CVPR2023) . This discussion will help position the proposed approach within the existing literature, establish connections, and provide valuable insights for potential comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHeZNNf9MN", "reviewer": "Reviewer_4ymw", "summary": "The paper introduces a novel curriculum-enhanced attention distillation approach to enhance the training of Graph Transformers (GTs). This method leverages attention coefficients as knowledge representations for more effective knowledge transfer from a limited graph transformer (LGT) teacher to a general graph transformer (GGT) student. The approach also incorporates out-of-class entropy and top-k pruning to promote active exploration and the surpassing of the teacher's performance by the student. Additionally, curriculum distillation is introduced to manage the attention focus dynamically between in-class and out-of-class learning phases, thereby improving the overall performance. The proposed method has been empirically validated across multiple datasets, demonstrating its efficacy in improving the GTs training process and their generalization capabilities", "questions": "- The loss function is complex with a few components. In Table 4 the authors evaluated their pipeline's performance sensitivity to each of the components. My first question is how sensitive the model is to variation of these hyperparameters (e.g. $\\gamma$, $\\beta$)? Both in case of stability and performance.\n- Can authors please comment on the scalability of this model? Does it scale to bigger datasets? If yes, have they tried running their model on those?\n- Have the authors witnessed a difference in student or teacher behavior in datasets with relatively high homophily values differences?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written and presented.\n- Empirical results are promising.", "weaknesses": "- The novelty is limited. Not a big concern, but the paper combines a few well-known methods. \n- The approach is a bit complicated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1e2hJBYx5W", "reviewer": "Reviewer_imQL", "summary": "The paper addresses the challenge of training Graph Transformers for semi-supervised node classification with limited labeled data. The authors propose a curriculum-enhanced attention distillation method that utilizes a Local GT teacher and a Global GT student. By introducing the concepts of in-class and out-of-class, and incorporating out-of-class entropy and top-k pruning, the authors facilitate the student's exploration under the teacher's guidance. Inspired by human learning, the method gradually allows for more out-of-class exploration while maintaining a dynamic balance. Extensive experiments demonstrate the effectiveness of the approach, surpassing state-of-the-art methods on seven public graph benchmarks.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors identified the limitations of GTs in the task of semi-supervised node classification and effectively addressed them through knowledge distillation, which is a reasonable and effective approach.\n2. The proposed two significant improvements, out-of-class entropy and top-k pruning, can constrain the student’s behavior during out-of-class exploration and enhance its generalization capability.\n3. The experiments demonstrate enhanced performance and improved generalization capability of the method.", "weaknesses": "1. My major concern is the motivation of the distillation of GTs. Why not apply distillation on other graph networks to obtain more significant performance, e.g. GCNs? Can the proposed method be applied on other networks?\n2. In Ablation, which components are the most important? Why \"w.o. $\\gamma$\" , \"Global uniform distribution\" bring such damage for the performance?\n3. Why not compare to some general KD methods for classification task, e.g. KD, AT, FitNets, DKD, CRD?\n4. Please provide the details of Teacher and student, including architecture, parameters, etc. If the utilized \"Graph Transformer\" equal to \"Graphormer\" or \"Nodeformer\"? If not, why not conduct distillation on them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)