Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification
Authors: Yihong Luo
Yuhan Chen
Siya Qiu
Yiwei Wang
Chen Zhang
Yan Zhou
Xiaochun Cao
Jing Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Sharpness-Aware Minimization; Graph Neural Networks; Few-Shot Node Classification
Primary area: graph_neural_networks

Abstract: Graph Neural Networks (GNNs) have shown superior performance in node classification. However, GNNs perform poorly in the Few-Shot Node Classification (FSNC) task that requires robust generalization to make accurate predictions for unseen classes with limited labels. To tackle the challenge, we propose the integration of Sharpness-Aware Minimization (SAM)--a technique designed to enhance model generalization by finding a flat minimum of the loss landscape--into GNN training. The standard SAM approach, however, consists of two forward-backward steps in each training iteration, doubling the computational cost compared to the base optimizer (e.g., Adam). To mitigate this drawback, we introduce a novel algorithm, Fast Graph Sharpness-Aware Minimization (FGSAM), that integrates the rapid training of Multi-Layer Perceptrons (MLPs) with the superior performance of GNNs. Specifically, we utilize GNNs for parameter perturbation while employing MLPs to minimize the perturbed loss so that we can find a flat minimum with good generalization more efficiently. Moreover, our method reutilizes the gradient from the perturbation phase to incorporate graph topology into the minimization process at almost zero additional cost. To further enhance training efficiency, we develop FGSAM+ that executes exact perturbations periodically. Extensive experiments demonstrate that our proposed algorithm outperforms the standard SAM with lower computational costs in FSNC tasks. In particular, our FGSAM+ as a SAM variant offers a faster optimization than the base optimizer in most cases. In addition to FSNC, our proposed methods also demonstrate competitive performance in the standard node classification task for heterophilic graphs, highlighting the broad applicability.

Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification
Yihong Luo & Yuhan Chen",
Siya Qiu 1.2 Yiwei Wang*., Chen Zhang Yan Zhou", Xiaochun C Tang -2f The Hong Kong University of Science and Technology
The Hong Kong University of Science and Technology (Guangzhou)
School of Computer Science and Engineering, Sun Yat-sen University
4 University of California, Merced 6 University of California, Loa Angeles Createlink Technology
7 School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University
Abstract
Graph Neural Networks (GNNs) have shown superior performance in node clas- sification. However, GNNs perform poorly in the Few-Shot Node Classification (FSNC) task that requires robust generalization to make accurate predictions for unseen classes with limited labels. To tackle the challenge, Awaree tion of Sharpness- Aware Minimization (SAM)-a technique designed to enhance model generalization by finding flat minimum of the loss landscape-inte GNN training. The standard SAM approach, however, consists of two steps in each training iteration, doubling the computational cost compared to the base optimizer (e.g., Adam). To mitigate this drawback, we introduce a novel algorithm, Fast Graph Sharpness- Aware Minimization (FGSAM), that integrates the rapid training Multi-l Perceptrons (MLPs) with the superior perfor- mance of GNNs. Specifically, utilize GNNs for parameter perturbation while employing MLPs to minimize the perturbed loss so that we can find flat mini- mum with good generalization more efficiently. Moreover, our method reutilizes the gradient from the perturbation phase to incorporate graph topology into the minimization process at almost zero additional cost. To further enhance train- ing efficiency, we develop FGSAM+ that executes exact perturbations periodi- cally. Extensive experiments demonstrate that our proposed algorithm outperforms the standard SAM with lower computational costs in FSNC tasks. In particu- lar, our FGSAM+ as SAM variant offers faster optimization than the base optimizer in most cases. In addition to FSNC, our proposed methods also demon- strate competitive performance in the standard node classification task for het- erophilic graphs, highlighting the broad applicability. The code is available at /ettps:/ggihhng com/ dtwtma///aaaaaeeeee MNeurIPS24
propose the integra-
Introduction
Graph Neural Networks (GNNs) have received significant interest in recent years due their powerful ability in various graph learning tasks, e.g., node classification. Numerous GNNs have been developed accordingly 20 35 6. [5]. Despite their successes, GNNs, like traditional neural networks, tend to be over- wwer-paramteriieed often requiring extensive labeled data for training to ensure generalization. However, in real-world networks, many node classes have few labeled instances, which can lead
"Equal Contribution
1 CCoresponding Author: Xiaochun Cao and Jing Tang.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to GNNs overfitting, resulting in poor generalization in these limited labeled classes. Recently, an increasing amount of research is focusing on developing superior GNNs, e.g., Meta-GCN 42] AMM-GNN 37, GPN [ and TENT [8, for Few-Shot Node Classification (FSNC) which aims to classify nodes from new classes with limited labelled instances.
Intuitively, training GNNs for FSNC requires robust model generalization ability for recognizing unseen classes from a small number of labelled examples. Motivated by the success of the recently proposed Sharpness-Awan
Minimization (SAM) for improving models' generalization in the vision domain [2, we suggest incorporating SAM into training GNNs for addressing FSNC tasks. The core idea of SAM is to perturb the model parameters to find flat minima of the loss landscape, thereby making the model more generalizable. However, key drawback of SAM is that it requires executing two nerward-backwer consumption compared to general optimizers like Adam. Some works 9 23 10 have been proposed to accelerate SAM, but none of them are crafted for graphs, i.e., not leveraging the graph properties for accelerating SAM.
steps to complete one optimization step. resulting in twice the time
This paper mainly focuses on efficient GNN train- ing in FSNC scenarios by leveraging SAM for improving the generalization of GNNs on unseen classes. To tackle the high training cost issue of SAM, we utilize the connection between GNNs and
62.5
55%
$2.0
42.0
6:5
4:0
41.
57.5
Adam
discarding Message-Passingg (MP) are equivalent to MLPs with faster training and worse performance in general to acceler- ate training. Specifically. we propose Fast Graph Sharpness-Awe ware Minimization (FGSAM) that uses GNNs for perturbing parameters and employs MLPs (i.e., GNNs discarding MP) to minimize perturbed training loss. This speeds up training at the cost of dropping graph topology information during minimizing the perturbed loss. Interestingly, we find that the gradient computed in parameter perturbation can be reused when minimizing loss to explicitly reintroduce topology information with negligible extra cost. Moreover, we can add back MP during inference to improve performance. To further reduce the computational cost, propose FGSAM+ which conducts an exact FGSAM-update at every k steps. As shown in Fig. [ empirical results in FSNC tasks show that our proposed FGSAM and FGSAM+ methods outperform both Adam and SAM, and meanwhile FGSAM+ is even faster than Adam. In addition, we evaluate the proposed methods in node classification, showing strong results, especially in heterophilic graphs which are known to be challenging for GNNs 30 [7]. This indicates that our proposed methods can effectively improve the GNN's s generalization capability for better performance.
SAM
60.
FGSAM
670
+GSAM
1006
200/
107%
126/
2335
100%
Training time compared to base optimirer l.e., Adamm (86)
Figure 1: Comparison of average accuracy and training time across datasets on different GNNs. The closer to the top left corner, the better.
The contributions of this paper can be summarized as follows.
We study the application of SAM in FSNC tasks.
We propose FGSAM that improves generalization in an efficient way by leveraging GNNs for sharpness-awan perturbation parameters and employing MLPs to expedite training. We further propose an enhanced version named FGSAM+, which conducts the actual FGSAM at every à steps and approximates it in the intermediate steps.
We demonstrate strong empirical results of the proposed methods across tasks.
2 Preliminary
Graph Neural Networks. Let G = (V,E) denotes an undirected graph, V = {ma)]_ is the node
set and & V x V is the edge set. A E gnxan is the adjacency matrix. Let X = {x;] E a be the initial node feature matrix, where do is the initial dimension, and Y = {vr) E Bex denotes the ground-truth node label matrix, where C' denotes the number of classes and y: is the one-hot encoding of node v label y. Let (L) be the output of the last layer of an L-layer GCN, the prediction probability matrix Y = softmax (H)))) is the final output of node classification.
Few-Shot Node Classification. In the FSNC task, the entire set of node classes can be divided into two disjoint subsets: base classes set Chase and novel classes set Cnowet, such that C = Chase U Gaoeel and Chase O Cnowel = Ø. There are sufficient labeled nodes Clases while there are only a limited number of labeled nodes in Crovel. FSNC task aims to learn model using the sufficient labeled nodes from

---

GNN Loss Landscape
GNN (PSNC WIFGSAM)
Traii Loss
Tes: LoGS
(NC)
(PSNCI
GNN
PeorMLP
Lose
2.5
HLP
2.0
2.0
350
150
262
50
100
Episodes
Episades
(b) Visualization of Loss Curve Figure 2: (a): Loss landscape visualization of GNN across tasks and optimizers, (b): Loss of GNN, MLP and its PeerMLP on the test set over the training process. In these experiments. MLP and PeerMLP share the same weight space as GNN but are trained without message-passing.
(a) Visualization of Loss Landscape
Chase. enabling it to accurately predict unlabeled nodes (i.e., query nodes Q) in Crovel, with limited labeled instances (i.e., support nodes S) from Gnovel-
Sharpness-Awars Minimization (SAM). SAM [12] is effective method to improve model's generalization. Let Dr = {x) y+) be the training dataset, following distribution D. Given a model parameterized by and commonly used loss function (e.g., cross-entropy loss) f, instead 1 - {( Ys: w). SAM aims to minimize the population loss Cr(w) B(a.y))D(((az y; w) by minimizing the vanilla training loss as well as the loss sharpness (i.e., find parameters whose neighbors within the Ey ball also have low training loss D as follows:
of directly minimizing training loss LD,
w" arg min max |e pSp
[C ((), + (D,(w)) 21 [
arg min max ||e
LD. (w + +6) 11/20/1
where p is the radius of the ep ball, and p 2 0 (usually p = 2). this way, the model can converge to flat minima in loss landscape (wr"), making the model more generalizable [[2]. For efficiency, SAM applies first-order Taylor expansion and classical dual norm problem to obtain the approximation:
Vw (w)
é = P V (w)ll
arg max LD, (w + +).
Finally, SAM computes the gradient w.r.t. perturbed model w + é for update w in Eq. (1
Vw max LD, (w + c) VwCD, +) ~ (wLo,(w) w+e-
(3)
Se
Additional Related Works. The effectiveness of SAMs and its variants have been widely verified in computer vision area 112 21 9 23 [0, [] Specifically, LookSAM (23 speeds up the SAM by periodically conducting exact perturbation, and Sharp-MAML [1] firstly focusing on meta-learning tasks. However, there limited work on developing SAM for graphs. WT-AWP (39 is the first SAM-like work that applied to GNN and gives a theoretical analysis of generalization bound on graphs. Compared to these works, our proposed FGSAM is crafted for graphs by its unique property. enabling the first SAM-like algorithm that can be faster than the base optimizer. Our work also shares some similarities with existing works [7 40 that explore the connection between GNNs and MLPs. However, they attributed the claim that introducing MP to MLP can improve performance during evaluation the powerful generalization ability of MP. contrast, we prove that for the linear case with synthetic graphs, whether there is MP or not, both will converge to the same optimal solution, taking a solid step toward understanding the underlying reasons.
3 Methodology
In this section, we propose Fast Graph Sharpness- Aaare Minimization (FGSAM), an efficient version of SAM for GNNs, aiming to reduce the training time when using SAM FSNC tasks while improving model's generalization.
3.1 Motivating Analysis
SAMs are series of new general training scheme used to improve the model's generalization, thus it intuitive to use SAM in FSNC tasks. However, there no work studying how to apply SAM to FSNC tasks. So our first question is: Q1: Can SAM benefit sho- node classification tasks?
3

---

Table 1: Time consumption of 200 episodes training (sec.) of baseline w/ and w/ MP (only consider feed-forward and -backward).
CoraFull
DBLP
ogbn-A
Bseline
Backbone
5N3K
10N3K
5N3K
10N3K
5N3K
ION3K
Metn-GCN
GNN
9.56
9.38
17.61
17.50
41.09
40.96
PeerMLP
1.11
1.17
1.35
1.54
1.02
1.17
A key property of FSNC is that the GNNs need to be generalized to unseen classes (i.e., novel classes), and the GNNs often converge to a relatively low loss on the training set, but the final performance depends on the GNNs' generalization ability. To demonstrate this intuitively, we plot the GNN's loss landscape of novel classes under the FSNC setting and of the test set under the NC setting (Fig. 2a). following previous work [22] The loss landscape of GNN under the FSNC setting is sharp and not smooth, with many local minima, in contrast to the flat and smooth loss landscape of GNN under the NC setting. This to some extent indicates that the FSNC setting poses a greater challenge to GNNs, which is consistent with our prior knowledge. Hence, applying SAM-like techniques can intuitively improve the generalization of GNN and enhance its performance.
However, another problem arises: training GNN on FSNC is already slow, and the core drawback of SAM is that it requires twice the training cost compared to Adam or SGD. Q2: Can we find way to reduce the SAM training cost based on GNN properties?
It is well known that the training speed of GNNs is slower than MLPs, mainly due to the notorious MP that causes significant time consumption, yet MP is essential for improving GNN performance. Removing the MP from GNNs fgn {X, A) J: w) turns them into MLPs fmip (X; w), which is an intriguing connection. As shown in Tab. T and Fig. 2b MLPs without the burden of MP demonstrate substantial training time advantage under the same settings as GNNs and can achieve nearly the same performance as GNNs on the training set, however, they perform significantly worse on the test set, revealing their poor generalization performance.
Inspired by previous work [7,, is appealing to remove MP during training, but reintroduce it in inference (PeerMLP). Although reintroducing MP after training can improve the performance, it still cannot surpass GNNs' (Fig. 2b). This may be because of the lack of graph topology information in training. Hence, propose minimizing training loss on PeerMLPs but minimizing the sharpness according to GNNs, implicitly incorporating the graph topology information in training. This allows the model to quickly converge to the vicinity of local minima and further converge to flat GNN local minima through a GNN's s sharpness-aware approach. By doing so, we not only introduce SAM to enhance the model's ' generalization ability and the information w.r.t graph topology but also leverage the intriguing connection between MLPs and GNNs to improve training speed.
3.2 FGSAM
We elaborate our proposed method Fast Graph Setarpness-Aaare Minimization (FGSAM). For the ease of reference, Fig. Ba visualizes the framework of FGSAM, so does to its enhanced version FGSAM+. There are two forward- backarrddbwwddddddd steps in the FGSAM-update.
Step 1: Graph sharpness-aware perturbation. The first forward-backwart step is served for computing the maximum perturbation é (Eq. 22). where we propose to perturb parameters with MP (GNN),
gann
Vw (w; fgm) u WWCC(WW (w; fym)ll P P C( gmn w),, Y)
V (f G w) Y)
(4)
Step 2: Minimizing perturbed loss. We propose minimize the perturbed loss by removing the MP (PeerMLP) to speed up training, i.c..
w= x arg min Lx (ur + é; fmip) arg min L(Smlp (X;u w + E), Y)
(5)
arg min L(f (x. I): w + E), Y).
It is clear that minimizing the loss on PeerMLPs is equivalent to minimizing the loss on GNNs ignoring the topology information. As demonstrated in Sec. Sec.B.1| intuitively the proposed approach can make model convergence near the local minima easily due to the connection between MLPs and GNNs, and perturbing parameters with MP can find the good flat minima of GNNs (see Fig. 2a).
Reintroducing Graph Topology in Minimization with Free Lunch. While reintroducing the MP in evaluation can improve performance, its absence during the minimization process may result in

---

Difference of Gradient
24
Meal
0.2
50
100
150
200
250
FGSAM
FGSAM+
Steps
(a) Visualization of FGSAM and FGSAM+
(b) Gradient difference curves
Left (a): The solid line indicates that the gradient is computed on the corresponding model, while the dashed line indicates the opposite. Right (b): The difference of gradients (i.e., 11g-11 - ge 12). It can be seen that 9. and 9g change much slower than 9a and gh. across the training process, thus can be reused in the intermediate steps.
Figure 3:
sub-optimal results. Incorporating MP directly into the minimization is computationally expensive, leading us to employ MLP to minimize the perturbed loss. Fortuitously, the gradient w.r.t MP is computed during the perturbation step, offering an opportunity for computational savings. We propose to capitalize on the already available gradient information from the first step by reusing it in the optimization procedure, as formalized in the following optimization target:
w arg min (> x Lg(w; (w; fgm) + [x( + é; fmip)) X20
(6)
This formulation implies that the computational cost of involving MP in the optimization is mitigated since the forward and backward passes are precomputed in the initial step. Thus, we effectively inte- grate graph topology into the minimization process almost without incurring additional computational expense, akin to receiving a free lunch. See detailed FGSAM in Algorithm
Adaptation to MAML Models. Model-Agmostic Meta-Learning (MAML) is widely used in FSNC tasks 38, involving two separate update steps in one MAML-update: learning task-relevan, knowledge, and ii) meta-update for iark-irrelevant update. This is different from standard gradient descent. Hence for integrating the FGSAM into the MAML models, we propose treating the MAML-update process as single entity, and applying the FGSAM-update only once simplifies the implementation, This contrasts with the Sharp-MAMI. applied separately in the two stages.
pre-training for
D. where the SAM-update
3.3 FGSAM+
Although the training time of FGSAM can be largely faster than naive SAM by ignoring the MP in minimizing perturbed loss, it still requires full approach need an extra computation cost for a aaward-backward step of PeerMLP, compared to the base optimizer.
step of GNN, which makes
Fortunately, the forward- backward step of GNN is mainly for perturbing parameters in FGSAM, thus we can further reduce the training time while maintaining performance, by employing FGSAM-update at every step (i.e., perturb parameters at every step) and reusing the preserved gradients from parameters perturbation into the intermediate steps
23
Eq.
3)
can be rewritten as:
V Do (w) m+t RJ Vw
R
V (o,(w) - pl V Lo,
In this way, SAM--radient 9s is composed by the vanilla gradient Ww (w) and the gradient of the fz-norm of vanilla gradient V.. v..lIV... V'
(w)ll-
This suggests that SAM-gradient g. = CwCD, (w) w+e can be divided into two orthogonal
parts [23]: 9h (in the direction of vanilla gradient = V WwLD, ) is used to minimize the loss value, and flatness-gradient ge used adjust the updates towards flat region. So ga and g. can be easily obtained if g, and g are given:
|lgs|| cos 9 igl
9.
9
9
gh
=11.11
9. 9, 9h
g.
g
where 0 is the angle between Is and gh. As illustrated in [23] 9v changes much slower than 9- and gh, thus we can compute and preserve g. at every steps, and reuse it to approximate gs in intermediate steps.
5

---

However, in our case, there exists a clear gap between the model used for perturbing (GNN) and for minimizing (PeerMLP). This is different from the approach in [23, which uses the same model for both. Thus we use an extra PeerMLPP forward-backward step to get another gmlp for computing Ie to reduce the gap:
g milp gm = V Wmfppfm (X; w)) = V Vw fmip), g. =
-wwC(w; fmip) lw+te
Note that the é is obtained by perturbing parameters with MP Eq. 9s and gilpe are obtained without MP, thus there still exists a gap.
Moreover, since we reintroduce graph topology (Eq. (6) in minimization, we propose to further use the extra PeerMLP step reuse graph topology for better performance. Specifically, we conduct the gradient w.r.t. topology information by projection as follows:
gulpp
9: = g - ||gsm||| cos((") g
(10)
where 0' is the angle between gath and grirp This can be reused in a similar way as ge when approximating FGSAM-update in the intermediate steps. We further conduct experiments to verify whether the 9g and g. will change slowly so that they can be reused for speed up in our approach. We plot the change of ga, Ih- Iv and gg (Fig. (3b) and the results show that the projected gradient both go and gg on parameters perturbed with MP shows much more stable pattern and slower changes than 9s and gt. indicating the feasibility of updating go and go every lt steps and reusing for the intermediate steps. We present the detailed FGSAM+ in Algorithm in Appendix [B]
Since we need an extra PeerMLP forward-backwart step at every step, the overall computation cost of our approach, FGSAM+, will be x the computation cost of GNNs plus (1 ) the computation cost of MLPs on average.
Analysis of Toy Case
In this section, we employ the Contextual Stochastic Block Model (CSBM) to analyze why minimizing perturbed training loss without MP can work to some extent, which is the underlying mechanism of FGSAM. The CSBM has been widely used to analyze of the properties of GNN (2 26).
Specifically, we focus on CSBM model that contains K distinct classes C1, C2...., CK. The nodes within the resulting graphs are grouped into non-overlapping sets C1, C2....,C............ Ck. each set representing one of the K classes. The generation of edges governed by probability p within the same class and probability between different classes. For any given node i, we sample its initial features 2; E R. from Gaussian distribution denoted by æ; ~ NN,,, I), where the mean μ #4 E R' corresponds node i belonging to set Ck, and k is an element of 21, K). Furthermore, the condition ||ı - -1رر||2 = D holds true for all j belonging to (1, K}. with D being a positive constant. Graphs that arise from this specified CSBM model are referred as K K-classes CSBM. After applying MP operation, the resultant features for node i are denoted by h,
The neighborhood label distribution D, of node i is K-dimensions vector, where D.(j] I[i C2)p + (1 - T[(i E Cz))q. Based on the neighborhood label distribution, consider the MP operation as h, degla) ΣjEN()) Di, we have: h; ~ N
(p=))ue p+(K-1y0
deg()) where i E C and çt =
My Based the distribution of h; and æy, can obtain following theorem:
Theorem 4.1 (The effectiveness of removing MP in minimization). Consider a K-classes CSBM, the optimal linear classifiers for both original features a, and filtered feutures h4 are the same.
Detailed proof is in Appendix C The theorem tells us that under the linear case, whether the MP layer is used or not, the optimal decision bound is the same. Hence, this encourages us to learn the weight of transformation layers without MP to speed up training. However, the real graph is more complex and we do not use a linear classifier, thus we propose to perform the graph sharpness-aware perturbation which implicitly involves the information of neighbors.
5 Experiments
We verify the effectiveness of our proposed FGSAM and FGSAM+ in this section. We first conduct experiments demonstrate that our proposed algorithms achieve better performance compared to SAM which requires twice the training time. Then we show that our proposed algorithms can achieve faster training speed compared to base optimizers (e.g., Adam). Next, we also conduct extra studies and an extra task to show the robustness and potential applications of our proposed algorithms.

---

Table 2: Accuracy and Time consumption the baseline with different optimizer. The best and the runner-up are denoted as boldface and underlined, respectively. '5N3K' denotes 5 5-way 3-shot setting. Time consumption of 200 episodes of training (sec., only consider
is also shown.
Setiin
SNEK
SN3K
Caratull HONSK
10NEK
800
Oree
SN3K
SN5K
DRLP
10N3
10N5
000
Avg Uiee
SN3K
SN5K
ogba-orkit 10N3
10N5K
600
Arg
Hine
Meia-GUN
To.25
13:00
51.19
58.5
56.35
9.65
$3.00
55.21
65.10
70.85
76.15
17.53
-9.33
16.37
10.08
25.20
00
64
RESAM
T1.25
75-47
54.71
54.75
TWI0S
$3.50
S5TRE
06.31
11.23
76.77
35.30
64,89
15.19
31.30
41: 7377
FGSAM+
71.58
58.73
67.31
4.50
08.97
12.15
10.62
57.08
58006
33.36
43:05
AMM--GNS
72.92
89.44
37:36
52.20
51.05
15.00
$1.01
K3.68
F6.40
11:31
75.55
25.15
58.05
57.79
23:71
26.34
41.30
43.33
SAM
53.34
$0.34
16.20
58.15
50.15
10.57
11.42
EGSAM
79.18
52.59
62.81
66.56
10.00
31:34
$5.00
70.39
71.32
71.00
18.26
33.822
50.49
33.50
34.85
42.20
23.19
192 III-MAMI modes
$7.30
42:12
SAM
67.35
65.02
53.05
52.50
59.82
3.62
75:44
72.56
67.38
57.38
23.09
6.75
58.65
39.91
39.92
63.67
13:95
w: F.GAAM
09.54
00.37
91.85
56.00
63.31
2.33
98.10
79.11
08:30
50.44
34.43
0.00
57.5
58:25
47.67
48.20
$2.92
K.57
FCSAM
$0.40
57.24
56.10
53.30
3002
70.00
19.31
2.56
53.39
58.04
45.59
49.49
52.53
TENT
71.24
$3.29
56.00
62.4
11.36
67.34
47.14
54.85
57.00
13.90
SAM
11.38
75.35
55.85
61.5
66.35
22.00
83.03
68.96
15.02
37.45
12.5 06
63.58
20
50.76
55.31
59 72
26.43
FGSAM
77.10
26.23
53.85
63.1
53.35
28.28
23.30
16.13
71.11
73.01
25.21
20.55
03.84
71.35
53.33
57.88
61.3
33.00
FGSAM+
72.85
$8.37
7372
15.58
53.56
16:85
Table 3: Comparison between SAM variants regarding accuracy and time consumption (10N3K).
Settings
SN3K
CoraFull
LONUK
SN3K
DBLP
10N
SN3K
aghn-arXiv
LONAK
Arg
Rosilin: Backhome
Optimizer
(9k)
1(4)
(5)
1.64
(%5
1150
(5)
1(6)
(%5
1(4)
(5)
1(65
(5i
Adam
65.25
1.54
$1.45
1.57
76.05
3.26
65.41
3.29
55.35
7.57
42.72
7.67
59.21
4.27
SAM
67.28
3.68
$5.05
3.55
79.44
6.76
67.88
6.76
56.18
13.90
39.91
13.95
60.95
5.77
ESAM
67.32
3.75
$3.99
3.60
77.58
6.53
06.54
6.83
54.51
16.05
36.65
16.14
59.04
5.85
GNN
LockSAM
68.38
2.91
$6.26
2.85
79.24
5.29
69.32
5.29
56.33
12:25
45.0
45.42
12.19
62.
6.79
GPN
AE-SAM
67.48
2.93
$1.27
2.81
79.34
5.37
67.25
5.24
56.43
12.36
43.51
12.25
60.96
6.78
FGSAM
69.54
2.33
57.85
2,401
$1.10
3.97
68.50
4.03
57.59
8.56
47.67
8.58
63.54
4.95
PGSAM+
69:40
1.42
57.74
2,00
$0.02
2.43
08.90
2.59
57.39
4.68
46.59
4.64
63.35
3,00
PerMMLP
SAM
Adam
65.80
66.18
0.45
0.74
49.87
$1.69
0.43
1.01
77.20
76.41
0.39
0.71
65
68.39
0.76
0.47
$1.78
49.09
0.33
0.69
35.
42.79
0.81
0.36
57.03
$9.17
0.41
0.79
5.1 Experiment Settings
Baseline. We evaluate our proposed FGSAM and FGSAM+ on SOTA models. The existing models can be divided into two main categories: MAML and non-MAML methods. Two representative models are selected from each category, respectively. as baselines for evaluation (Meta-GCN [42]
and AMM-GNN [37 for MAML models, and GPN [8] and TENT (38) for non-MAML models).
Datasets. We conduct evaluations three widely used real-world benchmark node classification datasets: CoraFull [5), DBLP and ogbn-arXiv [18]. and we use the train/val/test split as in (34
and 24 The comprehensive statistics of datasets shown in Tab. in Appendix [D.]]
Implementation Details. We implement our model by PyTorch [29 and conduct experiments on an RTX-3090Ti. We use Optuna (2] to search the hyper-parameters for each setting. See Appendix D.2 for detailed FSNC learning protocol.
5.2 Evaluation on Real-World Datasets
The results of different models across datasets are summarized in Tab. 2 All the models share a 2- layers architecture with 16 hidden channels. It can be seen that our proposed algorithms FGSAM and FGSAM+ provide better performance than Adam in most cases, and provide comparable performance with SAM. These results support our claim that FGSAM and FGSAM+ can find local minima with better generalization properties. Note that message- -aaggggggpaiiggggg is only used in perturbing parameters, not involved in parameters update (i.e., MLPs). The results further indicate that implicitly involving graph topology in training can make PeerMLPs outperform GNNs. See Appendix D.3 for details.
5.3 Time Consumption
To demonstrate the training speed advantage of our proposed algorithm, we summarize the training time for different models using various optimization methods across three datasets (Tab. ( The results indicate that our proposed algorithm FGSAM demonstrates only slight increase in training cost compared to Adam in most cases. Furthermore, our enhanced version FGSAM+ outperforms Adam in terms of speed in the majority of scenarios. It is worth mentioning that our proposed algorithms achieve superior or comparable performance when compared to both Adam and SAM. See Appendix DD4 for detailed results.
Limitation. For models composed of many non-GNN components (e.g., TENT), the training time on FGSAM+ may be still longer than that on Adam, since it is hardly further reduced.
7

---

Hidden Channels
Layers
Noily Feature
Nosky Edge
64
0b
We 12%
Number of Hodden Channes
Yumber
Standard Deviation of Nosse
Ratio of Nokey Edge
Figure 4: Performance of GPN trained by Adam and FGSAM+ with different settings. Left: Results
with various hidden channels. Middle Left: Results with various model depths. Middle Right: Results with features perturbed by noise of varying standard deviations. Right: Results with edges subjected to various noise ratios.
5.4 Comparison of the Variants of SAM
Training with Different Optimizer. We compare the performance of Meta-GNN and GPN training with different variants of SAM, including original SAM, ESAM (2], LookSAM 23 AE-SAM [19], our proposed FGSAM and FGS AM (Tab. We observe an anomalous phenomenon where ESAM, as an efficient variant of SAM, actually trains slower than SAM. This is because ESAM sorts the sample losses and selects suitable subset at each iteration, an operation that is negligible for image tasks; however, for graph tasks, since GNNs are relatively smaller, the proportion of time consumed by the sorting step is significant, leading an increase in training time. As shown in Tab. proposed method greatly reduces the training time, based on the relationship between GNN and MLP, while maintaining and even achieving superior performance, compared to other optimizers, indicating ours' high efficiency and effectiveness.
The Impact of Perturbing Parameters with Message-Passing, A key point of our work is that we perform parameter perturbation using GNNs, while PeerMLPs (i.e., without message-passing) are used to minimize the perturbed loss. This is significantly distinct from previous SAM methods which shared the same model for both parameter perturbation and loss minimization. So a natural question arises: to what extent does approach benefit from performing parameter perturbation using GNNs? We thus compare our approach to PeerMLPs training with Adam and vanilla SAM. Note that message-passing would be reintroduced during validation and test. From Tab. although the training time of PeerMLPs is shorter than that of GNNs, GNNs outperform their PeerMLPs in most cases. Despite that using PeerMLPs can accelerate the training of GNNs, the topology information is still very important for learning node representations. Thus our proposed FGSAM+ is a better solution, achieving better trade-off between efficiency and performance.
5.5
Ablation Studies
We further verify the consistent effectiveness of our method compared Adam across different settings regarding model implementation and graph property. Due to the computational resource restriction, all experiments here were conducted using GPN on the CoraFull with the 5-way 3-shot setting. We provide additional experiments (e.g., the effect of update interval k) in the Appendix E
The Impact of Network Structure. Here we investigate the effect of hidden dimension and the number of layers on the performance (on the left of Fig. 4. GPN with Adam requires higher hidden dimension (128) achieve relatively high accuracy, whereas GPN with FGSAM+ can attain SOTA even with a small hidden dimension (16). With respect to the number of layers, GPN with FGSAM+ consistently performs better within the range of 1~8 compared GPN with Adam, demonstrating the effectiveness of our proposed method (middle left of Fig, 4
The Impact of Noisy Features and Edges. Here investigate the effect of randomly adding Gaussian noise to features and randomly adding edges during testing (on the middle right and the right of Fig. 4 Specifically, for noisy features, we randomly add Gaussian noise with varying standard deviations to the node features. Meanwhile, for noisy edges, we uniformly and randomly introduce additional edges into the original structure. The results show that GPN with FGSAM+ method can still achieve relatively high performance, compared to GPN with Adam. These results effectively verify the robustness of our proposed method.
5.6 Additional Task on Conventional Node Classfication
Our proposed FGSAM+ also has the potential to be extended to other domains. To demonstrate this, we evaluate the performance of the FGSAM+ on the standard node classification task on both homophilic and heterophilic graphs. For homophilic graphs, we utilize three well-established citation networks: Cora, Citeseer, and Pubmed [32 13] For heterophilic graphs, we include page-page
8

---

Table 4: Results on nine real-world node classification benchmark datasets: Mean accuracy (% ).
Model
Optimizer
Cora
Citeseer
Pubmed
Chamelen
Squirrel
Actor
Cornell
Texas
Wisconsin
Aug
Adam
77.25
55.29
GCN
SAM
FGSAM (ours)
88.36
88.42
77.60
77.30
89.36
58.79
66.16
65.57
53.95
52.51
28.59
29.88
67.30
61.89
63.24
62.70
54,51
55.69
64.48
65,73
PGSAM+ (ours)
88.32
77.52
89.13
64,56
51.14
29.66
68.11
61.62
54.71
64.97
Adam
87.67
76.09
33.74
84.51
OnphSAGB
SAM
FGSAM (ours)
88.36
87.69
76.84
77.13
89.75
89.25
51.34
50.92
39.12
37.44
34.53
33.83
82.43
78,92
81.35
80.27
86.47
84.31
70,05
68.79
PGSAM+ (ours)
88.16
77.21
89.71
50.94
38.87
34.70
$1.35
79.46
86.47
69.65
Adam
76.37
31.46
GAT
SAM
PGSAM (ours)
88.60
88.49
76.78
76.98
87.63
87.24
47.87
46.82
32.35
31.61
29.49
30.41
61.89
59.46
65.95
62.16
55, 29
59.41
59.70
61.23
FGSAM+ (ours)
88.70
77.10
87.74
48.07
32.69
30,60
62.16
64.86
58.04
61.11
networks from Wikipedia, specifically the Chameleon and Squirrel datasets [1], netorr.etworr,
namely Actor [30], and web pages networks, namely Cornell, Texas and Wisconsin [30]. See
Appendix E. for statistics of these datasets. We use data splits (48%/32%/20s)) provided by (30]
and set k- = 2 for FGSAM+. We select three representative baselines, namely the classical GCN 20)
GAT 35 with learnable MP operation, and GraphSAGE [66] with complex MP operation, to
demonstrate the effectiveness of FGSAM and FGSAM+.
As shown in Tab. 4 both FGSAM and FGSAM+ generally outperform Adam and SAM across base models, indicating the potential wide application of our method. We observed that the proposed method achieves greater improvement
heterophilic graphs compared to homophilic graphs, and heterophilic graphs are generally considered more challenging. This indicates that our method can effectively enhance the generalization capability of GNNs. We also provide additional experiments of integrating FGSAM+ with prompt-based FSNC [33 in the Appendix E.3
5.7 Additional Study
Training Loss .... Adwe
SAM
We observe that both FGSAM and FGSAM+ gen- erally outperform the standard SAM across tasks (FSNC and standard node classification). This an interesting finding, as our FGSAM and FGSAM+
(Ours) p=0.05
Taining Lass
SA1 p=0.5
FCSAMA (Oursi pm02
p=2.5
PCSAAC4 [Duss) pm2.5
5A00 p.5.0)
algorithm remove message-passing during the min- imization of the perturbed loss, which is expected to hurt performance. We attribute these counter- intuitive results to the mitigation of the imbalance adversarial game. The training process of SAM-like algorithms entails an adversarial game similar to that in Generative Adversarial Nets (GANs) [4] Prior studies 3 4 28 have demonstrated that imbalanced adversarial games in GANs can give rise to worse results. Both FGSAM and FGSAM+ employ distinct models for perturbation and minimization, which can help alleviate the extent of imbalance. These factors may explain the observed performance discrepancies among the compared algorithms. To verify the explanation, we conduct experiments varying the hyper-parameter p. Specifically, we graphically illustrate the comparative training loss of SAM and FGSAM+ over range of p values in Fig. which reveals that while SAM struggles to converge with higher P values, FGSAM+ consistently achieves convergence. Moreover, it is established that higher o value is conducive larger could potentially enhance performance. Consequently, FGSAM+ is capable of mitigating the imbalanced games issue and tolerating a larger p. which contributes its enhanced performance.
FCSAM (Oursi pm5.
25
100
12b
150
175
200
Epoch
Figure 5: Training loss curves related to differ- ent p across optimizers.
tighter generalization bound, suggesting that a
6 Conclusion
In this work, we study the application of Sharpness- Aware Minimization (SAM) in FSNC to improve model's generalization, since the key for FSNC is to generalize the model to unseen samples. In order to alleviate the heavy computation cost of SAM, we utilize the connection between MLPs and GNNs and use MLPs to accelerate the training of GNNs. However, the low generalization and lack of using graph topology of MLPs also limit its performance. Hence we propose to apply GNNs to perturb parameters for generalization and use MLPs to minimize the perturbed training loss for conducting the proposed FGSAM. Moreover, we reuse the GNN gradient in perturbation in minimization for better including topology information. We further reduce the training time by conducting exact FGSAM update at every & steps and approximate FGSAM's gradient with reusing information in the intermediate steps. Finally, the extensive experiments demonstrate the effectiveness and efficiency of our proposed methods.

---

Acknowledgements
Jing Tang's work is partially supported by National Key R&D Program of China under Grant No. 2023YFF0725100 10. by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410 and U22B2060, by National Language Commission under Grant No. WT145-39, by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2023A15151 10131, by Guangzhou Municipal Science and Technology Bureau under Grant No. 2023A03J0667 and 2024A04J4454, and by Createlink Technology Co., Ltd. Xiaochun Cao's work is supported in part by National Natural Science Foundation of China (No. 62411540034), in part by Shenzhen Science and Technology Program (Grant No. KOTD202211010935555
References
[1] Momin Abbas, Quan Xiao, Lisha Chen, Pin- Yu Chen, and Tianyi Chen. Sharp-maml: Sharpness- aware model-agnostic meta learning. In International conference on machine learning, pages 10-32. PMLR, 2022.
[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2633 2019.
[3] Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adver- sarial networks. In International Conference on Learning Representations, 2017.
[4] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. International conference on machine learning, pages 214-223. PMLR, 2017.
[5] Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsu- pervised inductive learning via ranking. In ICLR, 2018
[6] Yuhan Chen, Yihong Luo. Jing Tang, Liang Yang, Siya Qiu, Chuan Wang, and Xiaochun Cao. Lsgnn: Towards general graph neural network in node classification by local similarity. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23. pages 3550-3558. International Joint Conferences Artificial Intelligence Organization, 2023. Main Track.
[7] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovie. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations. https///openreview nel/forum, 2021.
[8] Kaize Ding, Jianling Wang, Jundong Li, Kai Shu, Chenghao Liu, and Huan Liu. Graph prototypical networks for few-shot learning on attributed networks. In CIKM, 2020.
[9] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient awwre minimization for improved training of neural networks. arXiv preprint arXiv:2110.03141, 2021.
[10] Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness- aware training for free. Advances in Neural Information Processing Systems, 35:23439-223 204--,345,,,,
[11] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learing for fast adapta- tion of deep networks. In JCML, 2017.
[12] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[13] Lise Getoor. Query-driven active surveying for collective classification. 2012.
[14] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.

---

[15] Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. In Proceedings of the web conference 2021, pages 2559-2567, 2021.
[16] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, 2017.
[17] Xiaotian Han, Tong Zhao, Yozen Liu, Xia Hu, and Neil Shah. Mlpinit: Embarrassingly simple gnn training acceleration with mlp initialization. arXiv preprint arXiv:2210.00102, 2022.
[18] Weihua Hu, Matthias Fey, Marinka Zitikk Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurlPS, 2020.
[19] Weisen Jiang. Hansi Yang. Yu Zhang, and James Kwok. An adaptive policy to employ sharpness- aware minimization. In The Eleventh International Conference on Learning Representations. 2022.
[20] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint wrwiv:/609....... 1609.02907 2016.
[21] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914. PMLR, 2021.
[22] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[23] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsich, and Yang You. Towards efficient and scalable sharpness-awen aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[24] Yonghao Liu, Mengyu Li, Ximing Li, Fausto Giunchiglia, Xiaoyue Feng, and Renchu Guan. Few-shot node classification on attributed networks with graph meta-learning, In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 471-481, 2022.
[25] Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang- Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, editors, Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pages 417-428. ACM, 2023.
[26] Sitao Luan, Chenging Hua, Minkai Xu, Qincheng Lu, Jiaqi Zhu, Xiao Wen Chang, Jie Fu, Jure Leskovec, and Doina Precup. When do graph neural networks help with node classification: In- vestigating the homophily principle on node distinguishability. arXiv preprint arXiv:2304. 14274, 2023.
[27] Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? arXiv preprint 2rXiv:2106.061 2021.
(28) Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, 32, 2019.
[30] Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint 2rXiv:2002.05224 2020.
11

---

[31] Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi-Scale Attributed Node Embedding. Journal of Complex Networks, 9(2), 2021.
[32] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Ai Magazine, 2008.
[33] Xiangguo Sun, Hong Cheng, Jía Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopu- los, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 2120-2131. ACM, 2023.
[34] Zhen Tan, Song Wang, Kaize Ding, Jundong Li, and Huan Liu. Transductive linear probing: novel framework for few-shot node classification. In Learning on Graphs Conference, pages 4-1. PMLR, 2022.
[35] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1rXiv:1710.10900 2017.
[36] Kuansan Wang. Zhihong Shen, Chiyuan Huang, Chich-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 2020.
[37] Ning Wang, Minnan Luo, Kaize Ding, Lingling Zhang, Jundong Li, and Qinghua Zheng. Graph few-shot learning with attribute matching. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, 2020.
[38] Song Wang. Kaize Ding, Chuxu Zhang, Chen Chen, and Jundong Li. Task-adaptive few-shot node classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1910-1919, 2022.
[39] Yihan Wu, Aleksandar Bojchevski, and Heng Huang. Adversarial weight perturbation improves generalization in graph neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 10417-10425, 2023.
[40] Chenxiao Yang, Qitian Wu, Jiahua Wang, and Junchi Yan. Graph neural networks are inherently good generalizers: Insights by bridging GNNs and MLPs. In The Eleventh International Conference on Learning Representations, 2023.
[41] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR, 2016.
[42] Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng. Meta-gnn: On few-shot node classification in graph meta-learning. In CIKM, 2019.
[43] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvomek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08066 2022.
12

---

A Potential Broader Impact
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
B Algorithm
Algorithm 1 Training with FGSAM and FGSAM+.
Require: G. Chuse. learning rate 77, radius FGSAM update interval k,, adaptive ratio
Ensure: A flat minimum solution in.
Initialize weights wt: for f. + 0 to
do Sample training task from & and Chasei only for FGSAM
Vanilla grad ganh = V LT, (Wii fgan);
Perturbed weights é p [[geuT
geta
FGSAM-grad 9EGSAM = Again + Vio, LT, (wws; Smip) wm te5
### only for FGSAM+ if t% then
# the actual FGSAM-update
Vanilla grad g V w LT. (Wi, /gnn
Vanilla grad mipp = V w,, LT, (we; mipp );
Perturbed weights é p -
Topology-grad g3 = g
|g
g g
g mp
gm
1 min
[
SAM-grad 9s V w. LT; (w) fmip)
w +
Flatness gradd go 9: |1
g
g
g
11g.
rGSAM-grad gEGSAM Xg gmn
9si
else
# approximate FGSAM- -radiiggrdientt
Vanilla grad ml = V w. LT, (w))
Fmip): );
Approx gnn-grad gmm = g + g 1
ga
Approx FGSAM-grad gEGSAM mlp + ag 9-
1g
Agsun
end if
Update weights: Wc+1 W: OFGSAMi end for
i t- WT.
C Proof
The linear classifier for K-classification problems be formulated K(K-1) binary classification problems.
Hence study the classification between class C, and Cp without loss of generality.
The distribution of original features from different classes follows:
2; N I) Co
(11)
2;; N I) ; C
13

---

The distribution of filtered features from different classes follows:
(p 4) No + q
h4 ~ A
i Co.
p+ (K 1)q
deg (i)
(12)
(p (p- q) pp + q I
h.
E
p+ (K 1 deg (i)
For simplicity, we denote ñáo (p-0)uu-aKa and P (p 11,4,, p+(K-1y
Following 27). the optimal classifier of original features constructs decision bound P = { (e/w"" x b), where w Me Mo -
|l. b Mb M4 Similarly, the optimal classifier of filtered
features constructs a decision bound p' (h/w" h -w where w' P.. 14 P.. B
b' t4
And we have io - up w b b':
p-q p-(K-1Jo
(Mo - Mp), hence we have w = w'. Then we verify whether
w
w 'T
No + ptp
2
(p 9) + q
(p - My + qgK
w
2
++ (K 1)g
++ (K - 1)
w
(Mo + mp) (1- 2))
(Mo Hp))
(13)
+ (1 {) 2 [T 14 + f - Mo + 11 + 1p)
w
2 no + 1kp) + (1 2) (1=
Ho + Hp
2
1
no + [bp) + 2) w² (1-
Ho + Up
2
where À pI'"-
Then we show (1to - 1p))
ii - -otttele O.
From ||p.: 14jl12 D, have:
| - -1l12 111pp -1112,
(14)
which gives:
(No 12) (Mo (t) =(1p a1) (Hz fL)
1 Mo 2nd + µ P u Mo 24 T + p p
T
(15)
1 un 2/1 fr u Ms 21 u μ
Hence we have:
(po - -1p) TT
N + Mp
2
M/ T jt
Mo Hp
T
T Mo + Hy
11o
11 ft + php
(16)
1 T
1. i
11 Mo
11 14 11p11 1bp
Combining Eq. 13 and Eq. (16). we have:
w TT D' wnth
(17)
which means p = P
This completes the proof.
14

---

Table 5: Statistics of evaluation datasets
Datasets
# Nodes
Edges
# Features
# Classes
Class Split
CoraFull
19,793
63,421
8,710
70
40/15/15
DBLP
40,672
288,270
7,202
137
80/27/30
ogbn-arXiv
169,343
1,157,799
128
40
20/10/10
Table 6: Hyper-parameters Search Space.
Hyyper-paamees
Search Space
MAML-based models:
learning rate
{0.05, 0.01, 0.001, 0.0001)
weight decay
{0.0, 0.001, 0.0005)
dropout p a
{0.0, 0.1, 0.3, 0.5, 0.7, 0.9)
(0.01, 0.05, 0.1, 0.15, 0.2, 0.5, 0.8, 1.0, 1.2)
(0.5, 0.7, 0.9)
non-MAML models:
learning rale finetune
{0.5, 0.1, 0.01, 0.001) {0.05, 0.01, 0.003, 0.001, 0.0001)
learning rate meta
weight decay
(0.0, 0.001, 0.0005)
dropout P a
{0.0, 0.1, 0.3, 0.5, 0.7, 0.91
(0.01, 0.05, 0.1, 0.15, 0.2, 0.5, 0.8. 1.0, 1.2)
(0.5, 0.7, 0.9)
D Experiments details
D.1 Datasets Description
CoraFull is extension of the prevalent dataset 'Cora' 41..
citation network dataset.
On this graph, nodes represent papers and edges represent citation links. The nodes are labeled on the paper topics. Node attributes obtained using bag-of- ooggoofwordss for the title and abstract of the paper.
DBLP is also a citation network, where nodes represent papers and edges represent the citation between papers. Specifically, the node attributes are generated by the abstract and the node labels are based on the paper venues.
ogbn-arXiv is citation network among all Computer Science arXiv papers based on MAG [36]. Node represent papers and edges are citations links. The node attributes are obtained using Akip-gram on abstract of papers. The nodes are labeled by the subject area.
D.2 Implementation Details
Specifically, we implement our model by PyTorch 29 and conduct experiments 24GB Nvidia RTX3090Ti, according to the training protocol Algorithm Repeat number R 5, patience P 10, SAM update interval & 2, validation interval I 10, validation number V 20, test number W 100. For MAML models max epochs T = 500, and for non-MAML model max epochs T = 1000. We evaluate method under various settings, i.e., N (5, 10), K {3, 5). but set N = [2,5), for Coauthor-CS CS dataset. We use Optuna 2 for prree--prraeeers searching for all models with various optimizers, the search space is shown in Tab.
Note that we further split Chase into two disjoint class set: training class set Crr and validation class set Cral, such that Cruse Cr U C.ai and Cr O Ceal Ø. Overall, we use Crr and Cval for train and validation in the meta-training stage, respectively, and use Crovel for meta-test. We split into Cr, Con and Cnovel according to the class split ratio in Tab.
15

---

Algorithm 2 Training Protocol of FSNC Task
Require: 9, Cr. Cual, Cravel, repeat number R. max epochs T, patience P. validation interval I, validation number V, test number W.
Ensure: A trained model f, model's accuracy
Initialize f.8 8.84 4- (}.
repeat R times
for p + to R - 1 do
Initialize Sheat f- O, Stest - U, p 0:
# meta-training for 0
# training
Sample training task T: {St, 2+} from Cr:
Optimize model f on T;;
# validation
if t% = ( then
Sample V validation tasks Tval from Cval; Compute mean accuracy &val on Tn by f; Syal Shest then
Shest 8val, +- 0:
else
p p+ end if
# early-stop
p = then
break;
end if end if
end for
# meta-test
Sample W test tasks Test from Crovel;
Compute mean accuracy Stest on these tasks using model f;
s= s Stesti
end for f f f,&+ mc..
Table 7: Accuracy on the baseline with different optimizer. '5N3K' denotes 5-way 3-shot setting.
Corsfall
DBLP JONOK
agbn-arXiv
5NIK
LONSK
$4.57.02
70.35244
54.77
52.90
25.10.15
31.79..0
45.53
39.30-14
$2.66.14
57.05-4
28.92-
31.03-104
POSAM+
71.34uu
58.73
01.61
12
94.24:10
68.77u
72.18
-2
58.05.54
7292
57.29 47.55
-1.02-14
60.40.200
71.31
11.95
57.79-13
28.71-4.
68.472m
53
50.54
23
2m
49.42
50.75
30.57
7167
60.15am
60.11
24.0
62
71.70am
48.69
22.57
w/ FGSAM+
72.79
18.18
59.59
62.61
11.24
85.07.3
70.37
21.32
51.02
50,69.903
33.60
24.05
65:33gm
50.45
76.05
65
43.72
w/
67.28141
55.06.1
52,50
79,64119
77.66
17.88
87.78128
56 18
24
30.91
92:10
w/ PGSAM
69.5
89
30:37
$7.85
56.49
10
79.61
68.50
60.46
$7.58
58.23
47.67
20
w FGSAM+
69.40
69.96.2e
57.74
36.10
50,02 us
79,09
6854
09.51
57.39
28.04:2
46.59.124
49,49 1324
TENT
71.24a2s
75.49.1.
57.29
60.35-11
82.74-am
690042e
71.79.2a
60.44.5m
67.54-21
47.14-4a
54.88.mm
wi SAM
71.28iee
75.29
36.86113
61.5-19
S2.13
85.10
68961
73.02.1x
63.58i2a
69.30
50.79.u15
55.21
w/ EGSAM
71.10g
76.72
57.862m
63.71220
50.90gm
20.3g
73.41.r
63.8
15.
51.32
57.08g10
w/ PGSAM+
72.5.mm
73.77
58,37am
63.04.m
53.64
85.07
71.15
73.72am
66.20ggu
62.14.i
50,66un
53.56am
D.3 Evaluation Results with Standard Deviation
In Tab. 7 and Tab. we present the detailed results of Tab. and Tab. 7 with standard deviation, respectively.
16

---

Table 8: Results on nine real- woold node classification benchmark datasets: Mean accuracy (%). The
best results are denoted as boldface.
Model
Optimizer
Cora
Citeseer
Pubmed
Chameleon
Squirrel
Actor
Cornell
Texas
Wisconsin
Adam
88.36
17.25
88.71
65.06
52.49
28.54acas
61.08
60.27
GON
SAM
FGSAM
a8.36
s8.42
27.30
77.60
89.36.1.
28.79.14
66. 16
65.57
$2.51
53.95
28.59
29. 49s8
61.89 -9
67 67.30
62.70
63.24
54.51 1442
55.69 4431
FGSAM+
88.32gr
77.52gm0s
89. s
64.50 64.56
51.14g
68.1
61.62ax0
Adam
87.67
76.09
89. 15
50.33
37.61g
33.74
11
78.38
84.51
GrahhSAGE
SAM
87.69atm
89.25 89.25 26.44
50.92atm
57,44211
33 33.83
78. 22.92
80.27
84, 2443
88.36
77.13gmu
g1.34
34.53
82.4
81,35
86.47
FGSAM+
88. 16
77.21
89.71
50.94gt
38.8721a
34.70aes:
81.35
79.462-4
86.4
88 38.32
76.37
87.48.1m
46.51
31.46au
29.45atm:
19
62.16
a5.49
GAT
SAM
FGSAM
88.49
88.60
76.78 r044
76.98
87.6 47.63atm
17524
47.87
46.82
29.49 31.61
32.35g
30.41
59.46
61. 61.89
65.95
62.16
55.29 165
59.41
FGSAM+
88.70g13
77.10
87.74
48.07.11
30.60arm
58. 58.04
Table 9: Time consumption comparison. The results stands for the time (sec.) consumed in 200 episodes training (only consider the feed-forward and -backward).
Corafull
DBLP
ogbu-arXiv
Setting
SN3K
SNSK
1ON3K
I0N5K
5N3K
SNSK
10N3K
1ONSK
5N3K
SN5K
JON3K
IONSK
Meta-GCN
9.56
9.58
9.38
9.40
17.61
17.60
17:50
17.59
41.09
40.98
40.96
40.92
SAM
19.16
82.54
82.65
82.40
82.58
w/ FGSAM
10.91
10.82
10.79
10.80
19.15
19.22
19.11
19.12
42.50
42.54
42.45
42.45
FGSAM+
6.58
6.48
6.51
6.48
10.77
10.77
10.51
10.44
22.17
22.21
22.02
22.06
AMM-GNN
15.03
15.04
14.94
15.00
26.71
26.74
26.72
26.76
42.27
42.39
42.30
42,37
SAM
30.91
30.93
30.66
30.83
54.63
54,85
54.69
54.87
84.71
85.12
84.77
85,13
FGSAM
17.55
17.89
17.51
17:44
30.08
30.13
30.28
30.16
44.32
44.45
44.43
44.46
FGSAM+
9,99
10.05
9.97
9.99
16.14
16.44
16.25
16.22
23.24
23,24
23.13
23.15
GPN
1.84
1.93
1.87
1.92
3.26
3.26
3.29
3.29
7.67
7.74
7.67
7.73
SAM
3,68
3.69
3.55
3.58
6.76
6.80
6.76
6.81
15.90
16.02
15.95
16.06
2.33
2.19
2.40
2.39
3.95
4.03
8.58
8.58
w/ FGSAM+
1.62
1.48
2.06
2.14
2.43
2,51
2.59
2.70
4.68
4.66
4.64
4,65
TENT
7,58
8:29
13.14
14:49
7.79
8.70
13.65
15.30
9.21
9.98
15.53
16.87
SAM
15.05
16.89
26.67
29.50
15.98
17.49
27.70
30.25
19.28
20.53
31.78
34.
14,85
17.60
w/ FGSAM+
11.13
11.04
19.10
19.15
11.47
11.50
19.89
19.64
12.53
12.61
21.14
21.15
D.4 The Full Results of Time Consumption
Here we present the detailed results of training time consumption of different optimizers across various datasets. Tab, Iindicates that our proposed algorithm FGSAM demonstrates only slight increase in training cost compared to Adam in most cases. Furthermore, our enhanced version FGSAM+ outperforms Adam in terms of speed in the majority of scenarios. worth mentioning that our proposed algorithms achieve superior or comparable performance when compared to both Adam and SAM.
As mentioned before, for models composed of many non-GNN components (e.g. TENT), the training time FGSAM+ may be still longer than that on Adam, since is hardly further reduced.
E Additional Experiments
E.1
Statistics Of Benchmark Datasets Node Classification
Table 10: Benchmark datasets statistics for node classification
Cora
Citeseer
Pubmed
Chameleon
Squirrel
Actor
Cornell
Texas
Wisconsin
Nodes
2708
3327
19717
2277
$201
7600
183
183
251
Edges
5278
4552
44324
18050
108536
15009
149
162
257
# Classes
7
6
3
5
5
5
5
Features
1433
3703
500
2325
2089
932
1703
1703
1703
H(())
0.81
0.74
0.80
0.28
0.24
0.38
0.57
0.41
0.45
17

---

Table 11: Performance of different update interval k.
Corafull
DBLP
acc (%)
time (s)
acc (%)
time (s)
GPN FGSAM
69.54
2.33
80.10
3.97
2
69.40
1.62
80.02
2.43
GPN w/ FGSAM+
5
70.02
0,93
78.10 10
1.49
10
67.03
0.69
75.91
1.24
Table 12: Performance of prompt-based FSNC on Citeseer.
shots
shots
Setting
acc (%)
acc (%)
Fl
ProG 133
59.50 60.33
57.75
76.50
76.61
FGSAM+
58.43
77.00
77.21
E.2 The Effect of Update Interval in FGSAM+
Here we study the effect of update interval à in FGSAM+. can be observed from Tab, I that as k increases, the performance decreases, but meanwhile training time also decreases. This indicates that the possibility of choosing achieve a better trade-off between performance and efficiency. We note that the performance drop with increasing k seems to be larger compared to LookSAM [23 in computer vision tasks. This indicates the importance of the perturbation step in FGSAM+, as not only introduces information about flat minima, but also incorporates neighbor information in training. Therefore, we recommend setting as prior optimal update interval to avoid large information loss.
E.3 Integrating with Prompt-Based FSNC
Recently, there are many prompt-based methods 25 have been developed, showing promising performance in FSNC. Hence, we investigate how our method performs in such prompt- based FSNC task. Note that under this setting, the proposed method used in prompt tuning instead of training. As shown in Tab. 12 our method improves the baseline [3 with remarkable margin.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed some limitations about the proposed FGSAM+ in Sec. 5.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We have provided assumptions and proofs in the main body and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Necessary information provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Data is publicly available and code is also available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccpppbccccceec guwdes/ddeeeeee Cooeeeoodeesssseeeeeee http:///////uuddddd//eee/eeeess/
ssiniiiicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Detailed are provided in the paper (main body and appendix). Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Due to the page limit, the standard deviation is provided in the appendix (Tab. 7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is provided Sec. 5.1 and Sec.
5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We confirm that we have adhered to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential broader impacts in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no high-risk issues in our model trained for FSNC tasks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all the related papers, the papers of models and datasets
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will submit the code used along with the supplement materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "Y4bJkpdpv3", "reviewer": "Reviewer_nij7", "summary": "The paper proposes a method for few-shot learning on graphs leveraging sharpness-aware minimization (SAM) from the vision community. The paper explains SAM as a technique for gradient perturbation during training to push the parameters to \"flatter\" regions of the loss space in hopes of achieving better generalization in the few shot setting. However, SAM is inherintely slower, requiring two backward/forward passes during training; one to compute the loss gradient and a second to determine the perturbation direction. The authors propose multiple ideas to overcome the added computation and to leverage graph topology. The authors strategically substitute Graph Neural Networks (GNNs) with Multilayer Perceptions (MLPs) to avoid the burden of message passing in some parts of their algorithm while preserving graph topologies in others. The authors propose a second scheme that leverages iterations of an approximate perturbation scheme with periodic exact evaluations of SAM. The authors present ablation studies on multiple SOTA models and 9 datasets and show improvements in training times and accuracy.", "questions": "(1) I find the paragraph containing lines 137-145 hard to read due to how the ideas are presented:\n\n\"Inspired by previous work...we propose to remove MP during training, but reintroduce it..Reintroducing MP after training can improve the performance significanly but still cannot surpass GNNs...\"\"Henve we propose minimizing training loss on PeerMLPs but minimizing the sharpness according to GNNs.\n\nThere seems to be some logical oscillation in the text. First it seems that a scheme similar to PeerMLP is uses where the GNN is removed during training and injected back during inference. In the next few sentences, this seems to be abandoned and the GNN brought back for the SAM portion of training. I believe the experiments and ideas really did implement the latter formulation but either I'm misunderstanding or this paragraph could use some work to be clearer.\n\n(2) In table 8 in the Appendix, I don't believe the best results are actually in boldface. Maybe this was due to my printer or perhaps the table needs to be reformatted?", "limitations": "No limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present multiple novel approaches to overcome computational challenges applying SAM techniques to structured data to achieve better generality in the few shot case. I only have a few style suggestions at the beginning of the paper but otherwise the paper does a good job of explaining multiple, complex ideas in a clear way. The paper does present a significant result, blending ideas from PeerML and SAM from the vision community to contribute to making structured models more generalizable.", "weaknesses": "I think the paper is very good and clear in most cases. The only weaknesses I would note (and this may be subjective) would be to make some style improvements in the beginning of the paper.\n\n(1) In the abstract (line 14) Moreover, our method ingeniously reutilizes... is a bit grandiose; I would prefer a more scientific tone and delete \"Moreover\" and \"ingeniously\"; \"Our method reutilizies...\n\n(2) Remove \"for the first time\" in 20. It is clear that the ideas are novel.\n\n(3) Line 50 is the first time Message Passing is abbreviated as MP. I would follow the convention used in the paper for the introduction of other abbreviations where the first letter of the abbreviated character is in bold and capitalized.\n\n(4) Line 93 \"as follows\" is not followed by an equation but another sentence. Maybe the equation should be presented after the \"as follows\" text and the next sentence come after the equation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback and valuable comments on our manuscript. We found that your comments are really valuable to further improve our manuscript. Below, we address each of your concerns and questions:\n\n> **Q1:** I think the paper is very good and clear in most cases. The only weaknesses I would note (and this may be subjective) would be to make some style improvements in the beginning of the paper.\n\n**Response to Q1:** Thanks for your careful pointing out. We will improve our writing in the revision.\n\n> **Q2:** I find the paragraph containing lines 137-145 hard to read due to how the ideas are presented:\n\"Inspired by previous work...we propose to remove MP during training, but reintroduce it..Reintroducing MP after training can improve the performance significantly but still cannot surpass GNNs...\"\"Henve we propose minimizing training loss on PeerMLPs but minimizing the sharpness according to GNNs.\nThere seems to be some logical oscillation in the text. First it seems that a scheme similar to PeerMLP is uses where the GNN is removed during training and injected back during inference. In the next few sentences, this seems to be abandoned and the GNN brought back for the SAM portion of training. I believe the experiments and ideas really did implement the latter formulation but either I'm misunderstanding or this paragraph could use some work to be clearer.\n\n**Response to Q2:** Thank you for your valuable feedback, which has highlighted an area of our text that could benefit from greater clarity. In response, we would like to clarify that our approach indeed adopts the latter formulation mentioned, wherein MP is partially removed during training, with the reintroduction of GNN for the SAM portion of the training. The intent of our original discussion was to illustrate the relative performance of PeerMLP, acknowledging its merits yet also its limitations, thereby motivating our proposed method. To address the issue raised, we will revise the mentioned text to ensure the presentation of our idea is more clear and logical.\n\n> **Q3:** In Table 8 in the Appendix, I don't believe the best results are actually in boldface. Maybe this was due to my printer or perhaps the table needs to be reformatted?\n\n**Response to Q3:** Thanks for pointing out this! Upon review, we found that the boldfacing issue in Table 8 was indeed a formatting oversight on our part. We will correct this in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ohbXAgkCL", "reviewer": "Reviewer_4EHj", "summary": "This paper focuses on efficient graph neural network (GNN) training in few-shot node classification (FSNC) problem by extending sharpness-aware minimization (SAM) for reducing the computational cost and improving the generalization of GNNs on unseen classes. The training phase is accelerated by perturbing the parameters of GNN and then minimizing the perturbation loss of GNN without the message passing mechanism (MLP). Experiments have been conducted to verify the effectiveness and efficiency of the proposed method.", "questions": "1. Since this work focuses on solving FSNC problems, the authors should provide a comparison with the state-of-the-art FSNC methods, such as COSMIC [3] and COLA [4], in theoretical analysis and experiments.\n\n[3] Contrastive meta-learning for few-shot node classification. SIGKDD, 2023\n\n[4] Graph contrastive learning meets graph meta learning: A unified method for few-shot node tasks. WWW, 2024\n\n2. Why adding 25% noisy edges yields better results than adding only 15%? in Figure 4? The authors should discuss or explain this issue in details.", "limitations": "The authors describe some limitations of the proposed method in time consumption section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting to incorporate SAM technique with GNN by removing MP during training and reintroducing MP in inference, which is reasonable to improve the generalization of FSNC.\n2. The proposed method could be conveniently integrated into existing GNN-based few-shot node classification models, which is verified in the experiments.\n3. The carefully designed experiments highlight the effectiveness of the proposed method in reducing the computational costs of GNN training.\n4. The paper is well-organized and explains the method very clearly. In addition, the landscape visualization and toy case analyses also make the motivation and ideas easy to understand.", "weaknesses": "1. There are related work on applying SAM to few-shot tasks (Sharp-MAML) [1] and performing SAM every $k$ steps [2], and the innovation of this work is not significant compared with the above work.\n\n[1] Sharp-maml: Sharpness-aware model-agnostic meta learning. ICML, 2022\n\n[2] Towards efficient and scalable sharpness-aware minimization. CVPR, 2022\n\n2. The experiments only compare with small-sample node classification models and variants of SAM from 2022, lacking comparisons with the latest baseline methods of FSNC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and reviews. We provide a point-by-point response below. Hope this can address your concern and make things clearer.\n\n> **W1:** There are related work on applying SAM to few-shot tasks (Sharp-MAML) [1] and performing SAM every k steps [2], and the innovation of this work is not significant compared with the above work.\n\n**Response to W1:**\nBoth Sharp-MAML [1] and LookSAM [2] focus on vision data, while our work is crafted in the context of graph data. Specifically, we utilize GNNs for parameter perturbation while employing MLPs to minimize the perturbed loss. \nOur experimental results demonstrate that directly applying SAM-like algorithms from the vision domain to graphs does not yield satisfactory performance (refer to Table 2, Table 3, and Figure 5), while our method is not only faster but also better. This indicates the necessity for SAM variants that are specifically designed for graph data.\nFurthermore, although sharp-MAML applies SAM to few-shot tasks, their algorithm is only designed for MAML models. In contrast, our work can be applied to both MAML and non-MAML methods in GNN-based FSNC tasks.\nMost notably, from the perspective of SAM, our work is crafted for graphs by its unique property, enabling the first SAM-like algorithm that can be faster than the base optimizer. This effectively turns SAM's core drawback of slower training speed into an advantage, distinguishing our work from previous SAM-like works.\n\n> **Q1:** Lacking comparisons with the latest baseline methods of FSNC such as COSMIC [3] and COLA [4], in theoretical analysis and experiments. \n\n**Response to Q1:**\nOur proposed FGSAM is fundamentally orthogonal to GNN-based FSNC models, such as COSMIC and COLA, due to its unique positioning as a SAM-like optimizer designed to leverage the intrinsic properties of GNNs. \nWe selected classical and widely used FSNC models and NC models for evaluation. The strategy is borrowed from previous SAM works, where they also select classical and widely used models in the vision domain. We believe the selected baseline is diverse enough to verify the effectiveness of our proposed methods. \n\nHowever, we are glad to provide additional experiments on integrating FGSAM with state-of-the-art FSNC methods, such as COSMIC [3].\n\n|           | Corafull |  | DBLP  |  |\n|-----------|----------|------|-------|------|\n|           | acc      | time | acc   | time |\n| **COSMIC**   | 75.74    | 3.95 | 80.80 | 3.78 |\n| **COSMIC+FGSAM**  | 77.11    | 5.66 | 81.93 | 5.00 |\n| **COSMIC+FGSAM+** | 76.99    | 3.95 | 81.46 | 3.35 |\n\nDue to the time limitations, we evaluate our proposed method on COSMIC with 5N3K setting only on CoraFull and DBLP. As shown in the above table, FGSAM/FGSAM+ can effectively improve the performance of COSMIC, demonstrating the superiority of our proposed method.\n\n> **Q2:** Why adding 25\\% noisy edges yields better results than adding only 15\\%? in Figure 4? \n\n**Response to Q2:** \nFirstly, we found a typo in the coordinate axis labels in the original manuscript. The correct coordinate axis should be 0\\%, 1\\%, 5\\%, 10\\%, 15\\%, not as previously misstated. Consequently, the observed phenomenon is that adding 10\\% noisy edges yields better results than adding only 5\\%.\n\nWe appreciate the reviewer's interest in this intriguing phenomenon. In response, we conduct similar experiments on the AMM-GNN to validate the consistency of this phenomenon across different models. The results are presented in the following table:\n\n| edge noise         | 0\\%   | 1\\%   | 5\\%   | 10\\%  | 15\\%  |\n|--------------------|-------|-------|-------|-------|-------|\n| **AMM-GNN**   | 72.92 | 70.50 | 69.98 | 70.33 | 68.41 |\n| **AMM-GNN+FGSAM+** | 72.79 | 71.94 | 70.40 | 70.85 | 68.73 |\n\n\nIt can be seen that adding 10\\% noisy edge also yields better results than adding 5\\%, similar to the results on GPN. Hence, we suppose that this phenomenon may be attributed to the uniform introduction of a 10\\% noisy edge, which perhaps aligns more closely with certain inherent characteristics of the dataset, thereby mitigating the extent of performance degradation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O8KBDtX1Fq", "reviewer": "Reviewer_phcW", "summary": "The paper introduces an innovative approach to improve the generalization capabilities of Graph Neural Networks (GNNs) in few-shot node classification tasks. The authors propose a novel algorithm, Fast Graph Sharpness-Aware Minimization (FGSAM), which incorporates sharpness-aware minimization (SAM) techniques into GNN training but reduces the typical computational overhead by integrating multilayer perceptrons (MLPs) for efficiency. This method not only demonstrates superior performance on few-shot learning tasks compared to traditional techniques but also offers significant computational advantages, particularly in reducing training times.", "questions": "1. What is the standard deviations for the results in table 4? Are the improvements significant?\n\n2. Is there any theoretical justification for PeerMLP?\n\n3. What is g in equation (8)?\n\n4. The proof of theorem 4.1 is confusing. Do you assume all nodes have the same degree deg(i)?", "limitations": "the authors adequately addressed the limitations", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "originality: medium\nquality: medium\nclarity: medium\nsignificance: medium", "weaknesses": "Some experiments are missing and the proof need to be double checked carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and reviews. We provide a point-by-point response below. Hope this can address your concern and make things clearer.\n\n> **Q1:** What is the standard deviations for the results in table 4? Are the improvements significant?\n\n**Response to Q1:** The detailed standard deviations for the results presented in Table 4 are provided in Table 8, located in Appendix D.3. In order to assess the significance of the observed improvements, we performed paired t-tests on the results obtained from ten repeated trials comparing SAM with FGSAM and FGSAM+, respectively, across each dataset. The null hypothesis for these tests was that the performance of SAM would not be inferior to that of FGSAM or FGSAM+. The outcomes of these statistical tests are encapsulated in the **Table A** and **Table B**. In these tables, the plus symbol denotes levels of statistical significance, with +, ++, and +++ corresponding to significance levels of 5\\%, 1\\%, and 0.1\\%, respectively. The results indicate that both FGSAM and FGSAM+  achieve statistically significant improvement compared with SAM in most cases.\n\n**Table A. FGSAM vs SAM**\n|          | Cora | Citeseer | Pubmed | Chameleon | Squirrel | Actor | Cornell | Texas | Wisconsin |\n|-----------|------|----------|--------|-----------|----------|-------|---------|-------|-----------|\n| **GCN**       | -    | +        | +++    | +         | ++       | +++   | +++     | ++    | ++        |\n| **GraphSAGE** | +++  | +++      | +++    | +++       | +++      | +++   | +++     | +++   | +++       |\n| **GAT**       | +    | ++       | ++     | ++        | +++      | +++   | +++     | +++   | +++       |\n\n**Table B. FGSAM+ vs SAM**\n|          | Cora | Citeseer | Pubmed | Chameleon | Squirrel | Actor | Cornell | Texas | Wisconsin |\n|-----------|------|----------|--------|-----------|----------|-------|---------|-------|-----------|\n| **GCN**       | -    | +        | +++    | -         | ++       | ++    | +++     | -     | +         |\n| **GraphSAGE** | ++   | +++      | +++    | -         | +++      | +++   | +++     | -     | +++       |\n| **GAT**       | +    | ++       | +++    | ++        | +++      | +++   | +++     | +++   | +++       |\n\n\n> **Q2:** Is there any theoretical justification for PeerMLP?\n\n**Response to Q2:** Our Theorem 4.1 in the paper demonstrates that under the linear case, whether the MP  layer is used or not, the optimal decision bound is the same in the context of K-classes CSBM. We believe this theorem can give some valuable insight in using PeerMLP. Moreover, our approach reintroduces the graph topology in minimization. This is essentially different from PeerMLP which does not involve any graph topology information in training.\n\n\n> **Q3:** What is g in equation (8)?\n\n**Response to Q3:** The term $g$ is defined as $\\nabla_w L_{D_{\\text{tr}}}(w)$, referred to line 189 in page5. This represents the *vanilla gradient* of the loss function $L$ with respect to the weights $w$, computed on the training dataset $D_{\\text{tr}}$.\n\n\n> Q4: The proof of theorem 4.1 is confusing. Do you assume all nodes have the same degree deg(I)?\n\n**Response to Q4:** We do not assume all nodes having the same degree. The theorem and its proof are based on the definition of the K-classes CSBM, as detailed between Lines 218 and 226 of our manuscript. In the context of CSBM, while it is true that the expected degree of nodes within the same class is identical, it is crucial to note that this does not necessitate uniformity in the actual degrees across all nodes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is concerned with few shot node classification setting of graph neural networks. The difficulty of such task lies in quick and generalizable adaptation of the model using only few labels. To enhance such generalization, authors propose to incorporate sharpness aware minimization (finding flat local optima in the loss surface) during GNN adaptation. Direct application of SAM is computationally expensive (2x), and authors propose new modifications to SAM. Namely, substitution GNNs with multilayer perceptrons (for loss minimization)  and a setting when perturbations are approximated for few iterations and subsequently recalculated exactly. Provided experiments demonstrate that the method is not only significantly faster than the original SAM but also outperforms it in terms of accuracy.
This paper is clearly and well written and the proposed method was extensively evaluated via carefully crafted experiments and ablations, demonstrating superior performance. Reviewers found that this method has a clear practical application and while not very "novel", it is witty at applying tried and tested technique in a new setting (GNN) in a way that significantly reduces the computational burden.
Decision: Accept (poster)