Title: Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Sharpness-Aware Minimization; Graph Neural Networks; Few-Shot Node Classification
Abstract: Graph Neural Networks (GNNs) have shown superior performance in node classification. However, GNNs perform poorly in the Few-Shot Node Classification (FSNC) task that requires robust generalization to make accurate predictions for unseen classes with limited labels. To tackle the challenge, we propose the integration of Sharpness-Aware Minimization (SAM)--a technique designed to enhance model generalization by finding a flat minimum of the loss landscape--into GNN training. The standard SAM approach, however, consists of two forward-backward steps in each training iteration, doubling the computational cost compared to the base optimizer (e.g., Adam). To mitigate this drawback, we introduce a novel algorithm, Fast Graph Sharpness-Aware Minimization (FGSAM), that integrates the rapid training of Multi-Layer Perceptrons (MLPs) with the superior performance of GNNs. Specifically, we utilize GNNs for parameter perturbation while employing MLPs to minimize the perturbed loss so that we can find a flat minimum with good generalization more efficiently. Moreover, our method reutilizes the gradient from the perturbation phase to incorporate graph topology into the minimization process at almost zero additional cost. To further enhance training efficiency, we develop FGSAM+ that executes exact perturbations periodically. Extensive experiments demonstrate that our proposed algorithm outperforms the standard SAM with lower computational costs in FSNC tasks. In particular, our FGSAM+ as a SAM variant offers a faster optimization than the base optimizer in most cases. In addition to FSNC, our proposed methods also demonstrate competitive performance in the standard node classification task for heterophilic graphs, highlighting the broad applicability.

Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification

Abstract

Graph Neural Networks (GNNs) have shown superior performance in node classification. However, GNNs perform poorly in the Few-Shot Node Classification (FSNC) task that requires robust generalization to make accurate predictions for unseen classes with limited labels. To tackle the challenge, we propose the integration of Sharpness-Aware Minimization (SAM)-a technique designed to enhance model generalization by finding a flat minimum of the loss landscape-into GNN training. The standard SAM approach, however, consists of two forward-backward steps in each training iteration, doubling the computational cost compared to the base optimizer (e.g., Adam). To mitigate this drawback, we introduce a novel algorithm, Fast Graph Sharpness-Aware Minimization (FGSAM), that integrates the rapid training of Multi-Layer Perceptrons (MLPs) with the superior performance of GNNs. Specifically, we utilize GNNs for parameter perturbation while employing MLPs to minimize the perturbed loss so that we can find a flat minimum with good generalization more efficiently. Moreover, our method reutilizes the gradient from the perturbation phase to incorporate graph topology into the minimization process at almost zero additional cost. To further enhance training efficiency, we develop FGSAM+ that executes exact perturbations periodically. Extensive experiments demonstrate that our proposed algorithm outperforms the standard SAM with lower computational costs in FSNC tasks. In particular, our FGSAM+ as a SAM variant offers a faster optimization than the base optimizer in most cases. In addition to FSNC, our proposed methods also demonstrate competitive performance in the standard node classification task for heterophilic graphs, highlighting the broad applicability. The code is available at https://github.com/draym28/FGSAM_NeurIPS24

1 Introduction

Graph Neural Networks (GNNs) have received significant interest in recent years due to their powerful ability in various graph learning tasks, e.g., node classification. Numerous GNNs have been developed accordingly [20, 35, 6, 15]. Despite their successes, GNNs, like traditional neural networks, tend to be over-parameterized, often requiring extensive labeled data for training to ensure generalization. However, in real-world networks, many node classes have few labeled instances, which can lead

†

to GNNs overfitting, resulting in poor generalization in these limited labeled classes. Recently, an increasing amount of research is focusing on developing superior GNNs, e.g., Meta-GCN [42], AMM-GNN [37], GPN [8] and TENT [38], for Few-Shot Node Classification (FSNC) which aims to classify nodes from new classes with limited labelled instances.

Intuitively, training GNNs for FSNC requires robust model generalization ability for recognizing unseen classes from a small number of labelled examples. Motivated by the success of the recently proposed Sharpness-Aware Minimization (SAM) for improving models' generalization in the vision domain [12], we suggest incorporating SAM into training GNNs for addressing FSNC tasks. The core idea of SAM is to perturb the model parameters to find flat minima of the loss landscape, thereby making the model more generalizable. However, a key drawback of SAM is that it requires executing two forward-backward steps to complete one optimization step, resulting in twice the time consumption compared to general optimizers like Adam. Some works [9, 23, 10] have been proposed to accelerate SAM, but none of them are crafted for graphs, i.e., not leveraging the graph properties for accelerating SAM. Meta-GCN AMM-GNN GPN TENT

This paper mainly focuses on efficient GNN training in FSNC scenarios by leveraging SAM for improving the generalization of GNNs on unseen classes. To tackle the high training cost issue of SAM, we utilize the connection between GNNs and MLPs-GNNs discarding Message-Passing (MP) are equivalent to MLPs with faster training and worse performance in general-to accelerate training. Specifically, we propose F ast G raph S harpnessA ware M inimization ( FGSAM ) that uses GNNs for perturbing parameters and employs MLPs (i.e., GNNs discarding MP) to minimize perturbed training loss. This speeds up training at

62.5

★

★

69.0

62.5

63.0

68.5

62.0

62.0

61.5

62.0

68.0

61.5

61.0

61.0

67.5

Adam

SAM

61.0

60.5

FGSAM

60.0

67.0

★

FGSAM+

60.5

60.0

66.5

100%

200%

100%

200%

100%

200%100%

200%

Training time compared tobase optimizer (i.e.,Adam)(%)

the cost of dropping graph topology information during minimizing the perturbed loss. Interestingly, we find that the gradient computed in parameter perturbation can be reused when minimizing loss to explicitly reintroduce topology information with negligible extra cost. Moreover, we can add back MP during inference to improve performance. To further reduce the computational cost, we propose FGSAM+ which conducts an exact FGSAM-update at every k steps. As shown in Fig. 1, empirical results in FSNC tasks show that our proposed FGSAM and FGSAM+ methods outperform both Adam and SAM, and meanwhile FGSAM+ is even faster than Adam. In addition, we evaluate the proposed methods in node classification, showing strong results, especially in heterophilic graphs which are known to be challenging for GNNs [30, 7]. This indicates that our proposed methods can effectively improve the GNN's generalization capability for better performance.

The contributions of this paper can be summarized as follows.

We study the application of SAM in FSNC tasks.

We propose FGSAM that improves generalization in an efficient way by leveraging GNNs for sharpness-aware perturbation parameters and employing MLPs to expedite training.

We further propose an enhanced version named FGSAM+, which conducts the actual FGSAM at every k steps and approximates it in the intermediate steps.

We demonstrate strong empirical results of the proposed methods across tasks.

2 Preliminary

Graph Neural Networks. Let G = ( V , E ) denotes an undirected graph, V = { v i } n i =1 is the node set and E ⊆ V × V is the edge set. A ∈ R n × n is the adjacency matrix. Let X = { x i } n i =1 ∈ R n × d 0 be the initial node feature matrix, where d 0 is the initial dimension, and Y = { y i } n i =1 ∈ R n × C denotes the ground-truth node label matrix, where C denotes the number of classes and y i is the one-hot encoding of node v i 's label y i . Let H ( L ) be the output of the last layer of an L -layer GCN, the prediction probability matrix ˆ Y = softmax ( H ( L ) ) is the final output of node classification.

Few-Shot Node Classification. In the FSNC task, the entire set of node classes C can be divided into two disjoint subsets: base classes set C base and novel classes set C novel, such that C = C base ∪C novel and C base ∩ C novel = ∅ . There are sufficient labeled nodes in C base, while there are only a limited number of labeled nodes in C novel. FSNC task aims to learn a model using the sufficient labeled nodes from

Train Loss

Test Loss

GNN Loss Landscape

(NC)

GNN Loss Landscape

(FSNC)

GNN Loss Landscape

(FSNC w/FGSAM)

2.0

GNN

2.2

PeerMLP

2.1

LOSS

1.5

MLP

2.0

1.9

1.0

1.8

0.5

1.7

1.6

50

100

150

200

50

100

150

200

Episodes

Episodes

C base, enabling it to accurately predict unlabeled nodes (i.e., query nodes Q ) in C novel, with limited labeled instances (i.e., support nodes S ) from C novel .

Sharpness-Aware Minimization (SAM). SAM [12] is an effective method to improve model's generalization. Let D tr = { ( x i , y i ) } n i =1 be the training dataset, following distribution D . Given a model parameterized by w and a commonly used loss function (e.g., cross-entropy loss) ℓ , instead of directly minimizing training loss L D tr ( w ) = 1 n ∑ n i =1 ℓ ( x i , y i ; w ) , SAM aims to minimize the population loss L D ( w ) = E ( x , y ) ∼D [ ℓ ( x , y ; w )] by minimizing the vanilla training loss as well as the loss sharpness (i.e., find parameters whose neighbors within the ℓ p ball also have low training loss L D tr ) as follows:

where ρ is the radius of the ℓ p ball, and p ≥ 0 (usually p = 2 ). In this way, the model can converge to flat minima in loss landscape ( w ∗ ), making the model more generalizable [12]. For efficiency, SAM applies first-order Taylor expansion and classical dual norm problem to obtain the approximation:

Finally, SAM computes the gradient w.r.t. perturbed model w + ˆ ϵ for update w in Eq. (1):

Additional Related Works. The effectiveness of SAMs and its variants have been widely verified in computer vision area [12, 21, 9, 23, 43, 10, 1]. Specifically, LookSAM [23] speeds up the SAM by periodically conducting exact perturbation, and Sharp-MAML [1] firstly focusing on meta-learning tasks. However, there is limited work on developing SAM for graphs. WT-AWP [39] is the first SAM-like work that applied to GNN and gives a theoretical analysis of generalization bound on graphs. Compared to these works, our proposed FGSAM is crafted for graphs by its unique property, enabling the first SAM-like algorithm that can be faster than the base optimizer. Our work also shares some similarities with existing works [17, 40] that explore the connection between GNNs and MLPs. However, they attributed the claim that introducing MP to MLP can improve performance during evaluation to the powerful generalization ability of MP. In contrast, we prove that for the linear case with synthetic graphs, whether there is MP or not, both will converge to the same optimal solution, taking a solid step toward understanding the underlying reasons.

3 Methodology

In this section, we propose Fast Graph Sharpness-Aware Minimization (FGSAM), an efficient version of SAM for GNNs, aiming to reduce the training time when using SAM in FSNC tasks while improving model's generalization.

3.1 Motivating Analysis

SAMs are a series of new general training scheme used to improve the model's generalization, thus it is intuitive to use SAM in FSNC tasks. However, there is no work studying how to apply SAM to FSNC tasks. So our first question is: Q1: Can SAM benefit few-shot node classification tasks?

Akey property of FSNC is that the GNNs need to be generalized to unseen classes (i.e., novel classes), and the GNNs often converge to a relatively low loss on the training set, but the final performance depends on the GNNs' generalization ability. To demonstrate this intuitively, we plot the GNN's loss landscape of novel classes under the FSNC setting and of the test set under the NC setting (Fig. 2a), following previous work [22]. The loss landscape of GNN under the FSNC setting is sharp and not smooth, with many local minima, in contrast to the flat and smooth loss landscape of GNN under the NC setting. This to some extent indicates that the FSNC setting poses a greater challenge to GNNs, which is consistent with our prior knowledge. Hence, applying SAM-like techniques can intuitively improve the generalization of GNN and enhance its performance.

However, another problem arises: training GNN on FSNC is already slow, and the core drawback of SAM is that it requires twice the training cost compared to Adam or SGD. Q2: Can we find a way to reduce the SAM training cost based on GNN properties?

It is well known that the training speed of GNNs is slower than MLPs, mainly due to the notorious MP that causes significant time consumption, yet MP is essential for improving GNN performance. Removing the MP from GNNs f gnn ( { X , A } ; w ) turns them into MLPs f mlp ( X ; w ) , which is an intriguing connection. As shown in Tab. 1 and Fig. 2b, MLPs without the burden of MP demonstrate a substantial training time advantage under the same settings as GNNs and can achieve nearly the same performance as GNNs on the training set, however, they perform significantly worse on the test set, revealing their poor generalization performance.

Inspired by previous work [17], it is appealing to remove MP during training, but reintroduce it in inference ( PeerMLP ). Although reintroducing MP after training can improve the performance, it still cannot surpass GNNs' (Fig. 2b). This may be because of the lack of graph topology information in training. Hence, we propose minimizing training loss on PeerMLPs but minimizing the sharpness according to GNNs, implicitly incorporating the graph topology information in training. This allows the model to quickly converge to the vicinity of local minima and further converge to flat GNN local minima through a GNN's sharpness-aware approach. By doing so, we not only introduce SAM to enhance the model's generalization ability and the information w.r.t graph topology but also leverage the intriguing connection between MLPs and GNNs to improve training speed.

3.2 FGSAM

We elaborate our proposed method Fast Graph Sharpness-Aware Minimization ( FGSAM ). For the ease of reference, Fig. 3a visualizes the framework of FGSAM, so does to its enhanced version FGSAM+. There are two forward-backward steps in the FGSAM-update.

Step 1: Graph sharpness-aware perturbation. The first forward-backward step is served for computing the maximum perturbation ˆ ϵ (Eq. (2)), where we propose to perturb parameters with MP (GNN), i.e.,

Step 2: Minimizing perturbed loss. We propose to minimize the perturbed loss by removing the MP (PeerMLP) to speed up training, i.e.,

It is clear that minimizing the loss on PeerMLPs is equivalent to minimizing the loss on GNNs ignoring the topology information. As demonstrated in Sec. 3.1, intuitively the proposed approach can make model convergence near the local minima easily due to the connection between MLPs and GNNs, and perturbing parameters with MP can find the good flat minima of GNNs (see Fig. 2a).

Reintroducing Graph Topology in Minimization with Free Lunch. While reintroducing the MP in evaluation can improve performance, its absence during the minimization process may result in

𝜆𝒈 ୥୬୬

𝒈௦

𝒈௩ 𝒈௛

𝒘

𝒘൅𝝐 ො

𝒘௧ାଵ

𝒘௧ାଵ

୊ୋୗ୅୑

𝒈௦

𝒈𝒢

𝒈 ୥୬୬

𝒈 ୫୪୮

𝒈௦

𝒈௩ 𝒈௛

𝒘

𝒘൅𝝐 ො

𝒘௧ାଵ

𝒈௦

𝒈 ୥୬୬

𝒈 ୫୪୮

𝒈𝒢

𝒘௧ାଵ

୊ୋୗ୅୑

𝜆𝒈 ୥୬୬

𝐌𝐋𝐏

𝐆𝐍𝐍

FGSAM+

𝒈௦

𝒘

𝒘൅𝝐 ො

𝒘௧ାଵ

𝒘௧ାଵ

୊ୋୗ୅୑

𝒈௦

𝒈 ୥୬୬

𝜆𝒈 ୥୬୬

𝒈௦

𝒘

𝒘൅𝝐 ො

𝒘௧ାଵ

𝒈௦

𝒈 ୥୬୬

𝒘௧ାଵ

୊ୋୗ୅୑

𝜆𝒈 ୥୬୬

𝐌𝐋𝐏

𝐆𝐍𝐍

FGSAM

(b) Gradient difference curves

sub-optimal results. Incorporating MP directly into the minimization is computationally expensive, leading us to employ MLP to minimize the perturbed loss. Fortuitously, the gradient w.r.t. MP is computed during the perturbation step, offering an opportunity for computational savings. We propose to capitalize on the already available gradient information from the first step by reusing it in the optimization procedure, as formalized in the following optimization target:

This formulation implies that the computational cost of involving MP in the optimization is mitigated since the forward and backward passes are precomputed in the initial step. Thus, we effectively integrate graph topology into the minimization process almost without incurring additional computational expense, akin to receiving a free lunch . See detailed FGSAM in Algorithm 1.

Adaptation to MAML Models. Model-Agnostic Meta-Learning (MAML) [11] is widely used in FSNC tasks [8, 38], involving two separate update steps in one MAML-update: i) pre-training for learning task-relevant knowledge, and ii) meta-update for task-irrelevant update. This is different from standard gradient descent. Hence for integrating the FGSAM into the MAML models, we propose treating the MAML-update process as a single entity, and applying the FGSAM-update only once simplifies the implementation. This contrasts with the Sharp-MAML [1], where the SAM-update is applied separately in the two stages.

3.3 FGSAM+

Although the training time of FGSAM can be largely faster than naïve SAM by ignoring the MP in minimizing perturbed loss, it still requires a full forward-backward step of GNN, which makes our approach need an extra computation cost for a forward-backward step of PeerMLP, compared to the base optimizer.

Fortunately, the forward-backward step of GNN is mainly for perturbing parameters in FGSAM, thus we can further reduce the training time while maintaining performance, by employing FGSAM-update at every k step (i.e., perturb parameters at every k step) and reusing the preserved gradients from parameters perturbation into the intermediate steps [23]. Eq. (3) can be rewritten as:

In this way, SAM-gradient g s is composed by the vanilla gradient ∇ w L D tr ( w ) and the gradient of the ℓ 2 -norm of vanilla gradient ∇ w ∥∇ w L D tr ( w ) ∥ .

This suggests that SAM-gradient g s = ∇ w L D tr ( w ) | w +ˆ ϵ can be divided into two orthogonal parts [23]: g h (in the direction of vanilla gradient g = ∇ w L D tr ( w ) ) is used to minimize the loss value, and flatness-gradient g v is used to adjust the updates towards a flat region. So g h and g v can be easily obtained if g s and g are given:

where θ is the angle between g s and g h . As illustrated in [23], g v changes much slower than g s and g h , thus we can compute and preserve g v at every k steps, and reuse it to approximate g s in intermediate steps.

However, in our case, there exists a clear gap between the model used for perturbing (GNN) and for minimizing (PeerMLP). This is different from the approach in [23], which uses the same model for both. Thus we use an extra PeerMLP forward-backward step to get another g mlp for computing g v to reduce the gap:

Note that the ˆ ϵ is obtained by perturbing parameters with MP Eq. (4), g s and g mlp are obtained without MP, thus there still exists a gap.

Moreover, since we reintroduce graph topology (Eq. (6)) in minimization, we propose to further use the extra PeerMLP step to reuse graph topology for better performance. Specifically, we conduct the gradient w.r.t. topology information by projection as follows:

where θ ′ is the angle between g gnn and g mlp . This can be reused in a similar way as g v when approximating FGSAM-update in the intermediate steps. We further conduct experiments to verify whether the g G and g v will change slowly so that they can be reused for speed up in our approach. We plot the change of g s , g h , g v and g G (Fig. 3b) and the results show that the projected gradient both g v and g G on parameters perturbed with MP shows a much more stable pattern and slower changes than g s and g h , indicating the feasibility of updating g v and g G every k steps and reusing it for the intermediate steps. We present the detailed FGSAM+ in Algorithm 1 in Appendix B.

Since we need an extra PeerMLP forward-backward step at every k step, the overall computation cost of our approach, FGSAM+, will be 1 k × the computation cost of GNNs plus (1+ 1 k ) × the computation cost of MLPs on average.

4 Analysis of Toy Case

In this section, we employ the Contextual Stochastic Block Model (CSBM) to analyze why minimizing perturbed training loss without MP can work to some extent, which is the underlying mechanism of FGSAM. The CSBM has been widely used to analyze of the properties of GNN [27, 26].

Specifically, we focus on a CSBM model that contains K distinct classes c 1 , c 2 , . . . , c K . The nodes within the resulting graphs are grouped into n non-overlapping sets C 1 , C 2 , . . . , C K , each set representing one of the K classes. The generation of edges is governed by a probability p within the same class and a probability q between different classes. For any given node i , we sample its initial features x i ∈ R l from a Gaussian distribution denoted by x i ∼ N ( µ , I ) , where the mean µ = µ k ∈ R l corresponds to node i belonging to set C K , and k is an element of { 1 , 2 , . . . , K } . Furthermore, the condition || µ i -µ j || 2 = D holds true for all i, j belonging to { 1 , 2 , . . . , K } , with D being a positive constant. Graphs that arise from this specified CSBM model are referred to as K -classes CSBM. After applying a MP operation, the resultant features for node i are denoted by h i .

The neighborhood label distribution D i of node i is a K-dimensions vector, where D i [ j ] = I ( i ∈ C j ) p +(1 -I ( i ∈ C j )) q . Based on the neighborhood label distribution, consider the MP operation as h i = 1 deg ( i ) ∑ j ∈N ( i ) x i , we have: h i ∼ N ( ( p -q ) µ k + qK ¯ µ p +( K -1) q , I deg ( i ) ) , where i ∈ C k and ¯ µ = ∑ K

j =1 µ j K . Based on the distribution of h i and x i , we can obtain following theorem:

Theorem 4.1 (The effectiveness of removing MP in minimization) . Consider a K-classes CSBM, the optimal linear classifiers for both original features x i and filtered features h i are the same.

Detailed proof is in Appendix C. The theorem tells us that under the linear case, whether the MP layer is used or not, the optimal decision bound is the same. Hence, this encourages us to learn the weight of transformation layers without MP to speed up training. However, the real graph is more complex and we do not use a linear classifier, thus we propose to perform the graph sharpness-aware perturbation which implicitly involves the information of neighbors.

5 Experiments

We verify the effectiveness of our proposed FGSAM and FGSAM+ in this section. We first conduct experiments to demonstrate that our proposed algorithms achieve better performance compared to SAM which requires twice the training time. Then we show that our proposed algorithms can achieve faster training speed compared to base optimizers (e.g., Adam). Next, we also conduct extra studies and an extra task to show the robustness and potential applications of our proposed algorithms.

5.1 Experiment Settings

Baseline. We evaluate our proposed FGSAM and FGSAM+ on SOTA models. The existing models can be divided into two main categories: MAML and non-MAML methods. Two representative models are selected from each category, respectively, as baselines for evaluation ( Meta-GCN [42] and AMM-GNN [37] for MAML models, and GPN [8] and TENT [38] for non-MAML models).

Datasets. We conduct evaluations on three widely used real-world benchmark node classification datasets: CoraFull [5], DBLP and ogbn-arXiv [18], and we use the train/val/test split as in [34] and [24]. The comprehensive statistics of datasets are shown in Tab. 5 in Appendix D.1.

Implementation Details. We implement our model by PyTorch [29] and conduct experiments on an RTX-3090Ti. We use Optuna [2] to search the hyper-parameters for each setting. See Appendix D.2 for detailed FSNC learning protocol.

5.2 Evaluation on Real-World Datasets

The results of different models across datasets are summarized in Tab. 2. All the models share a 2layers architecture with 16 hidden channels. It can be seen that our proposed algorithms FGSAM and FGSAM+ provide better performance than Adam in most cases, and provide comparable performance with SAM. These results support our claim that FGSAM and FGSAM+ can find local minima with better generalization properties. Note that message-passing is only used in perturbing parameters, not involved in parameters update (i.e., MLPs). The results further indicate that implicitly involving graph topology in training can make PeerMLPs outperform GNNs. See Appendix D.3 for details.

5.3 Time Consumption

To demonstrate the training speed advantage of our proposed algorithm, we summarize the training time for different models using various optimization methods across three datasets (Tab. 2). The results indicate that our proposed algorithm FGSAM demonstrates only a slight increase in training cost compared to Adam in most cases. Furthermore, our enhanced version FGSAM+ outperforms Adam in terms of speed in the majority of scenarios. It is worth mentioning that our proposed algorithms achieve superior or comparable performance when compared to both Adam and SAM. See Appendix D.4 for detailed results.

Limitation. For models composed of many non-GNN components (e.g., TENT), the training time on FGSAM+ may be still longer than that on Adam, since it is hardly further reduced.

Hidden Channels

Layers

NoisyFeature

Noisy Edge

72

72

69

Adam

Accuracy(1%)

71

68

68

68

FGSAM+

64

60

67

66

66

56

65

64

66

52

48

64

63

62

44

62

60

65

16 64 128

256

512

1

2

4

6

8

0

0.01

0.05

0.1

0.15

0%

1%

5%

10%

15%

NumberofHiddenChannels

Number of Layers

StandardDeviationofNoise

Ratio ofNoisyEdge

5.4 Comparison of the Variants of SAM

Training with Different Optimizer. We compare the performance of Meta-GNN and GPN training with different variants of SAM, including original SAM, ESAM [9], LookSAM [23], AE-SAM [19], our proposed FGSAM and FGSAM+ (Tab. 3). We observe an anomalous phenomenon where ESAM, as an efficient variant of SAM, actually trains slower than SAM. This is because ESAM sorts the sample losses and selects a suitable subset at each iteration, an operation that is negligible for image tasks; however, for graph tasks, since GNNs are relatively smaller, the proportion of time consumed by the sorting step is significant, leading to an increase in training time. As shown in Tab. 3, our proposed method greatly reduces the training time, based on the relationship between GNN and MLP, while maintaining and even achieving superior performance, compared to other optimizers, indicating ours' high efficiency and effectiveness.

The Impact of Perturbing Parameters with Message-Passing. A key point of our work is that we perform parameter perturbation using GNNs, while PeerMLPs (i.e., without message-passing) are used to minimize the perturbed loss. This is significantly distinct from previous SAM methods which shared the same model for both parameter perturbation and loss minimization. So a natural question arises: to what extent does our approach benefit from performing parameter perturbation using GNNs? We thus compare our approach to PeerMLPs training with Adam and vanilla SAM. Note that message-passing would be reintroduced during validation and test. From Tab. 3, although the training time of PeerMLPs is shorter than that of GNNs, GNNs outperform their PeerMLPs in most cases. Despite that using PeerMLPs can accelerate the training of GNNs, the topology information is still very important for learning node representations. Thus our proposed FGSAM+ is a better solution, achieving a better trade-off between efficiency and performance.

5.5 Ablation Studies

We further verify the consistent effectiveness of our method compared to Adam across different settings regarding model implementation and graph property. Due to the computational resource restriction, all experiments here were conducted using GPN on the CoraFull with the 5-way 3-shot setting. We provide additional experiments (e.g., the effect of update interval k ) in the Appendix E.

The Impact of Network Structure. Here we investigate the effect of hidden dimension and the number of layers on the performance (on the left of Fig. 4). GPN with Adam requires a higher hidden dimension (128) to achieve relatively high accuracy, whereas GPN with FGSAM+ can attain SOTA even with a small hidden dimension (16). With respect to the number of layers, GPN with FGSAM+ consistently performs better within the range of 1 ∼ 8 compared to GPN with Adam, demonstrating the effectiveness of our proposed method (middle left of Fig. 4).

The Impact of Noisy Features and Edges. Here we investigate the effect of randomly adding Gaussian noise to features and randomly adding edges during testing (on the middle right and the right of Fig. 4). Specifically, for noisy features, we randomly add Gaussian noise with varying standard deviations to the node features. Meanwhile, for noisy edges, we uniformly and randomly introduce additional edges into the original structure. The results show that GPN with FGSAM+ method can still achieve relatively high performance, compared to GPN with Adam. These results effectively verify the robustness of our proposed method.

5.6 Additional Task on Conventional Node Classfication

Our proposed FGSAM+ also has the potential to be extended to other domains. To demonstrate this, we evaluate the performance of the FGSAM+ on the standard node classification task on both homophilic and heterophilic graphs. For homophilic graphs, we utilize three well-established citation networks: Cora , Citeseer , and Pubmed [32, 13]. For heterophilic graphs, we include page-page

networks from Wikipedia, specifically the Chameleon and Squirrel datasets [31], actor-network, namely Actor [30], and web pages networks, namely Cornell , Texas and Wisconsin [30]. See Appendix E.1 for statistics of these datasets. We use data splits (48%/32%/20%) provided by [30], and set k = 2 for FGSAM+. We select three representative baselines, namely the classical GCN [20], GAT [35] with learnable MP operation, and GraphSAGE [16] with complex MP operation, to demonstrate the effectiveness of FGSAM and FGSAM+.

As shown in Tab. 4, both FGSAM and FGSAM+ generally outperform Adam and SAM across base models, indicating the potential wide application of our method. We observed that the proposed method achieves greater improvement on heterophilic graphs compared to homophilic graphs, and heterophilic graphs are generally considered more challenging. This indicates that our method can effectively enhance the generalization capability of GNNs. We also provide additional experiments of integrating FGSAM+ with prompt-based FSNC [33] in the Appendix E.3.

5.7 Additional Study

We observe that both FGSAM and FGSAM+ generally outperform the standard SAM across tasks (FSNC and standard node classification). This is an interesting finding, as our FGSAM and FGSAM+ algorithm remove message-passing during the minimization of the perturbed loss, which is expected to hurt performance. We attribute these counterintuitive results to the mitigation of the imbalance adversarial game. The training process of SAM-like

algorithms entails an adversarial game similar to that in Generative Adversarial Nets (GANs) [14]. Prior studies [3, 4, 28] have demonstrated that imbalanced adversarial games in GANs can give rise to worse results. Both FGSAM and FGSAM+ employ distinct models for perturbation and minimization, which can help alleviate the extent of imbalance. These factors may explain the observed performance discrepancies among the compared algorithms. To verify the explanation, we conduct experiments varying the hyper-parameter ρ . Specifically, we graphically illustrate the comparative training loss of SAM and FGSAM+ over a range of ρ values in Fig. 5, which reveals that while SAM struggles to converge with higher ρ values, FGSAM+ consistently achieves convergence. Moreover, it is established that a higher ρ value is conducive to a tighter generalization bound, suggesting that a larger ρ could potentially enhance performance. Consequently, FGSAM+ is capable of mitigating the imbalanced games issue and tolerating a larger ρ , which contributes to its enhanced performance.

6 Conclusion

In this work, we study the application of Sharpness-Aware Minimization (SAM) in FSNC to improve model's generalization, since the key for FSNC is to generalize the model to unseen samples. In order to alleviate the heavy computation cost of SAM, we utilize the connection between MLPs and GNNs and use MLPs to accelerate the training of GNNs. However, the low generalization and lack of using graph topology of MLPs also limit its performance. Hence we propose to apply GNNs to perturb parameters for generalization and use MLPs to minimize the perturbed training loss for conducting the proposed FGSAM. Moreover, we reuse the GNN gradient in perturbation in minimization for better including topology information. We further reduce the training time by conducting exact FGSAM update at every k steps and approximate FGSAM's gradient with reusing information in the intermediate steps. Finally, the extensive experiments demonstrate the effectiveness and efficiency of our proposed methods.

References

Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. Sharp-maml: Sharpnessaware model-agnostic meta learning. In International conference on machine learning , pages 10-32. PMLR, 2022.

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2623-2631, 2019.

Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations , 2017.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In ICLR , 2018.

Yuhan Chen, Yihong Luo, Jing Tang, Liang Yang, Siya Qiu, Chuan Wang, and Xiaochun Cao. Lsgnn: Towards general graph neural network in node classification by local similarity. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 , pages 3550-3558. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations. https://openreview. net/forum , 2021.

Kaize Ding, Jianling Wang, Jundong Li, Kai Shu, Chenghao Liu, and Huan Liu. Graph prototypical networks for few-shot learning on attributed networks. In CIKM , 2020.

Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint arXiv:2110.03141 , 2021.

Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. Advances in Neural Information Processing Systems , 35:23439-23451, 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 , 2020.

Lise Getoor. Query-driven active surveying for collective classification. 2012.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Zhichun Guo, Chuxu Zhang, Wenhao Yu, John Herr, Olaf Wiest, Meng Jiang, and Nitesh V Chawla. Few-shot graph learning for molecular property prediction. In Proceedings of the web conference 2021 , pages 2559-2567, 2021.

William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS , 2017.

Xiaotian Han, Tong Zhao, Yozen Liu, Xia Hu, and Neil Shah. Mlpinit: Embarrassingly simple gnn training acceleration with mlp initialization. arXiv preprint arXiv:2210.00102 , 2022.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS , 2020.

Weisen Jiang, Hansi Yang, Yu Zhang, and James Kwok. An adaptive policy to employ sharpnessaware minimization. In The Eleventh International Conference on Learning Representations , 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpnessaware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning , pages 5905-5914. PMLR, 2021.

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems , 31, 2018.

Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12360-12370, 2022.

Yonghao Liu, Mengyu Li, Ximing Li, Fausto Giunchiglia, Xiaoyue Feng, and Renchu Guan. Few-shot node classification on attributed networks with graph meta-learning. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval , pages 471-481, 2022.

Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, editors, Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023 , pages 417-428. ACM, 2023.

Sitao Luan, Chenqing Hua, Minkai Xu, Qincheng Lu, Jiaqi Zhu, Xiao-Wen Chang, Jie Fu, Jure Leskovec, and Doina Precup. When do graph neural networks help with node classification: Investigating the homophily principle on node distinguishability. arXiv preprint arXiv:2304.14274 , 2023.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? arXiv preprint arXiv:2106.06134 , 2021.

Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 2794-2802, 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287 , 2020.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-Scale Attributed Node Embedding. Journal of Complex Networks , 9(2), 2021.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. Ai Magazine , 2008.

Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023 , pages 2120-2131. ACM, 2023.

Zhen Tan, Song Wang, Kaize Ding, Jundong Li, and Huan Liu. Transductive linear probing: A novel framework for few-shot node classification. In Learning on Graphs Conference , pages 4-1. PMLR, 2022.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies , 2020.

Ning Wang, Minnan Luo, Kaize Ding, Lingling Zhang, Jundong Li, and Qinghua Zheng. Graph few-shot learning with attribute matching. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management , 2020.

Song Wang, Kaize Ding, Chuxu Zhang, Chen Chen, and Jundong Li. Task-adaptive few-shot node classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1910-1919, 2022.

Yihan Wu, Aleksandar Bojchevski, and Heng Huang. Adversarial weight perturbation improves generalization in graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10417-10425, 2023.

Chenxiao Yang, Qitian Wu, Jiahua Wang, and Junchi Yan. Graph neural networks are inherently good generalizers: Insights by bridging GNNs and MLPs. In The Eleventh International Conference on Learning Representations , 2023.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Fan Zhou, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng. Meta-gnn: On few-shot node classification in graph meta-learning. In CIKM , 2019.

Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065 , 2022.

A Potential Broader Impact

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

B Algorithm

C Proof

The linear classifier for K-classification problems can be formulated as K ( K -1) 2 binary classification problems.

Hence we study the classification between class C o and C p without loss of generality.

The distribution of original features from different classes follows:

The distribution of filtered features from different classes follows:

For simplicity, we denote ˜ µ 0 = ( p -q ) µ o + qK ¯ µ p +( K -1) q and ˜ µ p = ( p -q ) µ p + qK ¯ µ p +( K -1) q .

Following [27], the optimal classifier of original features constructs a decision bound P = { x | w T x -w T b } , where w = µ o -µ p 2 / || µ o -µ p 2 || , b = µ o + µ p 2 . Similarly, the optimal classifier of filtered features constructs a decision bound P ′ = { h | w ′ T h -w ′ T b } , where w ′ = ˜ µ o -˜ µ p 2 / || ˜ µ o -˜ µ p 2 || , b ′ = ˜ µ o +˜ µ p 2 .

And we have ˜ µ o -˜ µ p = p -q p +( K -1) q ( µ o -µ p ) , hence we have w = w ′ . Then we verify whether w T b = w ′ T b ′ :

where λ = p -q p +( K -1) q .

Then we show ( µ o -µ p ) T ( ¯ µ -µ o + µ p 2 ) = 0 .

From || µ i -µ j || 2 = D , we have:

which gives:

Hence we have:

Combining Eq. (13) and Eq. (16), we have:

which means P = P ′ .

This completes the proof.

D Experiments details

D.1 Datasets Description

CoraFull is an extension of the prevalent dataset 'Cora' [41], a citation network dataset. On this graph, nodes represent papers and edges represent citation links. The nodes are labeled on the paper topics. Node attributes are obtained using bag-of-words for the title and abstract of the paper.

DBLP is also a citation network, where nodes represent papers and edges represent the citation between papers. Specifically, the node attributes are generated by the abstract and the node labels are based on the paper venues.

ogbn-arXiv is a citation network among all Computer Science arXiv papers based on MAG [36]. Node represent papers and edges are citations links. The node attributes are obtained using skip-gram on abstract of papers. The nodes are labeled by the subject area.

D.2 Implementation Details

Specifically, we implement our model by PyTorch [29] and conduct experiments on 24GB Nvidia RTX3090Ti, according to the training protocol Algorithm 2. Repeat number R = 5 , patience P = 10 , SAM update interval k = 2 , validation interval I = 10 , validation number V = 20 , test number W = 100 . For MAML models max epochs T = 500 , and for non-MAML model max epochs T = 1000 . We evaluate our method under various settings, i.e., N = { 5 , 10 } , K = { 3 , 5 } , but we set N = { 2 , 5 } for Coauthor-CS dataset. We use Optuna [2] for hyper-parameters searching for all models with various optimizers, the search space is shown in Tab. 6.

Note that we further split C base into two disjoint class set: training class set C tr and validation class set C val, such that C base = C tr ∪ C val and C tr ∩ C val = Ø. Overall, we use C tr and C val for train and validation in the meta-training stage, respectively, and use C novel for meta-test. We split C into C tr , C val and C novel according to the class split ratio in Tab. 5.

Algorithm 2 Training Protocol of FSNC Task

D.3 Evaluation Results with Standard Deviation

In Tab. 7 and Tab. 8 we present the detailed results of Tab. 2 and Tab. 7 with standard deviation, respectively.

D.4 The Full Results of Time Consumption

Here we present the detailed results of training time consumption of different optimizers across various datasets. Tab. 9 indicates that our proposed algorithm FGSAM demonstrates only a slight increase in training cost compared to Adam in most cases. Furthermore, our enhanced version FGSAM+ outperforms Adam in terms of speed in the majority of scenarios. It is worth mentioning that our proposed algorithms achieve superior or comparable performance when compared to both Adam and SAM.

As mentioned before, for models composed of many non-GNN components (e.g. TENT), the training time on FGSAM+ may be still longer than that on Adam, since it is hardly further reduced.

E Additional Experiments

E.1 Statistics Of Benchmark Datasets In Node Classification

E.2 The Effect of Update Interval k in FGSAM+

Here we study the effect of update interval k in FGSAM+. It can be observed from Tab. 11 that as k increases, the performance decreases, but meanwhile training time also decreases. This indicates that the possibility of choosing k to achieve a better trade-off between performance and efficiency. We note that the performance drop with increasing k seems to be larger compared to LookSAM [23] in computer vision tasks. This indicates the importance of the perturbation step in FGSAM+, as it not only introduces information about flat minima, but also incorporates neighbor information in training. Therefore, we recommend setting k = 2 as the prior optimal update interval to avoid large information loss.

E.3 Integrating with Prompt-Based FSNC

Recently, there are many prompt-based methods [25, 33] have been developed, showing promising performance in FSNC. Hence, we investigate how our method performs in such a prompt-based FSNC task. Note that under this setting, the proposed method is used in prompt tuning instead of training. As shown in Tab. 12, our method improves the baseline [33] with a remarkable margin.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed some limitations about the proposed FGSAM+ in Sec. 5.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided assumptions and proofs in the main body and appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Necessary information is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Data is publicly available and code is also available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Detailed are provided in the paper (main body and appendix).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Due to the page limit, the standard deviation is provided in the appendix (Tab. 7).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information is provided in Sec. 5.1 and Sec. 5.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that we have adhered to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the potential broader impacts in Appendix A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There are no high-risk issues in our model trained for FSNC tasks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all the related papers, the papers of models and datasets we use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will submit the code we used along with the supplement materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces an innovative approach to improve the generalization capabilities of Graph Neural Networks (GNNs) in few-shot node classification tasks. The authors propose a novel algorithm, Fast Graph Sharpness-Aware Minimization (FGSAM), which incorporates sharpness-aware minimization (SAM) techniques into GNN training but reduces the typical computational overhead by integrating multilayer perceptrons (MLPs) for efficiency. This method not only demonstrates superior performance on few-shot learning tasks compared to traditional techniques but also offers significant computational advantages, particularly in reducing training times.

STRENGTHS:
originality: medium
quality: medium
clarity: medium
significance: medium

WEAKNESSES:
Some experiments are missing and the proof need to be double checked carefully.

QUESTIONS:
1. What is the standard deviations for the results in table 4? Are the improvements significant?

2. Is there any theoretical justification for PeerMLP?

3. What is g in equation (8)?

4. The proof of theorem 4.1 is confusing. Do you assume all nodes have the same degree deg(i)?

LIMITATIONS:
the authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on efficient graph neural network (GNN) training in few-shot node classification (FSNC) problem by extending sharpness-aware minimization (SAM) for reducing the computational cost and improving the generalization of GNNs on unseen classes. The training phase is accelerated by perturbing the parameters of GNN and then minimizing the perturbation loss of GNN without the message passing mechanism (MLP). Experiments have been conducted to verify the effectiveness and efficiency of the proposed method.\n\nSTRENGTHS:\n1. It is interesting to incorporate SAM technique with GNN by removing MP during training and reintroducing MP in inference, which is reasonable to improve the generalization of FSNC.\n2. The proposed method could be conveniently integrated into existing GNN-based few-shot node classification models, which is verified in the experiments.\n3. The carefully designed experiments highlight the effectiveness of the proposed method in reducing the computational costs of GNN training.\n4. The paper is well-organized and explains the method very clearly. In addition, the landscape visualization and toy case analyses also make the motivation and ideas easy to understand.\n\nWEAKNESSES:\n1. There are related work on applying SAM to few-shot tasks (Sharp-MAML) [1] and performing SAM every $k$ steps [2], and the innovation of this work is not significant compared with the above work.\n\n[1] Sharp-maml: Sharpness-aware model-agnostic meta learning. ICML, 2022\n\n[2] Towards efficient and scalable sharpness-aware minimization. CVPR, 2022\n\n2. The experiments only compare with small-sample node classification models and variants of SAM from 2022, lacking comparisons with the latest baseline methods of FSNC.\n\nQUESTIONS:\n1. Since this work focuses on solving FSNC problems, the authors should provide a comparison with the state-of-the-art FSNC methods, such as COSMIC [3] and COLA [4], in theoretical analysis and experiments.\n\n[3] Contrastive meta-learning for few-shot node classification. SIGKDD, 2023\n\n[4] Graph contrastive learning meets graph meta learning: A unified method for few-shot node tasks. WWW, 2024\n\n2. Why adding 25% noisy edges yields better results than adding only 15%? in Figure 4? The authors should discuss or explain this issue in details.\n\nLIMITATIONS:\nThe authors describe some limitations of the proposed method in time consumption section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for few-shot learning on graphs leveraging sharpness-aware minimization (SAM) from the vision community. The paper explains SAM as a technique for gradient perturbation during training to push the parameters to \"flatter\" regions of the loss space in hopes of achieving better generalization in the few shot setting. However, SAM is inherintely slower, requiring two backward/forward passes during training; one to compute the loss gradient and a second to determine the perturbation direction. The authors propose multiple ideas to overcome the added computation and to leverage graph topology. The authors strategically substitute Graph Neural Networks (GNNs) with Multilayer Perceptions (MLPs) to avoid the burden of message passing in some parts of their algorithm while preserving graph topologies in others. The authors propose a second scheme that leverages iterations of an approximate perturbation scheme with periodic exact evaluations of SAM. The authors present ablation studies on multiple SOTA models and 9 datasets and show improvements in training times and accuracy.\n\nSTRENGTHS:\nThe authors present multiple novel approaches to overcome computational challenges applying SAM techniques to structured data to achieve better generality in the few shot case. I only have a few style suggestions at the beginning of the paper but otherwise the paper does a good job of explaining multiple, complex ideas in a clear way. The paper does present a significant result, blending ideas from PeerML and SAM from the vision community to contribute to making structured models more generalizable.\n\nWEAKNESSES:\nI think the paper is very good and clear in most cases. The only weaknesses I would note (and this may be subjective) would be to make some style improvements in the beginning of the paper.\n\n(1) In the abstract (line 14) Moreover, our method ingeniously reutilizes... is a bit grandiose; I would prefer a more scientific tone and delete \"Moreover\" and \"ingeniously\"; \"Our method reutilizies...\n\n(2) Remove \"for the first time\" in 20. It is clear that the ideas are novel.\n\n(3) Line 50 is the first time Message Passing is abbreviated as MP. I would follow the convention used in the paper for the introduction of other abbreviations where the first letter of the abbreviated character is in bold and capitalized.\n\n(4) Line 93 \"as follows\" is not followed by an equation but another sentence. Maybe the equation should be presented after the \"as follows\" text and the next sentence come after the equation?\n\nQUESTIONS:\n(1) I find the paragraph containing lines 137-145 hard to read due to how the ideas are presented:\n\n\"Inspired by previous work...we propose to remove MP during training, but reintroduce it..Reintroducing MP after training can improve the performance significanly but still cannot surpass GNNs...\"\"Henve we propose minimizing training loss on PeerMLPs but minimizing the sharpness according to GNNs.\n\nThere seems to be some logical oscillation in the text. First it seems that a scheme similar to PeerMLP is uses where the GNN is removed during training and injected back during inference. In the next few sentences, this seems to be abandoned and the GNN brought back for the SAM portion of training. I believe the experiments and ideas really did implement the latter formulation but either I'm misunderstanding or this paragraph could use some work to be clearer.\n\n(2) In table 8 in the Appendix, I don't believe the best results are actually in boldface. Maybe this was due to my printer or perhaps the table needs to be reformatted?\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}