Title: An Efficient Memory Module for Graph Few-Shot Class-Incremental Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: graph few-shot class-incremental learning

graph incremental learning

graph few-shot learning
Abstract: Graph incremental learning has gained widespread attention for its ability to mitigate catastrophic forgetting for graph neural networks (GNN). Conventional methods typically require numerous labels for node classification. However, obtaining abundant labels is often challenging in practice, which makes graph few-shot incremental learning necessary. Current approaches rely on large number of samples from meta-learning to construct memories, and heavy fine-tuning of the GNN parameters that lead to the loss of past knowledge. These result in significant memory consumption and loss of past knowledge information, respectively. To tackle these issues, We introduce Mecoin to efficient construct and Preserve memory. For efficient storage and update of class prototypes, Mecoin use Structured Memory Unit (SMU) to cache prototypes of the seen classes and update new class prototypes through interaction between nodes and the cached prototypes by Memory Construction module(MeCo). Besides, to avoid extensive parameter fine-tuning and forgetting, we introduce a Memory Representation Adaptive Module called MRaM to separate the learning of prototypes and class representations and use Graph Knowledge Interchange Module (GKIM) to injects past knowledge information into GNN. We analyze the effectiveness of our paradigm from the perspectives of generalization error, and discuss the impact of different distillation methods on model performance through experiments and VC-dimension. By comparison with other related methods, we validate that Mecoin achieves higher accuracy and lower forgetting rate.

An Efficient Memory Module for Graph Few-Shot Class-Incremental Learning

Abstract

Incremental graph learning has gained significant attention for its ability to address the catastrophic forgetting problem in graph representation learning. However, traditional methods often rely on a large number of labels for node classification, which is impractical in real-world applications. This makes few-shot incremental learning on graphs a pressing need. Current methods typically require extensive training samples from meta-learning to build memory and perform intensive finetuning of GNN parameters, leading to high memory consumption and potential loss of previously learned knowledge. To tackle these challenges, we introduce Mecoin, an efficient method for building and maintaining memory. Mecoin employs Structured Memory Units to cache prototypes of learned categories, as well as Memory Construction Modules to update these prototypes for new categories through interactions between the nodes and the cached prototypes. Additionally, we have designed a Memory Representation Adaptation Module to store probabilities associated with each class prototype, reducing the need for parameter fine-tuning and lowering the forgetting rate. When a sample matches its corresponding class prototype, the relevant probabilities are retrieved from the MRaM. Knowledge is then distilled back into the GNN through a Graph Knowledge Distillation Module, preserving the model's memory. We analyze the effectiveness of Mecoin in terms of generalization error and explore the impact of different distillation strategies on model performance through experiments and VC-dimension analysis. Compared to other related works, Mecoin shows superior performance in accuracy and forgetting rate. Our code is publicly available on the MecoinGFSCIL.

1 Introduction

In the field of graph learning, conventional methods often assume that graphs are static[1]. However, in the real world, graphs tend to grow over time, with new nodes and edges gradually emerging. For example, in citation networks, new papers are published and cited; in e-commerce, new products are introduced and updated; and in social networks, new social groups form as users join. In these dynamic contexts, simply updating graph representation learning methods with new data often leads to catastrophic forgetting of previously acquired knowledge.

Despite numerous methods proposed to mitigate the catastrophic forgetting problem in Graph Neural Networks(GNNs)[2, 3, 4], a critical and frequently neglected challenge is the scarcity of labels for newly introduced nodes. Most current graph incremental learning methods [5, 6] combat catastrophic forgetting by retaining a substantial number of nodes from previous graphs to preserve prior knowledge. However, these methods become impractical in graph few-shot class-incremental learn-

ing(GFSCIL) scenarios due to the limited labeled node samples. Some methods[7, 1] employ regularization to maintain the stability of parameters critical to the graph's topology. Yet, in GFSCIL, the label scarcity complicates accurate assessment of the relationship between parameter importance and the underlying graph structure, thus increasing the difficulty of designing effective regularization strategies. Consequently, the issue of label scarcity hampers the ability of existing graph continual learning methods to effectively generalize in graph few-shot learning scenarios.

GFSCIL presents two critical challenges: 1)How can we empower models to learn effectively from limited samples? 2)How can we efficiently retain prior knowledge with imited samples? While the first challenge has been extensively explored in graph few-shot learning contexts[8, 9], this paper focuses on the second. Currently, discussions on the latter issue within GFSCIL are relatively scarce. Existing methods[10, 11] primarily focus on enhancing models' ability to learn from few-shot graphs and preserve prior knowledge by learning class prototypes through meta-learning and attention mechanisms. However, to strengthen inductive bias towards graph structures and learn more representative class prototypes, these approaches require caching numerous training samples from the meta-learning process for subsequent GFSCIL tasks. This caching not only consumes significant memory but also imposes substantial computational costs. Furthermore, these methods extensively adjust parameters during prototype updates, risking the loss of previously acquired knowledge[12, 13]. These challenges underscore the need for innovative strategies that maintain efficiency without compromising the retention of valuable historical data-achieved through minimal memory footprint, high computational efficiency, and limited parameter adjustments.

To address the aforementioned challenges, we introduce Mecoin, an efficient memory construction and interaction module. Mecoin consists of two core components: the Structured Memory Unit(SMU) for learning and storing class prototypes, and the Memory Representation Adaptive Module(MRaM) for dynamic memory interactions with the GNN. To effectively leverage graph structural information, we design the Memory Construction module(MeCs) within the SMU. MeCs facilitates interaction between features of the input node and prototype representations stored in the SMU through self-attention mechanisms, thereby updating sample representations. Then, it utilizes the local structural information of input nodes to extract local graph structural information and compute a local graph structure information matrix(GraphInfo). The updated sample representations and GraphInfo are used to calculate class prototypes for the input nodes. These newly obtained prototypes are compared with those stored in the SMU using Euclidean distance to determine whether the input samples belong to seen classes. If a sample belongs to a seen class, the corresponding prototype in the SMU remains unchanged. Conversely, if a sample belongs to an unseen class, its calculated prototype is added to the SMU.

To address catastrophic forgetting in class prototype learning caused by model parameter updates, we introduce the MRaM mechanism within Mecoin. MRaM stores probability distributions for each class prototype, allowing direct access via indexing once input nodes are processed through MeCs and corresponding class prototypes are retrieved from SMU. This mechanism separates class prototype learning from node probability distribution learning, effectively mitigating forgetting issues caused by parameter updates. Additionally, to enhance the maintenance and updating of knowledge base, we integrate a Graph Knowledge Interaction Module (GKIM) within MRaM. GKIM transfers information about identified classes from MRaM to GNN and extracts knowledge of new classes from GNN back to MRaM, facilitating continuous knowledge updating and maintenance.

Contributions. The main contributions of this paper are as follows: i) We design Mecoin, a novel framework that effectively mitigates catastrophic forgetting in GFSCIL by integrating the SMU and MRaM; ii) We design the SMU, which efficiently learns class prototypes by facilitating interaction between node features and existing class prototypes, while extracting local graph structures of input nodes; iii) We propose the MRaM, which reduces the loss of prior knowledge during parameter fine-tuning by decoupling the learning of class prototypes from node probability distributions; iv) We analyze the benefits of separating class prototype learning from node probability distribution learning, considering generalization error bounds and VC dimensions. We also explore how different MRaM-GNNinteraction patterns affect model performance; v) Through extensive empirical studies, we demonstrate Mecoin's significant advantages over current state-of-the-art methods.

GNN Encoder

Graph data

Classifier

Prototype

Representations

=

=

matching

Memory losses

for keeping memory

Updating losses

for new knowledge

class

representation

(a) Graph neural network

(c) Structured Memory Unit

(f) Memory Representation Adaptive Module

=

…

…

Interaction with

old knowledge

extracts local graph

structure information

(b) MeCs

New class

representation

(d) GKIM

2 Notation

Let G 0 = ( V 0 , E 0 ) be the base graph with node set V 0 and edge set E 0 . We consider T temporal snapshots of G 0 , each corresponding to a GFSCIL task or session. Denote S = { S 0 , S 1 , . . . , S T } as the set of sessions including the pre-training session S 0 , and C = { C 0 , C 1 , . . . , C T } as the family of class sets within each session. The graph under session S i ( i = 1 , 2 , ..., T ) is denoted as G i = ( V i , E i ) , with node feature matrix and adjacency matrix represented by X i = ( x 1 i , ..., x |V i | i ) ⊤ ∈ R |V i |× d and A i ∈ R d × d respectively. For session S i , let X tr i ∈ R K | C i |× d and Y tr i be the features and corresponding labels of the training nodes respectively, where K is the sample size of each class in C i , thus defining a C i -way K -shot GFSCIL task. Let Y i be the label space of session S i , and we assume that the label spaces of different sessions are disjoint, i.e., Y i ∩ Y j = ∅ if i = j . Our goal is to learn a model f θ across successive sessions that maintains strong performance in the current session and also retains memory of the past sessions.

3 Efficient Memory Construction and Interaction Module

In this section, we present a comprehensive overview of our proposed framework, Mecoin. Unlike previous methods that are hampered by inefficient memory construction, low computational efficiency, and extensive parameter tuning-which often lead to the loss of prior knowledge-Mecoin enhances the learning of representative class prototypes. It achieves this by facilitating interaction between input nodes and seen class prototypes stored in the SMU, while integrating local graph structure information of the input nodes. Moreover, Mecoin decouples the learning of class prototypes from their corresponding probability distributions, thereby mitigating the loss of prior knowledge during both the prototype learning and classification processes. Fig. 1 illustrates the architecture of Mecoin.

3.1 Structured Memory Unit

To acquire and store representative class prototypes, we develop SMU within Mecoin. Let M be the set of class prototypes { m 0 , m 1 , . . . , m n -1 } , where n = | C T -1 | refers to the total number of classes learned from the past T -1 sessions, with each m i ∈ R k ( ∀ i ∈ [ n ] ). For current session S T ,

̸

the training node features X tr T are encoded through a pre-trained GNN model g ϕ :

where Z T ∈ R ( | C T | K ) × h . During the training process, the parameters of the pre-trained GNN, denoted as ϕ , remain fixed, and the training set used for pre-training is excluded from the subsequent GFSCIL tasks. To mitigate the significant memory usage resulting from caching metalearning samples and incorporate as much graph structural information as possible, we design MeCs within the SMU. MeCs merges graph structure information from the past sessions with that of the current session by interacting the encoded features of node Z T with the class prototypes in M . Specifically, MeCs firstly facilitates the interaction between Z T and the SMU-stored prototypes M 0: T -1 ≜ ( m 0 , m 1 , . . . , m n -1 ) ⊤ through a self-attention mechanism:

where W Q ∈ R h × m , W K ∈ R k × m , W V ∈ R k × h are learnable weight matrices and H T ∈ R ( | C T | K ) × h . Subsequently, to reduce memory consumption, we perform dimensionality reduction on H T through Gaussian random projection, resulting in ˜ H T ∈ R ( | C T | K ) × r , where 0 < r < k < h . MeCs then extracts local graph structure information of G T via the vanilla self-attention on X tr T , and preserves this information in the GraphInfo G T :

where G T ∈ R ( | C T | K ) × ( k -r ) and is integrated with ˜ H T by concatenation:

To determine the class prototypes under session S T , we perform k -means clustering on U T :

where M T ∈ R | C T |× k . In addition, to improve the utilization of graph structural information, we optimize U T by the edge loss L edge defined as follows:

For each node feature(i.e. each row) in X i , We identify the nearest class prototype by minimizing the Euclidean distance:

where u l i is the l -th row in U i .

3.2 Memory Representation Adaptive Module

In the traditional GFSCIL paradigm, adapting to evolving graph structures requires continuous learning and updating of class prototypes based on the current task's graph structure, alongside classifier retraining. This process, which involves adjusting parameters during class prototype learning, can lead the classifier to forget information about past categories, exacerbating the model's catastrophic forgetting. To address this, we introduce the MRaM within Mecoin. MRaM tackles this challenge by decoupling class prototype learning from class representation learning, caching probability distributions of seen categories. This separation ensures that class prototype updates don't affect the model's memory of probability distributions for nodes in seen categories, thus enhancing the stability of prior knowledge retention [14, 15, 16].

To maintain the model's memory of the prior knowledge, we introduce GKIM into MRaM for information exchange between Mecoin and the model. Specifically, let P = { p 0 , p 1 , . . . , p n -1 } denote the class representations learned from the GNN and stored in MRaM, where each p i corresponds to its respective class prototype m i . For a node feature x s from the seen classes with its class representation p x s , let p MLP x s be the category predicted probabilities learned from GNN and MLP, then GKIM transfers the prior knowledge stored in Mecoin to the model through distillation that based on the memory loss function:

where N s is the total number of samples from the seen classes.

Furthermore, to update the category representations of new classes in MRaM, GKIM updates the newly learned knowledge from the model to Mecoin via distillation with the update loss function in Eq.9. For any node feature x u from the unseen classes, its category representation is randomly initialized as p 0 x u , and its predicted probability vector is p MLP x u . Through distillation, p 0 x u is updated to incorporate the new classes' representations which are thus stored in Mecoin.

where N u is the total number of samples from the unseen classes. The overall loss of Mecoin consists of the MLP classification loss L cls , L edge , L memory and L update .

3.3 Theoretical Analysis

In this section, we analyze the advantages of Mecoin and how it improves models generalization from the perspective of generalization error. Besides, we also provide insights from the viewpoint of VC-dimension by comparing GKIM with non-parametric methods and MLPs in classifying category representations stored in MRaM, and distilling Mecoin as a teacher model with GNN model.

What are the advantages of Mecoin over other models? In few-shot learning, the limited training data and the overall samples in current session S T often have different distributions, which leads to overfitting. However, Mecoin can mitigate overfitting since it has a lower bound of generalization error than other corresponding models (Thm.1). Before introducing our theoretical result, we first supplement some notations. For the current session S T , let X T and Y T be the sample space and label space of X tr T respectively, and T T = { ( x i T , y i T ) } N i =1 be the training samples. Let F be a hypothesis class, f θ M and ˆ f ∈ F be Mecoin and other corresponding models trained on T T , and assume that the inputs x T ∈ X T undergo distributional shifts through any function g ϵ . Then by comparing the generalization error bounds of f θ M with ˆ f , we demonstrate in Thm.1 (proof in Appendix B.1) that Mecoin excels in distributional shifts, indicating its stronger generalization capability. The result is given in Thm.1, which is derived from theorem 3.1 in [12].

Theorem 1: For any model f ∈ { f θ M , ˆ f } trained on T T , denote R as its generalization error, then there exists a constant c such that for any δ > 0 , the following holds with probability at least 1 -δ :

where R ϵ = 1 N ∑ y T ∈Y T ∑ m ∈ I y T M |I y T m | E z [ E ϵ [ ℓ ( f ( g ϵ ( x T )) , y T )] -ℓ ( f ( x T ) , y T ) | z ∈ C y T m ] , B ˆ f = 1 N ∑ y T ,m 2 E T T ,ξ [sup ˆ f ∈F ∑ |I y m T | i =1 ξ i ℓ ( ˆ f ( x i T ) y i T ) | x i T ∈ C m , y i T = y T ] + c √ ln(2 e/δ ) / 2 N where { ξ i } i are i.i.d. random variables uniformly taking values in {-1 , 1 } , and ℓ is the loss function L Mecoin , z = ( x T , y T ) , C y T m = { ( x , y ) ∈ X T ×Y T | y = y T , m = argmin i ∈ [ N ] d ( k α ( x ) , m i ) } , I y T m = { i ∈ [ N ] | x i T ∈ C m , y i T = y T } , C m = { x ∈ X T | m = argmin i ∈ [ |M| ] d ( k α ( x , m i ) } , I y T M = m ∈ [ |M| ] | |I y T m | ≥ 1 } and k α is the MeCs operation .

Why use GKIM to interact with GNN models? Unlike traditional knowledge distillation techniques that rely on high-capacity teacher models, GKIM uses probability distributions stored in MRaM to preserve node of seen class distributions. This prevents knowledge loss in GNN during teacher model training. For unseen classes, the classifier learns and stores their probability distributions in MRaM. Notably, updating unseen class distributions and extracting seen class distributions can also use non-parametric methods [12]. Thus, we must examine GKIM's advantages over conventional distillation and non-parametric methods. We analyze the VC dimension when category representations in MRaM are distilled into models, comparing scenarios where MRaM, non-parametric methods, and multi-layer perceptrons (MLPs) act as teacher models.

Theorem 2: If n class representations are selected from GKIM, i.e. n is the input size of GNN, then the VC dimension of GKIM is:

where H is the number of hidden neurons in MLP and p denotes the number of parameters of GNN.

In the above theorem, we take the common voting classifier as an example for non-parametric methods. It averages the category representations stored in MRaM and updates them through backpropagation. While this method has lower complexity, it is highly sensitive to the initialization of category representations due to its reliance solely on the average while computing the prediction probabilities. When using MLP to update category representations, its parameters require fine-tuning to accommodate new categories that will lead to the forgetting of prior knowledge. In contrast, GKIM exhibits a higher VC-dimension compared to the other two methods, indicating superior expressive power. Additionally, GKIM selectively updates category representations in MRaM locally, preserving the model's memory of prior knowledge.

4 Experiments

In this section, we will evaluate Mecoin through experiments and address the following research questions: Q1 ). Does Mecoin have advantages in the scenarios of graph few-shot continual learning? Q2 ). How does MeCs improve the representativeness of class prototypes? Q3 ). What are the advantages of GKIM over other distillation methods?

4.1 Graph Few-Shot Continual Learning (Q1)

Experimental setting. We assess Mecoin's performance on three real-world graph datasets: CoraFull, CS, and Computers, comprising two citation networks and one product network. Datasets are split into a Base set for GNN pretraining and a Novel set for incremental learning. Tab.1 provides the statistics and partitions of the datasets. In our experiments, we freeze the pretrained GNN parameters and utilize them as encoders for subsequent few-shot class-incremental learning on graphs. For CoraFull, the Novel set is divided into 10 sessions, each with two classes, using a 2-way 5-shot GFSCIL setup. CS's Novel set is split into 10 sessions, each with one class, adopting a 1-way 5-shot GFSCIL configuration. Computers' Novel set is segmented into 5 sessions, each with one class, also using a 1-way 5-shot GFSCIL setup. During the training process, the training samples for session 0 are randomly selected from each class of the pre-trained testing set, with 5 samples chosen as the training set and the remaining testing samples used as the test set for training. It is worth noting that for each session, during the testing phase, we construct a new test set using all the testing samples of seen classes to evaluate the model's memory of all prior knowledge after training on the new graph data. Our GNN and GAT models feature two hidden layers. The GNN has a consistent dimension of 128, while GAT varies with 64 for CoraFull and 16 for CS and Computers. Training parameters are set at 2000 epochs and a learning rate of 0.005.

Baselines. 1) Elastic Weight Consolidation (EWC) [17]-imposes a quadratic penalty on weights to preserve performance on prior tasks. 2) Learning without Forgetting (LwF) [18]retains previous knowledge by minimizing the discrepancy between old and new model outputs. 3) Topology-aware Weight Preserving (TWP) [7]-identifies and regularizes parameters critical for graph topology to maintain task performance. 4) Gradient Episodic Memory (GEM) [19]-employs an episodic memory to adjust gradients during learning, preventing loss increase from prior tasks. 5) Experience

Replay GNN (ER-GNN) [3]- incorporates memory replay into GNN by storing key nodes from prior tasks. 6) Memory Aware Synapses (MAS) [20] evaluates parameter importance through sensi-

HAG-Meta

Geometer

OURS

(a)

0

10

20

30

40

50

60

70

Average Accuracy

2

4

6

8

10

(b)

20

30

40

50

60

70

80

90

HAG-Meta

Geometer

OURS

HAG-Meta

Geometer

OURS

(c)

0

100

200

300

400

500

600

Memory Usage

tivity to predictions, distinct from regularization-based EWC and TWP. 7) HAG-Meta [10],a GFSCIL approach, preserves model memory of prior knowledge via task-level attention and node class prototypes. 8) Geometer [11]- adjusts attention-based prototypes based on geometric criteria, addressing catastrophic forgetting and imbalanced labeling through knowledge distillation and biased sampling in GFSCIL.

Experimental results. We execute 10 tests for each method across three datasets, varying the random seed, and detailed the mean test accuracy in Tables 2, 3, and 4. Key findings are as follows:1) Mecoin surpasses other benchmarks on CoraFull, CS, and Computers datasets, exemplified by a 66.72% improvement over the best baseline on the Computers dataset across all sessions. 2) Mecoin maintains a performance edge and exhibits reduced forgetting rates, substantiating its efficacy in mitigating catastrophic forgetting. 3) Geometer and HAG-Meta perform poorly in our task setting, likely because, to ensure a fair comparison, the models do not use any training data from the metalearning or pre-training processes during training. The computation of class prototypes in these two methods heavily relies on the graph structure information provided by this data. Additionally, in Fig.2, we compare the results of Geometer and HAG-Meta tested under the experimental conditions given in their papers with the results of Mecoin on the CoraFull dataset. The experimental results indicate that our method still achieves better performance and forgetting rates than these two methods under low memory conditions, demonstrating the efficiency of our method in terms of memory.

4.2 MeCs for Memory (Q2)

We conduct relevant ablation experiments on MeCs across the CoraFull, CS, and Computers datasets:1) Comparing the impact of MeCs usage on model performance and memory retention; 2) Analyzing the effect of feature of node interaction with class prototypes stored in SMU on model

performance and memory retention;3)Assessing the impact of using graphInfo on model performance and memory retention. We conduct a randomized selection of ten classes from the test set, extracting 100 nodes per class to form clusters with their corresponding class prototypes within the SMU. Subsequently, we assess the efficacy of MeCs and its constituent elements on model performance and the rate of forgetting. In cases where the MeCs was not utilized, we employ the k-means method to compute class prototypes. In experiment, we use GCN as backbone. The experimental parameters and configurations adhered to the standards established in prior studies.

Experimental Results. The results of the ablation experiments on CoraFull and CS are shown in Fig.3 and results on other datasets are shown in Appendix.A. We deduce several key findings from the figure:1) The class prototypes learned by MeCs assist the model in learning more representative prototype representations, demonstrating the effectiveness of the MeCs method. 2) The difference in accuracy between scenarios where graphInfo is not used and where there is no interaction with class prototypes is negligible. However, the scenario without graphInfo exhibits a higher forgetting rate, indicating that local graph structural information provides greater assistance to model memory. This may be because local structural information reduces the intra-class distance of nodes, thereby helping the model learn more discriminative prototype representations.

10

Matching of test samples to SMU prototypes under different methods

10

5

0

5

10

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

With MeCs, Ave_acc:70.80%, PD:17.26%

10

5

0

5

10

10

5

0

5

10

No Inter, Ave_acc:67.78%, PD:23.96%

10

5

0

5

10

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

No GraphInfo, Ave_acc:67.61%, PD:24.83%

5

0

5

10

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

No MeCs, Ave_acc:63.38%, PD:39.34%

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class Centers

10

5

0

5

10

10

5

0

5

10

With MeCs, Ave_acc:80.02%, PD:12.81%

10

5

0

5

10

10

5

0

5

10

No Inter, Ave_acc:78.49%, PD:13.27%

10

5

0

5

10

10

5

0

5

10

No GraphInfo, Ave_acc:79.19%, PD:12.36%

10

5

0

5

10

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

No MeCs, Ave_acc:76.94%, PD:11.37%

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class Centers

Matching of test samples to SMU prototypes under different methods

10

5

0

5

10

With MeCs, Ave_acc:79.04%, PD:19.03%

10

5

0

5

10

No Inter, Ave_acc:78.31%, PD:18.55%

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

No GraphInfo, Ave_acc:78.26%, PD:18.76%

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

No MeCs, Ave_acc:77.89%, PD:18.71%

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class Centers

Matching of test samples to SMU prototypes under different methods

5

0

5

10

10

5

0

5

10

5

0

5

10

10

5

0

5

4.3 GKIM for Memory (Q3)

In our experimental investigation across three distinct datasets, we scrutinized the influence of varying interaction modalities between MRaM and GNN models on model performance and the propensity for forgetting. We delineate three distinct scenarios for analysis: 1) Class representations stored in MRaM are classified using non-parametric methods and used as the teacher model to interact with the GNN. 2) Class representations stored in MRaM are classified using MLP and Mecoin is employed as the teacher model to transfer knowledge to the GNN model. 3) GNN models are deployed in isolation for classification tasks, without any interaction with Mecoin. In this experiment, we use GCN as backbone. Throughout the experimental process, model parameters were held constant, and the experimental configurations were aligned with those of preceding studies. Due to space constraints, we include the results on the Computers dataset in the Appendix.A.

Experimental Results . It is worth noting that due to the one-to-one matching between MRaM and SMU via non-parametric indexing, there is no gradient back-propagation between these two components. This implies that updates to MRaM during training do not affect the matching between node features and class prototypes. Our experimental findings are depicted in Fig.4, and they yield several key insights: 1) GKIM outperforms all other interaction methods, thereby substantiating its efficacy. 2) The second interaction mode exhibits superior performance compared to the first and third methods. This is attributed to the MLP's higher VC-dimension compared to the non-parametric

2

4

6

8

10

CoraFull

0

20

40

60

80

OURS

no-mecoin

non-para

mlp

2

4

6

8

10

CS

0

20

40

60

80

100

OURS

no-mecoin

non-para

mlp

OURS

no-mecoin

non-para

mlp

CoraFull

0

10

20

30

40

50

60

70

80

PD

16.24

80.12

22.74

36.15

OURS

no-mecoin

non-para

mlp

CS

0

20

40

60

80

PD

37.41

88.56

42.93

38.11

method, which gives it greater expressive power to handle more complex sample representations. However, the use of MLP results in a higher forgetting rate compared to the non-parametric method. This is because when encountering samples from new classes, the parameters of MLP undergo changes, leading to the loss of prior knowledge. For the third method, extensive parameter finetuning leads to significant forgetting of prior knowledge. Method one performs less effectively than GKIM, primarily because GKIM, with its larger VC-dimension, ensures that the process of updating class representations does not affect representations of other classes stored in MRaM.

5 Conclusion

Current GFSCIL methods typically require a large number of labeled samples and cache extensive past task data to maintain memory of prior knowledge. Alternatively, they may fine-tune model parameters at the expense of sacrificing the model's adaptability to current tasks. To address these challenges, we propose the Mecoin for building and interacting with memory. Mecoin is made up of two main parts: the Structured Memory Unit (SMU), which learns and keeps class prototypes, and the Memory Representation Adaptive Module (MRaM), which helps the GNN preserve prior knowledge. To leverage the graph structure information for learning representative class prototypes, SMU leverages MeCs to integrate past graph structural information with interactions between samples and the class prototypes stored in SMU. Additionally, Mecoin introduces the MRaM, which separates the learning of class prototypes and category representations to avoid excessive parameter fine-tuning during prototype updates, thereby preventing the loss of prior knowledge. Furthermore, MRaMinjects knowledge stored in Mecoin into the GNN model through GKIM, preventing knowledge forgetting. We demonstrate our framework's superiority in graph few-shot continual learning with respect to both generalization error and VC dimension, and we empirically show its advantages in accuracy and forgetting rate compared to other graph continual learning methods.

References

Junshan Wang, Guojie Song, Yi Wu, and Liang Wang. Streaming graph neural networks via continual learning. In Proceedings of the 29th ACM international conference on information &knowledge management , pages 1515-1524, 2020.

Hongbo Bo, Ryan McConville, Jun Hong, and Weiru Liu. Ego-graph replay based continual learning for misinformation engagement prediction. In 2022 International Joint Conference on Neural Networks (IJCNN) , pages 01-08. IEEE, 2022.

Fan Zhou and Chengtai Cao. Overcoming catastrophic forgetting in graph neural networks with experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 4714-4722, 2021.

Yilun Liu, Ruihong Qiu, and Zi Huang. Cat: Balanced continual graph learning with graph condensation. In 2023 IEEE International Conference on Data Mining (ICDM) , pages 11571162. IEEE, 2023.

Xikun Zhang, Dongjin Song, and Dacheng Tao. Ricci curvature-based graph sparsification for continual graph representation learning. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Xikun Zhang, Dongjin Song, and Dacheng Tao. Hierarchical prototype networks for continual graph representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4622-4636, 2022.

Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 8653-8661, 2021.

Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.04043 , 2017.

Kaize Ding, Jianling Wang, Jundong Li, Kai Shu, Chenghao Liu, and Huan Liu. Graph prototypical networks for few-shot learning on attributed networks. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 295-304, 2020.

Zhen Tan, Kaize Ding, Ruocheng Guo, and Huan Liu. Graph few-shot class-incremental learning. In Proceedings of the fifteenth ACM international conference on web search and data mining , pages 987-996, 2022.

Bin Lu, Xiaoying Gan, Lina Yang, Weinan Zhang, Luoyi Fu, and Xinbing Wang. Geometer: Graph few-shot class-incremental learning via prototype representation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining , pages 1152-1161, 2022.

Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Curtis Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. In International Conference on Machine Learning , pages 34431-34455. PMLR, 2023.

Biqing Qi, Xinquan Chen, Junqi Gao, Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Interactive continual learning: Fast and slow thinking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12882-12892, 2024.

Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 , 2018.

Biqing Qi, Junqi Gao, Xingquan Chen, Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Contrastive augmented graph2graph memory interaction for few shot continual learning. arXiv preprint arXiv:2403.04140 , 2024.

Biqing Qi, Junqi Gao, Xinquan Chen, Dong Li, Weinan Zhang, and Bowen Zhou. Sr-cis: Self-reflective incremental system with decoupled memory and reasoning. arXiv preprint arXiv:2408.01970 , 2024.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV) , pages 139-154, 2018.

Marek Karpinski and Angus Macintyre. Polynomial bounds for vc dimension of sigmoidal and general pfaffian neural networks. Journal of Computer and System Sciences , 54(1):169-176, 1997.

Stefanie Jegelka. Theory of graph neural networks: Representation and learning. In The International Congress of Mathematicians , 2022.

A Ablation Experiments

In Section 4.2, we presented the relevant ablation experiments on the CoraFull and CS dataset. Below are the experimental results of the model on the Computers datasets.

5

0

5

10

5

0

5

10

With MECO, Ave_acc:79.22%, PD:12.02%

10

5

0

5

10

8

6

4

2

0

2

4

6

No Inter, Ave_acc:78.57%, PD:13.35%

8

6

4

2

0

2

4

6

8

6

4

2

0

2

4

No GraphInfo, Ave_acc:79.27%, PD:12.44%

10

5

0

5

10

10

5

0

5

10

No MECO, Ave_acc:78.21%, PD:12.94%

Category 0

Category 1

Category 2

Category 3

Class Centers

Matching of test samples to SMU prototypes under different methods

The experimental results in this section are similar to those on the CoraFull and CS dataset, and relevant experimental analyses can be referred to in the main text. Additionally, we designed related ablation experiments on the dimensions of GraphInfo and its integration position with node features. Detailed results are shown in tab.5, 6. From these results, we can draw the following conclusions:1)It is evident from the tables that concatenating GraphInfo to the left of node features with a dimension of 1 yields the best performance; 2)The model's performance decreases as the dimension of GraphInfo increases, while the forgetting rate generally exhibits a decreasing trend. This indicates that local graph structural information provides some assistance to the model's memory. When the dimension of class prototypes is fixed, an increase in the dimension of GraphInfo implies less graph structural information extracted from prototypes of seen classes in past sessions. Consequently, MeCs integrates less graph structural information, making it difficult for the model to learn representative class prototypes, thereby limiting the model's performance.

Furthermore, for Section 4.3, the relevant ablation experiment results on the Computers dataset can be found in Figure 8.

80

1

2

5

6

0

20

40

60

80

OURS

no-mecoin

non-para

mlp

mlp

0

10

20

30

40

50

60

70

PD

12.02

76.53

21.01

12.61

OURS

no-mecoin

non-para

3

4

Computers

Computer

B Proof of Theorems

B.1 Proof of Theorem 1

Proof. By instituting C y T m with C y k , C m with C k , I y T m with I y k , I y T M with I y K , m i with K i , and k α with k β ◦ z α , our proof is similar to the proof of theorem 3.1 in [12] (please refer to [12] for more details). For f ∈ { f θ M , ˆ f } , through similar procedure in the proof of [12], we obtain the coarse upper bound

of generalization error R :

We then further conduct the tighter generalization bound for ˆ f and f θ M separately based on Eq.13. Consider the case of f = ˆ f , for the second term of right hand side of equation (13), according to Lemma 4 of (Pham et al., 2021) and the fact that ∑ y T ∈Y T ∑ m ∈ I y T M |I y T m | = N , we obtain that for any δ > 0 , with probability at least 1 -δ ,

where R y T ,m ( l ◦ ˆ F ) = E T T ,ξ [sup ˆ f ∈F ∑ |I y m T | i =1 ξ i ℓ ( ˆ f ( x i T ) y i T ) | x i T ∈ C m , y i T = y T ] , and { ξ i } i are independently and identically distributed random variables that randomly taking values in {-1 , 1 } . Moreover, using Cauchy-Schwarz inequality, we have that

Then by ln( |Y T ||M| /δ ) ≤ max(1 , ln( |Y T ||M| )) ln( e/δ ) , we obtain the final tighter generalization upper bound of ˆ f :

For the case of f = f θ M , the second term in the right hand side of equation (13) equals 0. In fact, based on the definition of C y T m , I y T m and I y T M , and the matching method (i.e. based on the smallest Euclidean distance) between the input node features x T and the class prototypes m in Mecoin, we have that

where p T is the value that is uniquely corresponding to the class prototype m = k α ( x T ) , and h θ represents the knowledge interchange process in GKIM.

□

B.2 Proof of Theorem 2

proof : When we use a simple non-parametric decoder function which uses average pooling to calculate the element-wise average of all the fetched memory representation and then applies a softmax function to the output, the decoder is the same as a single hidden layer nets with fixed input weights. This network can be formulated as follows

where A i is the i -th row of weight matrix, b i is the bias, c i is the output layer weights. For average pooling, c i and b i is 0, and A i is 1 n . So is easy to prove that the VC-dimension is n + 1 . Then, according to [21], the VC-dimension of MLP with one hidden layer is n 2 H 2 . Then we consider the case when graph structure information is induced. When we use the distillation technique to inject graph structure information into GKIM via graph neural networks, the upper bound on the capacity of memory representation in GKIM is the expressive capacity of the graph neural networks. Thus, according to [22], we have that the VC-dimensinon of GKIM with graph structure information is p 2 n 2 H 2 . In summary, when graph structure information is introduced, the upper bound of the VC dimension of GKIM is increased and its expressive power is also enhanced.

C Parameters and Devices

The relevant experimental parameters in this paper were determined through grid search. The parameter settings for each dataset are shown in Table 7.

The environment in which we run experiments is:

CPU information:24 vCPU AMD EPYC 7642 48-Core Processor

GPU information:RTX A6000(48GB)

D Limitations

While our method has shown promising performance in graph few-shot incremental learning, there are still some issues that need to be addressed: 1) Our approach assumes that the graph structures across all sessions originate from the same graph. However, in real-world scenarios, data from different graphs may be encountered, and we have not thoroughly explored this issue; 2) Although our method currently maintains model performance with only a small number of samples, further validation is needed to assess whether it can still achieve excellent performance under low-resource conditions where graph structural information is scarce.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Contributions are detailed in the last paragraph of the Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We give the limitations in Appendix D.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theoretical results are detailed in Section3.3 and Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We report all settings and details of reproducible experiments in Section 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will make our code public after the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We report all settings and details of experiments in Section 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because we repeated each experiment in the paper 10 times and reported the mean of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix C of the paper reports the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the thesis complied with the NeurIPS Code of Ethics in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our approach does not address social impacts and this question is not applicable to our approach.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use explicitly are mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Matching of test samples to SMU prototypes under different methods

10

5

0

5

10

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

With MeCs, Ave_acc:70.80%, PD:17.26%

10

5

0

5

10

10

5

0

5

10

No Inter, Ave_acc:67.78%, PD:23.96%

10

5

0

5

10

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

No GraphInfo, Ave_acc:67.61%, PD:24.83%

5

0

5

10

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

No MeCs, Ave_acc:63.38%, PD:39.34%

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class Centers

10

5

0

5

10

10

5

0

5

10

With MeCs, Ave_acc:80.02%, PD:12.81%

10

5

0

5

10

10

5

0

5

10

No Inter, Ave_acc:78.49%, PD:13.27%

10

5

0

5

10

10

5

0

5

10

No GraphInfo, Ave_acc:79.19%, PD:12.36%

10

5

0

5

10

12.5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

No MeCs, Ave_acc:76.94%, PD:11.37%

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class Centers

Matching of test samples to SMU prototypes under different methods

Figure 7: Caption for Image 2

10

5

0

5

10

10

5

0

5

10

With MeCs, Ave_acc:79.04%, PD:19.03%

10

5

0

5

10

5

0

5

10

No Inter, Ave_acc:78.31%, PD:18.55%

10

5

0

5

10

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

No GraphInfo, Ave_acc:78.26%, PD:18.76%

10

5

0

5

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

No MeCs, Ave_acc:77.89%, PD:18.71%

Class 0

Class 1

Class 2

Class 3

Class 4

Class 5

Class 6

Class 7

Class 8

Class 9

Class Centers

Matching of test samples to SMU prototypes under different methods

GNN Encoder

Graph data

Classifier

Prototype

Representations

=

=

matching

Memory losses

for keeping memory

Updating losses

for new knowledge

class

representation

(a) Graph neural network

(c) Structured Memory Unit

(f) Memory Representation Adaptive Module

=

…

…

Interaction with

old knowledge

extracts local graph

structure information

(b) MeCs

New class

representation

(d) GKIM

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of graph few-shot class-incremental learning, proposes a novel framework named Mecoin to address the challenges of catastrophic forgetting and overfitting. Mecoin includes two key componets: the structured memory unit (SMU) for storing and updating prototypes and the memory representation adaptive module (MRaM) for separating the learning of prototypes and class representations. Based on SMU, Mecoin efficiently maintain representative prototypes preventing the overfitting problem caused by few-shot setting, and rely on MRaM, Mecoin avoids heavily finetuning the model parameters which solved the catastrophic forgetting problem in class-incremental learning.\n\nSTRENGTHS:\n1.Stores past knowledge efficiently with low storage space by using the SMU.\n2.Avoids catastrophic forgetting by separating the learning of prototypes and class representations.\n3.The experiments show significant improvement compared to other state-of-the-art methods.\n\nWEAKNESSES:\n1.The abbreviations of the proposed method are confusing. For example, in some places, the Memory Representation Adaptive Module is abbreviated as MRaM, while in others, it is MRAM. Additionally, the full model name, Mecoin, is too similar to one of its submodules, the Memory Construction module (MeCo).\n2.Fig. 1 is confusing; it doesn’t show the details and advantages of the SMU and MRaM structures.\n3.In Sec. 3.1, Eq. 6 lacks explanatory information. How does Eq. 6 integrate edge information into the loss function?\n4.The proof of Theorem 1 is too brief. I recommend the authors provide detailed derivation rather than citing a lemma from another paper.\n5.In Sec. 3.2, Eqs. 8 and 9 are incomprehensible. I assume the GKIM maintains past knowledge and updates new knowledge with different elements 𝑝. For example, if the dataset has 100 classes, \n𝑝1:60 would be maintained for past knowledge and 𝑝61:100 for new knowledge. Thus, 𝑁 should be 60. It’s challenging to explain how new knowledge is updated in Eq. 9.\n6.In Sec. 4.1, the process of pretraining the GNN needs to be explained. Is the pretraining phase a standard graph node classification task or a few-shot node classification task? Additionally, in Tab. 3, the comparison method MAG-Meta is mentioned twice; I guess one should be Geometer. Also, if the proposed method is named Mecoin, why does Mecoin appear among the comparison methods in Tabs. 2, 3, and 4?\n7.In Sec. 4.2, the toy dataset is too simple. Sampling only 4 classes makes it difficult to demonstrate the ability of different settings to handle overlapping problems. I recommend sampling at least 10 classes.\n\nQUESTIONS:\nPlease check the weaknesses\n\nLIMITATIONS:\nThe paper is poor-writing and many places lack detailed explanations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
The authors focus on graph few-shot class-incremental learning. The authors first introduce Mecoin to efficiently construct and preserve memory. To avoid extensive parameter finetuning and forgetting, the authors introduce a memory representation adaptive module called MRaM to separate the learning of prototypes and class representations. Besides, the authors propose Graph Knowledge Interchange Module (GKIM) to injects past knowledge information into GNN. Additional analyses illustrate the effectiveness of the methods.

STRENGTHS:
1. The paper is well-written, and the motivations for each part are clear.
2. The reported performance out-performs many baselines.
3. The theoretical and experimental analyses successfully illustrate the effectiveness of the methods.

WEAKNESSES:
1. The paper does not report error bars, standard deviations or provide detailed information about the statistical significance of the experiments, which is important for understanding the reliability and variability of the results.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
The paper has a section discussing several limitations in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

comments: SUMMARY:
This paper mainly focuses on graph few-shot class-incremental learning. To alleviate the significant memory consumption and catastrophic forgetting of old knowledge, it proposes to store only old class prototypes and update class prototypes by considering the interaction among nodes and prototypes. Furthermore, it gives a theoretical analysis on the generalization error. Experimental results show that the proposed method can achieve better results.

STRENGTHS:
The proposed idea of storing old class prototypes and updating class prototypes by considering their interaction with node features seems reasonable, as it has been proved to be useful in few-shot class-incremental learning. Experiments are relative sufficient to show its efficacy.

WEAKNESSES:
In general, this paper needs further polish, from my point of view. Just name a few, the abbreviations MRaM and MRAM are not consistent; the adjacency matrix A is incorrect; some sentences need careful attention. In addition, the motivation or the reason why the proposed method uses self-attention and MRAM is not clearly explained.

QUESTIONS:
1.What are the meta-learning samples? There is no definition or explanation about them. It would be much better if more explanation would be given. 
2.Why using Gaussian random projection to reduce the dimensionality, instead of others?
3.Why concatenating G_T and H_T^p? Could the authors give more explanation?
4.What is the difference between class prototype and class representation? It is very hard to tell from the paper.

LIMITATIONS:
The authors have discussed the limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13