An Efficient Memory Module for Graph Few-Shot Class-Incremental Learning

Authors: dong  li
aijia  zhang
junqi  gao
biqing  qi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:52:25

Abstract: Graph incremental learning has gained widespread attention for its ability to mitigate catastrophic forgetting for graph neural networks (GNN). Conventional methods typically require numerous labels for node classification. However, obtaining abundant labels is often challenging in practice, which makes graph few-shot incremental learning necessary. Current approaches rely on large number of samples from meta-learning to construct memories, and heavy fine-tuning of the GNN parameters that lead to the loss of past knowledge. These result in significant memory consumption and loss of past knowledge information, respectively. To tackle these issues, We introduce Mecoin to efficient construct and Preserve memory. For efficient storage and update of class prototypes, Mecoin use Structured Memory Unit (SMU) to cache prototypes of the seen classes and update new class prototypes through interaction between nodes and the cached prototypes by Memory Construction module(MeCo). Besides, to avoid extensive parameter fine-tuning and forgetting, we introduce a Memory Representation Adaptive Module called MRaM to separate the learning of prototypes and class representations and use Graph Knowledge Interchange Module (GKIM) to injects past knowledge information into GNN. We analyze the effectiveness of our paradigm from the perspectives of generalization error, and discuss the impact of different distillation methods on model performance through experiments and VC-dimension. By comparison with other related methods, we validate that Mecoin achieves higher accuracy and lower forgetting rate.

Main paper: 
 introduction: 
 In the field of graph learning, conventional methods often assume that graphs are static [1] . However, in the real world, graphs tend to grow over time, with new nodes and edges gradually emerging. For example, in citation networks, new papers are published and cited; in e-commerce, new products are introduced and updated; and in social networks, new social groups form as users join. In these dynamic contexts, simply updating graph representation learning methods with new data often leads to catastrophic forgetting of previously acquired knowledge.Despite numerous methods proposed to mitigate the catastrophic forgetting problem in Graph Neural Networks(GNNs) [2, 3, 4] , a critical and frequently neglected challenge is the scarcity of labels for newly introduced nodes. Most current graph incremental learning methods [5, 6] combat catastrophic forgetting by retaining a substantial number of nodes from previous graphs to preserve prior knowledge. However, these methods become impractical in graph few-shot class-incremental learn-ing(GFSCIL) scenarios due to the limited labeled node samples. Some methods [7, 1] employ regularization to maintain the stability of parameters critical to the graph's topology. Yet, in GFSCIL, the label scarcity complicates accurate assessment of the relationship between parameter importance and the underlying graph structure, thus increasing the difficulty of designing effective regularization strategies. Consequently, the issue of label scarcity hampers the ability of existing graph continual learning methods to effectively generalize in graph few-shot learning scenarios.GFSCIL presents two critical challenges: 1)How can we empower models to learn effectively from limited samples? 2)How can we efficiently retain prior knowledge with imited samples? While the first challenge has been extensively explored in graph few-shot learning contexts [8, 9] , this paper focuses on the second. Currently, discussions on the latter issue within GFSCIL are relatively scarce. Existing methods [10, 11] primarily focus on enhancing models' ability to learn from few-shot graphs and preserve prior knowledge by learning class prototypes through meta-learning and attention mechanisms. However, to strengthen inductive bias towards graph structures and learn more representative class prototypes, these approaches require caching numerous training samples from the meta-learning process for subsequent GFSCIL tasks. This caching not only consumes significant memory but also imposes substantial computational costs. Furthermore, these methods extensively adjust parameters during prototype updates, risking the loss of previously acquired knowledge [12, 13] . These challenges underscore the need for innovative strategies that maintain efficiency without compromising the retention of valuable historical data-achieved through minimal memory footprint, high computational efficiency, and limited parameter adjustments.To address the aforementioned challenges, we introduce Mecoin, an efficient memory construction and interaction module. Mecoin consists of two core components: the Structured Memory Unit(SMU) for learning and storing class prototypes, and the Memory Representation Adaptive Module(MRaM) for dynamic memory interactions with the GNN. To effectively leverage graph structural information, we design the Memory Construction module(MeCs) within the SMU. MeCs facilitates interaction between features of the input node and prototype representations stored in the SMU through self-attention mechanisms, thereby updating sample representations. Then, it utilizes the local structural information of input nodes to extract local graph structural information and compute a local graph structure information matrix(GraphInfo). The updated sample representations and GraphInfo are used to calculate class prototypes for the input nodes. These newly obtained prototypes are compared with those stored in the SMU using Euclidean distance to determine whether the input samples belong to seen classes. If a sample belongs to a seen class, the corresponding prototype in the SMU remains unchanged. Conversely, if a sample belongs to an unseen class, its calculated prototype is added to the SMU.To address catastrophic forgetting in class prototype learning caused by model parameter updates, we introduce the MRaM mechanism within Mecoin. MRaM stores probability distributions for each class prototype, allowing direct access via indexing once input nodes are processed through MeCs and corresponding class prototypes are retrieved from SMU. This mechanism separates class prototype learning from node probability distribution learning, effectively mitigating forgetting issues caused by parameter updates. Additionally, to enhance the maintenance and updating of knowledge base, we integrate a Graph Knowledge Interaction Module (GKIM) within MRaM. GKIM transfers information about identified classes from MRaM to GNN and extracts knowledge of new classes from GNN back to MRaM, facilitating continuous knowledge updating and maintenance.Contributions. The main contributions of this paper are as follows: i) We design Mecoin, a novel framework that effectively mitigates catastrophic forgetting in GFSCIL by integrating the SMU and MRaM; ii) We design the SMU, which efficiently learns class prototypes by facilitating interaction between node features and existing class prototypes, while extracting local graph structures of input nodes; iii) We propose the MRaM, which reduces the loss of prior knowledge during parameter fine-tuning by decoupling the learning of class prototypes from node probability distributions; iv) We analyze the benefits of separating class prototype learning from node probability distribution learning, considering generalization error bounds and VC dimensions. We also explore how different MRaM-GNN interaction patterns affect model performance; v) Through extensive empirical studies, we demonstrate Mecoin's significant advantages over current state-of-the-art methods.  
notation: 
 Let G 0 = (V 0 , E 0 ) be the base graph with node set V 0 and edge set E 0 . We consider T temporal snapshots of G 0 , each corresponding to a GFSCIL task or session. Denote S = {S 0 , S 1 , . . . , S T } as the set of sessions including the pre-training session S 0 , and C = {C 0 , C 1 , . . . , C T } as the family of class sets within each session. The graph under session S i (i = 1, 2, ..., T ) is denoted as G i = (V i , E i ), with node feature matrix and adjacency matrix represented by X i = (x 1 i , ...,x |Vi| i) ⊤ ∈ R |Vi|×d and A i ∈ R d×d respectively. For session S i , let X tr i ∈ R K|Ci|×d and Y tr i be the features and corresponding labels of the training nodes respectively, where K is the sample size of each class in C i , thus defining a C i -way K-shot GFSCIL task. Let Y i be the label space of session S i , and we assume that the label spaces of different sessions are disjoint, i.e., Y i ∩ Y j = ∅ if i ̸ = j. Our goal is to learn a model f θ across successive sessions that maintains strong performance in the current session and also retains memory of the past sessions. 
efficient memory construction and interaction module: 
 In this section, we present a comprehensive overview of our proposed framework, Mecoin. Unlike previous methods that are hampered by inefficient memory construction, low computational efficiency, and extensive parameter tuning-which often lead to the loss of prior knowledge-Mecoin enhances the learning of representative class prototypes. It achieves this by facilitating interaction between input nodes and seen class prototypes stored in the SMU, while integrating local graph structure information of the input nodes. Moreover, Mecoin decouples the learning of class prototypes from their corresponding probability distributions, thereby mitigating the loss of prior knowledge during both the prototype learning and classification processes. Fig. 1 illustrates the architecture of Mecoin.To acquire and store representative class prototypes, we develop SMU within Mecoin. Let M be the set of class prototypes {m 0 , m 1 , . . . , m n-1 }, where n = |C T -1 | refers to the total number of classes learned from the past T -1 sessions, with each m i ∈ R k (∀i ∈ [n]). For current session S T , the training node features X tr T are encoded through a pre-trained GNN model g ϕ :EQUATIONwhere Z T ∈ R (|C T |K)×h . During the training process, the parameters of the pre-trained GNN, denoted as ϕ , remain fixed, and the training set used for pre-training is excluded from the subsequent GFSCIL tasks. To mitigate the significant memory usage resulting from caching metalearning samples and incorporate as much graph structural information as possible, we design MeCs within the SMU. MeCs merges graph structure information from the past sessions with that of the current session by interacting the encoded features of node Z T with the class prototypes in M. Specifically, MeCs firstly facilitates the interaction between Z T and the SMU-stored prototypes M 0:T -1 ≜ (m 0 , m 1 , . . . , m n-1 ) ⊤ through a self-attention mechanism:EQUATIONwhereW Q ∈ R h×m , W K ∈ R k×m , W V ∈ R k×h are learnable weight matrices and H T ∈ R (|C T |K)×h. Subsequently, to reduce memory consumption, we perform dimensionality reduction on H T through Gaussian random projection, resulting in HT ∈ R (|C T |K)×r , where 0 < r < k < h.MeCs then extracts local graph structure information of G T via the vanilla self-attention on X tr T , and preserves this information in the GraphInfo G T :EQUATIONwhere r) and is integrated with HT by concatenation:G T ∈ R (|C T |K)×(k-EQUATIONTo determine the class prototypes under session S T , we perform k-means clustering on U T :EQUATIONwhere M T ∈ R |C T |×k . In addition, to improve the utilization of graph structural information, we optimize U T by the edge loss L edge defined as follows:EQUATIONFor each node feature(i.e. each row) in X i , We identify the nearest class prototype by minimizing the Euclidean distance:EQUATIONwhere u l i is the l-th row in U i .In the traditional GFSCIL paradigm, adapting to evolving graph structures requires continuous learning and updating of class prototypes based on the current task's graph structure, alongside classifier retraining. This process, which involves adjusting parameters during class prototype learning, can lead the classifier to forget information about past categories, exacerbating the model's catastrophic forgetting. To address this, we introduce the MRaM within Mecoin. MRaM tackles this challenge by decoupling class prototype learning from class representation learning, caching probability distributions of seen categories. This separation ensures that class prototype updates don't affect the model's memory of probability distributions for nodes in seen categories, thus enhancing the stability of prior knowledge retention [14, 15, 16] .To maintain the model's memory of the prior knowledge, we introduce GKIM into MRaM for information exchange between Mecoin and the model. Specifically, let P = {p 0 , p 1 , . . . , p n-1 } denote the class representations learned from the GNN and stored in MRaM, where each p i corresponds to its respective class prototype m i . For a node feature x s from the seen classes with its class representation p xs , let p M LP xs be the category predicted probabilities learned from GNN and MLP, then GKIM transfers the prior knowledge stored in Mecoin to the model through distillation that based on the memory loss function:EQUATIONwhere N s is the total number of samples from the seen classes.Furthermore, to update the category representations of new classes in MRaM, GKIM updates the newly learned knowledge from the model to Mecoin via distillation with the update loss function in Eq.9. For any node feature x u from the unseen classes, its category representation is randomly initialized as p 0 xu , and its predicted probability vector is p M LP xu . Through distillation, p 0 xu is updated to incorporate the new classes' representations which are thus stored in Mecoin.EQUATION)where N u is the total number of samples from the unseen classes. The overall loss of Mecoin consists of the MLP classification loss L cls , L edge , L memory and L update .EQUATION)In this section, we analyze the advantages of Mecoin and how it improves models generalization from the perspective of generalization error. Besides, we also provide insights from the viewpoint of VC-dimension by comparing GKIM with non-parametric methods and MLPs in classifying category representations stored in MRaM, and distilling Mecoin as a teacher model with GNN model.What are the advantages of Mecoin over other models? In few-shot learning, the limited training data and the overall samples in current session S T often have different distributions, which leads to overfitting. However, Mecoin can mitigate overfitting since it has a lower bound of generalization error than other corresponding models (Thm.1). Before introducing our theoretical result, we first supplement some notations. For the current session S T , let X T and Y T be the sample space and label space of X tr T respectively, and T T = {(x i T , y i T )} N i=1 be the training samples. Let F be a hypothesis class, f M θ and f ∈ F be Mecoin and other corresponding models trained on T T , and assume that the inputs x T ∈ X T undergo distributional shifts through any function g ϵ . Then by comparing the generalization error bounds of f M θ with f , we demonstrate in Thm.1 (proof in Appendix B.1) that Mecoin excels in distributional shifts, indicating its stronger generalization capability. The result is given in Thm.1, which is derived from theorem 3.1 in [12] .Theorem 1: For any model f ∈ {f M θ , f } trained on T T , denote R as its generalization error, then there exists a constant c such that for any δ > 0, the following holds with probability at least 1 -δ:EQUATIONwhereR ϵ = 1 N y T ∈Y T m∈I y T M |I y T m |E z [E ϵ [ℓ(f (g ϵ (x T )), y T )] -ℓ(f (x T ), y T )|z ∈ C y T m ], B f = 1 N y T ,m 2E T T ,ξ [sup f ∈F |I y T m | i=1 ξ i ℓ( f (x i T )y i T )|x i T ∈ C m , y i T = y T ] + c ln(2e/δ)/2Nwhere {ξ i } i are i.i.d. random variables uniformly taking values in {-1, 1}, and ℓ is the loss functionL Mecoin , z = (x T , y T ), C y T m = {(x, y) ∈ X T × Y T | y = y T , m = argmin i∈[N ] d(k α (x), m i )}, I y T m = {i ∈ [N ] | x i T ∈ C m , y i T = y T }, C m = {x ∈ X T | m = argmin i∈[|M|] d(k α (x, m i )}, I y T M = m ∈ [|M|] | |I y T m | ≥ 1}and k α is the MeCs operation. Why use GKIM to interact with GNN models? Unlike traditional knowledge distillation techniques that rely on high-capacity teacher models, GKIM uses probability distributions stored in MRaM to preserve node of seen class distributions. This prevents knowledge loss in GNN during teacher model training. For unseen classes, the classifier learns and stores their probability distributions in MRaM. Notably, updating unseen class distributions and extracting seen class distributions can also use non-parametric methods [12] . Thus, we must examine GKIM's advantages over conventional distillation and non-parametric methods. We analyze the VC dimension when category representations in MRaM are distilled into models, comparing scenarios where MRaM, non-parametric methods, and multi-layer perceptrons (MLPs) act as teacher models.Theorem 2: If n class representations are selected from GKIM, i.e. n is the input size of GNN, then the VC dimension of GKIM is:V CD =    O(n + 1) voting classifier O(p 2 H 2 ) MLP O(p 2 n 2 H 2 ) GKIM (12)where H is the number of hidden neurons in MLP and p denotes the number of parameters of GNN.In the above theorem, we take the common voting classifier as an example for non-parametric methods. It averages the category representations stored in MRaM and updates them through backpropagation. While this method has lower complexity, it is highly sensitive to the initialization of category representations due to its reliance solely on the average while computing the prediction probabilities. When using MLP to update category representations, its parameters require fine-tuning to accommodate new categories that will lead to the forgetting of prior knowledge. In contrast, GKIM exhibits a higher VC-dimension compared to the other two methods, indicating superior expressive power. Additionally, GKIM selectively updates category representations in MRaM locally, preserving the model's memory of prior knowledge. 
experiments: 
 In this section, we will evaluate Mecoin through experiments and address the following research questions: Q1). Does Mecoin have advantages in the scenarios of graph few-shot continual learning? Q2). How does MeCs improve the representativeness of class prototypes? Q3). What are the advantages of GKIM over other distillation methods?4.1 Graph Few-Shot Continual Learning (Q1)Experimental setting. We assess Mecoin's performance on three real-world graph datasets: Cora-Full, CS, and Computers, comprising two citation networks and one product network. Datasets are split into a Base set for GNN pretraining and a Novel set for incremental learning. Tab.1 provides the statistics and partitions of the datasets. In our experiments, we freeze the pretrained GNN parameters and utilize them as encoders for subsequent few-shot class-incremental learning on graphs. For CoraFull, the Novel set is divided into 10 sessions, each with two classes, using a 2-way 5-shot GFSCIL setup. CS's Novel set is split into 10 sessions, each with one class, adopting a 1-way 5-shot GFSCIL configuration. Computers' Novel set is segmented into 5 sessions, each with one class, also using a 1-way 5-shot GFSCIL setup. During the training process, the training samples for session 0 are randomly selected from each class of the pre-trained testing set, with 5 samples chosen as the training set and the remaining testing samples used as the test set for training. It is worth noting that for each session, during the testing phase, we construct a new test set using all the testing samples of seen classes to evaluate the model's memory of all prior knowledge after training on the new graph data. Our GNN and GAT models feature two hidden layers. The GNN has a consistent dimension of 128, while GAT varies with 64 for CoraFull and 16 for CS and Computers. Training parameters are set at 2000 epochs and a learning rate of 0.005. Baselines. 1) Elastic Weight Consolidation (EWC) [17] -imposes a quadratic penalty on weights to preserve performance on prior tasks.2) Learning without Forgetting (LwF) [18] retains previous knowledge by minimizing the discrepancy between old and new model outputs. 3) Topology-aware Weight Preserving (TWP) [7] -identifies and regularizes parameters critical for graph topology to maintain task performance. 4) Gradient Episodic Memory (GEM) [19] -employs an episodic memory to adjust gradients during learning, preventing loss increase from prior tasks. 5) Experience Replay GNN (ER-GNN) [3] -incorporates memory replay into GNN by storing key nodes from prior tasks. 6) Memory Aware Synapses (MAS) [20] evaluates parameter importance through sensi- tivity to predictions, distinct from regularization-based EWC and TWP. 7) HAG-Meta [10] ,a GFS-CIL approach, preserves model memory of prior knowledge via task-level attention and node class prototypes. 8) Geometer [11] -adjusts attention-based prototypes based on geometric criteria, addressing catastrophic forgetting and imbalanced labeling through knowledge distillation and biased sampling in GFSCIL.Experimental results. We execute 10 tests for each method across three datasets, varying the random seed, and detailed the mean test accuracy in Tables 2, 3 , and 4. Key findings are as follows:1) Mecoin surpasses other benchmarks on CoraFull, CS, and Computers datasets, exemplified by a 66.72% improvement over the best baseline on the Computers dataset across all sessions. 2) Mecoin maintains a performance edge and exhibits reduced forgetting rates, substantiating its efficacy in mitigating catastrophic forgetting. 3) Geometer and HAG-Meta perform poorly in our task setting, likely because, to ensure a fair comparison, the models do not use any training data from the metalearning or pre-training processes during training. The computation of class prototypes in these two methods heavily relies on the graph structure information provided by this data. Additionally, in Fig. 2 , we compare the results of Geometer and HAG-Meta tested under the experimental conditions given in their papers with the results of Mecoin on the CoraFull dataset. The experimental results indicate that our method still achieves better performance and forgetting rates than these two methods under low memory conditions, demonstrating the efficiency of our method in terms of memory.We conduct relevant ablation experiments on MeCs across the CoraFull, CS, and Computers datasets:1) Comparing the impact of MeCs usage on model performance and memory retention;2) Analyzing the effect of feature of node interaction with class prototypes stored in SMU on model performance and memory retention;3)Assessing the impact of using graphInfo on model performance and memory retention. We conduct a randomized selection of ten classes from the test set, extracting 100 nodes per class to form clusters with their corresponding class prototypes within the SMU. Subsequently, we assess the efficacy of MeCs and its constituent elements on model performance and the rate of forgetting. In cases where the MeCs was not utilized, we employ the k-means method to compute class prototypes. In experiment, we use GCN as backbone. The experimental parameters and configurations adhered to the standards established in prior studies.Experimental Results. The results of the ablation experiments on CoraFull and CS are shown in Fig. 3 and results on other datasets are shown in Appendix.A. We deduce several key findings from the figure:1) The class prototypes learned by MeCs assist the model in learning more representative prototype representations, demonstrating the effectiveness of the MeCs method.2) The difference in accuracy between scenarios where graphInfo is not used and where there is no interaction with class prototypes is negligible. However, the scenario without graphInfo exhibits a higher forgetting rate, indicating that local graph structural information provides greater assistance to model memory. This may be because local structural information reduces the intra-class distance of nodes, thereby helping the model learn more discriminative prototype representations. In our experimental investigation across three distinct datasets, we scrutinized the influence of varying interaction modalities between MRaM and GNN models on model performance and the propensity for forgetting. We delineate three distinct scenarios for analysis: 1) Class representations stored in MRaM are classified using non-parametric methods and used as the teacher model to interact with the GNN. 2) Class representations stored in MRaM are classified using MLP and Mecoin is employed as the teacher model to transfer knowledge to the GNN model. 3) GNN models are deployed in isolation for classification tasks, without any interaction with Mecoin. In this experiment, we use GCN as backbone. Throughout the experimental process, model parameters were held constant, and the experimental configurations were aligned with those of preceding studies. Due to space constraints, we include the results on the Computers dataset in the Appendix.A.Experimental Results. It is worth noting that due to the one-to-one matching between MRaM and SMU via non-parametric indexing, there is no gradient back-propagation between these two components. This implies that updates to MRaM during training do not affect the matching between node features and class prototypes. Our experimental findings are depicted in Fig. 4 , and they yield several key insights: 1) GKIM outperforms all other interaction methods, thereby substantiating its efficacy. 2) The second interaction mode exhibits superior performance compared to the first and third methods. This is attributed to the MLP's higher VC-dimension compared to the non-parametric method, which gives it greater expressive power to handle more complex sample representations. However, the use of MLP results in a higher forgetting rate compared to the non-parametric method. This is because when encountering samples from new classes, the parameters of MLP undergo changes, leading to the loss of prior knowledge. For the third method, extensive parameter finetuning leads to significant forgetting of prior knowledge. Method one performs less effectively than GKIM, primarily because GKIM, with its larger VC-dimension, ensures that the process of updating class representations does not affect representations of other classes stored in MRaM.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We report all settings and details of reproducible experiments in Section 4 and Appendix C. 
conclusion: 
 Current GFSCIL methods typically require a large number of labeled samples and cache extensive past task data to maintain memory of prior knowledge. Alternatively, they may fine-tune model parameters at the expense of sacrificing the model's adaptability to current tasks. To address these challenges, we propose the Mecoin for building and interacting with memory. Mecoin is made up of two main parts: the Structured Memory Unit (SMU), which learns and keeps class prototypes, and the Memory Representation Adaptive Module (MRaM), which helps the GNN preserve prior knowledge. To leverage the graph structure information for learning representative class prototypes, SMU leverages MeCs to integrate past graph structural information with interactions between samples and the class prototypes stored in SMU. Additionally, Mecoin introduces the MRaM, which separates the learning of class prototypes and category representations to avoid excessive parameter fine-tuning during prototype updates, thereby preventing the loss of prior knowledge. Furthermore, MRaM injects knowledge stored in Mecoin into the GNN model through GKIM, preventing knowledge forgetting. We demonstrate our framework's superiority in graph few-shot continual learning with respect to both generalization error and VC dimension, and we empirically show its advantages in accuracy and forgetting rate compared to other graph continual learning methods.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will make our code public after the paper is accepted. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our approach does not address social impacts and this question is not applicable to our approach. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
acknowledgement: 
 This work is supported by the National Science and Technology Major Project (2023ZD0121403). We extend our gratitude to the anonymous reviewers for their insightful feedback, which has greatlycontributed to the improvement of this paper. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets used in the paper are properly credited and the license and terms of use explicitly are mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.  
 
 Appendix: In Section 4.2, we presented the relevant ablation experiments on the CoraFull and CS dataset. Below are the experimental results of the model on the Computers datasets. Figure 5 : From left to right are the results of GKIM, without using GraphInfo, node features not interacting with class prototypes in SMU and without using MeCs , when performing the graph small-sample continuous learning task in the Computers dataset, four randomly selected categories from session1 and 400 randomly selected samples from the four categories are clustered at the class center of the class prototypes bit class centers obtained from the learning during the training process.The experimental results in this section are similar to those on the CoraFull and CS dataset, and relevant experimental analyses can be referred to in the main text. Additionally, we designed related ablation experiments on the dimensions of GraphInfo and its integration position with node features. Detailed results are shown in tab.5, 6. From these results, we can draw the following conclusions:1)It is evident from the tables that concatenating GraphInfo to the left of node features with a dimension of 1 yields the best performance; 2)The model's performance decreases as the dimension of GraphInfo increases, while the forgetting rate generally exhibits a decreasing trend. This indicates that local graph structural information provides some assistance to the model's memory. When the dimension of class prototypes is fixed, an increase in the dimension of GraphInfo implies less graph structural information extracted from prototypes of seen classes in past sessions. Consequently, MeCs integrates less graph structural information, making it difficult for the model to learn representative class prototypes, thereby limiting the model's performance. Furthermore, for Section 4.3, the relevant ablation experiment results on the Computers dataset can be found in Figure 8 . M with I y K , m i with K i , and k α with k β • z α , our proof is similar to the proof of theorem 3.1 in [12] (please refer to [12] for more details). For f ∈ {f M θ , f }, through similar procedure in the proof of [12] , we obtain the coarse upper bound of generalization error R:EQUATIONWe then further conduct the tighter generalization bound for f and f M θ separately based on Eq.13. Consider the case of f = f , for the second term of right hand side of equation ( 13), according to Lemma 4 of (Pham et al., 2021) and the fact that y T ∈Y T m∈I y T M |I y T m | = N , we obtain that for any δ > 0, with probability at leastEQUATIONwhere p T is the value that is uniquely corresponding to the class prototype m = k α (x T ), and h θ represents the knowledge interchange process in GKIM. □proof: When we use a simple non-parametric decoder function which uses average pooling to calculate the element-wise average of all the fetched memory representation and then applies a softmax function to the output, the decoder is the same as a single hidden layer nets with fixed input weights. This network can be formulated as followsEQUATIONwhere A i is the i-th row of weight matrix, b i is the bias, c i is the output layer weights. For average pooling, c i and b i is 0, and A i is 1 n . So is easy to prove that the VC-dimension is n + 1. Then, according to [21] , the VC-dimension of MLP with one hidden layer is n 2 H 2 . Then we consider the case when graph structure information is induced. When we use the distillation technique to inject graph structure information into GKIM via graph neural networks, the upper bound on the capacity of memory representation in GKIM is the expressive capacity of the graph neural networks. Thus, according to [22] , we have that the VC-dimensinon of GKIM with graph structure information is p 2 n 2 H 2 . In summary, when graph structure information is introduced, the upper bound of the VC dimension of GKIM is increased and its expressive power is also enhanced.The relevant experimental parameters in this paper were determined through grid search. The parameter settings for each dataset are shown in Table 7 . The environment in which we run experiments is:• CPU information:24 vCPU AMD EPYC 7642 48-Core Processor • GPU information:RTX A6000(48GB)While our method has shown promising performance in graph few-shot incremental learning, there are still some issues that need to be addressed: 1) Our approach assumes that the graph structures across all sessions originate from the same graph. However, in real-world scenarios, data from different graphs may be encountered, and we have not thoroughly explored this issue; 2) Although our method currently maintains model performance with only a small number of samples, further validation is needed to assess whether it can still achieve excellent performance under low-resource conditions where graph structural information is scarce.Justification: Theoretical results are detailed in Section3.3 and Appendix B.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper addresses the problem of graph few-shot class-incremental learning and proposes a novel framework named Mecoin to tackle the challenges of catastrophic forgetting and overfitting. Mecoin comprises two key components: the Structured Memory Unit (SMU) for storing and updating prototypes, and the Memory Representation Adaptive Module (MRaM) for separating the learning of prototypes and class representations. The SMU helps maintain representative prototypes, effectively preventing overfitting in few-shot settings. Meanwhile, MRaM mitigates catastrophic forgetting by avoiding extensive fine-tuning of model parameters in class-incremental learning.

During the rebuttal period, the authors satisfactorily addressed the reviewers' concerns, particularly those regarding the methodology presentation and experimental setup. The problem tackled in the paper is important, and the proposed solution is novel. Therefore, we recommend acceptance of this paper. However, the authors are strongly encouraged to (1) significantly improve the writing, especially in the method section, to enhance clarity and (2) include the additional empirical results provided during the rebuttal period to strengthen the paper’s contributions.

Reviews: review: paper_summary: paper_summary: This paper mainly focuses on graph few-shot class-incremental learning. To alleviate the significant memory consumption and catastrophic forgetting of old knowledge, it proposes to store only old class prototypes and update class prototypes by considering the interaction among nodes and prototypes. Furthermore, it gives a theoretical analysis on the generalization error. Experimental results show that the proposed method can achieve better results.
strength_weakness: strength_weakness: In general, this paper needs further polish, from my point of view. Just name a few, the abbreviations MRaM and MRAM are not consistent; the adjacency matrix A is incorrect; some sentences need careful attention. In addition, the motivation or the reason why the proposed method uses self-attention and MRAM is not clearly explained.
questions: questions: 1.What are the meta-learning samples? There is no definition or explanation about them. It would be much better if more explanation would be given. 
2.Why using Gaussian random projection to reduce the dimensionality, instead of others?
3.Why concatenating G_T and H_T^p? Could the authors give more explanation?
4.What is the difference between class prototype and class representation? It is very hard to tell from the paper.
limitations: limitations: The authors have discussed the limitations in the paper.
paper_summary: paper_summary: The authors focus on graph few-shot class-incremental learning. The authors first introduce Mecoin to efficiently construct and preserve memory. To avoid extensive parameter finetuning and forgetting, the authors introduce a memory representation adaptive module called MRaM to separate the learning of prototypes and class representations. Besides, the authors propose Graph Knowledge Interchange Module (GKIM) to injects past knowledge information into GNN. Additional analyses illustrate the effectiveness of the methods.
strength_weakness: strength_weakness: 1. The paper does not report error bars, standard deviations or provide detailed information about the statistical significance of the experiments, which is important for understanding the reliability and variability of the results.
questions: questions: See Weaknesses.
limitations: limitations: The paper has a section discussing several limitations in the appendix.
paper_summary: paper_summary: This paper tackles the problem of graph few-shot class-incremental learning, proposes a novel framework named Mecoin to address the challenges of catastrophic forgetting and overfitting. Mecoin includes two key componets: the structured memory unit (SMU) for storing and updating prototypes and the memory representation adaptive module (MRaM) for separating the learning of prototypes and class representations. Based on SMU, Mecoin efficiently maintain representative prototypes preventing the overfitting problem caused by few-shot setting, and rely on MRaM, Mecoin avoids heavily finetuning the model parameters which solved the catastrophic forgetting problem in class-incremental learning.
strength_weakness: strength_weakness: 1.The abbreviations of the proposed method are confusing. For example, in some places, the Memory Representation Adaptive Module is abbreviated as MRaM, while in others, it is MRAM. Additionally, the full model name, Mecoin, is too similar to one of its submodules, the Memory Construction module (MeCo).
2.Fig. 1 is confusing; it doesn’t show the details and advantages of the SMU and MRaM structures.
3.In Sec. 3.1, Eq. 6 lacks explanatory information. How does Eq. 6 integrate edge information into the loss function?
4.The proof of Theorem 1 is too brief. I recommend the authors provide detailed derivation rather than citing a lemma from another paper.
5.In Sec. 3.2, Eqs. 8 and 9 are incomprehensible. I assume the GKIM maintains past knowledge and updates new knowledge with different elements 𝑝. For example, if the dataset has 100 classes, 
𝑝1:60 would be maintained for past knowledge and 𝑝61:100 for new knowledge. Thus, 𝑁 should be 60. It’s challenging to explain how new knowledge is updated in Eq. 9.
6.In Sec. 4.1, the process of pretraining the GNN needs to be explained. Is the pretraining phase a standard graph node classification task or a few-shot node classification task? Additionally, in Tab. 3, the comparison method MAG-Meta is mentioned twice; I guess one should be Geometer. Also, if the proposed method is named Mecoin, why does Mecoin appear among the comparison methods in Tabs. 2, 3, and 4?
7.In Sec. 4.2, the toy dataset is too simple. Sampling only 4 classes makes it difficult to demonstrate the ability of different settings to handle overlapping problems. I recommend sampling at least 10 classes.
questions: questions: Please check the weaknesses
limitations: limitations: The paper is poor-writing and many places lack detailed explanations.

Comments: title: The rebuttal has addressed part of concerns.
Thanks for the Response
Request for Detailed Accuracy Metrics to Evaluate Class Incremental Learning Performance
comment: Dear Reviewer:

We are honored by your thorough and detailed review. Your valuable suggestions have greatly enhanced our reflection on the paper and significantly improved the quality and depth of our research.

Your insightful feedback has been instrumental in further elevating the paper's quality. Additionally, your recommendation to report accuracy metrics in a more detailed manner will aid us in better analyzing the plasticity-stability dilemma in continual learning.

We particularly appreciate the professionalism and constructive feedback you have demonstrated during the rebuttal phase. Your suggestions have allowed us to improve the paper and ultimately receive your endorsement. This is both an encouragement and an inspiration to us, and we believe these improvements will make our work more appealing.

Thank you again for your support and assistance.

Sincerely.
Dear Reviewer:

We greatly appreciate your detailed review and professional feedback on our paper. We especially thank you for pointing out the issue regarding the omission of error reporting in the experimental section. Based on your valuable comments, we have made the necessary improvements, updated the experimental report, and ensured the accuracy and completeness of the results.

We are pleased to receive your positive evaluation, which is crucial for advancing this research direction. Your guidance has helped us enhance the quality of our paper and better meet academic standards.

Thank you again for your support and assistance.

Sincerely.
Dear Reviewer:

We greatly appreciate your valuable feedback and suggestions during the review process. Your professional insights and thoughtful evaluation are highly significant to us. We are pleased to see your positive assessment of our paper. Based on your comments, we have made the following revisions:

- **Refinement of Paper Expression**: We have re-examined our paper and made further modifications to the logic, sentence structure, and word choice.
- **Standardization of Symbols and Abbreviations**: We carefully reviewed the symbols and abbreviations used in the paper and have standardized and unified them.
- **Improved Method Description**: We have provided a more detailed introduction of the proposed method, Mecoin, clarifying the roles of each component and their interrelationships, and included more detailed explanations of the objectives and operations in the method section.

We believe these changes will further enhance the quality of our paper. Thank you again for your support and assistance.

Sincerely.
After seeing the response from the authors, part of my concerns has been addressed. Since the rest of the reviewers liked the paper and also considering my low confidence, I tend to raise my rating to borderline accept. But I do encourage the authors to better polish their paper, especially the Method section, to make it easier to follow.
The authors response resolves my concerns, I decide to keep my rating.
Thank you for further acknowledging and supporting our paper, as well as for your approval of our previous response. Your feedback is extremely valuable to us.  
We agree that the plasticity-stability dilemma remains a challenge to overcome, and we will continue to explore and address this issue in future research. Regarding your suggestion to further refine the manuscript, we will carefully consider and implement it. We plan to review the paper’s expression and logic, particularly in the methodology section, to ensure clarity and readability. We will improve the paper with the following measures:  
- **Optimize Paper Expression**: We will modify the descriptions of paragraphs and sections according to the paper’s logic, such as word choice and sentence structure, while appropriately adding relevant background introductions to make the expressions more complete, clear, and easy to understand.  
- **Improve Method Description**: We will enhance the introduction of the proposed method, Mecoin, to clarify the roles of each component and their interrelationships; refine the explanation of technical terms, and carefully check the standardized use of symbols. We will also articulate the theory and its conclusions more clearly to help readers better understand our method and contributions.  

Thank you once again for your invaluable suggestions. We believe that with these adjustments, our paper will better highlight its scientific contributions and be more accessible to a broader audience.
I appreciate your considerate response to my latest concern regarding the detailed accuracy metrics for class incremental learning. The additional data and insights you have provided have sufficiently addressed my queries, and I am now convinced of the paper’s contributions to the field. From the result I can see the challenge of plasticity-stability dilemma still exists and needs to be overcome.

As we move forward, I would like to suggest a final refinement to enhance the readability and coherence of the manuscript, especially, the method section. I kindly ask that you carefully rephrase the text to ensure that the paper is as clear and easy to understand as possible. This will not only benefit the peer-review process but also make the paper more accessible to a broader audience, including practitioners and researchers who may not be as deeply familiar with the subject matter.

Your attention to detail in revising the manuscript will help to clarify the methodologies and findings, making the paper’s strengths even more apparent. I believe that with these adjustments, the paper will be well-positioned for acceptance and will have a significant impact on the community.
Thank you very much for your thorough review of our paper and for acknowledging our response. Your valuable feedback aids in a more comprehensive and clearer evaluation of our model. Since our experimental setup is consistent with previous papers [1][2], we did not report the specific accuracy for new and old classes in each session in detail. We fully agree that the evaluation method is crucial for a comprehensive understanding of our model's incremental learning capability. Based on your suggestion, we will add class accuracy metrics for each session in the results section.  
We apologize that due to word limits, we cannot present all experimental results (three tables, with a total of 153 columns and 19 rows each). In incremental learning tasks, as the number of sessions increases, the model’s ability to remember previous knowledge typically declines. Therefore, the newer sessions are better for evaluating the model's retention of past knowledge. Consequently, we have chosen to showcase the results of the last session for each dataset. The results are as follows:

|methods|backbone|dataset||||||CoraFull||||||
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
| \ | \ | \ ||||||session10||||||
| \ | \ | \ |session0|session1|session2|session3|session4|session5|session6|session7|session8|session9|session10|
|ergnn|GCN|CoraFull|16.47|9.45|3.65|7.72|5.29|8.85|3.57|4.93|6.44|6.34|4.47|
||GAT|CoraFull|15.07|8.08|2.17|9.16|6.33|9.03|1.13|5.99|7.883|7.11|4.93|
|lwf|GCN|CoraFull|9.15|6.45|4.86|5.58|6.24|6.39|9.22|9.81|2.88|7.81|1.49|
||GAT|CoraFull|9.15|6.45|4.86|5.58|6.24|6.39|9.22|9.81|2.88|7.81|1.49|
|gem|GCN|CoraFull|9.23|7.11|8.93|5.87|6.39|6.65|9.48|9.95|3.92|7.57|2.95|
||GAT|CoraFull|8.52|7.17|8.61|3.58|6.25|6.78|8.89|9.76|2.96|6.12|2.38|
|EWC|GCN|CoraFull|8.48|7.87|8.39|4.02|7.08|7.25|9.01|9.52|2.28|6.71|1.94|
||GAT|CoraFull|14.92|7.31|7.47|9.22|6.76|8.32|8.92|4.09|7.97|6.92|4.28|
|MAS|GCN|CoraFull|15.75|8.38|9.47|9.52|6.27|8.37|9.17|4.48|7.63|7.74|4.6|
||GAT|CoraFull|45.96|40.11|41.69|22.09|19.25|17.76|17.21|16.38|10.41|11.51|12.34|
|TWP|GCN|CoraFull|14.12|10.43|7.72|5.28|8.27|8.65|9.79|5.79|2.45|6.82|5.83|
||GAT|CoraFull|12.35|9.89|7.54|5.61|7.46|8.45|8.47|5.44|2.31|5.71|5.92|
|Geometer| \ |CoraFull|9.9|9.79|8.84|6.01|8.87|8.98|8.52|5.12|3.33|4.13|4.58|
|HAG-Meta| \ |CoraFull|60.74|50.18|45.16|32.86|21.43|24.42|16.95|24.21|11.45|20.68|13.28|
|ours|GCN|CoraFull|76.52|36|58.23|24.18|21.43|33.51|7.63|23.12|12.25|32.23|14.09|
||GAT|CoraFull|76.52|44|58.35|32.009|36.54|22.68|34.73|19.89|5.5|36.74|20.88|

|methods|backbone|dataset||||||CS||||||
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
| \ | \ | \ ||||||session10||||||
| \ | \ | \ |session0|session1|session2|session3|session4|session5|session6|session7|session8|session9|session10|
|ergnn|GCN|CS|0|0|0|0|0|0|0|0|0|24.39|100|
||GAT|CS|0|0|0|14.83|6.28|2.48|5.48|2.81|21.78|33.34|100|
|lwf|GCN|CS|0|0|0|0|0|0|0|0|12.36|11.13|100|
||GAT|CS|0|0|0|17.76|13.6|7.02|5.34|6.15|21.61|31.87|100|
|gem|GCN|CS|0|0|0|0|0|0|0|0|0|15.59|100|
||GAT|CS|0|0|0|0|0|7.55|0|0|0|15.35|100|
|EWC|GCN|CS|0|0|0|0|0|0|0|0|0|17.12|100|
||GAT|CS|0|0|0|0|16.24|18.04|17.84|10.97|20.95|32.12|100|
|MAS|GCN|CS|0|0|0|0|0|0|0|0|0|12.89|98.23|
||GAT|CS|0|0|41.43|41.17|35.06|52.37|48.78|31.29|25.53|38.02|100|
|TWP|GCN|CS|0|0|0|0|0|0|0|0|0|26.97|100|
||GAT|CS|0|8.52|40.34|38.16|30.12|49.97|44.12|20.88|18.23|24.8|94.56|
|Geometer| \ |CS|2.85|13.45|8.28|10.69|8.74|8.09|3.75|3.79|17.92|31.63|86.57|
|HAG-Meta| \ |CS|9.89|8.91|7.31|2.54|5.69|3.73|7.53|6.41|4.87|5.98|4.65|
|ours|GCN|CS|90.26|44.2|40.91|22.6|58.17|81.48|59.37|42.08|34.92|35.95|51.68|
||GAT|CS|91.51|47.28|43.1|23.84|61.43|80.11|63.79|45.33|37.47|39.74|54.92|

|methods|backbone|dataset|||Computers||||
|-|-|-|-|-|-|-|-|-|
| \ | \ | \ |||session5||||
| \ | \ | \ |session0|session1|session2|session3|session4|session5|
|ergnn|GCN|Computers|0|0|0|0|0|100|
||GAT|Computers|0|0|0|0|0|100|
|lwf|GCN|Computers|0|0|0|0|0|100|
||GAT|Computers|0|0|0|0|2.87|100|
|gem|GCN|Computers|0|0|0|0|0|100|
||GAT|Computers|0|0|0|0|0|100|
|EWC|GCN|Computers|0|0|0|0|0|100|
||GAT|Computers|0|0|0|0|0|100|
|MAS|GCN|Computers|0|0|0|0|0|100|
||GAT|Computers|0|0|0|12.86|8.98|100|
|TWP|GCN|Computers|0|0|0|0|0|100|
||GAT|Computers|0|0|0|0|0|100|
|Geometer| \ |Computers|0|0|0|0|0|87.55|
|HAG-Meta| \ |Computers|0|0|0|0|0|80.69|
|ours|GCN|Computers|71.37|75.15|48.88|77.69|68.24|79.36|
||GAT|Computers|82.42|73.26|42.7|77.69|71.24|60.98|

From the experimental results, it is evident that our method outperforms the baseline methods in both the older and the new classes, highlighting the advantages of our approach. If you would like to see results from any other session, we can provide them in a follow-up response. We will update the full experimental results in the appendix of the revised paper.  
[1] Graph few-shot class-incremental learning.  
[2] Geometer:Graph few-shot class-incremental learning via prototype representation.
Dear Authors,

Thank you for your thorough rebuttal, which has successfully addressed most of my concerns. However, I would like to raise an additional point regarding the evaluation of the few-shot class incremental learning approach presented in your paper.

A well-recognized challenge in the field of class incremental learning is the model’s ability to maintain high accuracy for the initial session classes while often struggling with the classes introduced in subsequent sessions. The metrics you have provided give an overall picture of the model’s performance but do not sufficiently highlight this specific issue.

To better assess the model’s capability in this regard, I kindly ask that you report the accuracy metrics in a more granular fashion. Specifically, I would like to see the accuracy for the classes of each session reported separately. For instance, at session 2, it would be beneficial to have the following:

The accuracy for the classes introduced in sessions 0 and 1.
The accuracy for the classes introduced in session 2.

This division will allow us to more clearly evaluate the model’s performance on both the older and the new classes, providing a more comprehensive understanding of its incremental learning capabilities.

Thank you for considering this request, and I look forward to your response.