Title: Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: continual learning

knowledge graph
Abstract: Continual learning in computer vision faces the critical challenge of catastrophic forgetting, where models struggle to retain prior knowledge while adapting to new tasks.
Although recent studies have attempted to leverage the generalization capabilities of pre-trained models to mitigate overfitting on current tasks, models still tend to forget details of previously learned categories as tasks progress, leading to misclassification. To address these limitations, we introduce a novel Knowledge Graph Enhanced Generative Multi-modal model (KG-GMM) that builds an evolving knowledge graph throughout the learning process. Our approach utilizes relationships within the knowledge graph to augment the class labels and assigns different relations to similar categories to enhance model differentiation. During testing, we propose a Knowledge Graph Augmented Inference method that locates specific categories by analyzing relationships within the generated text, thereby reducing the loss of detailed information about old classes when learning new knowledge and alleviating forgetting. Experiments demonstrate that our method effectively leverages relational information to help the model correct mispredictions, achieving state-of-the-art results in both conventional CIL and few-shot CIL settings, confirming the efficacy of knowledge graphs at preserving knowledge in the continual learning scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes KG-GMM, which leverages a Knowledge Graph to enrich relational information for class incremental learning. It also introduces knowledge graph-augmented inference to preserve detailed knowledge and reduce forgetting. Experiments demonstrate its effectiveness in both class incremental and few-shot class incremental learning.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n1. The paper is well-written, with clear and visually appealing figures. \n\n1. The authors provide a thorough background on Knowledge Graphs and Generative Multi-modal models for CIL, making the paper easy to follow.\n     \n## **Weaknesses**\n1. It is uncertain whether the proposed method effectively addresses the two challenges of leveraging LLMs in CIL, as mentioned in the introduction.\n\n1. The use of CIFAR-100 and subsets of ImageNet-21K is too limited to validate the general effectiveness of the proposed approach.\n\n1. The method section lacks explanation on how each component contributes to class incremental learning, particularly for learning new classes or retaining old ones. As a result, the method appears to be merely a combination of well-pretrained models and a knowledge graph built from large datasets.\n\nQUESTIONS:\nQ1. While leveraging a knowledge graph to enrich relational information may help address the challenge of classifying at a higher semantic level, it remains unclear whether the proposed method adequately addresses the issue of losing generalization ability, because the proposed method still trains only simple linear layers.    \n\nQ2. As the knowledge graph is ImageNet-based, there is a concern of potential information leakage in the experiments conducted on ImageNet subsets (Tiny-ImageNet, ImageNet-R, Mini-ImageNet), since the model may have prior knowledge of both seen and unseen classes. Similarly, CIFAR-100 shares some class labels with ImageNet, raising the same concerns. It would strengthen the evaluation if the method were tested on fine-grained datasets, such as FGVC-Aircraft or Herbarium 19.\n\nQ3. Which model is used for the \"Zero-shot\" performance at the experiments (Table 1 and Table 2)? Additionally, in Table 2, why is the performance drop observed in the zero-shot evaluation, which assesses the performance without training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of class-incremental learning (CIL) by leveraging generative multimodal models (GMMs). To overcome the limitations of traditional GMMs, the paper integrates a knowledge graph into the generative framework. The incorporation of the knowledge graph enhances the retention of knowledge from old tasks, providing significant benefits in both the training and testing phases. Experimental results on standard CIL and few-shot CIL scenarios demonstrate substantial improvements over existing state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The idea of integrating a knowledge graph into class-incremental learning (CIL) is both meaningful and novel. Current methods suffer from forgetting due to limited attention to knowledge relationships. This paper offers a simple yet effective solution by enhancing high-level graph knowledge with images.\n\n2. The paper is well-structured, clearly presenting the concepts of Knowledge Graph Enhanced Learning and Knowledge Graph Augmented Inference. It effectively demonstrates the novel techniques applied in both the training and testing phases.\n\n3. The performance across multiple benchmarks underscores the effectiveness of the approach, achieving performance gains of approximately 2-4%.\n\n4. The ablation study, along with the time complexity analysis, highlights a favorable trade-off between accuracy and computational cost.\n\nWeaknesses:\n\n1. The motivation and explanations in some sections could be further clarified and strengthened (see Questions 1-3 for specific examples).\n\n2. A discussion on how the proposed methods can be extended to other relevant scenarios as GMM models continue to develop would significantly enhance the paper's impact.\n\n3. Including some failure cases would provide a more comprehensive evaluation of the method and give readers a better understanding of its limitations.\n\nQUESTIONS:\n1. Could you clarify the main motivation behind the \"Trim to Key Relationships\" step? Is it feasible to use all possible relationships, or are there specific constraints guiding this choice?\n\n2. Why does the \"GMM + Description Labels\" approach not perform as well, whereas GMM with a knowledge graph yields better results?\n\n3. The \"Inference Time\" line in Fig. 5 is somewhat confusing. Could you provide further clarification on what this represents?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo mitigate catastrophic forgetting in continual learning, this paper constructs an evolving knowledge graph to leverage relational information between classes, using KG-based labels during training and a KG-augmented strategy to refine predictions at inference. Experiments on benchmark datasets demonstrate strong performance in both conventional and few-shot continual learning settings, outperforming state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- By incorporating knowledge graphs, the model captures semantic relationships between classes, enabling more discriminative feature learning for similar categories in incremental tasks. \n\n- The KG-augmented inference strategy uses relational queries to guide model predictions, reducing misclassifications by leveraging prior knowledge. This effectively addresses the limitation of generative models drifting to higher-level categories. \n\n- Experiments on diverse datasets and CIL settings (both conventional and few-shot CIL) validate the effectiveness of the proposed method. \n\nWeaknesses:\n\n- The KG trimming strategy (e.g., selecting top-two relations or two-hop connections) introduces subjectivity, potentially omitting discriminative relations critical for fine-grained class distinctions. Moreover, the approach relies heavily on ConceptNet, which may lack task-specific semantic relations in specialized domains (e.g., medical imaging or robotic vision), limiting generalizability to niche applications.\n- While the study demonstrates KG-GMM’s superiority over GMM in exemplar-free settings, it does not evaluate whether the method maintains this advantage when competing with GMM augmented by data replay. Whether the KG’s benefits can surpass those of data replay, which can be validated by additional experiments.\n\nQUESTIONS:\nThis paper introduces multi-modal models for continual learning, and existing multi-modal model based continual learning studies are suggested to be mentioned. \n\n[1] Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models. \n\n[2] MLLM-CL: Continual Learning for Multimodal Large Language Models.\n\n[3] HiDe-LLaVA: Hierarchical decoupling for continual instruction tuning of multimodal large language model.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces KG-GMM, a novel knowledge graph-enhanced generative multi-modal framework designed to alleviate catastrophic forgetting in class-incremental learning (CIL). By incrementally constructing a task-aware commonsense knowledge graph, the model uses structured relational information to enrich label representation and guide relation-aware reasoning during inference. Extensive experiments on conventional and few-shot CIL benchmarks show that KG-GMM consistently outperforms previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The idea of combining generative multi-modal LLMs with a dynamically growing knowledge graph for CIL is innovative. It addresses a critical limitation of prior generative CIL methods: the tendency to produce generic or biased predictions due to loss of class-specific detail. \n\n2. The paper introduces a well-structured knowledge graph construction mechanism that selects informative relations per class and guides both learning and inference.\n\nWeaknesses: \n\n1. The approach heavily relies on ConceptNet, a commonsense KG. The proposed method is only applicable to datasets with category labels, like Domainnet.\n\n2. The paper could benefit from a deeper qualitative analysis of failure cases. Some examples are shown in Fig. 4, but more insights into when the method fails to disambiguate similar classes (e.g., due to misleading relations) would be valuable.\n\nQUESTIONS:\n1. Consider exploring the impact of the choice of relation types (e.g., only IsA, AtLocation) on performance.\n\n2. It is necessary to emphasize how relations of second hop supplement information when relation of one hop is not enough.\n\n3. Similar classes in different tasks may influence the results. Thus, it would be helpful to run experiments on different seeds on TinyImageNet and ImageNet-R.\n\nLIMITATIONS:\nThe paper doesn’t include a discussion about the limitations. The suggestions are provided in Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}