Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning
Authors: Xusheng Cao
Haori Lu
Linlan Huang
Fei Yang
Xialei Liu
Ming-Ming Cheng
Venue: NeurIPS 2025 poster ()
Keywords: continual learning
knowledge graph
Primary area: deep_learning

Abstract: Continual learning in computer vision faces the critical challenge of catastrophic forgetting, where models struggle to retain prior knowledge while adapting to new tasks.
Although recent studies have attempted to leverage the generalization capabilities of pre-trained models to mitigate overfitting on current tasks, models still tend to forget details of previously learned categories as tasks progress, leading to misclassification. To address these limitations, we introduce a novel Knowledge Graph Enhanced Generative Multi-modal model (KG-GMM) that builds an evolving knowledge graph throughout the learning process. Our approach utilizes relationships within the knowledge graph to augment the class labels and assigns different relations to similar categories to enhance model differentiation. During testing, we propose a Knowledge Graph Augmented Inference method that locates specific categories by analyzing relationships within the generated text, thereby reducing the loss of detailed information about old classes when learning new knowledge and alleviating forgetting. Experiments demonstrate that our method effectively leverages relational information to help the model correct mispredictions, achieving state-of-the-art results in both conventional CIL and few-shot CIL settings, confirming the efficacy of knowledge graphs at preserving knowledge in the continual learning scenarios.

Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning
Xusheng Cao", Haori Lu², Linlan Huang", Fei Yang 1.2g Xialei Liu Ming-Ming Cheng!
NKIARI, Shenzhen Futian, VCIP, CS, Nankai University
[caoxusheng, luhaori, huanglinlan @ moaail.nanaai.ddun [feiyang, xialei, cmm] @ mankai.edu.cr
Abstract
Continual learning in computer vision faces the critical challenge of catastrophic forgetting, where models struggle retain prior knowledge while adapting new tasks. Although recent studies have attempted to leverage the generalization capabilities of pre-trained models mitigate overfitting on current tasks, models still tend to forget details of previously learned categories as tasks progress, leading misclassification. To address these limitations, we introduce novel Knowl- edge Graph Enhanced Generative Multi- motal model (KG-GMM) that builds an evolving knowledge graph throughout the learning process. Our approach utilizes relationships within the knowledge graph to augment the class labels and assigns different relations to similar categories enhance model differentiation. During testing, we propose a Knowledge Graph Augmented Inference method that locates specific categories by analyzing relationships within the generated text, thereby reducing the loss of detailed information about old classes when learning new knowledge and alleviating forgetting. Experiments demonstrate that our method effectively leverages relational information to help the model correct mispredic- tions, achieving state-of tfte-offthhe-at results in both conventional CIL and few-shot CIL settings, confirming the efficacy of knowledge graphs at preserving knowledge in the continual learning scenarios.
1 Introduction
Continual learning without forgetting old knowledge [34] has been thriving in the ever-changing world. Traditional approaches in this area typically employ three categories [33] of methods to prevent forgetting: replay-based methods 25 40 57 that retain a portion past data, architecture-bant
methods [32 42 58 that progressively expand the model, and regularization-based methods [7]
that prevent excessive changes in model parameters. Recently, methods based on large-scale pre- trained models [2] 39) have attracted more attention due to their superior performance than the traditional train maii-from--se www--fooo-sratcc methods. For example, prompt-based methods [44] (66) [55]
[4] have
been proposed that use small number of parameters to learn "general" and "specific" knowledge without altering the pre- pre-trained backbone. SLCA [62 proposes to use a small learning rate for the backbone to preserve the generalizability and larger learning rate for the classifier to accommodate new classes. Yet, these methods do not leverage textual information of the class labels or the relationships between the learned classes.
To leverage the rich information lies in text modality, Continual CLIP proposes to use Frozen CLIP to conduct predictions based on image-text similarities. RAPF [8 uses CL.PP (39 text encoder to detect and pull apart similar classes. PROOF (67] proposes to use tasc-fcc projections on both image and text encoder. MoE-Adapters [59] utilize the Mixture-of-Experts adapters on top of the
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

pre-trained CLIP model and an Auto-Selector to route for different input. CLIP-CIL 29 proposes to fine-tune an additional adapter after the backbone to learn new classes. However, in the more challenging exemplar-free scenario, these methods still face problems with classification bias towards the newly learned knowledge and forgetting of previously learned knowledge.
In contrast, GMM [5] addresses the bias prob-
Describe the detail (colur, species, location, texture, etc) of this photo.
lem by discarding the classifier and directly us-
What is this photo?
ing a Large Language Model (LLM) to generate predicted text. GMM employs image-text pairs as input and only fine-tunes a linear layer, effec- tively mitigating bias and leveraging the LLM's capability to understand and generate human- interpretable text. However, two challenges per- sist during the learning process.
Frouco LLM
Frouco LLM
Granny Smith
Apple Ted Fruit
This apple
Lemon Pincapple
Apple husrolor Green Apple Atlocation
Text Embeddinn
Knowledge Gruph
Similarity Prediction
Augmented Inference
Pinesppll
Granny Smith
Firstly, fine-tuning a small linear layer with single, fixed format could cause the Multi-modal LLM to lose its generalization capabilities, caus- ing it to output text only in one format "This is photo of [CLS]" while losing the ability to describe details about the image content in terms of colors, background, texture, etc. Sec- ondly, since the model lacks exposure to images from previous tasks, it tends to classify all images encountered into higher-level categories (knowledge obtained in the pre-training phase). For example, if the model learns "Granny Smith" in the initial task, it can accurately output "Granny Smith" for images of that class during immediate testing. However, after learning other similar categories like "Pineapple" or "Lemon" in subsequent tasks without replaying exemplars, the model tends to respond, "This a photo an apple" when presented with green apple images during testing and misclassifying it to newly learned text related label "Pineapple" as shown in Fig.
GMM.
(b) KG-GMM.
Figure The difference of inference pipeline between the generative-based baseline GMM [ and our Knowledge Graph enhanced generative multi-modal model (KG-GMM).
To address these challenges, we turn to common sense knowledge graphs [19, which are structured collections of factual triples, organized as (head, relation, object), e.g., (Granny Smith, IsA, Fruit) that capture relationships between entities. Compared to the black-box generation process of L.LMs (36.. knowledge graphs encode rich interrelationships information in the form of human-readal For example, ConceptNet 45 is a general common sense knowledge graph integrating triplets that cover nearly all visual recognition datasets (including both coarse-graines and fine-grained ones) with efficient query operations. We believe that a knowledge graph that incrementally expands as tasks increase would be lightweight and straightforward while retaining the knowledge structures learned by the model to prevent forgetting.
language.
Building on this idea, we propose Knowledge Graph Enhanced Generative Multi modti-modal Model for inasemnncreeetaal learning. It enables the model to focus more on the factual content within the images, providing descriptive output rather than direct guesses. Additionally, during inference, we construct a subgraph from the model's original textual output and compare with the existing graph to identify the specific category to which the image belongs. Specifically, we use common-sense knowledge graph to incrementally build a sub-graph during continual learning by storing class- relevant triplets. Training employs relations (not plain text) as ground truth labels. During testing, associative keywords (e.g., ISA, AtLocation) guide the model to output detailed image facts instead of direct guesses, enhancing precision by recalling prior relational knowledge from the graph. This enables structured retention and retrieval of learned relationships, improving answer specificity.
The main contributions of this paper are:
We propose using an ever-expanding knowledge graph to help the model distinguish similar classes across different tasks based on relationships, providing more references when discriminating similar classes.
During inference, we propose to guide the model with relation words to output factual de- scriptions rather than direct guesses. preventing forgetting through associative relationships.
Experiments on multiple datasets and settings demonstrate that our method can help the model alleviate forgetting with minimal training cost.
2

---

2 Related Work
2.1 Class Incremental Learning
Class incremental learning is method where model learns new classes sequentially over time without forgetting the previously learned classes. Early methods of class-incremental learning often involved training from scratch [24 53
69 [4 60].
With the advent of pre-trained models, using them starting point for continual learning has shown great effectiveness. Initially, models pre-trained on ImageNet are typically fine-tuned using parameter- efficient methods to incorporate new knowledge. such as prompt- techniques [4T [55] [6
44] and
low-rank adapters [3] 26 Some approaches involve fine-tuning specific modules of the model [66]
or setting differential learning rates [62]. Building on this, guidance from textual modality information
further alleviates forgetting. The CLIP model, with its strong zero-shot capabilities, has significantly
benefited class-incremental learning [49] and garnered considerable attention. This is often achieved
by adding parameter modules, such as linear layers [29] [8] or attention modules [20] 67], to the original model to directly adjust the well-learned features.
Currently, generative multi-mode modal models demonstrate powerful capabilities at mitigating the for- getting problem in continual learning. Simple fine-tuning on these models can effectively retain old knowledge while learning new classes (5). However, without replay, the GMM still tends to forget details of the classes learned in former tasks. Our method tackles this issue by incorporating rich relationship information in knowledge graphs to enhance the GMM learning and inference process.
2.2 Knowledge Graph
Knowledge graphs are structured representations of information where entities (nodes) are inter- connected through relationships (edges), allowing for complex querying and inference over linked data. There are four main types of knowledge graphs [3] Encyclopedic KGs that cover general knowledge (Wikidata (5I). Common Sense KGs (ConceptNet [45] ) that capture everyday concepts and objects, Domain-specific KGs tailored to specialized fields (UMLS B for medical domain), and Multi-modal KGs [3]] that integrate various data types such text and images. We mainly focus the Common Sense KG due to its broad scope that covers most classes in image datasets.
With the development of LLM, there has been research work that combines the generation capability of LLM models with the structured, rich factual knowledge stored in knowledge graphs. The combinations mainly fall into three categories: KG-enhanced LLMs (27 38 63) involve embedding KGs to improve LLMs by enhancing understanding and reasoning of the knowledge learned by LLMs. LLM-augmented KGs [6 23 64 leverage LLMs for different KG-based tasks such knowledge graph completion, graph-to-text generation, and question answering. The integration of LLMs and KGs 22 22 47) enables bidirectional reasoning grounded in both data and structured knowledge, as the two systems collaborate symbiotically to mutually enhance each other's 's capabilities.
While there have been studies integrating knowledge graphs with LLMs, most of these focus on single-task scenarios. The potential of combining LLMs and knowledge graphs to enhance knowledge preservation in continual learning scenarios remains unexplored.
3 Method
In this section, we begin by introducing the setup of slass-incremental learning and revisiting the training and testing procedures of GMM (5). Then, we present our method from two perspectives: knowledge graph enhanced learning and knowledge graph augmented inference.
3.1 Preliminaries
Class Incremental Learning (CIL). CIL. is a paradigm that focuses on the progressive acquisition of knowledge across disjoint of classes. Let A' denote the input space and y represent the label
space. At each incremental time step t, the learning algorithm receives new dataset D; {(2,, y:) I; E Xe,w E V.J, where }i is set of novel classes such that 3% n (U/))))) = 0. The objective is
to construct a classifier fo X U Vis that predicts labels overall observed classes up time t.
3

---

Fine-tuning
Inference
Stepl: get original prediction
Knowledge
Graph
+
Apple T&A Frur, Apple RecelvcAction Eaten Apple Allocction Store Apple has Color Green
Step2: Eatily prediction based Graph
Vinual
Grany Smith Piseapple
HauColer
Grees Pizza
Fruil.
[Gmany Smith. Pineapple]
ReceiveAction Eaten.
(Granny Smith, Pineapple]
Beerhhe citatie the ploti coker, spedics location, bockgronn, di:
olcationn Store.
[Graany Smith, Pineapple]
Gimnn Smith
has Color Green.
(Grany Smith]
p 0g
Step3: Retrieval augmented message
*
Prezen
Serids
Step4: Prediction
Cross Entropy Loss
Granyy
Simillarity Prediction
Goldifics
Froxx
Lesenable
Piacapli
Chiandiik
Figure 2: Left: In knowledge graph-enhanced learning, image embedding, knowledge-enhancee ground-truth embedding, and question embedding are input to Frozen LLM to generate predicted embedding; cross-entropp loss updates the linear layer. Right: In knowledge graph-augmente inference, relations from predicted text are extracted to create graph-augmented text, producing the final prediction.
Generative Multi-modal Models for CIL. In order to effectively harness the textual information embedded within the graph, we need to leverage language models capable of extracting text embed- dings that can be aligned with image features. Currently. two continual learning frameworks can incorporate textual features: those based on CLIP 220 29 18 and those utilizing large language
models (LLMs) [ Given our need for more nuanced text understanding and encoding-decoding abilities, we adhere to the continual learning framework outlined in GMM [5]
The GMM method's 'ss training pipeline involves using frozen image encoder fene and text encoder fiett to process the image-text pairs {(X,,, i, ) for each task where N, means the total number of image-text pairs in task and S, is the text labels of classes in task t. For each image Tg. Jenc generates an image embedding e; - Senc (ar;; Oenc), and for each class, GMM utilize the BERT tokenizer to get the corresponding embedding s,. Then, a question embedding q is also computed by the BERT model from the question text 9. The image embedding e, concatenated with the ground-trute embedding s; and question embedding is input to a frozen LLM to get the predicted tokens sequentially based on prior tokens:
Th
P(S ..., Sm F4 q.s)
II P(syle,
(1)
Then GMM computes Cross Entropy Loss to make the predicted token close to the GT token:
LCE Σ sj log
(2)
where sj the ground truth and šy is the prediction.
At inference time, GMM uses the fine-tuned model to get the text prediction based on the test image and instruction question. Then text encoder freu is used to measure the cosine similarity between predicted labels and ground truth to determine the final classification as:
pred arg max( (s), (S))
As discussed in the introduction, GMM may misclassify predicted text to its word-related class label instead of its meaning-related ground truth (e.g., apple pineapple instead of apple Granny Smith). So we propose to use relations in common sense graph to instruct the model output relation relation-aware aware text, thus helping locate the predicted text to its ground truth. We begin by introducing the Knowledge graph construction process, followed by how to utilize this graph for training and inference.
3.2 Knowledge Graph Enhanced Learning
Knowledge Graph Construction. In order to extract relations for each encountered new category, we follow 7SL-KG (55) to query three tables: nodes, relations, and edges from the ConceptNet. database based on the ILSVRC-2IK [10] classes within two-hop relations.

---

Then, we have a common sense knowledge graph at hand presented as G (E, where E represents entities, R represents relations and F ((),r,)) E E represents facts contained in this graph, in which h means head entity and means tail object. In the continual learning process, we gradually build dataset-specifie graph in the order in which classes appear at each given task.
To better illustrate our method, we divide our graph construction process into two steps:
Step 1: Build a temporary knowledge graph. Suppose we are currently in task t of continual learning. We initialize
= E felee T(S.),
(4)
where St denotes new class nodes, and ((,)) extracts non-class nodes from relation triplets involving S,. Then we build temporary graph for task t:
G =
(5)
containing all entities, relations, and fact triplets observed up to task t.
Step 2: Trim to Key Relationships. To enable efficient training and improved dis- crimination between similar classes, we proposed to trim the temporary knowledge graph G2 to a smaller one G²;=
Australia
Related
Fruit
Granny Smith
Actión
Receive Action
Pincapple
Hascbllr
AdLecation
AfLaration
AlLacation
G² = (E', Fi,,F)
(6)
that only contains few unique relations that best distinguish it from similar old classes. Here, EY is the trimmed entity set:
Green
Store
Fish
Goldrish
AsA
Clowafish
E E ci=felee E,, e
U 1=1
E
(7)
Reledec
Accoration
Animal
Allocattoon
Gold
Water
River
R2 and Fi are relations and facts that are associated with the current entity set Gy,- For example, as illustrated choose "AtLocation Store" and "AtLoca- tion Pizza" instead of previously used "IsA Fruit" and "ReceiveAction Eaten" by class "Granny Smith" " from task 1. The whole construction process of G₁ is detailed in Algo. Lines 4-6.
Node Task
Relation of Task,-1
Node
Relation Task,
Fig- [ we
Other Node
Relaticn
Figure 3: The knowledge graph construction process. Rectangle nodes represent class nodes, while round ones represent non-class nodes. Blue rectangles represent classes encountered in task - 1, while the orange ones represent classes from task t. Blue Arrows represent relations used for learning task 1, while the orange ones represent relations used for task
There exist circumstances where classes from later tasks (maybe the last task) have no direct relation to use; that is, all directly related tail nodes have been taken by for- mer encountered classes. We tackle this problem by using relations of the second hop in the common sense knowledge graph (c.g., we use "Clownfish ReatedTo Water RelatedTo River" instead of the previously occupied node "Water").
After obtaining the distinct graph that has different relations for each class, we then concatenate m triplets of each class together and input them into the model as the ground truth text in the image-text pairs as shown in the left side of Fig. fine-tuning the linear layer after the image encoder with Cross Entropy loss in Eq.
3.3 Knowledge Graph Augmented Inference
The inference process is illustrated on the right side of Fig. 2 by the fine-tuned L.LM with instructing questions "Describe details of this photo from color, species, location, background, etc.". Then we decompose the raw output into triplets: {(hp. Tps 0p) p
We first get the relation-aware output
5

---

Algorithm/ 1 Graph Construction For Task /
Input: St
Class names of task t
require: G = {E, R,
Entire Knowledge Graph
require: Et-1
Entity node for task t
require: Gz-1
> Subgraph for task 1
1: E, = L Ufelee T(S:))
> Init Eg b Init subgraph for task > Iterate each new classes D Find unused facts
2: G. = [E, R4 Fi-1]
3: for entity in St do
4: for (h, r, 0) in F do
5: if h -- and (h, r, $ F then
6:
G = [E,, R2 Ur, P. (h,r,))
7:
return G2
>> Return the subgraph for task t
After obtaining the predicted relations, we remove the head entities hp and use the relation pairs (Tpp0p)) }p)) to search within the subgraph G. constructed for the current task. The head entity that appears most frequently in the search results is considered the predicted head entity:
h = arg max
Σ [(h, E Gol.
(8)
1=1
where T[|] the indicator function that equals if the triplet is correct ( (h, Op) E F2) and otherwise.
Although this graph-based prediction h can be treated the final prediction, it neglects the rich information contained in the original raw text output s. To leverage both the knowledge reservation ability provided by the graph and the text understanding and reasoning ability provided by LLM, we prepend this graph-based prediction l s to obtain the graph-augmented output:
where @ denotes the concatenation operation, meaning that 8a is formed by placing h directly before This augmented sentence is then input into the text encoder to perform similarity prediction between the encoded 8, and all class features encountered so far to obtain the final prediction:
pred = arg max sim (Sren((Sa) stene())) cES.
(10)
Experiments
4.1 Experiments Setup
Datasets. We test our model on two commonly used continual learning benchmarks: Tiny-ImageNN and ImageNet-R, R, and two few-shot continual learning benchmarks: CIFARI00 and Mini- Mini-ImageNet. We also conduct experiments on fine-grad one-grained and medical imaging datasets, and the corresponding settings and detailed results are provided in the supplementary material.
For conventional continual learning, we follow the two standard configurations used in GMM 5 BO, in which all classes are equally divided among different tasks, and B 100 (i.c. Tiny-ImageNet) in which the first task contains 100 classes (half of the dataset) and the rest are equally divided into subsequent tasks. For few-shot continual learning, we follow the data splits proposed by [48]. For both datasets, we divide the data into two parts: a base session and incremental sessions. The base session consists of 60 classes with full access to all associated data. Each incremental session follows a 5-way 5-shot setting, introducing 5 new classes with only samples per class.
We build our expanding knowledge graph based on ConceptNet, which is a large-scale, multilingual knowledge graph that represents common-sense relationships between words and phrases in natural language. It consists of over 8 million nodes and approximately 21 million edges, connecting concepts through 50 relationships including "IsA", "PartOf". "UsedFor" and "HasProperty". etc.
Implementation details. For the Knowledge Graph part, we follow 7SI.KKG 35 to use a 2-hop ImageNet-based knowledge graph extracted from ConceptNet (45) as the initial G described in

---

Table 1: The performance comparison results between method and other methods on Tiny- ImageNet and ImageNet-R.
Tiny-Image Mny-ImageNet
ImageNet-R
Type
Method
Exemplar
B100-3
tasks
B100- 10
tasks
B100-20
tasks
B0-10 tasks
Avg
Avg
Last
Avg
EWC
19.01
6.00
15.82
3.79
12.35
4.73
35,00
LwF 25
X
22.31
7.34
17.34
4.73
12.48
4.26
38.50
iCaRL
45.95
34.60
43.22
33.22
37.85
27.54
Conventional
BBLL
47.17
35.12
45.03
34.64
40,41
29.72
UCIR
50.30
39.42
48.58
37.29
42.84
30.85
PASS 69
X
49.54
41.64
47.19
39.27
42.01
32.93
DyTux 0
55.58
47.23
52.26
42.79
46,18
36.21
Continual-CLI -L
X
70.49
70.55
66.43
70.51
66.43
72.00
L2P 56
83.53
78.32
76.37
65.78
68.04
52.40
72.92
Discriminative
DualPrompt
551
85.15
81.01
81.38
73.73
73.45
60.16
68.82
PT models
CODA-Prompt
X
85.91
82.80
75.28
66.32
73.88
MoB-CLIP 59
81.12
76.81
80.23
76.35
79.96
75.77
80.87
RAPF 17
78.64
74.67
77.42
73.57
76.29
72.65
80.28
Lincar Probe
X
74.38
65.40
69.73
58.31
49.72
45.17
Zero-shot
58.16
53.72
58.10
53.72
58.13
$3.72
67.38
Generative PT models
GMM 5
X
83,42
76.98
82.49
76.51
81.70
76.03
80.72
KG-GMM (Ours)
X
86.17
81.86
84.37
78.16
83.18
78.32
section 3.2 G contains 574270 nodes E, 50 relations R and 1380131 edges F. For the Generative
Multi-modal model part, we follow GMM to use the MiniGPT-4 (8 framework the image and text encoder. the BO setting of all datasets, we employ a 200-iteration warmup with a learning rate of 3e-6 and learning rate from 3e-5 to 3e-6 with cosine decay scheduler in the following fine-tuning phase. In the B100 setting, we first employ a learning rate of 3e-6, and then on the subsequent tasks, adopt lower learning rate of 3e-7, both employing cosine decay scheduler.
Baselines and evaluating metrics. We follow GMM to compare with three different categories
of methods, including conventional train-from-scrandd based 40 discriminative pre-trained based [56
15
65
58
25 69
55
4 and generative pre-trained based [5].
The evaluation
metrics for the experiments are defined as follows: "Avg" represents the model's average accuracy across all tasks, while "Last" denotes the model's accuracy on all test sets after fine-tuning the final task. For few-shot continual learning, we also add a new metric Harmonic Accuracy (37] (HAcc) to check the balanced performance between the base and new classes. A# 2xAnXAA where Ao is Aot.An
the acc of base classes and A, is the average of all classes. We report an average accuracy of three runs based on three different class orders. Please refer to the Supplementary Material for detailed results of different orders.
4.2 Experiments on Conventional CIL
In Tab. [. we present experiments on 100 base classes with 5, 10. and 20 incremental tasks settings on Imag-lmageNet, and B0- 10 tasks setting on ImageNet-R. Our KG-GMM demonstrates superior performance across all settings of these two datasets. On Imne-ImaeeNet, our method consistently outperforms others in all settings, particularly we outperform GMM by 2.93% on average in three settings in terms of I Last accuracy. On ImageNet-R, our model achieved an impressive last task performance of 84. 29%, surpassing the previous SOTA GMM by 3.57%, the previous best prompt- based method CODA-Prompt by 10.41%.
4.3 Experiments on Few-shot CIL
In Tab. we present the comparison results of our method against other baselines on Mini-Imagetie for few-shot class incremental learning. The table shows that our model lags behind several ImageNet- 21K pre-trained methods in the base session, achieves state-of-the-an results all subsequent eight sessions. Specifically, in the final session, our method reaches an accuracy of 78.07%, outperforming DualPrompt (55) by 21.26% points, CODA-Prompt 44 by 16.93% points, and the previous best GMM [5] by 2.89% points. Notably, our model leads GMM by 1.64% points in the first incremental
7

---

Table 2: Comparison results of our method with other conventional baselines and methods the Imaieem and CIFAR100 for few-shot class incremental learning. The table includes one base task and eight incremental tasks. PD is the performance drop between the first and last session. indicates our re-implementation based on PILOT [46].
Mini ImageNet
CIFARI00
Method
4
8
PDJ
HAcef
4
8
PDJ
HAcef
iCaRL
61.31
30.49
17.21
44.10
32.45
64.10
27.93
13.73
50.37
41.45
EEIL
61.31
33.14
19.58
41.73
28.43
64.10
28.96
15.85
48.25
32.43
LUCIR
61.31
25.68
14.17
47.14
35.65
64.10
31.61
13.54
50.56
39.67
TOPIC
61.31
37.48
24.42
36.89
32.98
64.10
40.11
29.37
34.73
25.23
CEC
72.00
56.70
47.63
24.37
15.96
73.07
58.09
49.14
23.93
22.46
F2M
72.05
56.71
47.84
24.21
19.21
71.45
57.76
49.35
22.06
19.37
MetaFSCIL
72.04
57.58
49.19
22.85
14.35
74.50
59.48
49.97
24.53
3.60
Entropy-reg
71.84
57.01
48.21
23.63
19.29
74.40
59.71
50.14
24.26
11.53
L2P"
94.12
70.94
56.83
37.29
0.00
91.22
68.66
54.89
36.33
0.00
DualPrompt"
93.97
70.61
56.80
37.17
0.10
91.08
68.45
54.67
36.41
0.10
CODA-Prompt"
95.37
74.47
61.14
34.23
0.00
93.55
71.91
59.32
34.23
0.00
Zero-shot
58.08
58.19
54.95
3.13
52.47
74.13
72.59
67.93
6,20
64.75
GMM
89.35
83.61
75.18
14.17
71.45
91.53
85.65
81.47
10.06
75.43
KG-GMM (Ours)
90.99
85.63
78.07
12.93
74.81
91.83
86.23
82.37
9.46
79.68
session and extends this lead to 2.89 points in the final session, indicating that our approach more effectively mitigates forgetting through the integration LLM and knowledge graphs
Tab. also presents the comparison results of our method against other baselines on CIFARI00 for few-shot class incremental learning. The results indicate that our method also achieves superior performance on low-resolution datasets, outperforming the previous state-of of ofe---f-thhhhee the-art method GMM by 0.90% points,the traditional
wor--ffoo-ssarre method Entropy-reg (28) by 32. 32.23% points, and the state-of-tha of-the-art prompt-based method CODA- Prrompt 44 by 23.05% points the final session. Note that although our method shows only slight improvement over GMM in the last session, it achieves a 4.25% gain Harmonic Accuracy. This indicates that our approach better balances learning new classes while retaining previously learned ones.
Table 3: Ablation study results of the description labels and the Graph- Augmented Inference on ImageNet-R B0-10 tasks.
Table 4: Time complexity analysis on the last task of B0-10 tasks setting of the ImageNet-R dataset.
Method
Accuracy
Relation Stored
r=
r=2
== 3 (Ours)
r=t
GMM
80.72
Avg text leugth (c)
30
51
75
102
GMM+Descriptions Labels
80.95
Generation (8)
6.54
12.63
KG-GMM w/o Graph-Augmented Inference
82.77
Graph Inference (ms)
Storage (MI)
0
0.05
0.07
0.08
KG-GMM
84,29
Accuracy (%)
81.03
81.92
84,29
84,31
4.4 Further Analysis
Ablation study on different components. In Tab. 3 present ablation studies to validate the effectiveness our method. We first use the GPT-3.5 generated descriptions (using the same prompt question
method) for each class as the text in the image-text pair of the original GMM. The results show that without the explicit guidance of the relation keywords, the rich information brought by the text has almost no improvement over the simple text labels used in the original GMM. Besides, show that using only the head entity h provided by the search results in KG (third row in Tab. 3] shows considerable improvement (82.77% 77%), but combined with the text output, our proposed KG-GMM achieves the best results 84.29%.
Ablation on Max tokens. In Fig. we present the comparative analysis of maximum token limits on model performance and inference time. The experiments were conducted on ImageNet-R, comparing

---

Image
Ground-Truth label
GMM predicted text
GMM prediction
KG-GMM predicted text
KG-GMM Prediction
This is a photo apple.
Apple IsA Fruit
Granny Smith
Pincapple
Apple HasColor Green Apple ocation Store
Granny Smith
This photo of shark.
Shark RelatedTo Head RelatedTo Hammer Stark All ocation Sea
Hammerhead
Great White Shark
Hammerhend
Goldfinch
This photo of bird.
Humminghird
Bird RelaledTo Gold
Bird Acctionn Rackyard
Goldfinch
Bird IsA Finch
This photo fox.
Fox IsA animal
Fox Squireel
Red Fox
Fox HasConiext movies Fox RelatedTo Red
Red Fox
Horse RelatedToo barn HorseSynonym
Afghun Hound
This photo of horse.
Zebru
Barn
Horse AtLocation farm yard
Figure Text examples of our methods against the original GMM.
our KG-GMM method with the baseline approach GMM+Descriptions that utilizes class descriptions generated by GPT-3.5 for data augmentation. Inference time was measured as the averaged processing duration per batch (size=64), Notably, our method achieves near-optimal performance at 20 tokens, while GMM+Descriptions exhibits slower performance growth with increasing tokens. This indicates that the knowledge graph- graph-augmented GMM produces more informative outputs at lower token constraints, demonstrating two key advantages: 1) Enhanced information efficiency through KG- enhanced training enables effective knowledge condensation, and 2) Superior performance attained with reduced computational overhead.
Time complexity analysis. In Tab. 4 we present the additional storage and inference costs associated with selecting different numbers of relations r per class with a total batch size of 64. For
Accuracy vs. Inference Time
85.0
O, we used GMM's default prompt, "This a photo of [CLS]", while for 2 and r = 3, the average generated text lengths correspond to 51 and 75 characters, respec- tively. "Generation cost" represents the time required to generate text, measured in seconds, "Storage" de- notes the additional memory usage from the knowl- edge graph, measured in MB, and "Graph inference" indicates the time needed to obtain the graph-based prediction h, measured in milliseconds.
82.5
80.0
77.5
Inference Time
75.0
72.5
70.0
67.5
KC GMN (Acouracy Gelie seso piees (ccurry)
65.0
irfevence Time
1
100
As shown in Tab. the storage and inference over- head of the knowledge graph is minimal (0.07MB and 0.53ms for = 3). The primary cost arises during the inference phase (generating longer texts increases inference time with higher values of r). Balancing inference time with performance gains, we select 3 our final hyperparameter. We also conduct experiments analysing memory overhead and training time between our method and the original GMM. Experimental results demonstrate that compared to the original GMM, our method requires maximum memory footprint of 169 KB, while utilizing this additional memory allocation to preserve exemplars yields no corresponding performance improvement. We refer the readers to the Section A.3 for detailed experiments.
Figure 5: Model Accuracy vs. Inference Time comparison regarding max tokens con- figured during inference.
Illustration of corrected predictions. In Fig. we present some visual examples our proposed KG-GMM in terms of the predicted text and final prediction in the BO-10 tasks setting of the ImageNet-R dataset. We can see that the GMM can still recognize shark but lose the ability to classify it to the right subclass (hammerhead). Instead, our KG-GMM can extract rich information from the relation contained the predicted text and locate to the right class. We refer the readers to the supplementary materials for more visualization results including failure cases.

---

Conclusions
In this paper, propose KG-GMM. a novel method combine MLLMs and knowledge graphs to tackle the catastrophic forgetting problem in continual learning. Our method gradually builds graph the process of learning new classes, assigning each class with r different relations to enhance discriminability between semantic similar classes. During inference, we use the generated relations to locate the specific class, combined with the predicted text, our KG-GMM can effectively preserve much of the LLM's generalization ability while providing more accurate category predictions for given test images. Extensive experiments show that our method outperforms the state-of-the-are baselines for exemplar- free class incremental learning.
10

---

Acknowledgments
This work is funded by NSFC (NO. 62206135, 62225604), Young Elite Scientists Sponsor- ship Program by CAST (2023QNRC001) the Fundamental Research Funds for the Central Universities (Nankai Universitiy, 070-63233085), Shenzhen Science and Technology Program (JCYJ202408131 11100244081111330440 14237048), "Science and Technology Yongjiang 2035" key technology break- through plan project (20252053), Chinese government- guided local science and technology develop- ment fund projects (scientific and technological achievement transfer and transformation projects) (254Z0102G). Computation is supported by the Supercomputing Center of Nankai University.
References
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[2] Nourhan Bayasi, Jamil Fayyad, Alceu Bissoto, Ghassan Hamarnch, and Rafeef Garbi. Biaspruner: Debiased continual learning for medical image classification. In International Conference on Medical Image Computing and Conputer-Asiisedd Intervention, pages 90-101. Springer, 2024.
[3] Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppL -2270, 2004.
[4] Xing Cao and Yun Liu. Relmkg: reasoning with pre-trained language models and knowledge graphs for complex question answering. Applied Intelligence, pages 1-15, 2022.
[5] Xusheng Cao, Haori Lu, Linlan Huang, Xialci Liu, and Ming-Ming Cheng. Generative multi-modal models good class incremental learners. IEEE Computer Vision and Pattern Recognition (CVPR), 2024.
[6] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End and incremental learning. In Proceedings of the European conference on computer vision (ECCV), pages 233 243,248,777 2018.
[7] Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), 2018
[8] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafseil: A meta-learning approach for few-shot class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14166-14175, 2022.
[9] Riccardo de Lutio, John Y Park. Kimberly A Watson, Stefano D' Aronco. Jan D Wegner, Jan J Wieringa, Melissa Tulig, Richard L Pyle, Timothy Gallaher. Gillian Brown, al. The herbarium 2021 half-earth challenge datasct and machine lcarning competition. Frontiers Plant Science, 12:787127, 2022.
[10] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[11] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[12] Chao Feng, Xinyu Zhang, and Zichu Fei. Knowledge solver: Teaching Ilms to search for domain knowledge from knowledge graphs. arXiv preprint arXiv:2309.03111 2023.
[13] Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal- free continual learning. In ECCV, 2024.
[14] Alex Gomez-Villa, Dipam Goswami, Kai Wang, Bagdanov Andrew, Bartlomicj Twardowski, and Joost van de Weijer. Exemplar-free continual representation learning via learnable drift compensation. In European Conference on Compuder Vision, 2024.
[15] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In CVPR, 2019.

---

[16] Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang. Ziyi Yang. Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun. Empowering language models with knowledge graph reasoning for open-domain question answering, In EMNLP, pages 9562-9581, 2022.
[17] Linlan Huang, Xusheng Cao, Haori Lu, and Xialei Liu. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. In European Conference on Computer Vision, pages 214-231. Springer, 2024.
[18] Linlan Huang, Xusheng Cao, Haori Lu, and Xialei Liu. Class dass-incrrmmnnal learning with clip: Adaptive representation adjustment and parameter fusion. In ECCV, 2024.
[19] Filip llievski, Pedro Szekely, and Bin Zhang. Cskg: The commonsense knowledge graph. In The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6-10, 2021, Proceedings 18, pages 680-696. Springer, 2021.
[20] Saurav Jha, Dong Gong, and Lina Yao. CLAP4CLII: Continual learning with probabilistic finetuning for vision-language models. In Thirty-eighth Conference on Neural Information Processing Systems, 2024.
[21] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[22] Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. Structgpt: A general framework for large language model to reason over structured data, arXiv preprint 07X6v233SS.09655 2023.
[23] Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. ICLR 2023, 2023.
[24] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351 3526, 2017.
[25] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analyxis and machine intelligence, 40 (0(12):2935-299 2017.
[26] Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23638- 23647, 2024.
[27] Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. KagNet: Knowledge-awee aware graph networks for commonsense reasoning. In EMNLP-IJCNLP, pages 2829-2839, 2019.
[28] Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class- incremental learning via antropy-regularieen data-free replay. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[29] Xialei Liu, Xusheng Cao, Haori Lu, Jia wen Xiao, Andrew D. Bagdanov, and Ming-Ming Cheng. Class incremental learning with pre-trained vision-language models, 2023.
[30] Liu, Hui Li, Alberto Garcia-Duran Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. Mmkg: multi-modal knowledge graphs. In The Semantic Web: 16th International Conference, ESWC 2019, Portorot, Slovenia, June 2-6, 2019, Proceedings 16, pages 459-474. Springer, 2019.
[31] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- trneigraieed visual classification of aircraft. arXiv preprini 1rXiv:1306.515.1 2013.
[32] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, 2018.
[33] Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Plass-incremente learning: survey and performance evaluation on image classification. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513 5533, 2022.
[34] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, pages 109-165. Elsevier, 1989.
[35] N. V. Nayak and S. H. Bach. Zero-shot learning with common sense knowledge graphs. Transactions on Machine Learning Research (TMLR), 2022.
12

---

[36] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering. 2024.
[37] Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C. Lovell. Few-shot class-incremental learning from an open-set perspective. In European Conference on Computer Vision, pages 382-397. Springer, 2022.
[38] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In EMNLP-IJCNLP, pages 2463-2473, 2019.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMIR, 2021.
[40] Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
[41] Anurag Roy, Riddhiman Moulick, Vinay Verma, Saptarshi Ghosh, and Abir Das. Convolutional prompting meets language models for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[42] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In ICML, 2018.
[43] Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima. Advances neural information processing systems, 34:6747-677 6761, 2021.
[44] James Seale Smith, Leonid Karlinsky. Vyshnavi Gutta, Paola Cascante- Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909-11919, 2023.
[45] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, 2017.
[46] Hai-Long Sun, Da-Wei Zhou, Han-Jia Yc, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXi::2309.07117 17, 2023.
[47] Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.0763: 2023.
[48] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12183-12192, 2020.
[49] Vishal Thengane, Salman Khan, Munawar Hayat, and Fahad Khan. Clip model is an efficient continual learner. arXiv:2210.03114, 2022.
[50] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[51] Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85, 2014.
[52] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[53] Fu-Yun Yun Wang, Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. arxiv preprint XXXi:2204.04666 2022.
[54] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang. Hang Su, and Jun Zhu. Hierarchical decom- position of prompt- based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 2023.
13

---

[55] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi. Ruoxi Sun. Han Zhang, Chen-Yo Yu Lee, Xiaoqi Ren. Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. ECCV, 2022.
[56] Zifeng Wang. Zizhao Zhang. Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Learning prompt for continual learning. CVPR, 2022.
[57] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 382, 2019.
[58] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning, In CVPR, 2021.
[59] Jiazuo Yu. Yunzhi Zhuge, Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of lision-aagguae models via mixture-of-experss adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219-2322 232300, 2024.
[60] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng. Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6982 6991, 2020.
[61] Chi Zhang, Nan Song. Guosheng Lin. Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12455-12464, 2021.
[62] Gengwei Zhang, Liyuan Wang. Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[63] Mengmei Zhang, Mingwei Sun, Peng Wang. Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang, and Chuan Shi. Graphtranslator: Aligning graph model to large language model for open-ended tasks. In Proceedings of the ACM on Web Conference 2024, pages 1003-1014, 2024.
[64] Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning, and Jure Leskovec. Greaselm: Graph reasoning enhanced language models. In ICLR, 2022.
[65] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Taa Xia. Maintaining discrimination and faimess in class incremental learning. In CVPR, 2020.
[66] Da- Wei Zhou, Win Cai, Han-Jia Yc, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incrementie learning with praitrained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 2024.
[67] Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[68] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[69] Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng- Liu. Prototype augmentation and self- supervision for incremental learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5871-5880, 2021.
14

---

A Appendix supplemental material
A.1 Different class order
Since the constructed sub-graph is influenced by the order in which classes are encountered, different class orders result in different sub-graphs. To evaluate this effect, conducted multiple experiments with varying class sequences, and the results are presented in Tab. 3lnd Tab, 6 As shown, while our model's performance fluctuates due to the order variations, consistently maintains high level of accuracy, demonstrating the robustness of our method.
Table 5: Comparison results between our method and other methods on Tiny-Im Inny-Im and ImageNet- R under different class orders.
Time-ImzgeNet
ImapeNee-R BU Iiisss
Type
Method
Exemphr
B100:3 lacks
B100 10
usks
B300220
aass
Avg
Last
Arg
Last
Avg
Last
Last
EWC
23
19.01
6.00
15.82
3.79
12:35
4.73
35,00
X
12.48
Conventional
SCaRL
+1
45.95
34.60
43.22
33.22
37.85
27.54
47.17
29.72
UCIR 15
50.30
39,42
48.58
37.29
42.54
30.85
X
Dylos 11
55.58
47.23
52.26
42.79
46.18
36.21
Cominunl-C111
X
1.29 56
83.53
7X.32
76.37
65.78
68.04
$2.40
72.92
Diseriminative modcs
DraiProngt
85.15
$1.01
81,38
73.73
73:45
60.16
68.82
CODA-Prupp
85.01
77.43
MoECLIP 9
81.12
76.81
80.23
76.35
79.96
75.77
80.87
78.44
Linear Pribe
74.38
65.40
69.93
58.31
60.14
49.72
45.07
GMM S
$3.42
76.98
82.49
76.51
$1.90
76.00
80.72
nodals
KG-GIM (Ouns) oxde22
$6.35
$2.00
$4.45
78647
83.43
7M.66
84.62
KG
86.27
83.13
78.36
KG GMM (Ouns)
50.17 0.21
41.86
0.70
$4.37
0.33
78.16
0.55
83.18 ct 01:19
78.32 0.29
84.29 0.27
Table 6: Comparison results of our method with other conventional baselines and methods on the Mmai-le dataset for few-shot class incremental learning under different class orders. The table includes one base task and eight incremental tasks. PD is the performance drop between the first and last session. indicates our re-implementation based on PILOT 46
8
PD1
HAust
37.63
mm 2
61.31
46.5 54
44.00
37.28
33.14
27.12
36.10
21.57
19.58
41.75
78.03
21:35
TOPIC 44
61.31
$0.09
45.17
41.16
37.48
35.52
32.19
28.05
24.42
76.59
32
F25
43
72.05
63.47
63.16
59.70
56.71
53,77
51.11
40.21
47.84
24.21
19:23
$2.90
Batneyy mg 10
71.8
07.13
03.21
99.77
3701
53555
31.55
19.53
14.21
23.63
19.29
122
94.12
87.20
$3.99
75.67
70.94
65.76
63.11
90
96.31
17.29
75.11
65.44
CODA Proupi"
44
95.37
83.85
$2.69
77.57
74.47
70.16
66.46
63.73
61.14
34,25
0.0
Zerossie
$7.76
$7.89
55.19
58.26
3,13
GMMM
99.35
88.40
86.11
$5.07
83661
$1.35
75.97
77.34
75.18
14.17
71.45
KG GMM ooder
91.54
98.14
85.13
$7.53
85.42
33.69
$1.45
80.33
76443
13.11
74,57
KG-GMM ang
00:90
89 01 081
88
11:11.1.000
0.00
81.99 0.63
8561 011
3370-4 0.33
81 17.1 0m
7866
0.19
15.07
007
13:93
031
7681
0.0
A.2
Experiments fine-grained and medical datasets
We conducted exploratory experiments on several fine- tne-grained datasets and medical imaging datasets, detailed below:
A.2.1 Datasets and settings
CUB-200 [22 is a fine-grained bird species dataset with detailed part-level attributes. We con- struct knowledge graph using each class's most certain attributes (confidence score 4), such as has bill shape:: or has has_bill_sheee bill_shape:ccoee resulting in approximately 17 relations per class after graph construction method, All 200 classes equally separated into 10 tasks.
15

---

HAM-10000 [50] is medical image dataset of 10,000 pigmented skin lesions of 7 classes for melanoma classification. Since we could find a suitable knowledge graph for medical data, we instead used DeepSeek-01 to generate 10 representative attributes for each of the classes (such as color color_doooe doninonce::lggh brown, background _kkn::No pigmentation), serving as their semantic descriptions or attribute sets. We follow each.
2 to separate 7 classes in 3 tasks, containing [2,2,3] classes
FVGC-Aircraft 3 is dataset comprising 100 different aircraft classes. As no suitable external knowledge graph was available, we utilized the structured metadata provided with the dataset. Specifically, each aircraft class is described using three hierarchical levels: 41 manufacturers, 70 families, and 100 variants. These are represented as three relations hasManufacturer, hasFamily, and has Variant- with each class associated with exactly one relation at each level, resulting in three relations per class.
Herbarium [9] is a dataset containing 46,000 herbarium specimens across more than 680 species within the flowering plant family Melastomataceae. Due the limited rebuttal time, it was difficult process the entire dataset, so we randomly selected 10 species and used DeepSeek-O1 to generate six attribute-based relations: hasLeafShape, hasLealMargin. Vesiea hasFlowersColor, hasStem, and hasFruuit, each with corresponding class-specific properties. The selected classes were evenly divided into tasks, with species per task.
Table 7: Performance comparison across fine-grained and medical datasets.
Datasets
CUB-200
HAM-10000
FGVC-Aireraft
Herbarium19
Avg
Last
Avg
Last
Avg
Last
Avg
Last
GMM
49.34
40.39
79.22
63.81
51.63
47.25
86.34
79.88
KG-GMM
52.65
43.84
81.45
64.34
51.88
47.88
88.59
81.14
The experimental results in Tab. demonstrate that, with sufficiently rich KG information (CUB-200, HAM-10000 our method consistently improves upon GMM. When relevant knowledge is limited (as in the case of FVGC-Aircraft) the performance gains from our method are relatively modest.
A.3 Analysis on training overhead
In Tab. we present more detailed comparisons on memory footprint (additional memory occu- pied by graph-based labels), training overhead (training time per batch), KG time (time used for constructing sub-graph G)), and the final accuracy. The experimental results demonstrate that the graph-based labels incur only marginal overhead compared to word-based labels, requiring at most 169KB of additional memory and 0.07 seconds of extra processing time. Notably, when convert- ing this 169KB capacity into exemplars for memory replay, the performance improvement proved statistically insignificant (from 80.7 50.72 to 80.91). This observation substantiates that our proposed method achieves substantial performance enhancements while maintaining minimal computational and storage overhead.
Table 8: Training overhead analysis on ImageNet-R.
Method
Memory Usage
Training Time
KG Time
Acc
GMM
244K
0.36
0
80.72
KG-GMM (task 1)
332K
0.40
2.46
KG-GMM (task 6)
368K
0.41
2.57
KG-GMM (task 10)
413K
0.43
2.73
84.29
GMM + exemplar
484K
0.36
0
80.91
A.4 More visualization results
In Fig. we provide additional visual examples of our methods against the original GMM. It can be observed that KG-GMM utilizes attributes such as the bird's color and location to refine its
16

---

Image
Ground- Truth label
predicted prediction
KG-GMM prodicted KG-GMM Prediction
This is photo of apple.
Apple
Granny Smith
Pincapple
Apple HazColor
Granny Smith
Apple AtLoocaiinn Store
white shark
This photo a shark.
white shark
Shark Relatedio Great
Animal
Great white shark
Shark ALLocation Ozean
This photo of
Shark Related Head
Hammerhcad
white stark
RelatefTo Hammer
Hammerhccad
AtLocation
Goldfinch
This photo bird.
Humminghird
Bind RelatedTo
Bind AtLocation Dackyard Bind IsA Finch
Goldfinch
Tree frog
photo toad
Newt
Toad RelatedTo Frog
Tead AtLocation Tree
Tree liog
Toad lsA Astplubina
Spider IsA, Arachmid
Tarantula
This photo spider.
Spider web
Spider AtLocation USA
Tarantula
Spider RelatefTo Torantula
Snake IsA ceptile Stake AtLocation 200
This is photo of
Cobra
Iguana
Iguana
Snake Relatedlo python
Hound
a hound.
photo
Afghan Hound
Hound Capuble(if bark
Hound IsA dog
Bengle
Figure 6: More text examples of our methods against the original GMM. The first four examples illustrate how our method corrects the model drift in continual learning. The last two examples with red boxes, showcase where both our method and GMM made incorrect predictions.
classification to the finer-grained category of "goldfinch." In contrast, the original GMM, although retaining knowledge of the broader category, loses the ability distinguish finer details after learning additional concepts. However, for certain categories that belong to the same broader class (e.g.. hound) and have nearly identical characteristics (i.e., similar relations), our method may still make errors, though it typically misclassifies them into closely related categories (e.g.. misclassifying Basset Hound Beagle rather than Afghan Hound based text similarity). We believe that if similar classes were more distinctly separated with different relations or finer-grained graphs with detailed attributes were incorporated, it would further enhance our method's performance.
A.5
Limitations and Border Impact
There are still several limitations in our work. As our work represents the first attemptt to integrate knowledge graphs with continual learning, experiments were primarily conducted on established continual learning benchmarks. We acknowledge that collecting KGs for some specialized datasets (e.g., fine-grained classification datasets and medical datasets) is tricky and remains unexplored. Furthermore, given that our primary baseline comparison focuses on GMM [5), our experiments were limited consistent backbone architectures (EVA-CLIP and Vicuna-7B). Employing more advanced visual-language models could yield performance improvements. With the rapid development of LLMs, continual learning methods will become more important across various applications. Knowledge graphs emerge as valuable yet previously underutilized tool for mitigating generalization loss during continual finettuning of LLMs. Exploring optimal methodologies for KG integration presents promising research direction worthy of in-depth investigation.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction in Section accurately reflect our contributions.
|
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Appendix A.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yee Yes]
Justification: All information detailed in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
All datasets can be obtained on the internet. We will soon open source our code.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https sonideSSumisssiooP.e
//nips. //nips.cc/pil ctt..cccpuubll ctc/public/guudeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide specific details of all experiments in Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide experiment results on three different class orders in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the NeurlPS Code of Ethics, and conduct research with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix A.5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets
cited in section 4
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe the usage of the Vicuna 7B in Section
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
24


Reviews:
[{"review_id": "gCKK6yHnzy", "reviewer": "Reviewer_SjoA", "summary": "This paper introduces KG-GMM, a novel knowledge graph-enhanced generative multi-modal framework designed to alleviate catastrophic forgetting in class-incremental learning (CIL). By incrementally constructing a task-aware commonsense knowledge graph, the model uses structured relational information to enrich label representation and guide relation-aware reasoning during inference. Extensive experiments on conventional and few-shot CIL benchmarks show that KG-GMM consistently outperforms previous methods.", "questions": "1. Consider exploring the impact of the choice of relation types (e.g., only IsA, AtLocation) on performance.\n\n2. It is necessary to emphasize how relations of second hop supplement information when relation of one hop is not enough.\n\n3. Similar classes in different tasks may influence the results. Thus, it would be helpful to run experiments on different seeds on TinyImageNet and ImageNet-R.", "limitations": "The paper doesn’t include a discussion about the limitations. The suggestions are provided in Questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. The idea of combining generative multi-modal LLMs with a dynamically growing knowledge graph for CIL is innovative. It addresses a critical limitation of prior generative CIL methods: the tendency to produce generic or biased predictions due to loss of class-specific detail. \n\n2. The paper introduces a well-structured knowledge graph construction mechanism that selects informative relations per class and guides both learning and inference.\n\nWeaknesses: \n\n1. The approach heavily relies on ConceptNet, a commonsense KG. The proposed method is only applicable to datasets with category labels, like Domainnet.\n\n2. The paper could benefit from a deeper qualitative analysis of failure cases. Some examples are shown in Fig. 4, but more insights into when the method fails to disambiguate similar classes (e.g., due to misleading relations) would be valuable.", "was_revised": true, "final_justification": "The responses address my concerns. Thus the final rating was upgraded.", "rebuttal": "Thank you for your valuable comments. We address your questions or concerns below.\n\n> **W1: Regarding reliance on ConceptNet and category labels.**\n\nWe choose ConceptNet because it integrates multiple structured knowledge sources (e.g., WordNet, DBPedia, OpenCyc) and offers rich, diverse, and readily accessible commonsense knowledge. It covers a wide range of entities and relations relevant to standard continual learning benchmarks, making it a practical and effective choice.\n\nTo address scenarios where ConceptNet may not be applicable, we extend our approach in two ways: (1) For datasets with rich attribute annotations (e.g., CUB-200), we construct custom knowledge graphs based on these attributes. (2) For datasets lacking such annotations (e.g., HAM-10000), we utilize large pretrained models to generate visually grounded concepts and relations to form dataset-specific graphs. We include the corresponding setups and results to demonstrate the flexibility and generalizability of our method beyond ConceptNet.\n\n- CUB-200 is a fine-grained bird species dataset with detailed part-level attributes. We construct a knowledge graph using each class's most certain attributes (confidence score 4), such as has_bill_shape::hooked or has_bill_shape::cone, resulting in approximately 17 relations per class after our graph construction method. All 200 classes are equally separated into 10 tasks.\n- HAM-10000 is a dermatoscopic image dataset of 10,000 pigmented skin lesions of 7 classes for melanoma classification. Since we could not find a suitable knowledge graph for medical data, we instead used DeepSeek-VL-O1 to generate 10 representative attributes for each of the 7 classes (such as color_dominance::light brown, background_skin::No pigmentation), serving as their semantic descriptions or attribute sets. We follow [A] to separate 7 classes in 3 tasks, containing [2,2,3] classes each. \n\n[A] BiasPruner: Debiased Continual Learning for Medical Image Classification, MICCAI 2024\n\n| datasets | CUB-200 |      | HAM-10000 |      |\n|:--------:|:------:|:----:|:--------:|:----:|\n|          |   Avg  | Last |    Avg   | Last |\n|    GMM   | 49.34       | 40.39     |  79.22        |  63.81    |\n|  KG-GMM  |  52.65      |  43.84    |  81.45        |   64.34   |\n\nAs shown in our experimental results, when relevant and visually discernible attributes (e.g., color, shape, pattern) are available, our model yields noticeable improvements over GMM, demonstrating that our approach is adaptable and benefits from contextual knowledge beyond ConceptNet.\n\n> **W2: More qualitative analysis of failure cases.**\n\nWe have analyzed several representative failure cases, which will be added to the revised manuscript. These cases reveal that while the integration of the knowledge graph helps the model attend to factual visual cues (e.g., red, AtLocation garage, RelatedTo barn), it can also introduce semantically misleading associations.\n\nFor example, the model may misclassify fire_engine as pickup_truck due to shared contextual relations (e.g., both being AtLocation garage), or confuse fox_squirrel with red_fox due to the shared visual feature red. These errors suggest that while the relations are factually correct, they may not always align with the target classification semantics, especially when visual similarity overlaps with noisy or broad knowledge links. We are incorporating this analysis into the revised version to better contextualize the limitations and potential improvements for future work.\n\n| class ID | class name | model output | prediction |\n|:--------:|:----------:|------------|:----------:|\n|   n03345487        |  fire_engine               |  fire_engine. fire_engine AtLocation garage. fire_engine RelatedTo truck. fire_engine IsA machine.            |   pickup_truck         |\n|  n02088094         |   afghan_hound         |     horse. horse RelatedTo barn. horse Synonym horse. horse AtLocation farm_yard         |  barn          |\n|  n02356798         |   fox_squirrel             |  red_squirrel. red_squirrel DerivedFrom red. red_squirrel RelatedTo arctic_fox. red_squirrel IsA fox            |     red_fox       |\n|  n02363005         |   beaver                     |   fox. fox IsA animal. fox HasContext movies. fox RelateTo cartoons           |     red_fox       |\n\n>**Q1: About the choice of relation types.**\n\nWe compare using a fixed set of relations (only IsA, IsA + AtLocation, IsA + AtLocation + HasProperty) with our method that selects key relations adaptively. Our approach performs best. This is because rare classes, such as West Highland white terrier, often lack common relations like IsA or AtLocation, leaving only the class name and leading to information loss. Moreover, fixed relation sets tend to produce overlapping links, where multiple classes are associated with the same entities, increasing classification ambiguity. Therefore, selectively retaining specific relations without considering their availability per class leads to suboptimal results.\n\n| relations | IsA | IsA + AtLocation | IsA + AtLocation + HasProperty| Ours |\n|:--------:|:----------:|:------------:|:----------:|:----:|\n| Last Acc  | 81.23    | 81.75                 |   82.79     |   84.29   |\n\n> **Q2: On the importance of the second-hop relations.**\n\nWe aim to select relations that best distinguish all seen classes. When certain relations have already been used by previous learned classes and the current class lacks available one-hop relations, we explore second-hop relations. This has two key benefits: it avoids reusing the same relations, which could increase classification ambiguity, and it encourages the model to focus on distinctive features when differentiating similar classes, thus improving classification performance.\n\nTo validate this, we conducted comparative experiments across three settings:\n\n- One-hop relations (fallback to class name if no relation),\n\n- Class name only (GMM), and\n\n- Our adaptive method with second-hop relations.\n\n| Method| one hop |   class names(GMM) | Ours     |\n|:--------:|:------:|:--------:|:----:|\n|    Last Acc| 83.68       |   80.72        |  84.29    |\n\nOur method yields a 0.61 absolute improvement over the one-hop setting using the same number of relations, confirming the added value of second-hop relations in enhancing model discrimination and overall performance.\n\n>**Q3: Regarding different class orders.**\n\nWe agree that in our method, the class order may affect the continual learning performance; thus, we conducted experiments on different seed-based class orders in Appendix Sec A.1. The results show that despite slight variations in results across different orderings, our model maintains stable improvements over other baselines."}, {"review_id": "mXlY0NdCRO", "reviewer": "Reviewer_sh7m", "summary": "To mitigate catastrophic forgetting in continual learning, this paper constructs an evolving knowledge graph to leverage relational information between classes, using KG-based labels during training and a KG-augmented strategy to refine predictions at inference. Experiments on benchmark datasets demonstrate strong performance in both conventional and few-shot continual learning settings, outperforming state-of-the-art methods.", "questions": "This paper introduces multi-modal models for continual learning, and existing multi-modal model based continual learning studies are suggested to be mentioned. \n\n[1] Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models. \n\n[2] MLLM-CL: Continual Learning for Multimodal Large Language Models.\n\n[3] HiDe-LLaVA: Hierarchical decoupling for continual instruction tuning of multimodal large language model.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- By incorporating knowledge graphs, the model captures semantic relationships between classes, enabling more discriminative feature learning for similar categories in incremental tasks. \n\n- The KG-augmented inference strategy uses relational queries to guide model predictions, reducing misclassifications by leveraging prior knowledge. This effectively addresses the limitation of generative models drifting to higher-level categories. \n\n- Experiments on diverse datasets and CIL settings (both conventional and few-shot CIL) validate the effectiveness of the proposed method. \n\nWeaknesses:\n\n- The KG trimming strategy (e.g., selecting top-two relations or two-hop connections) introduces subjectivity, potentially omitting discriminative relations critical for fine-grained class distinctions. Moreover, the approach relies heavily on ConceptNet, which may lack task-specific semantic relations in specialized domains (e.g., medical imaging or robotic vision), limiting generalizability to niche applications.\n- While the study demonstrates KG-GMM’s superiority over GMM in exemplar-free settings, it does not evaluate whether the method maintains this advantage when competing with GMM augmented by data replay. Whether the KG’s benefits can surpass those of data replay, which can be validated by additional experiments.", "was_revised": true, "final_justification": "The author's revisions successfully addressed my concerns. I maintain my positive score.", "rebuttal": "Thank you for your valuable comments. We address your questions or concerns below.\n> **W1.1: About KG trimming strategy in fine-grained classes.** \n\nIn fine-grained classification, while distinguishing between classes may require more nuanced relations, including too many or overly complex ones can overwhelm the LLM, increase the likelihood of hallucinations, and degrade inference efficiency. Therefore, although slightly more relations may be necessary to capture subtle distinctions, careful trimming remains essential. It ensures that only the most informative and discriminative knowledge is retained, maintaining a balance between precision, model robustness, and computational efficiency.\n\n> **W1.2: About adaptability in specialized domains.**\n\nWe choose ConceptNet because it integrates multiple structured knowledge sources (e.g., WordNet, DBPedia, OpenCyc) and offers rich, diverse, and readily accessible commonsense knowledge. It covers a wide range of entities and relations relevant to standard continual learning benchmarks, making it a practical and effective choice.\n\nTo address scenarios where ConceptNet may not be applicable, we extend our approach in two ways: (1) For datasets with rich attribute annotations (e.g., CUB-200), we construct custom knowledge graphs based on these attributes. (2) For datasets lacking such annotations (e.g., HAM-10000), we utilize large pretrained models to generate visually grounded concepts and relations to form dataset-specific graphs. We include the corresponding setups and results to demonstrate the flexibility and generalizability of our method beyond ConceptNet.\n\n- CUB-200 is a fine-grained bird species dataset with detailed part-level attributes. We construct a knowledge graph using each class's most certain attributes (confidence score 4), such as has_bill_shape::hooked or has_bill_shape::cone, resulting in approximately 17 relations per class after our graph construction method. All 200 classes are equally separated into 10 tasks.\n- HAM-10000 is a dermatoscopic image dataset of 10,000 pigmented skin lesions of 7 classes for melanoma classification. Since we could not find a suitable knowledge graph for medical data, we instead used DeepSeek-VL-O1 to generate 10 representative attributes for each of the 7 classes (such as color_dominance::light brown, background_skin::No pigmentation), serving as their semantic descriptions or attribute sets. We follow [A] to separate 7 classes in 3 tasks, containing [2,2,3] classes each. \n\n[A] BiasPruner: Debiased Continual Learning for Medical Image Classification, MICCAI 2024\n\n| datasets | CUB-200 |      | HAM-10000 |      |\n|:--------:|:------:|:----:|:--------:|:----:|\n|          |   Avg  | Last |    Avg   | Last |\n|    GMM   | 49.34       | 40.39     |  79.22        |  63.81    |\n|  KG-GMM  |  52.65      |  43.84    |  81.45        |   64.34   |\n\nAs shown in our experimental results, when relevant and visually discernible attributes (e.g., color, shape, pattern) are available, our model yields noticeable improvements over GMM, demonstrating that our approach is adaptable and benefits from contextual knowledge beyond ConceptNet.\n\n> **W2: About comparison with GMM augmented by data replay.**\n\n\nAs shown below, our method outperforms GMM augmented by data replay on Tiny-ImageNet under both 100-5 and 100-10 task settings. Moreover, our method also achieves consistent performance improvements when augmented with exemplars.\n\n|        | exemplar | Tiny-ImageNet | Tiny-ImageNet     |   Tiny-ImageNet             |    Tiny-ImageNet  |    Tiny-ImageNet           |   Tiny-ImageNet   | ImageNet-R  |\n|--------|:-----:|:---------------:|:------:|:----------------:|:------:|:---------------:|:------:|:-------------:|\n|        |     | B100-5 tasks  |  B100-5 tasks    | B100-10 tasks | B100-10 tasks     | B100-20 tasks | B100-20 tasks     | B0-10 tasks |\n|        |     | Avg           | Last | Avg            | Last | avg           | Last | Last        |\n| GMM    |  &cross;   | 83.42              | 76.98     | 82.49               | 76.51     |  81.70             | 76.03     |  80.72           |\n| GMM    |  &check;   | 84.16              | 78.46     |  83.95              | 78.64      |  84.23             | 79.17     | 89.41            |\n| KG-GMM | &cross;    |  86.17             | 81.86     | 84.37               | 78.16     | 83.18              | 78.32     |  84.29           |\n| KG-GMM | &check;    |  **87.54**             | **82.35**     | **85.12**               | **79.32**     |  **85.17**            | **80.34**     | **90.11**             |\n\n\n> **Q1: About incorporating existing studies.**\n\nWe will carefully revise the manuscript and incorporate the relevant literature into our work."}, {"review_id": "2i821Jx3Gv", "reviewer": "Reviewer_ErR5", "summary": "This paper addresses the challenge of class-incremental learning (CIL) by leveraging generative multimodal models (GMMs). To overcome the limitations of traditional GMMs, the paper integrates a knowledge graph into the generative framework. The incorporation of the knowledge graph enhances the retention of knowledge from old tasks, providing significant benefits in both the training and testing phases. Experimental results on standard CIL and few-shot CIL scenarios demonstrate substantial improvements over existing state-of-the-art methods.", "questions": "1. Could you clarify the main motivation behind the \"Trim to Key Relationships\" step? Is it feasible to use all possible relationships, or are there specific constraints guiding this choice?\n\n2. Why does the \"GMM + Description Labels\" approach not perform as well, whereas GMM with a knowledge graph yields better results?\n\n3. The \"Inference Time\" line in Fig. 5 is somewhat confusing. Could you provide further clarification on what this represents?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of integrating a knowledge graph into class-incremental learning (CIL) is both meaningful and novel. Current methods suffer from forgetting due to limited attention to knowledge relationships. This paper offers a simple yet effective solution by enhancing high-level graph knowledge with images.\n\n2. The paper is well-structured, clearly presenting the concepts of Knowledge Graph Enhanced Learning and Knowledge Graph Augmented Inference. It effectively demonstrates the novel techniques applied in both the training and testing phases.\n\n3. The performance across multiple benchmarks underscores the effectiveness of the approach, achieving performance gains of approximately 2-4%.\n\n4. The ablation study, along with the time complexity analysis, highlights a favorable trade-off between accuracy and computational cost.\n\nWeaknesses:\n\n1. The motivation and explanations in some sections could be further clarified and strengthened (see Questions 1-3 for specific examples).\n\n2. A discussion on how the proposed methods can be extended to other relevant scenarios as GMM models continue to develop would significantly enhance the paper's impact.\n\n3. Including some failure cases would provide a more comprehensive evaluation of the method and give readers a better understanding of its limitations.", "was_revised": true, "final_justification": "After reading the author's reply, I decided to keep my score.", "rebuttal": "Thank you for your valuable comments. We address your questions or concerns below.\n\n> **W1.1: About the motivation behind the \"Trim to Key Relationships\" step.**\n\nThe motivation behind the \"Trim to Key Relationships\" step is twofold. First, it aims to retain only the most essential relationships so that the model can identify specific categories without generating excessively long outputs. Second, it focuses on preserving the most discriminative relationships to avoid different categories sharing the same relationships, which could lead to ambiguity.\n\n> **W1.2: Why does \"GMM + Description Labels\" not perform well.**\n\nThe original GMM approach uses CLIP to match the text output of the MLLM model, but CLIP performs suboptimally with longer text inputs like description labels, as well as with retrieval tasks within the same modality [A]. In contrast, GMM combined with a knowledge graph benefits from more structured and concise attribute outputs, which allows for more accurate localization using the knowledge graph itself.\n\n[A] Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion\n\n> **W1.3: Clarify the meaning of  Fig. 5.**\n\nThe green and blue lines correspond to accuracy and should be read against the left y-axis, while the orange line represents inference time and should be read against the right y-axis. As the maximum number of tokens increases, the generated descriptions become more detailed, leading to higher classification accuracy. However, the time required to generate a certain number of tokens is relatively fixed per token. Therefore, both the green and blue lines share the same orange line for inference time. We will improve the clarity of Fig.5 in the revised version.\n\n> **W2: About the discussion on how the method generalizes to broader scenarios.**\n\nTo verify the adaptability of our method across broader scenarios, we conducted experiments on several fine-grained and medical imaging datasets. The specific experimental settings are as follows:\n\n- CUB-200 is a fine-grained bird species dataset with detailed part-level attributes. We construct a knowledge graph using each class's most certain attributes (confidence score 4), such as has_bill_shape::hooked or has_bill_shape::cone, resulting in approximately 17 relations per class after our graph construction method. All 200 classes are equally separated into 10 tasks.\n- HAM-10000 is a dermatoscopic image dataset of 10,000 pigmented skin lesions of 7 classes for melanoma classification. Since we could not find a suitable knowledge graph for medical data, we instead used DeepSeek-VL-O1 to generate 10 representative attributes for each of the 7 classes (such as color_dominance::light brown, background_skin::No pigmentation), serving as their semantic descriptions or attribute sets. We follow [B] to separate 7 classes in 3 tasks, containing [2,2,3] classes each.\n\n[B] BiasPruner: Debiased Continual Learning for Medical Image Classification, MICCAI 2024\n\n| datasets | cub200 |      | HAM10000 |      |\n|:--------:|:------:|:----:|:--------:|:----:|\n|          |   Avg  | Last |    Avg   | Last |\n|    GMM   | 49.34       | 40.39     |  79.22        |  63.81    |\n|  KG-GMM  |  52.65      |  43.84    |  81.45        |   64.34   |\n\nAs shown in our experimental results, when relevant and visually discernible attributes (e.g., color, shape, pattern) are available, our model yields noticeable improvements over GMM, demonstrating that our approach is adaptable and benefits from contextual knowledge beyond ConceptNet.\n\n> **W3: About failure cases analysis.**\n\nWe have analyzed several representative failure cases, which will be added to the revised manuscript. These cases reveal that while the integration of the knowledge graph helps the model attend to factual visual cues (e.g., red, AtLocation garage, RelatedTo barn), it can also introduce semantically misleading associations.\n\nFor example, the model may misclassify fire_engine as pickup_truck due to shared contextual relations (e.g., both being AtLocation garage), or confuse fox_squirrel with red_fox due to the shared visual feature red. These errors suggest that while the relations are factually correct, they may not always align with the target classification semantics, especially when visual similarity overlaps with noisy or broad knowledge links. We are incorporating this analysis into the revised version to better contextualize the limitations and potential improvements for future work.\n\n\n| class ID | class name | model output | prediction |\n|:--------:|:----------:|------------|:----------:|\n|   n03345487        |  fire_engine               |  fire_engine. fire_engine AtLocation garage. fire_engine RelatedTo truck. fire_engine IsA machine.            |   pickup_truck         |\n|  n02088094         |   afghan_hound         |     horse. horse RelatedTo barn. horse Synonym horse. horse AtLocation farm_yard         |  barn          |\n|  n02356798         |   fox_squirrel             |  red_squirrel. red_squirrel DerivedFrom red. red_squirrel RelatedTo arctic_fox. red_squirrel IsA fox            |     red_fox       |\n|  n02363005         |   beaver                     |   fox. fox IsA animal. fox HasContext movies. fox RelateTo cartoons           |     red_fox       |"}, {"review_id": "9N2NV8NjMZ", "reviewer": "Reviewer_qKfb", "summary": "This paper proposes KG-GMM, which leverages a Knowledge Graph to enrich relational information for class incremental learning. It also introduces knowledge graph-augmented inference to preserve detailed knowledge and reduce forgetting. Experiments demonstrate its effectiveness in both class incremental and few-shot class incremental learning.", "questions": "Q1. While leveraging a knowledge graph to enrich relational information may help address the challenge of classifying at a higher semantic level, it remains unclear whether the proposed method adequately addresses the issue of losing generalization ability, because the proposed method still trains only simple linear layers.    \n\nQ2. As the knowledge graph is ImageNet-based, there is a concern of potential information leakage in the experiments conducted on ImageNet subsets (Tiny-ImageNet, ImageNet-R, Mini-ImageNet), since the model may have prior knowledge of both seen and unseen classes. Similarly, CIFAR-100 shares some class labels with ImageNet, raising the same concerns. It would strengthen the evaluation if the method were tested on fine-grained datasets, such as FGVC-Aircraft or Herbarium 19.\n\nQ3. Which model is used for the \"Zero-shot\" performance at the experiments (Table 1 and Table 2)? Additionally, in Table 2, why is the performance drop observed in the zero-shot evaluation, which assesses the performance without training?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## **Strengths**\n1. The paper is well-written, with clear and visually appealing figures. \n\n1. The authors provide a thorough background on Knowledge Graphs and Generative Multi-modal models for CIL, making the paper easy to follow.\n     \n## **Weaknesses**\n1. It is uncertain whether the proposed method effectively addresses the two challenges of leveraging LLMs in CIL, as mentioned in the introduction.\n\n1. The use of CIFAR-100 and subsets of ImageNet-21K is too limited to validate the general effectiveness of the proposed approach.\n\n1. The method section lacks explanation on how each component contributes to class incremental learning, particularly for learning new classes or retaining old ones. As a result, the method appears to be merely a combination of well-pretrained models and a knowledge graph built from large datasets.", "was_revised": true, "final_justification": "Thank you for the rebuttal. It has addressed all of my concerns, especially regarding the plasticity–stability dilemma in continual learning and the validation of the method’s general effectiveness. I have therefore decided to raise my score to borderline accept.", "rebuttal": "Thank you for your valuable comments. We address your questions or concerns below.\n> **W1&Q1: Whether our method addresses the two challenges of leveraging LLMs in CIL.**\n\nOur method is designed to address two major challenges in using MLLMs for continual learning:\n\n- Fixed-format degeneration: Standard GMM setups often produce one-word responses due to training with only class label names. Our approach mitigates this by training with relational textual descriptions, which encourages the MLLM to retain its expressive capacity and generate more descriptive, attribute-rich outputs.\n\n- Semantic forgetting: MLLMs tend to regress to generic categories over time (e.g., predicting apple instead of Granny Smith). We address this by guiding inference with relational prompts and using attribute-based reasoning via a knowledge graph, enabling finer-grained class predictions even after multiple learning stages.\n\nRegarding the reviewer’s concern on generalization and the use of a simple linear layer:\nWhile we employ a lightweight linear classifier for efficiency and continual adaptation, it is important to note that the core generalization capacity lies in the frozen MLLM backbone, which is pretrained on massive and diverse image-text data. Our method does not aim to retrain or overfit a small model, but rather to prompt the MLLM to utilize its pretrained capabilities more effectively by supplying structured, discriminative knowledge. Moreover, our experiments (refer to **W2&Q2**) across diverse datasets (including fine-grained benchmarks like CUB-200 and HAM-10000) demonstrate that the method maintains good generalization and transferability.\n\n> **W2&Q2: About more experiments outside the ImageNet-21K scope.**\n\nWe conducted exploratory experiments on several fine-grained datasets and medical imaging datasets, as detailed below: \n- CUB-200 is a fine-grained bird species dataset with detailed part-level attributes. We construct a knowledge graph using each class's most certain attributes (confidence score 4), such as has_bill_shape::hooked or has_bill_shape::cone, resulting in approximately 17 relations per class after our graph construction method. All 200 classes are equally separated into 10 tasks.\n- HAM-10000 is a medical image dataset of 10,000 pigmented skin lesions of 7 classes for melanoma classification. Since we could not find a suitable knowledge graph for medical data, we instead used DeepSeek-O1 to generate 10 representative attributes for each of the 7 classes (such as color_dominance::light brown, background_skin::No pigmentation), serving as their semantic descriptions or attribute sets. We follow [A] to separate 7 classes in 3 tasks, containing [2,2,3] classes each. \n- FVGC-Aircraft is a dataset comprising 100 different aircraft classes. As no suitable external knowledge graph was available, we utilized the structured metadata provided with the dataset. Specifically, each aircraft class is described using three hierarchical levels: 41 manufacturers, 70 families, and 100 variants. These are represented as three relations—hasManufacturer, hasFamily, and hasVariant—with each class associated with exactly one relation at each level, resulting in three relations per class.\n- Herbarium is a dataset containing 46,000 herbarium specimens across more than 680 species within the flowering plant family Melastomataceae. Due to the limited rebuttal time, it was difficult to process the entire dataset, so we randomly selected 10 species and used DeepSeek-O1 to generate six attribute-based relations: hasLeafShape, hasLeafMargin, hasLeafVeins, hasFlowersColor, hasStem, and hasFruit, each with corresponding class-specific properties. The selected classes were evenly divided into 5 tasks, with 2 species per task.\n\n[A] BiasPruner: Debiased Continual Learning for Medical Image Classification, MICCAI 2024\n\n| datasets | CUB-200 |      | HAM-10000 |      | FVGC-aircraft| | Herbarium 19| |\n|:--------:|:------:|:----:|:--------:|:----:|:----:|:----:|:----:|:----:|\n|          |   Avg  | Last |    Avg   | Last | Avg   | Last | Avg   | Last | \n|    GMM   | 49.34       | 40.39     |  79.22        |  63.81    | 51.63  | 47.25   | 86.34  | 79.88 |\n|  KG-GMM  |  52.65      |  43.84    |  81.45        |   64.34   |   51.88  |  47.88 |  88.59  |81.14    | \n\nThe experimental results above demonstrate that, with sufficiently rich KG information (CUB-200, HAM-10000), our method consistently improves upon GMM. When relevant knowledge is limited (as in the case of FVGC-Aircraft), the performance gains from our method are relatively modest. \n\n>**W3: About the explanation on how our method addresses the plasticity and stability dilemma in continual learning.**\n\nTraditional CIL approaches mitigate forgetting by (1) storing exemplars, (2) expanding model architectures, or (3) constraining parameter updates. GMM takes a different route by reformulating classification as a generation problem, removing the static classifier and mitigating its bias toward new classes. However, as discussed earlier, this introduces two challenges: degeneration into fixed-format outputs and semantic drift toward generic categories.\n\nOur KG-GMM is designed to address these challenges and contributes to both retaining old knowledge (stability) and learning new classes (plasticity) in the following ways:\n\n- Relation-enriched descriptions during training help maintain the expressiveness of the MLLM and reduce degeneration, enhancing plasticity for new tasks.\n\n- Graph-augmented inference provides structured prompts that guide the model to recover fine-grained class details, directly supporting stability by retrieving specific attributes of previously learned classes.\n\nEmpirically, our method shows stronger gains after multiple tasks are introduced, indicating enhanced knowledge retention. As shown in Table 2,  on Mini-ImageNet, improvement after Task 0 is modest (from 89.35 to 90.99), but becomes more evident by the final task (from 75.18 to 78.07). On CIFAR-100, gains grow from 91.53 to 91.83 (Task 0) and from 81.47 to 82.37 (final task).\nThis performance trajectory confirms that KG-GMM’s components are not merely plug-ins, but actively contribute to mitigating forgetting and preserving discriminative knowledge across tasks.\n\n> **Q3: About \"zero-shot\" baseline and why it has a performance drop.**\n\nThe baseline \"zero-shot\" follows the usage in the GMM paper, where MiniGPT-4 is directly prompted with the question \"What is this a photo of?\"  and the given test image. The model's textual response is then encoded using a CLIP text encoder and compared against all currently seen class embeddings—the most similar one is selected as the final predicted class. As continual learning progresses and the number of learned classes increases, the model is required to distinguish among more classes in a single step. As a result, even zero-shot models experience a noticeable drop in performance."}]

Author rebuttal:

Meta-review: The paper proposes to use knowledge graph generation to maintain the model's knowledge or previous tasks in the context of Incremental classification. The authors build on top MiniGpt4 and consider several tasks with no exemplars while only updating the linear projection layer. 

Limitations mainly included the reliance on Concept Net for knowledge graph generation and concept classes, and other experimental limitations that the authors have addressed in the rebuttal.

All reviewers appreciated the contribution of the paper and the rebuttal has addressed their concerns.
Decision: Accept (poster)