Beyond Synthetic Replays: Turning Diffusion Features into Few-Shot Class-Incremental Learning Knowledge

## Abstract

Abstract Few-shot class-incremental learning (FSCIL) is challenging due to extremely limited training data; while aiming to reduce catastrophic forgetting and learn new information. We propose Diffusion-FSCIL, a novel approach that employs a text-to-image diffusion model as a frozen backbone. Our conjecture is that FSCIL can be tackled using a large generative model’s capabilities benefiting from 1) generation ability via large-scale pre-training; 2) multi-scale representation; 3) representational flexibility through the text encoder. To maximize the representation capability, we propose to extract multiple complementary diffusion features to play roles as latent replay with slight support from feature distillation for preventing generative biases. Our framework realizes efficiency through 1) using a frozen backbone; 2) minimal trainable components; 3) batch processing of multiple feature extractions. Extensive experiments on CUB-200, miniImageNet, and CIFAR-100 show that Diffusion-FSCIL surpasses state-of-the-art methods, preserving performance on previously learned classes and adapting effectively to new ones. Our code is available at here .

## 1 Introduction

Catastrophic forgetting remains a significant challenge when sequentially training deep neural networks. Continual learning methods [3, 26] tackle this issue by adapting models to new tasks while minimizing performance degradation on previously learned ones. A more practical variant, few-shot class-incremental learning (FSCIL) [59, 52], has gained attention, which trains models to incrementally learn new classes from very limited samples while tackling the harsher catastrophic forgetting problem. However, many existing FSCIL methods [64, 37, 61, 57, 49, 2, 47] struggle with critical challenges such as feature misalignment across classes and persistent catastrophic forgetting, both of which compromise the retention of prior learned knowledge.

To address these challenges, we propose a novel FSCIL framework that leverages a generative pre-trained model as the backbone. However, unlike the discriminative models, no prior studies have effectively leveraged the full strength of generative backbones for FSCIL to our knowledge. This is presumably due to the conventional use of generative models as image generators, which may fail to diminish the domain gap between the real and synthesized images. Taking a step forward, we hypothesize that generative models pre-trained on large-scale data, such as Stable Diffusion [40], present a viable alternative to traditional discriminative backbones as an improved use-case emerges, particularly. Additionally, thanks to the architectural nature (*e.g*., UNet [41]), extracted representations offer richer, multi-scale features armed with more diversity; text encoders can flexibly enhance representation quality.

Figure: Figure 1: FSCIL performance comparison. Our method significantly outperforms all competing methods on the standard FSCIL benchmarks, including CUB-200, miniImageNet, and CIFAR-100, which demonstrates its effectiveness in mitigating catastrophic forgetting. Best view with digital zoom.
Refer to caption: /html/2503.23402/assets/x1.png

This paper substantiates a novel FSCIL model using Stable Diffusion (SD) in a proposed refined manner.
Our method extracts intermediate representations through inversion and generation processes (*i.e*., forward and backward processes, respectively) in diffusion models, which ensures consistent feature extraction across learning sessions. To address feature misalignment and mitigate forgetting, we reutilize textual inversion [12] to determine the nature of unseen tasks. By embedding task-specific textual cues, our approach facilitates better adaptation and understanding of novel tasks, allowing for more informed updates to the model. Notice that all feature extraction is performed simultaneously using the backbone with a set of image latents and corresponding text prompts to realize efficiency. Finally, we adopt the strategy of [57] to learn novel class prototypes to maintain proper class separation. Despite using SD as the backbone, our method’s actual training process is designed to be lightweight by freezing SD. This ensures 1) faster convergence reducing training time; 2) robustness to overfitting enabling efficient adaptation without compromising past knowledge.

Experiments are conducted by following the standard protocol of FSCIL [49, 57, 62, 2]. We train our models on CUB-200 [53], miniImageNet [42], and CIFAR-100 [24] and compare them with the stae-of-the-art methods. Ours outperforms competing models by large margins as shown in [Fig. 1](#S1.F1) while showing remarkably low computational costs, with only about 6M trainable parameters and faster training time comparable to recent state-of-the-art models.

## 2 Related Work

### 2.1 Few-shot class-incremental learning

Few-shot class-incremental learning (FSCIL) [59, 52] extends the class-incremental learning paradigm [3, 26] to scenarios where each new class arrives with only a handful of samples, often causing severe catastrophic forgetting and overfitting. Common strategies [64, 37, 61, 57, 49, 2, 47] include developing robust prototypes or feature representations that effectively balance old and new classes. Some methods employ meta-learning [66, 6, 4, 67, 15] to rapidly adapt to novel few-shot classes, while others leverage replay strategies [27, 1, 5, 8, 39], either storing small exemplar sets or using additional generative models. Our method also adopts a generative replay strategy but uniquely leverages a large-scale text-to-image diffusion [40] backbone, requiring no extra buffer for synthetic images or extra networks [25, 18] for precisely control.

### 2.2 Diffusion models and their applications

Diffusion models [17, 46, 40] have recently emerged as powerful generative models, and their subsequent researches draw attention in various fields. The key to their success lies in the attention mechanism, incorporating conditioned text information and latent features to generate high-quality samples. Building upon the power of architecture and pre-trained knowledge, several works [48, 14, 29, 60, 63] have successfully demonstrated the rich and expressive nature of diffusion features. Therefore, these methods give rise to various works in generation fields [51, 11], 2D vision problem [30, 54, 23, 54], image/video editing [13, 45, 33] and so on. In this paper, we also adopt the potential of the diffusion models to facilitate our FSCIL work, which differs from previous methods using poor generalization knowledge of images.

## 3 Background

This section briefly reviews preliminaries for few-shot class-incremental learning (FSCIL) and text-to-image (T2I) diffusion models, which are our problem setting and main component to constitute the overall pipeline, respectively. We then introduce our rationale to exploit the diffusion model in the context of FSCIL, via the pilot study.

### 3.1 Preliminary

Few-shot class-incremental learning. Few-shot class-incremental learning (FSCIL) [59, 52] aims to
incrementally train models on new class images with limited data (*i.e*., 5-shots or 10-shots for each class) without forgetting previously acquired knowledge. FSCIL consists of sequential training sessions $\{\mathcal{S}^{0},\mathcal{S}^{1},\dots,\mathcal{S}^{s}\}$. In the base session ($\mathcal{S}^{0}$), a model is trained on relatively abundant data $\mathcal{D}^{0}$ covering base classes $\mathcal{C}^{0}$. In incremental sessions ($\mathcal{S}^{s\geq 1}$), new classes $\mathcal{C}^{s}$ appear with a few-shot dataset $\mathcal{D}^{s}$. Existing approaches [62, 4, 8, 39, 2] maintain a small exemplar buffer $\mathcal{M}$ to mitigate the catastrophic forgetting. Thus, training at session $\mathcal{S}^{s}$ uses $\mathcal{T}^{s}=\mathcal{D}^{s}\cup\mathcal{M}^{1:s-1}$, with evaluation performed on all classes learned so far: $\mathcal{C}^{(0:s)}=\bigcup_{k=0}^{s}\mathcal{C}^{k}$. This paper follows the standard training and evaluation protocols of FSCIL [49, 57, 62, 2].

Text-to-image (T2I) diffusion models. A milestone generative model, Stable Diffusion (SD) [40], learns multi-scale visual representations from input images, and the extracted features are versatile. SD encodes images into latent representations $\mathbf{z}_{0}$ via a variational autoencoder (VAE) [10]. During the *inversion* (forward) process, Gaussian noise is gradually added to $\mathbf{z}_{0}$ across discrete timesteps $t\in[0,\text{T}]$, resulting in a fully noised latent $\mathbf{z}_{\text{T}}$. Conversely, the *generation* (reverse) process iteratively denoises from timestep T back to 0, guided by text prompts p. Formally, text prompts p are first embedded into textual embeddings $\mathbf{w}$ using a text encoder [38]. The denoising UNet [41] $\boldsymbol{\epsilon}_{\theta}$ is trained by minimizing the following loss:

$$ $\mathcal{L}_{\text{SD}}=\mathbb{E}_{\mathbf{z}_{t},t,\boldsymbol{\epsilon}\sim\mathcal{N}(0,1)}\Bigl{[}|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}(\mathbf{z}_{t},t,\mathbf{w})|_{2}^{2}\Bigr{]}.$ (1) $$

Since architectures like UNet inherently possess a multi-scale nature, multi-scale visual representations at varying spatial resolutions can be extracted from intermediate layers. Following prior works [54, 29, 23], we extract multi-scale visual features from intermediate decoder layers (1 to 12). Due to these layers producing feature maps of different spatial resolutions, resizing and unifying their resolutions before aggregation. Apart from being trained on massive image datasets, the extracted multi-scale representations provide diversity, which ranges from semantic-level representations to fine-grained details. Our method leverages this benefit.

Figure: Figure 2: Generative vs. discriminative models for FSCIL. We compare FSCIL models with different backbones, such as Stable Diffusion [40], DINOv2 [36], and OpenCLIP [19]. Extra replay denotes generated replay images by using SD. We reveal 1) SD has high potential surpassing strong DINOv2 and OpenCLIP; 2) a conventional approach like using generated replays or additional parameters [18] fail to retain previous knowledge; 3) using more generated replays with SD neither helps the SD backbone nor a discriminative backbone (ResNet).
Refer to caption: /html/2503.23402/assets/x2.png

### 3.2 Motivation

Our conjecture.
FSCIL fundamentally aligns with classification, which makes a discriminative model a natural choice for the backbone. Scaling up a classification model pre-trained on larger datasets has been a common strategy, leveraging their learned representations as implicit replay mechanisms.
However, generative models like SD [40] remain underexplored in this context despite their success in tasks like open-vocabulary segmentation [30], some representation learning [63, 54, 23, 29], and class incremental learning [32, 20, 22], which are not FSCIL.

Motivated by the effectiveness of diffusion models in prior scenarios, we argue that SD could be a compelling backbone for FSCIL that can surpass discriminative backbones that prevail. We claim that a key challenge is how to use them effectively, as utilizing them effectively to generate replays would presumably fail to fully exploit their potential. SD offers merits of 1) using learned features pre-trained on massive-scale data, 2) extracting multi-scale representations, and 3) using a text encoder that could enhance representational flexibility. We argue that those strengths position diffusion models as a compelling alternative to traditional discriminative architectures [9, 36, 19].

Pilot study.
We explore the potential of SD to rival strong discriminative models such as DINOv2 [36] or OpenCLIP [30]. Furthermore, we empirically validate the conjecture that existing diffusion-based approaches, which are primarily used for generating replays to mitigate forgetting, often fail.
Our experimental setup follows the standard FSCIL training and evaluation protocols; we minimally append a simple prototype classifier [57] to each backbone for a controlled experiment without any bells and whistles.

[Fig. 2](#S3.F2) presents the results of our study, highlighting 1) the potential of SD and 2) how the straightforward use of SD can incur failure in FSCIL. Extra replay samples (*i.e*., synthetic images) generated by SD using basic text prompts (*e.g*., “A photo of {class-name}”) as suggested in [20], prove ineffective despite increasing their quantity.
Additionally, we investigate the use of additional adaptations through Low-Rank Adaptation (LoRA), continuously adding new parameters to adapt SD for better capturing incremental knowledge. However, even with this targeted adaptation, improvements remain marginal, underscoring the inherent limitations of naïvely utilizing diffusion models for FSCIL. These findings suggest that straightforward
knowledge sharing fails to fully unlock the potential of diffusion models for better FSCIL performance. This study encourages us to devise a refined method that leverages the capability of SD more effectively. We believe that using a large pre-trained model is closer to ideal if no further adaptation (*e.g*., ImageNet pre-training) is needed. Once SD’s potential is revealed, using it like DINOv2 as task-agnostic features might be a viable option.

Figure: Figure 3: Schematic visualization of our framework. We visualize the proposed network architecture and the signal flows during the inversion (*i.e*., forward) and generation (*i.e*., backward) processes. We use Stable Diffusion (SD) [40] as a frozen feature extractor with the input latents to extract the multi-scale representations through both the inversion and generation processes. Four types of features [$\mathbf{F}_{\text{inv}}$, $\mathbf{F}_{\text{syn}}$, $\mathbf{F}_{\text{aug}}$, $\mathbf{F}_{\text{gen}}$] are extracted from SD’s inversion and generation processes (see Sec. [4.1](#S4.SS1), [4.2](#S4.SS2) and [4.3](#S4.SS3)). $\mathbf{F}_{\text{gen}}$ is extracted for training incremental sessions only, with all features obtained in a single forward propagation. The extracted features are aggregated through the aggregation network and transferred to lightweight convolution layers (*i.e*., Conv block) and multi-layer perceptrons (*i.e*., MLP block). Detailed architectures are provided in the [Appendix B](#A2). The classifier is a prototype-based classifier [57]. Notice that the frozen SD does not participate in training; only the lightweight remaining part (*i.e*., only ≅6M parameters) is trained throughout the entire training.
Refer to caption: /html/2503.23402/assets/x3.png

## 4 Method

In this section, we propose a novel framework for FSCIL that leverages SD as a frozen backbone. Our method comprises four key components: 1) a backbone architecture extracting multi-scale deep visual features ([Sec. 4.1](#S4.SS1)), 2) a class-specific generative replay mechanism ([Sec. 4.2](#S4.SS2)), 3) an augmented representation strategy for enriching sparse incremental samples ([Sec. 4.3](#S4.SS3)), and 4) additional training strategies and procedure ([Sec. 4.4](#S4.SS4)). A schematic illustration of the backbone architecture is provided in [Fig. 3](#S3.F3).

### 4.1 Extracting inversion and generation features

Inspired by denoising diffusion implicit models (DDIM) [46] and its application [29, 54], we extract multi-scale visual representations from SD’s frozen UNet $\boldsymbol{\epsilon}_{\theta}$. We reinforce the feature extraction step by using both *inversion* and *generation* processes, also known as forward and backward diffusion processes. It is designed to simultaneously extract SD’s *inversion* and *generation* features within a single forward pass so that another extra forward process is unnecessary.

Inversion process.
Given the image latent variable $\mathbf{z}_{0}$ at 0-th timestep, we first perform the *inversion* process to obtain the noised latent variable $\mathbf{z}_{t}$ at $t$-th timestep. By feeding $\mathbf{z}_{t}$ to UNet $\boldsymbol{\epsilon}_{\theta}$, we obtain the 12 layered multi-scale features; however, we exclude the lowest-resolution layers (from $1$ to $3$) to avoid overly coarse features with presumably limited semantic details and thereby generating $9$-layered features as $\{\mathbf{f}_{4},\mathbf{f}_{5},\dots,\mathbf{f}_{12}\}$.
To effectively leverage structural and semantic information across the layers, we aggregate the features using an aggregation network $g^{\text{agg}}$ similar to [29, 54]. Specifically, the aggregation network $g^{\text{agg}}$, which is composed of convolutional bottleneck layers, infers weights $\beta_{l}$ to each layer’s feature $\mathbf{f}_{l}$ and combine features to generate inversion feature $\mathbf{F}_{\text{inv}}$, as $\mathbf{F}_{\text{inv}}=\sum_{l=4}^{12}\beta_{l}\mathbf{f}_{l}$.

Generation process. Afterwards, we reverse the process through the *generation* process, denoising $\mathbf{z}_{t}$ back to $\mathbf{z}_{0}$. Specifically, we feed the latent variable $\mathbf{z}_{t}$ into UNet $\boldsymbol{\epsilon}_{\theta}$ at timestep $t$ and extract multi-scale features from layers 4–12 as $\{\mathbf{f}^{\prime}_{4},\mathbf{f}^{\prime}_{5},\dots,\mathbf{f}^{\prime}_{12}\}$. We then aggregate the synthesized features in a manner similar to the inversion process by $\mathbf{F}_{\text{syn}}=\sum_{l=4}^{12}\beta_{l}\mathbf{f}^{\prime}_{l}.$ In both inversion and generation processes, text prompts guide the extraction of visual features.
Subsequently, we feed both $\mathbf{F}_{\text{inv}}$ and $\mathbf{F}_{\text{syn}}$ into a lightweight network consisting of a convolutional layer $g^{\text{conv}}$, a two-layer MLP $g^{\text{MLP}}$, and a prototype-based classifier [57].

Discussion on diffusion steps. We opt for using a single *inversion* and *generation* step ($t=1$) within the full timestep range ($t=\text{T}$) for efficiency, which is sufficient to achieve comparable performance when compared to the features using multiple timesteps. Intuitively, using multiple timesteps can enrich semantics, but we argue that the extracted features easily become redundant. This also coincides with the results of [23, 54], and thus, we use the single *inversion* and *generation* steps to extract diffusion features ($\textbf{F}_{\text{inv}},\textbf{F}_{\text{syn}}$) throughout the experiment.

### 4.2 Text-based class-specific generative features

Thanks to our architecture, which fully leverages the generative capability of SD, we naturally enable unlimited replay of previously learned classes by synthesizing class-specific features conditioned solely on text prompts p, eliminating the need to store synthetic images. Specifically, starting from a fully noised latent variable $\mathbf{z}_{\text{T}}$ at the final diffusion timestep ($t=\text{T}$), we iteratively denoise it back to $\mathbf{z}_{0}$, obtaining the generative features $\mathbf{F}_{\text{gen}}$ at the final denoise step ($\textbf{z}_{1}\to\textbf{z}_{0}$). However, naïvely employing simple text prompts p (*e.g*., “A photo of {class-name}”) typically results in generated features that fail to capture subtle, fine-grained class characteristics and poorly preserve previously learned class knowledge, limiting replay effectiveness (failure cases are provided in [Appendix C](#A3)).

To address this issue, we propose optimizing discriminative, class-specific prompts $\textbf{p}^{*}$ for each class. Taking advantage of our backbone’s ability to generate data solely conditioned on text prompts without additional modules, we create class-specific prompts that explicitly capture the unique characteristics of each class. To achieve this, we employ a class-specific prompt tuning strategy inspired by Gal et al. [12]. Specifically, for each class $c\in\mathcal{C}$, we define unique textual prompts $\textbf{p}^{*}_{c}$ associated with a learnable textual embedding $\mathbf{w}_{c}^{*}$. The embeddings $\mathbf{w}_{c}^{*}$ are initialized from the embedding of a class-name prompt (*e.g*., “cardinal”, “apple”) and optimized using the original SD loss defined in [Eq. 1](#S3.E1), based on only a few images from class $c$:

$$ $\small\mathbf{w}_{c}^{*}=\arg\min_{\mathbf{w}_{c}}\mathbb{E}_{\mathbf{z}_{t},t,\boldsymbol{\epsilon}\sim\mathcal{N}(0,1)}\Bigl{[}\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}(\mathbf{z}_{t},t,\tau_{\theta}(\textbf{p}^{*}_{c}))\|_{2}^{2}\Bigr{]},$ (2) $$

where both $\boldsymbol{\epsilon}_{\theta}$ and the text encoder $\tau_{\theta}$ are kept frozen during optimization. By minimizing this objective with a learnable parameter $\mathbf{w}_{c}$, we embed the visual concept of class $c$ directly into the textual embedding space of SD. Consequently, once learned, the optimized prompts $\textbf{p}_{c}^{*}$ for the specific class $c$ (*e.g*., “A photo of $\textbf{p}^{*}_{c}$”) can be directly utilized in the full generation process ($\mathbf{z}_{\text{T}}\to\mathbf{z}_{0}$), synthesizing class-specific generative features $\mathbf{F}_{\text{gen}}$ that precisely capture class identities, thus enabling unlimited replay of previously learned classes without synthetic image storage burdens.

Figure: Figure 4: Visualizations of augmented features $\textbf{F}_{\text{aug}}$ at varying noise timesteps. We visualize the diffusion features by injecting different noises into the latent of a real image. (a) Original images, (b) Noise-injected timesteps $t$, (c) Generated images after denoising (for visualization purposes only), and (d–f) Corresponding extracted features $\textbf{f}_{l}$ from the UNet’s decoder layers at the final denoising step.
Refer to caption: /html/2503.23402/assets/x4.png

### 4.3 Noise-augmented generative featues

Beyond replaying previously learned class information, our generative capability naturally extends to enriching data representations. Inspired by [31], we propose an image-to-image augmentation strategy to address limited data diversity in FSCIL.
Specifically, we encode a real image $x$ into a latent $\mathbf{z}_{0}$ using a pre-trained VAE, and directly inject Gaussian noise into the latent at an intermediate timestep $t\sim(0,\text{T}]$ using the DDIM [46] scheduler, producing a partially noised latent variable $\mathbf{z}_{t}$. Subsequently, we apply the generation process by progressively denoising the latent variable from timestep $t$ down to 0, resulting in a latent variable that structurally resembles the original latent $\mathbf{z}_{0}$ while incorporating diverse variations.
Finally, we extract augmented features $\mathbf{F}_{\text{aug}}$ at the final denoising step ($\mathbf{z}_{1}\to\mathbf{z}_{0}$), guided by class-specific prompts $\mathbf{p}_{c}^{*}$, thereby yielding representations that maintain semantic fidelity to the original image while introducing beneficial variations for improved generalization.

Balancing fidelity and diversity. The choice of the noise injection timestep $t$ critically determines the trade-off between fidelity and diversity in the augmented features $\mathbf{F}_{\text{aug}}$. Timesteps closer to 0 introduce minimal noise, yielding features closely resembling the original latent and thus limiting augmentation diversity. Conversely, timesteps closer to T introduce significant noise, potentially generating unrealistic features beyond the model’s representational capability. To effectively balance this trade-off, we discretize the total diffusion steps T into $m$ intervals and randomly sample noise injection timesteps from:

$$ $\small t\sim\left\{\tfrac{\text{T}}{m},\;\tfrac{\text{2T}}{m},\;\dots,\;\tfrac{(m-1)\text{T}}{m},\;\text{T}\right\},\;\text{where}\;m>1.$ (3) $$

By varying this intermediate noise injection timestep $t$, we produce diverse yet realistic augmented features $\textbf{F}_{\text{aug}}$ through augmented noises, effectively enhancing generalization in scenarios with insufficient visual representations. We visually illustrate examples of augmented features $\mathbf{F}_{\text{aug}}$ extracted at varying noise injection timesteps in [Fig. 4](#S4.F4). Furthermore, visualizations of the four feature (*i.e*., $\mathbf{F}_{\text{inv}}$, $\mathbf{F}_{\text{syn}}$, $\mathbf{F}_{\text{gen}}$, and $\mathbf{F}_{\text{aug}}$) differences are provided in [Appendix D](#A4).

### 4.4 Training strategies

Our training process consists of two stages: base session ($\mathcal{S}^{0}$) and incremental sessions ($\mathcal{S}^{s{\geq}1}$). The overall training procedure is algorithmically summarized in the Appendix and in the rest of this sub-section.

Base session $\mathcal{S}^{0}$. We utilize three types of features (*i.e*., $\mathbf{F}_{\text{inv}}$, $\mathbf{F}_{\text{syn}}$, and $\mathbf{F}_{\text{aug}}$) during the base session training. We efficiently employ $\mathbf{F}_{\text{aug}}$ only at the final training epochs as a brief fine-tuning step to minimize the burden of applying the entire generation process in the base session.

Incremental sessions $\mathcal{S}^{s{\geq}1}$.
During the incremental sessions training, only the MLP layer $g^{\text{MLP}}$ is trained with the few-shot data $\mathcal{D}^{s}$, while the aggregation network $g^{\text{agg}}$ and convolutional layers $g^{\text{conv}}$ remain frozen. We extract four types of features (*i.e*., $\mathbf{F}_{\text{inv}}$, $\mathbf{F}_{\text{syn}}$, $\mathbf{F}_{\text{gen}}$, and $\mathbf{F}_{\text{aug}}$) from the same aggregation network $g^{\text{agg}}$ during the incremental sessions. Unlike the base session, we additionally utilize the class-specific generative features $\mathbf{F}_{\text{gen}}$ to effectively mitigate catastrophic forgetting.

**Table 1: Performance comparison of FSCIL methods on CUB-200 [53]. AA denotes the average accuracy and FI represents the accuracy improvement at the final session. ^† denotes that values are from [57]. Note that our method does not pre-train the trainable layers on ImageNet-1K; we report the improved results when performing it as done in other methods. The bold numbers indicate the best results, and the second-best results are underlined.**
| Methods | Venue | Session Acc. (%) $\uparrow$ | AA (%) $\uparrow$ | FI |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |  |  |  |  |
| $\text{TOPIC}^{\dagger}$ [50] | CVPR’20 | 68.7 | 62.5 | 54.8 | 50.0 | 45.3 | 41.4 | 38.4 | 35.4 | 32.2 | 28.3 | 26.3 | 43.9 | +26.3 |
| $\text{Self-promoted}^{\dagger}$ [66] | CVPR’21 | 68.7 | 61.9 | 57.4 | 52.7 | 50.2 | 46.9 | 44.7 | 43.1 | 40.2 | 39.6 | 37.3 | 49.3 | +32.9 |
| $\text{CEC}^{\dagger}$ [58] | CVPR’21 | 75.9 | 71.9 | 68.5 | 63.5 | 62.4 | 58.3 | 57.7 | 55.8 | 54.8 | 53.5 | 52.3 | 61.3 | +17.9 |
| DSN [55] | TPAMI’22 | 76.1 | 72.2 | 69.6 | 66.7 | 64.4 | 62.1 | 60.2 | 58.9 | 57.0 | 55.1 | 54.2 | 63.3 | +16.0 |
| $\text{LIMIT}^{\dagger}$ [65] | TPAMI’22 | 76.3 | 74.2 | 72.7 | 69.2 | 68.8 | 65.6 | 63.6 | 62.7 | 61.5 | 60.4 | 58.5 | 66.7 | +11.7 |
| $\text{FACT}^{\dagger}$ [64] | CVPR’22 | 75.9 | 73.2 | 70.8 | 66.1 | 65.6 | 62.2 | 61.7 | 59.8 | 58.4 | 57.9 | 56.9 | 64.4 | +13.3 |
| $\text{MetaFSCIL}^{\dagger}$ [6] | CVPR’22 | 75.9 | 72.4 | 68.8 | 64.8 | 63.0 | 60.0 | 58.3 | 56.8 | 54.8 | 53.8 | 52.6 | 61.9 | +17.6 |
| $\text{ALICE}^{\dagger}$ [37] | ECCV’22 | 77.4 | 72.7 | 70.6 | 67.2 | 65.9 | 63.4 | 62.9 | 61.9 | 60.5 | 60.6 | 60.1 | 65.8 | +10.1 |
| ERDFR [27] | ECCV’22 | 75.9 | 72.1 | 68.6 | 63.8 | 62.6 | 59.1 | 57.8 | 55.9 | 54.9 | 53.6 | 52.4 | 61.5 | +11.9 |
| CLOM [67] | NeurIPS’22 | 79.6 | 76.1 | 72.9 | 69.8 | 67.8 | 65.6 | 63.9 | 62.6 | 60.6 | 60.3 | 59.6 | 67.2 | +10.6 |
| $\text{NC-FSCIL}^{\dagger}$ [57] | ICLR’23 | 80.5 | 76.0 | 72.3 | 70.3 | 68.2 | 65.2 | 64.4 | 63.3 | 60.7 | 60.0 | 59.4 | 67.3 | +10.8 |
| SubNetwork [21] | ICLR’23 | 78.1 | 74.6 | 71.4 | 67.5 | 65.4 | 62.6 | 61.1 | 59.4 | 57.5 | 57.2 | 56.8 | 64.7 | +13.4 |
| BiDistFSCIL [62] | CVPR’23 | 79.1 | 75.4 | 72.8 | 69.1 | 67.5 | 65.1 | 64.0 | 63.5 | 61.9 | 61.5 | 60.9 | 67.3 | +9.3 |
| SAVC [47] | CVPR’23 | 81.9 | 77.9 | 75.0 | 70.2 | 70.0 | 67.0 | 66.2 | 65.3 | 63.8 | 63.2 | 62.5 | 69.4 | +7.7 |
| OrCo [2] | CVPR’24 | 75.6 | 66.8 | 64.1 | 63.7 | 62.2 | 60.3 | 60.2 | 59.2 | 58.0 | 54.9 | 52.1 | 61.5 | +18.1 |
| CLOSER [34] | ECCV’24 | 79.4 | 75.9 | 73.5 | 70.5 | 69.2 | 67.2 | 66.7 | 65.7 | 64.0 | 64.0 | 63.6 | 69.1 | +6.6 |
| Yourself [49] | ECCV’24 | 83.4 | 77.0 | 75.3 | 72.2 | 69.0 | 66.8 | 66.0 | 65.6 | 64.1 | 64.5 | 63.6 | 69.9 | +6.6 |
| Ours | – | 84.2 | 78.0 | 76.6 | 73.3 | 71.5 | 69.9 | 69.4 | 69.2 | 66.6 | 65.9 | 64.3 | 71.7 | – |
| + ImageNet pre-train | – | 86.5 | 80.9 | 79.7 | 77.0 | 74.9 | 73.3 | 73.2 | 72.8 | 70.6 | 70.9 | 70.2 | 75.7 | – |

Training loss. To train the model, we mainly adopt the dot-regression (DR) loss $\mathcal{L}_{\text{DR}}$ from [57, 56] and use this loss when training the network using three types of features (*i.e*., $\mathbf{F}_{\text{inv}}$, $\mathbf{F}_{\text{syn}}$, and $\mathbf{F}_{\text{aug}}$). However, for the class-specific generative feature $\mathbf{F}_{\text{gen}}$, we additionally use the knowledge distillation loss to properly train the network. Despite the high fidelity of synthesized features $\mathbf{F}_{\text{gen}}$ produced by optimized class-specific prompts, these generated features inherently exhibit subtle discrepancies compared to real-data features, introducing domain gaps or generative artifacts. To mitigate these minor artifacts and ensure closer alignment with the real-data feature distribution, we introduce a lightweight MLP-based feature distillation strategy. Specifically, we transfer knowledge from a frozen teacher model $\phi_{\text{t}}(\cdot)$ to a student model $\phi_{\text{s}}(\cdot)$ by minimizing the cosine distance between their intermediate representations:

$$ $\small\mathcal{L}_{\text{distill}}=1-\cos\Bigl{(}\phi_{\text{t}}(\mathbf{F}_{\text{gen}}),~\phi_{\text{s}}(\mathbf{F}_{\text{gen}})\Bigr{)},$ (4) $$

which measures the cosine distance between their intermediate representations. Through such alignment, this distillation step mitigates artifacts from imperfect generations while preserving previously learned knowledge. Notably, this process only requires transferring a lightweight MLP.

As a result, our train loss is the DR loss, combined with the distillation loss $\mathcal{L}_{\text{distill}}$ from [Eq. 4](#S4.E4) to transfer previously acquired knowledge. Thus, the total training loss at incremental session $\mathcal{S}^{s{\geq}1}$ is defined as
$\mathcal{L}_{\text{total}}^{s}=\mathcal{L}_{\text{DR}}+\beta^{s}\mathcal{L}_{\text{distill}},$
where $\beta^{s}$ is a session-dependent weighting factor balancing the two loss terms. We progressively adjust $\beta^{s}$ based on the accumulated amount of previously learned knowledge, gradually increasing its value to reinforce the retention of past information as incremental learning proceeds. Specifically, we employ a linear growth strategy
$\beta^{s}=\beta_{\text{init}}+\frac{s}{\mathcal{S}}(1.0-\beta_{\text{init}}),$
where $\beta_{\text{init}}$, $s$, and $\mathcal{S}$ denote the initial distillation coefficient, the current incremental session, and the total number of the sessions, respectively.

### 4.5 Inference

At the inference stage, we utilize the inversion feature $\mathbf{F}_{\text{inv}}$ extracted via a single-step inversion ($\mathbf{z}_{0}\to\mathbf{z}_{1}$) with a null text prompt. Additional generative processes are omitted, to ensure the efficiency.

## 5 Experiment

### 5.1 Setup

Datasets. We use three benchmarks, including a challenging fine-grained benchmark CUB-200 [53], which consists of 200 bird species with 11,788 images at 224×224 resolution. Second, *mini*ImageNet [42] consists of 100 classes, each containing 500 training and 100 test images with a resolution of 84×84, is used. Finally, we use CIFAR-100 [24], which has 100 classes as well, where the images are 32×32 resolution. Following the standard protocol [57, 47, 37], we use 100 base classes for CUB-200 with 10 incremental sessions (*i.e*., 10-way 5-shot). For *mini*ImageNet and CIFAR-100, we adopt 60 base classes followed by 8 incremental sessions (*i.e*., 5-way 5-shot).

Evaluation metrics. We evaluate using several metrics. Session accuracy measures the model’s performance at each $i$-th session. Average accuracy (AA) represents the mean accuracy across all FSCIL sessions from 0-th session to $i$-th session, showing how well the model generalizes across both new and old classes. Final accuracy improvement (FI) measures the performance gap between our method and the other methods in the last session.

**Table 2: Performance comparison of FSCIL methods on *mini*ImageNet [42]. AA denotes the average accuracy and FI represents the improvement of the final session. ^† denotes that values are from [57]. The bold numbers indicate the best results, and the second-best results are underlined.**
| Methods | Venue | Session Acc. (%) | AA (%) $\uparrow$ | FI |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |  |  |  |  |
| $\text{TOPIC}^{\dagger}$ [50] | CVPR’20 | 61.3 | 50.1 | 45.2 | 41.2 | 37.5 | 35.5 | 32.2 | 29.5 | 24.4 | 39.6 | +41.0 |
| $\text{Self-promoted}^{\dagger}$ [66] | CVPR’21 | 61.5 | 63.8 | 59.5 | 55.5 | 52.5 | 49.6 | 46.7 | 43.8 | 41.9 | 52.7 | +23.5 |
| $\text{CEC}^{\dagger}$ [58] | CVPR’21 | 72.0 | 66.8 | 63.0 | 59.4 | 56.7 | 53.7 | 51.2 | 49.2 | 47.6 | 57.8 | +17.8 |
| FSIL-GAN [1] | MM’22 | 69.9 | 62.9 | 59.8 | 58.9 | 57.1 | 54.1 | 50.6 | 48.1 | 46.1 | 56.4 | +19.4 |
| $\text{LIMIT}^{\dagger}$ [65] | TPAMI’22 | 72.3 | 68.5 | 64.3 | 60.8 | 58.0 | 55.1 | 52.7 | 50.7 | 49.2 | 59.1 | +16.2 |
| FACT [64] | CVPR’22 | 72.6 | 69.6 | 66.4 | 62.8 | 60.6 | 57.3 | 54.3 | 52.2 | 50.5 | 60.7 | +14.9 |
| $\text{MetaFSCIL}^{\dagger}$ [6] | CVPR’22 | 72.0 | 68.0 | 63.8 | 60.3 | 57.6 | 55.2 | 53.0 | 50.8 | 49.2 | 58.9 | +16.2 |
| $\text{C-FSCIL}^{\dagger}$ [15] | CVPR’22 | 76.4 | 71.1 | 66.5 | 63.3 | 60.4 | 57.5 | 54.8 | 53.1 | 51.4 | 61.6 | +14.0 |
| ERDFR [27] | ECCV’22 | 71.8 | 67.1 | 63.2 | 59.8 | 57.0 | 54.0 | 51.6 | 49.5 | 48.2 | 58.0 | +7.4 |
| $\text{ALICE}^{\dagger}$ [37] | ECCV’22 | 80.6 | 70.6 | 67.4 | 64.5 | 62.5 | 60.0 | 57.8 | 56.8 | 55.7 | 64.0 | +9.7 |
| CLOM [67] | NeurIPS’22 | 73.1 | 68.1 | 64.2 | 60.4 | 57.4 | 54.3 | 51.5 | 49.4 | 48.0 | 58.5 | +17.4 |
| $\text{NC-FSCIL}^{\dagger}$ [57] | ICLR’23 | 84.0 | 76.8 | 72.0 | 67.8 | 66.4 | 64.0 | 61.5 | 59.5 | 58.3 | 67.8 | +7.1 |
| SubNetwork [21] | ICLR’23 | 79.7 | 74.3 | 70.0 | 66.4 | 63.2 | 60.0 | 57.4 | 55.4 | 54.1 | 64.5 | +11.3 |
| BiDistFSCIL [62] | CVPR’23 | 74.7 | 70.4 | 66.3 | 62.8 | 60.8 | 57.2 | 54.8 | 53.7 | 52.2 | 61.4 | +13.2 |
| SAVC [47] | CVPR’23 | 81.1 | 76.1 | 72.4 | 68.9 | 66.5 | 63.0 | 59.9 | 58.4 | 57.1 | 67.1 | +8.3 |
| OrCo [2] | CVPR’24 | 83.3 | 75.3 | 71.5 | 68.2 | 65.6 | 63.1 | 60.2 | 58.8 | 58.1 | 67.1 | +7.3 |
| CLOSER [34] | ECCV’24 | 76.0 | 71.6 | 68.0 | 64.7 | 61.7 | 58.9 | 56.2 | 54.5 | 53.3 | 62.8 | +12.1 |
| Yourself [49] | ECCV’24 | 84.0 | 77.6 | 73.7 | 70.0 | 68.0 | 64.9 | 62.1 | 59.8 | 59.0 | 68.8 | +6.4 |
| Ours | – | 89.2 | 82.7 | 78.7 | 75.7 | 73.6 | 70.7 | 68.3 | 66.7 | 65.5 | 74.6 | – |

Implementation details.
We employ a pre-trained Stable Diffusion (SD) v1.5 [40] with a classifier-free guidance scale of 7.5 [16] and DDIM [46] scheduler. For the inversion process, we use the prompt “Null” (for CUB-200, we use “Bird” to reflect its bird-specific composition) during training to prepare inference as accessing detailed class names is impossible. For the generation process during training, we use the class names (*e.g*., “apple”, “beaver”) for each dataset. Furthermore, all images are resized to $512\times 512$ for diffusion feature extraction, consistent with SD’s requirements. For CUB-200, we do not pre-train our model on ImageNet [7] unlike previous methods [57, 49, 6, 37, 62], though we report the ImageNet pre-trained results as well (see [Tab. 1](#S4.T1)). Following the methods [39, 62, 8, 44, 2] that utilize exemplar buffer, we store only one sample per class in the memory buffer $\mathcal{M}$.

**Table 3: Ablation study results on miniImageNet and CUB-200 datasets. The Bold marked number indicates the best result.**
|  | Methods | miniImageNet | CUB-200 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AA | Acc. | Base | Inc. | AA | Acc. | Base | Inc. |  |
| (a) | $\mathbf{F}_{\text{inv}}$ | 69.7 | 57.4 | 75.4 | 30.4 | 70.0 | 60.6 | 74.8 | 46.7 |
| (b) | (a) + $\mathbf{F}_{\text{syn}}$ | 70.5 | 58.7 | 77.5 | 30.6 | 70.5 | 61.2 | 75.3 | 47.3 |
| (c) | (b) + $\mathbf{F}_{\text{gen}}$ | 71.5 | 60.2 | 77.5 | 33.3 | 74.3 | 67.1 | 80.7 | 53.7 |
| (d) | (c) + $\mathbf{F}_{\text{aug}}$ | 74.6 | 65.5 | 81.3 | 41.8 | 75.7 | 70.2 | 80.2 | 60.4 |

### 5.2 Comparison with other methods

[Tab. 1](#S4.T1) and [Tab. 2](#S5.T2) report our results on CUB-200 and miniImageNet, respectively, where we achieve consistent state-of-the-art results across all sessions. Specifically, we achieve an average accuracy of 75.7% for CUB-200 and 74.6% for miniImageNet, demonstrating that our approach can actively leverage diffusion features to achieve robust performance improvements even with few-shot samples.

Notably, in the CUB-200 experiment, where other most prior methods [49, 2, 34, 62, 64, 57, 27] typically utilize ImageNet [7] for pre-training, our method significantly surpasses the previous state-of-the-art approaches such as Yourself [49], OrCo [2], and CLOSER [34], even though we do not perform ImageNet [7] pre-training. This demonstrates that effectively utilizing a generative backbone, despite not being trained specifically for classification, enables our method to surpass traditional discriminative designs.

CIFAR-100 results in [Tab. A](#A1.T1) (full results provided in Appendix.) show that ours’ base session ($\mathcal{S}^{0}$) performance is initially lower compared with Yourself [49] and NC-FSCIL [57] up to session 3. However, our method successfully maintains the performance across upcoming sessions and shows significant improvements in incremental sessions, ultimately achieving the highest performance in the last session with 59.9% and 69.0% average accuracy, which achieves a new state-of-the-art result.

**Table 4: Ablation study of initial distillation value $\beta_{\text{init}}$ on miniImageNet. $\beta^{s}=0$ denotes the case where distillation is not used. Bold indicates the best results.**
| $\beta_{init}$ | AA | Acc. | Base | Inc. |
| --- | --- | --- | --- | --- |
| $\beta^{s}=0$ | 71.5 | 60.2 | 77.9 | 33.7 |
| 0.0 | 74.5 | 65.3 | 81.1 | 41.6 |
| 0.1 | 74.6 | 65.5 | 81.3 | 41.8 |
| 0.5 | 74.0 | 65.1 | 83.0 | 38.2 |
| 0.7 | 73.5 | 62.1 | 83.7 | 29.6 |

### 5.3 Ablation study

We conduct ablation studies on key components using four metrics: average accuracy (AA), session accuracy (Acc.) at the last session, base class ($\mathcal{C}^{0}$) accuracy (Base), and incremental class ($\mathcal{C}^{s\geq 1}$) accuracy (Inc.).

On main components.
[Tab. 3](#S5.T3) summarizes the results of our ablation study on the *mini*ImageNet and CUB-200 datasets. Starting from the baseline in (a), adding $\textbf{F}_{\text{syn}}$ in (b) improves AA by 0.8% on miniImageNet and 0.5% on CUB-200, and similarly enhances Acc. at the last session, confirming the value of synthesized features in enhancing generalization. Introducing class-specific generative features $\textbf{F}_{\text{gen}}$ in (c) significantly boosts Inc. and reduces forgetting, clearly visible through improvements in AA and Acc. metrics for both datasets. Lastly, employing partial diffusion-based augmentation $\mathbf{F}_{\text{aug}}$ in (d) notably boosts incremental class accuracy, ultimately achieving the highest AA (74.6%, 75.7%) and Acc. (65.4%, 70.2%) on *mini*ImageNet and CUB-200 respectively. These results demonstrate that our proposed components effectively mitigate the model’s forgetting, significantly improving generalization from limited samples.

On progressive distillation.
[Tab. 4](#S5.T4) investigates the impact of the initial value $\beta_{\text{init}}$ for distillation loss $\mathcal{L}_{\text{distill}}$. We observe lower initial values ($\beta_{\text{init}}\in\{0.0,0.1\}$) facilitate incremental learning by effectively balancing the prior knowledge and new class information. Conversely, higher ones ($\beta_{\text{init}}\in\{0.5,0.7\}$) may prioritize preserving base class $\mathcal{C}^{0}$ knowledge (higher Base) yet strikingly degrade performance on new classes (lower Inc.). This highlights that excessively strong distillation from the beginning limits effective adaptation to incremental classes. Based on the study, we use a lower initial value $\beta_{\text{init}}{=}0.1$ in our experiments. Furthermore, when compared to $\beta^{s}=0$, we can see that $\mathcal{L}_{\text{distill}}$ is helpful for improving the overall accuracy.

**Table 5: Ablation study of diffusion noise interval $m$ and single noise strength on miniImageNet. Left: multi-interval noise strengths ($m>1$); right: results for a single noise strength constant ($t=\lambda\text{T}$). The best results are marked in bold.**
| Multi-interval noise strengths |  |  |  |  |
| --- | --- | --- | --- | --- |
| $m$ | AA | Acc. | Base | Inc. |
| 3 | 74.1 | 65.4 | 79.9 | 41.7 |
| 5 | 74.6 | 65.5 | 81.3 | 41.8 |
| 7 | 74.0 | 65.2 | 81.2 | 41.1 |

On noise variants.
[Tab. 5](#S5.T5) explores the impact of varying the noise interval $m$ for augmented features $\mathbf{F}_{\text{aug}}$. Employing interval $m{=}5$ achieves optimal performance by effectively balancing diversity and semantic fidelity. For the noise interval $m{>}5$, performance declines as the generated augmented features $\mathbf{F}_{\text{aug}}$ become overly similar and redundant. Conversely, single noise approaches ($t\in\{0.3\text{T},0.5\text{T},0.7\text{T}\}$) yield notably lower incremental accuracy due to the reduced diversity of $\mathbf{F}_{\text{aug}}$ caused by consistently similar transformations of original latent $\textbf{z}_{0}$. These results emphasize that systematically varying noise strengths is crucial to enhancing feature diversity and improving generalization in FSCIL.

**Table 6: Computational cost comparison. We compare training time versus final accuracy across all incremental sessions on CUB-200 with Yourself [49]. eff.ver denotes a variant limiting the training iterations and proportion of generative features. Ours matches the accuracy with significantly faster training and achieves much higher performance with extended training time.**
| Model | Training Time (min) | Last Acc. (%) |
| --- | --- | --- |
| Yourself [49] | $\approx 1236$ | 63.6 |
| Ours (eff. ver) | $\approx\textbf{162}$ | 63.5 |
| Ours | $\approx 2070$ | 70.2 |

### 5.4 Computational costs - training time

One may question the training cost of our method using SD compared to recent approaches in practice.
[Tab. 6](#S5.T6) reports training time versus final accuracy across all incremental sessions on CUB-200 compared with Yourself [49]. Our method inherently involves complete generative processes ($\textbf{F}_{\text{gen}}$ and $\textbf{F}_{\text{aug}}$), potentially increasing training complexity compared to purely discriminative approaches. We introduce an efficient variant (dubbed eff.ver), which restricts the generative features used in $\textbf{F}_{\text{gen}}$ and $\textbf{F}_{\text{aug}}$ along with reducing the total training iterations minimally.
Compared to the previous SOTA method Yourself [49], our efficient variant exhibits comparable accuracy with greater efficiency. This result confirms that our framework effectively balances accuracy and training efficiency, offering flexibility to available computational resources.

## 6 Conclusion

In this paper, we have tackled the challenging few-shot class incremental learning (FSCIL) task with a meticulously designed method based on a pre-trained text-to-image diffusion model. We presumed a frozen diffusion model is efficient and well-suited for FSCIL due to its high expressiveness and flexibility. We have proposed extracting complementary discriminative features generated through diffusion processes: 1) inversion and generation; 2) flexible text guidance-based diffusion; 3) noise-augmented diffusion features. The generated features help mitigate catastrophic forgetting during the incremental sessions.
We observed that our pipeline achieves state-of-the-art results on FSCIL benchmarks (*i.e*., CUB-200, miniImageNet, CIFAR-100), with empirical studies confirming the complementary role of each feature type.
Finally, we demonstrate that our framework is not inefficient but rather enjoys computational efficiency. We hope our method becomes an improved baseline for future research aiming to utilize diffusion models to tackle FSCIL.

Limitations. While our method has demonstrated striking performance, the computational costs should be continuously managed. We reported the competitive training time of our method, which appears comparable to a competing method even without fine optimizations. Further optimization would indeed offer more advantages for speeding up.

## References

- Agarwal et al. [2022]
Aishwarya Agarwal, Biplab Banerjee, Fabio Cuzzolin, and Subhasis Chaudhuri.
Semantics-driven generative replay for few-shot class incremental learning.
In *ACM MM*, 2022.
- Ahmed et al. [2024]
Noor Ahmed, Anna Kukleva, and Bernt Schiele.
Orco: Towards better generalization via orthogonality and contrast for few-shot class-incremental learning.
In *CVPR*, 2024.
- Cauwenberghs and Poggio [2000]
Gert Cauwenberghs and Tomaso Poggio.
Incremental and decremental support vector machine learning.
In *NeurIPS*, 2000.
- Chen and Lee [2021]
Kuilin Chen and Chi-Guhn Lee.
Incremental few-shot learning via vector quantization in deep embedded space.
In *ICLR*, 2021.
- Cheraghian et al. [2021]
Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi.
Semantic-aware knowledge distillation for few-shot class-incremental learning.
In *CVPR*, 2021.
- Chi et al. [2022]
Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang.
Metafscil: A meta-learning approach for few-shot class incremental learning.
In *CVPR*, 2022.
- Deng et al. [2009]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In *CVPR*, 2009.
- Dong et al. [2021]
Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong.
Few-shot class-incremental learning via relation knowledge distillation.
In *AAAI*, 2021.
- Dosovitskiy et al. [2021]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *ICLR*, 2021.
- Esser et al. [2021]
Patrick Esser, Robin Rombach, and Bjorn Ommer.
Taming transformers for high-resolution image synthesis.
In *CVPR*, 2021.
- Fang et al. [2023]
Chuan Fang, Yuan Dong, Kunming Luo, Xiaotao Hu, Rakesh Shrestha, and Ping Tan.
Ctrl-room: Controllable text-to-3d room meshes generation with layout constraints.
*arXiv*, 2023.
- Gal et al. [2022]
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.
An image is worth one word: Personalizing text-to-image generation using textual inversion.
*arXiv*, 2022.
- Gu et al. [2024]
Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang.
Videoswap: Customized video subject swapping with interactive semantic point correspondence.
In *CVPR*, 2024.
- Hedlin et al. [2024]
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi.
Unsupervised semantic correspondence using stable diffusion.
In *NeurIPS*, 2024.
- Hersche et al. [2022]
Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi.
Constrained few-shot class-incremental learning.
In *CVPR*, 2022.
- Ho and Salimans [2022]
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
*arXiv*, 2022.
- Ho et al. [2020]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *NeurIPS*, 2020.
- Hu et al. [2022]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
In *ICLR*, 2022.
- Ilharco et al. [2021]
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.
OpenCLIP, 2021.
- Jodelet et al. [2023]
Quentin Jodelet, Xin Liu, Yin Jun Phua, and Tsuyoshi Murata.
Class-incremental learning using diffusion model for distillation and replay.
In *ICCVW*, 2023.
- Kang et al. [2023]
Haeyong Kang, Jaehong Yoon, Sultan Rizky Hikmawan Madjid, Sung Ju Hwang, and Chang D Yoo.
On the soft-subnetwork for few-shot class incremental learning.
In *ICLR*, 2023.
- Kim et al. [2024]
Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tiruneh, and Seungryul Baek.
Sddgr: Stable diffusion-based deep generative replay for class incremental object detection.
In *CVPR*, 2024.
- Kondapaneni et al. [2024]
Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona.
Text-image alignment for diffusion-based perception.
In *CVPR*, 2024.
- Krizhevsky et al. [2009]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images, 2009.
- Li et al. [2023]
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.
Gligen: Open-set grounded text-to-image generation.
In *CVPR*, 2023.
- Li and Hoiem [2017]
Zhizhong Li and Derek Hoiem.
Learning without forgetting.
*TPAMI*, 2017.
- Liu et al. [2022]
Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang.
Few-shot class-incremental learning via entropy-regularized data-free replay.
In *ECCV*, 2022.
- Loshchilov and Hutter [2019]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *ICLR*, 2019.
- Luo et al. [2024]
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell.
Diffusion hyperfeatures: Searching through time and space for semantic correspondence.
In *NeurIPS*, 2024.
- Marcos-Manchón et al. [2024]
Pablo Marcos-Manchón, Roberto Alcover-Couso, Juan C SanMiguel, and Jose M Martínez.
Open-vocabulary attention maps with token optimization for semantic segmentation in diffusion models.
In *CVPR*, 2024.
- Meng et al. [2021]
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
Sdedit: Guided image synthesis and editing with stochastic differential equations.
*arXiv*, 2021.
- Meng et al. [2024]
Zichong Meng, Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, and Yanzhi Wang.
Diffclass: Diffusion-based class incremental learning.
In *ECCV*, 2024.
- Mou et al. [2024]
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang.
Dragondiffusion: Enabling drag-style manipulation on diffusion models.
In *ICLR*, 2024.
- Oh et al. [2024]
Junghun Oh, Sungyong Baik, and Kyoung Mu Lee.
Closer: Towards better representation learning for few-shot class-incremental learning.
In *ECCV*, 2024.
- OpenAI [2023]
OpenAI.
Chatgpt.
[https://chat.openai.com](https://chat.openai.com), 2023.
Large language model.
- Oquab et al. [2023]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.
Dinov2: Learning robust visual features without supervision.
*arxiv*, 2023.
- Peng et al. [2022]
Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell.
Few-shot class-incremental learning from an open-set perspective.
In *ECCV*, 2022.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *ICML*, 2021.
- Rebuffi et al. [2017]
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.
icarl: Incremental classifier and representation learning.
In *CVPR*, 2017.
- Rombach et al. [2022]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *CVPR*, 2022.
- Ronneberger et al. [2015]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In *MICCAI*, 2015.
- Russakovsky et al. [2015]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge.
In *IJCV*, 2015.
- Samuel et al. [2024]
Dvir Samuel, Rami Ben-Ari, Matan Levy, Nir Darshan, and Gal Chechik.
Where’s waldo: Diffusion features for personalized segmentation and retrieval.
In *NeurlPS*, 2024.
- Shi et al. [2021]
Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu.
Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima.
In *NeurlPS*, 2021.
- Shi et al. [2024]
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai.
Dragdiffusion: Harnessing diffusion models for interactive point-based image editing.
In *CVPR*, 2024.
- Song et al. [2020]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
*arXiv*, 2020.
- Song et al. [2023]
Zeyin Song, Yifan Zhao, Yujun Shi, Peixi Peng, Li Yuan, and Yonghong Tian.
Learning with fantasy: Semantic-aware virtual contrastive constraint for few-shot class-incremental learning.
In *CVPR*, 2023.
- Tang et al. [2023]
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.
Emergent correspondence from image diffusion.
In *NeurIPS*, 2023.
- Tang et al. [2024]
Yu-Ming Tang, Yi-Xing Peng, Jingke Meng, and Wei-Shi Zheng.
Rethinking few-shot class-incremental learning: Learning from yourself.
In *ECCV*, 2024.
- Tao et al. [2020]
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong.
Few-shot class-incremental learning.
In *CVPR*, 2020.
- Tewel et al. [2024]
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon.
Training-free consistent text-to-image generation.
*ACM TOG*, 2024.
- Tian et al. [2024]
Songsong Tian, Lusi Li, Weijun Li, Hang Ran, Xin Ning, and Prayag Tiwari.
A survey on few-shot class-incremental learning.
*Neural Networks*, 2024.
- Wah et al. [2011]
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.
The caltech-ucsd birds-200-2011 dataset, 2011.
- Wang et al. [2024]
Tianfu Wang, Guosheng Hu, and Hongguang Wang.
Object pose estimation via the aggregation of diffusion features.
In *CVPR*, 2024.
- Yang et al. [2022a]
Boyu Yang, Mingbao Lin, Yunxiao Zhang, Binghao Liu, Xiaodan Liang, Rongrong Ji, and Qixiang Ye.
Dynamic support network for few-shot class incremental learning.
*TPAMI*, 2022a.
- Yang et al. [2022b]
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao.
Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network?
In *NeurIPS*, 2022b.
- Yang et al. [2023]
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao.
Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning.
In *ICLR*, 2023.
- Zhang et al. [2021]
Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu.
Few-shot incremental learning with continually evolved classifiers.
In *CVPR*, 2021.
- Zhang et al. [2023]
Jinghua Zhang, Li Liu, Olli Silven, Matti Pietikäinen, and Dewen Hu.
Few-shot class-incremental learning: A survey.
*arXiv*, 2023.
- Zhang et al. [2024]
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang.
A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence.
In *NeurIPS*, 2024.
- Zhao et al. [2021]
Hanbin Zhao, Yongjian Fu, Mintong Kang, Qi Tian, Fei Wu, and Xi Li.
Mgsvf: Multi-grained slow versus fast framework for few-shot class-incremental learning.
*TPAMI*, 2021.
- Zhao et al. [2023a]
Linglan Zhao, Jing Lu, Yunlu Xu, Zhanzhan Cheng, Dashan Guo, Yi Niu, and Xiangzhong Fang.
Few-shot class-incremental learning via class-aware bilateral distillation.
In *CVPR*, 2023a.
- Zhao et al. [2023b]
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu.
Unleashing text-to-image diffusion models for visual perception.
In *CVPR*, 2023b.
- Zhou et al. [2022a]
Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan.
Forward compatible few-shot class-incremental learning.
In *CVPR*, 2022a.
- Zhou et al. [2022b]
Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan.
Few-shot class-incremental learning by sampling multi-phase tasks.
*TPAMI*, 2022b.
- Zhu et al. [2021]
Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha.
Self-promoted prototype refinement for few-shot class-incremental learning.
In *CVPR*, 2021.
- Zou et al. [2022]
Yixiong Zou, Shanghang Zhang, Yuhua Li, and Ruixuan Li.
Margin-based few-shot class-incremental learning with class-level overfitting mitigation.
In *NeurIPS*, 2022.

## Appendix A Comparative study with other backbones

This section presents a further quantitative analysis based on our pilot study in [Sec. 3.2](#S3.SS2), providing a more detailed comparison with other large-scale pre-trained backbones. We visualize our method’s results alongside other large-scale pre-trained models such as DINOv2 [36] and OpenCLIP [19]. While those large backbones excel at general-purpose feature extraction, they often struggle to capture fine-grained or localized details as effectively as generative approaches. In contrast, Stable Diffusion (SD) utilizes a coarse-to-fine pixel-level reconstruction process, resulting in more detailed and robust visual features [29, 48, 43]. As shown in [Fig. A](#A1.F1), although SD initially lags slightly behind DINOv2 and OpenCLIP in the base session accuracy, it demonstrates superior knowledge retention, consistently outperforming them across incremental sessions. Ultimately, our approach amplifies this advantage, achieving the highest final accuracy (61.8%), significantly mitigating catastrophic forgetting compared to DINOv2 (45.3%) and OpenCLIP (48.2%).

The qualitative PCA analysis in [Fig. B](#A1.F2) demonstrates the advantages of extracting multi-scale features from SD. By leveraging multiple layers from SD, we can capture diverse visual characteristics from general structural patterns in earlier layers to detailed, fine-grained cues in later layers. We believe the extracted multi-scale features become semantically meaningful representations. As evidence, our proposed model leveraged the multi-layer SD features, taking advantage of their complementary strengths to outstandingly enhance FSCIL performance.

Figure: Figure A: Comparison of large pre-trained backbones. We evaluate several backbones – DINOv2, OpenCLIP, and SD, appended by the prototype classifier [56, 57]. We borrow the results from [Fig. 2](#S3.F2). Beyond the potential of SD we have observed in [Fig. 3](#S3.F3), our proposed method further exploits its strengths to maximize performance and achieve substantial gains over competing backbones.
Refer to caption: /html/2503.23402/assets/x5.png

**Table A: Performance comparison of FSCIL methods on CIFAR-100 [24]. AA denotes the average accuracy, and FI represents the accuracy improvement at the final session. ^† denotes that values are from [57].**
| Methods | Venue | Session Acc. (%) $\uparrow$ | AA (%) $\uparrow$ | FI |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |  |  |  |  |
| $\text{TOPIC}^{\dagger}$ [50] | CVPR’20 | 64.1 | 55.9 | 47.1 | 45.2 | 40.1 | 36.4 | 34.0 | 31.6 | 29.4 | 42.6 | +30.5 |
| $\text{Self-promoted}^{\dagger}$ [66] | CVPR’21 | 64.1 | 65.9 | 61.4 | 57.5 | 53.7 | 50.8 | 48.6 | 45.7 | 43.3 | 54.5 | +16.6 |
| $\text{CEC}^{\dagger}$ [58] | CVPR’21 | 73.1 | 68.9 | 65.3 | 61.2 | 58.1 | 55.6 | 53.2 | 51.3 | 49.1 | 59.5 | +10.8 |
| FSIL-GAN [1] | MM’22 | 70.1 | 64.4 | 57.2 | 55.2 | 54.3 | 51.9 | 50.1 | 47.9 | 46.6 | 55.3 | +4.6 |
| $\text{DSN}^{\dagger}$ [55] | TPAMI’22 | 73.0 | 68.8 | 64.8 | 62.6 | 59.4 | 57.0 | 54.0 | 51.6 | 50.0 | 60.1 | +9.9 |
| $\text{LIMIT}^{\dagger}$ [65] | TPAMI’22 | 73.8 | 72.1 | 67.9 | 63.9 | 60.7 | 57.8 | 55.7 | 53.5 | 51.2 | 61.8 | +8.7 |
| FACT [64] | CVPR’22 | 74.6 | 72.1 | 67.6 | 63.5 | 61.4 | 58.4 | 56.3 | 54.2 | 52.1 | 62.2 | +7.8 |
| $\text{MetaFSCIL}^{\dagger}$ [6] | CVPR’22 | 74.5 | 70.1 | 66.8 | 62.8 | 59.5 | 56.5 | 54.4 | 52.6 | 50.0 | 60.8 | +9.9 |
| $\text{C-FSCIL}^{\dagger}$ [15] | CVPR’22 | 77.5 | 72.4 | 67.5 | 63.3 | 59.8 | 57.0 | 54.4 | 52.5 | 50.5 | 61.6 | +9.4 |
| ERDFR [27] | ECCV’22 | 74.4 | 70.2 | 66.5 | 62.5 | 59.7 | 56.6 | 54.5 | 52.4 | 50.1 | 60.8 | +9.8 |
| $\text{ALICE}^{\dagger}$ [37] | ECCV’22 | 79.0 | 70.5 | 67.1 | 63.4 | 61.2 | 59.2 | 58.1 | 56.3 | 54.1 | 63.2 | +5.8 |
| CLOM [67] | NeurIPS’22 | 74.2 | 69.8 | 66.2 | 62.4 | 59.3 | 56.5 | 54.4 | 52.2 | 50.3 | 60.6 | +9.6 |
| $\text{NC-FSCIL}^{\dagger}$ [57] | ICLR’23 | 82.5 | 76.8 | 73.3 | 69.7 | 66.2 | 62.9 | 61.0 | 59.0 | 56.1 | 67.5 | +3.8 |
| SubNetwork [21] | ICLR’23 | 80.0 | 75.8 | 71.8 | 67.4 | 64.1 | 60.9 | 59.1 | 56.9 | 54.8 | 65.6 | +5.1 |
| BiDistFSCIL [62] | CVPR’23 | 79.5 | 75.4 | 71.8 | 68.0 | 65.0 | 62.0 | 60.2 | 57.7 | 55.9 | 66.1 | +4.0 |
| SAVC [47] | CVPR’23 | 78.8 | 73.3 | 69.3 | 64.9 | 61.7 | 59.3 | 57.1 | 55.2 | 53.1 | 63.6 | +6.8 |
| OrCo [2] | CVPR’24 | 80.1 | 68.2 | 67.0 | 61.0 | 59.8 | 58.6 | 57.0 | 55.1 | 52.2 | 62.1 | +7.7 |
| CLOSER [34] | ECCV’24 | 75.7 | 71.8 | 68.3 | 64.6 | 61.9 | 59.3 | 57.5 | 55.4 | 53.3 | 63.1 | +6.6 |
| Yourself [49] | ECCV’24 | 82.9 | 76.3 | 72.9 | 67.8 | 65.2 | 62.0 | 60.7 | 58.8 | 56.6 | 67.0 | +3.3 |
| Ours | – | 82.3 | 76.1 | 72.6 | 68.6 | 68.0 | 66.0 | 64.9 | 62.4 | 59.9 | 69.0 | – |

Figure: Figure B: PCA visualization comparison. We extract the features from the diverse backbones pre-trained on miniImageNet [42] to analyze feature differences.. Our method uses both earlier and later features of the decoder; the earlier layers’ features have similar representation power compared with other large models, and later ones illustrate the details of the input image, which cannot be distinguished in the features from other backbones.
Refer to caption: /html/2503.23402/assets/x6.png

Figure: Figure C: Architectural details of our methods. Except for the backbone we used, our method consists of mainly three components: 1) convolutional layer ($g^{\text{conv}}$), 2) MLP layer ($g^{\text{MLP}}$), and 3) aggregation network ($g^{\text{agg}}$).
Refer to caption: /html/2503.23402/assets/x7.png

For the experiment here, we employ Stable Diffusion v1.5 [40] for the baseline SD and our method, and the ViT-L/14 model for both DINOv2 and OpenCLIP. All backbone models utilize the same prototype-based classifier [57] to ensure fairness in comparison. Training parameters are consistent across experiments, including batch size (16), learning rates ($10^{-4}$ for both MLP network $g^{\text{MLP}}$ and aggregation network $g^{\text{agg}}$), and a cosine annealing scheduler.

## Appendix B Additional implementation details

Experimental details.
We provide additional implementation details for our main experiments presented in [Sec. 5](#S5), as well as experimental results on CIFAR-100 ([Tab. A](#A1.T1)). We train the model using AdamW [28] with a weight decay of $10^{-4}$. The initial learning rate is set to $3\times 10^{-3}$ for the MLP layers ($g^{\text{MLP}}$) and $10^{-3}$ for the aggregation bottleneck network ($g^{\text{agg}}$). During the base session ($\mathcal{S}^{0}$) training, we train all components: the aggregation bottleneck network ($g^{\text{agg}}$), convolutional layer ($g^{\text{conv}}$), and MLP layers ($g^{\text{MLP}}$). In training incremental sessions ($\mathcal{S}^{s\geq 1}$), we train only the MLP layers while keeping the aggregation bottleneck network and convolutional layers frozen. We apply standard data augmentation techniques including random resizing, rotation, color jittering, and horizontal flipping.

For each dataset, we implement specific session splits and training schedules. On CUB-200 [53], we train the base session for 150 epochs, adding 5 classes per incremental session (classes 55–100) with a batch size of 64. During incremental sessions, we set the MLP layers ($g^{\text{MLP}}$) learning rate to $6\times 10^{-3}$. On miniImageNet, the base session runs for 170 epochs, adding 5 classes per session (classes 105–140), with a batch size of 64 and an MLP learning rate of $6\times 10^{-3}$ during incremental sessions. On CIFAR-100 [24], we train the base session for 200 epochs, using a batch size of 80. Incremental sessions use an MLP learning rate of $4\times 10^{-3}$.

Architectural details.
Detailed architectures of our subnetworks (*i.e*., $g^{\text{conv}}$, $g^{\text{MLP}}$, $g^{\text{agg}}$) are visualized in [Fig. C](#A1.F3). Specifically, the convolutional block ($g^{\text{conv}}$) consists of a convolutional layer followed by Batch Normalization (BN), SiLU activation, and a bottleneck layer for efficient feature projection. The MLP block ($g^{\text{MLP}}$) is structured with two linear layers, each followed by Group Normalization (GN) and SiLU activation functions. Lastly, the aggregation network ($g^{\text{agg}}$), inspired by prior works [29, 54], integrates multi-layer SD features through layer-wise upsampling, convolutional processing, element-wise addition, and final L2 normalization, enabling effective utilization of multi-scale diffusion features.

Class-specific prompt details.
Certain class labels (*e.g*., “ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle”) contain multiple synonymous object names, making it challenging to encode them clearly into a single token embedding. To resolve this ambiguity, we utilize a large language model (LLM) [35] to unify these labels into a concise single-word label. Each single-word embedding derived from this simplified label is then used to initialize a learnable textual embedding $w_{c}^{*}$ for each class $c$, subsequently optimized to produce the final class-specific prompts $\textbf{p}_{c}^{*}$. By employing single-word embeddings, we ensure that the semantic concept of each label is captured consistently and efficiently, eliminating redundancy or semantic dilution that occurs when multi-word labels are split across multiple tokens. The complete set of templates used for optimizing the learnable class-specific prompts $\textbf{p}^{*}$ and corresponding hyper-parameters are provided in [Tab. B](#A3.T2) and [Tab. C](#A3.T3), respectively.

Additionally, for CIFAR-100 [24], we adjust the synthesized image resolution to 384, as its original images ($32\times 32$ pixels) are too small for effective diffusion-based feature extraction. This adaptation is necessary given SD’s pre-training resolution of 512 pixels. In contrast, such adjustments are unnecessary for CUB-200 and *mini*ImageNet, as their native image resolutions ($224\times 224$ and $84\times 84$, respectively) are sufficient for effective feature extraction by the diffusion model.

Overall training procedure.
We summarize the overall training procedure algorithmically in [Algorithm 1](#alg1), clearly illustrating the training flow and the utilized features in both base ($\mathcal{S}^{0}$) and incremental sessions ($\mathcal{S}^{s\geq 1}$).

Figure: Algorithm 1 Overall training procedure

## Appendix C Effectiveness of the optimized prompts

Here, we highlight the importance of utilizing optimized class-specific prompts $\textbf{p}^{*}$ through a qualitative evaluation of SD’s generative capability. To clarify this, we visually compare two prompt cases: features extracted using a simple, generic textual prompt (*e.g*., “A photo of {class-name}”) and features extracted using our optimized, class-specific prompts $\textbf{p}^{*}$ (described in [Sec. 4.2](#S4.SS2)). Specifically, as illustrated in [Fig. D](#A4.F4) and [Fig. E](#A4.F5), when conditioned on a naïve prompt (*e.g*., “A photo of {class-name}”), SD frequently generates images that either roughly capture the overall appearance but fail to generate details of object’s attributes (columns 2), or completely fail to reproduce objects related to the given prompt (columns 5). In contrast, employing optimized class-specific prompts $\textbf{p}^{*}_{c}$ for various class $c$ significantly enhances the generative quality, enabling precise denoising of latent variables from $\mathbf{z}_{\text{T}}$ back to $\mathbf{z}_{0}$, thus yielding features $\mathbf{F}_{\text{gen}}$ that accurately reflect class identities.

**Table B: Examples of template for textual inversion process. “{}” is a placeholder to input the class to generate according to the FSCIL sessions.**
| Templates |
| --- |
| “a photo of a {}”, “a rendering of a {}”, “a cropped photo of the {}”, “the photo of a {}”, “a photo of a clean {}”, “a photo of a dirty {}”, “a dark photo of the {}”, “a photo of my {}”, “a photo of the cool {}”, “a close-up photo of a {}”, “a bright photo of the {}”, “a cropped photo of a {}”, “a photo of the {}”, “a good photo of the {}”, ‘a photo of one {}”, “a close-up photo of the {}”, “a rendition of the {}”, “a photo of the clean {}”, “a rendition of a {}”, “a photo of a nice {}”, “a good photo of a {}”, “a photo of the nice {}”, “a photo of the small {}”, “a photo of the weird {}”, “a photo of the large {}”, “a photo of a cool {}”, “a photo of a small {}” |

**Table C: Hyper-parameters for optimizing class-specific prompts.**
| Hyper-params | CUB-200 [53] | *mini*ImageNet [42] | CIFAR-100 [24] |
| --- | --- | --- | --- |
| Batch size | 1 | 1 | 1 |
| Warm-up iter. | 200 | 200 | 200 |
| Learning rate | $10^{-4}$ | $10^{-4}$ | $10^{-4}$ |
| Training iter. | 2000 | 2000 | 2000 |
| Resolution | $512$ | $512$ | $384$ |
| Emb. vec. size | 5 | 7 | 7 |

## Appendix D Comparison between diffusion features

Visualization of multi-scale features. We present detailed visualizations of the multi-scale SD features utilized in our framework in [Fig. F](#A4.F6). We visualize the four feature types (*i.e*., $\mathbf{F}_{\text{inv}}$, $\mathbf{F}_{\text{syn}}$, $\mathbf{F}_{\text{aug}}$, and $\mathbf{F}_{\text{gen}}$) extracted from different layers (from $\mathbf{f}_{4}$ to $\mathbf{f}_{12}$) of SD, given the same input image on CUB-200. For the augmented features $\mathbf{F}_{\text{aug}}$, we inject Gaussian noise at timestep $0.5\text{T}$ (halfway point of the diffusion process) to clearly illustrate how partial noise affects the resulting feature representation.

“Generated image” shown in [Fig. F](#A4.F6) is the result of the generative feature extraction ($\mathbf{F}_{\text{gen}}$), obtained using optimized class-specific prompts $\textbf{p}_{c}^{*}$ (specifically, (a) Herring_Gull and (b) Ring_billed_Gull). These generated images are shown only for visualization purposes and are not explicitly utilized during training or inference.

Visualization across noise injection timesteps.
Additionally, we visually show how the injected noise at intermediate timesteps $t$ over the full diffusion time T impacts the augmented features $\mathbf{F}_{\text{aug}}$. As shown in [Fig. G](#A4.F7), we illustrate $\mathbf{F}_{\text{aug}}$ at four distinct timesteps: $t=\{\frac{\text{T}}{4},\frac{2\text{T}}{4},\frac{3\text{T}}{4},\text{T}\}$. For each timestep, we provide the original and generated images (for visualization purposes only). The augmented features $\mathbf{F}_{\text{aug}}$ are presented from the top-left to the bottom-right, corresponding sequentially from layer $\mathbf{f}_{4}$ to $\mathbf{f}_{12}$.

Figure: Figure D: Visualization of synthesized images for generative features $\mathbf{F}_{\text{gen}}$ on CUB-200 [53]. From left to right, we show the reference images, generated images using a naïve prompt (*e.g*., “A photo of {class-name}”), and generated images obtained using our optimized, class-specific prompts $\textbf{p}^{*}$. Columns 2 and 5 illustrate typical failures when employing naïve prompts, where synthesized images either lack precise details (column 2) or fail entirely to match the reference class (column 5). In contrast, Columns 3 and 6 clearly demonstrate improved synthesis quality when optimized class-specific prompts $\textbf{p}^{*}$ are utilized. These visualizations explicitly highlight the necessity of optimized prompts for accurate generative feature extraction. (Note: This figure is provided solely for qualitative visualization purposes.)
Refer to caption: /html/2503.23402/assets/x8.png

Figure: Figure E: Visualization of synthesized images for generative features $\mathbf{F}_{\text{gen}}$ on miniImageNet [42]. From left to right, we show the reference images, images generated by naïve prompts, and images generated by optimized class-specific prompts $\textbf{p}^{*}$. Naïve prompt-based generation often leads to insufficient detail capture (column 2) or completely incorrect generation (column 5). However, using optimized prompts $\textbf{p}^{*}$ significantly improves the generative results, closely reflecting class-specific characteristics (column 3 and 6). We emphasize that these results are visualized solely to qualitatively illustrate differences in prompt effectiveness.
Refer to caption: /html/2503.23402/assets/x9.png

Figure: (a)
Refer to caption: /html/2503.23402/assets/x10.png

Figure: Figure G: SD intermediate feature visualization across different diffusion timesteps $t$. For each noise-injected timestep ($\frac{\text{T}}{4},\frac{2}{4}\text{T},\frac{3}{4}\text{T},\text{T}$), we provide original images and their corresponding generated images after denoising (for visualization purposes only). Visualizations of augmented features $\mathbf{F}_{\text{aug}}$ extracted from the UNet’s decoder layers ($\mathbf{f}_{4}$ to $\mathbf{f}_{12}$) at the final denoising step are sequentially arranged from top-left to bottom-right.
Refer to caption: /html/2503.23402/assets/x12.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: see above
soundness: 3
strengths: + using LDM for feature extraction instead of generative replay is interesting
+ the proposed method and its usage of different clean and noised LDM features makes sense
+ ablation on the different feature components from the ovearll method
+ easy to read with clear figures
summary: this paper investigates the few-shot class-incremental learning (FSCIL) problem and introduces a diffusion-model-based approach. instead of only using the latent diffusion model (LDM) for generative replays, the proposed method uses LDM for feature extraction, and shows that this LDM-based feature extractor can achieve competitive performance when compared to widely adopted options like DINOv2. by integrating the clean feature, the text-conditioned one-step noised feature, and multi-step noised features, the proposed method achieves good performance on multiple benchmarks.
weaknesses: - the reviewer is not convinced by the claim "DINOv2-G achieves higher initial accuracy due to larger capacity and explicit pre-training on CUB-200" (L188). SD is also trained on web data (most likely including CUB dataset). plus, the proposed method needs at least four forward passes on UNet and one forward pass on VAE encoder, so most like more compute than ViT-G in DINOv2
- following the above point, please consider report the inference FLOPs cost for the overall system and compare with existing methods, since the four forward passes do seem expensive

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: N/A
soundness: 2
strengths: a. This paper is well written and easy to follow.

b. It is interesting to focus on Stable Diffusion in CL.
summary: This paper concentrates on Few-shot class-incremental learning (FSCIL), and proposes Diffusion-FSCIL based on Stable Diffusion (SD). Main contributions are summarized as:

a. Proposing  Diffusion-FSCIL, a framework that fully exploits SD as a unified backbone for FSCIL.

b. Extensive experiments to prove the effectiveness of proposed.
weaknesses: There are three main concerns:

a. I respectfully disagree with the motivation of this work. In section of introduction, it is mentioned that classical generative replay (GR) CL methods have the drawbacks that dependeding on separate discriminative backbones. This is not a drawback, because classical GR methods consider the situation that a customer has its own model (i.e., separate discriminative backbone). Classical GR methods aim to mitigate catastrophic forgetting of this model by adopting a generator. Therefore, this is not a drawback.

b. Instead, i think the proposed method is limited to Stable Diffusion (SD). All operations are based on SD. It is ok to propose a SD-bsed methods. But it doesn't mean that the proposed method deals with the issue of classical GR. Instead, classical GR is general to customer's backbone.

c. I think this paper is lack of technical novelty and the motivation of each operation in the proposed method, such as One-step inversion feature, One-step synthetic feature, is not clear. Why does this work use these operations? What is the motivation?

Other concern:

d. Section 5.4 reports the time cost cross incremental sessions. However, the most time cost occurs at base session.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. What is the performance of model with stable backbone on entire dataset and using similar training used for other models but with backbone of SD.
    
2. Clarification on how F_{gen} is distilled and used is needed. For further details see weakness point.
    
3. For more points refer to weakness
soundness: 2
strengths: 1. Utilizing a generative model as an encoder backbone for class incremental learning is a novel idea that has been proposed by the author.
    
2. There is no extra buffer space required by the model for the previously learned class for replaying the images to avoid catastrophic forgetting.
    
3. Using the latent features directly for replay during training is an interesting approach since the latent features captures semantics of the class rather than focusing on generating images following exact pixel distribution from data.
    
4. The model trained requires approximately 6M parameters.
summary: The paper proposes Diffusion-FSCIL, a novel framework that leverages Stable Diffusion not merely as a generative replay tool but as a unified backbone for few-shot class-incremental learning. Instead of generating and storing synthetic images, the method extracts four types of multi-scale latent features—inversion, synthetic, augmented, and generative—directly from SD’s U-Net to capture complementary semantic and structural information. These features enable efficient replay, generalization, and knowledge retention using lightweight networks (~6M parameters) while keeping SD frozen. Extensive experiments on CUB-200, miniImageNet, and CIFAR-100 benchmarks show that Diffusion-FSCIL outperforms prior FSCIL methods, demonstrating that diffusion models can act as both powerful generative and discriminative foundations for incremental learning
weaknesses: 1. There is a possibility of data leakage since the stable diffusion backbone used by the authors are entirely frozen which has already been trained on very large dataset.
    
2. In line 188, the claim is that DinoV2 is pretrained on CUB dataset, but in DinoV2 paper appendix C table 18 shows that for only for retrieval pretraining that dataset is used and not for pretraining. Which pretrained model did the authors use and if it is not retrieval pretrained then this claim doesn’t hold true and that raises the question on the claim made by the authors.
    
3. In Pilot Study, what is the approach used for training other models in continual setup besides Stable Diffusion Backbone. What parameters of the other models were tuned and how?
    
4. F_{gen} is not present during the Session 0 and introduced from session 1 (line 321) then how is it able to capture features from the classes trained in Session 0 since there is no network to distill from or use weights from. This is also highlighted in algorithm presented in Appendix where in base training F_{gen} is not present.
    
5. For each instance of data, forward pass generates the latent from classes for replay (F_{gen}), how is the class chosen from previous sessions for generating latent. Also, does it cover every single class that has been seen during the previous sessions?
    
6. Is there a particular reason why in CUB-200 there is severe degradation in other SD based models compared to the proposed approach but on miniImagenet or CIFAR 10, the degradation in performance from Session 0 to final session is almost similar.
    
7. For reproducibility, in line 318 what is the number of epochs that the F_{aug} is trained for.
    
8. For reproducibility, how is single label combination is done. What is the prompt or technique used by the authors to combine the labels into single one as claimed in line 745 of Appendix B.
    
9. The claim made in line 750, talks about “By employing single-word embeddings, we ensure that the semantic concept of each label is captured consistently and efficiently eliminating redundancy or semantic dilution that occurs when multi-word labels are split across multiple token.” Is there any experiment or results backing this support since it is dependent on tokenization process? For eg. in gpt-4-32k tokenizer “ring billed gull” uses 4 tokens, “Ring_billed_Gull” (used by the authors) uses 5 token and “RingBilledGull” also uses 5 tokens.
    
10. Some notational clarification needs to be done for eg. In line 146 S is used for denoting total number of session, instance of each incremental session.
    
11. \beta_{l} is used as weighing for each of the aggregation module. Are the weights being shared by each feature type or separate for each feature type.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Why does Stable Diffusion (SD) retain knowledge better than discriminative models like DINOv2 or CLIP in incremental sessions?
2. Why use four types of features instead of learning a single unified representation?
soundness: 3
strengths: 1.The idea of using SD as a frozen feature extractor instead of a generative replay buffer is original and well-motivated.
2. The use of multi-scale U-Net features from SD is technically sound and well-justified. The class-specific prompt optimization and controlled noise injection strategies are clever and effective.
3. The paper is well-written, with clear structure and intuitive illustrations.
summary: This paper introduces Diffusion-FSCIL, a novel few-shot class-incremental learning (FSCIL) framework that leverages Stable Diffusion (SD) as a unified frozen backbone for feature extraction. Unlike prior works that use SD solely for generating synthetic replay images, this approach extracts four complementary types of features (inversion, synthetic, augmented, and generative) directly from the diffusion process in latent space. The method achieves state-of-the-art performance on CUB-200, miniImageNet, and CIFAR-100, while maintaining efficiency with only ~6M trainable parameters.
weaknesses: 1. While the empirical results are strong, the paper lacks theoretical analysis on why  SD features are more robust to forgetting compared to discriminative models like DINOv2.
2. All experiments are conducted on small-scale image classification datasets.
3. There is no discussion on how the method would adapt to other generative models (e.g., DiT, CLIP-guided diffusion, autoregressive models).


Decision: Reject