ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion
Authors: Yingjun Du
Zehao Xiao
Shengcai Liao
Cees G. M. Snoek
Venue: NeurIPS 2023 poster (2023)
Keywords: Meta-learning
few-shot learning
diffusion model
prototype
TL;DR: We introduce ProtoDiff, a novel framework that leverages a task-guided diffusion model during the meta-training phase to gradually generate prototypes, thereby providing efficient class representations.

Abstract: Prototype-based meta-learning has emerged as a powerful technique for addressing few-shot learning challenges. However, estimating a deterministic prototype using a simple average function from a limited number of examples remains a fragile process. To overcome this limitation, we introduce ProtoDiff, a novel framework that leverages a task-guided diffusion model during the meta-training phase to gradually generate prototypes, thereby providing efficient class representations. Specifically,  a set of prototypes is optimized to achieve per-task prototype overfitting, enabling accurately obtaining the overfitted prototypes for individual tasks.
Furthermore, we introduce a task-guided diffusion process within the prototype space, enabling the meta-learning of a generative process that transitions from a vanilla prototype to an overfitted prototype. ProtoDiff gradually generates task-specific prototypes from random noise during the meta-test stage, conditioned on the limited samples available for the new task. Furthermore, to expedite training and enhance ProtoDiff's performance, we propose the utilization of residual prototype learning, which leverages the sparsity of the residual prototype. We conduct thorough ablation studies to demonstrate its ability to accurately capture the underlying prototype distribution and enhance generalization. The new state-of-the-art performance on within-domain, cross-domain, and few-task few-shot classiﬁcation further substantiates the beneﬁt of ProtoDiff.

ProtoDiff: Learning to Learn Prototypical Networks by Task-Guided Diffusion
Yingjun Du', Zehao Xiao', Shencai Liao², Cees G. M. Snoek 'AIM Lab, University of Amsterdam PInception Institute of Artificial Intelligence
Abstract
Prototype-based meta-learning has emerged powerful technique for addressing few-shot learning challenges. However, estimating a deterministic prototype using simple average function from a limited number of examples remains fragile process. To overcome this limitation, we introduce ProtoDiff, a novel framework that leverages a task-guided diffusion model during the meta-traine veta-training phase gradually generate prototypes, thereby providing efficient class representations. Specifically, set of prototypes is optimized to achieve per-task prototype overfit- ting, enabling accurately obtaining the overfitted prototypes for individual tasks. Furthermore, we introduce a tusk-guided diffusion process within the prototype space, enabling the meta-learning of generative process that transitions from a vanilla prototype an overfitted prototype. ProtoDiff gradually generates task- specific prototypes from random noise during the meta-test stage, conditioned on the limited samples available for the new task. Furthermore, to expedite training and enhance ProtoDiff's s performance, type learning, which leverages the sparsity of the residual prototype. We conduct thorough ablation studies to demonstrate its ability accurately capture the under- lying prototype distribution and enhance generalization. The new state-of-the-art performance within-domain, cross-domain, and few-task few-shot classification further substantiates the benefit ProtoDiff.
propose the utilization of residual proto-
Introduction
This paper considers prototype-based meta-learning, where models are trained swiftly adapt to current tasks and perform classification through metric-based comparisons between examples and newly introduced variables the prototypes of the classes. This approach, rooted in works by Reed [34] and further developed by Snell el al. [42], generalizes deep learning models to scenarios where labeled data is scarce. The fundamental idea is to compute the distances between queried examples and class prototypes and perform classification based on these distances to predict classes for queried examples. Derived from the prototypical network [42], several prototype- based methods have demonstrated their effectiveness in few-shot learning [2, 8, 11, 61]. However, prototypes are estimated from a limited number of sampled examples, they may not accurately capture the overall distribution [54, 61]. Such biased distributions could lead to biased prototypes and subsequent classification errors, suggesting that the current prototype modeling approach might lack the ability to represent universal class-level information effectively. Rather than relying on points in the prototype embedding space by using simple average function, we propose to generate the distribution of prototypes for each task.
To overcome the challenges of estimating deterministic prototypes from limited examples in few-shot learning, propose PiftoDiff, innovative framework that leverages task-guided diffusion model during the meta-training phase. ProtoDiff introduces a three-step process to address these limitations comprehensively. Firstly, optimize set of prototypes to achieve per-task prototype overfitting,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

po(# 1
Bulldag
g( 24-1)
obearved during meta-trinning ony
observed during meta-training
process
Figure 1: Computational graph of ProtoDiff. In the initial stage, the computation of the vanilla prototype 2 is performed based the support set S'. Subsequently, through the process of diffusion sampling, the diffused prototype Zz-1 is derived from the combination of z and Finally, the prediction y for the query set Q' generated by utilizing the diffused prototype Zo in conjunction with the query set x. The diffusion forward and sampling processes are indicated by dashed and solid arrows within the gray rectangular area.
ensuring accurate retrieval of overfitted prototypes specific to individual tasks. These overfitted prototypes serve the ground truth representations for each task. Secondly, task-guided diffusion process is implemented within the prototype space, enabling meta-leamnin of a generative process that smoothly transitions prototypes from their vanilla form to the overfitted state. ProtoDiff generates task-specific prototypes during the meta-test stage by conditioning random noise on the limited samples available for the given new task. Finally, we propose residual prototype learning, which significantly accelcrates training and further enhances the performance of ProtoDiff by leveraging the sparsity of residual prototypes. The computational graph of ProtoDiff, illustrated Figure I, showcases the sequential steps involved in the forward diffusion process on the prototype and the generative diffusion process. The resulting generated prototype, denoted as Zy. is utilized for query prediction. By incorporating probabilistic and task sosk-gddddd prototypes, ProtoDiff balances adaptability and informativeness, positioning it as a promising approach for augmenting few-shot learning in prototype- based meta-learning models.
To validate the effectiveness of ProtoDiff, we conduct comprehensive experiments on three distinet few-shot learning scenarios: within-domani cross-domain, and few-task few-shot learning. Our findings reveal that ProtoDiff significantly outperforms state-of-the-art prototype-basee meta-learning models, underscoring the potential of task-guided diffusion to boost few-shot learning performance. Furthermore, we provide detailed analysis of the diffusion mechanism employed in ProtoDiff, showcasing its ability to capture the underlying data structure better and improve generalization. This thorough investigation highlights the strengths of our approach, demonstrating its potential to offer a more effective solution for few-shot learning tasks in various applications and settings.
Preliminaries
Before detailing our ProtoDiff methodology, we first present the relevant background on few-shot classification, the prototypical network, and diffusion models.
Few-shot classification. We define the N-way K-shot classification problem, which consists of support sets S and query set Q. Each task T³, also known as an episode, represents a classification problem sampled from task distribution p(T). The way of an episode denotes the number of classes within the support sets, while the shot refers to the number of examples per class. Tasks are created from dataset by randomly selecting a subset of classes, sampling points from these classes, and subsequently dividing the points into support and query sets. The episodic optimization approach [50] trains the model iteratively, performing one episode update a time.
Prototypical network. We develop our method based on the prototypical network (ProtoNet) by Snell et al. [42]. Specifically, the ProtoNet leverages non- parammrrettic classifier that assigns query point to the class having the nearest prototype in the learned embedding space. The prototype z" of object class is obtained by: z === Σ. (x(+x)), where (x(*)) is the feature embedding of the support sample x which is usually obtained by convolutional neural network. For each query sample x", the distribution over classes is calculated based on the softmax over distances to the prototypes of all classes in the embedding space:
exp(-d(foox9))
ply = c(x9) = E. exp( hop(-d(foor d(fo(x)) 2²²)))
(1)

---

where y" denotes a random one-hot vector, with y indicating its n-th element, and d(..-) is some (Euclidean) distance function. Due its non-parametric nature, the ProtoNet enjoys high flexibility and efficiency, achieving considerable success in few-shot learning. To avoid confusion, we omit the superscript e for the prototype z in this subsection.
Diftusion model. In denoising diffusion probabilistic models [16], a forward diffusion process, q(x,{22;-1) is characterized as Markov chain that progressively introduces Gaussian noise each time step 6, beginning with clean image wo ~ q( The subsequent forward diffusion process is formulated as follows:
( = II (x) where (xx//=1-1) == Naxx v -1
where {:3)T- is variance schedule. By defining a;:=1 - and o;: II sample time step t, denoted as æt, can be generated in single step as follows:
== Oxs the forward diffused
I Vā + VI where E~
Since the reverse of the forward step, ((x,-1/xx) is computationally infeasible, the model learns to maximize the variational lower bound using parameterized Gaussian transitions, po (ate (ax)), where the parameter is denoted as 6. Consequently, the reverse process is approximated as a Markov chain with the learned mean and fixed variance, starting from a random noise FT ~ N (ar: : 0, I):
(a po (o II po 1
(4)
where
po(#:-112z;) == Ni I), (o( = 1 (m.
1 - Ote
co(2;; 1))
(5)
Va, VI
Here, Ea( t) the diffusion model trained by optimizing the following objective function:
Co Et. [|ce col + v1 a,c, t)]
(6)
Upon completing the optimization, the learned score function integrated into the generative (or reverse) diffusion process. To sample from pe pe(#t-1/z3) [x,), one can perform the following:
2--1 1 100 (2 t) + o,e
(o(re) + ore.
Vat
VI az.
In the case of conditional diffusion models [22, 38], the diffusion model eg in equations (6) and (7) is substituted with eg (c,,, Vā + vI agC, t), where represents the corresponding conditions, e.g., other images, languages, and sounds, etc. Consequently, the matched conditions strictly regulate the sample generation in supervised manner, ensuring minimal changes the image content. Dhariwal and Nichol [7] suggested classifier-guided image translation to generate the images of the specific classes along with pre-trained classifier. Taking inspiration from conditional diffusion models that excel at generating specific images using additional information, we introduce the task- guided diffusion model that generates prototypes of specific classes by taking into account contextual information from various few-shot tasks. Note that our task-gune mask-guided diffusion model operates not on an image æ, but on the prototype
3 Methodology
This section outlines our approach to training prototypical networks via task-guided diffusion. We begin by explaining how to obtain task-specific overfitted prototypes Section 3.1. Next, we introduce the task-guided diffusion method for obtaining diffused prototypes in Section 3.2. In the
same section, we also introduce residual prototype learning, which accelerates training. Figure visualizes the diffusion process of diffused prototypes by our ProtoDiff.
3

---

Diftusion process
vasis
21
p90b-0-12
brob=847
1ro8-0.66
probcc
Figure Visualization the diffusion process. ProtoDiff randomly selects certain areas to predict during the diffusion process, with the lowest probability at the beginning time step. As time progresses, the prototype gradually aggregates towards the dog, with the highest probability at t=0.
Postional encoding
Figure 3: ProtoDiff illustration. We first obtain an overfitted proto- type by the per-task prototype over- fitting. We then noise overfitted prototype, which inputs the diffusion forward process. The input of task-guide diffusion in- cludes the vanilla prototype and a random time step f. The resulting output is the diffused prototype z;.
Task Bak-guidl Culdted ai Moosa
Zt
Per-task
Colfoie
everfitting
Support set
Query image
Protstypes:
Varilla
Ovarlited
Diflused
Nkised
Time slep:
3.1 Per-task prototype overfitting
Our approach utilizes modified diffusion process as its foundation. Firstly, a meta-learner, denoted by fo(T")=/(T)) Φ), is trained on the complete set of tasks, T. Here, øi corresponds to the learned weights of the model over the entire task set. Subsequently, fine-tuning is performed on each individual task T to obtain task-specific overfitted prototypes denoted by z ===== (S*, Q¹). This involves running the meta-learner fo on the support set S and query set Q' the task T
Ti Σ (x.y)-T
LCE(J(S) x²" ))).")
T
where Lce is the cross-entropy loss minimized in the meta-learner training. The support set St and query set Q'=(x"', ,y"' correspond the data used for fine-tuning task T During fine-tuning, we obtain the o' through several iterations of gradient descent. We illustrate the overall framework of the per-task prototype learning in the appendix.
Our proposed method, PiftoDiff, for few-shot learning, relies on acquiring task-specific overfitted prototypes that be considered as the "optima" prototypes due their high confidence in final predictions, approaching value of 1. To achieve this, we employ fine-tuning of the meta-learner and extract task-specific information that surpasses the accuracy and reliability of generic prototypes used in the meta-training stage. However, accessing the query set in the meta-test stage is not feasible. Hence, we need rely on the vamilla prototype to meta-learn the process of generating the overfitted prototype during the meta-traine training stage. In the forthcoming section, we will introduce the task -uided diffusion model, which facilitates the learning process transitioning from the vanilla prototype to the overfitted prototype.
3.2 Task-guided diffusion
Our generative overfitted prototype is based diffusion [43], robust framework for modeling the prototypes instead images. The length of the diffusion process determines the number of forward passes needed to generate new prototype, rather than the dimensionality of the data. Our model uses diffusion to progressively denoise the overfitted prototype zz.
Meta-training phase. Diffusion models can be configured to predict either the signal or the noise when presented with a noisy input [16,31) Previous research the image domain has suggested that noise prediction is superior to signal prediction. However, we discovered empirically that signal prediction performs better than noise prediction in our experiments, so we parameterized the diffusion model output the prototype.

---

By N(01) BT-1
Zo
Figure Diffusion sam- pling during meta-test. Sam- pling starts from random noise zr and gradually de- noises it the diffused proto- types Xt. At each time step t is sampled by taking the Zt vanilla prototypes 2, and time step inputs. The diffused prototypes zo are used pre- dict the query set.
Task- Task-guided -uided
Task- bask-guided diffusion
Task fask-guided -uli-guiddddddd diffusion
Bird
Query image
Support set
T
t
T- 1
t
0
Random nolsa
Varilla prototypes
Difuuee prototypes
Tima stop
The generative diffusion process is devised to reconstruct the overfitted prototype z" by iteratively operating on a random noise vector Zr ~ (0, I). which possesses the same dimensions z². This process continues with 24, where i. denotes the number of diffusion steps. Consequently, the reconstructed Zo should exhibit proximity to z" for given task.
Specifically, during the forward diffusion process at time step t, we obtain the noised overfitted prototype 2, Subsequently, we input the noised prototype 2,, the vanilla prototype 2, and the time step / into the task-guided diffusion. This yields the diffused prototype &z, which then allows us predict the final results of the query using Equation ()). For each task, our task-guided diffusion entails two components: the variational lower bound Lair for the diffused prototype, and the cross-entropy loss LCE.
The objective is to minimize the simplified variational lower bound, which involves predicting the denoised overfitted prototype:
Lai =1= zo(Vā,z + ārc, t)
(9)
Here, ze(:;) is implemented by the transformer model [49] operating on prototype tokens from z". 2, and the time step.
By utilizing equation (I), we derive the final prediction ÿa using the diffused prototype z,. The ultimate objective is expressed as follows:
IQI Σ (
L
Z441, (ogg()))") 2r]] + -lzz" - zo(Vaiz + vI - -ā,e ž, t.
(10)
where 3 represents hyperparameter, and [Q] denotes the query size.
To prepare the two input prototypes x² and ă for processing by the transformer, we assign each position token inspired by [10]. We also provide the scalar input diffusion timestep as individual tokens to the transformer. To represent each scalar as a vector, we use a frequency-
encoding scheme [28]. Our transformer architecture is based on GPT-2 [33]. The decoder in the final layer of our ProtoDiff maps the transformer's ' output to the diffused prototype. Note that only the output tokens for the noised overfitted prototypes 2, decoded to predictions. The overall meta-training phase of Pift is shown in Figure 3.
Meta-test phase. We can not obtain the overfitted prototype during the meta-test phase since we can not access the query set. Thus, diffusion sampling begins by feeding-in Gaussian noise 2r as the 2 input and gradually denoising it. Specifically, to handle new task 7= (S, 2}. we first compute the vanilla prototype à using the support set S. We then randomly sample noise from (O, I). We input both žand to the learned task-guided diffusion model to obtain the diffused prototype ZT Zp 2, T). After T' iterations, the final diffused prototype can be obtained as Zo 20 z, to). Once the final diffused prototype Zu is obtained,
calculate the final prediction 5 using equation (1). Figure 4 illustrates sampling in the meta-test stage. We also provide the detailed algorithm of the meta-training and meta-test phase in the appendix.
Residual prototype learning. To further enhance the performance of ProtoDiff, we introduce a residual prototype learning mechanism in our framework. We observe that the differences between the overfitted prototype z" and vanilla prototype are not significant, as the vector z" contains many zeros. Therefore, propose predict the prototype update z - instead of directly predicting
5

---

the overfitted prototype z' itself. This approach also enables us to initialize ProtoDiff to perform the identity function by setting the decoder weights zero. Moreover, we find that the global residual connection, combined with the identity initialization, significantly speeds up training. By utilizing this mechanism, we improve the performance of ProtoDiff in few-shot learning tasks.
4 Related Work
Prototype-based meta-learning. Prototype-based meta-learning is based on distance metrics and generally learns a shared or adaptive embedding space in which query images are accurately matched to support images for classification. It relies on the assumption that a common metric space is shared across related tasks and usually does not employ an explicit base learner for each task. By extending the matching network [50] to few-shot scenarios, Snell et al. [42] constructed prototype for each class by averaging the feature representations of samples from the class in the metric space. The classification matches the query samples prototypes by computing their distances. To enhance the prototype representation, Allen et al [2] proposed an infinite mixture of prototypes to adaptively represent data distributions for each class, using multiple clusters instead of a single vector. Oreshkin et al [32] proposed a task-dependent adaptive metric for few-shot learning and established prototype classes conditioned on task representation encoded by a task embedding network. Yoon et al [60] proposed a few-shot learning algorithm aided by a linear transformer that performs task-specific null-space projection of the network output. Graphical neural petwork-based models generalize the matching methods by learning the message propagation from the support set and transferring to the query set [13]. FEAT [58] was proposed to leverages samples from all categories within a task to generate prototype, capitalizing on the intrinsic inter-class information to derive a more discriminative prototype. In contrast, our ProtoDiff, while using the overfitted prototype supervision, only employs samples from single category to generate the new prototype. This means we are not tapping into the potential informative context provided by samples from other categories. Additionally, our ProtoDiff employs the diffusion model to progressively generate the prototype, whereas FEAT does not utilize any generative model for prototype creation. Prototype- based methods have recently been improved in various ways [4, 46, 58, 61]. To the best of our knowledge, we are the first to propose a diffusion method to generate the prototype per task, rather than deterministic function.
Diffusion models. These models belong to a category of neural generative models that utilize stochastic diffusion processes [43, 44], much like those found in thermodynamics. In this framework, a gradual introduction of noise applied a sample of data. neural model then learns to reverse the process by progressively removing noise from the sample. The model denoises initial pure noise to obtain samples from the learned data distribution. Ho et al. [16] and Song el al. [44] contributed to advancements in image generation, while Dhariwal and Nichol [7] introduced classifier- llassifier-guided diffusion for a conditioned generation. GLIDE later adapted this approach|
[31], enabling conditioning on textual CLIP representations. Classifier-free guidance [17] allows for conditioning with balance between fidelity and diversity, resulting in improved performance [31]. Since the guided diffusion model requires large number of image-annotation pairs for training. Hu et al. [19] propose self- guided diffusion models. Recently, Hyperdiffusion [12, 27] was proposed in the weight space for generating implicit neural fields and 3D reconstruction. In this paper, we introduce ProtoDiff, prototype-based meta-learning approach within a task-guided diffusion model that incrementally improves the prototype's expressiveness by utilizing a limited number of samples.
5 Experiments
In this section, we assess the efficacy of ProtoDiff in the context of three distinct few-shot learning scenarios: within-domain few-shot learning, cross-domain few-shot learning, and few-task few-shot learning. For the within-domain few-shot learning experiments, we apply our method to three specific datasets: minilmagenet [50], tiered/magenet [35], and ImageNet-800 [5]. Regarding cross- domain few-shot learning, we utilize minilmagenet [50] as the training domain, while testing is
conducted on four distinct domains: CropDisease [30], EuroSAT [15], ISIC2018 [47], and ChestX [52]. Furthermore, few-task few-shot learning [56] challenges the common assumption of having abundant tasks available during meta-training. To explore this scenario, we perform experiments on
four faw- meta- meta-lcarning challenges: minilmagenet-S [50], ISIC| [29]. DermNet-S [6], and Tabular

---

Table 1: Benefit of ProtoDiff on mindlmagenet, tieredImagenet and ImageNet-800. The results of the Classifier- Baseline and Seta--aaelline are provided by Chen et al. [5]. ProtoDiff consistently achieves better performance than two baselines on datasets and settings.
minilmagenet
fieredImagenet
Imagenet-800
Method
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Classificr- Siffferr-asllinee [S]
58.91 1:025
77.76-00.7
68. 09:227::.2
13.74
86.07=021
96. 14:0008
[5]
63. 17 20:23
79.26-t0.17
68 62.222
83.74=2.18
89.70=z.1
96. 10.000
ProtoDiff
66.63 10:21
83.48 10.15
72.95 :0024
85.15 :0:18
92.13 0020
98.21 :0008
Vanile
VAE
Normalizing flows
ProteGift wib resiciall
ProtoDifi residuul
Orarlmed
Prote8:33
Probe0.43
Probwo.52
Probed C1
Probeo
Proboo 3D
Figure 5: Visualization of the different reconstructed overfitted prototypes using different generative models. The vanilla prototype focuses only on the overall features of the bird and overlooks the finer details of the robin. In contrast, the overfitted prototype can highlight specific features such as the tail and beak. While VAE and normalizing flow can generate prototypes of certain parts, the diffusion method can generate prototypes that more closely resemble the overfitted prototype. With residual prototype learning, PiftoDiff achieves better.
Murris [3]. For a comprehensive understanding of the datasets used in each setting, we provide detailed descriptions in the Appendix.
Implementation details. In our within-domain experiments, we utilize a Conv-4 and ResNet- 12 backbone for minilmagenet and fieredlmagenet. A ResNet-50 is used for ImageNet-800. We follow the approach described in [5] to achieve better performance and initially train a feature extractor on all the meta-training data without episodic training. Standard data augmentation techniques are applied, including random resized crop and horizontal flip. For our cross-domain experiments, we use ResNet- 10 backbone extract image features, which is a common choice for cross-domain few-shot classification [14, 57]. The training configuration for this experiment is the same as the within-domail training. For few-task few-shot learning, we follow [56] using a network containing four convolutional blocks and a classifier layer. The average within -oomiin cross-domain, few-task few-shot classification accuracy (%, top-1) along with 95% confidence intervals are reported across all test query sets and tasks. Code available at: https: / //gi thub. com/
www.ccoom////.// /D////DUUU-// uva--va///e wwwpwaaaa/aaaaaie ProtoDiff.
Benefit of ProtoDiff. Table compares our ProtoDiff with two baselines demonstrate its effec- tiveness. The first Classifier- is a classification model trained on the entire label set using classification loss and performing few-shot tasks with the cosine nearest-centroid method. The Meta-Baseline [5] consists of two stages. In the first stage, a classifier is trained on all base classes, and the last fully connected layer is removed to obtain the feature encoder. The second stage is meta-learning, where the classifier is optimized using episodic training. The comparison between the Baseline and Meta-Baseline highlights the importance of episodic training for few-shot learning. PrftoDiff consistently outperforms Meta-Baseling [5] by a large margin on all datasets and shots. The task lask- diffusion model employed in our ProtoDiff generates more informative prototypes. leading to improvements over deterministic prototypes.
Benefit of diffusion model. To confirm that the performance gain of our ProtoDiff model can be attributed to the diffusion model, we conducted the experiments only using MLP and transformers as -enerrenreaeve models. We also compared it with two widely used generative models: the variational autoencoder (VAE) [21] and normalizing flows [36]. VAE learns a low- dimensional representation of input data to generate new samples. Normalizing flows are more recent and learn a sequence of invert- ible transformations to map simple prior distribution to the data distribution. We ob- tained the task-specific prototype in the meta- test stage, conditioned on the support samples and (0, I), by first acquiring an overfit-
Table 2: Benefit of diffusion model over (non-)generative models on minilmagenet.
minilmagenet
Method
1-shot
5-shot
w/o generative model [$]
63.17.1023
79.26 10:17
w/ MLP
64,15 15021
80.23
w/ Transformer
64.97+021
+1.28+0.14
w/ VAE
14.2510022
$0.1310.1
w/ Normalizing flows
65.11 1:.22
81.9610.17
w/ Diffusion
66.63 :0.2
83,48 60.15
7

---

ted prototype z' and then using it as the ground truth to train VAE and normalizing flows in the meta-training stage. The experimental results reported in Table 2 show that the diffusion model outperforms all variants in terms of accuracy. Specifically, our diffusion model improves accuracy by 18% compared to VAE and 1.52% compared to normalizing flows. Furthermore, we reconstructed the overfitted prototype using different generative models in Figure 5. The vanilla prototype focuses solely on the bird's features and overlooks the finer details of the robin. In contrast, the overfitted prototype can emphasize specific features such as the tail and beak, resulting in better discrimination of the robin from other bird species. While VAE and normalizing flows can generate prototypes of specific parts, our diffusion method can generate prototypes that more closely resemble the overfitted prototype, leading to improved performance. Our findings suggest that diffusion models hold promise as more practical approach for few-shot learning, they can model complex distributions and produce informative prototypes.
Effect of the residual prototype. The incorporation of residual prototypes in ProtoDiff presents a viable approach for reducing computational costs. This is attributable to the fact that the residual
prototype exclusively encapsulates the disparity be- tween the overfitted prototype and the original pro- totype, while the latter can be readily reused in each episode. Consequently, the calculation of the over- fitted prototype is only necessitated once during the meta- meta-training stage. Notably, the residual prototype often encompasses numerous zero values, thereby further expediting the training process. Table 6a il- lustrates the superior accuracy achieved by PiftoDiff with residual prototype learning in comparison to both the vanilla model and ProtoDiff without the residual prototype. The training progress of ProtoDiff is visually represented in Figure 6b, demonstrating its accelerated training capabilities when compared to alternative methods. These results indicate the effec- tiveness of residual prototype learning in capturing task- -secciccc information previously unaccounted for by the vanilla prototypes. Thus, the inclusion of resid- ual prototype learning in ProtoDiff not only expedites the training process but also serves as a straightfor- ward and effective approach to enhance the overall performance of ProtoDiff.
(a) Accuracy (%) on various settings.
miniImagennt
Method
1-shot
5-shot
Vanilla
63.17 64.75 66.63 2031
ProtoDiff w/o residual
80.76
ProtoDiff w/ residual
83.48 15
Posadlifi wi costall
Pronttif wis miitrll
10
10a
6 10 16 20
25
Eough
(b) Optimization curves.
Figure 6: Effect of residual prototype on minilmagenet. Utilizing ProtoDiff with resid- ual prototype not only accelerates the training pro- cess but also surpasses the performance of both the vanilla prototype and the non-residual protolype.
Analysis of uncertainty. In the process of estimating the likeliiid of input data, the choice of the number of Monte Carlo samples becomes a critical hyperparameter. Typically, a larger number of samples tends to provide a more accurate approximation and better generalization to unforeseen prompts. In our experiments, we employed ProtoDiff, incorporating the sampling of multiple prototypes in the final phase of the diffusion process. The results of these ex-
and FEAT [58] with
Table 3: Analysis of uncertainty. Increasing the number of Monte Carlo samples provides good improvement on minilmagenet.
periments are provided in Table 3.
Notably, we observe substan- tial performance boost with an in- crease in the number of samples. For instance, a sample size of 50 results in significant enhance- ment in I-shot accuracy, raising it from 66.63 for + ProtoDiff to 72.25 for FEAT + ProtoDiff.
Meta- baseline + ProtoDiff
FEAT + ProtoDiff
1-shot
5-shot
I-shot
5-shot
1
66.63=n21
83.48
85.16 -8.97=025
10
68.02-0.17
84.95
70. 10
86
20
68.9120.1
85.74
61.07=015
17
50
69. 10.11
86.12.10.10
72.25
88.32::012
100
69.13
86.07+0.10
72.21
88.37=0.12
1000
1
86.11
12
18.13-0012
Visualization of the prototype adaptation process. We undertake a qualitative analysis of the adaptation process of our ProtoDiff framework during the meta-test time. We employ 3-way 5-shot setup on the miniImageeee dataset for more comprehensive visualization of the adaptation process. To achieve this, depict the vanilla prototypes, the diffused prototypes at various timesteps, and instances from the support and query sets. All examples are first normalized to a length of I and
8

---

Table 5: Cross-domain few-shot comparison on four benchmarks. ProtoDiff is consistent top-performer
CropDiseases
EuroSAT
ISIC
ChestX
Method
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
ProtoNet [42]
57.57 #0.5
79.72
54.19
73.29
29.62
39.57=0.5
22.30
24.05
GNN [10]
57.19
83.12=0.0
50
78.69
30.14 00:30
42.54=0.0
21.94=0:00
23.87=020
AFA [2]]
67.61 20:50
88.06=030
63.12:050
85.58
33.21
46.01 =0:40
22.92
25.02=:030
ATA [5]]
67.47 40.50
90,59
61.35
83.75=0.40
33,21
44,91 =0.40
22.10=00.0
24.32
HVM
[1]]
65.13
40.45
87.65=0.31
61.97=00.4
74.88
33.87
42.05
22.94:0.7
27.15 +0.45
This paper
68.93 10.31
90.15 8.031
65,93 10.34
87.25
34,97 10.33
45.65 L0.31
23.01 10:45
28.54 10.4
subsequently projected onto two-dimensional space using t-SNE [48]. Figure 7 displays the instance representations from both the query and support sets that are cross and circle, the vanilla prototypes that are denoted by four-point star symbol, an the diffused prototypes that are de- noted by stars. Arrows depict the adaptation pro- cess of the diffused prototypes, and each color corresponds each respective class. Initially, the diffused prototypes are randomly allocated based on a Gaussian distribution, positioning them considerably from the optimal prototypes for each class. As the diffusion process contin- ues, these diffused prototypes incrementally approach the optimal prototypes, resulting in more distinctive representations for different classes. In contrast, the vanilla prototypes lack distinguisha- bility as they are computed using a single deterministic function based solely on the support set. This observation underscores the importance of the prototype adaptation process attaining improved performance in few-shot classification tasks.
-2
Qany proteppps
Tooss probegpes
Orkost probagaos
-2
Figure 7: t-SNE visualization of ProtoDiff adaptation process on 3-way 5-shot task.
Within-domain few-shot. We evaluate our method on few-shot classification within domains, in which the training domain is consistent with the test domain. The results are reported in Table 4. In this comparison, we apply Set- Feat [1] with ProtoDiff to experiment since SetFeat is the current otate-of- the-art model based on prototype- based meta-learning. Our ProtoDiff consistently achieves state-of-the-- -rrt performance on all datasets under var- ious shots. The better performance confirms that ProtoDiff can achieve more informative and higher quality prototypes to perform better for few- shot learning within domains.
Table 4: Within-domain few-shot comparison on minilmagened and rieredImagenet. ProtoDiff performs well on all datasets.
minilmaaenntt
féere/Imagenet
Method
1-shot
5-shot
1-shot
5-sbot
TapNer [59]
61.65=011
76.36=0010
63.08-011
80.26=0:12
CTM [26]
62.05
78.63
64.78.n11
81.05-003
MetaOptNett [2]]
62.64
78.63
65.81 _07
81.75
Meta- Buselin [S]
63.17 bozz
79.26.u.17
68.62 :0.27
83.74:0a
CAN [1S]
63.85 Loves
79.44.1m
69.89
84.23_00
Meia DeepBDC [S3]
67.34:040
84.46.1.22
72. 72.34:0.44 :0.4
87.31
SUN [.]
67, 85
83,25
72.99=0.50
86.74
SctFeat [i]
68. 60
82.71 20.40
73.63-055
87.59=037
This paper
71.25
83.95
75.97
88.75
Cross-domain few-shot. We also evaluate our method in the cross- few-shot classification, where the training domain is different from test domain. Table 5 shows the evaluation of ProtoDiff on four datasets, with 5-way --shot and 5-way 5-shot settings. ProtoDiff also achieves competitive performance on all four cross-domain few-shot learning benchmarks for each setting. On FuroSAT [15], our model obtains high recognition accuracy under different shot configurations, outperforming the second best method, AFA [20], by significant margin of 2.82% for 5-way 1-shot. Even on the most challenging ChestX [52], which has a considerable domain gap with minilmageNet,, our model achieves 28. 08.54% accuracy in the 5-way 5-shot setting, surpassing the second-best HVM [[]] by 1.39%. The consistent improvement across all benchmarks and settings confirms the effectiveness of PrftoDiff for coms-domain few-shot learning.
Few-task few-shot. We evaluate ProtoDiff on the four different datasets under 5-way 1-shot and 5-way 5-shot in Table 6. In this comparison, we apply MLTI [56] with ProtoDiff to experiment. Our method achieves state-of-the-art performance on all four few-task meta-learning benchmarks under each setting. On minilmagenet-S, our model achieves 44.15% under 1-shot, surpassing the

---

Table 6: Few-task few-shot comparison on four benchmarks. ProtoDiff consistently ranks among the top- performing models.
minlImagenet-S
ISIC
Dermnet-S
Tabular Murris
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
I-shot
5-shot
ProtoNet [42]
36.26=0.00
50.72:0.00
58.56
66,25 2000
44.21 2:775
60.33
80.03
89.20 1:066
Meta- Dropour [23]
38.32_017
52.53
58.40
67.32.100
44.30
60.86 000.66.775
78.18_0.
89, 89.25...0 10.5
MetaMix [55]
39.80-087
53.35
59.66
68.97::0.0
46.06:085
62.97:070
79.56=091
88.88
Meta Interpolation [25]
40.28
53.06
MLTI [S6]
41.36-0073
55.34-002
62.82
+1.52+00.8
49.38
65.19.0.7
81.89
90.12
MetuModulation [45]
-3.21
57.26
65.61
76.40
50.45
67.05
83.13
91.23
This paper
44.75 0.70
58.18
0072
66.13 1.04
76.23 80.81
51.53 :0:5
67.97
84.03 0.80
91.35 80.57
best method MetaModulation [45], by a margin of 1.54%. Even on the most challenging DermNet- S, which forms the largest dermatology dataset, our model delivers 51.53% the 5-way 1-shot setting. The consistent improvements on all benchmarks under various configurations confirm that approach is also effective for few-task meta-learning.
Limitations. Naturally, our proposal also comes with limitations. Firstly, the diffusion model employed in our approach necessitates substantial number of timesteps to sample the diffused pro- totype during the meta-test stage. Although we somewhat alleviate this issue by utilizing DDIM [44] sampling. it still requires more computational resources than vanilla prototypes. Secondly, obtain- ing the overfitted prototype involves fine-tuning, which inevitably leads to increased training time. While this fine-tuning step contributes to the effectiveness of our model, it comes with an additional computational cost. In meta-training and meta-testing ProtoDiff is slower by factors of ProtoNet 5x and 15x in terms of wall-clock times per task. More detailed time numbers are provided in the Appendix. As part of future work, we will investigate and address these limitations to further enhance the applicability and efficiency of our approach.
Conclusion
We proposed ProtoDiff, novel approach to prototype-based meta-learning that utilizes task- guided diffusion model during the meta-training phase to generate efficient class representations. We addressed the limitations of estimating deterministic prototype from limited number of examples by optimizing a set of prototypes to accurately represent individual tasks and training a task-guided diffusion process model each task's underlying distribution. Our approach considers both probabilistic and task-guided prototypes, enabling efficient adaptation to new tasks while maintaining the informativeness and scalability of prototypical networks. Extensive experiments on three distinct few-shot learning scenarios: within-domain, cross-domain, and few-task few-shot learning validated the effectiveness of ProtoDiff. Our results demonstrated significant improvements in classification accuracy compared to rate-of-thh-are meta-learning techniques, highlighting the potential of lask-guided diffusion in augmenting few-shot learning and advancing meta meta-learning.
Acknowledgment
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
References
[1] A. Afrasiyabi, H. Larochelle, J.-F. Lalonde, and C. Gagné. Matching feature sets for few-shot image classification. In CVPR, pages 9014-9024, 2022.
[2] K. R. Allen, F. Shelhamer, H. Shin, and J. B. Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICMI., pages 232-241, 2019. 2019.
[3] K. Cao, M. Brbic, and J. Leskovee. Concept learners for few-shot learning. ICLR, 2020.
10

---

[4] T. Cao, M. Law, and S. Fidler. A theoretical analysis of the number of shots in few-shot learning. arXiv preprint arXiv: 1909.1122.. 2019.
[5] Y. Chen, Z. Liu, H. Xu, T. Darrell, and X. Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In ICCV, pages 9062-9071, 2021.
[6] Dermnet. Dermnet dataset. https: bttps://dermnerr com/ 2016.
[7] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 34:8780- 8794, 2021.
[8] N. Ding, Wang, Y. Fu, G. Xu, Wang, Xie, Shen, Huang. H.-T. Zheng, and R. Zhang. Prototypical representation learning for relation extraction. arXiv preprint arXiv: 1647, 2021.
[9] B. Dong, P. Zhou, S. Yan, and W. Zuo. Self-promoted supervision for few-shot transformer. In ECCV, pages 329-347. Springer, 2022.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbomn X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[11] Y. Du, X. Zhen, L. Shao, and C. G. M. Snoek. Hierarchical variational memory for few-shot learning across domains. In ICLR, 2022.
[12] Z. Erkoç, F. Ma, Q. Shan, M. Nießner, and A. Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. arXiv preprint arXiv:2303.110111 2023.
[13] V. Gareia and Bruna. Few-shot learning with graph neural networks. In ICLR, 2018
[14] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, J. R. Smith, K. Saenko, T. Rosing, and
R. Feris. A broader study of cross-domain few-shot learning. In European Conference Computer Vision, 2020.
[15] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
A novel dataset and deep learning
[16] Ho. A. Jain, and P. Abbeel. Denoising diffuusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
[17] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv: 1.XX::2207.125999 2022.
[18] R. Hou, H. Chang, M. Bingpeng, S. Shan, and X. Chen. Cross attention network for few-shot classification. NeurlPS, pages 4005-4016. 2019.
[19] V. Hu, D. W. Zhang, Y. M. Asano, J. Burghouts, and C. G. M. Snoek. Self-guided diffusion models. In CVPR, 2023.
[20] Y. Hu and J. Ma. Adversarial feature augmentation for cross-domain few-shot classification. In ECCV, pages 20-37. Springer, 2022.
[21] D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6144 2013.
[22] G. Kwon and J. C. Ye. Diffusion-based image translation using disentangled style and content representation. arXiv preprint arXiv:2209./522 /5264, 2022.
[23] H. B. Lee, T. Nam, E. Yang, and S. J. Hwang. Meta dropout: Learning to perturb latent features for generalization. In ICLR, 2020.
[24] K. Lee, Maji, Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In CVPR, pages 10657-10665, 2019.

---

[25] S. Lee, B. Andreis, K. Kawaguchi, J. Lee, and S. Hwang. Set-based meta mntaa-nterppolaton for few-task meta-learning In NeurIPS, 2022.
[26] H. Li, D. Eigen, S. Dodge, M. Zeiler, and X. Wang. Finding task-relevant features for few-shot learning by category traversal. In CVPR, pages 1-10. Computer Vision Foundation IEEE, 2019.
[27] S. S. Lutati and L. Wolf. Ocd: Learning to overfit with conditional diffusion models. arXiv preprint arXiv:2210.00471, 2022.
[28] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[29] M. A. A. Milton. Automated skin lesion classification using ensemble of deep neural networks in isic 2018: Skin lesion analysis towards melanoma detection challenge. arXiv preprint arXiv: 1901. 10802, 2019.
[30] S. P. Mohanty, D. P. Hughes, and M. Salathé. Using deep learning for image-based plant disease Frontiers in plant science, 7:1419, 2016.
detection.
[31] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and
M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 22. 10741, 2021.
[32] B. Oreshkin, P. R. López, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, pages 721-731, 2018.
[33] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[34] S. K. Reed. Pattern recognition and categorization, Cognitive psychology, 3(3):882 407, 1972.
[35] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and S. Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint 1803.00000.0061 2018.
[36] D. Rezende and S. Mohamed, Variational inference with normalizing flows. In ICML, pages 1530-1538. PMLR, 2015.
[37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211225) 2015.
[38] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho. I. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH, pages 1-10, 2022.
[39] H. Sasaki, C. G. Willcocks, and T. P. Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358. 2021.
[40] V. G. Satorras and J. B. Estrach. Few-shot learning with graph neural networks. In ICLR, 2018
[41] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618-626, 2017.
[42] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In NewrIPS, pages 4077-4087, 2017.
[43] J. Sohl-Dickstein, F. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pages 2256-2265. PMLR, 2015.
[44] Song, C. Meng, and Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
12

---

[45] W. Sun, Y. Du, X. Zhen, F. Wang, L. Wang, and S. Cees, G.M. Metamodulation: Learning variational feature hierarchies for few-shot learning with fewer tasks. ICML. PMLR, 2023.
[46] E. Triantafillou, Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint D0.Xiv::903.000900 2019.
[47] Tschandl, C. Rosendahl, and H. Kittler. The ham 10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[48] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[49] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. In NeurlPS, pages 6000-6010, 2017.
[50] O Vinyals, C. Blundell, T. Lillicrap. D. Wierstra, et al. Matching networks for one shot learning.
In NeurIPS. pages 3630-3638, 2016.
[51] H. Wang and Z.-H. Deng. Cross-domain few-shot classification via adversarial task augmenta- tion. arXiv preprini arXiv:2104. 14385, 2021.
[52] X.
Wang, Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In CVPR, pages 2097- 2106, 2017.
[53] Xie, F. Long, J. Lv, Q. Wang, and P. Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In CVPR, pages 797 2022.
[54] S. Yang, L. Liu, and M. Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint Ttt::2001.06399. 2021.
[55] H. Yao, L.-K. Huang, L. Zhang, Y. Wei, L. Tian, J. Zou, J. Huang, et al. Improving generalization in meta-learning via task augmentation. In ICML, pages 11887-11897. PMLR, 2021.
[56] H. Yao, L. Zhang, and Finn. Meta-learning with fewer tasks through task interpolation. arXiv preprint arXiv:2106.028 02695, 2021.
[57] H. Ye, H. Hu, D. Zhan, and F. Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020.
[58] H.-J Ye, H. Hu, D.-C. Zhan, and Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8808-888 8817, 2020.
[59] S. Yoon, J. Seo, and Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In ICML, 2019.
[60] S. W. Yoon, J. Seo, and J. Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In ICML, pages 7115-7123, PMLR, 2019.
[61] X. Zhen, Y. Du, H. Xiong, Q Qiu, C. G. M. Snoek, and L. Shao. Learning to learn variational semantic memory. NeurIPS, 2020.
13

---

A Algorithms
We describe the detailed algorith ms for meta-training and meta-test of ProtoDiff as following Algorithm and 2, respectively:
Algorithm Meta-training phase of ProtoDiff.
Input: p(T): distribution over tasks; 0: diffusion parameters; &i feature extractor parameters; T: diffusion steps.
Output: zş: diffusion network, fa: feature extractor.
1: repeat
2: Sample batch of tasks T ~ p(T)
3: for all T do
4: Sample support and query set {S¹ Q²) from T
5: Compute the vanilla prototype = [(S))
6:
repeat
7:
Take some gradient step on VLCE(S', 2¹), updating @+
8:
until (CE 2.) converges
9:
Compute the overfitted prototype by using the updated Φ¹, z = (8.)
10:
t & Uniifrrm(1....
11:
e (0, 1)
B. (T--)++ (000-)))--((111)) 0r =
āe 1-,
Br,
13:
2 (a ) V-
14:
Compute diffusion loss Laitt with Equation (9) and the final loss Lys, with Equation (10) Update {ø, 0) + (6,0) 16.0) ETinp(T) Lys using query data of each task.
end for
17: until fo and Zœ converges
Algorithm Meta-test phase ProtoDiff.
Input: (S, Q): meta-test task, Zo: trained diffusion network parameters, fo: trained feature extractor network parameters, T: diffusion steps.
1: 2r ~ (
2: Compute vanilla prototype å with support set f(S)
3: for t........ do
4:
(0,
10 (0- 04 1 Be, a. 11
5: B.
1-1 Zi-1 zol (2c, ž)
6:
7: end for
8: Compute the final prediction y by with Equation (1) based 2g
B Per-task prototype overfitting architecture
To enhance our comprehension the Per-task prototype overfitting part, we propose a succinet architectural representation depicted in Figure 8. The initial step entails the computation of the conventional prototypes ž for a meta-training task. Subsequently, Equation (1) is employed to calculate the predictions for the query sample. The backbone's parameters are subsequently updated through iterations. Through the utilization of parameters from the final iteration, we ultimately obtain the prototypes z" that exhibit overfitting characteristics.
C Residual prototype learning architecture
In order to gain more comprehensive understanding of our residual prototype learning, we have crafted succinct architecture diagram illustrated Figure 9. Our proposition involves the prediction of the prototype update, denoted zz 2. instead of directly predicting the overfitted prototype
14

---

X iterations
Support set
Gradient descent
X1
S
z
Z
Backbone
y 9
[0.7, 0.2, 0.1]
CE
[1. o, 0]
Query image X
y !
Vanilla prototypes
Overfitted prototypes
Query features
Figure Per-task prototype overfitting.
z". This distinctive approach also allows us to initialize ProtoDiff with the capability to perform identity function, achieved by assigning zero weights to the decoder. Notably, we have discovered that the amalgamation of global residual connection and identity initialization substantially expedites the training process. By harnessing this mechanism, have successfully enhanced the performance of ProtoDiff in the context of few-shot learning tasks.
Task-guided diffusion
E
Z
Z
Z
Noise
Vanilla prototypes
Residual
Overfitted prototypes
Time step
Diffused prototypes
Figure 9: Residual prototype learning.
D Datasets
Within-domain few-shot. For this setting we focus on 5-way 1-shot/5-shot tasks, which aligns
with previous research [42]. The within-domain few-shot experiments are performed on three
datasets: minilmagenet [50] fieredlmagenet [35], and ImageNet-800) [5]. minilmagenet consists of 100 randomly selected classes from VLSVRRC2012 [37], while tiere//magenet is composed 608
classes that are grouped into 34 high-level categories. We measure the accuracy of 600 tasks randomly
15

---

sampled from the meta-test set to evaluate the performance. Following [5], we also evaluate our model on ImageNet- 800, dataset obtained by randomly dividing the 1,000 classes of ILSVRC-2012 into 800 base classes and 200 novel classes. The base classes consist of images from the original training set, while the novel classes comprise images from the original validation set.
Cross-domain few-shot. In the 5-way 5-shot cross-domain few-shot classification experiments, the training domain is minilmagenet [50], and the testing conducted on four different domains. These
domains are CropDisease [30], which contains plant disease images; EuroSAT [15], a collection of
satellite images; ISIC2018 [47], consisting of dermoscopic images of skin lesions, and ChestX [52],
a dataset of X-ray images.
Few-task few-shot.
Few-task few-shot learning [56] challenges the common meta- veta-training as-
sumption of having many tasks available. We conductt experiments on four few-task meta-learning
challenges, namely mindImagenet-S [50], ISIC [29], DermNet-S [6]. and Tabular Murris [3]. To
reduce the number of tasks and make it comparable to previous works, we followed [56] by limiting the number of meta-training classes to 12 for minilmagenet-S, with 20 meta-test classes. ISIC [29] involves classifying dermoscopic images across nine diagnostic categories, with 10,015 images available for training in eight different categories, of which we selected four as meta-training classes. DermNet [6], which contains over 23,000 images of skin diseases, was utilized to construct Dermnet- by selecting 30 diseases as meta-trainin classes, following [56]. Finally, Tabular Murris [3], which deals with cell type classification across organs and includes nearly 100,000 cells from 20 organs and tissues, was utilized select 57 base classes as meta-trait geta-training classes, again following the same approach as [56].
E Implementation details
In our within-domain experiments, we utilize a Conv-4 and ResNet-12 backbone for minilmagenet and maared A ResNet-50 is used for ImageNet-800. We follow the approach described in [5] to achieve better performance and initially train a feature extractor on all the meta-training data without episodic training. We use the SGD optimizer with a momentum of 0.9, a learning rate starting from 0.1, and decay factor of 0. For minilmagenet, we train for 100 epochs with batch size of 128, where the learning rate decays at epoch 90. For tieredImageNet, we train for 120 epochs with a batch size of 512, where the learning rate decays at epochs 40 and 80. Lastly, for ImageNet-800. we train for 90 epochs with batch size of 256, where the learning rate decays at epochs 30 and 60. The weight decay is 0.0005 for ResNet- 12 and 0.0001 for ResNet-50. Standard data augmentation techniques, including random resized crop and horizontal flip, are applied. For episodic training, we use the SGD optimizer with a momentum of 0.9, a fixed learning rate of 0.001, and a batch size of 4, meaning each training batch consists of 4 few-shot tasks calculate the average loss. For our cross domain experiments, we use a ResNet- 10 backbone to extract image features, which is a common choice for cross-domain few-shot classification [14, 57]. The training configuration for this experiment is the same as the within-domain minilmagenet training. For few-task few-shot learning, we follow [56] using network containing four convolutional blocks and a classifier layer. Each block comprises a 32-filter 3 x 3 convolution, a batch normalization layer, a ReLU nonlinearity. and a 2 x max pooling layer. All experiments are performed on a single A100 GPU, each taking approximately 20 hours. We will release all our code.
F
Visualization of diffusion process
The ProtoDiff method utilizes a task-guided diffusion model to generate prototypes that provide efficient class representations, as discussed in the previous section. To better understand the ef- fectiveness of our proposed approach, we provide a visualization by Grad-Cam [41] in Figure of the diffusion process, demonstrating how ProtoDiff gradually aggregates towards the desired class prototype during meta-traie trainiin. The vanilla prototype shown in the first row on the left, which does not exclusively focus on the guitar. In contrast, the overfitted prototype in the second row on the left provides the highest probability for the guitur. ProtoDiff, with the diffusion process, randomly selects certain areas to add noise and perform diffusion, resulting in prototype that gradually moves towards the guitar with the highest probability at t=0. Moreover, ProtoDiff with residual learning produces more precise prototype. The comparison between these different prototypes demonstrates
16

---

Diflusion procees leerning
Ventaa
100
tel0
tedy
Pppo
Probi
Diffusion with residual leaming
Garrtted
tel0
ted0
Input
0:35
PHot 0.48
Figure 10: Visualization of the diffusion process. The first row on the right shows the vanilla prototype, which does not exclusively focus on the guitar. In contrast, the overfitted prototype in the second row on the right provides the highest probability for the guitar. ProtoDiff randomly seleets certain diffusion process, with the lowest probability at the beginning time step. As time progresses, prototype gradually aggregates towards the guitar, with the highest probability at t=0. comparison, ProtoDiff with residual learning produces a more precise prototype.
predict during the
the effectiveness the ProtoDiff diffusion process in generating a more accurate and informative prototype for few-shot learning tasks.
G
More results
Effect of transformer structure Our ProtoDiff model is constructed using design inspired by GPT-2. This includes a 12-layer transformer, a 512-dimensional linear transformation, an attention mechanism with 16 heads, and an MLP with hidden dimensionality 512. These configurations are in line with GPT-2's default parameters. The configuration files can be accessed our code repository for more detailed parameter setup. Our experiments in the table 7a and 7b highlight that our model achieves optimal performance with these settings.
Table 7: Effect of transformer structures
(a) Results on the different transformer structures.
(b) Results the different heads numbers.
miniTmagenet 5-shot
minilmagenet
Structures
1-shot
Nodes numbers
1-shot
5-shot
3-layers
62.17 10025
78.93.0.11
node
63.28 10.24
79.97 10.15
6-layers
63.25 10.22
79.63 10.15
node 4
64,43 10.22
80.13
9-layers
65.21
80.13440
node
15.9111023
81.91
1a-lrs
66.63 -0221
83.48
node 16
66.63 :0:21
83.48 10.15
15-layers
64.15 +0.25
80.93 10.18
Effect of different time steps We have selected the diffusion time step for the diffusion process to be 100. We' ve adopted the DDIM sampling strategy to accelerate prediction with. This effecctivlyy reduces the total sample requirement to just 10. We've conducted comparative experiments using varying diffusion times and intermediate intervals. As presented in the tables 8 and 9, observe that as the diffusion timesteps increase, both the performance and the inference time increase. Simultaneously, when is increased, the inference time decreases, making the process more efficient.
Effeet of more support images To prove more support images during meta-training to obtain more accurate prototypes, we conducted an experimental comparison using different numbers of support sets during meta-training. The results in the table 10 illustrate that augmenting the number of support images for each class during the intaa-rrainig phase enhances performance across various shots.
17

---

Table 8: Results with the different diffusion timesteps minilmagenets..
dim(r)= 10, Timesteps
1-shot
5-shot
1-shot inference time
5-shot inference time
==10
63.98
80.12 10:15
ms
ms
==
65.93-.021
82.93
5 ms
ms
T= 100
66.23
83.48
9 ms
14 ms
=
66.65 #:223
83.59-0.14
53
93
== 1000
66. 0.220
83.97 20117
102 ms
192 ms
Table 9: Results with the different diffusion timesteps on minilmaagenets.
dim(r), T= 100
1-shot
5-shot
Speed
r ==
66.78 =0.21
83.62
10.14
99 ms
T T= 5
66.15 =0223
83.44
45 ms
r == 10
66.63
83.48
10.15
ms
T 20
66.12 =0221
82.79
10.13
ms
== 100
63.15 0..23
81.27
10014
ms
Meta-training and meta-test wall-clocks times per task We have compiled the wall-clock time for both the meta-training and meta-testing phases of ProtoDiff and compared these against the respective times for ProtoNet in the Tables 1a and llb. In meta-training and meta-testing wall-clock times per task, our PrftoDiff is slower by factors of ProtoNet 5x and 15 x, respectively. As part of future work, we will investigate and address this limitation to further enhance the efficiency of our approach.
Rationale for Conditioning We utilize the "vanilla" prototype as guiding condition, gradually enabling the diffusion model to generate class-specific prototypes. We also explored two alternative strategies for conditioning in the table 12: one without any conditions and another with learned class embeddings. When not conditioned, the performance tends to be arbitrary due to the absence of class-specific cues during diffusion. On the other hand, employing learned class embeddings as condition yielded subpar results compared to the vanilla prototype, potentially due to outliers in each class's 's learned embeddings.
Results of our pre-trained model We also give the results of our own pre- pre-trained model before applying ProtoDiff to provide a clear comparison and demonstrate our method's improvements. We have prepared the results of our pre- pre-trained model (trained with CE loss on the whole training set) and will present table 13.
18

---

Table Results of different support sets for each class.
minilmagenet 1-shot
Structures
5-shot
66.63=a211
73.12
70.25 10.22
83.48
10.13
73.21.0021 1:021
84.72 10.13
82.21 1:023
84.98
10.17
Table 11: Meta traaattraiinng and meta-test wall-clock times each task.
(a) Mcta-training.
(b) Meta-test.
minilmagenet 5-shot
minilmagenct 5-shot
Meta-training
1-shot
Meta-test
1-shot
ProtoNet
0.6 ms
1.2 ms
ProtoNet
0.6 ms
1.2 ms
ProtoDiff
3
5.3 ms
ProtoDiff
ms
14 ms
Table 12: Results of different conditions.
minilmagenet
Conditions
1 1-shot
5-shot
w/o conditions
20.27
22.32
Learnt class embeddings
66.17 40.22
81.17-0016
Vanilla prototype
66.63 +0.21
83.48
Table 13: Results of our pre-trained model on different datasets
minilmagenet I-shot
tiered/magenet
Imagenet-80N
S-shot
1-shot
5-shot
1-shot
5-shot
Pretrained from Chen al [}]
Classifier-Basee
58.91
77.76
68.07.00
83.74.n1s
86.07
96. n
Pretrained Chen al. [S]
ProtoDiff
66.63 -9.1
72.95 -834
85.15
92.13-020
Pretrained from
Classifer- Baseline
58.75
77.86
68.15-025
83,95
85.97-022
96.03
Pretrained from ours
ProtoDiff
66.49 0.25
83.51 18.4
73.01
1824
85.72 k..7
92.05 0.21
98.11
0.18
19


Reviews:
[{"review_id": "9EOIxOUPRN", "reviewer": "Reviewer_zHCT", "summary": "This paper tries to improve Prototypical Network in few-shot learning. Specifically, the authors use a diffusion model to predict optimal prototypes from initial prototypes. In each iteration of the meta-training phase, the optimal prototypes are obtained by several inner loops on the current task, then a diffusion model takes the initial prototypes as input, and predicts the difference between the optimal and initial prototypes. The final loss is used to meta-learn the feature extractor and the diffusion model. The experiments show that the proposed method can improve ProtoNet to some degree.", "questions": "See Weakness above.", "limitations": "The authors have stated some of the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Introducing diffusion models into few-shot learning and using it to translate features is interesting.\n- The use of residual prototype learning is clever.", "weaknesses": "- In essence, the key idea is to learn a module to predict optimal prototypes from initial prototypes. Thus this module is a set-to-set function, and does not need to be stochastic, e.g., a MLP or transformer.  From this point, there is no motivation of using generative models like diffusion models, considering that no probabilistic inference is used during training or testing. The motivation of this paper is thus very weak.\n- In fact, back in 2019, FEAT algorithm [1] already exploited such an idea, using a shallow transformer to transform initial prototypes into near-optimal prototypes without using any probabilistic models. While being much more efficient (no diffusion operations, no inner gradient loops, and much fewer parameters used) and being less tuned for hyperparameters, FEAT can achieve performance comparable to ProtoDiff.  Similarly, CrossTransformer [2] also uses a shallow transformer to transform initial prototypes into near-optimal prototypes, but in addition, considers spatial information. Thus the novelty and the value of the paper are quite limited.\n- The writing of the method section needs to be significantly improved. Before looking at the algorithm 1-2 in the appendix, I struggle to understand the pipeline of the method, and some notations are confusing. For example, there are clear mistakes in the notations in lines 112-116.\n- In Table 1, the result of Classifier-Baseline is borrowed from Meta-Baseline. Instead, the authors need to show the results of their own pre-trained model (trained with CE loss on the whole training set) before the use of ProtoDiff to show how much improvement of their method brings.\n- In Table 2, only generative models are considered. As stated above, more deterministic feature transformation methods should be compared.\n\n[1] Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions. CVPR 2020.\n[2] CrossTransformers: spatially-aware few-shot transfer. NeurIPS 2020.\n\n======Post-rebuttal=======\n\nThe authors have addressed most of the concerns during the rebuttal. Thus I raise my score from 3->7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thank you for your time, effort, and constructive comments.***\n\n* **Motivation of ProtoDiff**\n\nAnswer: We acknowledge your insight regarding using a stochastic or diffusion model. Our choice to employ a generative model like the diffusion process extends beyond simply predicting optimal prototypes from initial ones. It also incorporates uncertainty into the few-shot learning process, which can be vital when dealing with limited samples.  We have also conducted the experiments only using MLP and transformers instead of using generative models in the following table.\n\n|miniImageNet|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|Vanilla|63.17|79.26|\n|MLP|64.15|80.23|\n|Transformer|64.97|81.28|\n|Diffusion|66.63|83.48|\n\nWhile an MLP or transformer can improve performance over vanilla prototypes, our experiments reveal that ProtoDiff outperforms these alternatives. Incorporating uncertainty through the generative model provides a nuanced advantage in capturing the underlying distribution, resulting in better performance. We will detail these experimental comparisons and further clarify the motivation behind our approach in the revised manuscript.\n\n* **Limited novelty**\n\nAnswer: While FEAT and CrossTransformer also utilize a function to transform vanilla prototypes into near-optimal ones, they lack the intermediate supervision provided by \"overfitted\" prototypes, which can guide the transformed prototype toward an optimal state. Our model uniquely learns an \"overfitted\" prototype first and then leverages a diffusion model to capture this learning process, ensuring a more refined transition to the optimal prototype. Besides the diffusion process to generate diffused prototypes, our model provides a new insight to leverage the “overfitted” prototype as supervision. This approach enables our system to utilize the overfitted prototype as guidance, thus implementing meta-learning as a continuous learning process.  We will incorporate these papers into our discussion of related work and in our comparisons. We believe this is of community interest. \n\n* **Writing of the method**\n\nAnswer: We regret that the notations in the method section were confusing, and we appreciate pointing out the specific mistakes in lines 112-116. We have taken your feedback to heart, fixed these typos, and conducted a thorough review of all notations and equations in the manuscript.\n\n* **Pre-trained model**\n\nAnswer: We agree that it is essential to showcase the results of our own pre-trained model before applying ProtoDiff to provide a clear comparison and demonstrate our method's improvements. In response to your suggestion, we have prepared the results of our pre-trained model (trained with CE loss on the whole training set) and will present it in a new table. This information will be included in the revised manuscript to provide a more transparent and comprehensive evaluation of ProtoDiff.\n\n|||mini  1-shot|mini   5-shot|tiered 1-shot|tiered  5-shot|800  1-shot|800  5-shot|\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n|Pretrained from Chen et al.|Classifier-Baseline|58.91|77.76|68.07|83.74|86.07|96.14|\n|Pretrained from Chen et al.|ProtoDiff|66.63|83.48|72.95|85.15|92.13|98.21|\n|Pretrained from ours|Classifier-Baseline|58.75|77.86|68.15|83.95|85.97|96.03|\n|Pretrained from ours|ProtoDiff|66.49|83.51|73.01|85.72|92.05|98.11|\n\n* **More deterministic feature transformation methods results**\n\n We have also conducted the experiments using MLP and transformers instead of generative models in the first question (**Motivation of ProtoDiff**).\nWhile an MLP or transformer can improve performance over vanilla prototypes, our experiments reveal that ProtoDiff outperforms these alternatives. \n\n\n\n"}, {"review_id": "vI8mrtHI2x", "reviewer": "Reviewer_ZGD4", "summary": "This paper proposes a prototype-based meta-learning method called ProtoDiff. During meta-learning stage, ProtoDiff introduces a task-guided diffusion process, learning a generative process conditioned on vanilla prototype for more robust prototype representation. ProtoDiff is extensively validated on within-domain, cross-domain, and few-task few-shot classification.  \n\n", "questions": "See \"Weaknesses\" section. ", "limitations": "Though the authors discuss the limitations of the proposed ProtoDiff, i.e., heavy computational cost, it is not clear how ProtoDiff is expensive compared to the baselines. I suggest the authors provide wall-clock time of meta-training/meta-testing of ProtoDiff against the ProtoNet. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The diffusion models have been very successful generative models, attracting great interests very recently. It is interesting to see the authors introduce the diffusion models into few-shot learning regime, for addressing the limitation of unrobust prototype estimation given a limited number of training examples.\n\n(2) The proposed ProtoDiff is extensively evaluated on a variety of few-shot learning problems, including within-domain, cross-domain, and few-task few-shot classification. The ablation study is carefully and thoroughly designed for validating the effectiveness. \n", "weaknesses": "(1) The diffusion model for prototype learning is the key in the proposed ProtoDiff; however, some technical details on the diffusion model are missing. \n\nAbout the architecture, how many layers are used in the encoder and what about dimension of linear transform and the number of heads in the attention module and what about the hidden dimensionality in MLP? What the proposed ProtoDiff will perform if the design parameters vary?\n\nAbout the diffusion process, how do you select T? How does different T affect the performance and speed of the proposed ProtoDiff?\n\n(2) Some other concerns.\n\nIn ProtoDiff, the ground truth prototypes, called overfitting prototypes, are obtained by finetuning the network for several steps. Besides that strategy, have you ever considered, during meta-training, using more support images (e.g. 8 or 16) by which one may get more accurate prototypes? As suggested in ProtoNet, it is beneficial to train with a higher way that will be used at meta-test.  \n\nIn table 3, the baseline of ProtoNet is SetFeat; therefore, I suggest the authors change “This paper’’ to “SetFeat+ProtoNet”. How do you implement ProtoNet in Tables 4 and 5? Is it based on SetFeat or ProtoNet? Please clarify. \n\nIn equation (10), $\\mathbf{z}$ is not defined; the subscript $t$ of $\\sum$ is missing. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thank you for your time, effort, and constructive comments.***\n\n(1) **Technical details are missing** \n\nAnswer: Our ProtoDiff model is constructed using a design inspired by GPT-2. This includes a 12-layer transformer, a 512-dimensional linear transformation, an attention mechanism with 16 heads, and an MLP with a hidden dimensionality 512. These configurations are in line with GPT-2's default parameters. The configuration files can be accessed in our code repository for more detailed parameter setup. Our experiments in the subsequent tables highlight that our model achieves optimal performance with these settings. We will ensure this information is included in the revised manuscript for clarity.\n\n\n|transformer structures|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|3-layer|62.17$\\pm$ 0.25|78.93$\\pm$0.17|\n|6-layer|63.25$\\pm$ 0.22|79.63$\\pm$0.15|\n|9-layer|65.21$\\pm$ 0.21|80.13$\\pm$0.18|\n|12-layer|66.63$\\pm$ 0.21|83.48$\\pm$0.15|\n|15-layer|64.15$\\pm$ 0.25|80.93$\\pm$0.18|\n\n\n|Heads numbers|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|1|63.28$\\pm$ 0.24|79.97$\\pm$0.15|\n|4|64.23$\\pm$ 0.22|80.13$\\pm$0.14|\n|8|65.91$\\pm$ 0.23|81.91$\\pm$0.16|\n|16|66.63$\\pm$ 0.21|83.48$\\pm$0.15|\n\n We have selected the diffusion time step **T** for the diffusion process to be 100. We've adopted the DDIM sampling strategy to accelerate prediction with $dim(\\tau) = 10$. This effectively reduces the total sample requirement to just 10. We've conducted comparative experiments using varying diffusion times and intermediate intervals $dim(\\tau)$. As presented in the subsequent tables, we observe that as the diffusion timesteps increase, both the performance and the inference time increase. Simultaneously, when $dim(\\tau)$ is increased, the inference time decreases, making the process more efficient. \n\n\n\n|$dim(\\tau)=10$, Timesteps|1-shot|5-shot|1-shot inference time|5-shot inference time|\n|:-----:|:-----:|:-----:|:-----:|:-----:|\n|T = 10|63.98$\\pm$ 0.24|80.12$\\pm$0.15|1ms|2ms|\n|T = 50|65.93$\\pm$ 0.21|82.93$\\pm$0.17|5ms|7ms|\n|T = 100|66.63$\\pm$ 0.21|83.48$\\pm$0.15|9ms|14ms|\n|T = 500|66.65$\\pm$ 0.23|83.59$\\pm$0.14|53ms|93ms|\n|T = 1000|66.74$\\pm$ 0.20|83.97$\\pm$0.17|102ms|192ms|\n\n|$dim(\\tau)$, T=100|1-shot|5-shot|speed|\n|:-----:|:-----:|:-----:|:-----:|\n|1|66.78$\\pm$ 0.21|83.62$\\pm$0.14|99ms|192ms|\n|5|66.15$\\pm$ 0.23|83.44$\\pm$0.16|45ms|92ms|\n|10|66.63$\\pm$ 0.21|83.48$\\pm$0.15|9ms|14ms|\n|20|66.12$\\pm$ 0.21|82.79$\\pm$0.13|5ms|7ms|\n|100|63.15$\\pm$ 0.23|81.27$\\pm$0.14|1ms|2ms|\n\n\n(2) **More support images**\n\nAnswer:  We appreciate your suggestion of using more support images during meta-training to obtain more accurate prototypes, as also recommended in ProtoNet. To shed more light on this, we've conducted an experimental comparison using different numbers of support sets during meta-training. The results in the subsequent table illustrate that augmenting the number of support images for each class during the meta-training phase enhances performance across various shots. We intend to integrate these findings into the revised version of our manuscript.\n\n\n\n|Number of support sets for each class|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|1|66.63$\\pm$ 0.21|73.12$\\pm$0.15|\n|5|70.25$\\pm$ 0.22|83.48$\\pm$0.13|\n|8|73.91$\\pm$ 0.21|84.72$\\pm$0.13|\n|16|82.21$\\pm$ 0.23|84.98$\\pm$0.17|\n\n**Clarified Tables 4 and 5**\n\nAnswer: Thank you for your keen suggestion. In our revision, we will change the label accordingly. For Table 4, the implementation is based on AFA [19]. Hence we will appropriately change \"This paper\" to \"AFA + ProtoDiff\". Similarly, in Table 5, the implementation is based on MLTI [51], and the label \"This paper\" will be replaced with \"MLTI + ProtoDiff.\" \n\n**Equation typo**\n\nAnswer:  Thank you for your careful observation. In Equation (10), **z** should be denoted as $\\mathbf{z_t}$. Also, you're correct about the missing subscript for the summation symbol $\\sum$. The subscript should be $(x, y) \\sim Q$. Our revised manuscript will correct these typos to ensure clarity and precision.\n\n(3) **Limitations**\n\nAnswer:  We have compiled the wall-clock time for both the meta-training and meta-testing phases of ProtoDiff and compared these against the respective times for ProtoNet. In meta-training and meta-testing wall-clock times per task, our ProtoDiff is slower by factors of  ProtoNet  $5 \\times$ and $15 \\times$, respectively.\nAs part of future work, we will investigate and address this limitation to further enhance the efficiency of our approach.  We will incorporate these details into the revised manuscript for a more comprehensive comparison.\n\n|Meta-train|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|ProtoNet|0.6ms|1.2ms|\n|ProtoDiff|3ms|5.3ms|\n\n\n|Meta-test|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|ProtoNet|0.6ms|1.2ms|\n|ProtoDiff|9ms|14ms|\n\n\n"}, {"review_id": "iMGrXg2CtN", "reviewer": "Reviewer_8tyC", "summary": "This paper proposes using the diffusion model to learn a prototypical network. During the meta-train stage, an overfitted prototype (by fine-tuning the classifier until it gets a probability of 1 for the task) is obtained first, then given a vanilla prototype (by simply averaging the features) as input, the diffusion model is to denoise a random prototype into the overfitted prototype. This is because the overfitted prototype is unknown during the meta-test stage, therefore the diffusion model is to obtain this overfitted prototype by using the vanilla prototype as conditional information.", "questions": "- I believe Eq.8 has a typo.\n- What is the ratio of the increased training time over the conventional model?\n- What is the case when the number of available tasks is low and not diverse? Will the conventional model perform better or not?\n- How can this model be applied in production compared to conventional model?", "limitations": "- the author mentioned two limitations, one is the proposed model requires a substantial number of timesteps to sample the prototype during the meta-test stage. \n- secondly, obtaining the overfitted prototype requires increased training time.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- the whole design of leveraging the diffusion model to predict the prototype is quite interesting and novel. That is, overfitted prototype indeed is better than the vanilla prototype (can be seen as an \"optimal\" prototype for a given task), but obtaining it is not possible during the meta-test stage because the query set is not available. Hence, using the diffusion model to capture the distribution of how a vanilla prototype can be turned into an overfitted prototype is quite making sense.\n", "weaknesses": "- as the diffusion model is data-hungry, I believe when the number of available tasks is low and not diverse, this proposed model may not perform better than a conventional model. The author should also include this analysis to make the paper more complete.\n- the proposed model has increased training time due to overfitting the prototype and a high number of timesteps to predict the prototype, as mentioned by the author. The author should also include this analysis to let readers know how much increased time over conventional methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thank you for your positive feedback and for recommending acceptance of our paper.***\n\n* **Lower and not diverse available tasks**\n\nAnswer: We acknowledge the reviewer's point regarding the data-intensive nature of diffusion models. Indeed, performance can be impacted when faced with a limited number of training or less diverse tasks.\n Nevertheless, our original manuscript's Table 4 offers insights into cross-domain few-shot learning, indicating that even in less diverse scenarios, ProtoDiff maintains competitive results. \nTable 5 further illustrates our model's resilience under the constraints of the few-task few-shot learning framework. In addition, our approach consistently stands its ground against top-tier competitors, including MetaModulation (Sun et al.). We are confident these findings effectively address the issues highlighted and will provide more details in our revised version.\n\n|miniImageNet|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|MetaModulation (Sun et al.)|43.21|57.26|\n|MLTI + ProtoDiff|44.75|58.18|\n\nSun et al. \"MetaModulation: Learning Variational Feature Hierarchies for Few-Shot Learning with Fewer Tasks.\" ICML 2023.\n\n* **Increased time over conventional methods**\n\nAnswer: We appreciate your suggestion regarding the increased training time for the proposed model, which is a consequence of overfitting the prototype and the high number of timesteps required for predicting the prototype. To provide a clear comparison, we have analyzed the time consumed during both the meta-training and meta-testing phases of ProtoDiff and compared these results with the respective times for ProtoNet. \n In meta-training and meta-testing wall-clock times per task, our ProtoDiff is slower by factors of  ProtoNet  $5 \\times$ and $15 \\times$, respectively. Numerical results are provided in our response to *Reviewer ZGD4*, who asked the same question.\nWe will include this time analysis in the revised manuscript to understand our approach's efficiency comprehensively.\n\n* **Eq.8 typo**\n\nAnswer: Correct, the summation symbol $\\sum$ is missing over the task $T_i$. We have rectified this and will ensure a thorough review of all notations and equations in our manuscript to prevent any such oversights in the final draft. \n\n* **The ratio of the increased training time**\n\nAnswer: The training time for our proposed model, ProtoDiff, compared to the conventional model, is approximately 2.5 times longer. This increase is primarily due to the overfitting of the prototype and the diffusion process to predict the prototype.\n\n* **The number of available tasks is low and not diverse**\n\nAnswer: In scenarios where the number of tasks is limited and lacks diversity, the performance of conventional models might be questioned. However, in Table 4, our experiments focus on cross-domain few-shot classification and have already shown ProtoDiff's capabilities even when tasks are not diverse and exhibit domain shifts.\n Further, our experiments in Table 5 delved into the few-task few-shot challenges, where the number of available tasks is low. In both these situations, ProtoDiff consistently emerges as one of the leading models, underscoring its effectiveness under such constraints.\n\n* **How can this model be applied in production?**\n\nAnswer: Our ProtoDiff is designed to be adaptable in its application to any prototype-based meta-learning method, enhancing performance across different challenges, such as conventional few-shot learning, cross-domain few-shot learning, and few-task few-shot learning. The versatility of ProtoDiff allows it to handle various few-shot scenarios, providing a potential edge over standard methods. Such adaptability can be beneficial in diverse production environments.\n\n* **Limitations**\n\nAnswer: We have included the comparisons for the effect of the number of timesteps (see response to *reviewer ZGD4*) and increased training time.\n\n\n\n\n\n\n\n\n"}, {"review_id": "p0nNfT3lUW", "reviewer": "Reviewer_BNd7", "summary": "The paper introduces ProtoDiff, a novel addition to the family of Prototypical Network methods that uses a diffusion model to obtain better class prototypes at inference time.\n\nPrototypical Networks perform classification with metric-based comparisons between examples and “class prototypes”, variables computed based on support sets of class examples.\n\nIn ProtoDiff, a “vanilla” prototype is first computed based on the class support set. This vanilla prototype is then used as conditional input of a diffusion model, computing over T steps a residual between the vanilla prototype and a better, “diffused” prototype for the class. This diffused prototype is then used to perform traditional prototypical network classification. To train the diffusion model at meta-training time, one has to produce “overfitted” prototypes for tasks, based on both the query set and support set for the task.\n\nThe authors then demonstrate the effectiveness of their approach with a series of experiments for within-domain few-shot learning, cross-domain few-shot learning, and few-task few-shot learning.", "questions": "I would ask the authors to further elaborate on the positioning of their work within the meta-learning literature, arguing for the choice of conditioning the diffusion model on the existing “vanilla” prototype, and comparing it with possible alternative choices, such as specifically learnt class embeddings.", "limitations": "The authors satisfactorily discuss the limitations of their work in the main paper text.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The combination of prototypical networks and diffusion models is interesting: applying insights from recent developments in diffusion models to the field of meta-learning is still under-explored in the field.\n\nThe paper is clearly presented, the procedure itself is a novel combination of existing ideas, and is clearly motivated and easily argued for.\n\nThe experiments seem to support the claim that the diffusion model specifically is what is needed to edge out any remaining performance missing from the original prototypical network approach, in the form of a residual.", "weaknesses": "Overall, the paper doesn’t have any glaring flaw, if not for the fact that experimental gains seems marginal. In practice, the idea behind the paper seems to be about squeezing any additional percentage point of performance out of the existing technique of prototypical networks. This is especially apparent with the fact that the diffusion model is used to produce a residual, and is also itself conditioned on the “vanilla” prototype.\n\nAs for related work, it would be interesting to explore the connection with *Nava et al. (2022). Meta-Learning via Classifier (-free) Diffusion Guidance.* (https://arxiv.org/abs/2210.08942). They also apply diffusion models in the context of meta-learning, using a similar metric classification technique for the downstream tasks, even though they approach the problem with the goal of conditionally diffusing a model’s parameters, instead of class prototypes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Thank you for your time, effort, and constructive comments.***\n\n* **Weakness**\n\nAnswer:  Thank you for the thoughtful feedback and the reference to the work by Nava et al. We acknowledge the concern that the experimental gains might seem marginal and that the focus appears on incremental improvements within prototypical networks.\n\nHowever, it is essential to highlight that applying a diffusion model in ProtoDiff represents a novel way to generate prototypes, moving beyond simple performance squeezing. By using a task-guided diffusion model to produce a residual and allowing for task-specific prototype generation, ProtoDiff opens a new avenue in prototype learning. This approach offers a subtle but significant departure from conventional methods, allowing for more robust and nuanced class representations.\n\nAs for the connection with the paper by Nava et al., we agree that exploring similarities and differences between our approach and theirs would enrich our work. While our focus is on the diffusion of class prototypes, their method applies diffusion models in a broader context of the model's parameters. We will review this paper in depth and include a discussion in our revised manuscript to clarify how ProtoDiff fits within this broader research landscape.\n\n* **Questions**\n\nAnswer:\n**Position in the Meta-Learning Landscape**: As far as we know, our ProtoDiff is the first approach that integrates a diffusion model for generating prototypes in few-shot meta-learning. This design choice facilitates seamless integration with any metric-based meta-learning method for mutual benefit. \n\n\n**Rationale for Conditioning**: We utilize the “vanilla” prototype as a guiding condition, gradually enabling the diffusion model to generate class-specific prototypes. We also explored two alternative strategies for conditioning in the following table: one without any conditions and another with learned class embeddings.\nWhen not conditioned, the performance tends to be arbitrary due to the absence of class-specific cues during diffusion. On the other hand, employing learned class embeddings as a condition yielded subpar results compared to the vanilla prototype, potentially due to outliers in each class's learned embeddings.\nWe plan to elaborate on these insights in our revised manuscript.\n\n\n\n|Conditions|1-shot|5-shot|\n|:-----:|:-----:|:-----:|\n|W/o conditons|20.27$\\pm$ 0.25|22.32$\\pm$0.15|\n|Learnt class embeddings|66.17$\\pm$ 0.22|81.17$\\pm$0.16|\n|Vanilla prototype|66.63$\\pm$ 0.21|83.48$\\pm$0.15|\n\n"}]

Author rebuttal:

Meta-review: This paper proposes a prototype-based meta-learning method, ProtoDiff.  It designs a task-guided diffusion process to predict the more optimal prototypes from initial prototypes. Experimental results show the effectiveness of ProtoDiff on within-domain, cross-domain, and few-task few-shot classification tasks. All the reviewers unanimously accepted the paper, and so did the final decision. The authors should also revise their paper according to the reviewers' suggestions in the final version.
Decision: Accept (poster)