SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Prototype-based meta-learning. Prototype-based meta-learning is based on distance metrics and generally learns a shared or adaptive embedding space in which query images are accurately matched to support images for classification. It relies on the assumption that a common metric space is shared across related tasks and usually does not employ an explicit base learner for each task. By extending the matching network [50] to few-shot scenarios, Snell et al. [42] constructed a prototype for each class by averaging the feature representations of samples from the class in the metric space. The classification matches the query samples to prototypes by computing their distances. To enhance the prototype representation, Allen et al [2] proposed an infinite mixture of prototypes to adaptively represent data distributions for each class, using multiple clusters instead of a single vector. Oreshkin et al [32] proposed a task-dependent adaptive metric for few-shot learning and established prototype classes conditioned on a task representation encoded by a task embedding network. Yoon et al [60] proposed a few-shot learning algorithm aided by a linear transformer that performs task-specific null-space projection of the network output. Graphical neural network-based models generalize the matching methods by learning the message propagation from the support set and transferring it to the query set [13]. FEAT [58] was proposed to leverages samples from all categories within a task to generate a prototype, capitalizing on the intrinsic inter-class information to derive a more discriminative prototype. In contrast, our ProtoDiff, while using the overfitted prototype as supervision, only employs samples from a single category to generate the new prototype. This means we are not tapping into the potential informative context provided by samples from other categories. Additionally, our ProtoDiff employs the diffusion model to progressively generate the prototype, whereas FEAT does not utilize any generative model for prototype creation. Prototype-based methods have recently been improved in various ways [4, 46, 58, 61]. To the best of our knowledge, we are the first to propose a diffusion method to generate the prototype per task, rather than a deterministic function.
Diffusion models. These models belong to a category of neural generative models that utilize stochastic diffusion processes [43, 44], much like those found in thermodynamics. In this framework, a gradual introduction of noise is applied to a sample of data. A neural model then learns to reverse the process by progressively removing noise from the sample. The model denoises an initial pure noise to obtain samples from the learned data distribution. Ho et al. [16] and Song et al. [44] contributed to advancements in image generation, while Dhariwal and Nichol [7] introduced classifier-guided diffusion for a conditioned generation. GLIDE later adapted this approach [31], enabling conditioning on textual CLIP representations. Classifier-free guidance [17] allows for conditioning with a balance between fidelity and diversity, resulting in improved performance [31]. Since the guided diffusion model requires a large number of image-annotation pairs for training, Hu et al. [19] propose self-guided diffusion models. Recently, Hyperdiffusion [12, 27] was proposed in the weight space for generating implicit neural fields and 3D reconstruction. In this paper, we introduce ProtoDiff, a prototype-based meta-learning approach within a task-guided diffusion model that incrementally improves the prototype's expressiveness by utilizing a limited number of samples.
In this section, we assess the efficacy of ProtoDiff in the context of three distinct few-shot learning scenarios: within-domain few-shot learning, cross-domain few-shot learning, and few-task few-shot learning. For the within-domain few-shot learning experiments, we apply our method to three specific datasets: miniImagenet [50], tieredImagenet [35], and ImageNet-800 [5]. Regarding cross-domain few-shot learning, we utilize miniImagenet [50] as the training domain, while testing is conducted on four distinct domains: CropDisease [30], EuroSAT [15], ISIC2018 [47], and ChestX [52]. Furthermore, few-task few-shot learning [56] challenges the common assumption of having abundant tasks available during meta-training. To explore this scenario, we perform experiments on four few-task meta-learning challenges: miniImagenet-S [50], ISIC [29], DermNet-S [6], and Tabular
Table 1: Benefit of ProtoDiff on miniImagenet, tieredImagenet and ImageNet-800. The results of the Classifier-Baseline and Meta-Baseline are provided by Chen et al. [5]. ProtoDiff consistently achieves better performance than two baselines on all datasets and settings.
Method	miniImagenet	tieredImagenet	Imagenet-800
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
Classifier-Baseline [5]	58.91±0.23	77.76±0.17	68.07±0.29	83.74±0.18	86.07±0.21	96.14±0.08
Meta-Baseline [5]	63.17±0.23	79.26±0.17	68.62±0.27	83.74±0.18	89.70±0.19	96.14±0.08
ProtoDiff	66.63±0.21	83.48±0.15	72.95±0.24	85.15±0.18	92.13±0.20	98.21±0.08
Figure 5: Visualization of the different reconstructed overfitted prototypes using different generative models. The vanilla prototype focuses only on the overall features of the bird and overlooks the finer details of the robin. In contrast, the overfitted prototype can highlight specific features such as the tail and beak. While VAE and normalizing flow can generate prototypes of certain parts, the diffusion method can generate prototypes that more closely resemble the overfitted prototype. With residual prototype learning, ProtoDiff achieves better.
Murris [3]. For a comprehensive understanding of the datasets used in each setting, we provide detailed descriptions in the Appendix.
Implementation details. In our within-domain experiments, we utilize a Conv-4 and ResNet-12 backbone for miniImagenet and tieredImagenet. A ResNet-50 is used for ImageNet-800. We follow the approach described in [5] to achieve better performance and initially train a feature extractor on all the meta-training data without episodic training. Standard data augmentation techniques are applied, including random resized crop and horizontal flip. For our cross-domain experiments, we use a ResNet-10 backbone to extract image features, which is a common choice for cross-domain few-shot classification [14, 57]. The training configuration for this experiment is the same as the within-domain training. For few-task few-shot learning, we follow [56] using a network containing four convolutional blocks and a classifier layer. The average within-domain/ cross-domain, few-task few-shot classification accuracy (\%, top-1) along with  $95\%$  confidence intervals are reported across all test query sets and tasks. Code available at: https://github.com/YDU-uva/ProtoDiff.
Benefit of ProtoDiff. Table 1 compares our ProtoDiff with two baselines to demonstrate its effectiveness. The first Classifier-Baseline is a classification model trained on the entire label set using a classification loss and performing few-shot tasks with the cosine nearest-centroid method. The Meta-Baseline [5] consists of two stages. In the first stage, a classifier is trained on all base classes, and the last fully connected layer is removed to obtain the feature encoder. The second stage is meta-learning, where the classifier is optimized using episodic training. The comparison between the Baseline and Meta-Baseline highlights the importance of episodic training for few-shot learning. ProtoDiff consistently outperforms Meta-Baseline [5] by a large margin on all datasets and shots. The task-guided diffusion model employed in our ProtoDiff generates more informative prototypes, leading to improvements over deterministic prototypes.
Benefit of diffusion model. To confirm that the performance gain of our ProtoDiff model can be attributed to the diffusion model, we conducted the experiments only using MLP and transformers
as non-generative models. We also compared it with two widely used generative models: the variational autoencoder (VAE) [21] and normalizing flows [36]. VAE learns a low-dimensional representation of input data to generate new samples. Normalizing flows are more recent and learn a sequence of invertible transformations to map a simple prior distribution to the data distribution. We obtained the task-specific prototype in the meta-test stage, conditioned on the support samples and  $\epsilon \sim \mathcal{N}(0,I)$ , by first acquiring an overfit
Table 2: Benefit of diffusion model over (non-)generative models on miniImagenet.
Method	miniImagenet
1-shot	5-shot
w/o generative model [5]	63.17±0.23	79.26±0.17
w/ MLP	64.15±0.21	80.23±0.15
w/ Transformer	64.97±0.21	81.28±0.14
w/ VAE	64.45±0.22	80.13±0.15
w/ Normalizing flows	65.11±0.22	81.96±0.17
w/ Diffusion	66.63±0.21	83.48±0.15
ted prototype  $\mathbf{z}^*$  and then using it as the ground truth to train VAE and normalizing flows in the meta-training stage. The experimental results reported in Table 2 show that the diffusion model outperforms all variants in terms of accuracy. Specifically, our diffusion model improves accuracy by  $2.18\%$  compared to VAE and  $1.52\%$  compared to normalizing flows. Furthermore, we reconstructed the overfitted prototype using different generative models in Figure 5. The vanilla prototype focuses solely on the bird's features and overlooks the finer details of the robin. In contrast, the overfitted prototype can emphasize specific features such as the tail and beak, resulting in better discrimination of the robin from other bird species. While VAE and normalizing flows can generate prototypes of specific parts, our diffusion method can generate prototypes that more closely resemble the overfitted prototype, leading to improved performance. Our findings suggest that diffusion models hold promise as a more practical approach for few-shot learning, as they can model complex distributions and produce informative prototypes.
Effect of the residual prototype. The incorporation of residual prototypes in ProtoDiff presents a viable approach for reducing computational costs. This is attributable to the fact that the residual
prototype exclusively encapsulates the disparity between the overfitted prototype and the original prototype, while the latter can be readily reused in each episode. Consequently, the calculation of the overfitted prototype is only necessitated once during the meta-training stage. Notably, the residual prototype often encompasses numerous zero values, thereby further expediting the training process. Table 6a illustrates the superior accuracy achieved by ProtoDiff with residual prototype learning in comparison to both the vanilla model and ProtoDiff without the residual prototype. The training progress of ProtoDiff is visually represented in Figure 6b, demonstrating its accelerated training capabilities when compared to alternative methods. These results indicate the effectiveness of residual prototype learning in capturing task-specific information previously unaccounted for by the vanilla prototypes. Thus, the inclusion of residual prototype learning in ProtoDiff not only expedites the training process but also serves as a straightforward and effective approach to enhance the overall performance of ProtoDiff.
(a) Accuracy  $(\%)$  on various settings.
Method	miniImagenet
1-shot	5-shot
Vanilla	63.17±0.23	79.26±0.17
ProtoDiff w/o residual	64.75±0.22	80.76±0.16
ProtoDiff w/ residual	66.63±0.21	83.48±0.15
(b) Optimization curves.
Figure 6: Effect of residual prototype on miniImagenet. Utilizing ProtoDiff with a residual prototype not only accelerates the training process but also surpasses the performance of both the vanilla prototype and the non-residual prototype.
Analysis of uncertainty. In the process of estimating the log-likelihood of input data, the choice of the number of Monte Carlo samples becomes a critical hyperparameter. Typically, a larger number of samples tends to provide a more accurate approximation and better generalization to unforeseen prompts. In our experiments, we employed Meta-Baseline and FEAT [58] with ProtoDiff, incorporating the sampling of multiple prototypes in the final phase of the diffusion
Table 3: Analysis of uncertainty. Increasing the number of Monte Carlo samples provides a good improvement on miniImagenet.
Meta-baseline + ProtoDiff	FEAT + ProtoDiff
1-shot	5-shot	1-shot	5-shot
1	66.63±0.21	83.48±0.15	68.97±0.25	85.16±0.17
10	68.02±0.17	84.95±0.10	70.18±0.15	86.53±0.12
20	68.91±0.15	85.74±0.10	61.07±0.15	87.14±0.12
50	69.14±0.15	86.12±0.10	72.25±0.15	88.32±0.12
100	69.13±0.15	86.07±0.10	72.21±0.15	88.37±0.12
1000	69.11±0.15	86.11±0.10	72.12±0.15	88.13±0.12
process. The results of these experiments are provided in Table 3. Notably, we observe a substantial performance boost with an increase in the number of samples. For instance, a sample size of 50 results in a significant enhancement in 1-shot accuracy, raising it from 66.63 for Meta-Baseline + ProtoDiff to 72.25 for FEAT + ProtoDiff.
Visualization of the prototype adaptation process. We undertake a qualitative analysis of the adaptation process of our ProtoDiff framework during the meta-test time. We employ a 3-way 5-shot setup on the miniImageNet dataset for more comprehensive visualization of the adaptation process. To achieve this, we depict the vanilla prototypes, the diffused prototypes at various timesteps, and instances from the support and query sets. All examples are first normalized to a length of 1 and
Table 5: Cross-domain few-shot comparison on four benchmarks. ProtoDiff is a consistent top-performer.
Method	CropDiseases	EuroSAT	ISIC	ChestX
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
ProtoNet [42]	57.57±0.51	79.72±0.67	54.19±0.57	73.29±0.71	29.62±0.32	39.57±0.57	22.30±0.25	24.05±1.01
GNN [40]	57.19±0.50	83.12±0.40	54.61±0.50	78.69±0.40	30.14±0.30	42.54±0.40	21.94±0.20	23.87±0.20
AFA [20]	67.61±0.50	88.06±0.30	63.12±0.50	85.58±0.40	33.21±0.30	46.01±0.40	22.92±0.20	25.02±0.20
ATA [51]	67.47±0.50	90.59±0.30	61.35±0.50	83.75±0.40	33.21±0.30	44.91±0.40	22.10±0.20	24.32±0.20
HVM [11]	65.13±0.45	87.65±0.31	61.97±0.34	74.88±0.45	33.87±0.35	42.05±0.34	22.94±0.47	27.15±0.45
This paper	68.93±0.31	90.15±0.31	65.93±0.34	87.25±0.35	34.97±0.33	45.65±0.31	23.01±0.45	28.54±0.41
subsequently projected onto a two-dimensional space using t-SNE [48]. Figure 7 displays the instance representations from both the query and support sets that are cross and circle, the vanilla prototypes that are denoted by a four-point star symbol, an the diffused prototypes that are denoted by stars. Arrows depict the adaptation process of the diffused prototypes, and each color corresponds to each respective class. Initially, the diffused prototypes are randomly allocated based on a Gaussian distribution, positioning them considerably from the optimal prototypes for each class. As the diffusion process contin
ues, these diffused prototypes incrementally approach the optimal prototypes, resulting in more distinctive representations for different classes. In contrast, the vanilla prototypes lack distinguishability as they are computed using a single deterministic function based solely on the support set. This observation underscores the importance of the prototype adaptation process in attaining improved performance in few-shot classification tasks.
Within-domain few-shot. We evaluate our method on few-shot classification within domains, in which the training domain is consistent with the test domain. The results are reported in Table 4.
In this comparison, we apply SetFeat [1] with ProtoDiff to experiment since SetFeat is the current state-of-the-art model based on prototype-based meta-learning. Our ProtoDiff consistently achieves state-of-the-art performance on all datasets under various shots. The better performance confirms that ProtoDiff can achieve more informative and higher quality prototypes to perform better for few-shot learning within domains.
Figure 7: t-SNE visualization of ProtoDiff adaptation process on 3-way 5-shot task.
Table 4: Within-domain few-shot comparison on miniImagenet and tieredImagenet. ProtoDiff performs well on all datasets.
Method	miniImagenet	tieredImagenet
1-shot	5-shot	1-shot	5-shot
TapNet [50]	61.65±0.15	76.36±0.10	63.08±0.15	80.26±0.12
CTM [26]	62.05±0.55	78.63±0.06	64.78±0.11	81.05±0.52
MetaOptNet [24]	62.64±0.61	78.63±0.46	65.81±0.74	81.75±0.53
Meta-Baseline [5]	63.17±0.23	79.26±0.17	68.62±0.27	83.74±0.18
CAN [18]	63.85±0.48	79.44±0.34	69.89±0.51	84.23±0.37
Meta DeepBDC [53]	67.34±0.43	84.46±0.28	72.34±0.49	87.31±0.32
SUN [9]	67.80±0.45	83.25±0.30	72.99±0.50	86.74±0.33
SetFeat [1]	68.32±0.62	82.71±0.46	73.63±0.88	87.59±0.57
This paper	71.25±0.45	83.95±0.45	75.97±0.75	88.75±0.18
Cross-domain few-shot. We also evaluate our method in the cross-domain few-shot classification, where the training domain is different from test domain. Table 5 shows the evaluation of ProtoDiff on four datasets, with 5-way 1-shot and 5-way 5-shot settings. ProtoDiff also achieves competitive performance on all four cross-domain few-shot learning benchmarks for each setting. On EuroSAT [15], our model obtains high recognition accuracy under different shot configurations, outperforming the second best method, AFA [20], by a significant margin of  $2.82\%$  for 5-way 1-shot. Even on the most challenging ChestX [52], which has a considerable domain gap with miniImageNet, our model achieves  $28.54\%$  accuracy in the 5-way 5-shot setting, surpassing the second-best HVM [11] by  $1.39\%$ . The consistent improvement across all benchmarks and settings confirms the effectiveness of ProtoDiff for cross-domain few-shot learning.
Few-task few-shot. We evaluate ProtoDiff on the four different datasets under 5-way 1-shot and 5-way 5-shot in Table 6. In this comparison, we apply MLTI [56] with ProtoDiff to experiment. Our method achieves state-of-the-art performance on all four few-task meta-learning benchmarks under each setting. On miniImagenet-S, our model achieves  $44.15\%$  under 1-shot, surpassing the
Table 6: Few-task few-shot comparison on four benchmarks. ProtoDiff consistently ranks among the top-performing models.
miniImagenet-S	ISIC	Dermnet-S	Tabular Murris
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
ProtoNet [42]	36.26±0.70	50.72±0.70	58.56±1.01	66.25±0.96	44.21±0.75	60.33±0.70	80.03±0.90	89.20±0.56
Meta-Dropout [23]	38.32±0.75	52.53±0.70	58.40±1.16	67.32±0.95	44.30±0.81	60.86±0.75	78.18±0.90	89.25±0.56
MetaMix [55]	39.80±0.87	53.35±0.88	59.66±1.10	68.97±0.80	46.06±0.85	62.97±0.70	79.56±0.91	88.88±0.60
Meta Interpolation [25]	40.28±0.70	53.06±0.72	-	-	-	-	-	-
MLTI [56]	41.36±0.73	55.34±0.72	62.82±1.09	71.52±0.89	49.38±0.85	65.19±0.73	81.89±0.88	90.12±0.59
MetaModulation [45]	43.21±0.73	57.26±0.72	65.61±1.09	76.40±0.89	50.45±0.84	67.05±0.73	83.13±0.89	91.23±0.57
This paper	44.75±0.70	58.18±0.72	66.13±1.04	76.23±0.81	51.53±0.85	67.97±0.74	84.03±0.89	91.35±0.57
best method MetaModulation [45], by a margin of  $1.54\%$ . Even on the most challenging DermNet-S, which forms the largest dermatology dataset, our model delivers  $51.53\%$  on the 5-way 1-shot setting. The consistent improvements on all benchmarks under various configurations confirm that our approach is also effective for few-task meta-learning.
Limitations. Naturally, our proposal also comes with limitations. Firstly, the diffusion model employed in our approach necessitates a substantial number of timesteps to sample the diffused prototype during the meta-test stage. Although we somewhat alleviate this issue by utilizing DDIM [44] sampling, it still requires more computational resources than vanilla prototypes. Secondly, obtaining the overfitted prototype involves fine-tuning, which inevitably leads to increased training time. While this fine-tuning step contributes to the effectiveness of our model, it comes with an additional computational cost. In meta-training and meta-testing ProtoDiff is slower by factors of ProtoNet  $5 \times$  and  $15 \times$  in terms of wall-clock times per task. More detailed time numbers are provided in the Appendix. As part of future work, we will investigate and address these limitations to further enhance the applicability and efficiency of our approach.
We proposed ProtoDiff, a novel approach to prototype-based meta-learning that utilizes a task-guided diffusion model during the meta-training phase to generate efficient class representations. We addressed the limitations of estimating a deterministic prototype from a limited number of examples by optimizing a set of prototypes to accurately represent individual tasks and training a task-guided diffusion process to model each task's underlying distribution. Our approach considers both probabilistic and task-guided prototypes, enabling efficient adaptation to new tasks while maintaining the informativeness and scalability of prototypical networks. Extensive experiments on three distinct few-shot learning scenarios: within-domain, cross-domain, and few-task few-shot learning validated the effectiveness of ProtoDiff. Our results demonstrated significant improvements in classification accuracy compared to state-of-the-art meta-learning techniques, highlighting the potential of task-guided diffusion in augmenting few-shot learning and advancing meta-learning.
[1] A. Afrasiyabi, H. Larochelle, J.-F. Lalonde, and C. Gagné. Matching feature sets for few-shot image classification. In CVPR, pages 9014-9024, 2022.
[2] K. R. Allen, E. Shelhamer, H. Shin, and J. B. Tenenbaum. Infinite mixture prototypes for few-shot learning. In ICML, pages 232-241, 2019.
[3] K. Cao, M. Brbic, and J. Leskovec. Concept learners for few-shot learning. *ICLR*, 2020.