SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[4] T. Cao, M. Law, and S. Fidler. A theoretical analysis of the number of shots in few-shot learning. arXiv preprint arXiv:1909.11722, 2019.
[5] Y. Chen, Z. Liu, H. Xu, T. Darrell, and X. Wang. Meta-baseline: Exploring simple meta-learning for few-shot learning. In ICCV, pages 9062–9071, 2021.
[6] Dermnet. Dermnet dataset. https://dermnet.com/. 2016.
[7] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8794, 2021.
[8] N. Ding, X. Wang, Y. Fu, G. Xu, R. Wang, P. Xie, Y. Shen, F. Huang, H.-T. Zheng, and R. Zhang. Prototypical representation learning for relation extraction. arXiv preprint arXiv:2103.11647, 2021.
[9] B. Dong, P. Zhou, S. Yan, and W. Zuo. Self-promoted supervision for few-shot transformer. In ECCV, pages 329-347. Springer, 2022.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[11] Y. Du, X. Zhen, L. Shao, and C. G. M. Snoek. Hierarchical variational memory for few-shot learning across domains. In ICLR, 2022.
[12] Z. Erkoç, F. Ma, Q. Shan, M. Nießner, and A. Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. arXiv preprint arXiv:2303.17015, 2023.
[13] V. Garcia and J. Bruna. Few-shot learning with graph neural networks. In ICLR, 2018.
[14] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, J. R. Smith, K. Saenko, T. Rosing, and R. Feris. A broader study of cross-domain few-shot learning. In European Conference on Computer Vision, 2020.
[15] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[16] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
[17] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[18] R. Hou, H. Chang, M. Bingpeng, S. Shan, and X. Chen. Cross attention network for few-shot classification. In NeurIPS, pages 4005-4016, 2019.
[19] V. T. Hu, D. W. Zhang, Y. M. Asano, G. J. Burghouts, and C. G. M. Snoek. Self-guided diffusion models. In CVPR, 2023.
[20] Y. Hu and A. J. Ma. Adversarial feature augmentation for cross-domain few-shot classification. In ECCV, pages 20-37. Springer, 2022.
[21] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[22] G. Kwon and J. C. Ye. Diffusion-based image translation using disentangled style and content representation. arXiv preprint arXiv:2209.15264, 2022.
[23] H. B. Lee, T. Nam, E. Yang, and S. J. Hwang. Meta dropout: Learning to perturb latent features for generalization. In ICLR, 2020.
[24] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In CVPR, pages 10657-10665, 2019.
[45] W. Sun, Y. Du, X. Zhen, F. Wang, L. Wang, and S. Cees, G.M. Metamodulation: Learning variational feature hierarchies for few-shot learning with fewer tasks. In ICML. PMLR, 2023.
[46] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096, 2019.
[47] P. Tschandl, C. Rosendahl, and H. Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[48] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NeurIPS, pages 6000–6010, 2017.
[50] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In NeurIPS, pages 3630-3638, 2016.
[51] H. Wang and Z.-H. Deng. Cross-domain few-shot classification via adversarial task augmentation. arXiv preprint arXiv:2104.14385, 2021.
[52] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In CVPR, pages 2097-2106, 2017.
[53] J. Xie, F. Long, J. Lv, Q. Wang, and P. Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In CVPR, pages 7972-7981, 2022.
[54] S. Yang, L. Liu, and M. Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101.06395, 2021.
[55] H. Yao, L.-K. Huang, L. Zhang, Y. Wei, L. Tian, J. Zou, J. Huang, et al. Improving generalization in meta-learning via task augmentation. In ICML, pages 11887-11897. PMLR, 2021.
[56] H. Yao, L. Zhang, and C. Finn. Meta-learning with fewer tasks through task interpolation. arXiv preprint arXiv:2106.02695, 2021.
[57] H. Ye, H. Hu, D. Zhan, and F. Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020.
[58] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8808-8817, 2020.
[59] S. Yoon, J. Seo, and J. Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In ICML, 2019.
[60] S. W. Yoon, J. Seo, and J. Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In ICML, pages 7115-7123. PMLR, 2019.
[61] X. Zhen, Y. Du, H. Xiong, Q. Qiu, C. G. M. Snoek, and L. Shao. Learning to learn variational semantic memory. In NeurIPS, 2020.
We describe the detailed algorithm ms for meta-training and meta-test of ProtoDiff as following Algorithm 1 and 2, respectively:
Algorithm 1 Meta-training phase of ProtoDiff.
Input:  $p(T)$ : distribution over tasks;  $\theta$ : diffusion parameters;  $\phi$ : feature extractor parameters;  $T$ : diffusion steps.
Output:  $\mathbf{z}_{\theta}$ : diffusion network,  $f_{\phi}$ : feature extractor.
1: repeat
2: Sample batch of tasks  $\mathcal{T}^i\sim p(\mathcal{T})$
3: for all  $\mathcal{T}^i$  do
4: Sample support and query set  $\{\mathcal{S}^i,\mathcal{Q}^i\}$  from  $\mathcal{T}^i$
5: Compute the vanilla prototype  $\tilde{\mathbf{z}}^i = f(S^i)$
6: repeat
7: Take some gradient step on  $\nabla \mathcal{L}_{\mathrm{CE}}(\mathcal{S}^i,\mathcal{Q}^i)$ , updating  $\phi^i$
8: until  $\mathcal{L}_{\mathrm{CE}}(S_i,\mathcal{Q}_i)$  converges
9: Compute the overfitted prototype by using the updated  $\phi^i$ ,  $\mathbf{z}^{i,*} = f_{\phi^i}(S_i)$
10:  $t\sim \text{Uniform}(1\dots T)$
11:  $\epsilon = \mathcal{N}(\mathbf{0},\mathbf{1})$
12:  $\beta_{t} = \frac{10^{-4}(T - t) + 10^{-2}(t - 1)}{T - 1},\alpha_{t} = 1 - \beta_{t},\bar{\alpha}_{t} = \Pi_{k = 0}^{k = t}\alpha_{k}$
13:  $\hat{\mathbf{z}}_t^i = \sqrt{\bar{\alpha}_t} (\mathbf{z}^{i,*} - \tilde{\mathbf{z}}^i) + \sqrt{1 - \bar{\alpha}_t^2}\epsilon$
14: Compute diffusion loss  $\mathcal{L}_{\mathrm{diff}}$  with Equation (9) and the final loss  $\mathcal{L}_{\mathcal{T}^i}$  with Equation (10)
15: Update  $\{\phi, \theta\} \gets \{\phi, \theta\} - \beta \nabla_{\{\phi, \theta\}} \sum_{\mathcal{T}^i \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}^i}$  using query data of each task.
16: end for
17: until  $f_{\phi}$  and  $\mathbf{z}_{\theta}$  converges
Algorithm 2 Meta-test phase of ProtoDiff.
Input:  $\tau = \{S, \mathcal{Q}\}$ : meta-test task,  $\mathbf{z}_{\theta}$ : trained diffusion network parameters,  $f_{\phi}$ : trained feature extractor network parameters,  $T$ : diffusion steps.
1:  $\mathbf{z}_T\sim \mathcal{N}(\mathbf{0},\mathbf{I})$
2: Compute vanilla prototype  $\tilde{\mathbf{z}}$  with support set  $f(S)$
3: for  $t = T, \dots, 1$  do
4:  $\epsilon = N(\mathbf{0},\mathbf{1})$
5:  $\beta_{t} = \frac{10^{-4}(T - t) + 10^{-2}(t - 1)}{T - 1},\alpha_{t} = 1 - \beta_{t},\bar{\alpha}_{t} = \Pi_{k = 0}^{k = t}\alpha_{k}$
6:  $\mathbf{z}_{t - 1} = \mathbf{z}_{\theta}(\mathbf{z}_t,\tilde{\mathbf{z}},t)$
7: end for
8: Compute the final prediction  $\mathbf{y}^q$  by with Equation (1) based on  $\mathbf{z}_0 + \tilde{\mathbf{z}}$
B Per-task prototype overfitting architecture
To enhance our comprehension of the Per-task prototype overfitting part, we propose a succinct architectural representation depicted in Figure 8. The initial step entails the computation of the conventional prototypes  $\tilde{\mathbf{z}}$  for a meta-training task. Subsequently, Equation (1) is employed to calculate the predictions for the query sample. The backbone's parameters are subsequently updated through  $I$  iterations. Through the utilization of parameters from the final iteration, we ultimately obtain the prototypes  $\mathbf{z}^*$  that exhibit overfitting characteristics.
C Residual prototype learning architecture
In order to gain a more comprehensive understanding of our residual prototype learning, we have crafted a succinct architecture diagram illustrated in Figure 9. Our proposition involves the prediction of the prototype update, denoted as  $\mathbf{z}^{*} - \tilde{\mathbf{z}}$ , instead of directly predicting the overfitted prototype
Figure 8: Per-task prototype overfitting.
$\mathbf{z}^{*}$ . This distinctive approach also allows us to initialize ProtoDiff with the capability to perform the identity function, achieved by assigning zero weights to the decoder. Notably, we have discovered that the amalgamation of a global residual connection and the identity initialization substantially expedites the training process. By harnessing this mechanism, we have successfully enhanced the performance of ProtoDiff in the context of few-shot learning tasks.
Figure 9: Residual prototype learning.
Within-domain few-shot. For this setting we focus on 5-way 1-shot/5-shot tasks, which aligns with previous research [42]. The within-domain few-shot experiments are performed on three datasets: miniImagenet [50] tieredImagenet [35], and ImageNet-800 [5]. miniImagenet consists of 100 randomly selected classes from ILSVRC-2012 [37], while tieredImagenet is composed of 608 classes that are grouped into 34 high-level categories. We measure the accuracy of 600 tasks randomly
sampled from the meta-test set to evaluate the performance. Following [5], we also evaluate our model on ImageNet-800, a dataset obtained by randomly dividing the 1,000 classes of ILSVRC-2012 into 800 base classes and 200 novel classes. The base classes consist of images from the original training set, while the novel classes comprise images from the original validation set.
Cross-domain few-shot. In the 5-way 5-shot cross-domain few-shot classification experiments, the training domain is miniImagenet [50], and the testing is conducted on four different domains. These domains are CropDisease [30], which contains plant disease images; EuroSAT [15], a collection of satellite images; ISIC2018 [47], consisting of dermoscopic images of skin lesions, and ChestX [52], a dataset of X-ray images.
Few-task few-shot. Few-task few-shot learning [56] challenges the common meta-training assumption of having many tasks available. We conduct experiments on four few-task meta-learning challenges, namely miniImagenet-S [50], ISIC [29], DermNet-S [6], and Tabular Murris [3]. To reduce the number of tasks and make it comparable to previous works, we followed [56] by limiting the number of meta-training classes to 12 for miniImagenet-S, with 20 meta-test classes. ISIC [29] involves classifying dermoscopic images across nine diagnostic categories, with 10,015 images available for training in eight different categories, of which we selected four as meta-training classes. DermNet [6], which contains over 23,000 images of skin diseases, was utilized to construct Dermnet-S by selecting 30 diseases as meta-training classes, following [56]. Finally, Tabular Murris [3], which deals with cell type classification across organs and includes nearly 100,000 cells from 20 organs and tissues, was utilized to select 57 base classes as meta-training classes, again following the same approach as [56].
In our within-domain experiments, we utilize a Conv-4 and ResNet-12 backbone for miniImagenet and tieredImagenet. A ResNet-50 is used for ImageNet-800. We follow the approach described in [5] to achieve better performance and initially train a feature extractor on all the meta-training data without episodic training. We use the SGD optimizer with a momentum of 0.9, a learning rate starting from 0.1, and a decay factor of 0.1. For miniImagenet, we train for 100 epochs with a batch size of 128, where the learning rate decays at epoch 90. For tieredImagenet, we train for 120 epochs with a batch size of 512, where the learning rate decays at epochs 40 and 80. Lastly, for ImageNet-800, we train for 90 epochs with a batch size of 256, where the learning rate decays at epochs 30 and 60. The weight decay is 0.0005 for ResNet-12 and 0.0001 for ResNet-50. Standard data augmentation techniques, including random resized crop and horizontal flip, are applied. For episodic training, we use the SGD optimizer with a momentum of 0.9, a fixed learning rate of 0.001, and a batch size of 4, meaning each training batch consists of 4 few-shot tasks to calculate the average loss. For our cross-domain experiments, we use a ResNet-10 backbone to extract image features, which is a common choice for cross-domain few-shot classification [14, 57]. The training configuration for this experiment is the same as the within-domain miniImagenet training. For few-task few-shot learning, we follow [56] using a network containing four convolutional blocks and a classifier layer. Each block comprises a 32-filter  $3 \times 3$  convolution, a batch normalization layer, a ReLU nonlinearity, and a  $2 \times 2$  max pooling layer. All experiments are performed on a single A100 GPU, each taking approximately 20 hours. We will release all our code.
F Visualization of diffusion process
The ProtoDiff method utilizes a task-guided diffusion model to generate prototypes that provide efficient class representations, as discussed in the previous section. To better understand the effectiveness of our proposed approach, we provide a visualization by Grad-Cam [41] in Figure of the diffusion process, demonstrating how ProtoDiff gradually aggregates towards the desired class prototype during meta-training. The vanilla prototype is shown in the first row on the left, which does not exclusively focus on the guitar. In contrast, the overfitted prototype in the second row on the left provides the highest probability for the guitar. ProtoDiff, with the diffusion process, randomly selects certain areas to add noise and perform diffusion, resulting in a prototype that gradually moves towards the guitar with the highest probability at  $t = 0$ . Moreover, ProtoDiff with residual learning produces a more precise prototype. The comparison between these different prototypes demonstrates
Diffusion process with residual learning
Figure 10: Visualization of the diffusion process. The first row on the right shows the vanilla prototype, which does not exclusively focus on the guitar. In contrast, the overfitted prototype in the second row on the right provides the highest probability for the guitar. ProtoDiff randomly selects certain areas to predict during the diffusion process, with the lowest probability at the beginning time step. As time progresses, the prototype gradually aggregates towards the guitar, with the highest probability at  $t = 0$ . In comparison, ProtoDiff with residual learning produces a more precise prototype.
the effectiveness of the ProtoDiff diffusion process in generating a more accurate and informative prototype for few-shot learning tasks.
G More results
Effect of transformer structure Our ProtoDiff model is constructed using a design inspired by GPT-2. This includes a 12-layer transformer, a 512-dimensional linear transformation, an attention mechanism with 16 heads, and an MLP with a hidden dimensionality 512. These configurations are in line with GPT-2's default parameters. The configuration files can be accessed in our code repository for more detailed parameter setup. Our experiments in the table 7a and 7b highlight that our model achieves optimal performance with these settings.
Table 7: Effect of transformer structures
(a) Results on the different transformer structures.
Structures	miniImagenet
1-shot	5-shot
3-layers	62.17±0.25	78.93±0.17
6-layers	63.25±0.22	79.63±0.15
9-layers	65.21±0.21	80.13±0.18
12-layers	66.63±0.21	83.48±0.15
15-layers	64.15±0.25	80.93±0.18
(b) Results on the different heads numbers.
Nodes numbers	miniImagenet
1-shot	5-shot
node = 1	63.28±0.24	79.97±0.15
node = 4	64.43±0.22	80.13±0.14
node = 8	65.91±0.23	81.91±0.16
node = 16	66.63±0.21	83.48±0.15
Effect of different time steps We have selected the diffusion time step T for the diffusion process to be 100. We've adopted the DDIM sampling strategy to accelerate prediction with. This effectively reduces the total sample requirement to just 10. We've conducted comparative experiments using varying diffusion times and intermediate intervals. As presented in the tables 8 and 9, we observe that as the diffusion timesteps increase, both the performance and the inference time increase. Simultaneously, when is increased, the inference time decreases, making the process more efficient.
Effect of more support images To prove more support images during meta-training to obtain more accurate prototypes, we conducted an experimental comparison using different numbers of support sets during meta-training. The results in the table 10 illustrate that augmenting the number of support images for each class during the meta-training phase enhances performance across various shots.
Table 8: Results with the different diffusion timesteps on miniImagenets.
dim(τ) = 10, Timesteps	1-shot	5-shot	1-shot inference time	5-shot inference time
T =10	63.98±0.24	80.12 ±0.15	1 ms	2 ms
T = 50	65.93±0.21	82.93±0.17	5 ms	7 ms
T = 100	66.63±0.21	83.48±0.15	9 ms	14 ms
T = 500	66.65±0.23	83.59±0.14	53 ms	93 ms
T = 1000	66.74±0.20	83.97±0.17	102 ms	192 ms
Table 9: Results with the different diffusion timesteps on miniImagenets.
dim(τ), T = 100	1-shot	5-shot	Speed
τ =1	66.78±0.21	83.62 ±0.14	99 ms
τ =5	66.15±0.23	83.44±0.16	45 ms
τ =10	66.63±0.21	83.48 ±0.15	9 ms
τ =20	66.12±0.21	82.79 ±0.13	5 ms
τ =100	63.15±0.23	81.27 ±0.14	1 ms
Meta-training and meta-test wall-clocks times per task We have compiled the wall-clock time for both the meta-training and meta-testing phases of ProtoDiff and compared these against the respective times for ProtoNet in the Tables 11a and 11b. In meta-training and meta-testing wall-clock times per task, our ProtoDiff is slower by factors of ProtoNet  $5 \times$  and  $15 \times$ , respectively. As part of future work, we will investigate and address this limitation to further enhance the efficiency of our approach.
Rationale for Conditioning We utilize the "vanilla" prototype as a guiding condition, gradually enabling the diffusion model to generate class-specific prototypes. We also explored two alternative strategies for conditioning in the table 12: one without any conditions and another with learned class embeddings. When not conditioned, the performance tends to be arbitrary due to the absence of class-specific cues during diffusion. On the other hand, employing learned class embeddings as a condition yielded subpar results compared to the vanilla prototype, potentially due to outliers in each class's learned embeddings.
Results of our pre-trained model We also give the results of our own pre-trained model before applying ProtoDiff to provide a clear comparison and demonstrate our method's improvements. We have prepared the results of our pre-trained model (trained with CE loss on the whole training set) and will present it in table 13.
Table 10: Results of the different support sets for each class.
Structures	miniImagenet
1-shot	5-shot
1	66.63±0.21	73.12±0.15
5	70.25±0.22	83.48±0.13
8	73.91±0.21	84.72±0.13
16	82.21±0.23	84.98±0.17
Table 11: Meta-training and meta-test wall-clock times on each task.
(a) Meta-training.
Meta-training	miniImagenet
1-shot	5-shot
ProtoNet	0.6 ms	1.2 ms
ProtoDiff	3 ms	5.3 ms
(b) Meta-test.
Meta-test	miniImagenet
1-shot	5-shot
ProtoNet	0.6 ms	1.2 ms
ProtoDiff	9 ms	14 ms
Table 12: Results of different conditions.
Conditions	miniImagenet
1-shot	5-shot
w/o conditions	20.27±0.25	22.32±0.15
Learnt class embeddings	66.17 ±0.22	81.17±0.16
Vanilla prototype	66.63 ±0.21	83.48±0.15
Table 13: Results of our pre-trained model on different datasets
miniImagenet	tieredImagenet	Imagenet-800
1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
Pretrained from Chen et al. [+]	Classifier-Baseline	58.91±0.23	77.76±0.17	68.07±0.29	83.74±0.18	86.07±0.21	96.14±0.08
Pretrained from Chen et al. [+]	ProtoDiff	66.63±0.21	83.48±0.15	72.95±0.24	85.15±0.18	92.13±0.20	98.21±0.08
Pretrained from ours	Classifier-Baseline	58.75±0.21	77.86±0.17	68.15±0.28	83.95±0.17	85.97±0.22	96.03±0.09
Pretrained from ours	ProtoDiff	66.49±0.25	83.51±0.14	73.01±0.24	85.72±0.17	92.05±0.21	98.11±0.18