SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compared our approach to the following baselines: Euclidean is the standard Euclidean centroid calculated as the mean of the seeds. Normalized Euclidean is the same as Euclidean, but the centroid is projected to the sphere induced by the  $\chi$  distribution. Sphere Projection first normalizes the seed to a sphere with radius  $r = Mode(\chi)$ , then finds the centroid on a sphere by optimizing a point that minimizes the sum of geodesic paths between the seeds to the centroid, as presented in [9]. See supplemental for more details. NAO-path and NAO-centroid are our methods presented in sections 4.1 and 4.2, respectively. The high accuracy and low FID levels of NAO in Table 1 demonstrate that our interpolation approach outperforms other baselines in terms of image quality and content. We further put these results to test in downstream tasks (in Sec. 5.1-5.3).
Acc	FID
Interpolation methods
LERP	0.0	50.59
SLERP [48]	30.41	18.44
NAO-path (ours)	51.59	6.78
Centroid computation methods
Euclidean	0.0	54.88
Normalized Euclidean	27.95	37.04
Sphere Projection	40.81	14.28
NAO-centroid (ours)	67.24	5.48
Table 1: Comparing FID and accuracy of images generated by SD through sampling from different interpolation and centroid computation methods.
5.1 Rare-concept generation
Following [43] we compared different centroid optimization strategies in rare-concept generation.
Dataset. The evaluation is performed on ImageNet1k classes ordered by their prevalence in the LAION2B dataset [45]. LAION2B [45] is a massive "in the wild" dataset that is used for training foundation diffusion models (e.g. Stable Diffusion [38]).
Compared Methods. We conducted a comparative evaluation between different centroid estimation strategies. SeedSelect [43] is a baseline method where a seed is randomly sampled and no centroid is calculated. Other baselines were presented at the beginning of Section 5.
ImageNet1k in LAION2B
Methods	Many n=235 #>1M	Med n=509 1M>#>10K	Few n=256 10K>#	Total Acc	FID	\( {\widehat{T}}_{\text{Init }} \) (sec)	\( {\widehat{T}}_{\text{Opt }} \) (sec)
SeedSelect [43]	97.8	95.8	76.1	91.3	6.5	0	298
Euclidean	0.0	0.0	0.0	0.0	51.1	0	-
Euclidean + SeedSelect	0.0	0.0	0.0	0.0	65.7	0	inf
Normalized Euclidean	45.3	39.8	20.1	36.0	45.8	0	-
Normalized Euclidean + SeedSelect	88.1	84.8	70.9	82.0	19.2	0	285
Sphere projection	52.0	48.9	26.5	44.0	12.3	0.1	-
Sphere projection + SeedSelect	97.1	95.1	74.2	90.2	8.6	0.1	72
NAO-centroid (ours)	56.8	55.6	41.1	52.2	9.8	25	-
NAO-centroid + SeedSelect (ours)	98.5	96.9	85.1	94.3	6.4	25	29
Table 2: Image generation quality measured by the accuracy of a pre-trained classifier. Average per-class accuracy is reported separately for the head, tail, and middle classes. NAO produces the best performance in terms of image quality and is also  $\times 10$  faster.
(a)
Model	miniImageNet
5-way 5-shot	CIFAR-FS
5-way 5-shot
Label-Halluc [23]	67.04 ± 0.7	89.37±0.6
FeLMI [39]	86.08±0.4	89.47±0.5
SEGA [59] †	79.03±0.2	86.00±0.2
SVAE [58] ‡	80.70±0.2	78.89±0.3
Textual Inversion [20]* ‡	85.44±3.9	-
Stable Diffusion [38] ‡	85.05±0.5	90.87±0.5
DiffAlign [40] ‡	88.63±0.3	91.96±0.5
SeedSelect [43]	92.08±0.7	94.87±0.4
NAO-centroid (ours)	93.21±0.6	94.85±0.5
(b) Model	CUB 5-way 5-shot
TriNet [12]	84.10±0.4
FEAT [60]	82.90±0.2
DeepEMD [61]	88.69±0.5
MultiSem [46] †	82.9±n/a
SEGA [59] †	90.85±0.2
Stable Diffusion* [38]‡	90.61±0.5
SeedSelect [43]	96.01±0.4
NAO-centroid (ours)	97.92±0.2
(c)
Model	ImageNet-LT
CE	41.6
MetaSAug [27]	47.4
smDragon [42]	47.4
CB LWS [24]	47.7
DRO-LT [44]	53.5
Ride [56]	55.4
PaCO [13]	53.5
DRAGON [42] †	57.0
VL-LTR [52] !	70.1
Stable Diffusion [43] ‡	56.4
SeedSelect [43]‡	74.9±0.5
NAO-centroid (ours)	78.9±0.1
Table 3: (a)-(b) Few-shot recognition. Comparing NAO to prior work on few-shot learning benchmarks. We report our results with  $95\%$  confidence intervals on meta-testing split of the dataset. (c) Long-tail recognition. Comparing NAO to prior work on long-tail learning benchmark. Values are accuracy, obtained with a ResNet-50 backbone. * denote results provided by [43]. † for multi-modal methods that use class labels as additional information, ‡ for multi-modal methods that were also pre-trained on external datasets, and! for multi-modal methods that further finetuned foundation models. Our approach achieves the best results on all benchmarks.
miniImageNet	CIFAR-FS	CUB
TInit	TOpt	TTotal	TInit	TOpt	TTotal	TInit	TOpt	TTotal
SeedSelect [43]	-	276	276 sec	-	228 sec	228 sec	-	312 sec	312 sec
NAO-centroid (ours)	28 sec	25 sec	53 sec	29 sec	21 sec	50 sec	31 sec	29 sec	60 sec
Table 4: Comparing convergence time between SeedSelect [43] alone and SeedSelect initialized with seeds found using NAO. Note the five-fold reduction in runtime for our method.
Experimental Protocol. For every class in ImageNet, we randomly sampled subsets of training images, calculated their centroid in seed space, and generated an image using Stable Diffusion directly or as input to SeedSelect [43]. The class label was used as the prompt. This process is repeated until 100 images are generated for each class. We then used a SoTA pre-trained classifier [53] to test if the generated images are from the correct class or not (more details can be found in the supplementary). We use this measure to evaluate the quality of the generated image, verifying that a strong classifier correctly identifies the generated image class. We also report Mean FID score between the generated images and the real images, mean centroid initialization time  $\hat{T}_{Init}$  and mean SeedSelect optimization time until convergence  $T_{Opt}$  on a single NVIDIA A100 GPU. The results summarized in Table 2 show that our NAO method substantially outperforms other baselines, both in accuracy and in FID score. Furthermore, NAO gives a better initialization point to SeedSelect [43], yielding significantly faster convergence without sacrificing accuracy or image quality.
Next, we evaluate NAO as a semantic data augmentation method on two learning setups: (1) Few-shot classification, and (2) Long-tail classification. We aim to show that our approach not only achieves faster generation speed but also attains state-of-the-art accuracy results on these benchmarks.
Few-shot benchmarks typically provide a limited number of samples per class, along with the corresponding class labels. NAO is used as a semantic data augmentation approach: We use the given samples and generate many more samples from that class, which are then used to train a classifier.
Datasets. We evaluated NAO on three common few-shot classification benchmarks: (1) CUB200 [55]: A fine-grained dataset comprising 11,788 images of 200 bird species. The classes are divided into three sets, with 100 for meta-training, and 50 each for meta-validation and meta-testing. (2) miniImageNet [54]: A modified version of the standard ImageNet dataset [14]. It contains a total of 100 classes, with 64 classes used for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing. The dataset includes 50,000 training images and 10,000 testing images, with an equal number of images distributed across all classes. (3) CIFAR-FS [8]: Created from CIFAR-100 [26] by using the sampling criteria as miniImageNet. Has 64 classes for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing; each class containing 600 images.
Following all previous baselines, we report classification accuracy as the metric. We report our results with  $95\%$  confidence intervals on the meta-testing split of the dataset.
Compared Methods. We conducted a comparative evaluation of our approach with several state-of-the-art methods for few-shot learning. These methods fall into three categories based on their approach. (A) Methods that do not use pre-training nor use class labels during training: Label-Hallucination [23] and FeLMi [39]; (B) Methods that use class labels during training: SEGA [59]; and (C) Methods that utilize a classifier pre-trained on external data and also use class labels during training: SVAE [58], Vanilla Stable Diffusion (Version 2.1) [38], Textual Inversion [20], DiffAlign [40], and SeedSelect [43]. The last four methods are semantic data augmentation methods.
Experimental Protocol. For a fair comparison with prior work, we follow the training protocol in [40] and [43]. We generated 1,000 additional samples for each novel class using SeedSelect. It was initialized with seeds found with NAO using the centroid and interpolation samples of the few-shot images provided during meta-testing and prompted it with the corresponding class name. We used a ResNet-12 model for performing N-way classification and trained it using cross-entropy loss on both real and synthetic data.
Results. Tables 3a and 3b compare NAO with SoTA approaches on few-shot classification benchmarks: CUB, miniImageNet, and CIFAR-FS. NAO consistently outperforms all few-shot methods on CUB [55] and miniImageNet [54], and reaches comparable results to SeedSelect [43] on CIFAR-FS [8]. Table 4 further compares the mean run time of SeedSelect with and without NAO on these datasets, on a single NVIDIA A100 GPU. The results highlight the competence of our approach in generating rare and fine-grained classes, to reach top accuracy with a five-fold reduction in the runtime.
Datasets. We further evaluated NAO on long-tailed recognition task using the ImageNet-LT [32] benchmark. ImageNet-LT [32] is a variant of the ImageNet dataset [15] that is long-tailed. It was created by sampling a subset of the original dataset using the Pareto distribution with a power value of  $\alpha = 6$ . The dataset contains 115,800 images from 1,000 categories, with the number of images per class ranging from 5 to 1,280.
Compared Methods. We compared our approach with several state-of-the-art long-tail recognition methods. These methods fall into three categories based on their approach. (A) Long-tail learning methods that do not use any pretraining nor employ class labels for training: CE (naive training with cross-entropy loss), MetaSAug [27], smDragon [42], CB LWS [24], DRO-LT [44], Ride [56] and Paco [13]. (B) Methods that use class labels as additional information during training: DRAGON [42]. (C) Methods that were pre-trained on external datasets and use class labels as additional information during training: VL-LTR [52], Vanilla Stable Diffusion (Version 2.1) [38] and SeedSelect [43]. MetaSAug [27], Vanilla Stable Diffusion and SeedSelect [43] are semantic augmentations methods. Note that VL-LTR [52], compared to other models, further fine-tuned the pre-trained model (CLIP [35]) on the training sets.
Experimental Protocol. Following previous methods, we use a ResNet-50 model architecture, train it on real and generated data, and report the top-1 accuracy over all classes on class-balanced test sets.
Figure 6: Results for few-shot image recognition, comparing NAO+SeedSelect to previous approaches. Fine-tuning a CLIP classifier on SeedSelect generated images initialized with NAO consistently achieves SOTA results across all shot levels.
Results. Tables 3c evaluates our approach compared to long-tail recognition benchmarks. NAO reaches SoTA results albeit simple, compared to other complex baselines.
5.4 CLIP finetuning with Synthetic data for image classification
We follow [43, 62, 64] and further examine the advantage of using our method in the generation of synthetic images, as semantic augmentation, for the downstream task of few shot CLIP recognition. In order to deal with rare concepts we use our method together with SeedSelect showing SoTA results in this task [43]. To this end we use NAO as seed initialization for SeedSelect. In the context of this task, we are provided with a limited number of real training samples per class, along with their corresponding class names, and the goal is to fine-tune CLIP.
Experimental Setup. For a fair comparison we follow the same experimental protocol of [43]. Specifically, we generated 800 samples for each class using NAO+SeedSelect from the limited real training samples. We then fine-tune a pre-trained CLIP-RN50 (ResNet-50) through mix-training, incorporating both real and generated images.
Compared Methods. We compare our approach with the following baselines: Zeor-shot CLIP: Applying the pre-trained CLIP classifier without fine-tuning; CooP [64]: Fine-tuning a pre-trained CLIP via learnable continuous tokens while keeping all model parameters fixed; Tip Adapter [62]: Fine-tuning a lightweight residual feature adapter; CT & SD: Classifier tuning with images generated with SD. Textual Inversion [20]: Classifier tuning with images generated using personalized concepts (See Supp for implementation detail). Results for CT & SD were reproduced by us on SDv2.1 using the code published by the respective authors.
Results. Figure 6 shows results for the few-shot image recognition task. It demonstrates that initiating seeds using NAO improves SeedSelect performance on all shots.
This paper proposes a set of simple and efficient tools for exploring the seed space of text-to-image diffusion models. By recognizing the role of the seed norm in determining image quality based on the  $\chi$  distribution as prior, we introduce a novel method for seed interpolation and define a non-Euclidean metric structure over the seed space. Furthermore, we redefine the concept of a centroid for a set of seeds and present an optimization scheme based on the new distance function. Experimental results demonstrate that these optimization schemes, biased toward the  $\chi$  distribution mode, generate higher-quality images compared to other approaches. Despite the simplicity and effectiveness of our approach, there are several limitations to be aware of. Firstly, compared to standard interpolation and centroid calculation, it involves an additional optimization step. Secondly, our centroid and/or the samples along our interpolation paths may not produce plausible and semantically correct images on their own, necessitating the use of SeedSelect optimization. Lastly, although our method is expected to be applicable to all diffusion models, we specifically evaluated it with the open-source Stable Diffusion [38] model in this study.
[60] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020. 8
[61] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, 2020. 8
[62] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. ECCV, 2022. 10
[63] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In CVPR, 2019. 3
[64] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 2021. 10
[65] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. ECCV, 2020. 3