Norm-guided latent space exploration for text-to-image generation
Authors: Dvir Samuel
Rami Ben-Ari
Nir Darshan
Haggai Maron
Gal Chechik
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
few-shot learning
long-tail learning

Abstract: Text-to-image diffusion models show great potential in synthesizing a large variety of concepts in new compositions and scenarios. However, the latent space of initial seeds is still not well understood and its structure was shown to impact the generation of various concepts. Specifically, simple operations like interpolation and finding the centroid of a set of seeds perform poorly when using standard Euclidean or spherical metrics in the latent space. This paper makes the observation that, in current training procedures, diffusion models observed inputs with a narrow range of norm values. This has strong implications for methods that rely on seed manipulation for image generation, with applications to few-shot and long-tail learning tasks. To address this issue, we propose a novel method for interpolating between two seeds and demonstrate that it defines a new non-Euclidean metric that takes into account a norm-based prior on seeds. We describe a simple yet efficient algorithm for approximating this interpolation procedure and use it to further define centroids in the latent seed space. We show that our new interpolation and centroid techniques significantly enhance the generation of rare concept images. This further leads to state-of-the-art performance on few-shot and long-tail benchmarks, improving prior approaches in terms of generation speed, image quality, and semantic content.

Norm-guided latent space exploration for text-to-image generation
Dvir Samuel Rami Ben-Ari', Nir Darshan', Haggai Maron 3.4 Gal Chechik
Bar-lllan University, Ramat-Gan, Israel
OriginAA, Tel-Aviv, Israel
Techhion,, Haifa, Israel
NVIDIA Research, Tel-Aviv, Israel
Abstract
Text-to-image diffusion models show great potential in synthesizing large variety concepts in new compositions and scenarios. However, the latent space initial seeds is still not well understood and its structure shown to impact the generation of various concepts. Specifically, simple operations like interpolation of seeds perform poorly when using standard
and finding the centroid of Euclidean or spherical metrics in the latent space. This paper makes the observation that, current training procedures, diffusion models observed inputs with narrow range of norm values. This has strong implications for methods that rely on seed manipulation for image generation, with applications to few-shot and long- taill learning tasks. To address this issue, we propose a novel method for interpolating between two seeds and demonstrate that defines new -un-Fuclidean metric that takes into account norm- norm-hased prior seeds. We describe simple yet efficient algorithm for approximating this interpolation procedure and use it to further define centroids in the latent seed space. We show that new interpolation and centroid techniques significantly enhance the generation rare concept images. This further leads to state-of-the-art performance on few-shot and long-tail benchmarks, improving prior approaches in terms of generation speed, image quality, and semantic content.
Introduction
Text-to-image diffusion models demonstrate exceptional ability to generate new and unique images. They map random samples (seeds) from a high-ddmennsonal space, conditioned on a user- provided text prompt, to corresponding image. Unfortunately, the seed space, and the way diffusion models map it into the space of natural images are still poorly understood. This may have a direct effect generation quality. For example, diffusion models have difficulty generating images of rare concepts, and specialized methods have been proposed to resolve this issue, for example, by performing optimization in the seed space [43]. Our limited understanding of the seed space is further demonstrated by the fact that standard operations on seeds, such as interpolating between two seeds finding the centroid of given set of seeds, often result in low-quality images with poor semantic content (Figure left, first two rows). As a result, methods based on the exploration and manipulation of seed spaces face considerable challenge.
The aim of this paper is to propose simple and efficient tools for exploring the seed space and to demonstrate how these tools can be used to generate rare concepts. Our main observation is that specific property, the norm of the seed vector, plays a key role in how a seed is processed by the
"Correspondence to: Dvir Samuel <dvirsamuel gmail.com>
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

-051
LERP
SLERP
-0.64
Ours
-0.8
-0.98
SLERP
-1.2
-1.49
-1.9
Ours
-.59
-1.53
Figure (Left) Visual comparison of different interpolation methods between two seeds from high-dim space of StableDiiffssion [38]. Images generated using Stable Diffusion [38]. (Right) Paths found using Linear. Spherical, and likelihood-basee interpolation methods in 2D space, where the norm of samples has x distribution (log) PDF. Note that NAO interpolation path goes through the domain with the desired norm (yellow-green) Both linear interpolation and SLERP [48] produce seeds that result in images that are flat or just noise, whereas our approach is capable of generating images that are meaningful (quantified in Table I).
diffusion model. In more concrete terms, since seeds are sampled from a multidimensional Gaussian distribution, the norm of the seeds has X distribution. For high dimensional Gaussian distributions, such as the ones used by diffusion models, the X distribution strongly concentrated around a specific positive number. Consequently, diffusion models tend to favor inputs with this norm, resulting in lower-quality images when the norm of the seed differs from it.
To address this issue, we propose to use prior distribution over the norms in the seed space based on the x distribution to guide exploration. While prior- based exploration techniques for seed spaces have been proposed before [2, 4], the advantage of our prior is that it does not rely upon expansive estimation of the empirical data distribution nor on complex computations and hence can be applicd to very high dimensional latent spaces. Yet, show below, our prior still significantly improves exploration techniques in seed space.
As first step, we propose novel method for interpolating between two seeds. In contrast to Linear Interpolation (LERP) or Spherical Linear Interpolation (SLERP) [48], we formulate this problem as finding a likelihood-maxinee maximiziniiiii path in seed space according to the aforementioned prior. In addition to providing us with an interpolating path, we also demonstrate that the optimal value of this optimization problem defines new non-Euclidean metric structure over the seed space. Figure I compares our interpolation paths to two other frequently used interpolation methods in 2D and in image space. The improvement of the image quality along the path is evident. Specifically, the 2D example (right panel) illustrates that LERP and SLERP paths cross low- low-probability areas whereas path maintains high probability throughout. The same phenomenon is shown for images (left panel) where it is apparent that intermediate points in the paths generated by the baseline methods have significantly lower quality.
As next step, we build on our newly defined metric to define a generalized centroid for set of seeds. In contrast to the standard definition of the centroid in Fuclidean spaces, we define the centroid the point that minimizes the distances to the seeds according to the new distance function (also known as the Fréchet mean for that given metric). We show how to discretize the two optimization problems above and solve them using simple and efficient optimization scheme. We call our approach NAO for Norm-Aware Optimization.
We evaluate NAO extensively. First, we directly assess the quality of images generated by our methods, showing higher quality and better semantic content. Second, we use our seed space interpolation and centroid finding methods two tasks: (1) Generating images of rare concepts, and (2) Augmenting semantic data for few-shot classification and long-tail learning. For these tasks, our experiments indicate that seed initialization with our prior-guided approach improves SoTA performance and the same time has significantly shorter running time (up to X10 faster) compared to other approaches.

---

2 Related Work
Text-guided diffusion models. Text-guided diffusion models involve mapping random seed (noise) 27 and textual condition P to an output image 2g through denoising process [7,36,41]. The inversion of this process can be achieved using a deterministic scheduler (e.g. DDIM [50D) allowing for the recovery of the latent code 2T from given image 2g. See detailed overview in the supplemental.
Rare concept generation with text-to-image models. Diffusion models excel in toxt-to-image generation [7, 36, 41], but struggle with rare fine-grained objects (e.g.payphone or tiger-cat in StableDiffusion [38]) and compositions g.g.shaking hands) [28, 43]. Techniques like pre-trained image classifiers and text-driven gradients have been proposed to improve alignment with text prompts, but they require pre- pre-trained classifiers or extensive prompt engineering (17,22, 29, 33, 38,34,44 41, 57]. Other approaches using segmentation maps, scene graphs, strengthening tross-attention units also face challenges with generating rare objects [5, 10, 18, 19, 633. SeedSelect [43] is recent approach that optimizes seeds in the noise space generate rare concepts. However, it suffers from computational limitations and long generation times. This paper aims to address these limitations by developing efficient methods that significantly reduce generation time while improving the quality of generated images.
Latent space interpolation. Interpolation is a well-studied topic in computer graphics [21,49, 51]. Linear Interpolation (LERP) is commonly used for smooth transitions by interpolating between two points in straight line. Spherical Linear Interpolation (SLERP) [48]. on the other hand, offers computing interpolation along the arc of a unit sphere, resulting in smoother transitions along curved paths. Image interpolations in generative models obtained by three main approaches: (1) Linear or spherical interpolation between two latent vectors [1,37, 65), (2) Image-to-Iman translation approaches [31] and (3) Learning interpolation funciton metrics based on the data 11,25, 2.. [3] observed that linearly traveling a normally distributed latent space leads to sub- optimal results and proposed an interpolation based on Riemannian metric. [2] further proposed a methodology to approximate the induced Riemannian metric in the latent space with a locally conformally flat surrogate metric that is based on learnable prior. Note, that as opposed to our approach, these priors do not have closed-form solution, they work on relatively low dimensional latent space of VAE (constrained and compact latent space), and they learn the metric from the data itself. In this paper, we do not assume any of the above. We introduce novel interpolation approach that effectively use inherent structure of the latent space to achieve correct interpolation without any additional data, on the high-dimensional seed space of diffusion model.
Data Augmentation via Latent Space Exploration. Previous methods, such [16, 30] and [6], have proposed techniques for semantic data augmentation using the latent space of generative models. These methods involve imposing uniform latent representations and applying linear interpolation or learning mappings to sample specific areas in the latent space. However, these approaches require training generative models from scratch. In contrast, this paper demonstrates more efficient approach by utilizing the latent space of pre-trained diffusion model for creating data augmentations without the need for additional model fine-tuning.
3 A norm-based prior over seeds
We start with reviewing statistical properties of samples in seed latent space E Rdd, with d denoting the dimension of the space.
In diffusion models, it is common to sample = from a high-dimensional standard Gaussian distribution 27 AO(0, Ia). For such multivariate Gaussians, the L2 norm of samples has > x = || (22/222((((()) / where I(-)) is the Gamma function and || is the standard Euclidean norm.
a (Chi) distribution: || =
Importantly, as the dimension grows, the distribution of the norm tends to highly concentrated around the mean, since at a high dimension,
Probability
0.6
the variance approaches constant 0.5. This strong concentration is illustrated in the inset figure on the right, for d = 16384 = 1282, the seed dimension used by Stable Diffusion [38]. At this dimension, the mean is
0.3
0.0
120
130
140
||z1||
3

---

(a)
||
x0.7
x0.8
x0.9
x1
x1.1
x1.2
x1.3
(b)
80
RID
51
60
||
(c)
accuract
40
60 128 200
||z|||
Figure (a) Progressively changing the norm of a fixed seed, which initially has a norm of v = 128. Images are generated by Stable Diffusion [38]. The visual quality of generated images degrades as the norm diverges away from vd the left and right end. (b) Mean per-class FID of the generated images in relation to the seed norms (lower is better). (c) Mean per-class accuracy of the generated images, determined by a state-of-the-ar pre- pre-trained classifier, function of the seed norm. Plots (b) and (c) show the degradation of the image quality the seed norm moves away from vd
also very close to the mode of the distribution, and both approach vd 128. This property means that samples drawn from multi-variat high-dimensional Gaussian distribution concentrated around specific value
Vd. Our key observation is that diffusion models trained with inputs sampled from the above normal distribution, and therefore the models are only exposed to inputs with norm values close during training. We hypothesize that this causes the model to be highly biased toward inputs with similar norm values.
mode
We conducted several experiments validate this bias. First, inspected the visual quality of images generated with different norm values, all sharing the same direction of the seed vector. Figure 2(a) visually illustrates the sensitivity of the Stable Diffusion model to the input norm, showing that quality degrades as the norm drifts away from the mode. Second, we conducted a systematic quantitative experiment and measured the impact of seed norm image quality in terms of FID and classification accuracy scores using an ImageNet pre-trained classifier. Figures 2(b)-2(c), show close to the mode. Full details of these
again that image quality depends the seed having experiments are given in Section 5 and supplemental material.
We conclude that the norm of the seed constitutes a key factor in the generation of high-quality images. Below we describe how this fact can be used for seed optimization and interpolation.
Norm-guided seed exploration
Based on the above results, we define a prior over the seed space as P(er) = (112r))) This probability density function represents the likelihood of a seed with norm | || to be drawn from the Gaussian distribution. We now describe simple and efficient methods for seed interpolation and centroid finding using that prior.
4.1 Prior indueed interpolation between two seeds
We first tackle the task of finding an interpolation path between the seeds of two images. The derivation of this interpolation path illustrates the advantages of using the prior in a simple setup, and will also be used later for finding centroids for sets of seeds. As seen in Figure (see also Figure 3), linear interpolation path between seeds consists of seeds that yield low-quality images. Instead, we define a better path : [0. 1] + IRd as the solution to the following optimization problem: Given two images, I, and /2, and their corresponding inversion seeds, 2/ and zi. derived by inversion techniques (e.g. DDIM Inversion [50]), we aim to maximize the log-likelihood of that path under our prior, defined as the line-integral the log-likelihood of points on the path.
Equivalently, we minimize the negative log- likelihoddo of the path, which is strictly positive, yielding
inf - - (ogg P(r)ds s.t. 7(0) 21 x(1) 27
(1)

---

jep
Tiger-oat
LERP
SLERP
Ours
Figure 3: Qualitative comparison among different interpolation methods between two image seeds. "Jeep" a common concept while "Tiger cat" a rare concept. Images generated using SD [38].
Here, the infimum is taken with respect to all differentiable curves y and J, W W(y)ds denotes the line integral of function W : Id + IR over the curve y 1 We denote the optimal value obtained for optimization problem (1) as fz+, 24)). It turns out that f' defines a (non-euclidean) distance on the seed space. This is stated formally in the following proposition.
Proposition 1, For any distribution yielding strictly positive negative log-likelihood. and specifically when P is the x" distribution, then f(:-)) is distance function R
See the supplementary material for proof. It is important to note that the optimization problem does not only provide with a path that maximizes the log-likelihood of prior, but it also defines new metric structure on the seed space that will prove useful in Section 4.2.
To approximate the solution to problem (1) in practice, discretize the path into sequence of piece-wise linear segments, that connect a series points 2 = Zp..... In = 2 and replace the integral with its corresponding Riemann over that piece-wise linear path:
T; Fi-1
minimize 20...
log
| xz-1/l
2
s.t.
Z0 = 2T, 2/ C(x) - - 0, [1....,
To facilitate a good approximation of the continuous integration, we also constrain consecutive path points to be close (see implementation at the end of Sec. 4.2).
Figures 1-3 illustrate paths resulting from the optimization the discretized optimization problem. Our optimized path consistently produces higher-quality images compared to other methods. A quantitative evaluation given in Section 5.
4.2 Prior induced centroid
Having defined a new metric structure in seed space, now ready to tackle the problem of finding centroid of multiple seeds. To this end, we assume to have set of images (71, I,,... I&] with their inversions (2 .., 2+} images. For example, one possible use such a centroid would be to find good initialization for seed associated with rare concept based on a few images of that concept. This enables us merge information between seeds and generate a more reliable fusion of the images that would be achievable through basic interpolation between seeds.
-0.51
Euclidean
Normalized Euclidean
-0.64
Sphere Projection
and wish to find the centroid of these
-0.8
-0.98
-1.2
-1.49
BIEE
-1.9
-2.59
-11.53
Recall that the centroid of set of points is defined the point that minimizes the sum distances between all the points. Perhaps the simplest way define centroid in our case by using the Euclidean distance, where the
Figure 4: Comparing different centroid finding methods in 2D space on con- tour of the distribution (log) PDF.
When 7 is differentiable, the integral be calculated using the following formula: So W (())))

---

Input Images (Imagenet1k)
Normalized
Sphere
Sphers Projection SeodSelect
Ours BBedBeleec
Euclidean
Projection
Ours
Tiger
Rare concepts
Orygen. wasle
Fay- phone
C-Common concepts
T-skirt
Jeep
Figure 5: Comparing different centroid optimization approaches on common and rare concepts of IeageNetlk, strong baselines. We further initialized SeedSelect [43] with the centroids and it for up to 5 iterations (~200 sec on single A100 GPU). NAO manages to generate correct images for rare concepts cat in tiger-cat, a clear oxygen mask and payphone), even without SeedSelect. With SeedSelect the images are further improved. The effect is less emphasized on common concepts w.r.t to strong baselines.
centroid definition boils down to a simple formula the average of the seeds. Unfortunately, we show in Section 3 this definition of centroid is not suitable for our purposes due to the incompatibility of the centroid's ' norm with the diffusion model. Instead, we propose use generalization of the Euclidean centroid, called the Fréchet mean, which is induced by the distance function defined above. To achieve this, we formulate an optimization problem that seeks the centroid e IR that minimizes the distances f(c, 24-) all the seeds:
6, 2.7 th = argmin
Σ 1=1
log P(sj)ds)
where 7/ paths between the common centroid and the inversion z/.. This optimization problem can be discretized in the same way we discretized Equation (1), i.e. by defining all the paths as piecewise linear paths defined by sequence of points and adding constraints on the distances between successive points in the path. See the supplemental for a discretized equation.
Figure 4 illustrates example of centroid in 2D found with approach. Figure 5 shows images resulting from the centroids found seed space. Section provides quantitative results.
Application to seed optimization methods. Diffusion models often encounter significant imbalances the distribution of concepts within their training data [43]. This imbalance presents a challenge for standard pre-trained diffusion models, leading to difficulties in accurately generating rare concepts. One proposed solution, as suggested by [43], involves employing a seed optimization technique. SeedSelect begins with a randomly generated seed that produces incorrect image and progressively optimizes the seed until a plausible image is generated. However, the method described in [43] is time-consuming requiring up 5 minutes to generate single image. Our approach can serve as an initial point for SeedSelect to reduce substantially its optimization time by generating initial seeds that result in realistic and quality images.
In our case, given few images, generating new data can obtained in the following manner: First, find a centroid e* and the interpolation paths o between image inversion in the seed space. Next, new data is generated by sampling points along the paths from the givens seeds to the centroid, and using them initializations for SeedSelect.

---

Implementation We implemented simple optimization algorithm that optimizes the discretized problems using Pytorch and the Adam optimizer. To speed up convergence we initialize the op- timization variables: the centroid is initialized with the Euclidean centroid and path variables are initialized as the values of the linear path between the points and the centroid. We implement the constraints ((x) = [ 24-1l - 8 S 0 using a soft penalty term in the optimization, by the form a ReLU(C(z)) where a is papeer-rrreeter.. Note that there is a penalty for positive C(z),, when the constraint is not satisfied. We note that in practice there is no guarantee that this optimization scheme converges to the optimal value of the optimization problem, however, we see in practice that high-quality paths obtained.
5 Experiments
We evaluate our approach in terms of image quality and also consider generation time. We start by studying the quality and semantic content of images generated with our seeding approach and evaluate them in three applications: (1) rare concept generation and (2) semantic data augmentation, to be tested at enhancing few-shot classification, long-tail learning, and few-shot image recognition. (3) We further tested our model on video frame interpolation, showing the results in the supplementary material. An ablation study can also be found in the supplementary material.
Direct evaluation of interpolation and centroid finding: Table compares FID scores and the accuracy of images generated using different interpolation and centroid finding methods. For the interpolation experiment, we randomly selected a class from ImageNet and obtained a pair of images and their corresponding seeds through inversion [50]. For interpolation methods that require path optimization, we used paths with 10 sampled points. We then select three seeds along the path (also for LERP and SLERP), with uniform intervals, and feed them into StableDiffusion [38], to generate 3 new images per pair. We repeated the process above to obtain 100 images per class. For the centroid experiment, we used 3-25 seed points obtained from the inversion of additional images (randomly selected) from the train set. We repeated this process for 50 random ImageNet classes. Mean FID scores were then calculated (against real ImageNetlk images), along with mean per-class accuracy using pre-trained classifier. See supplementary material for more details. Our optimized path consistently produces higher-quality images compared to other methods.
We compared our approach to the following baselines: Euclidean is the standard Euclidean centroid calculated as the mean of the seeds. Normalized Euclidean is the same as Euclidean, but the centroid is projected to the sphere induced by the X distribution. Sphere Projection first normalizes the seed to a sphere with radius r = Mode(x), then finds the centroid on a sphere by optimizing point that minimizes the sum of geodesic paths between the seeds to the centroid, as presented in [9]. See supplemental for more details. NAO-path and NAO-centroid are our methods presented in sections 4.1 and 4.2, respectively. The high accuracy and low FID levels of NAO in Table demonstrate that our interpolation approach outperforms other baselines in terms of image quality and content. We further put these results to test in downstream tasks (in Sec. 5.1.5.3).
Ace
FID
Interpolation methods
LERP
0.0
50.59
SLERP [48]
30.41
18.44
NAO-path (ours)
$1.59
6.78
Centroid computation methods
Euclidean
0.0
54.88
Normalized Euclidean
27.95
37.04
Sphere Projection NAO-centroid (ours)
40.81
14.28
67.24
5.48
Table 1: Comparing FID and accuracy of images generated by SD through sam- pling from different interpolation and centroid computation methods.
5.1 Rare-concept generation
Following [43] we compared different centroid optimization strategies in rare-concept generation.
Dataset, The evaluation is performed on ImageNetlk classes ordered by their prevalence in the LAION2B dataset [45]. LAION2B [45] is massive "in the wild" dataset that is used for training foundation diffusion models (e.g. Stable Diffusion [38]).
Compared Methods. We conducted a comparative evaluation between different centroid estimation strategies. SeedSelect [43] is a baseline method where a seed is randomly sampled and no centroid calculated. Other baselines were presented at the beginning of Section
7

---

ImageNetlk in LAION2B
Table 2: Image generation quality measured by the ac- curacy of a pre-trained clas- sifier. Average per-class accuracy is reported sepa- rately for the head, tail, and middle classes. NAO pro- duces the best performance in terms of image quality and is also x 10 faster.
Methods
Many
Med
Few
Total Ace FID
Trai Tope (sec) (see)
n=235
u=509
n=256
#>1M
IM>R>10K 10K*#
SeedSelect [43]
97.8
95.8
76.
91.3
6.5
D
298
Euclideun
0.0
0.0
0.0
0.0
51.1
Euclidean SeedSelect
0.0
0.0
0.0
0.0
65.7
inf
Normalized Euclidean
45.3
39.8
20.1
36.0
45.8
Normalized Euclidean ScodSclect
88.1
84,8
70.9
82.0
19.2
285
Sphere projection
52.0
48.9
26.5
44.0
12.3
0.1
Sphere projection SeedSelect
97.1
0.1
72
NAO-centroid (ours)
56.8
55.6
41.1
52.2
9.8
25
NAO centroid SeedSelect (ours)
98.5
96.9
85.1
94.3
6.4
25
29
(c)
ImageNet-LLT
Model
minilnage/Nt
5-way 5-shot
5-way 5-aboo
CIFAR-FS
Model
5-way 5-chot
CUB
CE
MensSAug [2]]
41.6
47.4
Label-Halluc [25]
PeLMi [39]
67.04 0.7
86.08_0.4
89,37-066
89,47105
TriNet [12]
HEAT [60]]
82.90:2022
$4.10.00.4
smDragon
LWS [24]
47.4
SEGA [591
79.03_022
86.00=022
DMp [6]]
88.69+0.5
DRO-LT [44]
Ride [56]
SVAE [S8]
80.70-02
78.89:003
MutriSem (461
82.92n//
PACO [13]
53.5
Testual Invension 120 4
85.44_1.9
SPGA [59]
90.85+002
Stable Diffasion 1
85.05-0.5
90.87=0.5
Stille Diffision* [38]
90.61+00.5
DRAGON [42]
$7.0
DeftAlign [40]
88.63=033
91,96=00.5
70.1
SeedSelect (43)
92,08-007
94,87=044
ScedSeiect [:3]
96,01:004
Stable Diflssion [43]
56.4
NAO-centroid (ours)
93.21_.0.6
94,85_.0.5
NAO-centrodd
97,92:002
ScedSelect [43]
74.900.5
NAO-centrold (ours)
78.9+0.1
Table 3: (a)-(b) Few-shot recognition. Comparing NAO to prior work on few-shot learning bench- marks. We report our results with 95% confidence intervals on meta-testing split of the dataset. (c) Long-tail recognition. Comparing NAO prior work long-tail learning benchmark. Values accuracy, obtained with a ResNet-50 backbone. denote results provided by [43]. t for muulti-modl methods that use class labels additional information, ş for multi-modal methods that were also pre-trained on external datasets, and ! for multi-modal methods that further finetuned foundation models. Our approach achieves the best results on all benchmarks.
miniImageNet
CIFAR-FS
CUB
Timit
Too
Trotal
Tiaa
Toot
Troii
Tima
Topt
Trotat
SeedSelect [43]
276
276 sec
228 sec
228 sec
312 sec
312 sec
NAO-centroid (ours)
28 sec
25 soc
53 sec
sec
21 sec
50 sec
31 soc
29 see
60 sec
Table 4: Comparing convergence time between SeedSelect [43] alone and SeedSelect initialized with seeds found using NAO. Note the five-fold reduction in runtime for our method.
Experimental Protocol. For every class in ImageNet, we randomly sampled subsets of training images, calculated their centroid in seed space, and generated an image using Stable Diffusion directly or as input to SeedSelect [43]. The class label was used as the prompt. This process is repeated until 100 images are generated for each class. We then used a So'TA pre-trained classifier [53] to test if the generated images are from the correct class or not (more details can be found the supplementary). We use this measure to evaluate the quality of the generated image, verifying that strong classifier correctly identifies the generated image class. We also report Mean FID score between the generated images and the real images, mean centroid initialization time Tinat and mean SeedSelect optimization time until convergence Tope on single NVIDIA A100 GPU. The results summarized in Table 2 show that our NAO method substantially outperforms other baselines, both in accuracy and in FID score. Furthermore, NAO gives better initialization point SeedSelect [43], yielding significantly faster convergence without sacrificing accuracy or image quality.
Next, we evaluate NAO as semantic data augmentation method on two learning setups: (1) Few-shot classification, and (2) Long-tail classification. We aim to show that our approach not only achieves faster generation speed but also attains state-of-the-art accuracy results on these benchmarks.
8

---

5.2 Few-shot learning
Few-shot benchmarks typically provide limited number of samples per class, along with the corresponding class labels. NAO is used as a semantic data augmentation approach: We use the given samples and generate many more samples from that class, which are then used to train a classifier.
Datasets. We evaluated NAO on three common few-shot classification benchmarks: (1) CUB- 200 [55]: A fine-grained dataset comprising 11,788 images of 200 bird species. The classes are divided into three sets, with 100 for meta-training, and 50 each for meta-validation and meta-festing,
(2) minilmageNet [54]] A modified version of the standard ImageNet dataset [14]. It contains total of 100 classes, with 64 classes used for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing. The dataset includes 50,000 training images and 10,000 testing images, with an equal number of images distributed across all classes. (3) CIFAR-FS [8]: Created from CIFAR- 100 [26] by using the sampling criteria as minilmageNet. Has 64 classes for meta-trainnng, 16 classes for meta-validation, and 20 classes for meta-testing: each class containing 600 images.
Following all previous baselines, report classification accuracy the metric. We report our results with 95% confidence intervals on the meta-testing split of the dataset.
Compared Methods. We conducted comparative evaluation of our approach with several state- of-the-art methods for few-shot learning. These methods fall into three categories based on their approach. (A) Methods that do not use pre-training nor class labels during training: Label- Hallucination [23] and FeLMi [39]; (B) Methods that use class labels during training: SEGA [59]; and (C) Methods that utilize classifier pre-trained
external data and also use class labels during training: SVAE [S8], Vanilla Stable Diffusion (Version 2.1) [38]. Textual Inversion [20], DiffAlign [40], and SeedSelect [43]. The last four methods are semantic data augmentation methods.
Experimental Protocol. For a fair comparison with prior work, we follow the training protocol in [40] and [43]. We generated 1,000 additional samples for each novel class using SeedSelect. It was initialized with seeds found with NAO using the centroid and interpolation samples of the few-shot images provided during meta-testing and prompted it with the corresponding class name. We used ResNet-12 12 model for performing N-way classification and trained it using cross-entropy loss on both real and synthetic data.
Results. Tables 3a and 3b compare NAO with SoTA approaches on few-shot classification bench- marks: CUB, minilmageNet, and CIFAR-FS. NAO consistently outperforms all few-shot methods on CUB [55] and minilmageNet [54], and reaches comparable results to SeedSelect [43] on CIFAR- FS [8]. Table 4 further compares the mean run time of SeedSelect with and without NAO on these datasets, on a single NVIDIA A100 GPU. The results highlight the competence of our approach in generating rare and fine-grat fine-grained classes, to reach top accuracy with a five-fold reduction in the runtime.
5.3 Loig-tail learning
Datasets. We further evaluated NAO long-tailed recognition task using the ImageNet-LT [32] benchmark. ImageNet-LT [32] is a variant of the ImageNet dataset [15] that is long-tailed. It was created by sampling a subset of the original dataset using the Pareto distribution with power value of a = 6. The dataset contains 115,800 images from 1,000 categories, with the number of images per class ranging from 5 to 1.280.
Compared Methods. We compared our approach with several state-of-the-arr long-tail recognition methods. These methods fall into three categories based on their approach. (A) Long-tail learning methods that do not use any pretraining nor employ class labels for training: CE (naive training with cross-entropy loss), MetaSAug [27]. smDragon [42], CB LWS [24], DRO-LT [44], Ride [56] and Paco [13]. (B) Methods that use class labels as additional information during training: DRAGON [42]. (C) Methods that were pre-trained on external datasets and use class labels as additional information during training: VL-LTR [52], Vanilla Stable Diffusion (Version 2.1) [38] and SeedSelect [43]. MetaSAug [27]. Vanilla Stable Diffusion and SeedSelect [43] semantic augmentations methods. Note that VL-LTR [52], compared to other models, further tnne-tuned the pre-trained model (CLIP [35]) on the training sets.
Experimental Protocol. Following previous methods, we use a ResNet-50 model architecture, train it real and generated data, and report the top-1 accuracy over all classes class-balanced test sets.

---

ImageNet
CUB
75
65
Accuracy (%)
70
Accuracy (%)
65
55
60
Zero-shot CLIP
50
Zero shot CLI
Adapter
55
CT So
45
CT Texutal Inversion
SeedSelect
CT SeodSelect
50
SeedSelect
SeadSelect
10 12 14 16 # real samples per class
4 6 8
10 12 14 16 # real samples per class
2
4
8
Figure 6: Results for few-shot image recognition, comparing NAO+SeedSelect to previous approaches. Fine-tuning CLIP classifier SeedSelect generated images initialized with NAO consistently achieves SOTA results across all shot levels.
Results, Tables 3c evaluates our approach compared to long-tail recognition benchmarks. NAO reaches SoTA results albeit simple, compared to other complex baselines.
5.4 CLIP finetuning with Synthetic data for image classification
We follow [43,62, 64] and further examine the advantage of using our method in the generation of synthetic images, as semantic augmentation, for the downstream task of few shot CLIP recognition. In order to deal with rare concepts we use our method together with SeedSelect showing SoTA results in this task [43]. To this end we use NAO seed initialization for SeedSelect. In the context of this task, we are provided with a limited number of real training samples per class, along with their corresponding class names, and the goal is to fine-tune CLIP.
Experimental Setup. For fair comparison we follow the same experimental protocol of [43]. Specifically, we generated 800 samples for each class using NAO+SeedSelect from the limited real training samples. We then fine-tune pre-trained CLIP-RN50 (ResNet-50) through mix-training, incorporating both real and generated images.
Compared Methods. We compare our approach with the following baselines: Zeor-shot CLIP: Applying the pre-trained CLIP classifier without fine-tuning; CooP [64]: Fine-tuning pre-trained CLIP via learnable continuous tokens while keeping all model parameters fixed; Tip Adapter [62]: Fine-tuning lightweight residual feature adapter; CT & SD: Classifier tuming with images generated with SD. Textual Inversion [20]: Classifier tuning with images generated using personalized concepts (See Supp for implementation detail). Results for CT & SD were reproduced by us on SDv2. using the code published by the respective authors.
Results. Figure 6 shows results for the few-shot image recognition task. It demonstrates that initiating seeds using NAO improves SeedSelect performance all shots.
6 Conclusion
This paper proposes a set of simple and efficient tools for exploring the seed space of text-to-image diffusion models. By recognizing the role of the seed norm in determining image quality based on the distribution as prior, we introduce a novel method for seed interpolation and define non-Fuclidean metric structure over the seed space. Furthermore, we redefine the concept of seeds and present an optimization scheme based on the new distance function. Experimental results demonstrate that these optimization schemes, biased toward the x distribution mode, generate higher-quie tigher-quality images compared to other approaches. Despite the simplicity and effectiveness of our approach, there are several limitations be aware of. Firstly, compared standard interpolation and centroid calculation, it involves an additional optimization step. Secondly, our centroid and/or the samples along our interpolation paths may not produce plausible and semantically correct images on their own, necessitating the use of SeedSelect optimization. Lastly, although our method is expected to applicable to all diffusion models, we specifically evaluated with the open-source Stable Diffusion [38] model in this study.
centroid for a set
10

---

References
[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? ICCV, 2019. 3
[2] Georgios Arvanitidis, Bogdan Georgiev, and Berhhard Schölkopf. A prior-based approximate latent riemannian metric. In AISTAT, 2021. 2,
[3] Georgios Arvanitidis, Lars Kai Hansen, and Sptren Hauberg. Latent space oddity; on the curvature of deep
generative models. ICLR, 2018. 3
[4] Georgios Arvanitidis, Søren Hauberg. Philipp Hennig, and Michael Schober. Fast and robust shortest paths
manifolds learned from data. ArXiv, abs/1901.0777 007229, 2019. 2.3
[5] Omri Avrahami. Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. CVPR, 2023. 3
[6] Idan Azuri and Daphna Weinshall. Learning from small data through sampling an implicit conditional generative latent optimization model. ICPR, 2020.
[7] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine. Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint XXXi:22211.01222 2022.
[8] Luca Bertinetto, Joao F Henriques, Philipp HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. ICLR, 2019. 9
[9] Samuel R. Buss and Jay P. Fillmore. Spherical averages and applications to spherical splines and interpola- tion. ACM Trans. Graph., 2001.
[10] Hila Chefer, Yuval Alaluf, Yacl Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention- based semantic guidance for text-to-image diffuusion models. arXiv preprint arXiv:2301.13826 2023.
[11] Ting-Cong Chen, Xiaogang Xu, Zhuotao Tian, and Jiaya Jia. Homomorphic latent space interpolation for unpaired image-to-image translation. CVPR, 2019. 3
[12] Z. Chen, Yanwei Fu, Yinda Zhang, Yu-Gang Jiang. X. Xue. and Leonid Sigal. Multi-level semantic feature augmentation for one-shot learning. IEEE Transactions Image Processing, 28:4594-4605, 2018. 8
[13] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. ICCV,
2021. 8, 9
[14] J. Deng, W. Dong, R. Socher, I., Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In evpr, 2009.
[15] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 9
[16] Terrance Devries and Graham W. Taylor. Dataset augmentation in feature space. ICLR Workshop, 2017. 3
[17] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021.
[18] Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Frie Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. ICLR, 2023. 3
[19] Oran Gafni, Adam Polyak. Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022. 3
[20] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. Or. An image is worth one word: Personalizing text-to-image generation using textual inversion.
ICLR, 2023. 8, 9, 10
[21]
Dianyuan Han. Comparison of commonly used image interpolation methads. In ICCSEE, 2013.
[22] Jonathan Ho and Tim Salimans. Classifier--ree free diffusion guidance. NeurlPS workshop on Deep Generative
Models and Downstream Applications, 2021. 3
[23] Yiren Jian and Lorenzo Torresani. Label hallucination for few-shot classification. AAAL, 2022. 8,
[24] B. Kang. S. Xie. M. Rohrbach. M. Yan. A. Gordo. J. Feng, and Y. Kalantidis. Decoupling representation and classifier for tailed recognition. ICLR, 2020. 8, 9
[25] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability. and variation. ICLR. 2018. 3
[26] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009. 9
[27] Shuang I., Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. CVPR, 2021. 8, 9.9
[28] Nan Liu, Shuang Li, Yilun Du, Antonio Torralha, and Joshua Tenenhaum. Compositional visual generation with composable diffusion models. In ECCV, 2022. 3
[29] Vivian Liu and Lydia B Chilton. Design guidelines for prompt engineering text-to-image generative models.
In ACM SIGCHI, 2022.
[30] Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang. Site Li, Ping Jia, and Jane You. Data augmentation via latent space interpolation for image classification. In ICPR, 2018. 3
11

---

[31]
Yahui Liu, E. Sangineto. Yajing Chen. Linchao Bao, Haoxian Zhang, Niculae Sebe, Bruno Lepri. and Marco De Nadai. Smooth maage-to-image translations with latent space interpolations. ArXiv, 2022. 3 Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In CVPR, 2019. 9
[32]
[33] Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis dall-e 2. arXiv preprint
ar arXiv:2204.338D 2022. 3
[34] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided
diffusion models. NeurlPS, 2022. 3
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from
natural language supervision. In ICML, 2021. 9
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.066
2022. 3
[37] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel
Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. CVPR, 2020. 3
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björn Ommer. High-resolution
image synthesis with latent diffussion models. In CVPR, 2022. 4, 7, 8, 9, 10
[39] Aniket Roy, Anshul Shah, Ketul Shah, Prithviraj Dhar, Anoop Cherian, and Rama Chellappa. FeLMi: :
Few shot learning with hard mixup. In NeurIPS, 2022. 8.9
[40] Aniket Basu Ray, Anshul B. Shah, Ketul Shah, Anirban Roy, and Ramalingam Chellappa. Diffalign: Few-shot learning using diffusion based synthesis and alignment. ArXiv, abs/2212.05404, 2022. 8, 9
[41] Chitwan Saharia, William Chan, Saurahh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-
image diffusion models with deep language understanding. NeurlPS, 2022. 3
[42] Dvir Samuel, Yuval Atzmon, and Gal Chechik. From generalized zero-shot learning to long-tail with class
descriptors. WACV, 2021. 8, 9
[43] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. It all about where you start: Text-to-image generation with seed selection. ArXiv, abs/2304. 14530, 2023. 1, 6, 7, 8, 9, 10
[44] Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. ICCV, 2021. 8, 9
[45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open aarge-scale dataset for training next generation image-text models. NeurlPS, 2022. 7
[46] Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex Bronstein. Baby steps towards few-shot learning with multiple semantics. Pattern Recognition Letters, 2022. 8
[47] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou, Interpreting the latent space of gans for semantic face editing. CVPR, 2019.
[48] Ken Shoemake. Animating rotation with quaternion curves. In SIGGRAPH, 1985. 3,
[49]
Wan-Chi Siu and Kwok- Wai Hung. Review of image interpolation and super-resolution. In Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 2012. 3
[50]
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021. 3, 4,
[51] Philippe Thévenaz, Thierry Blu, and Michael Unser. Image interpolation and resampling. Handbook of medical imaging, processing and analysis, 2000, 3
[52] Changyao Tian, Wenhai Wang. Xizhou Zhu, Xiaogang Wang, Jifeng Dai, and Y. Qiao. V1-Itr: Learning class-wise visual-linguistic representation for long-tailed visual recognition. In Europcan Conference on Computer Vision, 2021. 8, 9
[53] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. ECCV, 2022. 8
[54] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. koray kavukcuoglu, and Daan Wierstra. Matching networks for shot learning. In NeurIPS, 2016. 9
[55] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD 2irds-200-2011 Dataset.
Technical Report -NS-TR-201 1-001, California Institute of Technology, 2011. 9
[56] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing
diverse distribution-awarr experts. ICLR, 2021. 8.9
[57] Zijie Wang, Fvan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery datasct for text-to-image generative models. arXiv preprint 1rXiv:2210.144 14996, 2022. 3
[58] Jingyi Xu and Hieu Le. Generating representative samples for few-shot classification. In CVPR, 2022. 8,
[59] Fengyuan Yang, Ruiping Wang, and Xilin Chen. Sega: semantic guided attention on visual prototype for shot learning. In WACV, 2022. 8.9 9
12

---

[60]
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020. 8
[61] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, 2020. 8
[62] Renrui Zhang. Rongyao Fang. Wei Zhang. Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. ECCV, 2022.
10
[63] Zhao, Lili Meng. Weidong Yin, and Leonid Sigal. Image generation from layout. CVPR, 2019.
[64] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 2021. 10
[65] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolci Zhou, In-domain gan inversion for real image editing. ECCV, 2020. 3
13


Reviews:
[{"review_id": "5W4AKS92Ue", "reviewer": "Reviewer_qXwY", "summary": "This paper makes the observation that current training procedures make diffusion models biased toward inputs with a narrow range of norm values. To address this issue, the authors propose a novel method for interpolating between two seeds and demonstrate that it defines a new non-Euclidean metric that takes into account a norm-based prior on seeds. The authors describe a simple yet efficient algorithm for approximating this metric and use it to further define centroids in the latent seed space. The effectiveness of the proposed approach is validated on generating images of rare concepts, and augmenting semantic data for few-shot classification and long-tail learning.", "questions": "Please refer to the weakness section.", "limitations": "The limitation is adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observation that current training procedures make diffusion models biased toward inputs with a narrow range of norm values is interesting and inspiring.\n2. The proposed approach is well-motivated and aligns with intuition.\n3. The observation and proposed approach have the potential to benefit many tasks related to the application of diffusion models.", "weaknesses": "1. The proposed approach only works well with seed optimization techniques such as SeedSelect, as indicated by Fig.5 and lines 195 - 203. This implies that the derived interpolation paths may not be optimal and it also introduces extra computational cost of the seed optimization.\n2. The proposed approach is demonstrated mainly on generating rare images and augmenting data for few-shot learning. However, I am more interested in more applications that might benefit from the proposed approach. For example, can the proposed approach be applied for video interpolation and video generation and perform better than previous approaches?\n3. Although significantly better than previous approaches, the results in Fig.3 indicate that the interpolation results of the proposed approach is still not perfect. Do the authors have insights on the reason for the imperfect interpolation results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach inspiring and interesting. We address your comments below.   \n\n#### **Q1: The proposed approach only works well with seed optimization.**\n**A1:** Let us explain where and why NAO works well without seed optimization and the relation between the two methods. First, we stress that seed optimization is only necessary for generating rare concepts (See SeedSelect [1] and in lines 29 and 62 of our paper). For rare concepts, diffusion models often fail. They may generate high-quality images but from a wrong category. Seed optimization presents a solution for this scenario.    \n\nNAO functions effectively without relying on seed optimization. This is demonstrated in Figures 1, 3, S3, and Table 1. NAO's interpolations and centroids enable the generation of semantically consistent images even for **common (head)** concepts where SeedSelect is unnecessary. For instance, head classes like \"tuna\" and \"jersey\" have multiple interpretations and can lead to the generation of incorrect objects. \"Tuna\" might denote either a \"tuna fish\" or a \"tuna can,\" while \"jersey\" could refer to a \"standard t-shirt\" or a \"sports team shirt\".  Given a few reference images with their concept name, NAO can generate images that belong to the correct domain.    \n\nTo further illustrate NAO’s advantage, we replicated the experiments from Table 1 solely on 50 randomly selected **common (head)** concepts, **without employing seed optimization**. Subsequently, we compared NAO to two alternate interpolation methods in the provided table. The results unequivocally confirm that NAO achieves significantly higher accuracy (in generating the correct concept) and improved FID.\n\n|                             | Acc (higher is better)           | FID (lower is better)|\n| -----------            |  ----------- | ----------- |\n| LERP                  | 0.00      | 60.60       |\n| SLERP                | 69.12   | 17.55        |\n| **NAO-path (ours)**| **88.91**   | **5.21**        |\n\n#### **Q2: Can the proposed approach be applied for video interpolation and video generation?**\n**A2:** We found the idea of applying our method to video interpolation intriguing, following the reviewer's suggestion. As a result, we applied our approach to the task of video frame interpolation, aiming to generate an intermediate frame between existing consecutive frames (I1 and I2) in a video sequence. Using NAO, we interpolated between the seed inversions of two given frame images, optimizing 50 points, and generating an image from the middle points. Example images in Figure G1 (see pdf attached to the general response to all reviewers) illustrate NAO's ability to create intermediate frames. To further demonstrate NAO’s capability, we conducted also a quantitative evaluation. To this end, we followed the experiment outlined in [2,3], focusing on the **DAVIS dataset**: a widely acknowledged benchmark for Video Frame interpolation tasks. The evaluation of predicted frames against the ground truth was carried out using established metrics like PSNR and SSIM. The table demonstrates that our interpolation approach achieves comparable results to existing methods specifically tailored for video interpolation, despite solely using a pre-trained text-to-image model. We appreciate the reviewer's insightful suggestion, as this presents a promising avenue for future research.\n\n|                             | PSNR   (higher is better)        | SSIM  (higher is better)|\n| -----------            |  ----------- | ----------- |\n| MCVD [2]                  | 18.646      | 0.705       |\n| LDMVFI  [3]              | 25.541   | 0.833       |\n| **NAO-path (ours)**| 25.413  | 0.813        |\n\n\n#### **Q3: Do the authors have insights on the reason for the imperfect interpolation results?**\n**A3:** As indicated in A1 and also mentioned in [1], rare concepts often have restricted regions within the seed space where plausible and correct images can be generated. This is evident in Figure 3, where NAO performs well for the common concept of a jeep, but for the rare concept of a tiger cat, *most* images appear satisfactory. Despite this complexity, our approach surpasses existing interpolation methods by consistently generating superior, semantically accurate images at all interpolation points.\n\n#### **References**\n[1].   Samuel et al. (2023), \"It's all about where you start: text to image generation with seed selection\".     \n[2].   Danier et al. (2023), \"LDMVFI: Video Frame Interpolation with Latent Diffusion Models\".    \n[3].   Voleti et al. (2022), \"MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation\".    \n[4].   Perazzi et al. (2016), \"A benchmark dataset and evaluation methodology for video object segmentation\".    \n"}, {"review_id": "NCOFm0h63g", "reviewer": "Reviewer_6Fzu", "summary": "This paper investigates a new method for interpolating in the seed space of diffusion models, which is the Gaussian distribution used to initialize the generation process. Experiments demonstrate that diffusion models struggle with generation when the norm of the input differs from the distribution of norms of random noise samples drawn from the starting distribution; this can happen if the input is a LERP or SLERP interpolation between two random samples. The paper proposes to define a prior over the seed space using a chi distribution, and finds a path such that the likelihood of each point on the path is maximized under the prior distribution. This is optimized in a discretized fashion. A similar method can be used to find the centroid between multiple images, which minimizes the likelihood of the paths from each latent to the centroid. These methods are used to generate additional data, and studied in the context of rare-concept generation, few-shot recognition, and long-tail recognition.", "questions": "- in L212, what are the constraints $c(x) \\geq 0$?\n- Figure 1 left was difficult to understand in the first pass. Perhaps it would help to make clearer that the color refers to log likelihood of the $\\chi$ distribution?\n- Table 4: should it be 21 seconds rather than minutes?\n- Are the optimization times stated per image in Table 4? \n- What is the total overall overhead for the long-tail experiments -- how many images are generated per class?", "limitations": "Limitations are addressed in the conclusion. The key limitation is that this method requires additional optimization to generate one image, on the order of 30-60 seconds. The most effective use case of this method seems to be in conjunction with SeedSelect to produce plausible images with less optimization time, as the NAO method alone does not always produce recognizable images, as shown in Table 2. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is demonstrated to generate effective images in limited data scenarios. By providing a few example images, the model can generate related images by interpolating between the inputs under high likelihood regions of the latent space.\n- The method is effective when combined with prior existing methods. SeedSelect optimizes an initial seed to match the concepts in a few given images. When using the NAO-centroid to initialize the seed, the results are more effective and optimization is faster compared to using SeedSelect alone. \n- Qualitatively, the results look compelling against other shown interpolations in the diffusion seed space. ", "weaknesses": "- It seems that the investigation is only performed on the input latent space, but I'm curious if feature interpolation in alternative latent representations would preserver a stronger image prior. For example, Asyrp[1] demonstrates that smooth changes can be obtained by manipulating the h-space of a diffusion model. I think this would be a worthwhile baseline to compare to, rather than just interpolations in the input latent space. \n- The method relies on optimization over over a set of points. I think more details on the optimization could be provided here -- for example, how do the results differ if the number of interpolation points is changed, or the optimization time changes? What is the variation in this optimization procedure? \n\n[1] Asyrp: https://arxiv.org/abs/2210.10960 ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach effective with compelling results. We address your comments below.\n\n#### **Q1: Comparing with Asyrp [1].**\n**A1**: We value the reviewer's suggestion to conduct a comparison between interpolation in the input space (our approach) and interpolation in a feature space (Asyrp). After a deep look, we found that a direct comparison is not feasible due to several reasons. First, The h-space lack properties of interpolation as it is primarily designed for editing of a predefined set of concepts. Second, the h-space edits carried out at each denoising step complicate the prospect of straightforward image interpolation between images in a single space. Despite these challenges, we attempted to interpolate (using different interpolation techniques and NAO) between h tensors of two images through all denoising steps. The outcome yielded corrupted images, underscoring the unsuitability of h-space for effective interpolations.\n\n#### **Q2: More details regarding optimization.**\n**A2:** Thank you for your valuable feedback. In response to this comment, we conducted additional experiments, replicating those performed in Table 1 of the main paper. We evaluated our approach by gradually increasing the number of optimization points and analyzed its impact on both FID scores and convergence time. The results demonstrate that an increase in the number of points leads to slightly improved FID scores, with only a marginal rise in optimization time. We will add this experiment and additional details regarding the optimization procedure in our revised manuscript/suppl. material. Variational analysis will be added too.\n\n\n|                             |  NAO-path |                |   NAO-centroid |           |\n| -----------            |  ----------- | -----------  |  ----------- | ----------- |\n|    #points            | FID (lower is better)           | $T_{init}$ (lower is better)| FID (lower is better)           | $T_{init}$ (lower is better) |\n|     10                   | 6.78+-0.1 | 21+-1 s | 5.48+-0.15 | 26+-1s |\n|       50                 | 6.7+-0.09| 23+-1 s|  5.47+-0.11 | 26+-1s |\n|         100             | 6.67+-0.06 | 24+-1 s | 5.43+-0.10 | 28+-1s|\n|         1000           | 6.61+-0.05| 27+-1 s| 5.41+-0.08| 30+-1s        |\n\n#### **Q3: What is the constraint c(x)≤0?.**\n**A3:** $c(x)$ corresponds to the constraint in Eq. 2, namely $|x_i-x_{i-1}|  \\leq \\delta$. We formulate this constraint as $c_{i}(x) = |x_i-x_{i-1}| -\\delta \\leq 0$. This constraint is part of the integral discretization (into sum of finite elements) in Eq. 2. In the implementation we enforce this constraint using a soft penalty term with ReLU function; that is ReLU($c(x)$) (as described in line 212). Now there is a penalty for positive $c(x)$ (when the constraint is not satisfied). We apologize for the brevity in lines 211-212. We will elaborate on the discretization description of this equation in suppl. material and make sure $c(x)$ is clearly defined in text in the revised version.\n\n#### **Q4: Figure 1 was difficult to understand.**\n**A4:** We thank the reviewer for their feedback. As pointed out by the reviewer, the colors in the figure correspond to the log-likelihood of the chi distribution. To address this concern, we will provide additional clarification on the figure and include more detailed explanations in the text for the camera-ready version.\n\n#### **Q5: Typo in Table 4.**\n**A5:** Yes, a typo,  it should be 21 sec. Thank you. Will be fixed in the camera-ready version.\n\n#### **Q6: Are the optimization times stated per image in Table 4?**\n**A6:** Yes, for a fair comparison, we provided the optimization time per image when employing SeedSelect for rare concepts. Notably, our approach significantly reduces the optimization time from 5 minutes to just ~25 seconds and also leads to reduced memory requirements, allowing concurrent generation of multiple images. It is important to mention that for common (head) concepts, optimization is *unnecessary*, and our approach enables direct image generation.\n\n#### **Q7: How many images are generated per class for long-tail learning?**\n**A7:**  We followed the experimental protocol of  [2] and generated samples for each class until the combined total of real and generated samples equaled the count of the class with the highest number of samples in the dataset, resulting in a uniform data distribution. For more details, please refer to Appendix F in the Supplementary material.\n\n#### **Q8: The method requires additional optimization to generate one image.**\n**A8:** This is correct for only rare concepts. For rare concepts, It requires additional optimization but NAO computational cost is significantly cheaper than the alternative, SeedSelect. While SeedSelect needs to backpropagate through the diffusion model NAO makes a simple path optimization process in the seed space. Please also refer to the response provided in the shared answer for all reviewers.\n\n#### **References**\n[1].   Kwon et al. (2023), \"Asyrp: Diffusion Models already have a Semantic Latent Space\".   \n[2].   Samuel et al. (2023), \"It's all about where you start: text to image generation with seed selection\".    \n"}, {"review_id": "yGNwe6o3En", "reviewer": "Reviewer_WuD2", "summary": "The paper observed that the seed (noise) for the trained diffusion model has a property that the norm of the seed, which follows the $\\chi$ distribution, is concentrated around a certain positive number, $\\sqrt{d}$ where $d$ denotes the seed dimension. Based on this observation, the paper proposed a way to explore the seed space (e.g. interpolation, centroid) using Norm-Aware Optimization (NAO), which defines the objective function as likelihood maximization in the seed space. The paper applied seed space exploration with NAO to generate rare concepts and augment semantic data for few-shot classification and long-tail learning.", "questions": "- It is unclear how many piece-wise linear paths are used to estimate (1) in experiments. Also, there is no discussion or analysis on $\\delta$. If one set small $\\delta$, the objective function (2) would be harder since the number of variables increases. If one sets large $\\delta$, the approximation will be inaccurate. The paper mentioned that they constrain consecutive path points to be close (lines 161-162), but the corresponding description is unclear. Especially, in line 212, what is the meaning of the constraints, $c(x)\\leq0$? There is no definition of the function $c(\\cdot)$ and the input argument $x$.\n- For the centroid estimation, there are at least two approximation gaps: one from discretizing the line-integral of log-likelihood (2) and the other from sub-optimal solution for (2). However, there is no analysis of the accuracy of the proposed distance function and centroid estimation, except for empirical performance whose accuracy seems to be dominated by the Euclidean centroid initialization.\n- For the rare-concept generation, the paper claims that SeedSelect with the initial point found by NAO-centroid achieves faster and better generation. However, NAO-centroid also initializes its centroid with the Euclidean centroid of inversion points whereas SeedSelect uses random initialization. For a fair comparison, the runtime and performance of the SeedSelect initialized with the Euclidean centroid should be compared. Note that the Euclidean centroid initialization of the NAO-centroid is “to speed up convergence” (in line 201).\n- In Table 4, CIFAR-FS $T_{Opt}$  might be 21 sec, not 21 min. And there is no unit for miniImageNet $\\bar{T}_{Opt}$.", "limitations": "Yes, the authors describe the limitations of the proposed method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper clearly stated contributions with comparisons with existing works on latent space exploration.\n- The proposed method relies on the inherent structure of latent space, which is defined as the normal distribution, so could be used generally under diffusion model literature.\n- Experiments are well-designed and easy to follow.", "weaknesses": "- Some experimental conditions (e.g. number of piece-wise linear paths) are unclear.\n- Although the proposed method uses multiple approximations, the paper does not provide analysis or experiments on the accuracy of the approximation.\n- Comparison for the rare-concept generation using centroid estimations seems not fair. Compared baseline is initialized randomly, but the proposed method is initialized with the Euclidean centroid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback, acknowledging the broad usability of our approach and the well-designed, easy-to-follow experiments.  We address your comments below.\n\n#### **Q1: It is unclear how many piece-wise linear paths are used.**\n**A1:**  For few-shot learning benchmarks we optimized paths with a length of 200 points between the centroid of the k-shot training samples and the samples themselves (see line 108 suppl. material). For long-tail learning, we optimized paths ranging in length from 10 to 200, adjusting based on the available number of samples for each class (line 121 suppl. material). Due to lack of space we present more details in Appendix F of the supplementary material.\n\n#### **Q2: What is the constraint c(x)≤0?**\n**A2:** $c(x)$ corresponds to the constraint in Eq. 2, namely $|x_i-x_{i-1}|  \\leq \\delta$. We formulate this constraint as $c_{i}(x) = |x_i-x_{i-1}| -\\delta \\leq 0$. This constraint is part of the integral discretization (into sum of finite elements) in Eq. 2. In the implementation we enforce this constraint using a soft penalty term with ReLU function; that is ReLU($c(x)$) (as described in line 212). Now there is a penalty for positive $c(x)$ (when the constraint is not satisfied). We apologize for the brevity in lines 211-212. We will elaborate on the discretization description of this equation in suppl. material and make sure $c(x)$ is clearly defined in text in the revised version.\n\n#### **Q3: Missing analysis for delta.**\n**A3**: We appreciate the reviewer's valuable feedback and their contribution to improving our paper. In our experiments, we set $\\delta$ to be $||z_1 - z_2||/n$, where $z_1$ and $z_2$ are seed inversions of real images ($x_1$ and $x_2$), and $n$ is the number of interpolation points to be optimized. In response to the reviewer's comment, we conducted an in-depth analysis to investigate the impact of different delta values on the FID of images generated along the interpolation path. Figure G2 (see pdf attached to the general response to all reviewers) presents the results of this analysis. The analysis is done with $n=100$ points. The findings reveal that excessively low or high delta values adversely affect the FID, as they either constrain the points too closely together, causing overlap, or spread them too far apart, leading the path into low-likelihood regions.\n\n#### **Q4: Analysis of the accuracy of the proposed distance function.**\n**A4:** Indeed solving the optimization problem in Equation (2) provides only an approximation of its continuous counterpart in Equation (1).  We agree that there are two main error components: (i) approximation error of the discrete integral compared to the continuous one; (ii) the optimization error when optimizing Equation 2. This is an important observation, and we will include a discussion of it in the paper.\nWith regards to (i), since $\\log P(x)$ is smooth, we expect the minimizer of Equation (1) to be smooth as well and this allows piecewise linear paths to approximate it with arbitrarily small error by using enough point in Equation (2). \nRegarding (ii), we refer to this concern in lines 213-214. As is typically done in deep learning, we employ first-order optimization methods to optimize a non-smooth function. Therefore, analyzing the error and convergence is a very difficult problem that is beyond the scope of this paper. Based on our experimental results and 2D visualizations, we believe that our optimization process converges to a satisfactory solution. From a practical standpoint, we have observed that although the numerical solutions may not be exact, the FID measures, which indicate the quality of the generated images, consistently demonstrate high quality. This is also evident from the example images displayed in Figure 3 of the main paper and Figure S3 in the supplementary material.\n\n#### **Q5: Comparison with SeedSelect initialized with the Euclidean centroid.**\n**A5:** We indeed compare our results with SeedSelect initialized with Euclidean centroid and also with the Normalized Euclidean centroid, for fair comparison. This is shown in Table 1 (main paper, Page 7, L3,5). The results indicate that the performance improvement achieved by NAO is not solely attributed to the Euclidean initialization, but rather to the optimization minimization problem formulated in Equation (2) of our paper. Actually, initiating centroids with methods other than NAO, hurts SeedSelect performance (see Table 1). In terms of classifier accuracy, *NAO+SeedSelect* outperforms *Euclidean+SeedSelect* and *Normalized Euclidean + SeedSelect* by +94.3% and +12.3%, respectively. Additionally, in terms of FID score, *NAO+SeedSelect* performs +12.8% better than *Normalized Euclidean + SeedSelect*. For few-shot and long-tail benchmarks, we employed the best strategy as identified in Table 1, namely, *NAO+SeedSelect*.\n\n#### **Q6: Typo in Table 4.**\n**A6:** Yes, it should have been 21 sec. Thank you. Will be fixed in the camera-ready version."}, {"review_id": "A6ySwSt4jg", "reviewer": "Reviewer_yJ89", "summary": "\tThis paper proposes a novel method for interpolating between two seeds and demonstrates that it defines a new non-Euclidean metric that takes into account a norm-based prior on seeds. This paper describes a simple yet efficient algorithm for approximating this metric and using it to further define centroids in the latent seed space, which helps generate rare concept images and leads to state-of-the-art performance on few-shot and long-tail benchmarks.", "questions": "-\tThere are some formatting errors in the article, such as the picture in the upper right corner of page 4 without icons and annotations.\n-\tI can not understand the interpolation path of 2D space. Hope to have more explanations of 2D space, and how to get the result such as in Figure 1 (right).\n-\tOther concerns have already been mentioned in Weakness.\n", "limitations": "Yes, the authors have addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThis paper first discusses the property of the seed, that is, the relationship between the norm of the seed and the quality of the generated image, which provides good theoretical support for the proposed new non-Euclidean metric.\n-\tThe newly proposed non-Euclidean metric combined with the centroid method has a good effect according to the experimental results and has been optimized to a certain extent for problems such as rare concept generation and long tail training.\n-\tFrom the seed level, the paper investigates the text-image generation problem of the diffusion model, and verify the feasibility of controlling the image generation from the seed level.\n", "weaknesses": "-\tIn the early stage, the norm problem of seed was elaborated and verified too much, but the core path optimization and centroid method did not elaborate enough.\n-\tJudging from the experimental results of the pictures in the article, the method in this paper does not seem to show a particularly great advantage, especially after adding the seed-selection method.\n-\tThe two interpolation methods compared in this article are very basic. There are more non-linear interpolation methods to compare, and the results after interpolation seem to be inconsistent with the actual results. In my understanding (also my experiment testing), no matter what kind of Gaussian noise the seed is, SDM can generate a relatively reasonable image, rather than an unnatural noisy image like the ones presented in Figure 1(left). \n-\tFrom the point of view of experimental design, the seed select method is also a key part, such as Figure 5, but it has not been explained in detail.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach effective with good theoretical support. We address your comments below.\n\n#### **Q1: Core path optimization and centroid method did not elaborate enough.**\n**A1:** We value the reviewer’s feedback to improve our paper. Due to lack of space, we provide more details on this in Sec. C in the Supplementary material. We will elaborate on this section in the final version with additional details.\n\n#### **Q2: The method lacks an advantage with seed-selection, based on results.**\n**A2:** Please refer to the shared response (A1) provided in the shared answer for all reviewers. NAO has two main advantages over SeedSelect: It is 10x faster, and it improves quality as measured by an improvement in terms of FID and accuracy.  See Table2 for the numbers.\n\n#### **Q3: The two interpolation methods compared in this article are very basic**\n**A3:** We compared to SoTA approaches in latent space interpolation, and agree that smarter methods can be used. However, we argue that being aware of the latent space structure plays a significant role, much more than using smarter interpolation methods. The key factor that was so far neglected in latent space interpolation is being aware of the distribution of samples in that space. This is the main contribution of this paper. We are open to evaluating novel methods upon request and appreciate the reviewer's observation, which we acknowledge as a potential avenue for future exploration.\n\n#### **Q4: No matter what kind of Gaussian noise the seed is, SD can generate a relatively reasonable image.**\n**A4:** Thank you for your comment, however, we would like to clarify that this may not be entirely accurate. SD generates reasonable images when the norm of the seeds is close to the mode of the chi-distribution (L 115-123 in the main paper). Randomly sampling from a Gaussian distribution and applying current interpolation techniques results, most of the time, with visually appealing images because the seed norms possess this property. However, in practical scenarios with **real images**, the properties of seeds obtained from inversions may not always be favorable, leading to the failure of current interpolation methods. Our approach considers this prior knowledge and selects seeds that allow SD to generate plausible results.\n\n#### **Q5: [1] was not been explained in detail.**\n**A5:** Indeed, [1] plays a crucial role in rare-concept generation. However, it is important to note that our approach does not depend on SeedSelect and operates independently. Nevertheless, in the final version, we will provide a more elaborate explanation of [1] and offer additional details about their methodology.\n\n#### **Q6: 2D space examples and Figure 1 are difficult to understand.**\n**A6:** We appreciate the reviewer's valuable feedback and are grateful for their efforts to improve our paper. In these figures, the colors correspond to the log-likelihood of the chi distribution. The black points represent real images, while the colored points represent interpolation paths/centroids between these real images. Our approach prioritizes points with high log-likelihood, leading to visually appealing results when using SD. Conversely, other methods disregard the inherent structure of the seed space, resulting in poor-quality images. We acknowledge the need for additional clarification on these figures and will provide more detailed explanations in the text for the final version of the paper and the camera-ready version.\n\n#### **Q7: Formatting errors and typos.**\n**A7:**  Thank you. Will be fixed in the camera-ready version.\n\n#### **References**\n[1].   Samuel et al. (2023), \"It's all about where you start: text to image generation with seed selection\""}]

Author rebuttal:
Dear Reviewers and ACs,    
We were happy to see that the reviewers have found our approach **"interesting and inspiring"**, **"well-motivated" (R1)**, and recognized its **potential to benefit various tasks related to diffusion models' applications (R1, R3)**. Additionally, they have acknowledged that our method is based on **"good theoretical support" (R4)** and have found our experiments to be **“well-designed”**, **“easy-to-follow” (R3)** and **"compelling" (R2)**.    
We have addressed the reviewer's concerns in our rebuttal and are open to further discussion. Your input has been instrumental in improving our paper.
    
    
### General response to all reviewers

NAO presents an innovative method to interpolate between seeds, revealing a new non-Euclidean metric that takes into account a prior over samples. Our efficient algorithm approximates this metric, facilitating centroid estimation in the latent seed space and enhancing the generation of rare concept images. 

#### **Q1: The proposed approach only works well with seed optimization.**
**A1:** Let us explain where and why NAO works well without seed optimization and the relation between the two methods. First, we stress that seed optimization is only necessary for generating rare concepts (See SeedSelect [1] and in lines 29 and 62 of our paper). For rare concepts, diffusion models often fail. They may generate high-quality images but from a wrong category. Seed optimization presents a solution for this scenario.    

NAO functions effectively without relying on seed optimization. This is demonstrated in Figures 1, 3, S3, and Table 1. NAO's interpolations and centroids enable the generation of semantically consistent images even for **common (head)** concepts where SeedSelect is unnecessary. For instance, head classes like "tuna" and "jersey" have multiple interpretations and can lead to the generation of incorrect objects. "Tuna" might denote either a "tuna fish" or a "tuna can," while "jersey" could refer to a "standard t-shirt" or a "sports team shirt".  Given a few reference images with their concept name, NAO can generate images that belong to the correct domain.    

To further illustrate NAO’s advantage, we replicated the experiments from Table 1 solely on 50 randomly selected **common (head)** concepts, **without employing seed optimization**. Subsequently, we compared NAO to two alternate interpolation methods in the provided table. The results unequivocally confirm that NAO achieves significantly higher accuracy (in generating the correct concept) and improved FID.

|                             | Acc (higher is better)           | FID (lower is better)|
| -----------            |  ----------- | ----------- |
| LERP                  | 0.00      | 60.60       |
| SLERP                | 69.12   | 17.55        |
| **NAO-path (ours)**| **88.91**   | **5.21**        |


#### **Q2: What is the constraint c(x)≤0?.**
**A2:** $c(x)$ corresponds to the constraint in Eq. 2, namely $|x_i-x_{i-1}|  \leq \delta$. We formulate this constraint as $c_{i}(x) = |x_i-x_{i-1}| -\delta \leq 0$. This constraint is part of the integral discretization (into sum of finite elements) in Eq. 2. In the implementation we enforce this constraint using a soft penalty term with ReLU function; that is ReLU($c(x)$) (as described in line 212). Now there is a penalty for positive $c(x)$ (when the constraint is not satisfied). We apologize for the brevity in lines 211-212. We will elaborate on the discretization description of this equation in suppl. material and make sure $c(x)$ is clearly defined in text in the revised version.

#### **References**
[1]    Samuel et al. (2023), "It's all about where you start: text to image generation with seed selection".



Meta-review: All four reviewers were in favor of paper acceptance after author rebuttals. The paper provides an interesting new approach exploring/understanding of the latent space of text-to-image models.
Decision: Accept (poster)