AID: Attention Interpolation of Text-to-Image Diffusion
Authors: Qiyuan He
Jinghao Wang
Ziwei Liu
Angela Yao
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion models
training-free
image interpolation
compositional generation
Primary area: diffusion_based_models
TL;DR: AID (Attention Interpolation of Diffusion) is a method that enables the text-to-image diffusion model to generate interpolation between different conditions with high consistency, smoothness and fidelity

Abstract: Conditional diffusion models can create unseen images in various settings, aiding image interpolation. Interpolation in latent spaces is well-studied, but interpolation with specific conditions like text or image is less understood. Common approaches interpolate linearly in the conditioning space but tend to result in inconsistent images with poor fidelity. This work introduces a novel training-free technique named \textbf{Attention Interpolation via Diffusion (AID)}. AID has two key contributions: \textbf{1)} a fused inner/outer interpolated attention layer to boost image consistency and fidelity; and \textbf{2)} selection of interpolation coefficients via a beta distribution to increase smoothness. Additionally, we present an AID variant called \textbf{Prompt-guided Attention Interpolation via Diffusion (PAID)}, which \textbf{3)} treats interpolation as a condition-dependent generative process. Experiments demonstrate that our method achieves greater consistency, smoothness, and efficiency in condition-based interpolation, aligning closely with human preferences. Furthermore, PAID offers substantial benefits for compositional generation, controlled image editing, image morphing and image-controlled generation, all while remaining training-free.

Attention Interpolation for Text-to-lmage Diffusion
Qiyuan He'
Jinghao Wang²
Ziwei Liu?
Angela Yao!.
National University of Singapore qhedu. edu. ag
S-LLab, Nanyang Technological University
ayaоθcomp.. edu.
(jinghao003, ziwei.
Abstract
Conditional diffusion models can create unseen images in various settings, aid- ing image interpolation. Interpolation latent spaces is well-studied, but in- terpolation with specific conditions like text or image is less understood. Com- mon approaches interpolate linearly in the conditioning space but tend result in inconsistent images with poor fidelity. This work introduces a novel training- free technique named Attention Interpolation via Diffusion (AID). AID has two key contributions: 1) fused inner/outer interpolated attention layer to boost im- age consistency and fidelity; and 2) selection of interpolation coefficients via a beta distribution to increase smoothness. Additionally, we present an AID vari- ant called Prompt-guided Attention Interpolation via Diffusion (PAID), which 3) treats interpolation condition-depenndnt generative process. Experiments demonstrate that our method achieves greater consistency, smoothness, and ef- ficiency in iudditinn-baee interpolation. aligning closely with human prefer- ences. Furthermore, PAID offers substantial benefits for compositional genera- tion, controlled image editing, image morphing and image-controlled generation, all while remaining training- free. Our code and demo are available at https://qy- htwpaa/awoo/ttteetttnntteooolloo
1 Introduction
Interpolation is common operation applied to generative image models. It generates smoothly transitioning sequences of images from one seed another within the latent space and facilitates applications in image attribute modification [40, data augmentation (38] and videos [46]. Interpo- lation has been investigated extensively [18
in VAES 20), GANs [ and diffusion mod- els [13]. Text-to-image diffusion models 35 are a new class of conditional generative models that generate high-quality images conditioned on textual descriptions. How to interpolate between distinct text conditions such as "a truck" and "a cat" (see Fig. (d)) is relatively under-explored. This issue is, however, crucial for various downstream tasks, such as conditional generation with multiple conditions 6 25 51 or cross-modality conditions [54] [66, as well for image editing [I] 48),
42
where precise control over impact of different conditions essential to achieve desired results.
This paper formulates the task of conditional interpolation and identifies three ideal properties for in- terpolating text-to-image diffusion models: thematic consistency, smooth visual transitions between adjacent images, and high-qualin, interpolated images. For instance, interpolating from "a truck" to "a car" should avoid irrelevant transitions (e.g., via "a bowl"). The sequence should change between the two conditions gradually and feature high-quality and high-fidelity images (vs. e.g. simple overlays of the truck and cat). These properties directly motivate our quantitative evaluation metrics for conditional interpolation: consistency, smoothness, and fidelity.
direct approach to traverse the conditioning space is interpolating in the text embedding itself [53 55 [6]. Such an approach often has sub-optimal results (see the first row of 2 A closer analysis reveals that interpolating the text embedding is mathematically equivalent to interpolating the keys and values of the cross-attention module between the text and image space. Our analysis further 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

(a) text to text: lady the sea of flowers..." to "Mobile Suit Gundam...""
(b) image image: Mona Lisa Taylor Swift
(c) "Dragon" to "Knight"
(d) "Truck" to "Car"
(f) "Photo of dog" to "Photo of a car", guided with
"A dog driving car" (lop). "A car with dog furry texture" (middle), and "A toy named dog-car" (bottom).
(e) "Ship" to "Airplane"
Figure I: Our approach enables text-to-image diffusion models to generate nuanced spatial and con- ceptual interpolations between different conditions including text (a, c-e) and image (b), with seamless transitions in layout, conceptual blending, and user-specified prompts guide the interpolation paths (f).
reveals that the keys and values in self-attention impose stronger influence than cross-attention, which may explain why text embedding interpolation fails to produce consistent results.
Based analysis, we introduce novel framework: Attention Interpolation of Diffusion (AID) models for conditional interpolation. AID enhances interpolation quality with (1) fused interpo- lated attention mechanism on both cross attenttttttionn and self-attention layers to improve consistency and fidelity and (2) Beta-distribution-bae based sample selection along the interpolation path for in- (3) Prompt-guided Attention Interpolation of Diffusion (PAID) models to further guide the interpolation via a text description of the path itself.
terpolation smoothness. Additionally, we introduce
highlight our approach's 's effec- tiveness (see samples in Fig. [ and more Appx. H) without any additional training.
Experiments on various state-of-the-art diffusion models 30 35 2
Human evaluators predominantly prefer our method over standard text embedding interpolation. We further show that our method can benefit various downstream tasks, such as composition generation, and boost the control ability of image editing. Our method is also compatible with image condi- tion (see Fig. (b)), which can be further used for more application such as image morphing and image- controlledd generation. This underscores the practical impact of the problem of conditional interpolation and our proposed solution. Our main contributions are:
Problem formulation conditional interpolation within the text-to-image diffusion model context and proposing evaluation metrics for consistency, smoothness, and fidelity;
A novel and effective training-free method AID for text-to-image interpolation. AID can be augmented with prompt-guided interpolation (PAID) to control specific paths between two conditions;
Extensive experiments highlight AID's improvements for text-based image interpolation. AID substantially improves interpolation sequences, with significant enhancements in fi- delity, consistency, and smoothness without any training. Human studies show strong preference for the AID;

---

We show that AID offers much better control ability for diffusion- based image editing, and it can be used for compositional generation with state-of-the-ar performance. It is also compatible with image condition, enabling more applications such as image morphing or controlling the scale of additional image prompt.
2 Related Work
Diffusion Models and Attention Manipulation. The emergence of diffusion models has signif- icantly transformed the text-to-image synthesis domain, with higher quality and better alignment with textual descriptions 35
Attention manipulation techniques have been instrumen- tal in unlocking the potential of diffusion models, particularly in applications such as in-painting and compositional object generation. These applications benefit from refined control over the at- tention maps, aligning the modifier and the target object more closely to enhance image coher- ence 1 [144 Furthermore, cross-frame attention mechanisms have shown promise aug- menting visual consistency within video generation frameworks utilizing diffusion models [17 B1]. These works suggest that the visual closeness of two generated images may be reflected in the sim- ilarity of their attention maps and motivates
37
33
study interpolation from an attention perspective.
Interpolation in Image Generative Models. Interpolation within the latent spaces of models such as GANs 8 and VAEs (20 has been studied extensively 4 18 46. More recently, explorations of diffusion model latent spaces allow realistic interpolations between real-world images (38 21].
Works to date, however, are limited to a single condition, and there is lack of research focused on interpolation under varying conditions. [Wang & Golland explored linear interpolation within text embedding to interpolate real-world images; however, this approach yields image sequence with diminished fidelity and smoothness. This gap underscores the need for further exploration of conditional interpolation in generative models.
3
Analysis of Conditional Interpolation
3.1 Text-to-Image Diffusion Models
Text-to-image diffusion models such as Stable Diffusion 335
30
generate images from specified
text. Consider the generation of an image for some specified text as an inference process denoted by f(2".c) The function f is an abstraction representing the denoising diffusion process, is representation of the conditioning text and 2" is randomly sampled latent seed. Usually, c is represented as CLIP text embedding [32]. while the z's over the denoising time steps of the generation are sampled from Gaussian distribution. More specifically, the inference carried out over 7 denoising time steps, the latent 2²-1 can be sampled conditionally, based on z
(I) where L represents the denoising time-step index, pa(2",c, c, is estimated by a UNet (36], and Ez c, ()) is determined by noise scheduler 113 42) After iterative sampling from z" to z the
Pa(2"~112 = Ni (,),,, Ex , c,t)), where 2' ~ (O,
c,
image is generated by decoder D, as ()=").
Attention is used in text-to-image diffusion models B3 29
37
in various forms. Cross-attention
is commonly used as the link from the text condition to the image generation. Specifically, given a latent variable E Rd, text condition Rd and the attention layer with matrices Wq E 1Rd, xd, WK E Rd.xda and Wy E Rdd Xd., the cross-attention is computed as
QKT Vd
A(2,c) Attn (am(QQA
K, Q,K,
softmax
VV,
where Q WT², K W.C. V WTe. (2)
Self-attention is also commonly used in state-of-the art text-to-diffusio models 30 35 2]. Self- attention is a special case of cross-attention and can also be computed with Eq. 2 as A z). In this case, the key and values are defined K W V W/= respectively. For brevity, we abuse the notation directly represent multi-head attention 47 and denote the attention layer as Attn(Q, K, V) in both cross-attention and self-attention scenarios.
3

---

(a) "an apple" to "a bed"
(b) "a lady wearing oxygen mask " "a
Figure 2: Results comparison between AID (the 1st row) and text embedding interpolation (the 2nd row). AID increases smoothness, consistency, fidelity significantly.
3.2 Text Embedding Interpolation
In this paper, we denote linear and spherical interpolation 49 21 as ry (5T7) [w; A. B) and r. (w; A, B) re- spectively, where E [0. 1] is the interpolation coefficient, and (A, B) are the interpolation anchors or end-points. Conditional interpolation differs from standard text-to-image generation in that there two text conditions e₁ and Cm. Each condition has its own respective latent seeds 21 and 2m The objective of conditional interpolation is to generate a sequence images {f1m = {41, Im In this sequence, the source images 1 (1) m = (zm, Cm) as described in Sec. 3.1
generated by the standard text-to-image generation, i.c,
Existing literature has shown that similarity in input space, including the latent seed and the embed- ding of the condition reflects the similarity in the output pixel space [9 49) Directly interpolating the text embedding is therefore straightforward approach that can be used to generate an in- terpolated image sequence [49 [6]. In text embedding interpolation, the text conditions {cI, Cm) and their latent seeds {=1, 2m are used as endpoints for interpolation and images are generated accordingly:
i-1 for {1...m)) m]. (3)
1== f(2, G) where 2, ::=r((u;; 21, 2m), G2 = 01, Cm); and w; m
Note that spherical interpolation is applied to estimate the latent seed 2; to ensure that Gaussian noise properties preserved (38]. In contrast, linear interpolation is applied to estimate the text condition [53 49 |6 24) For both 2; and c,, the interpolation coefficient w; sampled in uniform increments from to m.
Given that the text condition e directly propagates into the key and value and (Eq. interpo- lating between the text conditions 0i and Cm is equivalent to interpolating the associated keys and values in cross-attention. This is stated formally in the following proposition:
Proposition Given query Q from a latent variable z. keys and values {K1, V1} and {Kms Vm from text conditions {ci, Cm and linearly interpolated text conditions C₁ the resulting cross- attention module A(2, C2) is given by linearly interpolated keys and values K; and V₁:
A(z,c; = AQt K., V.), where K; = rz ri(w;; K1, Km) and V. = rr(w;; Vi, Vn), (4)
where is defined similarly as Eq.
The proof for Proposition 1 is given the Appx. A This proposition gives insight into how text embedding interpolation can be viewed manipulating the keys and values. Specifically, it is equivalent to interpolating the keys and values to generate the resulting interpolated image. It is worth noting that an analogous interpretation does not carry through for the query Q even though it also depends on some interpolated latent seed z². This is because 2 is estimated as an interpolation between the latent seeds 2 and 2/ while the latent 2f itself is progressively altered through the denoising process (see Eq.. 1).
3.3 Measuring the Quality of Conditional Interpolation
Text embedding interpolations work well when the conditions are semantically related, e.g., dogg" and "a car", but may lead to failures in less related cases. To better analyze the characteristics of the interpolated image sequences, we define three measures based on ideal interpolation qualities: consistency, smoothness, and image fidelity.
We use subscripts on to denote latent seed indices without expressing the T' denoising timesteps explic- itly, i.e. of and I

---

Perceptual Consistency. Ideally, the interpolated image sequence should transition from one source or endpoint to the other in perceptually direct and therefore consistent path. Similar [[), we use the average LPIPS metric [57] across all adjacent image pairs in the sequence to evaluate consistency. If denotes the L.IPS model, the consistency C of sequence /im defined as:
1 C(limi P) P)= m
4(1))
For example, consistent interpolation from "an apple" to "a bed" may pass through "an apple and a bed" but should not have intermediate stages like "a messy sketch" (see (a).
Perceptual Smoothness. A well-interpolated sequence should exhibit gradual and smooth transi- tion. We propose to apply Gini coefficients on the perceptual distance between each neighbouring pair of interpolated images to indicate smoothness. Gini coefficients [5] are conventional indicator of data imbalance [5 10 50] where higher coefficients indicate more imbalance. And the imbal- ance of perceptual distance of each neighbouring pair indicates low smoothness. Let G(X) denote the Gini coefficient of a set X = {I1, 42, ..., J. The smoothness S of a sequence 41m with P denoting the LPIPS model is defined as:
m-1
G(X) = Ei= E 24 Izl
S(Tmmi P)= = 1 G( U (4))))) d=1
(6)
2n 21=1
Fig. b) shows how a smooth interpolation sequence exhibits a gradual transition on the visual content (from "a lady wearing oxygen mask" to "lion" " in the top row) instead of one source image or end-point dominating the sequence (the "lion" in the bottom row).
Fidelity. Finally, any interpolated images should be of the same (high) quality conventionally gen- erated images. Following (38 [4]] we evaluate the fidelity of interpolated images with the Fréchet
Given n interpolated sequences {T [/2) _....... I(m) }. the fidelity
Inception Distance (FID) 12
F of the sequences is defined as the FID based on visual inception model 2 The FID between the source images and the interpolated images is defined as:
(ÜMP) 1 j=1
(1) F P
((m) = FIDM
For example, the interpolated sequence should have minimal artifacts (see Fig. aa)), where the top row clearly shows the appearance of the apple, whereas the bottom row does not.
3.4 Diagnosing Text Embedding Interpolation
Experimentally (see Sec. 5.1 we observe that text embedding interpolation sequences exhibit poor consistency and smoothness. The interpolated images are also commonly low in fidelity, with indi- rect and non-smooth transitions. Where do the failures of text embedding interpolation come from? We analyze the outputs from the perspective of spatial layouts and the selection of interpolation coefficients.
Spatial Layouts and Attention. Consistency is directly affected by the difference between the spatial layout of the source and interpolated images. One observation is that the spatial layout of interpolated images from text embedding interpolations is quite different from the source endpoints (see Fig. 2 (a) bottom row). Proposition links the text embedding interpolation to the cross- attention mechanism exclusively. However, the literature suggests that the spatial layout of the overall image is strongly linked to the self-attention mechanism 17 31.. As such, we hypothesize that cross-attention does not pose enough spatial layout constraints stand-alone. Instead, there is a need for a stronger link of the interpolation to self-attention, to allow more consistent spatial transitions.
As a simple test, we swap the keys and values from two text-lo-image generations. Consider two images I and I' generated from two text prompts and p'. We replace the keys and values from
TTyiccally, Inception v3 44 My is used.
5

---

PAID: Prompt guided Actention Interpolation of Diffasion
Power Berrphhr
"Aphonsatrua""
AstiiQ, AL,
Deta petor inlectioo
steps
Figure 3: An overview of PAID: Prompt-guided Attention Interpolation of Diffusion. The main com- ponents include: (1) Replacing both cross-attention self-attention when generating interpolated image by fused interpolated attention; (2) Selecting interpolation coefficients with Beta prior; (3) Inject prompt guidance in the fused interpolated cross-attention.
either the cross-attention self-attention layers in the generative process of I with that of I' to generate fcross and setr respectively. We then evaluate mean squared error (MSE) of the low- frequency components between I' and Leross or Leet- The results show that Leer closely resembles I', while Leress does not. More details and results are shown in Appx.
Selection of Interpolation Coefficients. Interpolation methods 52 formly spaced coefficients w2 on the interpolation path. Yet an observation from Fig. 2 (b) shows that uniformly spaced points in the text-embedding space do not lead to uniformly spaced images with smooth transitions. Small visual transitions may occur over large range of interpolation co- efficients and vice versa, which we can show quantitatively by comparing the perceptual distances between adjacent pairs uniformly spaced coefficients. This suggests that we should adopt non- uniform selection to ensure smoothness. More details and results are shown in the Appx. B
38 commonly select uni-
4 AID: Attention Interpolation of Text-to-Image Diffusion
The diagnosis in Sec. (3.4 directly leads us to make the following proposals for improving conditional interpolation. First, we motivated extend attention interpolation beyond cross-attention to self- attention as well (Sec. 4.1 and propose fused attention. Secondly, our diagnosis of the smoothness motivates us adopt non-uniform selection of interpolation coefficients encourage more even transitions (Sec. 4.2). Combining these two techniques, we propose AID: Attention Interpolation of text-lo-image Diffusion.
Finally, in an effort to give more precise control over the interpolation path, we introduce the use of prompt guidance for interpolation (Sec. 4.3). This further enhances AID as frompt-guidance AID (PAID). The full pipeline is shown in Fig.
4.1 Fused Interpolated Attention Mechanism
The analysis in Sec. 3.4 highlights that both cross-attention and self-attention likely play role in interpolating spatially consistent images. Proposition can be generalized to self-attention where the keys and values are derived from the latent instead of for enhancing spatial constraint. As such, we define a general form of inner-interpolated attention on the keys and values as follows:
Intp-Attm, (Qrs K1sm. Vimii wi
Attn Qú. (1 - K1 + W.Kmm (1 -w; + w. Vm). (8)
where Qi is derived from 2;. Note that Eq. g is equivalent to Eq. if (K1, Km) and (Vi, Vn) are derived from {c1, J. i.e. cross-attention; if they derived from {21, Zm then represents self-attention.
Instead of applying interpolation to the key and value, we can also interpolate the attention itself. We define this as outer-interpolated attention:
Intp-Attn (Qi, Kiam Viam ws) = w:) Aun (an(eer. (1,, K1, Ki, + w. Aun Qi, Km, Von). (9)

---

Similarly, Eq. can represent both cross- and self-attention, depending on if KK. and (V, Vn.) y are derived from (c or {21, [m respectively. More details on the differences be- tween inner and outer interpolation are given in the Appx. and AID-O for inner and outer interpolation respectively.
We denote the two versions as AID-I
While applying interpolation as defined in Eqs. X and
| for self-attention does lead to high spatial consistency, it also results in poor fidelity images. This is likely because directly replacing the self- attention mechanism with some interpolated version is too aggressive. Therefore, for self-attention, we maintain the source keys and values K; and V, from the interpolated 2, and concatenate them with the interpolated keys and values, as shown in Fig. 3 Denoting concatenation as [, we define fused attention interpolation, leading to a fused inner-interpolated attention:
Intp-Autff (Q:, K K1:, Vi,mi 10:) = Attn Qi, [(1 [11 w;) K ww Km; Ki], [(1 (1-mm)Vi++u Vn, Vi]
(10)
For self eeff-attention, as K2 is derived from 24, K; * (1-w;) w Km the same holds for V,. For
cross-attention, however, K; (1 - w; + w; w,KKm so fusing the two does not provide additional benefits. We note that there are opportunities for fusion with keys and values derived from other sources. We follow such strategy in Sec. [43 to inject additional text-based guidance.
Analogous to Eq. 9 we define a fused outer-interpolated attention:
Intp-Attn Q2: K1:m V1:mi W:) (1 - (Qi, [K1, K,), [V2, Vil)
(II)
+ +u. (Km, Ki), [Vm, Vil)
4.2 Non-Uniform Interpolation Coefficients
The analysis in Sec. 3.4 shows that interpolation coefficients should not be selected uniformly adopted in previous methods [T]] on the interpolation path. For more flexibility, we apply Beta distribution (t ax .3.. Beta distributions are conveniently defined within the range of [0, 1]. When a= and 1, Ps degenerates to a uniform distribution, which reverts to the original set- ting. When a > and 3 > 1, the distribution is concave (bell-shaped), with higher probabilities away from the end-points of O and 1, i.e. away from the source images. Finally, the selected points are adjustable based on alpha and beta values, to give higher preference towards one or the other source image (see Fig. 3).
Given the Beta prior represented as cumulative distribution function FB(w, a, B). we define a Beta- interpolation (ww 0, 1) as r(F (w, a, B)). where w ~ U(0, 1). Therefore, the distributed point with Beta prior becomes:
(r(0),
1,,,01",''70 a, B)), r (1 m
m a
- a, ()),)
(12)
In practice we employ dynamic selection process to adjust the a and 9 parameters of the Beta prior, and form the smoothest sequence from the explored coefficients. Further details are provided in Appendix D
4.3 Prompt Guided Conditional Interpolation (PAID)
Given two source inputs, the hypothesis space of interpolation paths is actually large and diverse. Yet most interpolation methods [52] 38) estimate one deterministic path. Can we control or specify the interpolation path? One possibility is to provide (third) conditioning text, which we refer to as a guidance prompt. To connect the interpolated sequence with the text in the guidance prompt g, we fuse the associated key K2 WT gg and value Vg WTg instead of the original K; and V; in the fused inner-interpolated attention Eq. for cross-attention:
Guide-Attnf (Qi, K1.m, Vi:m: Wi, Kg. Vz))
(13)
Attn (Q:, [[1- - W; i +
w;Km,, Kg]: [[1 w[--w;VVi + w; Vn, vol).
In practice, the guidance prompt is provided by users to choose the interpolation path conditioned on the text description as Fig. (f) shows. We demonstrate that the prompt-guided attention inter-
polation dramatically boosts the ability of compositional generation in Sec. 5.2
7

---

Datroot
Methed
Smoethnce (1)
Concitency (.)
Fideley ())
CIFAR-10
D1
TR
a,754
07511
0.4295
0,3645
118,05
K3.13
0.3531
0.344
13516
AID O
0,7831
0,2905
51.43
AID-1
0,7861
0.3271
101,13
8
0.366
43321
128,89
THI
0.7424
0.3867
142.58
62817
03482
135
LAION Aesthetics
DI
0,7511
0:6365
101.31
AID-O
0.7613
0.2948
K2.01"
AID-1
0S152*
0,5787
129.41
0.301 0.7905
$1.03
(b)
Table 1: Quantitative results of conditional interpolation. Quantitative results where the best performance marked as (*) and the worst marked as red. (a) Performance on CIFAR-10 1I and and AID-I both show significant improvement over the Text Embedding Interpolation (TEI). Though Denoising Interpolation (DI) achieves relatively high fidelity, there is a trade-off with very bad performance on consistency (0.4295). AID-O boosts the performance in terms of consistency and fidelity while AID-I boosts the perfor- mance of smoothness; (h) Ablation studies AID-O's components, showcase that the Beta prior enhances smoothness, attention interpolation heightens consistency, and self-attention fusion significantly elevates fi- delity.
AID-O
5 Experiments
Configuration and Settings. We evaluate quantitatively based on the three measures for conditional interpolation defined in Sec. 3.3 and user studies. Detailed experimental and application configura-
tions are given in Appxs. F and
We use Stable Diffusion 1.4 [35 as the base model to implement our attention interpolation mecha- nism for quantitative evaluation. In all experiments. 512 512 image generated with the DDIM Scheduler [42] and DPM Scheduler (26) within 25 timesteps. Additional qualitative results using
other state-of-the-art text-to-image diffusion models 30 23
are given in Appx.
5.1 Conditional Interpolation
Protocol, Datasets & Comparison Methods. For experiments in each dataset, we run 5 trials each with N = 100 iterations. In each iteration, we randomly select two conditions and generate an interpolation sequence with size = 7. We report the mean of each metric of the interpolation sequences over all trials as the final result. Our proposed framework is evaluated using corpora from dataset from the larger LAION-5B collection [39 To the
CIFAR-10 22 and the L.AIN--eeshhete best of our knowledge, the only related method is the text-embedding interpolation (TEI)
49 53 55
(see Sec. 3.2). We also compare with Denoising Interpolation (DI), which interpolates along the denoising schedule; more details in are given in the Appx. F
Results. We quantitatively evaluate our methods based the evaluation protocol as shown in Tab AID-O significantly increases the performance of all the evaluation metrics. AID-I achieves higher smoothness, AID-O has significant improvements in consistency (-20.3% on CIFAR- 10 and -23.9% on -estheesss and fidelity (-66.62 on CIFAR-10 10 and -60.37 on LAION-Acstheticn The fidelity of AID-I is poorer than AID-O and worse than Denoising Interpolation. However, AID-I achieves competitive qualitative results as shown by the user study.
Ablation Study. Tab. I shows ablations of the AID-O framework with CIFAR-10 10, focusing on three primary design elements: attention interpolation, self-attention fusion, and veta-interpolation. Results show that attention interpolation improves consistency while Beta-interpolation contributes to improvements in smoothness and self-attention fusion to enhance image fidelity. While atten- tion interpolation (without fusion with self-attention) with Beta-interpolation achieves the highest smoothness, it does so at the cost of fidelity. Similarly, AID without Beta interpolation achieves the strongest consistency but trades off smoothness (see Fig. Fig. (a) provides qualitative comparison between different ablation settings.
User Study. Using Mechanical Turk, we check for human preferences on four types of text sources: 1) near objects, such as dogs and cats; 2) far objects, such as dragons and bananas; 3) scenes, such as waterfalls and sunsets; and 4) scene and object, such as a sea of flowers with a robot. This variety provides a comprehensive assessment for both concept and spatial interpolation. We conducted 320 trials in total; in each trial, an independent evaluator was asked to select their preferred interpolation result. Tab. 2 shows that our method almost always preferred, though the preference is split across AID-I and AID-O depending on the type of text sources.

---

0.50
CoSH
COSMMCNI
Tasee wIS stanties
00:15
0.30
0.05
poke
(a)
(b)
Figure 4: Qualitative comparison of different ablation setting of AID. (a) Qualitative comparison between AID without fusion (1" row), AID with fusion (2td row), and AID with fusion and beta prior (3th row). Fus- ing interpolation with self-attention alleviates the artifacts of the interpolated image significantly, while beta prior increases smoothness based AID with fusion. (b) CLIP score of different methods composition generation.
Interpolation method
Near Object
Far Object
Scene
Object+Scene
biitting Method
Smoothness
TEI AID-I
8.75%
1.16%
0%
1.26%
P2P
0.3741
53.75%
50%
45.2%
45.57%
P2P + AID
0.8921 (0.5180
AID-O Hard to determine
36.25%
46.5%
51.90%
EDICT
0.5978
1,25%
2.32%
4.76%
1.26%
HDICT + AID
0.8486 (0.2508 1)
(a)
(b)
Table 2: Human evaluation results. (a): Human preference ratio of each method in different categories of interpolation, AID-I, and AID-O are dominantly preferred by TEI; (b): Smoothness of different editing methads, combined with AID boosts control ability on editing level.
5.2 Application
In this section, firstly introduce how to adapt methods into applications including image edit- ing control and compositional generation. We further extend to cross-modality conditions including image prompt well with IP-Adapter (54), which enables applications including image morphing and image-controlled generation. We provide more details Appendix.
Image Editing Control. Text-based image editing tries to modify image based textual description (see Fig. 5). Existing methods [16] [1 48) rely on text embedding interpolation to control the editing level. Training-free methods 48 struggle to control the editing level based on the text, while ours does not. We validate the control ability of our methods using Prompt-to- Prompt [11 (P2P) for synthesized image editing and EDICT [88
for real image editing.
We evaluate the ability to control the editing level using the smoothness metric defined in Sec. 3.3 using the image editing dataset presented in 48). Given an image with an editing level of and the original image with an editing level of 0, we use either TEI or AID-O to interpolate edited images with levels ranging from ( 8 8..... ) and assess the smoothness of the edited image sequence.
Quantitative results are reported in Tab. 2 (b). Our method greatly improves the smoothness of the edited image sequence, aligning with different editing levels and thereby enhancing the control ability for editing. As shown Fig. P2P alone cannot effectively control the editing level but combining it with AID allows for precise level adjustments.
Compositional Text-to-Image Generation. Compositional generation is highly challenging for text-to-image diffusion models (5
6 7 In our experiments, we focus on concept conjunction 6 generating images that satisfy two given text conditions. For example, given the conditions "a robot" and "a sea of flowers," the goal is to generate an image that aligns with both "a robot" and "a sea of flowers."
For compositional generation, we use PAID to interpolate between conditions Ci and 02 with the the same dataset for human
prompt guidance "c, AND oz". ". For quantitative evaluation, evaluation as Sec. 5.1 and CLIP scores [32 to evaluate if the generated images align with both conditions. We compare our methods with vanilla Stable Diffusion [35 30 and two other of the-art training-free free methods: Compositional Energy- based Model (CEBM) 25 and RRR 7
Fig. (b) shows that the CLIP score of our method is higher than previous methods for both Stable Diffusion 1.4 (55 and SDXI. R0) Moreover, our method produces fewer artifacts such merging the two objects together, as illustrated in Fig.

---

(a) dog sitting the grass."
(b) "A boy is angry happy."
Figure 5: Results of image editing control. Our method boosts the controlling ability over editing. The first of and (h) generated by P2P ATD while the second row is P2P TEI.
(a) Vanilla
(b) CEBM
(c) RRR
(d) PAID
Figure 6: Results of compositional generation. Images the left generated with "a deer" and "a plane"
based SD 1.4 [5] and images the right are generated with "a robot" and "a sea of flowers" based SDXL. 30 Compared to other methods, PAID-O properly captures both conditions with higher fidelity.
(a) Image morphing between real images
(c) "A boy is smiling." composition referenee Figure 7: Results of AID with image conditions. Our method is compatible with IP-Adapter for image- conditioned generation (a). In both global image prompt (b) and composition image prompt (c), from left to right the scale of additional image prompt slowly increases. The first illustrates results controlled by AID, while second row shows results achieved using scale setting provided by IP-Adapter.
(b) "A statuc is running." " global reference
Image Morphing and Image-Controlles Generation. Image morphing finds transitions between two images, while image-controlled generation creates images based on a text prompt with an addi- tional image prompt. To enable generation with image condition, adapt AID on IP-Adapter (54). IP-Adapter integrates image embeddings into cross-attention layers, allowing diffusion models to incorporate image prompts. For morphing, we use an empty text prompt and apply AID for smooth interpolation between image conditions. In image-controlled generation, AID adjusts the image prompt scale across endpoints, enhancing control.
Our method enables effective image interpolation (Fig. 7 (a)) and offers finer control than IP- Adapter. As shown in Fig. [ (b), AID maintains both text and image alignment, while in Fig. (c), it better preserves identity while following compositional references. Further comparisons are provided in Appendix G
Conclusion
In this work, we introduce a novel task: conditional interpolation within a diffusion model, along with its evaluation metrics, which include consistency, smoothness, and fidelity. We present novel approach, referred to as AID and PAID, designed to produce interpolations between images under varying conditions. This method significantly surpasses the baseline performance without train- ing, demonstrated through both qualitative and quantitative analysis. Our method is training- and broadens the scope of generative model interpolation, paving the way for new opportunities in various applications, such compositional generation and image editing control.
10

---

References
[1] Tim Brooks, Aleksander Holynski, and Alexii A Efros. Instructpix2pixx Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
[2] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o; Fast training of diffusion trans- former for photorealistic text-lo-image synthesis. In ICLR, 2024.
[3] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Woi Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems, 36, 2024.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pp. 248-255, Icee, 2009,
[5] Robert Dorfman. A formula for the gini coefficient. The review of economics and statistics, pp. 146-149, 1979.
[6] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy-based models, Advances in Neural Information Processing Systems, 16666776667, 2020.
[7] Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl- Dich-Diicsttin, Amaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and MCMC. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 8489-8510. PMLR, 2023.
[8] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1710.09412. 2014.
[9] Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-9208, 2021.
[10] Qiyuan He, Linlin Yang. Kerui Gu, Qiuxia Lin, and Angela Yao. Analyzing and diagnosing pose estimation with attributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4821-4830, 2023.
[11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross-attention control. In ICLR. OpenReview.net, 2023.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[15] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401 4410. 2019.
[16] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
11

---

[17] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint
2023.
[18) Siavash Khodadadeh, Sharare Zehtabian, Saeed Vahidian, Weijia Wang, Bill Lin, and Ladislau Bölöni. Unsupervised meta-learning through latent-space interpolation in generative models. arXiv preprint arXiv:2006.10236, 2020.
[19] Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets. Understanding ddpm latent codes through optimal transport. arXiv preprint arXiv:2202.07477 2022.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.61 14, 2013.
[21] Tetta Kondo, Shumpei Takezaki, Daichi Haraguchi, and Seiichi Uchida. Font style interpola- tion with diffusion models. arXiv preprint arXiv:2402.14311 2024.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Cagliostro Research Lab. Animagine xl 3.1. httpp://uuugnnngaaee..ccccooooo animagine-xle 1x-.ginneex--.1111 2024.
[24] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612- 2022.
[25] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[27] Weijian Luo, Tianyang Hu, Shifeng Zhang. Jiacheng Sun, Zhenguo Li, and Zhihua Zhang- Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion mod- els. Advances Neural Information Processing Systems, 36, 2024.
[28] OpenAl. GPT-4. https://openai. gpt 2023.
[29] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
[30] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
[31] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535, 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML,, Proceedings of Machine Learning Research, 2021.
[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 1(2)3, 2022.
[34] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assited Evervention-MICCAN 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedingg, Part III 18, pp.
Springer, 2015.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photore- alistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems. 35:36479-36494, 2022.
[38] Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm-guided latent space exploration for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[39] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-25229 2022.
[40] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disen- tangled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 44(4):2004-2018, 2020.
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
[42] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[43] Lukasz Struski, Jacek Tabor, Igor Podolak, Aleksandra Nowak, and Krzysztof Maziarz. Realism index: Interpolation in generative models with arbitrary prior. arXiv preprint 1rXiv:1904.03445, 2019.
[44] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
[45] Suryakanthi Tangirala. Evaluating the impact of gini index and information gain on classifi- cation using decision tree classifier algorithm. International Journal of Advanced Computer Science and Applications, 11(2):612-619 2020.
[46] Quang Nhat Tran and Shih- Yang. Efficient video frame interpolation using generative adversarial networks. Applied Sciences, 10(18):6245, 2020.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[48] Bram Wallace, Akash Gokul, and Nikhil Naik. EDICT: exact diffusion inversion via coupled transformations. In CVPR, pp. 22532-22541. IEEE, 2023.
[49] Clinton J Wang and Polina Golland. Interpolating between images with diffusion models. arXiv e-prints, pp. arXiv-2307, 2023.
[50] Jinghao Wang. Zhengyu Wen, Xiangtai Li, Zujin Guo, Jingkang Yang, and Ziwei Liu. Pair then relation: Pair-net for panoptic scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[51] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Compo- sitional -et--t synthesis with attention map control of diffusion models, arXiv preprint arXiv:2305. 13921, 2023.
13

---

[52] Cheng Yang, Lijing Liang, and Zhixun Su. Real-world denoising via diffusion model. arXiv preprint urXiv:2305.04455 2023.
[53] Zhaoyuan Yang, Zhengyang Yu, Zhiwei Xu, Jaskirat Singh, Jing Zhang, Dylan Campbell, Pe- ter H. Tu, and Richard Hartley. IMPUS: image morphing with perceptually-unifom sampling using diffusion models. In ICLR. OpenReview.net, 2024.
[54] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0672 2023.
[55] Kaiwen Zhang, Yifan Zhou, Xudong Xu, Bo Dai, and Xingang Pan. Diffmorpher: Unleashing the capability of diffusion models for image morphing, In CVPR, pp. 7912-7921. IEEE, 2024.
[56] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 586-595, 2018.
[58] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
14

---

A Preliminaries and formulation
Linear / Spherical Interpolation. Given tensor A and tensor B, the linear interpolation path rr(w)
where w E [0, 1] is defined as:
ri B) = (1 - w) + wB
(14)
The spherical interpolation is defined as:
(w:; A, B) sin(1 w) sin 0
sin wf
A
A +
B, O = arcos
(15)
sin 0
|||||||B||
Distinction on the Discrete Sequence and Continuous Path. Our formulation diverges from previous studies by concentrating on the assessment of discrete samples, referred to the interpolation sequence, instead of the continuous interpolation path. This is crucial because the quality of the interpolation sequence is determined not only by the interpolation path's 's quality but also by how select the exact sample along the interpolation path, which previous methods overlook. Additionally, the size of an interpolation sequence is often low in practical us- As result, our evaluation framework is specifically designed to cater to interpolation sequences.
High cersetency High smoothness
High coraatercy. Low amsaltrees
conistect,,yy High cresutrness
age 38 52
Figure 8: Difference between smoothness and consistency in measurement of discrete sequence.
This distinction significant when evaluating smooth- ness and consistency as Fig. 8 shows. While Perceptual Path Length (PPL) [15] indicates both smoothness and consistency on the continuous path, where the PPL of the blue path shorter than the green path, this does not hold in discrete sequences. The sequence can have bad smoothness even if it lies on smooth interpolation path (see the blue triangle).
Proof of Proposition 1. Proposition 1 indicates that interpolating text embedding linearly is equiv- alent to interpolating key and value in the cross-attention mechanism. The proof is straightforward by decomposing the formula of the attention layer as follows:
A(zi, G3) = Att K;, Vi)
Attn(Q;, W&G W ci)
1
Attn W T TT( m ; C1, Cm), W WpTl(
17 ; c1, Gm))
(16)
AttniQQ r
m WF W fm)) |
Wif,, c1, (F)m))
m
i-1
Attm(QQ. m
K1, Km), Ti
Vi, Von))
m
B Diagnosis of Text Embedding Interpolation
Controlled Experiments on the Key and Value of Attention. To conduct the analysis on the key and value of self-attention and cross-attention, we analyze the effect by replacement experiments. Specifically, given two conditions and c', we first generate I and I' accordingly. They replace all the key and value of either cross-attention or self-attention during the generation of l' to the key and value computed from I, which incurs two new generated images including T'ross and Inmyy. If self-attention is more important to constraint the spatial layout, The images obtained by replacing self-attention T'atf should be more similar to 7 compared I Terass-
To quantitatively verify this, consider two images sharing more similar spatial layouts should have lower differences in the low-frequency information. Therefore, we evaluate the difference in the spatial layout of the two images by directly evaluating the L2 loss the low-pass. images. Specifically, it can be written as:
1 - -IG(I:)) G(T::o)|1
Da 1':)
(17)
15

---

2550
020
g015
2.10
500
OGIT 0.00
(a) I (b) P'
(c) L'eross
(d) l'atf
Figure 9: Diagnosis of text embedding interpolation on spatial layout (a and adjacent distance (f). (a) Image generated by "a cat wearing sunglasses"; (b) Image generated by "a dog wearing sunglasses", (c) Replacing the cross-attention during generation of (b) by (a); (d) Replacing the self-attention during generation of (b) by (a); (e) Box plot of De [[LL, ('ross)) and Dao(I, (.elf) When fixing a query, the key and value in self-attention mostly determine the output of pixel space com- pared cross-attention. (f) The maximum adjacent distance and the average of other adjacent pairs.
where Gi(: o)) represents Gaussian blurring kernel with parameter a. We conduct our experiments based on the corpus in the form of class names of CIFAR-10 [22), which we introduce in Sec. 5.1 We run 100 trials and generate two images at each trial then compare the difference between (I (ett)) and Da l'ooss)..
Based on our empirical verification shown in Fig. 9 (c), 2a((,,'eef) 2 O, which indicates that the spatial layout of !'elf is almost the same as I, while Dai(I, T'sel) >> Da (I, (roos)) indicating key and value of self-attention impulses much stronger spatial constraints on the generation than cross-attention.
Non-smooth Distance Among Adjacent Pairs. Selecting uniformly distributed interpolation co- efficients in text embedding interpolation, commonly does not result in uniform visual transition in the pixel space. Instead, we found that small visual transitions may occur over large range of interpolation coefficients, and vice versa. To quantitatively verify this, randomly draw two text conditions from the same corpus of CIFAR-10 10 [22] and apply text embedding interpolation with uniformly distributed coefficients {0, 0.25, 0.5, 0.75, 1) generate interpolated images. Then evaluate our observation by comparing the maximum distance of four adjacent pairs and the average of other distances. As Fig. (f) shows, the maximum distance is often much larger than the average distance of other adjacent pairs, indicating that abrupt visual transition occurs in a short range of interpolation coefficients transition.
C Outer vs. Inner Attention Interpolation
Mathematical Induction. We start by comparing the formula of outer interpolated attention and inner interpolated attention. We expand the inner interpolated attention defined in Eq. Das follows:
(on-AAttn,QQ... K Kirm Viem (;)
Attn(Q,, (1 - K4 + ttK (1 +
Q:((1 Ki + tm
softmax
)[(1 - 4iV + 4 Vm)]
Vdle Qi((1 (1,K- + 4KKm] T VV
(18)
(1 - 44) softmax(
Vdl
Qil(1 - K1 + KtK,, T
+ t, softmax
Vm
Vdlk
Similarly, we expand the outer interpolated attention defined in Eq. 10
Intp-Attn, K1.m, Vi:mita)
(1 t2) Aun(Q,, K1, Vi) + tt Attn (Qi, Km, Vm)
(19)
K.KT Vi + softmax( Q.KT m V
= (1 4x) softmax
Vdl.
Comparing Eq. 18 and Eq. 19 above, the essential difference is: while inner attention interpola-
tion uses the same attention map softmax( ( +G, fusing source keys K₃ and Km for
Vds
16

---

(b) "Louis Henry Sullivan (September 3, 1856 April 14, 1924) was an American architect..." to "Modern Landscape Painting Zion by Johnathan Harris" " Qualitative results from LAION-Aesthetics. For each pair of prompts, the first row is the Input Interpolation, the second row is AID-O and the third row is AID-I. Our methods provide direct and smooth interpolation in spatial layout and style, with high fidelity.
(a) "Fox- Watercolor Art Print" -hri--Chriima--fomm-aaa Funiclll j60,"peg""
Figure 10:
(a) "Dog Oil Painting" " "Bird Chinese Painting"
(b) "bonana" to "pen"
Figure 11: Qualitative comparison between AID-O (the 1st row) and AID-I (the 2nd row). While AID-O prefers keeping the spatial layout, AID-I prefers interpolating the concept and style. Comparing the 4th column (h), AID-I properly captures "pen the shape of banana" while AID-O provides banana but the spatial layout the same as the pen.
different source value 1/1 and Vm. outer attention interpolation, on the other hand, using different attention maps for different source key and value. This may answer why the AID-I tends con- ceptual interpolation fusing the characteristics of two concepts into one target but AID-O tends to spatial layout interpolation allowing the simultaneous existence of two concepts in the interpolated image.
Qualitative Results. We observe that AID-I prefers interpolation the concept style. On the other hand, AID-O strongly enhances perceptual consistency and encourages interpolations in the spatial layout of images, Fig. shows. Even when interpolating between two very long prompts, both methods can achieve direct and smooth interpolations with high fidelity as Fig. T0 shows.
D Selection with Beta Prior
D.1 Intuition behind Beta Prior
Based on our analysis in Sec. 3.4 of interpolation coefficients. This indicates that need to select more points in that small range rather than uniformly select coefficients between [0, 1].
3.4. and Sec. B. the transition often occurs abruptly in small range
We hypothesize that this is because: different from interpolation in the latent space, which is only introduced the initial denoising steps, the diffusion model incorporates the text embedding for multiple denoising steps. This may amplify the influence of the source latent variable with higher coefficients. Therefore, when is close to or 1, r' r'(t) is closer to 0, leading to the intuition that we want to sample more mid-range t.
Based on our heuristics above and the empirical observation in Sec. B we apply Beta prior, which is bell-shaped distribution when and 3 are both larger than 1, encourage more coefficients smaller range of interpolation coefficients. Furthermore, we can de-bias the visual transition towards one endpoint to make it smoother by adjusting B, or vice versa.
D.2 Dynamic Selection
The main challenge of the entire selection procedure for interpolation coefficients lies in the time cost. Firstly, the ideal interpolation coefficients can vary depending on different combinations of conditions and even different latent seeds. Secondly, exploring each new interpolation coefficient
17

---

Algorithm Exploration with Beta prior
Input:
Exploration size nt, the initial Gaussian Noise 21, An. two conditions C1, Ca. generative
process represented as f(2 Fas C1 Gnj Wi), perceptual distance function P(.*). CDF of Beta distribution 1lla.9)
Output: An image sequence
Initialize the list of explored coefficients: w [0,
Initialize image sequence I = [f [21 2n. C1. Cui 0), fz1, Zn. C1, Caš 1)] Initialize the list of distance of each neighbouring pair:
d + [f Zn. Cni 0), [21, Zn, C1 Cni 1))]
Initialize the hyperparameter: a and B + 1, iteration number +- 0
for i < n do k argmax(d)
> Search for the current largest distance and get its index Select coefficient uniformly separate the distance under Beta prior p(a.3)
w/ pla.8) (I (wx) (wxx) +
(w2+1)/2)
Update exploration state
Remove dg from
d.insert(k:, P(f(21, 2ns C1, Cnj Wk), f21 Zny c1, Cni u rimm))) d.insert(k + ((f Zns cni w f(z,, on C1 Gns Wk+1)))
w. w.insert(k; w²)
Linsert((k,, f(21, En, C1, wn)w)))
Update Beta prior
Get target points: w + d/d s/d.)su(), w + accumulate(w) Curve fit: a, argmax, MLF (), w, w)
i+ =
end for return I
requires re-running the image generation process. To address these issues, we introduce Beta-based dynamic selection method to efficiently select satisfactory interpolation coefficients. This selection procedure divided into two stages as shown in Alg. [ and Alg. 2 In the first stage, we explore different coefficients based on a Beta prior and observations of perceptual distances. In the second stage, we search for a smooth image interpolation sequence from the images generated with the explored coefficients. Below, we refer to the exploration size n, and the size of the interpolation sequence as m.
During the exploration procedure, we maintain the following: 1) the currently ex- plored coefficients w; 2) the hyperparameters of the Beta distribution and 3; 3) the list of percep- tual distances d of each neighboring image pair generated by the explored coefficients; and 4) the generated images I.
Exploration.
In each iteration, we first select the neighboring image pair with the highest perceptual distance and explore new coefficient located between the selected two coefficients according to the Beta distribution. We then generate a new image with the chosen coefficient and compute its perceptual distance with both images in the selected pair. After that, update and 3 based the observations. Specifically, we aim to adjust the coefficients so that their differences are proportional the perceptual distances between neighboring images, which is our target. Using the currently explored coefficients and their corresponding target coefficients, we update a and / by fitting the cumulative distribution function. By repeating this process, obtain set of generated images to be used in the second stage.
Search. In the second stage, first compute the perceptual distance between each pair (not only neighboring pairs), represented as a weight matrix W. We reformulate the task as directed graph, where each image represents node I with E 21,2.... nn,, and edge Ei exists for any S i < j < n. Our goal is to find a path starting from I, to I, with fixed path length m, maximizing smoothness.
To solve this problem efficiently, we use a heuristic indicator-the difference between the maximum and minimum weights, i.e, the range of weight along the path-which reflects smoothness. This
18

---

Algorithm Search smoothest sequence
Input:
Image sequence I, interpolation size perceptual distance P(.,)), threshold Output: Smooth interpolation sequence
Initialize graph G = (V, F.), where V <- I and Pay + P(I, < Set binary search bounds: 0, h max(F)) min(F)
while f - >>>>660 do
D - (h + 1)/2
Initialize DP with size m)), where DP1,1 t (-00, 0o, [1])
P Each item includes (max,
min, path)
for = 1 m do
Start dynamic programming
for = 1= n do
Let maxx win 1) DP,,.
for j = to do
if Winin Pag Wmax then
Update Wmax + max Wmaxx Eag) ). Wmin <- min minii Eig)
if Wmax Wmin D and better path found then
DP, j (Wmax: Wmins I + [j])
end if
end if
end for end for end for DP,, R..m exists then h D. + mm
else 1
end end while return
indicator is bounded by the difference between the maximum and minimum weights of the entire graph, allowing us to perform binary search to find the lowest possible difference. Specifically, given a value of such difference, we search for path fulfilling the requirement with length m using dynamic programming. The algorithm is guaranteed to find the path with the minimal difference between the maximum and minimum weights. This equivalent to finding an image interpolation sequence with the minimal difference between the maximum and minimum perceptual distances among neighboring pairs.
The computation complexity of the search algorithm is O(n²m log(c)). where is the range of the perceptual distance. In practice, we choose the exploration size n 1.5m, which can already achieve very smooth results and the overhead is neglible compared to the cost of inference with diffusion model.
E Trade-off between Consistency and Fidelity via Warm-up Steps
We observe that early steps in denoising are essential to determine the spatial layout of the generated image. Thus, we can trade off between the effect of interpolation and prompt guidance by setting the number of warm-up steps. After several warm-up steps, we transform the attention interpolation into a simple generation process.
This design is based on the observation that early denoising steps of the generative model can deter- mine the image content to large extent as Fig. 12 shows. With only 5 initial steps (over total of 25 denoising steps) using "dog" as guidance (the 6th image in Fig. 12 the image content is already fixed as "dog", which means the influence of later denoising steps using "car" has very low influence to the image content generation.
Therefore, we can utilize this characteristic of the diffusion model to constrain spatial layout with AID the early stage denoising and then transit to geif-generation with the guided prompt to refine the details.
19

---

Effect of early denoising steps. The images are generated using 25 denoising steps. For the ith image shown in the row from left to right, it is generated by using "A photo of dog, best quality, extremely detailed" the first i denoising steps, then generated by using "A photo of car, best quality, extremely detailed" for the rest denoising steps.
Figure 12:
Interpolation sequence Comparison
Which image interpolation Is better? Consider the transition smooth the visual content is consistent with high quality.
Intepoitioo seguance
Ireepdatioon securece Irtepciatioe sequance
intepelation
Wirpoiatio
Figure 13: Screenshot of the survey layout. The user is prompted to choose the best interpolation sequence with high smoothness, consistency, and fidelity.
F
Auxiliary Experiments Details
Hardware Environments. All quantitative and qualitative experiments presented in this work conducted on a single H100 GPU and Float16 precision.
Perceptual Model Used in Evaluation Metrics. For consistency and smoothness, we follow con- ventional settings and choose VGG16 41 to compute LPIPS [57]. For fidelity, we adapt the Google v3 Inception Model 44 following previous literature to compute FID between source images and interpolated images.
Datasets. We introduce the details of CIFAR-10 and LAION-Aesthetics used for evaluating condi- tional interpolation here.
The CIFAR-10 dataset 22 comprises 60,000 32x32 color images distributed across 10 classes. This dataset is commonly used benchmark classification algorithms. In our context, we utilize the class names as prompts to generate images corresponding specific categories. The CIFAR-I0 corpus aids assessing the effectiveness of our framework, PAID, in handling brief prompts that describe clear-cut concepts.
CIFAR-10:
LAION-Aesthetics: We sample the LAION-Aesthetics dataset from the larger LAION- 5B collection [39] with aesthetics score over 6, curated for its high visual quality. Unlike CIFAR-10, this dataset provides extensive ground truth captions for images, encompassing lengthy and less direct descriptions. These characteristics present more complex challenges for text- based analysis. We employ the dataset to test our framework's 's interpolation capa- bilities in more demanding scenarios.
Selection Configuration. In terms of Bayesian optimization on a and in the beta prior to applying selection approach, we set the smoothness of the interpolation sequence as the objective target, [1, 15) as the range of both hyperparameters, 9 fixed exploration where a and /3 are chosen from {10, 12, 14), and 15 iterations optimize.
Denoising Interpolation. Denoising interpolation interpolates the images along the schedule. Specifically, given prompt A and prompt B and the number of denoising steps N, for inter- polation coefficient& we guide the generation with prompt A for the first [t:N] steps and guide with prompt B for the rest of steps.
Human Evaluation Details. To minimize bias towards particular style, we included an equal number of photorealistic and artistic prompts for each category. We conducted 320 trials in total. In each trial, an independent rater from Amazon Mechanical Turk evaluated the results and chose the best one among AID-I, AID-O, and text embedding interpolation (TEI). We show the layout of the human study survey in Fig. [3 For near object, the prompt sampled from: (l"a dog", "a cat"), ("a jeep",, "a sports car"], "J ["a lion", "a tiger"), ["a boy with blone hair", "a boy with black hair"]];
20

---

car
dlgg figure with car wheels insteed feet*
dog
Compositional Generation ] From text to text
X corgi"
Image Editing] From image to [text image]
statue is reading"
[ Image-Control Generation From text to [text + image]
Image Morphing ] From image to image
Our method combined with the inversion method [48] or IP-Adapter [54] can be further applied to several downstream tasks including image editing, image-control generation and image morphing.
Figure 14:
For far object, the prompt is sampled from: [["an astronaut", "a horse"], I"a girl", "a ballon"], l'a dragon", "a banana"), ["a computer", "a ship"]. ["a deer". "an airplane"]] }}: For scene, the prompt sampled from: [["sunset", "moonlit night"), momoolit night', 'forest'], ["forest", "lake"), ['lake', "sunset"]]; For scene and object, the prompt is sampled from: (["a robot", "sea of flowers"], ["aa deer", "urban street"], ['sea of flowers", "a deer"], ["urban street", "a robot"]).
G Details of Applications
In this section, we introduce to adapt our methods for four applications including composition gen- eration (interpolation between text conditions), image editing (interpolation from image condition to multi-modal condition). image morphing (interpolation between image conditions) and image- conditioned generation (interpolation from text condition to multi-modal condition), Fig. [4] shows, where the content orange box represents the input, and the content blue box repre- sents the output. We combine method with IP-AAapter [54] for the application of image-control generation and image morphing.
G.1 Composition Generation: Text to Text
interprets diffusion models as energy-based models in which the data distributions defined by the energy functions can combined, and generate compositional images by considering the generation from each condition separately and combining them at each denoising step.
CEBM [25]
RRR (7] concludes that the sampler (not the model) responsible for the failure in compositional generation and proposes new samplers, inspired by MCMC, which enables successful compositional generation, with an energy-based parameterization of diffusion models which enables Metropolis- corrected samplers.
Datasets, We use the same dataset for human evaluation introduced in Sec.
G.2 Image Editing Control: Image to (Text Image]
controls the prompt-to-promp editing, i.e.. editing on synthesized images while keeping spatial layout by borrowing the cross-attention map, i.e., query and key, from the generation of the edited image. P2P enables editing the focus content while keeping the spatial layout the same for synthesized images.
P2P [TI].
21

---

(b)
(second row) and the method by Wang et al. [9 (third row). Our approach achieves comparable superior performance to IMPUS, but with only 1/100th of the time cost.
Figure 15:
Results of image morphing. We compare our method (first row) with IMPUS 53
P2P + AID. For a given source prompt generating source images, we view another generation tra- jectory of edited images as whole and apply AID on the two-generation trajectory. Specifically, with P2P, the interpolation of AID between pross-attention only happens at the value vector while other components remain the same as the original method.
EDICT [48]. re-conceptualizes the denoising process through a series of coupled transformation layers, with each inversion process mirrored as such transformations. We denote the comprehensive process in EDICT of generating image / from latent under prompt C' as Ey C), and its inverse as (I, C). AID applied within these coupled layers during the denoising phase. We explore two applications: editing control and video frame interpolation.
EDICT AID. For given image I with source prompt C1 and target prompt Ct, we first derive its latent representation 2 = (I,,, C₁) by EDICT. To interpolate m images between C₁ and C,, replicate 21 across 21:m and employ AID for sequence generation.
Dataset. We follow the same dataset presented in [88 for quantitative evaluation. For synthesized images, each data is presented as source prompt and an editing prompt. For real images, each data is presented in a source image and editing prompt. Specifically, images of five classes "African Elephant, Ram, Egyptian Cat, Brown Bear, and Norfolk Terrier" from ImageNet [4] are taken and then we conduct four types of experiments: one involves editing "a photo of (animal I to "a photo of (animal 2)" (resulting in 20 species editing pairs in total); two involve contextual changes ("a photo of [nimall] in the snow" and "a photo of [animal) in a parking lot"); and one involves a stylistic change ("an impressionistic painting of the (animal]") )"). When this is applied to synthesized images, we use "a photo of the (animal)" as the source prompt.
G.3 Image Morphing: Image to Image
Image morphing involves smoothly transitioning from one image to another by blending and align- ing key features. Traditional approaches often require fine-tuning pretrained text-to-image diffusion models on individual samples [53] 551. which can be computationally intensive and time-consuming. In contrast, our method provides a training- free solution, allowing seamless image transitions with- out the need for fine-tuning, We compare approach with [51 [53].
AID for Image Morphing. We build on IP-Adapter (54), which adapts text-to-image diffusion mod- els to accommodate multi-modal conditions, including image and text inputs. IP-Adapter manages image prompts through cross artss-att layers, enabling our method to naturally extend to image mor- phing without additional training. For each morphing task, the provided image serves the prompt in Adaptrrerr with a null text input. Unlike approaches that rely on model fine-tuning [55] [53, our method operates entirely training-free,, making it more efficient alternative while delivering high-quality results.
Preliminary Results. As shown in the first row of Fig. 15 our method produces smooth, consis-
tent transitions for real-world images. comparison to the previous training-free approach by [49]
(third row in Fig. [5). our method achieves improved fidelity in the interpolated images. Further-
more, compared to IMPUS [53] (second row Fig. [5], which requires fine-tuning Stable Diffusion
with LoRA [1] -takinn approximately one hour per sample on a single A 100 GPU-our approach
22

---

STSI
(b) "A dog is playing with ball" Figure 16: Results of image-conditioned generation. IP-Adapter (second row) has difficulty properly scaling the influence of the additional image condition (see the last column in (a) and (b)). Our method (first row) achieves smoother control and greater subject consisteney, particularly evident in the statue's hair in (a).
(a) "A statue reading book"
training- frreing-free,, generating the entire interpolation sequence in about six minutes, achieving com- petitive performance with significantly lower computational cost.
G.4 Conei-Condi Generation: Text to (Text + Image]
longe-conditioned generation has emerged as an approach for guiding text-to-image models using supplementary control signals that are challenging to express through text alone, such as layout, subject, and style. Existing methods, including ControlNet [56) and IP-Adapter (54), often struggle to balance these additional conditions effectively, particularly when scaling their influence. Our method addresses these limitations, specifically improving upon IP-Adapter, which frequently fails to maintain a balance between the image and text inputs when both are used as conditions.
AID for Image- Conditoond Generation. Our approach utilizes the IP-Adapter integrated with the AID framework, similarly its use in image morphing. For image-conditionel generation, however, we begin with a null image prompt alongside text prompt, and at the interpolation endpoint, we incorporate both the image and text prompts.
Preliminary Results. Fig. 16 compares our method (first row) with IP-Adapter (second row). The leftmost images are generated without the additional image condition, while the rightmost images show the maximum influence of the image condition. IP-Adapter fails to maintain subject consis- tency, as shown by the statue's inconsistent hair (a), and struggles to scale the image condition effectively, especially in (b). In contrast, our method offers more consistent and smoother control over the subject, ensuring a better balance between image and text conditions.
H Auxiliary Qualitative Results
We show more qualitative results here using prompt guidance with inner attention interpolation. In this section, the results are obtained with Stable Diffusion 1.5 (35] and UniPCMullistepSched uler [58]. To enhance the visual ability, we use the negative prompt "monochrome, lowres, bad anatomy, worst quality, low quality"". To trade-off between perceptual consistency and effectiveness of prompt guidance, we use the first 10 denoising steps over 50 total denoising steps of Uni for warming up. As Fig. 17 and Fig. [8] show, our methods can generate image interpolation on differ-
ent concepts and paintings. We provide more examples in Fig. 19
And we provide more results
obtained by SDXL 30 and Animagine 3.0 (3 Fig. 20
I Distinction with Concurrent Works on Image Morphing
that also focus deep interpolation but the objective is different where their objective is on real-world image morphing where the main challenge to make the interpolation between real images is as good the interpolation between generated images. On the contrary, we focus on improving the quality of generative interpolation, which can be further used in their framework.
There are two concurrent works [5 53
specializes in generating image morphing through uniform perceptual sampling. The process begins with applying textual inversion to derive text embeddings. followed by fine-tuning the model with LoRA [14 based on specific image content. The generation of image sequences exccuted sequentially, ensuring the images are uniformly distributed.
IMPUS 53
23

---

DiffMorpher [27] initiates its process by training with LoRA, utilizing both prompts and source images. The method extends beyond simple interpolation of text embedding and latent space by also interpolating within LoRA parameters. They also explore attention interpolation, specifically within the realm of inner attention. They observe that attention interpolation across all denoising steps can introduce artifacts, which makes them adopt it with interpolating with other multiple components. On the contrary, we find that combining outer attention interpolation with self-attention fusing can significantly address this problem, and the performance is boosted without any fine-tuning, which emphasizes the difference.
IMPUS and DiffhMrrpher lack control over the specific interpolation path due to their reliance on interpolated text embeddings. Conversely, our method can plug to allow precise control over the interpolation path.
Furthermore, IMPUS and DiftMorpher necessitate fine-tuning during the testing phase achieve optimal performance to tackle the challenges from real images, requiring thousands of iterations to optimize LoRA or text embeddings for each interpolation sequence. Our method is more efficient for downstream tasks such as editing and video frame interpolation in training-free manner.
J Limitation and Social Impact
Limitation. Our method is post-hoe performed on a text-to-image diffusion model and the results are dependent on the ability of the base model.
Social Impact. Our method offers control over training-free image editing methods which initially have nearly no such ability. This is impactful to the practical usage of the text-to-image diffusion model. However, method also increases the compositional generation ability of the text-to-image model, which may make deepfake harder to detect.
24

---

Figure 17: Qualitative results of interpolation between animal concepts. For animal, we use "A photo of ponimal_namen high quality, extremely detailed" generate the corre- sponding source images. The guidance prompt is formulated as "A photo of an animal called (animal Sitimallnamm____ strong ability create compositional objects.
name - {l]aaniialll...mmm name_ B}. high quality, extremely detailed". PAID enables
Figure 18: Qualitative results of interpolation between different paintings. For painting, use "A painting of coinutingnamme high quality, extremely detailed" to generate the source images. The guided prompt is generated by GPT-4 (28 given description of source images, e.g., the guided prompt for the second row "A painting of Mona Lisa under Starry Night, high quality, extremely detailed".
25

---

Figure 19: More qualitative results generated by SD 1.5.
26

---

Figure 20: More qualitative results generated by Animagine 3.0 [23 (the 1st row) and SDXL
(from 2nd to 9th rows).
27

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency. research ethics, and societal impact. Do not re- move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] is generally preferable to "[No] it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g.. "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset general, answering "[No] or "[NA] not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just your best judgment and write a justification to elaborate. All supporting evidence can appear either the main paper the supplemental material, provided in appendix. If you answer [Yes]] to question, in the justification please point to the section(s) where related material for the question can be found.
used"). In
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the Appendix.
28

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ye]]
Justification: We provide the full experiments setting only relying open-source data and models the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primanily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide detailed experiment details in Sec. 5.1 and Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//ipss.cc/pbbic//uuieee Summ:://iddoSSSeseeeeee ississsisn ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
30

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the full settings in Sec. 5.3 and the Appendix for pyper-parammttr selection and inference details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Ys]]
Justification:
We provide the compute resource in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
31

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss the impacts in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
32

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pataussssspperswiiiii paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [Yes]
Justification: [NA]
33

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (i applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "rb2ejS4DCg", "reviewer": "Reviewer_oKyj", "summary": "In this paper the authors explore the task of interpolation between images in conditional diffusion modal. First the authors list out the three desirable properties of successful interpolation: perceptual consistency, smoothness, and image quality. The authors first introduce a method AID that incorporates three ideas: (1) interpolation should be done with both cross attention and self attention, (2) instead of a simple interpolation a fused interpolation should be performed, (3) the interpolation coefficients are sampled with a beta distribution instead of sampling uniformly.\n\nThe authors also introduce a second variation of their method called PAID (Prompt guided conditional interpolation) where the users can optionally add a prompt representing the intermediate image.", "questions": "Please see the weaknesses above.", "limitations": "The authors have included a limitations section in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and well organized. The authors do a good job first analyzing the issues with interpolating just the text prompt and then proposing a method that addresses them.\n\n- The visual results shown in the paper look impressive and the authors show that the method can also be applied to other tasks like image editing and compositional generation.\n\n- The authors use a comprehensive set of metrics that capture the 3 different aspects of good interpolation and show that the proposed method is helpful for each of the three metrics.", "weaknesses": "- One aspect of diffusion models that the authors have not considered here is the classifier free guidance and the use of negative prompts, which is standard practice for generating high quality images. Appendix H in supplement mentions that there are some results shown with negative prompt. But it would be useful to have a more detailed discussion of this in the main paper.\n- The authors show an ablation study in Table 1(b). However the discussion for these experiments is very brief. It would be useful if the authors could show some visual results corresponding to these ablation experiments and a more thorough discussion.\n\n(Minor):\n- “text-to-diffusion” → “text-to-image” L94", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer oKyj for the candid feedback and helpful suggestions. We provide our response to the reviewer's concerns in the following.\n\n> W1: One aspect of diffusion models that the authors have not considered here is the classifier free guidance and the use of negative prompts, which is standard practice for generating high quality images. Appendix H in supplement mentions that there are some results shown with negative prompt. But it would be useful to have a more detailed discussion of this in the main paper.\n\n\nThanks for the suggestion to add a discussion on classifier-free guidance and the use of negative prompts. In our quantitative experiments, we use a fixed scale 7.5 of classifier-free guidance for all generated images without negative prompt. In appendix H, we use common negative prompt \"monochrome, lowres, bad anatomy, worst quality, low quality\" to boost visual quality. Generally, the effect of classifier-free guidance scale and negative prompt for our method is similar to the base generative model. We will add the discussion to main paper.\n\n> W2: The authors show an ablation study in Table 1(b). However the discussion for these experiments is very brief. It would be useful if the authors could show some visual results corresponding to these ablation experiments and a more thorough discussion.\n\nThanks for this advice. We show the visual results of ablation study in Fig. 4 (a). As it shows, in the first row, pure attention interpolation will achieve high consistency but lack of fidelity. In the second row, fusing with self-attention can increase the fidelity. In the third row, selecting with Beta distribution can increase the smoothness further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qyXs87y15F", "reviewer": "Reviewer_oFS1", "summary": "In this work, the authors propose Attention Interpolation via Diffusion (AID), a novel, training-free technique for improving image interpolation under specific conditions like text or pose. Traditional methods using linear interpolation often produce inconsistent, low-fidelity images. AID enhances image consistency and fidelity with a fused interpolated attention layer and selects interpolation coefficients using a beta distribution for smoother results. An advanced variant, Prompt-guided Attention Interpolation via Diffusion (PAID), treats interpolation as a condition-dependent generative process. The authors include the experiments to demonstrate AID's consistency, smoothness, and efficiency in condition-based interpolation. The work also includes user study to show AID better aligns with human preferences and aids compositional generation and image editing control.", "questions": "1. For the fidelity metric in Eq. 7, one should expect the metric is maximized when the interpolation starts the same as image A, has an abrupt change from image A to B, and stays as image B for the remaining. This could be against the objective to generate perceptually consistent and smooth images. \n2. Based on proposition 1, text embedding interpolation can be viewed as manipulating the K/V which leads to suboptimal interpolation. However in (fused) inner-interpolation, still only K/V are manipulated. \n3. Is there a systematic way to determine the coefficient for Beta distribution, or is it a hyperparameter that needs tune for different models or prompts?\n4. How does applying interpolation module in self-attention or cross-attention alone affect the performance?", "limitations": "The work sufficiently addressed the limitations.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The work includes user study to compare performance of different models which is appreciated. \n2. The work is motivated to improve existing image interpolation methods based on their drawbacks.", "weaknesses": "1. The work proposes a few metrics to evaluate the quality of interpolated images. However, these metrics could be biased and fail to evaluate the quality of samples. More details are discussed in Questions. \n2. Though image interpolation can generate interesting visual effects. I feel the actual applications for such technique could be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer oFS1 for the feedback. We provide our response to the reviewer's concerns in the following.\n\n> W1 (Q1): For the fidelity metric in Eq. 7, one should expect the metric is maximized when the interpolation starts the same as image A, has an abrupt change from image A to B, and stays as image B for the remaining. This could be against the objective to generate perceptually consistent and smooth images.\n\nThanks for pointing this out. Our fidelity metric Eq. 7 is based on the Fréchet Inception Distance (FID).  As an aside, Eq. 7 is based on $n$ interpolated sequences in the dataset and not a single sequence.  Nonetheless, duplicating the $n$ source image pairs as the interpolations will lead to the max FID.\n\nHowever, such a degenerate setting is true of the FID measure in general and have been discussed in existing works [57,58]. \\emph{Any} generative model achieves the highest FID when it duplicates the reference image set.  Yet FID is widely accepted in the literature to evaluate fidelity for image generation [2,30,35] and image interpolation [38,50].  This is because existing works do not evaluate with FID alone; instead, good models must have high fidelty \\emph{and} diversity. In our case, we evaluate with fidelity, consistency, and smoothness.\n\nGiven the pervasive use of FID for evaluating fidelity, we believe our adoption of FID should not be considered a significant weakness. We are happy to add discussion in the revision about the importance of achieving strong performance on a diverse set of measures.\n\n> W2: Though image interpolation can generate interesting visual effects. I feel the actual applications for such technique could be limited.\n\nWe respectfully disagree. Sections 5.2 and 5.3 already show our method's effectiveness in two significant applications: image editing control and compositional generation.\n\nImage editing remains a prominent research topic over the past two years [11,49,59]. A main challenge is controlling the level of editing and Sec. 5.2 demonstrates that existing editing methods often fail. On the contrary, our method provides a fast and efficient solution for adjusting the editing level, which we consider a major contribution to downstream applications.\n\nCompositional generation [6,7,25] remains a challenging task for state-of-the-art generative models [2,30,35]. As shown in Section 5.3, our method significantly enhances the quality of compositional generation in a training-free manner.\n\nWe would like to emphasize that the numerous papers on these two application areas accepted at NeurIPS highlight the broad applicability of such techniques. Additionally, there are also many papers working on interpolation in generative model as well, which can benefit various applications [6,7,11,25,38,49,50,59].\n\n> Q2: Based on proposition 1, text embedding interpolation can be viewed as manipulating the K/V which leads to suboptimal interpolation. However in (fused) inner-interpolation, still only K/V are manipulated.\n\nIndeed, both forms of interpolation manipulate the K/V.  However, the manipulation targets of these two methods are different. Text embedding interpolation (Sec. 3.4) only manipulates the K/V in the cross-attention layer. (Fused) inner-interpolation manipulates the K/V of both cross- and self-attention (Sec. 4.1). Furthermore, the fused version also concatenates the original and manipulated K/V to boost fidelity. This is mentioned in Sec.4.1 (L193-L197). We will clarify in the revision to emphasize these differences.\n\n> Q3: Is there a systematic way to determine the coefficient for Beta distribution, or is it a hyperparameter that needs tune for different models or prompts?\n\nThe hyperparameter $\\alpha$ and $\\beta$ does need tuning for different prompts but can be selected automatically by Bayesian optimization as shown in the Appendix. D. In our experiments, the optimization starts from $\\alpha=T/2$ and $\\beta=T/2$ with 5 rounds where $T$ is the inference timesteps of diffusion model. In this case, the computation overhead is minimal for each sample.\n\n\n> Q4: How does applying interpolation module in self-attention or cross-attention alone affect the performance?\n\nInterpolating in cross-attention alone is equivalent to interpolating the text embedding (Sec. 3.4) and provides sub-optimal results (see Fig.2 and Tab.1 (a)).  Interpolating the self-attention alone leads to bad results similar to alpha interpolation. We will add one more section in the Appendix to discuss it in the revision.\n\nAdditional references:\n\n[57] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti,and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation,2024.\n\n[58] Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them, 2020.\n\n[59] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing, 2023.\n\nOther references are notated as the same as main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lXPHX5Fr7G", "reviewer": "Reviewer_THPg", "summary": "Summary and Contributions: The authors introduce a novel task called conditional interpolation, which is to generate interpolation images with various conditions like text or pose. They propose an attention-base (and prompt-guided) method to achieve conditional interpolation, and three evaluation metrics to assess the consistency, smoothness, and fidelity of generated images. Extensive experiments shows their method achieve the best performance in image interpolations of diffusion models.", "questions": "No coded is provided, which I think is important.", "limitations": "see weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The propose task(Conditional interpolation) is unexplored and interesting. Comom image interpolation task focus on generate transition images between two real-world images, but limit to one condition. The author introduce both text-embedding and attention mechanism to achieve better performance. \n2. The authors do a detail analysis of conditional interpolation, specifically text-embedding interpolation. They prove that text-embedding interpolation is equivalent to manipulating the keys and values in cross-attention module and find that doing similar operation in self-attention layer can significantly improve spatial consistency. \n3. The method can be used in image interpolation with distinct conditions like “a truck” and “a cat”. The authors also show their method can improve image-editing results when use editing methods like p2p.\n4. The authors introduce a third prompt to guide the interpolation between two prompts, which is interesting and useful.\n\nCorrectness: \nthe claims and method are correct.\nClarity: \nThe paper is well written\nRelation to Prior Work: \nThe paper is clearly discussed how their work differs from previous contributions.", "weaknesses": "1.The paper lack of a clear and detail definition of the conditional interpolation. The authors claims that they formulate a new task call conditional interpolation, which is doing interpolation under various condition, such as text and pose. But I am confused that the method they proposed only condition on text, how could it be various condition? What the definition about various condition? \n2.The comparations between baselines are inconsistent. The table 1 shows the result of TEI and DI, but the table 2 only show TEI.\n3.The qualitatively compare between baselines（TEI and DI）is lack, while the quantitative result of this two baseline are exist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer THPg for the constructive feedback. Here is the response to reviewer's concerns.\n\n> W1:The paper lack of a clear and detail definition of the conditional interpolation. The authors claims that they formulate a new task call conditional interpolation, which is doing interpolation under various condition, such as text and pose. But I am confused that the method they proposed only condition on text, how could it be various condition? What the definition about various condition?\n\nThanks for raising this point. We state in the title and introduction that this work targets text-to-image diffusion models where text serves as condition.  We will remove mentions of ``various'' conditions.  Note we give a clear definition of the task formulation and the evaluation metrics in Sec. 3.2. We are happy to further highlight these as definitions in the revision.\n\n> W2:The comparisons between baselines are inconsistent. Table 1 shows the result of TEI and DI, but table 2 only shows TEI. 3.The qualitative comparison between baselines (TEI and DI) is lacking, while the quantitative results of these two baselines exist.\n\nThanks for pointing this out. We are happy to add some results to the revision as reference for our own work.  \n\nNote we omitted DI for the qualitative results and human study because it is not used by competing methods; there is no comparison basis for Table 2 and the qualitative results.\n\n> Q1:No coded is provided, which I think is important.\n\nOur intention was to release the source code upon acceptance. We provide an anonymous link to our code in a seperate comment to AC following the rebuttal rule of NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mAjTBu1Pbq", "reviewer": "Reviewer_frah", "summary": "This paper proposes a training-free method for generation interpolation of diffusion models with attention manipulation. Targeting on the layout transition inconsistency and nonuniform step-wise transition, the paper proposes to extend the attention interpolation from previous cross-attention to self-attention, and adopts a beta distribution to ease the nonuniform transition. Additionally, three metrics are proposed to evaluate the interpolation qualities, involves consistency, smoothness and fidelity. Experiments show the effectiveness.", "questions": "1. What's the behavior difference between the inner-interpolation and outer-interpolation, is there any guide lines when to use the inner or outer?", "limitations": "The authors have adequately discuss the limitations and broader impacts of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The beta distribution for interpolation weights instead of linear weights seems promising.\n2. The proposed metrics seem sound to evaluate the interpolation quality.\n3. Experiments show the effectiveness of the proposed attention interpolation.", "weaknesses": "1. The application of the generation interpolation seems somewhat restricted, and the practical meaningness of the task is doubted, is the interpolation could benefit other generative applications, or isolated. \n2. Basically, the manipulation of the attention map has been broadly explored for image editing, such as [Masactrl; ICCV 2023], [InfEdit; CVPR 2024], etc., and the self-attention corresponding to the layout and the cross-attention corresponding to the semantic are basically common sense, the proposed AID adopts a similar fashion with reformulated interpolation task, where the contribution is somewhat insignificant.\n3. The discussion with training-based interpolation methods is suggested to be provided, such as [UniHDA; 2024].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer frah for the feedback. Here, we aim to clarify our contributions and the application of the proposed methods.\n\n> W1: The application of the generation interpolation seems somewhat restricted, and the practical meaningness of the task is doubted, is the interpolation could benefit other generative applications, or isolated.\n\nWe respectfully disagree with the assertion of limited applications. On the contrary, we believe the applications we mentioned are of significant importance. As discussed in Sections 5.2 and 5.3, the proposed method enhances performance in two critical areas: image editing control and compositional generation. These applications, and the value of exploring interpolation, have been recognized for their value in numerous existing works accepted in the top conference [6,7,11,25,38,49,50].\n\nTo the best of our knowledge, our method is the first to efficiently adjust the editing level, particularly in a tuning-free manner. Traditional editing methods are incapable of this, making our approach more practical for real-world applications, as demonstrated in Section 5.2.\n\nFurthermore, our method can be independently used for compositional generation in a tuning-free manner, outperforming previous state-of-the-art methods as shown in Section 5.3. It is noteworthy that even state-of-the-art models [2,30,35] struggle with generating two-concept images. Our method plays a crucial role in addressing this challenge and improving performance.\n\n> W2: Basically, the manipulation of the attention map has been broadly explored for image editing, such as [Masactrl; ICCV 2023], [InfEdit; CVPR 2024], etc., and the self-attention corresponding to the layout and the cross-attention corresponding to the semantic are basically common sense, the proposed AID adopts a similar fashion with reformulated interpolation task, where the contribution is somewhat insignificant.\n\nWe respectfully disagree with the assertion of insignificant distribution. \n\nAs acknowledged in Section 2, attention manipulation is extensively explored [1,3,11,31,34,51], particularly from a downstream application perspective such as image editing and compositional generation. The trade-off between edits and retaining parts of the original image is crucial for practical applications in image editing, as highlighted in [2,16]. This issue is also a focal point in compositional generation research [6,7,25].\n\nHowever, existing works recognize the trade-off issue as an open challenge without effectively addressing it. When attempting such trade-offs, they often rely on varying scales of classifier-free guidance and simple text embedding interpolation [16,49,57], which we demonstrate to be very ineffective and lacking generalization ability in Sections 3.4 and 5. [Masactrl; ICCV 2023] and [InfEdit; CVPR 2024] also lack this trade-off capability, and we will include additional results of more attention-based methods even for training-based methods in our revision. Given the extensive use of attention mechanisms, our work provides practical insights on controlling the level of attention manipulation, assisting existing works and addressing a critical gap in current research.\n\n> W3: The discussion with training-based interpolation methods is suggested to be provided, such as [UniHDA; 2024].\n\nOur method diverse from training-based interpolation methods [58,59] in mainly two aspects. The main advantage of training-based interpolation methods is that these kind of work often focus on cross-model / cross-domain area, enabling interpolation across domain. However, they often requires sample-wise fine-tuning to adapt the source generator, serving as the main challenge to real world application. On the contrary, our method only focus interpolating within text modality, but it is tuning-free and more efficient while keeping the competitive performance.\n\nWe kindly thanks for the advice, and will add more discussion with [59] together in the revision.\n\n> Q1: What's the behavior difference between the inner-interpolation and outer-interpolation, is there any guidelines when to use the inner or outer?\n\nThanks for raising this point.  Generally, we recommend to use outer interpolation (AID-O) when the interpolation is more related to spatial layouts, and (inner interpolation) when interpolating between distinct semantic concepts.  A discussion is provided in Appendix C and Fig. 10.\n\nAdditional References:\n\n[57] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity-preserving generation in seconds, 2024.\n\n[58] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators, 2021.\n\n[58] Hengjia Li, Yang Liu, Yuqi Lin, Zhanwei Zhang, Yibo Zhao, weihang Pan, Tu Zheng, Zheng Yang, Yuchun Jiang, Boxi Wu, and Deng Cai. Unihda: A unified and versatile framework for multi-modal hybrid domain adaptation, 2024.\n\nOther references are notated as the same as the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received somewhat mixed reviews.  The authors did not submit a rebuttal PDF but did provide rebuttal comments and participated in discussion with the reviewers.  The main concerns raised were around motivation and novelty.    The AC reviewed the paper in detail along with the reviews and discussion.  The AC disagrees with some of the reviewer and finds the proposed problem formulation to be interesting and potentially relevant given current interest in controllable image generation.  Further, as recognized by oKyj, the proposed method produces compelling results and is reasonably well evaluated.  As a result, the AC recommends acceptance.
Decision: Accept (poster)