IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models
Authors: Zhaoyuan Yang
Zhengyang Yu
Zhiwei Xu
Jaskirat Singh
Jing Zhang
Dylan Campbell
Peter Tu
Richard Hartley
Venue: ICLR 2024 poster (2024)
Keywords: image morphing
diffusion models
clip inversion
Primary area: visualization or interpretation of learned representations

Abstract: We present a diffusion-based image morphing approach with perceptually-uniform sampling (IMPUS) that produces smooth, direct and realistic interpolations given an image pair. The embeddings of two images may lie on distinct conditioned distributions of a latent diffusion model, especially when they have significant semantic difference. To bridge this gap, we interpolate in the locally linear and continuous text embedding space and Gaussian latent space. We first optimize the endpoint text embeddings and then map the images to the latent space using a probability flow ODE. Unlike existing work that takes an indirect morphing path, we show that the model adaptation yields a direct path and suppresses ghosting artifacts in the interpolated images. To achieve this, we propose a heuristic bottleneck constraint based on a novel relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness. We also propose a perceptually-uniform sampling technique that enables visually smooth changes between the interpolated images. Extensive experiments validate that our IMPUS can achieve smooth, direct, and realistic image morphing and is adaptable to several other generative tasks.

Published as a conference paper at ICLR 2024
IMPUS: IMAGE MORPHING WITH PERCEPTUALLY- UNIFORM SAMPLING USING DIFFUSION MODELS
Zhaoyuan Yang" Zhengyang ; Zhiwei Xu², Jaskirat Singh", Jing Zhang", Dylan Campbell Peter Tu', Richard Hartley²
GE Research" Australian National University?
{zhaoyuan yang, tue com {zhengyang. yu, zhiwei xu, jaskirat singh, jing. zhang, dylan. campbell, richard. hww.cchar....rarlyly.ly......e Ganu. edu au
ABSTRACT
We present a diffusion-based image morphing approach with perceptually- uniform sampling (IMPUS) that produces smooth, direct and realistic interpo- lations given an image pair. The embeddings of two images may lie distinct conditioned distributions of a latent diffusion model, especially when they have significant semantic difference. To bridge this gap, we interpolate in the locally linear and continuous text embedding space and Gaussian latent space. We first optimize the endpoint text embeddings and then map the images the latent space using a probability flow ODE. Unlike existing work that takes an indirect morph- ing path, we show that the model adaptation yields direct path and suppresses ghosting artifacts in the interpolated images. To achieve this, we propose heuris- tic bottleneck constraint based on a novel relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness. We also propose perceptually-unifool technique that enables visually smooth changes between the interpolated images. Extensive experiments validate that our IMPUS can achieve smooth, direct, and realistic image morphing and is adaptable to several other generative tasks.
sampling
INTRODUCTION
In various image generation tasks, diffusion probabilistic models (DPMs) (Ho et al. 2020 Song et al. 2221a b) have achieved state-of-the-art performance with respect to the diversity and fidelity of the generated images. With certain conditioning mechanisms (Dhariwal & Nichol 2021 Ho & Salimans 2022: Rombach et al. (002). DPMs smoothly integrate guidance signals from Yang et al. 2023) to perform conditional generation.
difterent modalitiss (Zhang |
Among different controllable signals, text guidance (Rombach al., 2022, Saharia al. 2022
Nichol et al.) 2022) is widely used due to its intuitive and flexible user control.
Different from other generative models, such generative adversarial networks (GANs) GGoodfel- low et al., 2014), DPMs have some useful properties such as invertibilityy. Firstly, ODE samplers (Lu al. 2022a Liu al. 20222 can approximate bijective mapping between given image Xg and
its Gaussian latent state XT by connecting them with a probability flow ODE (Song et al. 2021b). Secondly, the conditioning signal can be inverted with textual inversion (Gal
2023) which can
be used to obtain semantic embedding of an image. These properties make DPMs useful for image
manipulation, including the tasks of image editing (Meng al.) 2022 Brooks
2023 Hertz
et al. 2023), image-to-image translation (Tumanyan et al. [023 Parmar et al. 2023 and image
blending (Lunarring 2022 Avrahami et al. 2023 Pérez et 2023).
In this paper, we consider the task of image morphing, generating sequence of smooth, direct, and realistic interpolated images given two endpoint images. Morphing (Dowd) 2022] Nri [022] Zope& & Zope 2017), also known image metamorphosis, changes object attributes, such as shape or texture, through seamless transitions from one frame to another. Image morphing is usually
"Equal contribution, order determined via coin flip and may he listed in either order. Code is available at: https: https:///gituuu...m/////222222:

---

Published as a conference paper at ICLR 2024
Figure 1: Image Morphing with IMPUS. IMPUS achieves smooth, real, and direct interpolation between two images with perceptually-unifor transition. More results in Sec. 55 and Appendix.
achieved by using image warping and color interpolation (Wolberg 1998: Lee et al. 1996). where
the former applies geometric transformation with feature alignment constraint, and the latter
blends image colors. This has been well-studied in computer graphics (Lee et al. 1996 Seitz & 2014 Rajković 2023 Zope & Zope 2017 Shechtman et al. 2010) but less so in computer vision where the starting point is natural images.
Dyer, 1996 Zhu al., 2007 Wolberg 1998: Liao al.,
Inspired by the local linearity the CLIP (Radford al., 202I) I text embedding space (Liang al.
2022 Kawar et al., 2023) and the probability flow ODE formulation of DPMs (Song et al. (2221a) we propose the first image morphing approach with
sampling using diffusion
models (IMPUS). A similar setting explored by [Wang& & Golland (2023), which shows strong dependency on pose guidance and fine-grained text prompts, while our approach achieves better performance on smoothness, directness, and realism, without additional explicit knowledge, such as human pose information, being required. Some examples are visualized in Figure [
We summarize main contributions as: (1) propose robust open world automatic image morphing framework with diffusion models; (2) we discover that low-rank adaptation contributes significantly morphing task; (3) present an algorithm for perceptually-unifef encourages the interpolated images to have visually smooth changes; (4) we propose three evaluation criteria, namely smoothness, realism, and directness, with associated metrics to measure the quality of the morphing sequence; and (5) we explore potential applications of morphing in broader domains such as video interpolation, model evaluation as well as data augmentation.
sampling that
RELATED WORK
Text-to-image Generation.
The mainstream of current text-to-image generation models (Nichol al. 2022 Rombach al. 2022 (ahariia et al.) 2022) based diffusion models (Ho al., Song et al. 202 a b). especially the conditional latent diffusion models (Rombach et al, Given the flexibility of textual prompting (Radford et al. 2021 text-to-image generation
2020
2022
well studied to generate both high-quality and open world images (Ramesh et al. 2022). Text
inversion Gal al. (2023) enables capturing specific concepts present in real-world images.
Image Editing. Among image editing models (Meng et al. 2022 Kawar et al. 2023 Brooks et al.,
2023 Hertz al., 2023). text-guided image editing is a burgeoning field due the advances in
language models (Brown et al., 2020 Chowdhery et al. 2023
Touvron et al. (023). Editing for real
images enabled by latent state inversion (Song al. 2021a Mokady et al. 2023). Unlike image editing that requires specific guidance local global factor of variation. Image morphing has
requirements for guidance on particular subject or attribute variations between two different images.
Image-to- tnageeto-immag Translation. Image-to-image translation (Tumanyan et al. 2023 Parmar et al.) 2023 Su et al. 2023: Kim et al., 2023 Cheng et al. 2023) is usually done to transfer an image from source domain translation, which seeks to find a mapping between two domains, image morphing aims to achieve seamless and photo-realistic transitions between them.
target domain while preserving the content information. Unlike image-to maggeto--mmage
2

---

With the Published as a conference paper at ICLR 2024
Image Morphing, Human involvement is usually required for conventional image morphing (Lee et al. 1996, Seitz & Dyer 1996 Zhu et al. 2007 Wolberg 1998; Liao et al., 2014, Dowd 2022).
(2022) uses neural networks to learn the morph map between two images, which however fails to obtain a smooth morphing of images that have a large appearance difference. Alternatively, Miller & (200)); (ankoovie(2023)) [hechhmaan et al. 2010 | et al. (2020) achieve image morphing by minimizing the path energy on a Riemannian manifold to obtain geodesic paths between the initial and target images, and perceptual well as similarity constraints across the neighboring frames are enforced to achieve seamless image transitions. Most existing morphing works either require human intervention or are restricted to a narrow domain (see Appendix (e.andixBBBiiee for details). In this work, we present novel diffusion morphing framework that achieves image morphing with minimum human intervention and is capable of applying arbitrary real-world images.
BACKGROUND
3.1 PRELIMINARIES OF DIFFUSION MODELS
Denoising Diffusion Implicit Models (DDIMs). With some predefined forward process {4: (4hhelo 2e that gradually adds noises to image Xo. diffusion model approximates corresponding reverse process {px (et)xe(o.7] to recover the image from the noises. In this work, we use a DDIM (Song et al,
202 see Appendix A for more details. Given q (x; xo) A (xx; Bi,x,, (1 3,) I) and parameterized noise estimator co, we have the following update rule in the reverse diffusion process
X = VBBB1
X
V1
3,5g (xx)
B o, (x,)
+
TCEL
(1)
v
BB
"random noise"
"direction pointing
"predicted Xo"
where or is free variable that controls the stochasticity in the reverse process. By setting o, to O, we obtain the DDIM update rule, which can interpreted as the discretization of continuous- time probability flow ODE (Song et al. [2021a] Lu et al.) (022a).. Once the model is trained, this ODE update rule can be reversed to give a deterministic mapping between Xo and latent state (Dhariwal & Nichol 2021), given by
X
-V1- 3et1
3g
Eg
(xx).
8441
3g
A DDIM be trained using the same denoising objective DDPM (Ho al. 2020). sampled noise and a condition vector e, it is given by
min LDPM (x e; 0) = min Ex (x) (-
[Ike (x,, ()
(3)
8
Classifier-free Guidance (CFG). When a conditional and an unconditional diffusion model are jointly trained, inference time, samples can be obtained using CFG (Ho Salimans 2022). The predicted noises from the conditional and unconditional estimates are defined as
èg (Xe, f, e) == weg (x,, t, e) + (1 w) (x,, t, Ø)
(4)
where w the guidance scale that controls the trade-off between mode coverage as well as sample fidelity and Ø is null token used for unconditional prediction.
3.2 DESIDERATA FOR IMAGE MORPIINNG
The image morphing problem can be formulated as constrained Wasserstein barycenter problem (Simon Aberdam 2020 Agueh Carlier 201 Chewi et al., 2200)), where two images are interpreted two distributions ( and g(t) respectively. The optimal solution this problem an o-parameterized family of distributions given by
g(a) arg min(1 - a)M3 (500, + amz (eW)v)
(5)
3

---

Published as a conference paper at ICLR 2024
[0, 1] is the interpolation parameter, M is the target data manifold and v is a probability measure on M. The parameterized distribution clo) connects g(0) and g(1) (Villani, 2021), which leads to smooth and direct transition process. The formulation is informative but it is challenging to generalize such formulation to real world high dimensional images; thus, we explore an alternative, tractable method, without losing track of the desirable characteristics from such formulation. Based the objective of image morphing in existing works, we propose three important properties for high-quality image morphing.
where W2(+,)) :) is the 2- Wasserstein distance (Villani 2016),
Smoothness. Transition between two consecutive images from the generated morphing sequence should be seamless. Different from traditional image morphing methods that consider a physical transition process (Arad et al.. [994] Lee et al., [995) to achieve seamless transitions, we aim to estimate a perceptually smooth path with a less stringent constraint.
Realism.
The interpolated images should be visually realistic and distributed on a target manifold for image generation models. In other words, the images should be located in high-density regions according to the diffusion model.
Directness.
Similar to optimal transport that finds the most efficient transition between two prob- ability distributions, the morphing transition should be direct and have as little variation along the path as possible under the constraints of smoothness and realism. A desirable and direct morphing process should well fit human intuition for a minimal image morph.
4 IMAGE MORPHING WITH PERCEPTUALLY- SAMPLING
Given two image latent vectors X and x corresponding to an image pair, we aim to generate the interpolated sequence (x(=)) with a E [0, 1] such that it can well satisfy the three criteria Sec. 3.2 In the following, by e E R", and a diffusion timestep by E Z. We define a diffusion model parameterized by as Ep IR" x Z m - RR, where the forward and reverse processes are defined in Sec. [.1 In Sec. 4.1 we propose a semantically meaningful construction between the two images in the given (0) (1) image pair by interpolating their textual inversions (e(") e and latent states {xT XT J. Then, in Sec. 4.2 we show that model adaptation with novel heuristic bottleneck constraint can control the trade-off between the directness of the morphing path and the quality of interpolated images. In we introduce a perceptually-unifon sampling technique.
(0)
denote image latent encoding by X E R", a conditional text embedding
Sec. 4.3
4.1 INTERPOLATION IN OPTIMIZED TEXT-EMBEDDING AND LATENT- STATE SPACE
Interpolating Text Embeddings. Inverting the concepts in each image into optimized text em- beddings with a pre-trained DPM facilitates modeling conditioned distributions that are centered around the input images. Inspired by recent works (Liang et al., 2022 Kawar al., 2023) showing that the CLIP feature space is densely concentrated and locally linear, we choose to linearly interpo- late between optimized CLIP text embeddings e0)) and e!)) of the image pair. To address potential
nonlinearity & discontinuity between semantically distinct samples, given X and x( we initial- ize the embedding e by encoding a simple common prompt, set as <An image of [token]>, where [token] is either the root class of the pair, e.g., "bird" for "flamingo" and "ostrich", the concate- nation of the pair, e.g., "beetle car" for "beetle" and "car". Since two embeddings are initialized by common prompt, they stay closed after optimization, which is important for our locally linear assumption. The optimized text embeddings can be obtained by optimizing the DPM objective in Eq.
(0)
(0) = arg min min
LDPM 0) and (1)) = arg min (XP (1) .)
(6)
The optimized text embeddings ( and ( can well represent the concepts in the image pair. Particularly, the conditional distributions Po Px (0)) and po (x(e(")) closely reflect the degree of im- age variation of the image pair, where the images conditional distributions. Due to the local lincarity and continuity of the CLIP embedding space, ()) and po (x (1) can be smoothly bridged by linearly interpolating between the optimized embeddings, i.e., Po (x (()) := p>(x (1 - ()) + a)()), providing an alternative to the in- tractable problem of Eq. ( See Appendix E.T for analysis effects of common prompt.
highly possible be under their associated
4

---

Published as a conference paper at ICLR 2024
Interpolating Latent States. With the optimized text embeddings, we compute po(x(e(a)) to smoothly connect the two conditional distributions associated with the image pair. We then present latent state interpolation between the two image (0) and (1) We find that vanilla interpolation between X and x leads to severe artifacts in the interpolated images. Hence, we interpolate in the latent distribution M(xT: 0, I) by first deterministically mapping the images the distribution via the probability flow ODE in Sec. 3.1 As shown in Khrulkov et al. (2023), the diffusion pro- cess between Xo and XT can be seen state similarity can also reflect certain pixel-level similarity (see Appendix [E.2 for details), making it essential correctly interpolate the latent states, Following previous work (Wang et al. 2023a),
optimal transport mapping, which suggests that latent
we smoothly interpolate between (0) and Xg via spherical linear interpolation (slerp) to their la- tent states x/" and x/ to obtain interpolated latent state (a) sim(l-a)w xf
sin
sinw x1
sin w
where = arcoos X XTr /(Ix>) |||x{/"))) ID)) The denoised result Xo is then retrieved via the probability flow ODE by using the interpolated conditional distribution pe (x(el=))
(0)T (1)
(a)
4.2 MODEL ADAPTATION WITH HEURISTIC BOTTLENECK CONSTRAINT
Interpolating in the optimized text embedding and latent state space using the forward ODE in Sec. S.c. facilitates faithful reconstruction of images in the given image pair and a baseline morph- ing result in Wang & Golland (2023). However, this approach leads to severe image artifacts when the given images are semantically different, shown in Sec. [.1 We also observe that this baseline approach often fails to generate direct transitions by transiting via non-intuitive interpolated states.
To obtain direct and intuitive interpolations between the images in the given image pair, we in- troduce model adaptations on the image pair. This can limit the degree of image variation in the morphing process by suppressing high-density regions that are not related the given images. A standard approach is to fine-tune the model parameters using the DPM objective in Eq. (3) by ming (D e (0)) detail loss, as shown in Sec. S.c.5.2) indicating that the conditional density of the source model is prone to collapse to the image modes without being aware of the variations of their image regions. To alleviate this issue, we introduce model adaptation with a heuristic bottleneck constraint. We use the low-rank adaptation (LoRA) (Hu et al.] (2022) approach, which helps to retain mode coverage and reduces the risk of catastrophic forgetting (Wang et al., (023b) due to its low-rank bottleneck. The optimization problem is given by
(x))( e());)) 0). However, this vanilla fine-tuning leads to object
min (x)) (0) ()) + Al.) + CDMM (x, (0) ee:0 0 + Ale), s.t. rank(A0.) = re 28.
where LoRA rank is set be much smaller than the dimension of model weights. We find that the ideal rank w.r.t. quality of the morphs is based on a trade-off between directness and fidelity. Directness is affected by the sample diversity between endpoint images (e-g., low sample diversity occurs in image pairs that are intra-class or conceptually connected via a shared concept; large sample diversity occurs image pairs that are inter-class or conceptually related via a root class such as "mammal" for "lion" and "camel"). Fidelity is affected by the perceptual gap between endpoints, i.e., a larger perceptual gap between the endpoints can lead to higher risk quality loss after model adaptation with a larger rank due to mode collapse. Typically, larger LoRA rank can enhance the directness of morphs by supressing sample diversity, but may lead to low- low-fidelity results (e.g., omor-smooth morphs), which underscores the importance of finding a proper trade-off.
Relative Perceptual Path Diversity. Based on these qualitative observations, we define relative perceptual path diversity (rPPD) to evaluate the diversity along the morphing path, relative to the diversity of the endpoint images (see Appendix E.. for rationale). To achieve this, we use our baseline morphing result from Sec. [4. and consecutive pairs, relative to the perceptual difference between the endpoints, given by
and compute the average perceptual difference between the
Σ N- LPIPS (x (xt)), ((c)),((a++))) x(ar1))
(8) where LPIPS(:..) measures the perceptual difference between two images Zhang et al. 2018) and a; = i/(N 1) for [0, N 1] and N samples along the path. Empirialll, we set the LoRA rank to re [_ax((0[187__]]) where [:] returns the rounddown integer.
(N -N-1) LPIPS (x("), x(1)
5

---

Published as a conference paper at ICLR 2024
Unconditional Bias Correction. At inference time, achieve high text alignment, widely adopted strategy is to set a CFG scale w > 1 as in Eq. (4). While the aforementioned model adapta- tion strategy uses conditioned images and text embeddings, the unconditional distribution modeled by the same network is also changed. Due to parameter sharing, however, it causes an undesired bias. To address this, we consider strategies: 1) randomly discard conditioning (Ho & Salimans) 2022) 2) separate LoRA parameters for fine-tuning the unconditional branch x( and x("), We observe that the latter can provide more robust bias correction. Hence, we use additional LoRA parameters A0, with a small rank ro. i.e.,
min LDMM (x0 20:00 + A0)) LDPM (X 00 + A0h), s.t. ran)(A0)) = ro Alg
(9)
For inference, with 0g 0 A0e and Ou A + Alp. parameterize the noise prediction Ea (Xe, e, = WEB. (x,, 1, e) + (1 w)Ee (xx, ())
(10)
4.3 PERCEPTUALLY UNIFORM SAMPLING
In order produce a smooth morphing sequence, it desirable for consecutive samples to have sim- ilar perceptual differences. However, uniform sampling along the interpolated paths is not guaran- teed to produce constant rate of perceptual changes. We introduce a series of adaptive interpolation parameters }. The core idea is to use binary search to approximate a sequence of interpolation parameters with constant LPIPS differences. Details are provided in Appendix D.]
EXPERIMENTS
Evaluation Metries. We validate the effectiveness of our proposed method with respect to the three desired image morphing properties defined Sec. 3.2 1) Directness. For a morph se- quence with N consecutive interpolation parameters a₁ [0, 1]. we report the total LPIPS, given by Ee LPIPS(a(0.)) 2((0=1)), and the maximum LPIPS to the nearest endpoint, given by max, (IIP(_.IPS(2((0)) (0))), LPIPS(x((0.) )), as indicators of directness. 2) Realism. We report FID (Heusel et al. 2017) to measure the realism of generated images. However, note that this is an imperfect metric because the number of morph samples is limited and the samples are not i..d., leading to bias. 3) Smoothness. We report the perceptual path length (PPL) (Karras al., 2019) under uniform sampling, defined as PPL: E0-)(( LPIPS(x (a)), x(a+)))] where is a small constant. For our perceptually-uninn
sampling strategy, the total LPIPS measure is appro- priate for evaluating smoothness (see Appendix [.I for rationale). For a fair comparison with Wang & Golland (2023), we also use their evaluation setting: 1) the total LPIPS from uniformly-spacee pairs (PPL11/1)) and 2) the FID between the training and the interpolated images.
Datasets. The data used comes from three main sources: 1) benchmark datasets for image genera- tion, including Faces (50 pairs of random images for each subset of images from CelebA-HQ Kar- ras et al.] al,] (008)), Animals (50 pairs of random images for each subset of images from AFHQ (Choi al. 2020), including dog, cat, dog-cat, and wild), and Outdoors (50 pairs of church images from
L.SUN Yu et al., 2015)). 2) internet images, e.g., the flower and beetle car examples, and 3) 25 image pairs from Wang & Golland (2023).
Hyperparameters for Experiments. We set the LoRA rank for unconditional score estimates 2, and the default LoRA rank for conditional score estimates set to be heuristic (auto). The condi-
Table 1: Comparison with baseline.
PPL
FID 4
tional parts and unconditional parts are finetuned for 150 steps and 15 steps respectively. The fine- tune learning rate is set to 10 Hyperparameters for text inversion as well as guidance scales vary based on the dataset. See Appendix D for more details.
Wang & Golland 2023
143.25
198.16
IMPUS (ours)
92.58
148.45
5.1 COMPARISON WITH EXISTING TECHNIQUES
For a fair comparison with Wang & Golland (003). we use the same evaluation metrics and the same set of images and show the performance in Table 1 Our approach has clear advantage w.r.t.

---

Published as a conference paper at ICLR 2024
(a) Wang & Golland (2023
(b) IMPUS (ours)
Figure 2: Compared with Wang & Golland (2023), our method achieves better sample consistency and quality. See Appendix for more trajectories to see comparison in smoothness.
Table 2: Quantitative results on benchmark. The lower the better for all metrics. For endpoint reconstruction errors, we report the mean and maximum (in parentheses) of LPIPS at the endpoints.
Text Inv. Auto Rank Rank 4 Rank 32
LPIPS
LPIPS
FID
PPL
Endpoint Error
1.13
0.37
48.89
248.06
0.092 (0.308)
1.16
0.39
48.20
238.87
0.081 (0.137)
female malee
1.14
0.38
46.84
322.31
0.079 (0.132)
1.09
0.37
46,12
367.70
0.080 (0.169)
1.61
0.50
46.28
451.85
0.125 (0.206)
1.75
0.52
45.82
485 .48
0.1 (6.167)
cat+>dog
1.71
0.52
48.11
613.77
0.119 (0.245)
1.60
0.49
43,21
804.36
0.1 10.190)
1,46
0.39
35.71
533.70
0.113 (0.412)
1.79
0.42
36.52
1691.48
0.093 (0.248)
churche>chured
1.85
0.42
36.02
1577.45
0,088 (0.289)
1.66
0.41
34.15
2127.56
0,090 (0.266)
both metrics. A qualitative comparison is shown in Figure 2 where our method has better image consistency and quality with severe artifacts as in Wang & Golland (2023) given two drastically different images (see first row of images). See Appendix B for further comparison.
5.2 ADDITIONAL RESULTS AND ABLATION STUDIES
We perform a detailed analysis of the proposed approach with respect to different configurations and hyperparam- eters. Figure 3 shows the effectiveness of our method on Cel AFHQ, and LSUN-Church Total LPIPS (LPIPS,), Max LPIPS (LPIPSAM). Ms), and FID calculated based on the sampling with a constant LPIPS interval of 0.2. PPL is with 50 intervals uniformly sampled in [0,
(a) CelebA-HO meme
Ablation Study on Textual Inversion. Given the impor- tance of the text prompt for accurate conditional genera- tion, we have claimed that suitable text embedding is criti- cal for image morphing, which is optimized via text inver- sion. Alternatively, the optimized could be replaced by a fine-grained human-der -emaande text prompt.
(b) AFHQ cate
(c) LSUN churche+church
For the benchmark datasets, we select the subset of femalex-mmal, cate>dog, and churche<church. We ablate textual inversion by using prompt that describes the cate- gories and show results in Table 2 For femaler-mmal, we use the prompt "a photo of woman man". For cat+>dog,
Figure 3: Benchmark examples.
the prompt "a photo of dog or cat". In Table 2 without textual inversion, morphing sequences have a shorter and more direct path (smaller LPIPS and LPIPS but have larger reconstruction error at the endpoints. A poor reconstruction caused by the lack of textual inversion may lead to inferior morphing quality, see robust.
Appendix E.5 for details. Thus, text inversion with coarse initial prompt

---

Published as a conference paper at ICLR 2024
Table Ablation study on model adaptation with text inversion. All metrics are the lower the better.
Adaptation
L.PPSS
M
FID
PPL
Endpoint Error
1.45 0.05
0.41
0.01
52.76 = 4.28
196.30
41.78
0.192 _ 0.083
CelebA-HQ
1.07 + 0.07
0.37 + 0.01
43.59 + 5.16
201.87
26.34
0.083 = 0.001
2.27 0.20
0.55 1 0.03
42.67
1 12.39
414.10
50.76
0.254 1 0.007
AFHQ
1.90 + 0.18
0.54 + 0.03
36.79 = 11.87
679.96
159.04
0.130 + 0.007
1.94
0.40
42.48
663.33
0.184
LSUN
1.79
0.42
36.52
1691.48
0.093
(a) Without model adaptation
(b) Vanilla model adaptation
(c) LoRA with rank = 2
(d) LoRA with rank = 4
(e) LoRA with rank =
(I) LoRA with rank = 32
(g) LoRA with rank 2
(h) LoRA with rank =8
LoRA with rank 32
(i) LoRA with rank =
(k) LoRA with rank
LoRA with rank = 32
(m) Finetune conditioning
Randomly discard
(0) Separate LoRA
(p) Uniform sampling
Figure 4: Ablation studies under LoRA rank, unconditional noise estimates, and sampling methods.
LoRA rank: (a)-(f) intra-class on flowers with 7 0.44, dELP 0.15, (g)-(i) inter-class on beetle and car with = 0.39, deLı = 0.37 (i)-(l) inter-class on lion and camel with y = 0.49, deL 0.29. Unconditional noise estimates: (m) only finetune on conditioning, (n) randomly discard conditioning during finetune, (o) separate LoRA for unconditional estimates. Sampling methods: (o)-(p) perceptually uniform sampling versus uniform space sampling.
Ablation Study Model Adaptation. We further ablate model adaption and show results CelebA-HQ, AFHQ, and LSUN in Table 3 with heuristic LoRA rank for the bottleneck con- straint and textual inversion. Since AFHQ and Celeba-HQ have different settings mael dog+>dog, etc.), we report the mean and standard deviation of all settings, see Appendix[F]] for details. Overall, model adaptation leads to more direct morphing results (smaller LPIPS and LPIPS M) and realistic images (lower FID) and lower reconstruction error at endpoints. However, the higher PPL values reflect that uniform sampling may not work well with model adaptation.
We show performance w.r.t. LoRA ranks qualitatively in Figure 4 where the selected ranks by our heuristic bottleneck constraint are in bold. From the results, the ideal rank differs for different image pairs, and our proposed rPPD can select the rank with qualified trade-off between image quality and directness (we also show deLP embedding space for comparison). Quantitative analysis of LoRA rank on subsets of the benchmark datasets (femalet-mmale cat+>dog, church+<church) reported in Table 2 Three settings of LoRA rank are considered which are: 1) heuristic rank with bottleneck constraint, 2) rank 4, and 3) rank 32. We observe that higher LoRA ranks have better guantitative results on the benchmark dataset which different from the conclusion We conjecture that the scope and diversity of benchmark datasets are more limited than the internet images; thus, phenomenon such overly blurred interpolated images occurs less likely, resulting
cosine similarity as cosine distance in CLIP image
have from the internet-colleted images (see Figure 4
8

---

Published as a conference paper at ICLR 2024
in better performance on high LoRA ranks. Another observation is that reconstruction enror the endpoints does not decrease as the LoRA rank increases. We conjecture it is caused by the numerical error during the ODE inverse. We leave the study of these phenomena our future work.
Ablation Studies on Unconditional Noise Estimates and Perceptually Uniform Sampling. We demonstrate the effects of different unconditional score estimate settings and sampling model adaptation only applies to conditional estimates, the inferior update of the unconditional estimates (due to sharing of parameters) results in a color bias, see Figure mm). Hence, we consider both randomly discard condition-
methods in Figure mm)-(p).
ing (Ho & Salimans (022) and apply separate LoRA for unconditional score estimates.
In our experiments, applying a separate LoRA for unconditional score estimates leads to slightly higher image quality and is more robust to various guidance scales than randomly discard condition- ing, see Figure 4 n))(o) and Ap- pendix [E.6] for more details. We show example of interpolation result from uniform sampling in Figure 4p). Compared with Fig-
Usn
(a) GradCAM (morphing sequenee) (b) Logits (x: frame index)
Figure 5: Model explainability with IMPUS (ResNet50).
ure o), transitions between images in Figure 4pp) are less seamless, valid our sampling technique.
5.3 APPLICATION OF IMPUS TO OTHER TASKS
Novel Sample Generation and Model Evaluation. Given images from two different categories, our morphing method is able to generate a sequence of meaningful and novel interpolated images. Figure a) shows a morphing sequence from lion to an eagle. The interpolated images are creative and meaningful, which shows great potential in data augmentation. Our morphing technique can also be used for model evaluation and explainability. For example, the 4th image of the morphing sequence looks like eagle, but the highest logit (5igure(5]bb) is prairie fowl. With GradCAM (Sel-) varaju et al. 2017), the background seems distinguish the prairie fowl from the eagle.
Video Interpolation. Our morphing tools can also be directly applied for video interpolation. In Figure 6 given start and end frames, we can generate realistic interpolated frames. We claim that more interpolated images can further boost our performance for video interpolation.
Figure 6: Video interpolation of Jurassic World BBayona 2018) and EpicSlowMotionDDog (003
CONCLUSION
In this work, we introduce a diffusion- based image morphing approach with perceptually- sampling, namely IMPUS, to generate smooth, realistic, and direct interpolated images given an image pair. We show that our model adaptation with heuristic bottleneck constraint can suppress inferior transition variations and increase the image diversity with a specific directness. Our ex- periments validate that our approach can generate high-quality morphs and shows its potential for data augmentation, model explainability and video interpolation. Although IMPUS achieves high- quality morphing results, abrupt transitions occur when the given images have significant semantic difference. In some contexts, our approach generates incorrect hands as well (see Appendix E.8). Hence, more investigations are desirable in our future work.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research supported by the DARPA Geometries of Learning (GoL) program under the agree- ment No. HR001 12290075 and the DARPA Environment-driven Conceptual Learning (ECOLE) program under agreement No. HR001 12390061. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. government.
REFERENCES
Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM J. Math. Anal., 43(2):904-922 924, 2011.
Nur Arad, Nira Dyn, Daniel Reisfeld, and Yehezkel Yeshurun. Image warping by radial basis functions: Application to facial expressions. CVGIP Graph. Model. Image Process., 56(2):161- 172, 1994.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM Trans. Graph., 42(4):149:1114:111 2023.
Bayona. Jurassic world:
Fallen kingdom. Film, 2018. URLttts::/ imdb com/title/ Et4881806/ Directed by J.A. Bayona. Produced by Frank Marshall and Patrick Crowley. Universal Pictures.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pp. 18392-18402. IEEE, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Bin Cheng, Zuhao Liu, Yunbo Peng, and Yue Lin. General image-to-image translation with one-shot image guidance. arXiv preprint arXiv:2307.14333 2023.
Sinho Chewi, Tyler Maunu, Philippe Rigollet, and Austin Stromme, Gradient descent algorithms for wasse-wwsn barycenters. In Conference on Learning Theory, pp. 1276-1304. PMLR,
2020.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, pp. 3185-8194. Computer Vision Foundation IEEE, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240:11 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs image synthesis. In NeurIPS, pp. 8780-8794, 2021.
David Dowd. Python image morpher (pim). http://gjihlbcccomddddd//////yym 2022.
10

---

Published as a conference paper at ICLR 2024
EpicSlowMMotinDDog. Dog running in epic slow motion 2 [hd]. Online video, 2013. URL https: //youtu.be/_caok////
Noa Fish, Richard Zhang, Lilach Perry, Daniel Cohen-Or, Eli Shechtman, and Connelly Barnes. Image morphing with perceptual constraints and stn alignment. In Computer Graphics Forum, pp. 303-313, Wiley Online Library, 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR. OpenReview.net, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, pp. 2672-2680, 2014.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross-attention control. In ICLR. OpenReview.net, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, pp- 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, abw/2207.12598, 2022. doi: 22. 12598
htt.ps://dbi..or/11....
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. ICLR. OpenRe- view.net, 2022.
Ferenc Huszár. Gaussian distributions soap bubbles. hatpr/iiinaaauuusiiiiiiiiirrrrrr
http::/wwww.infeenne...whhhhh
2017.
András Jankovics. Automatic image morphing. http://giiii.cccmaaaliirsssssaaa 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR. OpenReview.net, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, pp.
Computer Vision Foundation / IEEE, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NeurlPS, 2022.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, pp. 6007- 6017. IEEE, 2023.
Valentin Khrulkov, Gleb V. Ryzhakov, Andrei Chertkov, and Ivan V. Oscledets. Understanding DDPM latent codes through optimal transport. In ICLR. OpenReview.net, 2023.
Beomsu Kim, Gihyun Kwon, Kwanyoung Kim, and Jong Chul Ye. Unpaired image-to-image trans- lation via neural schrödinger bridge. arXiv preprint arXiv:2305. 0.005/5000 15086, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In ICLR. OpenReview.net, 2023.
11

---

Published as a conference paper at ICLR 2024
Seungyong Lee, Kyungyong Chwa, and Sungyong Shin. Image metamorphosis using snakes and free-form deformations. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pp. 439-448, 1995.
Seungyong Lee, George Wolberg, Kyung Yong Chwa, and Sung Yong Shin. Image metamorphosis with scattered feature constraints. IEEE Trans. Vis. Comput. Craph., 2(4):337-334, 1996.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022.
Jing Liao, Rodolfo S. Lima, Diego Nehab, Hugues Hoppe, Pedro V. Sander, and Jinhui Yu. Au- tomating image morphing using structural similarity on a halfway domain. ACM Trans. Graph., 3365):168:1-168:22 12, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR. OpenReview.net, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). Open- Review.net, 2019.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095. 2022b.
Lunarring. Latent blending. http://ggtho...oom/luaarrr//eeet
2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Er- mon. Sdedit: Guided image synthesis and editing with stochastic differential equations. ICLR. OpenReview.net, 2022.
Michael I. Miller and Laurent Younes. Group actions, homcomorphisms, and matching: A general framework. International Journal of Computer Vision, 41:61-84, 2001.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, pp. 6038-6047. IEEE, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML., volume 162 of Proceedings of Machine Learming Research, pp. 16784-16804, PMLR, 2022.
Chigozie Nri. Differentiable morphing. https://github.com//// https///ttbb.oomm//ooo////ttttfp 2022.
Sanghun Park, Kwanggyoon Seo, and Junyong Noh. Neural crossbreed: neural based image meta- morphosis. ACM Trans. Graph., 39(6):22::11222 1-24:11,, 2020.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In SIGGRAPH (Conference Paper Track), pp. 11:1-11:11. ACM, 2023.
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 577-582, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Marko Rajković. Geodesics, Splines, and Embeddings in Spaces of Images. PhD thesis, Universitäts- und Landesbibliothek Bonn, 2023.
12

---

Published as a conference paper at ICLR 2024
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2)3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10674-10685. IEEE, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
Steven M Seitz and Charles R Dyer. View morphing. In Proceedings of the 23rd anmual conference on Computer graphics and interactive techniques, pp. 21-30, 1996.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, pp. 618-626. IEEE Computer Society, 2017.
Eli Shechtman, Alex Rav-Acha, Michal Irani, and Steve Seitz. Regenerative morphing. In CVPR, pp. 615-622, IEEE Computer Society, 2010.
Dror Simon and Aviad Aberdam. Barycenters of natural images constrained wasserstein barycenters for image morphing. In CVPR, pp. 7907-7916. Computer Vision Foundation / IEEE, 2020.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR. OpenReview.net, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR. OpenReview.net, 2021b.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR. OpenReview.net, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.13977 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven mmage-to-image translation. CVPR, pp. 1921-1930. IEEE, 2023.
C. Villani. Optimal Transport: Old and New. Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, 2016. ISBN 9783662501801. WtLphtts:://boo google. com. st/boooks?i==spSSDDA.
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
Clinton J. Wang and Polina Golland. Interpolating between images with diffusion models. arXiv
preprint 3rXiv:2307.125660 2023.
Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. Infodiffusion: Representation learning using information maximizing dif- fusion models. In ICML, volume 202 of Proceedings of Machine Learning Reseurch, pp. 36336- 36354. PMLR, 2023a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint arXiv:2305. 16213, 2023b.
George Wolberg. Image morphing: survey. The visual computer, 14(8-9):360-2 1998.
Yue Yang, Kaipeng Zhang, Yuying Ge, Wenqi Shao, Zeyue Xue, Yu Qiao, and Ping Luo. Align, adapt and inject: Sound-guided unified image generation. arXiv preprint arXiv:2306. 1/.3006//50 //544 2023.
13

---

Published as a conference paper at ICLR 2024
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 100.:36506.03355, 2015.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprini arXiv:2302.05543 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, pp. 586-595. Computer Vision Foundation IEEE Computer Society, 2018.
Lei Zhu, Yan Yang, Steven Haker, and Allen Tannenbaum. An image morphing technique based on optimal mass preserving mapping. IEEE Trans. Image Process., 16(6))1481-1)) 2007.
Bhushan Zope and Soniya Zope. survey of morphing techniques. International Journal of Advanced Engineering, Management and Science, 3(2):239773, 2017.
APPENDIX
A
PRELIMINARY ON DIFFUSION MODELS
Denoising Diffusion Implicit Models (DDIM). In this paper, DDIM (Song et al. 202 la
For a sample XG, the reverse step is defined as follows with a variable at
v1
Beco
(+)
(xx)
Xt-Ai 33-20
V
B of (xi)
+
B
"random noise"
"direction pointing to x.,"
"predicted xo"
(11)
By setting the free variable oe in Eq.
as O, we have a deterministic generative pass as follows
X X
3
X
(v
:1-s:
Be-
1
3r
B
Eg
(t)
(x:)
(12)
which can be seen as the discretization continuous- ODE (Song et al.
2021a)
lim A0
X1
P
Xt
B,-Ar
lim At-0
1 Bt
1- 3t
BI-DC
31-Ar
co
(xx)
(13)
dx(()= = -60
(t)
x(t)
do((),
(1)
where x(f) = Xy
and o(t)
1-8 D.
vBr
B FURTHER DISCUSSION ON COMPARISON WITH EXISTING METHODS
B.1 COMPARISON WITH WANG & GOLLAND 2023
We find the following major differences of Wang & Golland 20233 with our work: 1) task defini- tion Wang Golland (2023) focus on interpolation and novel image generation but not directness, which is an important factor for image morphing. An indirect morph could generate sequence of intermediate images irrelevant to the end point images (see Fig. g). which also restricts their down- stream applications. Directness of our method can lead to many downstream benefits, e.g., for video interpolation, it facilitates temporal coherency and consistency; for interpolation of an image and it's edited version, it align the image transition with the target editing direction allowing intuitive user control over the magnitude of change; for data augmentation, limits the image variations to be related the endpoint images, making the interpolated images be in-distribution; 2)
14

---

Published as a conference paper at ICLR 2024
(a) Interpolation result Wang & Golland (2023) without (left) and with (right) pose guidance
(b) Our image morphing results with generic text prompts (left) and model adaptation (right)
Figure 7: Compared with Wang & Golland 2023), our approach does not require extra guidance signals to achieve consistency in pose variation nor fine- -in-inrieed text prompts. Together with the pro- posed model adaptation strategy, our approach demonstrates superior image morphing performance.
the first to propose that model adaptation is the key component for morphing, Despite its simplicity, this technique has been shown in our work to provide extraordinary improvements both sample fidelity and directness. Meanwhile, unlike Wang & Golland (2023), the superior performance does not depend on careful selection for prompts (c.g., for boundary samples in Fig. 7 Wang & Golland 2023) uses two detailed prompts: <HDR photo of a person practicing martial arts, crisp, smooth, photorealistic, high-quulity wallpaper, ultra HD, detailed> and < <HDR photo of a person leaning over, crisp, smooth, high-quulity wallpaper>, while ish character>>), ), hyperparameters or random seeds. As shown in Fig. 7 it also enables our method to morph inputs with pose inconsistency, while Wang & Golland (2023) relies on an auxiliary pose estimation model to handle this problem: 3) The flexibility and robust performance shows the poten- tial of our method be adopted a wide range of applications. e.g., Sec. 5.3 of the main context we show that our work can be adopted for model evaluation and video interpolation. We provide further examples in Fig. 12 showing that our work can be seamlessly incorporated with an existing tatte-of-the-ar art image editing & translation pipeline (Brooks et al.) (023) that enables users to con- trol the magnitude of change. [Wang& & Golland (2023) does not have such flexibility; 4) We propose perceptually uniform sampling while Whnge & Golland (2023) uses standard uniform sampling. This has a significant impact on the quality of the results;
coarse prompt <a photo of a cartoon-
Figure 8: Indirect Interpolation from Wang & Golland 2023 generates irrelevant intermediate images which restrict application of downstream tasks.
We note that, in the original paper of Wang & Golland (202)), 26 pairs of images are claimed to be used, but there is access to only 25 pairs provided in their public Repository |https:// github. com/ Wwliintoowwan/CContte
We present additional comparison with Wang
& Golland (2023)
Fig. 9
B.2 COMPARISON WITH OTHER MORPHING METHODS
We compare our morphing method with following open source automatic image morphing methods: DiffMorph Nri
2022), Automatic Image Morphing (Jankovics 2020) (adapted from Python Image Morpher (Dowd) 2022 as well as Neural Crossbreed (Park et al. 2020) and show the examples in Fig. 10 Automatic Image Morphing is non-learning based, DiffMorph optimization-base while Neural Crossbreed is GAN-based. Since Neural Crossbreed requires image pairs relevant to the GAN training data, we only compare with them on the image pair provided in their paper. According to Fig. [0 a)-(c), morphing sequences generated by our method are smoother and more realistic compared with DiffMorph and Automatic Image Morphing. In Fig. d0) we show a failure case from Neural Crossbreed where the GAN model fails due to lack of training data. In summary, existing automatic morphing tools either create blurred and unrealistic intermediate images or cannot generalize to unseen data.
15

---

Published as a conference paper at ICLR 2024
(a) Wang & Golland (2023)
(b) Ours
Figure 9: Additional comparison results with Wang & Golland 20033). We rerun the code from
Wang & Golland (2023) to generate 5-image sequences for comparison.
ExiiMerpa
Gans
Cars
(a) Morphing examples from DittMorph
(b) Morphing examples from Auto Image Morph
UniiStepa
Nearal Enverraal (PAN Tuna))
Auzo Moeph: OpenCV + tythee Image Magler
(c) Morphing examples from paper
Morphing examples from Crossbreed
Figure 10: Comparison between different open source implementations for automatic morphing. The image pair in (a) collected from DiffMorph. The image pair in (b) collected from Automatic Image Morphing. The image pair in (c) collected from Wang & Golland (2023). The image pair in (d) collected from Park et al. (2020). Overall, existing automatic morphing tools either create blurred and unrealistic intermediate images or cannot generalize to unseen data.
C EFFECT OF PROPOSED COMPONENTS
In Fig. 11 we provide the results of progressively adding proposed components in work to the baseline pre-trained Stable Diffusion model, which uses user-provided prompts and discards the textual inversion and model adaptation: 1) Baseline, 2) Textual Inversion, 3) Textual Inversion Vanilla Adaptation, 4) Textual Inversion + LoRA Adaptation (rank=4), 5) Textual Inversion LoRA Adaptation (heuristic rank), 6) Textual Inversion LoRA Adaptation (heuristic rank) Uncondi-
16

---

Published as a conference paper at ICLR 2024
tional Bias Correction. We that the baseline model fails to reconstruct of the endpoint images and produces low fidelity samples. Textual inversion ensures a faithful reconstruction of the input endpoints. Model adaptation with LoRA can improve the perceptual directness of morphing path without suffering from the severe artifacts of vanilla model adaptation. An adaptively selected rank using the proposed rPPD metric shows improved directness than commonly used rank, avoid- ing unnatural factor variations, c.g., using rank 4, the brightness of the left man's ' shirt first increases then decreases, while when heuristic rank is used, the brightness varies naturally. Results with the proposed unconditioned bias correction strategy shows finer details (such as hands and clothes).
Baseline
Textual
Textual Inversion Vanilla Adaptation
Textual Inversion LoRA Adaptation (rank
Inputs
Textual Inversion LoRA Adaptation (heuristic rank)
Textual Inversion + LoRA Adaptation (heuristic rank)
+ Unconditioned Bias Correction
Figure 11: Results of progressively adding proposed components. The baseline is a pre-trained Stable-Diffusion which uses user-provided prompts and discards the textual inversion and modal adaptation process.
D FURTHER IMPLEMENTATION DETAILS
For all the experiments, we use use a latent space diffusion model (Rombach et al. 2022). with pre-trained weights from timizer (Loshchilov & Hutter 2019), and the learming rate is set as 0.002 for 2500 steps. For the benchmark dataset, we perform text inversion for 1000 steps. LoRA is trained with Adam opti- mizer (Kingma & Ba (0155) and the learning rate is set as 0.001. The hyperparameters of LPIPS is set to default settings.
Textual inversion is trained with AdamW op-
D.1 MODEL. ADAPTATION AND PERCEPTUALLY UNIFORM SAMPLING
The overall process of our proposed framework is presented in Algorithm Jand Algorithm respec- tively.
E JUSTIFICATION OF DESIGNS FOR EACH COMPONENT
In this section, we provide detailed justifications for designating the proposed components and strategies of our image morphing method, including prompting & interpolation strategies, endpoint image selection details as well as model sampling & estimation strategies.
E. COMMON PROMPT NITIAAIZATTINN
Our text embedding interpolation strategy is based upon the local linearity of CLIP embedding space (Liang et al., 2022). To address the potential nonlinearity between the optimized embeddings of semantically distinct image pairs, we a common prompt <An image of [token]> the semantic gap between the input endpoint images, where for semantically distinct pairs, [token] either a common root class (e.g., "animal") or concatenation of the pair (e.g., "poker man"). Since two prompt embeddings are initialized with the same common prompt, they stay closed after optimization. We find that this simple method can avoid the non- non-linear and discontinuous issue in
matter
Source from http://ngggniaaacooooomm////mssm
17

---

Published as a conference paper at ICLR 2024
Algorithm Finetuning & inference process of IMPUS
Require:
The pre-trained diffusion model to with text encoder fø. Learning rates 7,, 122- Input images Xb and x A common prompt for the endpoint images. Starting ending interpo- lation scales astorts Qendd Perceptual difference between consecutive sampling images ólpips- Error tolerance for binary search Ctal. Number of steps for text inversion Tine- Number of step for model adaptation Tadpt- Number of step for DDIM T. Textual Inversion (Sec.4.1)
1: // Phase
2: Initialize: e(0) + faly). e(1) 4 fo(y)
3: for E {1,... Time) do
4: Randomly sample time step & and random noise 6, NO
5: Adding noise to data x!0) + v V (0) + v - -1-1,1ee, + vBxx0 + V(I 3,)e,
(0)) + ( - mn e CDPM (x) ,e(0),
6:
e(1) ( V. e CDPM (1) e())))
7:
8: // Phase 2: Model Adaptation (Sec.4.2)
9: Calculating according to Eq. 8.
D. Heuristic rank selection
10: Setting LoRA rank +- 1887-6))
for E (1, ]
12:
Low-rank model adaptation according to Eq. 7 and Eq. 9 with learning rate 1/2- sampling
13: // Phase 3: perceptually-unie
Obtaining initial latent states x and xf" via Eq.
15: Cutist + Binarysearch(Astory aend, Ólpips: Ctol) for a in Quist. do
> Algorithm.2 b Sampling process
17: ela) + - a)) + ae()), (a) + sinfl-a)w
(11
sinw X7 + sin oud XT
> Interpolation
for from 1' to 1 do
X1 X--vI-bec (xx (xx, ( + /1 3 (0 (x),
(a)
x
(a) return {x0 facau
Algorithm Perceptually- Uniform Sampling with constant LPIPS
Input: asstort: starting alpha value; aendi ending alpha value; 8tpips perceptual difference between consecutive sampling images, txnl: error tolerance for binary search
Output: Cuiss: list of alpha.
procedure BINARY SEARHHYEARHHaarrr.. Geedd dipiper Etol)
2: Olfist & [astare], aeurr + Astart
> Initialization
3: while D(aeury Gend) > 8 do
Binary search of constant L.PIPS
4:
< Clearr 0:g2 &- Obends Osmid f- aeitas2
5: while LLIPS(xX(ee)) (((eu)) (iceerr)x(mmmn))) x((mid)) )- 5) cxal do
LPIPS(X (aewr)) x (amid)) > â then
6:
7:
0122 f- 401a
8:
else
9:
a/ f- 4a 2
10:
amid 4- 2:12
Ocear <- amidds Olisst +- Atiist U Qeur
D Append alpha the list
12:
Olistt &- Oliist @een
Append the ending alpha
return Olist
18

---

Published as a conference paper at ICLR 2024
Editing with Instruct-pix2pix: "make the kid happy"
Magnitude of attribute editing controlled by IMPUS
Editing with instruct-pix2pee "put her bar"
Magnitude of environment transfer controlled by IMPUS
Editing with Instruct-pix2pix: "make it snowing"
Magnitude of style transfer controlled IMPUS
Figure 12: Given a prompt-based image editing pipeline (Brooks et al. 2023 IMPUS generate a gradual transition between source and edited images. The transitions generated by IMPUS is well- aligned with the target editing direction of the prompt, which can achieve intuitive control over magnitude of change.
CLIP embedding space to some extent, where the usage of common prompt is crucial step which illustrated in Fig. 13
photo of lion" photo a camel"
photo animal"
"a photo poker" photo a man"
photo a poker man"
Figure 13: Given pair of inter-class images, setting separate prompts lead to severe artifacts (top left) or drastic transitions (bottom left), suggesting a non-Euclidean property in CLIP embedding space. While our common prompt strategy (right) leads to natural transition between endpoints.
E.2 INTERPOLATION IN LATENT STATE
The finding in Khrulkov al. (023) shows that zo and FT forms an Monge optimal transport in the 1.2 sense, this suggests that the similarity in latent state will also cause certain pixel-wise similarities between the generated image Zo. As shown Figure of Khrulkov et al. (2023), such similarity in latent state indeed reflect the structural information in generated images. We provide similar results in Fig. 14 where we fix the latent state Zr for generations using a Stable Diffusion model with different text prompts, from where the structure of generated images, which coincide with the finding of Khrulkov et al.) (2023). This finding indicates that latent state Xy reflect rich structural information about Fo. which makes
significant alignment between
19

---

Published as a conference paper at ICLR 2024
it essential to correctly interpolate FT in order to smoothly morph two images. We spherical linear interpolation (slerp) instead linear interpolation (lerp) to ensures that the interpolation can preserve the Euclidean norm of the interpolated latent state Xr (Huszár 2017). Empirically, find slerp works well while lerp often produces unnatural intermediate samples since the Fuclidean norm of interpolated latent states not preserved in lerp. See Fig. [3 for examples.
(8) "photo of cat'
(b) 'photo of dog'
(c) 'photo of tiger'
(d) 'photo of elephant'
Figure 14: Using the same latent state xy, the generated images Stable Diffusion using different prompts shows strong alignment in structural information.
(a) Result lerp latent state X
(b) Result of slerp latent state XT
Figure 15: Linear interpolation (lerp) in latent state Xy produces unnatural intermediate samples.
Samples with spherical linear interpolation (slerp) are natural-looking and consistent.
E.3 SELECTION STRATEGY FOR ENDPOINT IMAGES
Image pairs selected in our works are collected from three sources: (1) Example image pairs pro- vided in Wang & Golland ) (2) Additional images we collected from Internet; (3) Benchmark data for computer vision research. In terms of benchmark evaluation, based on a simple protocol, i.e., for a given benchmark dataset, we randomly select image pairs whose similarity is under a certain LPIPS threshold (0.7 for cat and dog: 0.7 for cat+>dog; 0.7 for wild; 0.6 for female and male; 0.6 for femalex->mle; 0.55 for church). The intuition behind this process is that, the endpoint images should have certain perceptual connection for the interpolated images between them to be meaningful. In Fig. [6 we provide counter example, showing that morphs between a pair of perceptually unrelated images can be meaningless.
select the image pairs
Das Dane
Gal Oraa
Doves
Figure 16: Morphing result between samples without perceptual connection may be meaningless.
E.4 QUALITY BOOSTING
Convex CFG Scheduling. Although the interpolation between initial states and inverted textual embeddings can facilitate smooth connection between inputs, however, observe that the inter- mediate samples in the morphing path often show lower saturation than the ones near the bound- aries. Intuitively, after model adaptation, the overall probability densities (x(x;, t) would con- centrate more on the boundary samples, leading to lower density for regions in between. In or- der to provide extra guidance cues guide the generation of intermediate samples via enforcing
20

---

Published as a conference paper at ICLR 2024
larger density ratio po(x,, t|c)/ (x t), propose convex CFG scheduling in the form of Wmin ) 0.5], which can boost the generation quality for intermediate
Wa = Wmar (Wmaa samples as shown in Fig.
Stochasticity Boosting. For semantically distinct input pairs, artifacts may appear in details the intermediate morphing results, where possible cause is ODE sampler with limited time steps. As shown in prior works (Karras et al., 2022; Kwon et al. (023), stochastic diffusion samplers often show improved quality than deterministic ones. Taking the reverse DDIM process in Eq. (11) as an example, a stochastic sampler sets a positive or, which requires extra "denoising" by the diffusion model for newly introduced Gaussian noises on top of "denoising" for initial noise, which may also correct the prediction errors from earlier denoising steps. However, setting a positive o throughout the denoising process may fail to provide a faithful preservation of input content. To find balance, using a total of 16-time steps, we find that setting as 0 for the initial 6 steps and setting for the rest 10 steps can faithfully retain the content from inputs while correcting some artifacts as shown in Fig. 17
(a) Morphing result of our work fixed CFG.
(b) Result w.o. stochasticity boosting.
(c) Results of our w.t. quality boosting.
Figure 17: Illustration on the effect of quality boosting.
Ablation Study CFG. We also study the effects of different hyperparameters, such the scale of classifier-free guidance. We observe that inversion-only morphing requires a stronger CFG scale, while inversion with adaptation requires less guidance. In most situations, a large CFG scale can degrade the morphing quality and lead to a longer morphing sequence, We provide ablation studies of CFG total LPIPS (Table. 4), Max LPIPS (Table. 5). FID (Table. (6) and PPL (Table. 7
E.5 FAILURE CASE WITHOUT TEXTUAL INVERSION
In our work, textual inversion is performed on the embedding of'a coarsely initialized prompt. Alter- natively, the optimized embedding could be replaced by a fine-grained human-designed text prompt. We find that using appropriate prompt shared by two images can improve the smoothness of the morphing path without loss of image quality (see the first example in Fig. [8]. However, for an ar- bitrary pair of images, it is usually difficult to choose such ideal prompt (see the second example Fig. [8 for failure cases where a human-labeled prompt not informative enough to reconstruct given images, resulting in inferior performance). Instead, our textual inversion with coarse initial prompt more flexible and robust.
E.6 FURTHER DETAILS OF SEPARATE LORA vs. RANDOMLY DISCARD CONDITIONING
During experiment, observe that using separate LoRA for unconditional score estimation per- forms more stable (compared with randomly discard conditioning during finetune) for image in- version. For example, on CelebA (female-male pairs), maximum end point reconstructed error in
21

---

Published as a conference paper at ICLR 2024
Input
photo of logo' w/'t
'simple elegant loga, vector graphics, 2D, symmetrical, centered, high quality' w/o textual inversion
Input
"a photo of
boy'
w/' textual inversion
"rick and morty, high quality, large eyes, expressive faces, wallpaper, artstation, highly detailed' w/o
Figure 18: Results w.o. textual inversion using detailed prompt. In the first example where an appropriate description is selected to be shared by two images, the morphing path shows improved directness w.o. textual inversion. In the example below, an inappropriate prompt leads an unfaith- ful reconstruction of inputs and downgraded quality.
terms of LPIPS is 0.137 for separate LoRA unconditional score estimates, the value for randomly discard conditioning during finetune is 0.211. A poor reconstruction may lead inferior quality; thus, we select separate LoRA for unconditional score estimation. We show two examples of recon- structed images Fig. 19 To enhance the difference, we blend real as well as reconstructed images (0.5 x real 0.5 x recon.) in Fig. 1)).. Other approaches could improve reconstruction quality as well, but applying separate LoRA for unconditional score estimates simple yet effective enough for our task.
E.7 INTUITION OF RELATIVE PERCEPTUAL PATH DIVERSITY
Fidelity and directness are two important criteria for our morphing task, but there is a trade-off between them: a blurred image sequence (generated from large rank) is likely be more direct than a image sequence with high fidelity. If pair of images has large perceptual difference, large rank adaptation could lead to blurred intermediate samples due overfitting. The design of rPPD score is based on the trade-off between fidelity (higher quality morphs) and directness (smaller total perceptually change). Directness: The accumulated LPIPS, which measures the directness of preliminary morph, is proportional to the LoRA rank for adaptation. Fidelity and Diversity: The perceptual difference of the endpoint images, related to quality (blurriness, diversity, fidelity) of inversely proportional to the LoRA
generated intermediate images after large rank adaptation, rank for adaptation. Thus, we use the ratio to control the trade-off between the two, and a larger rPPD score indicates larger LoRA A rank can be applied make the morphs more direct.
22

---

Published as a conference paper at ICLR 2024
Real imput
Sep. LoRA
Rand. discard
Sep. LoRA
Rand. discurd
(a) Real and reconstructed images
Blending real and reconstructed images (weight:
Figure 19: Comparison of reconstruction quality between (1) separate LoRA for unconditional score estimates and (2) randomly discard conditioning during finetune. The "ghost" effect from blending images indicated that randomly discard conditioning during finetune is less robust (compared with separate LoRA for unconditional score estimates) in terms of reconstruction.
E.8
FAILURE CASES
Although IMPUS achieves high-quality morphing results, abrupt transitions occur when the given images have significant semantic difference, Figure 20 (a). In cases like Figure 20 (b), our approach generates incorrect hands, which is common for diffusion models.
Figure 20: Failure cases of IMPUS
F DETAILED QUANTITATIVE RESULTS
We provide detailed tables in this section. Different scales of guidance are considered: (1) CFGI: 1.n: max:2, (2) CFG2: min:2 max:3, (3) CFG3: min:3 max:4, and (4) CFG4: min:4 max:5. Inv. represents text inversion and Adapt. represents model adaptation. The numbers reported in the main paper are the best numbers across all settings.
Table 4: Mean and standard deviation of total LPIPS.
Dulaset
Im: Only CTGI
Iac Only
CEC2
Im: Only CTGS
Inc. Only CTGI
Im:++Adhgt CEGI
Imce/Adggt CTG2 Adapi, Only CTGI Adap, Only CTGZ
male male
1.28
0.33
[.30
17.06
1.1
0.33
1.01
=
11:00
1.19
L.2 0.20
fomale e* famole
1.45 0.53
1.63
0.43
1.43 0-26
1.54 - 0.36
1.03 +.0.6
1.16 0.211
female male
1.50 0.36
1 0.38
1400 0.227
2.11= 0.30
1.66 1.16-
131 +01.8
1.13+
1,27 + 0.18
2.25 0.62
2.70
2.79 1.811
3.39
187
218
0.60
wild
258
2.33)
1.57
1.83 13++032
diunch
+.4.4
268
0.334
Table Mean and standard deviation of max LPIPS.
Damner
Only CFFT
Iav: Only
Daly
Im:4AAiap.
4Adape
Aciipt Only CRG Adiet Only
male
02012000 0106
0.43 2 0006
0.41 0.0
044 ILRS
(136 0.IS
(13
female
0.41 0.06
0:43 0.004
0.40 2: 0.05
0:43 +: 0.05
0.37 0.04
0.38 4: 0.04
female male
0.42 2: 0.04
0.45 1 0.00
0.42 1: 0.04
0.45 1 0.04
(.39 0.04
0.10 0.03
0.37-1000
0.39 0.04
0.57 0.06
0.6 + 0.06
0.59 + 0.07
061 + 0,07
(155 1.07
0.56 + 0.07
wild
23

---

Published as a conference paper at ICLR 2024
Table 6: Mean and standard deviation of FID.
Dames
Only
lay. Unly CFG2
Ualyy LFGS
Inx Dai CRN
Em-rAAcapt CFGZ
Adapt. Only Adiet Only
male
52,58 = 0.32
10.06 0.39
63:26 1 0126
72.49 4. 0.42
66.19 1: 0.25
49.33
0.27
famale
47.17
0.33
46.66
+ 0.31
$3.99 0.36
36.39 + 0.22
57.88
0.23
fomale male
53.55 0.29
61.39
59.41 0:033
07.82
2.32.....3. 0.38
48.30 + 0.25
48.00 0.25
48.89 0.25
50.2
16.53 = 0.26
0-229
40.5 +-330
45 0.34
32.08 + 0.21
35.00 +:00:00000 0.22
dog
61.45
12.59 0.38
15.17.0044
61.73 0.39
1: 0.25
50.15 0.28
wild
30.35
23.28
27:31 0.25
0.17
20.11 0.18
0:36
51,50 1:0.3
56
49.11
0.26
church
44.61
38.37
Table 7: Mean and standard deviation of PPL.
Danase
En:
Ims CFG2
In: Unly C3KG
In CRS4
Inv-AAdape CRGI 187.
Anc:-Acapt CTGZ
Adape CRGT
male
187.0
466
1M159
712.75
S8.94 1816.34
812.7 2705.22
425.31
323
fenule
4
275.70
258.72 628.28
505.10
1381.98
591.45 2 2551.81
179.65 375.86
305.33 094.54
female male
251.19
480.31
421.73 = 1238.64
901.66 = 4177.55
950.52 3007.18
238,87
557.75
394.47 = 916.87
248.06 E 766.20
381.27 1616.40
cat
432.71 + 83
676.33
2733.90
1451.31
9076.00
1977 007.90- 5408.94
710.23
1600.5
1 1196.00
2336
dog
475.35 817.85
1053
2423 10
2003.53
4091.1
2534 77 6712.08
918.50
2346.47
1674.60
4529.26
wild
41265
713.6
747.22
1327.47
1383.24
1071.43
1995
3781 30
005 09
1459-92
108% 10 2510.31
cardogg
315
63
99 1200
1131778
1332.01
1213
2399.28
485.08
903.8
uw.. 96 20W1.3
451.85
916.1
364.5 2018
charch
1236.84
4631.25
10460.97
36039.89
1691.48
380400
F.1 RATIONALE OF TOTAL LPIPS AS INDICATOR OF SMOOTHNESS FOR PERCEPTUALLY UNIFORM SAMPLING
We provide some rationale of how total LPIPS related to the (1) rate of change between consecutive images and (2) perceptual path length (PPL) here. Given two images (0) and x11), a diffusion model x (e2) = F(a,, (0) ())) generates an interpolated sample (between (0) and x()) based on the interpolation parameter og. For specified small delta LPIPS value ALPIPS (which is almost imperceptible by human), perceptually uniform sampling ensures that LPIPS(x x ((94))))= ALPIPS and the morphing process requires N interpolated samples to start from (² to (1)), We define this set of N (non-uniform) interpolation scale as A = {a1,..., aN }. and a set of N uniform interpolation scale as B = ( N+I. w then perceptually uniform sampling can be con- sidered as function G B + A such that 9 ((T+T) a;, and the generation of interpolated sam- ples (given perceptually uniform sampling) can be written as ((x)) F(G(3)), ("), (2)) where 3, E B. Thus, rate of change Aupies is always (N + 1)- ALPIPS which is the total LPIPS along the morphing sequence. One noteworthy observation is that N and ALPIPS are not independent: as N increase, ALPIPS will decrease. The intuition behind is that for fixed number of N interpo- lated samples (from (9) to (²))), models with larger total LPIPS will have a larger rate of change (in term of LPIPS) between two consecutive samples (less smooth given fixed number of interpo- lated points). Another metric for smoothness is PPL: PPL. = E,-U10,122 LPIPS(x xxa++))) Following notations from the previous paragraph, if we let E N+I. PPL for perceptually uniform sampling is PPL. E.-- + 1)2. ALPIPS] (N 1)2 ALPIPS (total LPIPS) Standard ALPIPS
PPL calculation requires E = N+1 < In terms of perceptually-unifoon uniform sampling. N should be defined in a manner such that ALPIPS are almost imperceptible by human. Using total LPIPS to approximate PPL may not be precise due to numerical approximation, but the trend should be similar (larger total LPIPS indicates larger PPL).
F.2 IMPACTS OF CLASSIFIER GUIDANCE (CFG) SCALES
We discover that, given fixed LPIPS difference for perceptually-uniform sampling, larger CFG scales are likely to generate longer sequences (less direct morphs). The reconstruction quality also decrease for large CFG scales, but overall the differences are not visually significant. We show an example of sequence generated from different CFG scale in Figure 21 Detailed tables in our appendix well.
24

---

Published as a conference paper at ICLR 2024
Source
CFG: 1.5
CFG: 3
CFG:
Figure 21: Serceptuall-Unnffom Uniform Sampling with different CFG scales. The difference LPIPS
between consecutive images are set to 0.2.
BEMAZO Zata
DOUBLE
1121
EIZZIA
CRIRZILL
Figure 22: Additional morphing results with 9 frames.
25

---

Published as a conference paper at ICLR 2024
Figure 23: 100 interpolated images of "flower".
Figure 24: Poker card human.
26

---

Published as a conference paper at ICLR 2024
G ADDITIONAL MORPHING RESULTS AND DOWNSTREAM APPLICATIONS
Long Trajectory Morphing Results
To better illustrate the smoothness of morphing with our
proposed IMPUS, illustrate examples of interpolated images in Fig. 22 We also illustrate 100 interpolated images using our uniform sampling in Fig. 23
More Examples of Downstream Applications
We show some additional examples of down-
stream applications Fig. 25 27
Figure 25: Interpolation of two video frames teleporting parrot.
Sabure
Figure 26: Model explainability with our morphing tool (Swin'T).
30
Figure 27: Model explainability with our morphing tool (Swin')).
H IMAGE ATTRIBUTION
25 pairs of images: httpps////tthib.cc//////////eeet
Net Net
27

---

Published as a conference paper at ICLR 2024
Flower:
Car:
Camel
pinty-thee-flammnoo//
Movie of http:///ww/o/////////////oooaaat
Running ottppp//r/oaaaaa///....0/0/20110
Parrot: h|tpg///sssg/////ooooooooooo////
Movie poster Barbie: http:///../////ooooooooo////1111
Movie poster / htt.a///aaaiii/aoo/o////oooooooa
28


Reviews:
[{"review_id": "2DQUGdGuSo", "reviewer": "Reviewer_NAVh", "summary": "This paper proposes a new method to generate smooth and realistic image morphing results using diffusion models. The method starts with a baseline approach based on textural embedding interpolation and latent interpolation. To advocate direct and plausible interpolation results, authors fine-tune the diffusion model based on the image pairs with a bottleneck constraint implemented by LoRA to retrain the mode coverage along the morphing path. The rank is selected adaptively based on a metric that measures the density along the morphing path. To produce a smooth morphing, interpolation parameters are adaptively determined to ensure uniform perceptual transition. The proposed method demonstrates improved image morphing results compared to the prior work, especially when the semantics of the image pairs differ a lot.", "questions": "The rank adaption based on relative perceptual path diversity (rPPD) seems not effective enough. Is it possible to use much simpler heuristics based on CLIP score difference?\n\nThe techniques proposed in the paper do not make a significant difference when used independently, but the visual difference relative to the prior work (Figure 2) is impressive. It will be useful to show the qualitative results that show how the result progressively evolves by adding each technique one by one. Such analysis will let the readers understand which component plays the role at most.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The method proposes several effective techniques to improve the prior diffusion-based baseline, yielding smooth, realistic and direct morphing results. The method works well on challenging cases as shown in Figure 1 and Figure 2 in the main text and Figure 2 in the Appendix, which are indeed impressive. Notably, these are achieved without auxiliary knowledge such as pose guidance as used in prior the method.\n\n- The proposed techniques are simple yet effective. The LoRA finetuning effectively improves the plausibility of the interpolation results without mode collapse to training data. The perceptually uniform sampling based on the LPIPS difference is a quite useful technique and will benefit the following works in this area.\n\n- The paper proposes novel metrics to measure the morphing quality in terms of directness, realism and smoothness. The quantitative comparisons are solid and sufficiently prove the effectiveness of the method.", "weaknesses": "- While the proposed method is effective, I think the contribution of the paper is relatively incremental. The framework is mostly built upon Wang and Golland 2023. Using LoRA to derive an input-aware diffusion model is relatively straightforward. And the proposed metrics are mostly based on prior work. Also, the LoRA rank adaption is heuristic and cannot be regarded as a major contribution.\n\n- The physical meaning of the relative perceptual path diversity (rPPD) score is not clear. Why the score is a ratio that divides the\nperceptual difference of the endpoint images? Moreover, the accumulated LPIPS difference cannot properly reflect the semantic difference of the input image pair. For example, the morphing for two bears that change a lot in location may be much easier than the morphing for two objects that differ significantly in semantics. To properly choose the LoRA rank, a much simpler metric that depends on CLIP score difference seems to suffice.\n\n- Some of the proposed techniques only bring marginal improvement. As shown in Table 2, the best model is more likely not to be the one with adaptive rank. And in Figure 4, the difference between (n) and (o) is not significant.\n\n- There is no quantitative study for the effectiveness of separate LoRA for unconditional score estimation. Also, there is no smoothness measure for perceptually uniform sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qwgNd0jkzq", "reviewer": "Reviewer_zxkV", "summary": "This paper proposes IMPUS, a diffusion-based image morphing method that can synthesize smooth and realistic transitions between two images. It employs a latent diffusion model with distinct conditional distributions for each image and interpolates in the locally linear and continuous text embedding space and Gaussian latent space. The method employs an adaptive bottleneck constraint based on a novel relative perceptual path diversity score to yield a direct path and suppresses ghosting artifacts in the interpolated images. Extensive experiments confirm IMPUS's effectiveness in image morphing and its potential for other image generation tasks.", "questions": "Why is data manifold and data distribution discuessed in methodologies but not covered in experiments? Would it be possible to visualize the data manifold or distribution with figures or give quantitative experiment resutls?\nWhy is the proposed method compared with only one existing diffusion based image interpolation method? Are there other similar works, can they be compared in fair experiment settings and criteria?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea of applying diffusion models on image morphing task is interesting and promising;\nThe proposed evaluation criterion Smoothness,  Realism and Directness are reasonable and effective;\nThe presented techniques are effective and the overall method is able to generate high qulity results;\nAblation studies are thorough and abundant, which fully reveals the function and influence of each components of the model", "weaknesses": "In the comparison section, the proposed method is only compared with one previous method, which seems to be not comprehensive.  Also, there's only qualitative comparison, while quantitative criterion and user study would better convey the difference.\nIn the realism criteria and section3, the authors discussed about manifold and distribution several times, but gave no illustration or discussion in the experiment section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FIqFF34H6R", "reviewer": "Reviewer_meGk", "summary": "This paper presents an image morphing approach using pre-trained text-to-image diffusion models. The key idea is to interpolate text embeddings and latent states of two images following low-rank adaptation of model weights. The intuition is that the fine-tuned model exposes a more direct morphing path, yielding more perceptually convincing interpolations. Qualitative and quantitative experiments are performed to assess several key aspects of the proposed method. Overall, the method achieves superior morphing results compared to baselines and further supports interesting applications.", "questions": "- It would be helpful if the authors can comment on the robustness of their method relative to the hyper-parameters in model adaptation and sampling.\n\n- Most of the image pairs used for experiments seem to have certain degree of spatial alignment. It would be helpful to know the rule of thumb for the selection of these image pairs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method makes use of a pre-trained text-to-image diffusion model for image morphing. The data prior captured by the diffusion model naturally constrains the morphing path on the image manifold, allowing perceptually convincing interpolations.\n\n- The paper provides a helpful discussion on the tradeoffs in model adaptation. It discovers that LoRA creates an effective bottleneck to reduce overfitting, and further devises a simple strategy to calculate LoRA ranks based on sample diversity along the morphing path.\n\n- The paper introduces an effective approach for perceptually uniform sampling. It is based on adaptive step size selection and yields smooth and visually pleasing morphing results. \n\n- Several metrics are defined to evaluate different aspects of image morphing, namely smoothness, realism and directness. The method outperforms baselines in qualitative and quantitative experiments, and shows potential for several interesting applications.", "weaknesses": "- Despite the impressive results, there are some vague claims between the lines which need further clarification.\n\na) Equation 5 formulates image morphing as walking on the geodesic path in the 2-Wasserstein sense. However, the actual implementation amounts to linear interpolation of CLIP text embeddings, which in no way reflects the theoretical formulation.\n\nb) For the interpolation of latent states, the paper cites an early work saying x0 and xT forms an optimal transport mapping. How does this justify the slerp interpolation between xT(0) and xT(1)?\n\n- The method only compares to Wang and Golland (2023), another diffusion-based method for image morphing, yet there are numerous methods that are not based on diffusion (or even not learning-based). I am curious about how the method compares to, for example, frequency domain morphing methods, and what are their typical failure modes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vsy5b91FNP", "reviewer": "Reviewer_UYJ7", "summary": "This paper presents a diffusion-based image morphing approach aiming to produce smooth, direct, and realistic interpolations given an image pair.  The authors interpolate in the locally linear and continuous text embedding space and Gaussian latent space. An \"adaptive\" bottleneck constraint based on a relative perceptual path diversity score that automatically controls the bottleneck size and balances the diversity along the path with its directness. Experiments verify the effectiveness to some extent. Though the method is clearly described, there exist some issues with both motivations and experiments.", "questions": "* In Eq. (5), $\\nu$ is a distribution, why it should belong to the data manifold $\\mathcal{M}$ which should be the set of samples?\n* The embedding space of CLIP is locally linear. Since the two semantically distinct text embeddings are not in the global linear space, does it contradict the linear interpolation between the optimized embedding spaces?\n* What is the logic for \"the diffusion process between $\\bf x_0$ and ${\\bf x}_T$ can be seen as an optimal transport mapping, therefore, we apply spherical linear interpolation\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Image morphing in computer graphics is useful. Image morphing of nature images seems visually interesting to me.\n* The method of image morphing with perceptually-uniform sampling using diffusion models is clearly described.\n* The paper is OK to read.", "weaknesses": "* The motivation for image morphing of conceptually different nature images is not clear to me. Why should we study the image morphing of natural images without any human involvement? To my understanding, it may be reasonable to morph images for a certain purpose, e.g., continuously changing the poses or colors, via human involvement. However, how can we guarantee the generated images fit our purpose without any human involvement?\n* The perceptually-uniform sampling is not reasonable to me. If I am right, morphing should be that given a uniform sequence in time-space, e.g., [0,1], and generating images \"uniformly\" changing between start and end images. In perceptually-uniform sampling, can we obtain the images corresponding to a given middle time?\n* The authors argue \"adaptive\" bottleneck constraint. However, I can not see how the rank is adaptively learned.\n* More discussion on the perceptual path diversity is encouraged. For example, why should we use this to measure the diversity? Why use the formulation in Eq. (8)? Does large $\\gamma$ imply higher diversity?\n* It would be better to provide an algorithm for both fine-tuning and inference.\n* To show the superiority of the proposed method, more baseline approaches should be compared in experiments, such as the image morphing approaches in the related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)