FreeMorph: Tuning-Free Generalized Image Morphing with Diffusion Model

Authors: Yukang Cao, Chenyang Si, Jinghao Wang, Ziwei Liu
Primary area: generative models
Keywords: image morphing, diffusion model, tuning-free method

Abstract:
We present **FreeMorph**, the first tuning-free method for image morphing that accommodates inputs with varying semantics or layouts. Unlike existing methods, which rely on fine-tuning pre-trained diffusion models and are limited by time constraints and semantic/layout discrepancies, FreeMorph delivers high-fidelity image morphing without extensive training. Despite its efficiency and potential, tuning-free methods still face challenges in maintaining high-quality image morphing due to the non-linear nature of the multi-step denoising process and bias inherited from the pre-trained diffusion model. In this paper, we introduce FreeMorph to address this challenge by integrating two key innovations. **1)** We first propose a **guidance-aware spherical interpolation** design that incorporates the explicit guidance from the input images by modifying the self-attention modules, addressing identity loss, and ensuring consistent transitions throughout the generated sequences. **2)** We further introduce a **step-oriented motion flow** that blends self-attention modules derived from each input image to achieve controlled and directional transitions that respect both input images. Our extensive evaluations demonstrate that FreeMorph outperforms existing methods with training that is 10X - 50X faster, establishing a new state-of-the-art for image morphing. The code will be released.

Decision: reject
Reviewer scores: 5, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730297098734

thread_type:
official_review

<thread_object>
creator: reviewer_qmrm

<content>
summary:
This paper propose FreeMorph, a tuning-free pipeline that can generate transitions between two images within 30 seconds. It proposes two components, guidance-aware spherical interpolation, and Step-oriented motion flow, which yields smooth transitions between source and target images, even at examples of cross semantics. Both visual and quantitative evaluations are provided.

strengths:
I think this paper is above the bar, with good results and reasonable techinques. Experiments are well designed, limitations are discussed.

weaknesses:
Some motivation parts are not that clear to me, see below

questions:
This paper targets at two issues related to the image morphing, identity loss and directional trasition. However, I do not see the importance of these two issues. Why they are important and worthing efforts for this task. 

For example, when not preserving the identity, what results would be like. Similarly, what is the issue when miss preserve directional transition. I understand that the smooth transition is important, is it the same as the meaning of directional transition?

 These are important questions to me when reading the introduction. I can see cool examples in the intro, but cannot figure out clearly the core issues that this paper handles. I try to solve this by see examples of compared other methods, and some of them are still OK during the transition. 

What is the successful rate, are the results cherry-picked
</content>
</thread_object>

</thread>
<thread>
creation date:
1730298914138

thread_type:
official_review

<thread_object>
creator: reviewer_ncsg

<content>
summary:
This paper presents a novel tuning-free image morphing approach, which utilizes the pre-trained stable diffusion and manipulates its self-attention to achieve high-fidelity morphing. Specifically, it proposes a guidance-aware spherical interpolation and step-oriented motion flow, and respectively apply them in different steps of the reverse diffusion/forward denoising processes. Based on the qualitative and quantitative results, the proposed approach outperform the existing methods.

strengths:
- The proposed tuning-free morphing approach is straightforward and effective. The modified interpolation of the self-attention is able to produce more natural transition between morphing sequences.
- A new dataset, Morph4Data, is presented and used in the evaluation. This dataset contains four categories, which helps in a detailed analysis on the image morphing methods.
- This paper presents and analyzes many baseline settings in the ablation study. This helps in understanding the specific role of each key design.

weaknesses:
- This paper is apparently not well prepared for publication. For example:
  - There are many errors in spelling (“ur” in line 122, “owever” in line 133, etc.) and formulations (missing “{“ and “}” in line 179, etc.). 
  - In equation (7), since m has the same size as z, what does it mean by “m=1”? 
  - Both “DeepMorpher” and “deepmorpher” appear in the paper.
  - The image captions of Fig.2 and Fig.3 is too simple. Reader have to move to the main text to try to understand the figures.
- In the introduction, it mentions a comprehensive evaluation system, including a new dataset and specialized metrics. But in Sec 4.1, it says “following IMPUS and DiffMorpher, we ... using the following metrics”. Where are the proposed specialized metrics? 
- This paper presents both the qualitative and quantitative evaluations. In the figures, the “spherical interpolation” looks good than the existing method DiffMorpher. But based on the quantitative evaluation in Table 1, “spherical interpolation” performs worse than DiffMorpher. Therefore, either the visual results are cherry-picked, or the evaluation metrics cannot adequately measure the results. Without a human perceptual study, it’s difficult to get a conclusion when comparing the two methods.
- The proposed algorithm contains a reverse diffusion process and a forward denoising process. In my understanding, the diffusion process of DDIM is to directly add noise without using the trained network. But in Algorithm 1, the modified attention mechanism is applied in the diffusion process. There should be more details to explain how to implement this diffusion process. In addition, why the forward denoising steps also have “for t=1 to T”?
- For the experiment “Ours(Var-A)” in line 469, what does it mean to “omit the original attention mechanism”? Completely remove the attention?

questions:
My main concerns are the evaluation metrics and the technical details about the reverse diffusion process. I'd like to increase my rating if there are convincing explanations or evidences in the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667296478

thread_type:
official_review

<thread_object>
creator: reviewer_jpft

<content>
summary:
This paper presents FreeMorph, a tuning-free method for image morphing that accommodates inputs with varying semantics and layouts. Several modules are proposed to enhance the quality of image interpolants, ensuring a smooth transition that remains consistent with the given images at both ends.

strengths:
1. This method does not require training, making it more accessible to regular users.
1. The authors have conducted extensive experiments to identify the best combination of modules and hyperparameters.
1. Although this is a task that is difficult to evaluate, I can appreciate the authors’ efforts to provide both quantitative and qualitative results.

weaknesses:
Related to the task
1. There are some unverified claims at the core of this task. For example, see lines 254-255.
1. What constitutes good image morphing is somewhat underdefined. The intended effect that this paper seeks to achieve feels too vague and abstract, making it challenging to identify a clear definition of effective image morphing.
1. Even with quantitative metrics like PPL, high scores do not necessarily indicate results that align with human preferences, which can be highly subjective.

Related to the proposed method
1. While the authors have introduced several modules, their use appears to have been determined empirically. The selection of hyperparameters and the final module composition could benefit from clearer justification, as they seem somewhat arbitrary at times.
1. Given that the method involves multiple components, the presentation could be more streamlined. It can be difficult to understand the specific problem each module aims to address. (Please see questions below for further clarification.)
1. The generated samples have pretty visible ghost artifacts. To me, that would not be considered as good image morphing.

questions:
1. What's the intuition behind Equation 5. How does it relate to identity preservation?
1. How does Equation 6 connect to motion flow? What do you mean by "step-oriented"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669389606

thread_type:
official_review

<thread_object>
creator: reviewer_xahp

<content>
summary:
The paper proposes a method for image interpolation with off-the-shelf T2I diffusion models. By modifying the attention modules using two proposed methods, the continuousness of the interpolations is improved, without having to fine-tune the diffusion model. The authors also introduce auxiliary tricks to further improve interpolation performance. The contribution of the various proposed components to improved interpolation performance is validated using an extensive ablation study, and the overall performance is compared qualitatively and quantitatively with some alternative approaches.

strengths:
The quantiative and qualitative experiments, especially the ablation study seem well-executed and thorough, with the exception of one important method missing (see weaknesses). I really appreciate the authors thoroughly evaluating across different levels of challenging interpolations and proving a large number of qualitative comparisons in the appendix.

The proposed method seems reasonable in construction and effective in practice.

In the comparisons performed by the authors, their proposed method seems to provide a clear improvement over the methods they compare against.

Besides small mistakes (see questions), the paper is generally well-structured, reasonably well-written and easy to follow.

weaknesses:
Attention Interpolation for Text-to-Image Diffusion (He et al., 2024) is cited in the related work section but never compared to, despite the methods seeming quite similar. Given that the authors were aware of this work at the time of submission, I would expect the authors to discuss how their and this method relate and incorporate it into quantitative and qualitative comparisons.

Similarly, Smooth Diffusion (Guo et al., CVPR 2024) is quite closely related and could also be compared to, although it requires additional tuning and thus seems a lot less relevant as a baseline. Demonstrating that the proposed method performs similarly to one requiring extra fine-tuning would strengthen the paper substantially.

questions:
Eq. 3 looks wrong to me. Defining it with j as an integer in the range [1, 5] does not result in a slerp, as far as I know.

3.4: I think using a DCT instead of an FFT could help improve quality by getting rid of the correlation between the pixels on the left and right (and top and bottom respectively) edge of the image.

The number of decimal digits in tables is excessive. Please reduce it to a reasonable number (more than 3 or 4 digits should not be necessary in most cases) to aid readability.

There are a bunch of small mistakes. I'd recommend going over the paper and fixing them for a potential camera-ready version, as the current state looks a bit sloppy. Some examples: l. 122: "ur" -> "Our", l. 130: "Recently, Recently,"; l. 132: "wang2023interpolating"; l. 194: wrong citation for LDM, l. 211: I think DiffMorph should be attributed to different people.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733069195392

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their time and effort in reviewing our paper!

Below, we summarize the changes made according to the reviews:

1. We compare our method with "Attention Interpolation for Text-to-Image Diffusion" and "Smooth Diffusion" to demonstrate our superiority (`#XAhp`).

2. We discuss more about Eq. (6) for slerp (`#XAhp`).

3. We show the difference between DCT and FFT and illustrate their influence on the generated image transitions (`#XAhp`).

4. We update the decimal digits in the tables and rectify the writing typos (`#XAhp`, `#NcSG`).

5. We discuss the nonlinearity of the multi-step denoising process (`#JPFT`).

6. We clarify what is a good image morphing (`#JPFT`).

7. We conduct user studies to evaluate different methods in a subjective manner (`#JPFT`, `#NcSG`, `#qmRm`).

8. We provide further explanation and rationale behind the design of our proposed modules, offering a detailed justification (`#JPFT`).

9. We explain the positive influence of Eq. (5) on identity preservation (`#JPFT`).

10. We clarify the reason for the name of our modules. We also modify their names for more accuracy (`#JPFT`).

11. We improve the caption for Fig.2 and Fig.3 (`#NcSG`).

12. We clarify the components of our introduced benchmark and improve the writing (`#NcSG`).

13. We clarify the process of reverse diffusion steps and forward denoising steps (`#NcSG`).

14. We clarify the design of our experiments "Ours (Var-A)" (`#NcSG`).

15. We illustrate the importance of smooth transition and identity preservation for image morphing (`#qmRm`).

16. We clarify the difference between smooth transition and directional transition (`#qmRm`).

We sincerely thank all reviewers and the AC(s) again for their valuable suggestions, which have greatly helped strengthen our paper.

If you have any further questions, we would be happy to discuss them!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a tuning-free method for image morphing that accommodates significantly different semantics and image layouts. The authors introduce two key techniques—guidance-aware spherical interpolation and step-oriented motion flow—that yield visually pleasing transitions between source and target images. However, in addition to multiple spelling and typesetting issues, the reviewers identified several technical concerns, particularly regarding comparisons with closely related research (e.g., T2I Diffusion and Smooth Diffusion), subjective evaluations, and the rationale behind key components. The authors provided feedbacks, yet three of them remained somewhat hesitant to recommend acceptance. The Area Chair also notes that external links were provided for some experimental results, but the risk revealing reviewers' identities may exist. Although the fourth reviewer’s assessment (Reviewer qmRm) was more positive, the Area Chair observed discrepancies between the review’s tone and the final rating, reducing its overall weight in the decision. Taking all feedback into account, especially the reservations of the first three reviewers, the Area Chair has decided to reject the manuscript. The authors are encouraged to resubmit to a future venue, including all necessary supportive materials within the official review package rather than relying on external websites.