NoiseDiffusion: Correcting Noise for Image  Interpolation  with Diffusion Models beyond Spherical Linear Interpolation
Authors: PengFei Zheng
Yonggang Zhang
Zhen Fang
Tongliang Liu
Defu Lian
Bo Han
Venue: ICLR 2024 spotlight (2024)
Keywords: Image Interpolation; Diffusion Models
Primary area: generative models

Abstract: Image interpolation based on diffusion models is promising in creating fresh and interesting images. 
Advanced interpolation methods mainly focus on spherical linear interpolation, where images are encoded into the noise space and then interpolated for denoising to images. 
However, existing methods face challenges in effectively interpolating natural images (not generated by diffusion models), thereby restricting their practical applicability. 
Our experimental investigations reveal that these challenges stem from the invalidity of the encoding noise, which may no longer obey the expected noise distribution, e.g., a normal distribution. 
To address these challenges, we propose a novel approach to correct noise for image interpolation, NoiseDiffusion. Specifically, NoiseDiffusion approaches the invalid noise to the expected distribution by introducing subtle Gaussian noise and introduces a constraint to suppress noise with extreme values. In this context, promoting noise validity contributes to mitigating image artifacts, but the constraint and introduced exogenous noise typically lead to a reduction in signal-to-noise ratio, i.e., loss of original image information. Hence, NoiseDiffusion performs interpolation within the noisy image space and injects raw images into these noisy counterparts to address the challenge of information loss. Consequently, NoiseDiffusion enables us to interpolate natural images without causing artifacts or information loss, thus achieving the best interpolation results.

Published as a conference paper at ICLR 2024
NOISEDIFFUSION:
CORRECTING NOISE FOR IMAGE INTERPOLATION WITH DIFFUSION MODELS BEYOND SPHERICAL LIN-
EAR INTERPOLATION
Pengfei Zheng' Yonggang Zhang² 'University of Science and Technology of China "TMLR Group, Hong Kong Baptist University Uniiversity of Technology Sydney *Sydney Centre, The University of Sydney
Zhen Fang² Tongliang Liu Defu Lian' Bo Han
ABSTRACT
Image interpolation based on diffusion models is promising in creating fresh and interesting images. Advanced interpolation methods mainly focus on spherical linear interpolation, where images are encoded into the noise space and then in- terpolated for denoising to images. However, existing methods face challenges in effectively interpolating natural images (not generated by diffusion models), thereby restricting their practical applicability. Our experimental investigations reveal that these challenges stem from the invalidity of the encoding noise, which may longer obey the expected noise distribution, e.g., normal distribution. To address these challenges, we propose a novel approach to correct noise for image interpolation, NoiseDiffissinn Specifically, NoiseDiffusion approaches the invalid noise to the expected distribution by introducing subtle Gaussian noise and introduces a constraint to suppress noise with extreme values. In this context, pro- moting noise validity contributes to mitigating image artifacts, but the constraint and introduced exogenous noise typically lead to a reduction in signal-to -t---oisse ratio, i.e., loss of original image information. Hence, NoiseDiffusion performs interpolation within the noisy image space and injects raw images into these noisy counterparts to address the challenge of information loss. Consequently, NoiseD- iffusion enables us to interpolate natural images without causing artifacts or infor- mation loss, thus achieving the best interpolation results. Our code is available at https: //tt:///ggttubb// com/imlree wtt.aa/o...ooo.oooooooooooooooao
Original Imagee
Introdacing Noise for Interpolation
Linear Spherical Interpolation Meehod
Nole-Diffadoo
Original Image
Figure 1: Comparison of images generated with different interpolation methods.
INTRODUCTION
Image interpolation is an exceptionally fascinating task, not only for generating analogous images but also for igniting creative applications, especially in domains like advertising and video gener- ation. At present, state-of-the-art generative models showcase the ability to produce intricate and captivating visuals, with many recent breakthroughs deriving from diffusion models (Ho et al. 2020
Song et al. 202 Rombach et al. 2022 Saharia et al. 2022b: Ramesh et al.) 2022). The potential
"Corresponding Author Defu Lian (liandefu @ustc.edu.cn)

---

Published as a conference paper at ICLR 2024
of diffusion models is widely acknowledged, but our knowledge, there has been relatively little research on image interpolation with diffusion models (Croitoru et al. 2023).
Within the realm of diffusion models, the prevailing technique for image interpolation is spherical linear interpolation (Song et al. b0) This approach shines when employed with images gener- ated by diffusion models. However, when extrapolated to natural images, the quality of interpolation results might fall short of expectations and frequently introduce artifacts, as depicted in Figure
We initially analyze the spherical linear interpolation process and attribute subpar interpolation sults to the invalidity of the encoding noise. This noise does not obey the expected normal distri- bution and may contain noise components at levels higher lower than the denoising threshold resulting in artifacts in the final interpolated images. Directly manipulating the mean and variance of the noise through translation and scaling straightforward approach to bring it closer to the desired distribution. However, this not only fails to improve the image quality but also results in the loss of image information. In addition, combined with the SDEdit method(Meng et al.) [2022), we directly introduce standard Gaussian noise for interpolation. While this method improves the quality of images, comes at the expense of introducing additional information, as depicted in Figure 4
To improve the interpolation results, we propose a novel approach to correct noise for image inter- polation, NoiseDiffiasio. Specifically, NoiseDiffusion approaches the invalid noise to the expected distribution by introducing subtle Gaussian noise and introduces constraint to suppress noise with extreme values. In this context, promoting noise validity contributes to mitigating image artifacts, but the constraint and introduced exogenous noise typically lead to reduction in signal-to-noise ratio, i.e., loss of original image information. Hence, NoiseDiffusion subsequently performs in- terpolation in the noisy image space and injects raw images into these noisy images tackle the information loss issue. These enhancements enable us to interpolate with natural images without artifacts, yielding the best interpolation results achieved to date. Considering the limited explo- ration of previous research in this field (Croitoru et al. (023), we hope that our research can provide inspiration for future research.
2 RELATED WORK
Diffusion Models Diffusion models create samples from the Gaussian noise using sequential de- noising steps. To date, diffusion models have been applied to various tasks, including image gen-
eration (Rombach et al. [002] Song & Ermon 2020) Nichol et al.] 2022 Jiang et al.. 2022), image
super-resolution (Saharia et al. 2022c Batzolis et al. [2021] Damels et al. (021), image inpaint- ing (Esser et al., (021). image editing (Meng et al. (022), and image-to-image translation (Saharia et al. 2022a). In particular, latent diffusion models (Rombach et al., 2022) excel in generating text-conditioned images, receiving widespread acclaim for their ability to produce realistic images.
Image Interpolation Earlier approaches, such StyleGAN (Karras al. 2019), allowed for in- terpolation using the latent variables of images. However, their effectiveness is constrained by the model's 's ability represent only a subset of the image manifold, presenting challenges when applied to natural images (Xia et al., 2022). What's s more, latent diffusion models can utilize prompts to in- terpolate the generated images (like Lunarring), but its interpolation potential on natural images has not yet been discovered. To the best of our knowledge, a method for interpolating natural images using latent variables with diffusion models has not been encountered.
PRELIMINARIES
In this section, we first introduce how to describe the diffusion model's noise injection and denoising process in the form of stochastic differential equations (SDEs). Building upon this, we provide a brief overview of how diffusion models are used for image interpolation and editing. Through image editing, we can implement an interpolation method that doesn't it require latent variables, that is, introducing Gaussian noise and then denoising. These methods form the foundation of the proposed approach, DifssDif
Noiie level of denoising
2

---

Published as a conference paper at ICLR 2024
3.1 THE DETAILS OF DIFFUSION MODELS
Perturbing Data With SDES (Song et al. b02 We denote the distribution of training data as Paata()), and the Gaussian perturbations applied to (ar by the diffusion model be de- scribed by the following stochastic differential equation expression:
dx: pl t)dt + (t)dw..
where E B.T)T>> > is fixed constant, {w, (w, denotes the standard Wiener process (a.k.a., Brownian motion), (u(--): Rd _2 is vector-valued function called the drift coefficient of ær, and (()) R - IR is scalar function known as the diffusion coefficient.
We denote the distribution of æ, as (r(e and consequently, Pu represents for the training data distribution Pdata and pr is an unstructured prior distribution that contains no information of pu-
Generating Samples By Reversing the SDEs (Song et al. b02 By starting from samples of pr and reversing the perturbation process, we can obtain samples process is also diffusion process and can be given by the reverse-time SDE (Anderson 1982):
Po. The reverse of diffusion
dx [p( t) (t) log at))du
where w is a standard Wiener process when time flows backwards from T to O, and dt is an in- finitesimal negative timestep. Once the score of each marginal distribution, V log Po(x). known for all f, we can derive the reverse diffusion process from Eq 2 and simulate it to sample from Po- And methods like stochastic Runge-Kutt Kutta (Kloeden et al. 1992) methods can used to solve this.
Probability Flow ODE (Song et al., 2021b) Diffusion models enable another numerical method for solving the reverse-time SDE. For diffusion processes, there exists a corresponding deterministic process whose trajectories share the same marginal probability densities (xx))) as the SDE. This deterministic process satisfies an ordinary differential equation (ODE):
dæ:- = [p(=,, t) 2 (f) Po
(3)
which can be determined from the SDE once scores are known. Usually we call the ODE in Eq the probability flow ODE.
3.2 IMAGE EDITING
Spherical Linear Interpolation In diffusion models, the prevailing image interpolation method is spherical linear interpolation (Song et al. 2021a
a (A) sin sin((11 X)0) (0) sin (1) s0n(O)
2T sin(O) ØT
where 0 = arccos( (ar
and À is coefficient that controls interpolation style between
(1) two images. zp can be either noisy image encoded from image = by integrating Eg 3 or randomly sampled standard Gaussian noise. After completing the interpolation of latent variables through the above equation, decoding can be achieved by integrating the corresponding ODE for
the reverse-time time SDE. In the rest of the paper, we use szerp(=;"" (0) x ,, A) to denote the spherical linear interpolation of the latent variables x; and (1) with the interpolation cocfficient A.
Image Editing with SDEdit (Meng et al. 2022) The SDEdit accomplishes image modifications by overlaying the desired alterations onto the image, introducing noise, and subsequently denoising the composite. This process ensures that the resulting image maintains a high level quality. For any given image æo, the SDEdit procedure is defined follows:
Sample
(a o² (to)) then produce 2o by solving Eq 2
For appropriately trained SDE models, trade-off between realism and faithfulness emerges when varying the values of to. When add more Gaussian noise and run the SDE for longer, the synthesized images are more realistic but less faithful. Conversely, adding less Gaussian noise and running the SDE produces synthesized images that more faithful but less realistic.
3

---

Published as a conference paper at ICLR 2024
Diffusion Model
Diftusion Model
(0) x1
(1)
Stapp
Iatepsatio Interpllation
Encode to Tatemi Sgever
largo
Deneise
inage
The spherical linear interpolation. Original images: The images on the left are natural images, whereas the images the right are generated by the diffusion model. Interpolation results: The images on the left and right are the interpolation results of natural images and images generated by diffusion model respectively.
Figure 2:
4
THE IMAGE INTERPOLATION METHODS
4.1 THE SPHERICAL LINEAR INTERPOLATION OF IMAGES
Let's start by introducing the process of spherical linear interpolation images. Given two images, the initial step involves encoding them into a latent space, i.c., Eq. [nd 5 Then, we can perform
spherical linear interpolation on the latent variables, i.c., Eq. 6 followed by denoising generate the interpolation results with Eq.
2, f (0) t),
(4)
f(a t), 2 (1)
(5)
0t s slerp (21 2i (1)
A),
(6)
to f (x),
In this context, we denote the Gaussian noise & (t)) and the original image as z
with i E {0, 1}. respectively. Accordingly, (z) represents the noisy image corresponding to the variable of the image in the latent space with noise level ((). Using the probability flow ODE for its stability and unique encoding capabilities, and we denote this encoding process as a function f. Similarly, we denote the decoding process as f which corresponds to denoising through the ODE associated with the reverse-time SDE.
encode into the latent space by integrating Eq
Examining Figure 2 we notice that the interpola- tion result derived from natural images (not gen- erated from diffusion model) displays notice- able artifacts. contrast- ing with the derived from images generated by the diffusion model, which is free from such imperfections.
Figure 3: The impact of noise levels. We added Gaussian noise with lev- els of (t)) [70, 75, 80, 85, 90] the image on the left. Subsequently, we applied denoising to each noisy image with the same noise level of (t') = 80, resulting denoised images on right.
4.2 THE REASON FOR FAILURE
To explore what kind of potential variables be better denoised, add Gaussian noise to the image at various noise levels (t), resulting in I; Zo + Gt, and then denoise them at the same noise level (t'), yielding tn f- (xx,t') t'). The results are shown in Figure
Based on the results depicted Figure 3 we observe that adding Gaussian noise matching the de- noising level produces high-quality images. However, when the noise level exceeds the denoising

---

Published as a conference paper at ICLR 2024
threshold, additional artifacts are introduced in the generated images. Conversely, when the noise level falls below the denoising threshold, the resulting images appear somewhat blurred, accompa- nied by noticeable loss of features.
This phenomenon is rather peculiar since, in the context of Gaussian distribution, points closer to the mean typically exhibit higher probability density. In other words, within the framework of the diffusion model, noisy images with lower noise levels (closer to the mean) should ideally be more effectively denoised. Building upon these observations, introduce Theorem provide an explanation for this phenomenon:
Theorem 1.
The standard normal distribution (0, I.) in high dimensions is close to the uniform distribution the sphere of radius vn.
The detailed proof process of Theorem can be found in Appendix A. Theorem I indicates that random variables following the standard normal distribution in high dimensions are primarily distributed on hypersphere. This is because, increases, but the volume in nigh-dimensional space gradually expands as we move away from the mean. This result neatly explains why only noisy images with noise levels matching the denoising threshold can produce high-quality results after denoising: During the training process, the model can only observe noisy images primarily reside on the hypersphere. Consequently, it can only effec- tively recover images of this nature.
approach the mean, the probability density
Building upon Theorem mismatch between noise levels and denoising threshold. The natural images encompass numerous features that the model has not previously encountered. Consequently, the latent variables do not obey the expected normal distribution, and may contain noise components at levels higher or lower than the denoising threshold, resulting low image quality after denoising. Inspired by SDEdit, we can directly introduce Gaussian noise to the images as a solution to this mismatch problem. Details are follows.
attribute the failure of spherical linear image interpolation the
4.3 INTRODUCING NOISE FOR INTERPOLATION
Here, we introduce the image interpolation method combined with SDEdit. When given two images, the method starts by introducing Gaussian noise at the same level to each of them. Following this, employ spherical linear interpolation and subsequently apply denoising:
(0)
(0)
a;
Do Ex
(8)
(1) (1)
v + ct.
(9)
2, s slerp (2 2i A),
(10)
tu f (xt, t).
(11)
The noise added to the images can be either the same or different. Shortly, we will demonstrate that they exhibit only minor distinctions. However, it is crucial to emphasize that since this image inter- polation method is based SDEdit, it unavoidably inherits the drawbacks of the SDEdit method,
as illustrated in Figure 4
The interpolation results presented in Figure indicate that the method can address the issue of poor image quality. However, when we add more Gaussian noise and denoise, the interpolated images, while maintaining the original style, exhibit a phenomenon resembling direct image overlay. Conversely, selecting less Gaussian noise and denoising, while ensuring realistic images, introduces additional information, ultimately resulting in interpolation failure.
4.4 NOISEDIFFUSION
Based on the experimental results above, we can conclude the following: when spherical linear in- terpolation is directly applied to natural images, the resulting images can better preserve the original features but may contain artifacts. Conversely, directly introducing noise for image interpolation may yield high-quality images but often causes the information loss issue. To integrate these two methods, we propose the following theorem.
5

---

Published as a conference paper at ICLR 2024
Diffusion Model
Natural
+
Innepstation Rosuh
X
Add Caussinn Noise et
Skep
Naturil
Intepodation
Result
Introducing noise for image interpolation. In the interpolated images, the top one repre- sents the interpolation result with less Gaussian noise, while the bottom one represents the interpo- lation result with more Gaussian noise.
Figure 4:
Theorem 2. In high-dimensional spaces, independent and isotropic random vectors tend to be al- most orthogonal.
The detailed proof process of Theorem 2 can be found in Appendix A.2 Based on Theorem
and Theorem 2 we proposed a new image interpolation method called NoiseDiffusion: Given two images, we begin by encoding them into the latent space and clip them to suppress noise with extreme values. Next, we synthesize the latent variables with Gaussian noise, combining them with the original images, and finally apply clipping and denoising to produce the interpolation results:
z; = clip(f(=)))) (t))
(12)
x (1) <l cf (1) .t))
(13)
(1) * #t; + B + (u a) tu + (v - B) xo +
(14)
a (clpp(m())
(15)
In these equations, a and 3 correspond to coefficients for image style, while μ and v serve as com- pensation coefficients to adjust the amount of original image information. Additionally, represents the lubrication coefficient, which can be used to adjust the amount noise to enhance image quality.
Ensuring that the formula Va² + f + 72 = is satisfied is crucial. Drawing from Theorem and Theorem 2 we can infer that for any three high-dimensional vectors hypersphere with a radius of ||r|] denoted as U1, and V3 the magnitude of the weighted sum U12 is given by || || = ||a
v + v2ll / 1 12
/22 cos 0 aa² |/||r|| Moreover, it is worth noting that the newly obtained vector 012 and the vector v3 also remain orthogonal. Consequently, represent the magnitude of the weighted sum of these vectors as: ||a 11 8 + 7- Uyll /a² 2 + f2 7 |r]]. While the denoised image in Figure displays some artifacts. the majority of its content remains clear. This observation implies that the latent variables of natural images V1
2 i
a (U) U2 (1) also tend be near the hypersphere. Therefore, considering that Gaussian noise Vs = 6 also resides on the hypersphere, it is crucial maintain the formula / 2/2+ + + =1
to ensure that the final synthesized latent variable also possesses the same properties.
4.5 BOUNDARY CONTROL.
According to the widely recognized statistical principle known as the empirical rule (also known as 68-95-99.7 rule) (Pukelsheim) 1994), which pertains the behavior of data within normal dis- tribution, approximately 99.7% of data points are located within three standard deviations from the mean. Consequently, considering analysis of how noise above the denoising threshold impacts images, data points exhibiting significant deviations from the mean are considered potential sources of image artifacts, hypothesis that will be validated in subsequent experiments. To mitigate their influence, we employ the following boundary control (clip) procedure:
Boundary,
if Pixel Value Boundary, if Pixel Value -Boundary, Pixel Value, otherwise.
Pixel Value
Boundary,

---

Published as a conference paper at ICLR 2024
4.6 THE CONNECTION OF METHODS
Here, we establish the relationship between our approach and other methods, highlighting the ad- vantages of our approach. To begin with, our method, when coupled with appropriate parameter choices, can be adapted into two other methods:
Spherical Linear Interpolation Combining Theorem 2
as high-dimensional random vectors are orthogonal, we can express spherical linear interpolation in the following form with 0 =
sin((1 sin((1-X)0) 110) zi
(0) + sin(X0) xz sin((1 A) "aaa +sin(2 2) )al (1)
x (A)
sin(O)
sin(0)
This is equivalent to our method with 7 0, 4== a sin((1 A). 3) and v sin A 2).
Introducing Noise for Interpolation We classify the method of introducing noise for image inter- polation into two categories, assuming that the noise level is substantially higher than that of the image, which is often the common case. In this scenario, we can show that our approach can be adapted into this method:
1. The noise added to the image is the same:
at = s eerp(ar Bt A) sin((1 11 -)) (0) sin(A 0) (1)
(0)
(1)
sin(O) 2.. + (0) ai.
(1 \)az (0)
Ax (1) -11 1)x0 (0)
Ax0
(1)
+ +11 x A)et
= (1 - \az0 + A
(1) ce..
This is equivalent to our method with a = 3 = = = -
2. The noise added to the image is different:
sin((1 A) 7)
sin (A- 7) (1)
= slerp(aa (0)
Dt A)
(1)
(0)
+
sin( + xx1
xt
sin(
sin((1 ))
2 0
(0) sin(1
2 )a0
(1)
+ sin((1 A)
2 ")c," + sin(2 7 (1)
2 3)(_")
sin((1 A) 2 m
(0) sin(X
π (1)
2 0
+ t.
This is equivalent to our method with a = 3 0, p = sin((1 .)) sin( })
v=
Compared with spherical linear interpolation, our method introduces Gaussian noise to better posi- tion latent variables on the hypersphere. In contrast to the approach of introducing noise for inter- polation, our method incorporates noise correction, which enables us to position latent variables on the hypersphere and remove artifacts with a smaller amount of Gaussian noise.
EXPERIMENTS
The SDE typically designed such that pr close a tractable Gaussian distribution TTx). We hereafter adopt the configurations in Karras et al. (0022), who set fı t) = 0 and (tt) = V2r. In this case, we have (x( = Paata (x:) & (0, TT). where & denotes the convolution operation, and ा (œ) = (0, 1). We conduct evaluations on diffusion models trained on LSUN Cat-256 and LSUN Bedroom-256 images a basis for our evaluation. We verify the effectiveness of our method on the Stable Diffusion (Rombach et al. 2022). ) and the results are provided in the Appendix C
5.1 THE LUBRICATING COEFFICIENT
We keep all other parameters unchanged and incrementally increase y from 0 to 1, as illustrated in Figure Upon observation, it is apparent that as y increases, the artifacts in the image gradually diminish, resulting in a notable enhancement in image quality. However, at the same time, the image gradually loses some of its original features and introduces additional information.
7

---

Published as a conference paper at ICLR 2024
y=
= v0.2
v0.4
Y v0.6
00.8 y =1
Figure 5: The impact of lubricating coefficient y.
5.2 THE CHANGE IN STYLE
As shown in Figure 6 we can change the style of images by modifying the values of a and B. In order to facilitate comparison with the results of spherical linear interpolation, we choose sin (3 A). 5 = cos (3 A) and y = 0. Additionally, more interpolation results are available in the
appendix (Figure 11 and Figure 12
Original Image
0.1
A 0.3
À 0.5
0.7
0.9
Original Image
Figure 6: The image style changes with the variation of A.
5.3 BOUNDARY CONTROL
We implemented boundary control the la- tent variables, and the results are depicted in Figure 8 It can be seen that the boundaries decrease, the artifacts on the im- age are greatly reduced, which substantially improves the quality of the images. Fur- thermore, we compared three boundary con- trol methods: control before interpolation, control after interpolation, and control be- fore and after interpolation. The results of the three methods are shown in Figure T0 Upon examination, can be observed that the method of applying constraints to latent vari- ables before and after interpolation more effective in reducing artifacts.
The effect of image scaling is demon strated in the top and bottom rows of images, show- casing results obtained by scaling the original im- age with scales = [10, 2, 0.5, 0.1]. the middle row, we present the outcomes of image interpola- tion, maintaining all parameters except for μ and v, which are adjusted to μ [10, 1, 0.5, 0.1).
Figure 7:
However, reducing the boundaries also leads to some loss of image features and darken- ing, which implies that boundary control can result in information loss. To address this is- sue, one effective strategy to incorporate the original image information in the noisy image space, as detailed below.
5.4
THE IMPACT OF IMAGE INFORMATION
Figure fillustrates the impact of modifying the information of the original images (i.e., modifying the values μ and v) on the interpolation results. can be observed that smaller values of and lead to darker images while increasing them results in overly bright pictures. Images obtained with
8

---

Published as a conference paper at ICLR 2024
Boundary-2.0 Boundary-2.2 Boundary-2.4 Boundary-2.6 Boundary-2.8 Boundary-3.0 Boundary-3.2 3.22
Figure 8: The impact of the boundary.
# 0.500 /
0.625 μ 0.750 == v=v 0.875 / 1.00 Original image
Figure 9: Supplementing the original image information.
smaller μ and y values exhibit similarities to those obtained with boundary control applied. Thus, by modifying the values (u and v, we may be able to mitigate the feature loss and darkening issues caused by boundary control, which can be seen in Figure What's more, our method ensures that noise levels meet the necessary threshold, but the information of images may exceed or fall short of the desired threshold because this is determined by and 3. Therefore, by adjusting the parameters yu and v, we can regulate the information of images, thereby improving the interpolation results.
5.5
FINAL RESULT
We collected images from the Internet and employed three different methods for image interpola- tion. Throughout the interpolation process, we maintained consistency in parameter settings, with detailed information available in the Appendix. From the interpolation results, we observe that our method effectively reduces artifacts and maximally preserves information compared directly ap- plying spherical linear interpolation. Furthermore, our approach outperforms methods involving noise introduction in preserving original image features, as illustrated in Figures 13 and 14
6 CONCLUSION
In this paper, we propose novel method that surpasses the limitations of spherical linear inter- polation. Our approach establishes unified framework for both spherical linear interpolation and directly introducing noise for interpolation methods, leveraging the strengths of each. Addition- ally, by imposing boundary control on noise and supplementing the original image information, method effectively tackles the challenges posed by noise levels exceeding falling below the denoising threshold. Through the correction of latent variables, our approach improves the interpo- lation results of natural images, achieving superior interpolation outcomes.
Limitation and future work. Our approach, like any method, is not without its drawbacks and constraints. Compared to directly introducing noise for interpolation, our method involves extra step: mapping the images to the latent variables. This additional overhead will double the pro- cessing time. However, this extra overhead leads to better feature preservation. Furthermore, our paper mainly focuses on image data. Accordingly, its effectiveness in other modalities has not been validated, which is potential limitation of our work. Thus, we will explore the possibility of our method in different modalities in our future work. We will also explore the possibility of apply- ing our method to different scenarios, such as a) investigating the interpolation between natural and adversarial images (Zhang et al., 2022). b) studying the interpolation among different envi- ronments (Arjovsky et al. 2201)), and c) exploring the interpolation between in-distribution and nut-of-distribution data (Fang et al. 2022). Moreover, it is exciting apply method to many interesting scenarios, like interpolation between different person images, interpolation for low-level computer vision (Zamir et al. 2021), and interpolation for video generation (Liu et al. 2024).

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGMENTS
The work was supported by grants from the National Key R&D Program of China (No. (02120011101) YGZ and BH were supported by the NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652, HKBU Fac- ulty Niche Research Areas No. RC-FNRA-IG12--23SCII
and HKBU CSD Departmental In- centive Scheme. TL is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, 1.7,,,P22 LP220200949, ICT90100031.
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 1982.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv, 2019.
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv, 2021.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Max Daniels, Tyler Maunu, and Paul Hand. Score-based generative neural networks for large-scale optimal transport. In NeurIPS, 2021.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. In NeurIPS, 2021.
Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-of-distribution detection learnable? In NeurIPS, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, In NeurlPS, 2020.
Yuming Jiang, Shuai Yang. Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics, 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeuriPS, 2022.
Peter E Kloeden, Eckhard Platen, Peter Kloeden, and Eckhard Platen. Stochastic differential equations. Springer, 1992.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv, 2024.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Friedrich Pukelsheim. The three sigma rule. The American Statistician, 1994.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv, 2022.
10

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPHI, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence 2022c.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021a.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In NeurIPS, 2020.
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, b021b.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf. and Kun Zhang. Causaladv: Adversarial robustness through the lens of causality. In ICLR, 2022.

---

Published as a conference paper at ICLR 2024
A PROOFS
A.1 THE PROOF OF THEOREM 1
Lemma 1. Let X coordinates X₃ that satisfy EX?
R" be random vector with independent, sub-gaussian
Then
||2 VT 22 S CK²
where = max; |X
C is an absolute constant and we define:
X Vv, inf ( > 0 I8 exp( X //t) - 2] lt inf [t > 0: E exp
X 2 /22) v 2]
Proof. For simplicity, we assume that K 2 1. We shall apply Bernstein's deviation inequality for the normalized sum of independent, mean zero random variables
X 1 =
-)
==1
Since the random variable X; is sub-gaussian, X? - is sub-exponential, and more precisely
||? - 11lv, - C|| l
= x
< CK2
Applying Bernstein's 's inequality, we obtain for any 2 that
P X 11 2 u} < 2 expp
K4 min(u², (in((u,u) u))
This is good concentration inequality for | from which we are going deduce concentration
inequality for X |. To make the link, we can use the following elementary observation that is valid for all numbers 2 O:
1
1| 2 5 implies | - 1| 2 max(6, 6,,8)
We obtain for any 8 2 0 that
P(
X
12
1
2 5} - P(|
X 1
1 N max 6, 8²)}
s 2xepl
en 5²) (for u = max 52))
K
Changing variables to t = 8, we obtain the desired sub- lob-gaussian tail
P | vnl 2 t] S 2 exxp
for all 2
K
As we know form Sub-gaussian properties, this is equivalent the conclusion of the theorem.
Theorem 1. The standard normal distribution NO, I,) in high dimensions is close to the uniform distribution the sphere of radius /n.
Proof. from Lemma 1, for the norm of g ~ N(0,, In) we have the following concentration inequal- ity:
Pilllll2- - vnl 2 t < exp( ctt) for all t 2 0
Let us represent ~ (0, I,) in polar form as
r0
where ||a||2 is the length and 0 g/ /lll|a is the direction of g.
Concentration inequality says that r= || ~ /n with high probability, so
g ad v n0 Unif( (
In other words, the standard normal distribution in high dimensions is close to the uniform distribu- tion the sphere of radius /n, i.e.
N(0, I.) = Vndl ~ Unif( Vnsm-1)
12

---

Published as a conference paper at ICLR 2024
A.2 THE PROOF OF THEOREM! 2
Definition 1. A random vector X in R" is called isotropic if
[(x) - In
where In denotes the identity matrix in R..
Lemma 2. A random vector X in Rn is isotropic if and only if
E E(X, m²² ||0112 for all E Ik""
Proof. Recall that two symmetric n x matrices A and B are equal if and only if x Ax = x Bæ for all æ E R". Thus X is isotropic if and only if
x (EXX Ja
I for all æ E R
The left side of this identity equals E(X, x)² and the right side | Lemma 3. Let X be isotropic random vector in TR Then
= n
Moreover, if X and Y are two independent isotropic random vectors in R", then E(X, Y)² =
Proof. To prove the first part, have
E| = EX" x
E E X)
(viewing X" X as a x matrix)
x
(by the cyelic property of trace)
=xx((Xx))) (by linearity)
= tr(In) (by isotropy)
n
To prove the second part, we use a conditioning argument. Fix a realization and take the conditional expectation (with respect to X) which denote Ex. The law of total expectation says that
E E(X, Y) |xXx Y)
where by Ey we of course denote the expectation with respect to Y. To compute the inner ex- pectation, we apply Lemma with æ Y and conclude that the inner expectation equals ||||] Thus
B(X, Y)² Ey ||
= n (by the first part of lemma)
Theorem 2. In high-dimensionul spaces, independent and isotropic random vectors tend to be al- most orthogonal
Proof. Let normalize the random vectors X and Y in Lemma setting
X and Y
X
Y
Lemma 3 is basically telling us that probability, which implies that
|| 2| VT, Y x vn and (X,Y) vn with high
Thus, in high-dimensione spaces independent and isotropic random vectors tend to be almost or- thogonal.
[
13

---

Published as a conference paper at ICLR 2024
B EXPERIMENTS WITH MODELS TRAINING IN SINGLE DOMAIN
Parameter choices To facilitate comparison with spherical linear interpolation results, we maintain the condition a/3 sin { >) / cos (3 A). and ensure that va² + 3 + = when comput- ing and B. And we set = 2.0 * az/ (a + $), = 2.0 * B/(a + 3). Additionally, several other parameters, albeit hyperparameters, have predefined ranges for user convenience. For instance, the boundary ranges from 2.0 to 2.4, 7 [0, vo.1]. Users only need to determine the value of À to specify the style of interpolation results.
B. THE IMPACT OF THE BOUNDARY
Boundary=2.0 Boundary=2.2 Boundary=2.4 Boundary=2.6 Boundary=2.8 Boundary=3.0 Boundary=3.22
The impact of the boundary. From top to bottom: controlling noise before inter- polation, controlling noise after interpolation, and controlling noise both before and after inter- polation. From left to right: the coefficient ratio of the noise boundary to the variance are 2.2, 2.6, 2.8, 3.0, 3.2].
Figure 10:
We compared three boundary control methods: control before interpolation, control after interpo- lation, and control before and after interpolation, as shown in Figure I0 From the figure, we can observe that all three methods introduced similar level of blurriness, indicating a loss of image information, and applying constraints to noise both before and after interpolation is more effective in reducing artifacts.
B.2 INTERPOLATION OF IMAGES WITH MODELS TRAINED ON LSUN BEDROOM-255 256
We searched online for images of bedroom and used a diffusion model trained exclusively on LSUN Bedroom-256 images for interpolation. We gradually increased the value of À to modify the style of the interpolation images and ensuring that other parameters are within the specified range. The results shown in Figurd
B.3 INTERPOLATION OF IMAGES WITH MODELS TRAINED ON LSUN CAT-256
We searched online for images of cat and used diffusion model trained exclusively on LSUN Cat-256 images for interpolation. We gradually increased the value of to modify the style of the interpolation images and ensuring that other parameters within the specified range. The results are shown Figure
B.4 COMPARISON OF RESULTS FROM DIFFERENT METHODS
We compared our method with spherical linear interpolation and the method of introducing noise for interpolation, using models separately trained LSUN Cat-256 and LSUNN Bedroom-22 256 datasets.
14

---

Published as a conference paper at ICLR 2024
Taput
Generated Interpolations
Input
Figure 11: Interpolation with natural images. By modifying A, our method can generate interpolated results with different image styles.
The results are displayed in Figure 13 and Figure [4 From the figures, it's clear that spherical linear interpolation introduces significant artifacts, while introducing noise for interpolation introduces extra information. In contrast, our method not only preserves the original image informations but also enhances the quality of images.
C EXPERIMENTS ON STABLE DIFFUSION
C.1 STABLE DIFFUSION
We extended our experiments on Stable Diffusion and compared it with other methods. Due to the differences the form of tx and (t) in Stable Diffusion, there have been significant changes in its latent variables. However, the challenges faced by different interpolation methods are simi- lar: spherical linear interpolation produces images with noticeable defects (Figure 2 Figure 25 while the method of introducing noise for interpolation introduces additional information (Figure T6
Figure 20p. Due to the highly unstructured latent space of the Stable Diffusion, it becomes chal- lenging to interpolate between two image samples, as depicted in Figure 15 Therefore, consider interpolating latent variables in the noisy image space, here we chose to interpolate the images when t 700.
C.2 EXPERIMENTAL RESULTS
We collected various images online to interpolate on Stable Diffuuson.. The results are shown be- low. To facilitate comparison with spherical linear interpolation results, we maintain the condition a/B sin (5 X) cos (5 >), and ensure that /a² + 32 + 72 = when computing a and B. Additionally, the boundary is set to 2.0, 7 E [0. v0.1], and 1.2*a/(a = 1.2 (12+ Users need determine the value of modify the style of interpolation results.
15

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
Figure 12: Interpolation with natural images. By modifying X, our method can generate interpolated results with different image styles.
Input
Generated Interpolations
Ingut
Figure 13: Comparison between spherical linear interpolation and our method. The top and left- most images represent the original images. The second row displays the results of spherical linear interpolation, while the third row shows the outcomes of our method.
16

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
Figure 14: Comparison between the method of introducing noise for interpolation and our method. The top and leftmost images show the original images. The second and third rows display the interpolation results obtained by directly introducing noise. The fourth row illustrates the outcomes of our method.
Input
Generated Interpolations
Input
Figure 15: Spherical linear interpolation results when the images are encoded into the noise space.
17

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
Figure 16: Interpolation results with Stable Diffusion (Introducing Noise) (1/5).
Inpul
Generaled Interpolations
Input
Figure 17: Interpolation results with Stable Diffusion (Introducing Noise) (2/5).
18

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
Figure 18: Interpolation results with Stable Diffusion (Introducing Noise) (3/5).
Input
Generated Interpolations
Input
BEN
OOO IO
100m
Figure 19: Interpolation results with Stable Diffusion (Introducing Noise) (4/5).
19

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
9
Figure 20: Interpolation results with Stable Diffusion (Introducing Noise) (5/5).
Input
Generated Interpolations
Inpur
Figure 21: Interpolation results with Stable Diffusion (Spherical Linear Interpolation) (1/5).
20

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
Figure 22: Interpolation results with Stable Diffusion (Spherical Linear Interpolation) (2/5).
Input
Generated Interpolations
Input
Figure 23: Interpolation results with Stable Diffusion (Spherical Linear Interpolation) (3/5).
21

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Input
Figure 24: Interpolation results with Stable Diffusion (Spherical Linear Interpolation) (4/5).
Input
Generated Interpolations
Inpul
Figure 25: Interpolation results with Stable Diffusion (Spherical Linear Interpolation) (5/5).
22

---

Published as a conference paper at ICLR 2024
Inpul
Generated Interpolations
Input
Figure 26: Interpolation results with Stable Diffusion (NoiseDiffusion) (1/5).
Input
Generated Interpolations
Inpur
Figure 27: Interpolation results with Stable Diffusion (NoiseDiffusion) (2/5).
23

---

Published as a conference paper at ICLR 2024
Tnput
Generated Interpalations
Tnput
Figure 28: Interpolation results with Stable Diffusion (NoiseDiffusson) (3/5).
Input
Generated Interpolations
Input
Figure 29: Interpolation results with Stable Diffusion (NoiseDiffusion) (4/5).
24

---

Published as a conference paper at ICLR 2024
Input
Generated Interpolations
Iaput
Figure 30: Interpolation results with Stable Diffusion (NoiseDiifuusio) (5/5).
25


Reviews:
[{"review_id": "bqxz63h7zj", "reviewer": "Reviewer_h8PS", "summary": "This paper addresses image interpolation based on diffusion models; it shows the limitation of the linear spherical interpolation (Song 2020), and then proposes a novel method. Specifically, the proposed method integrates the linear spherical interpolation and the linear spherical interpolation combined with SDEdit; it denoises the linear combination of the latent variables, the original images, and Gaussian noise. The performance of the proposed method is compared with those of the linear spherical interpolation and the linear spherical interpolation combined with SDEdit on bedroom and cat images.", "questions": "I would be happy to receive your feedback to the comments on Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "First of all, the experimental results qualitatively demonstrate that the proposed method works better than both the linear spherical interpolation and the linear spherical interpolation combined with SDEdit. Second, the proposed method generalizes the linear spherical interpolation and the linear spherical interpolation combined with SDEdit so that it includes both the methods in special cases. Third, the theoretical backgrounds are interesting; the limitation of the linear spherical interpolation is explained by Theorem 1, and the proposed method is supported by Theorem 2.", "weaknesses": "First, the proposed method has a number of empirical parameters (alpha, beta, gamma, mu, nu, and boundary). It would be difficult to appropriately set those parameters in practice, although the effects of each parameter is experimentally demonstrated. Second, the experimental evaluation is limited: only qualitative evaluation on small number of images. Third, the presentation of this paper could be improved.\nIn my opinion, Introduction is too short and the number of references are too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dsMwHasVZz", "reviewer": "Reviewer_YU98", "summary": "The paper explores and discusses the limitations of linear spherical interpolation in the context of image interpolation, particularly when applied to images not generated by diffusion models. It investigates how inappropriate noise introduction and estimation (overestimation and underestimation of noise level) affects image quality, leading to artifacts and loss of original style in the interpolated images. The study aims to address these limitations by proposing a novel interpolation approach that strategically imposes boundary constraints (i.e. clipping) on the introduced noise (at inversion steps), aligning with the hypersphere centered around the Gaussian distribution. This approach aims to mitigate noise artifacts and improve image quality, preserving the essential characteristics of the original image while efficiently removing artifacts. Additionally, the paper explores the integration of different interpolation methods, such as linear spherical interpolation and SDEdit, to overcome the identified limitations and achieve satisfactory results in both training and natural image domains.", "questions": "INITIAL REVIEW:\n\n• Correct the typo on page 2 after equation (1) where $g$ should be replaced with $\\sigma\".\n\n    • The rationale behind introducing the methods in section 2.1 needs clarification, establishing their relevance to the current work. Additionally, references for the methods introduced in this section are missing.\n\n    • Define $\\bm{\\epsilon}_t$ (as observed in equation (4)) since it hasn't been explicitly defined in the paper.\n\n    • Provide further clarification on whether \"Gaussian noise\" is i.i.d. (independent and identically distributed) or simply refers to AWGN (Additive White Gaussian Noise).\n\n    • Maintain consistency in the use of spaces after full stops and commas.\n\n-------------------------------------------------------------------------------------------------------------------------------------\n\nPOST-REBUTTAL ASSESSMENT:\nThe questions have been addressed in the revised paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "• Identifies and addresses limitations in current linear spherical interpolation techniques, particularly in handling natural images (not generated by diffusion models).\n\n    • Offers an approach to enhance image interpolation quality and address noise-related issues (i.e. noise artifacts and loss of style dues to a mismatch between the actual noise strength and its estimation) through and imposing a clipping constraint on the noise components.", "weaknesses": "INITIAL REVIEW:\n\n• The paper's contribution heavily relies on specific parameter selection (e.g., parameters in eq.(14), boundary values in section 3.5, and the interpolation coefficient $\\lambda$ in slerp), yet practical guidance for selecting these parameters remains unclear.\n\n    • Computational overhead and potential limitations of implementing the proposed method in practical scenarios lack detailed explanation. The experiments are conducted under a simplified scenario where the drift coefficient $\\mu(x_t,t)$ is set to zero.\n\n    • Further exploration is needed to assess the level of improvement and applicability in diverse domains or image types, as seen in Figure 6.\n\n    • The paper lacks explicit discussion on the generalizability of the method to various image sets.\n    • The previous works referenced (e.g., SDEdit) should be briefly described for reader convenience, without necessitating an external search for the referenced paper.\n\n    • There's a deficiency in the literature review concerning image generation based on diffusion probabilistic models, requiring more depth and exploration in this area.\n-------------------------------------------------------------------------------------------------------------------------------------\nPOST-REBUTTAL ASSESSMENT:\nThe concerns have been addressed in the revised paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ewwz9jSuEQ", "reviewer": "Reviewer_A19z", "summary": "A new approach to image interpolation with diffusion models is presented, where a new image is interpolated from two other images (in contrast to interpolating pixels within a single image).  While exisitng methods work reasonably well with images generated by a diffusion model, they do not work well with natural images not generated from the diffusion model.  Resulting interpolated images exhibit considerable artefacts.  Theory is introduced to explain the reason for the failure and suggest how to mitigate it.  A novel approach is presented to add appropriate noise to the natual images considered but at the same time ensure the interpolated image is faithful to the two images it is generated from.  A number of experiments are presented that demonstrate the improvement in interpolated image quality compared to the standard spherical interpolation approach.", "questions": "Could the authors propose a summary metric to qualtify the performance of their approach over a large test-data set?  At present results are presented for just a handful of test images.  While is seems very likely the improved performance of the proposed method will generalize to other data it would be more compelling if the authors could demonstrate this quantitatively.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is clear and well written.  The background and existing approaches are clearly explained, highlighting the tradeoff between realsim (i.e. avoiding artefacts) and faithfulness (i.e. ensuring consistency with the source images).  \n\nThe reason for the existing failure for natural images is related the mismatched noise levels, which arises due to the concentration of probability in high dimensions.  A numerical experiment (Fig. 3) is performed to support this hypothesis and Theorem 1 is presented to provide further support.  I believe Theorem 1 is a fairly well-known result.  Nevertheless, a proof is presented in Appendix A (I initially did not appreciate a proof was provided; in the main body of the article the reader should be pointed to Appendix A for the proof).\n\nFurther theory is presented in Theorem 2 to guide a solution to the problem (again, in the main body the reader should be pointed to the proof of Theorem 2 in Appendix B).  Based on the provided theory a novel diffusion-based image interpolation approach is presented, combining an SDEdit-like stage, followed by interpolation in latent space, followed by denoising with the diffusion model.  The interpolation approach introduces a number of free parameters to address contrast issues and trade-off consistence with each source image.  Standard spherical interpolation and linear interpolation combined with SDEdit are recovered as special cases for specific parameter choices.\n\nThe resulting interpolation method is simple but appears to work well.  I expect it to be of considerable practical use.", "weaknesses": "While a number of supporting experiments are presented, these could be more compelling.  Experiments are performed to demonstrate the impact of different parameter choices, which are helpful.  In terms of overall interpolation quality in comparison to existing techniuqes, a number of examples a presented where the proposed method is demonstrated to be visually superior to existing approaches.  While further examples are presented in the appendices, in total just a handful of images are considered.  Given the artefacts that arise with existing techniques, I don't doubt that the proposed method is superior.  However, to further support this it would be useful to have some metric to summarize performance over a large test data-set.  I appreciate this may not be trivial but I wonder if the authors have any thoughts on how they could assess performance over a large test-set?\n\nBoundary control methods are introduce in Section 4.4 but are not explained.  The appendix is referenced but that only contains an additional figure of results.  The boundry control methods should be discussed somewhere.  In general Appendix B feel incomplete.  While I appreciate papers should stand on their own, without Appendices, if that is the case then further detail needs to be provided in the main body.\n\nThere seem to be a number of minor typos throughout where spaces a missing, typically following a period or comma.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)