SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model

## Abstract

Abstract We present SoundMorpher, a sound morphing method that generates perceptually uniform morphing trajectories using a diffusion model. Traditional sound morphing methods models the intractable relationship between morph factor and perception of the stimuli for resulting sounds under a linear assumption, which oversimplifies the complex nature of sound perception and limits their morph quality. In contrast, SoundMorpher explores an explicit proportional mapping between the morph factor and the perceptual stimuli of morphed sounds based on Mel-spectrogram. This approach enables smoother transitions between intermediate sounds and ensures perceptually consistent transformations, which can be easily extended to diverse sound morphing tasks. Furthermore, we present a set of quantitative metrics to comprehensively assess sound morphing systems based on three objective criteria, namely, correspondence, perceptual intermediateness, and smoothness. We provide extensive experiments to demonstrate the effectiveness and versatility of SoundMorpher in real-world scenarios, highlighting its potential impact on various applications such as creative music composition, film post-production and interactive audio technologies. See SoundMorpher-demo for the listening demonstration.

## 1 Introduction

Sound morphing is a technique to create a seamless transformation between multiple sound recordings. The goal is to produce perceptual intermediate sounds that gradually change from one sound to another.
Sound morphing has a wide range of applications, including music compositions, synthesizers, psychoacoustic experiments to study timbre spaces (Caetano & Rodet, ; Hyrkas, ), and practical applications such as film post-production, AR or VR interactive games, and adaptive audio content in video games (Qamar et al., ; Siddiq, ).

Traditional sound morphing methods used the interpolation principle of sound synthesis technique, which relies on interpolating the parameters of a sinusoidal model (Tellman et al., ; Osaka, ; Williams et al., ).
Others make use of digital signal processing techniques to explore high-level audio features in the time-frequency domain to achieve more effective and continuous morphing (Williams et al., ; Brookes & Williams, ; Caetano & Rodet, ; ; Roma et al., ; Caetano, ).
However, these methods are limited to applications such as producing inharmonic and noisy environmental sounds (Gupta et al., ; Kamath et al., ).
Despite the increasing interest in applying machine learning to sound generation, there has only been limited exploration in sound morphing.
Recent approaches (Zou et al., ; Gupta et al., ; Kim et al., ; Kamath et al., ) have shown their superior effectiveness compared to traditional methods in various scenarios. However, we observed several critical limitations of those existing methods.
Firstly, they are primarily designed for static or cyclostationary morphing (see Sec. 3.1), limiting their applicability to dynamic sound transformations. Secondly, these approaches often lack sufficient quantitative evaluation, limiting further analysis of their effectiveness. Thirdly, they require training on task-specific datasets, which limits their application in different scenarios. Most importantly,
they typically assume a linear relationship between morphing factors and sound perception, and achieve smooth morphing by gradually changing the morph factor. This assumption oversimplifies the complex nature of sound perception, as gradually changing morph factors does not inherently result in smooth perceptual transitions.
To this end, our goal is to develop a method that achieves perceptually coherent morphing, ensuring seamless and natural sound transition.

In this paper, we introduce SoundMorpher, a sound morphing method that produces perceptually smooth and intermediate morphing, comprising the following key contributions.

- •
SoundMopher is the first open-world sound morphing method based on a pre-trained diffusion model, which integrates typical morph tasks such as static, dynamic and cyclostationary morphing.
Unlike prior works (Kim et al., ; Gupta et al., ), SoundMorpher can be broadly applied to various real-world tasks without requiring extensive retraining.
- •
We propose the sound perceptual distance proportion (SPDP), which explicitly connects morph factors and perceptual stimuli of morphed results. This allows SoundMorpher to produce morphing paths with a uniform change in perceptual stimuli, achieving more seamless perceptual transitions compared to existing methods (Kamath et al., ).
- •
We adapt a set of comprehensive quantitative metrics according to criteria proposed by Caetano & Osaka () for evaluation, addressing the lack of quantitative assessment for sound morphing systems (Caetano, ; Zou et al., ; Caetano & Rodet, ) and may offer insights for analyzing and comparing future sound morphing methods.
- •
We provide extensive experiments to demonstrate that SoundMorpher can be effectively applied to several potential applications in broader real-world scenarios, including musical instrument timbre morphing, music morphing and environmental sound morphing.

## 2 Related Work

In this section, we first present a detailed review of related works on sound morphing task. Then, we also briefly introduce tasks that are similar to sound morphing and clarify the differences.

Sound morphing.
Traditional sound morphing methods rely on interpolating parameters of a sinusoidal sound synthesis model (Tellman et al., ; Osaka, ; Williams et al., ; Primavera et al., ).
To achieve more effective and continuous morphing, Williams et al. (); Brookes & Williams (); Caetano & Rodet (; ); Roma et al. (); Caetano () target on exploring perceptual spectral domain audio features by digital signal processing techniques, such as MFCCs, spectral envelope, etc..
Others such as Kazazis et al. () involve a hybrid approach that extracts audio descriptors to morph accordingly and interpolate between the spectrotemporal fine structures of two endpoints according to morph factors.
Machine learning sound morphing methods offer advantages such as high morphing quality by leveraging semantic representation interpolation within a model instead of traditional audio feature interpolation.
Zou et al. () proposes a non-parallel many-to-one static timbre morphing framework that integrates and fine-tunes the machine learning technique (i.e., DDSP-autoencoder (Engel et al., )) with spectral feature interpolation (Caetano & Rodet, ).
Kim et al. () targets synthesizing music corresponding to a note sequence and timbre, which uses non-linear instrument embedding as timbre control parameters under a pretrained WaveNet (Engel et al., ) to achieve timbre morphing between instruments. Luo et al. () learns latent distributions of VAEs to disentangle representations for pitch and timbre of musical instrument sounds. Tan et al. () uses a GM-VAE to achieve style morphing to generate realistic piano performances in the audio domain following temporal style conditions for piano performances, which morphs the conditions such as onset roll and MIDI note into input audio. MorphGAN (Gupta et al., ) targets on audio texture morphing by interpolating within conditional parameters, and trained the model on a water-wind texture dataset.
A recent concurrent work by Kamath et al. () uses a pre-trained AudioLDM (Liu et al., ) to morph sound between two text prompts. In contrast, we focus on classical sound morphing, where the morphing process is performed directly between two given audios rather than between text prompts. A key advantage of our method is its ability to provide precise guidance during the morphing process, as the target audio delivers exact information on how the source sound should evolve—something that text prompts cannot always achieve, for example, morphing between two music compositions.

Synthesizer preset interpolation.
Synthesizer preset interpolation achieves sound morphing by developing models that compute interpolations within the domain of synthesis parameters for a black-box synthesizer (Le Vaillant & Dutoit, ; Dutoit et al., ; Le Vaillant & Dutoit, ). Unlike classical sound morphing, which perceptually blends two audio files into an intermediate sound, synthesizer preset interpolation treats the synthesizer as a non-differentiable black box, with presets composed of both numerical and categorical parameters. By smoothly interpolating between these presets, the task aims to achieve seamless morphing of synthesized sounds.

Text-to-audio editing.
Text-to-audio editing is the process of using text queries to edit audio.
With the success of diffusion models in image editing tasks, recent works target zero-shot audio editing with text instructions (Manor & Michaeli, ; Zhang et al., ; Lan et al., ) involving tasks such as inpainting, outpainting, timbre transfer, music genre transfer, or vocals removal.

Timbre transfer. Timbre transfer is a specific task that aims at converting the sound of a musical piece by one instrument (source) into the same piece played by another instrument (target). This concerns the task of converting a musical piece from one timbre to another while preserving the other music-related characteristics (Comanducci et al., ; Jain et al., ; Li et al., ).

Voice conversion and morphing.
Voice conversion (VC) involves modifying vocal characteristics of a source speech to match a target speaker, either by using target speeches or text (Li et al., ; Yao et al., ; Niu et al., ; Sheng et al., ). The primary objective of VC is to alter the vocal identity to closely resemble the target voice style, while preserving the linguistic content of the source speech. Voice morphing is a broader scope, focusing on blending or transforming one voice into another. This often involves creating an intermediate voice that incorporates characteristics of both source and target voices, allowing for gradual transitions between them (Sheng et al., ).

## 3 Preliminaries

### 3.1 Sound Morphing

Sound morphing aims to produce intermediate sounds as different combinations of model source sound $\hat{S}_{1}$ and target sound $\hat{S}_{2}$ (Caetano & Rodet, ; ),
which can be formulated as

$$ $M(\alpha,t)=(1-\alpha(t))\hat{S}_{1}+\alpha(t)\hat{S}_{2}$ (1) $$

Each step is characterized by one value of a single parameter $\alpha$, the so-called morph factor, which ranges between 0 0 and $1$, where $\alpha=0$ and $\alpha=1$ produce resynthesized source and target sounds, respectively.
Due to the intrinsic temporal nature of sounds, sound morphing usually involve three main types: dynamic morphing, where $\alpha$ gradually transfers from 0 0 to $1$ over time dimension (Kazazis et al., ),
static morphing, where a single morph factor $\alpha$ leads to an intermediate sound between source and target (Sethares & Bucklew, ), and cyclostationary morphing where several hybrid sounds are produced in different intermediate points (Slaney et al., ).

To solve the limitation on previous works that target on expensive perceptual evaluation only, Caetano & Osaka () proposes three objective criteria for sound morphing techniques:
(1) Correspondence. The morph is achieved by a description whose elements are intermediate between source and target sounds, highlighting semantic level transition;
(2) Intermediateness. The morphed objects should be perceived as intermediate between source and target sounds, evaluating perceptual level correlation;
(3) Smoothness. The morphed sounds should change gradually (i.e., ‘smoothly’) from source to target sounds, by the same amount of perception increment. Under the assumption of linear perceptual stimuli, adding the same factor should increase the same amount of perception.
In this study, we evaluate SoundMorpher according to the three criteria by a series of comprehensive objective quantitative metrics in addition to perceptual evaluation.

### 3.2 Latent Diffusion Model on Audio Generation

SoundMorpher utilizes a pretrained text-to-audio (TTA) latent diffusion model (LDM) (Rombach et al., ) to achieve sound morphing. This approach offers the advantage of performing various types of sound morphing without the need to train the entire model or use additional datasets. Specifically, we use AudioLDM2 (Liu et al., ), a multi-modality conditions to audio model. It employs a pre-trained variational autoencoder (VAE) (Kingma & Welling, ) to compress audio $x$ into a low-dimension latent space as VAE representations $z$. AudioLDM2 generates latent variables $z_{0}$ from a Gaussian noise $z_{T}$ given the condition $C$ and further reconstruct audio $\hat{x}$ from $z_{0}$ by VAE decoder and a vocoder (Kong et al., ).
AudioLDM2 uses an intermediate feature $Y$ as an abstraction of audio data $x$ to bridge the gap between conditions $C$ and audio $x$, named language of audio (LOA). The LOA feature is obtained by a AudioMAE (Huang et al., ; Tan et al., ) and a series of post-processing formulated as $Y=\mathcal{A}(x)$.
The generation function $\mathcal{G}(\cdot)$ is achieved by a LDM.
In the inference phase, AudioLDM2 approximates LOA feature by the given condition as $\hat{Y}=\mathcal{M}(C)$ using a fine-tuned GPT-2 model (Radford et al., ). Then generates audios conditioned on the estimated LOA feature $\hat{Y}$ and an extra text embedding $E_{T5}$ from a FLAN-T5 (Chung et al., ) with a LDM as $\hat{x}=\mathcal{G}(\hat{Y},E_{T5})$. We denote the conditional embeddings in AudioLDM2 as $E=\{\hat{Y},E_{T5}\}$, therefore, the generative process becomes $\hat{x}=\mathcal{G}(E)$.

Diffusion Models.
The LDM performs a forward diffusion process during training, which is defined as a Markov chain that gradually adds noise to the VAE representation $z_{0}$ over $T$ steps as $z_{t}=\sqrt{1-\beta_{t}}z_{t-1}+\sqrt{\beta_{t}}\epsilon_{t}$.
where $\epsilon_{t}\sim N(0,I)$ and noise schedule hyperparameter $\beta_{t}\in[0,1]$. Therefore, we can derive the distribution of $z_{t}$ given $z_{0}$ as $q(z_{t}|z_{0})=\sqrt{\gamma_{t}}z_{0}+\sqrt{1-\gamma_{t}}\epsilon_{t}$, where $\gamma_{t}=\prod_{t=1}^{t}1-\beta_{t}$.
The LDM learns a backward transition $\epsilon_{\theta}(z_{t},t)$ from the prior distribution $N(0,I)$ to the data distribution $z$, that predicts the added noise $\epsilon_{t}$ (Ho et al., ).
Following the objective function of denoising diffusion probabilistic models (DDPM) (Ho et al., ), the objective function for training AudioLDM2 is

$$ $\text{min}_{\theta}\mathcal{L}_{DPM}=\text{argmin}_{\theta}[\mathbb{E}_{z_{0}, E,t\sim\{1,...,T\}}||\epsilon_{\theta}(z_{t},E,t)-\epsilon_{t}||_{2}^{2}]$ (2) $$

To reduce computational demands on inference, AudioLDM2 uses
denoising diffusion explicit models (DDIM) (Song et al., ), which provides an alternative solution and enables significantly reduced sampling steps with high generation quality. The DDIM reverse diffusion process is

$$ $z_{t-1}=\sqrt{\gamma_{t-1}}(\frac{z_{t}-\sqrt{1-\gamma_{t}}\epsilon_{\theta}(z _{t},E,t))}{\sqrt{\gamma_{t}}})+\sqrt{1-\gamma_{t-1}-\sigma_{t}^{2}}\epsilon_{ \theta}(z_{t},E,t)+\sigma_{t}\epsilon_{t}$ (3) $$

We can revise a deterministic mapping between $z_{0}$ and its latent state $z_{T}$ once the model is trained (Dhariwal & Nichol, ; Yang et al., ) by the following equation

$$ $~{}\frac{z_{t+1}}{\sqrt{\gamma_{t+1}}}-\frac{z_{t}}{\sqrt{\gamma_{t}}}=(\sqrt{ \frac{1-\gamma_{t+1}}{\gamma_{t+1}}}-\sqrt{\frac{1-\gamma_{t}}{\gamma_{t}}}) \epsilon_{\theta}(z_{t},E,t)$ (4) $$

## 4 Method

Given a source and target audio pair $\{x^{(0)},x^{(1)}\}$,
sound morphing aims to generate intermediate sounds $x^{(\alpha(t))}$ between the audio pair given morph factors $\alpha\in[0,1]$. To account for the variation of $\alpha$ over time in Equation 1, we discretize the function $\alpha(t)$ where $t\in[0,T]$ into $N$ elements, resulting in a morphed sequence of sounds $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ based on $\{\alpha_{i}\}^{N}_{i=1}$.
According to the smoothness criteria proposed by Caetano & Osaka (), the desired sound morphing technique should have smooth linear perceptual stimuli when the morph factor $\alpha$ varies in the sequence $\{\alpha_{i}\}^{N}_{i=1}$.
Therefore, we define $p_{i}$ to represent the perceptual stimuli of the morphed audio $x^{(\alpha_{i})}$ given morph factor $\alpha_{i}$. However, the relationship $\mathcal{P(\cdot)}$ between morph factor $\alpha$ and perceptual stimuli $p$ is intractable. Our goal is to find a discrete morph factor sequence $\{\alpha_{i}\}^{N}_{i=1}$ such that for each transition, the perceptual stimuli difference $\Delta p$ is a constant value. Therefore, we formulate the problem as

$$ $p_{i+1}-p_{i}\equiv\mathcal{P}(x^{(\alpha_{i+1})})-\mathcal{P}(x^{(\alpha_{i}) })=\Delta p,~{}~{}i\in[1,...,N-1]~{}$ (5) $$

This formulation is a refined sound morphing problem where, rather than controlling morph factor $\alpha$, we control the constant perceptual stimuli difference $\Delta p$ to find the optimal trajectory with morph factors $\{\alpha_{i}\}^{N}_{i=1}$ that will achieve perceptually uniform sound morphing (^1^11See Appendix 7.1 for overall SoundMorpher pseudo algorithm pipeline and further implementation details.).

In Section 4.1 we introduce feature interpolation and model adaption with a pre-trained AudioLDM2. This method allows high-quality intermediate morph results to be obtained by controlling morph factor $\alpha$. To achieve perceptually uniform sound morphing as in Equation 5, we explore an explicit connection $\mathcal{P(\cdot)}$ between perceptual stimuli $p$ and morph factor $\alpha$ in Section 4.2. In Section 4.3, we provide extensions of our method on the different morphing methods discussed in Section 3.1 to show the advantages of perceptually uniform sound morphing.

### 4.1 Feature interpolation and model adaption

Interpolating optimized conditional embeddings.
We first introduce text-guided conditional embedding optimization strategy under a pre-trained AudioLDM2, which retrieves corresponding conditional embeddings $E$ of the given audio data. As mentioned in Section 3.2, AudioLDM2 accepts two conditional inputs: LOA feature $Y$ and text embedding $E_{T5}$. We denote $E=\{Y,E_{T5}\}$ as the overall conditional embedding inputs for AudioLDM2. The LOA feature $Y$ is an abstraction of audio data which is semantically structured, and $E_{T5}$ captures sentence-level of representations.
To retrieve corresponding conditional embeddings of the given audio data, we first obtain the latent variables $z_{0}^{(0)}$ and $z_{0}^{(1)}$ of audio $x^{(0)}$ and $x^{(1)}$ from the pre-trained VAE in AudioLDM2 pipeline. We initialize a simple common text prompt (e.g., ‘An audio clip of sound’) as a text guidance condition $C$ to obtain $E$ by GPT-2 encoder and FLAN-T5 encoder in AudioLDM2 pipeline, respectively, as $E^{(0)}$ and $E^{(1)}$. Instead of optimizing the model parameters, we freeze the model parameters and optimize the conditional embedding $E^{(0)}$ and $E^{(1)}$ by the denoising objective function in Equation 2

$$ $E^{(0)}=\text{arg min}_{E}\mathcal{L}_{DPM}(z_{0}^{(0)},E;\theta)\text{ and }E ^{(1)}=\text{arg min}_{E}\mathcal{L}_{DPM}(z_{0}^{(1)},E;\theta)$ (6) $$

The optimized conditional embeddings $E^{(0)}$ and $E^{(1)}$ fully encapsulate the abstract details of audios $x^{(0)}$ and $x^{(1)}$. Due to the semantically structured nature of the conditional embeddings, the conditional distributions $p_{\theta}(z|E^{(0)})$ and $p_{\theta}(z|E^{(1)})$ closely mirror the degree of audio variation between the audio pair. To explore the data distribution that conceptually intermediate between $z^{(0)}$ and $z^{(1)}$, we bridge these two distributions through linear interpolation. Specifically, we define the interpolated conditional distribution as $p_{\theta}(z|E^{(\alpha)}):=p_{\theta}(z|(1-\alpha)E^{(0)}+\alpha E^{(1)})$, where $\alpha\in[0,1]$.

Interpolating latent state.
The conditional embedding represents the conceptual abstract of audio data. However, we also wish to smoothly morph the content of the audio pair.
Following Song et al. () and Yang et al. (), we smoothly interpolate between $z_{0}^{(0)}$ and $z_{0}^{(1)}$ by spherical linear interpolation (slerp) to their starting noise $z_{T}^{(0)}$ and $z_{T}^{(1)}$ and further obtained the interpolated latent state $z_{T}^{(\alpha)}:=\frac{\text{sin}(1-\alpha)\omega}{\text{sin}\omega}z_{T}^{(0
)}+\frac{\text{sin}\alpha\omega}{\text{sin}\omega}z_{T}^{(1)}$, where $\omega=\text{arccos}(\frac{z_{T}^{(0)\intercal}z_{T}^{(1)}}{||z_{T}^{(0)}||||z
_{T}^{(1)}||})$. The denoised latent variable $z_{0}^{(\alpha)}$ is obtained by applying a diffusion denoising process on the interpolated starting noise $z_{T}^{(\alpha)}$ and conditioning on the interpolated conditional embedding $E^{(\alpha)}$. The final morphed audio result $x^{(\alpha)}$ is obtained from $z_{0}^{(\alpha)}$ by the VAE decoder and a vocoder.

Model adaption. Model adaptation helps to limit the degree of morphed variation by suppressing high-density regions that not related to the given inputs (Yang et al., ). We use LoRA (Hu et al., ) to inject a small amount of trainable parameters for efficient model adaptation. We fine-tune AudioLDM2 with LoRA trainable parameters using $z^{(0)}$ and $z^{(1)}$. See Appendix 7.2 for details.

### 4.2 perceptually uniform Sound Morphing

Sound perceptual distance proportion (SPDP). The relationship between morph factor $\alpha$ and perceptual stimuli $p$ is intractable.
Our goal is to establish an objective quantitative metric that links $p_{i}$ and $x^{(\alpha_{i})}$ as in Equation 5. This metric should satisfy two key conditions:
(1) the output $p$ should increase monotonically as $\alpha$ increases; (2) it should accurately represent perceptual differences between $x^{(\alpha)}$ and $\{x^{(0)},x^{(1)}\}$, ensuring a smooth transition through intermediate states.
Therefore, we propose the sound perceptual distance proportion between $x^{(\alpha)}$ and $\{x^{(0)},x^{(1)}\}$. We define $p_{i}\in\mathbb{R}^{2}$ as a 2D vector to represent the perceptual proximity of $x^{(\alpha_{i})}$ to both $x^{(0)}$ and $x^{(1)}$. Instead of extracting numerous audio features through traditional signal processing techniques, we use Mel-scaled spectrogram to capture perceptual and semantic information on audio. Mel-spectrogram (Tzanetakis & Cook, ) provides a pseudo-3D representation of audio signals, with one axis representing time and the other representing frequency on the Mel scale (Stevens et al., ), while the values denote the magnitude of each frequency at specific time points. The advantage of using Mel-spectrogram lies in the Mel filter banks, which map frequencies to equal pitch distances that correspond to how humans perceive sound (Sturm, ; Müller, ).
Denoting $x_{mel}^{(\alpha_{i})}$ as the Mel-spectrogram of audio $x^{(\alpha_{i})}$, the SPDP $p_{i}$ between two endpoint audios $x^{(0)}$ and $x^{(1)}$ given $\alpha_{i}$ is defined as

$$ $p_{i}=[\frac{||x_{mel}^{(\alpha_{i})}-x_{mel}^{(0)}||_{2}}{||x_{mel}^{(\alpha_ {i})}-x_{mel}^{(0)}||_{2}+||x_{mel}^{(\alpha_{i})}-x_{mel}^{(1)}||_{2}},\frac{ ||x_{mel}^{(\alpha_{i})}-x_{mel}^{(1)}||_{2}}{||x_{mel}^{(\alpha_{i})}-x_{mel} ^{(0)}||_{2}+||x_{mel}^{(\alpha_{i})}-x_{mel}^{(1)}||_{2}}]$ (7) $$

Binary search with constant SPDP increment.
To produce a perceptually smooth morphing trajectory with a constant perceptual stimuli increment, we use binary search to seek the corresponding $\{\alpha_{i}\}^{N}_{i=1}$ based on a constant
$\Delta p$. The target SPDP sequence $\{p_{i}\}_{i=1}^{N}$ is obtained by an interpolation $p_{i}=(1-\frac{i-1}{N-1})p^{(0)}+\frac{i-1}{N-1}p^{(1)}$,
where the two endpoints are $p^{(0)}=[0,1]^{T}$ and $p^{(1)}=[1,0]^{T}$. See Algorithm 2 in Appendix 7.3 for detail pseudo algorithm.

### 4.3 Controllable Sound Morphing with Discrete α 𝛼 \alpha italic_α Series

By controlling the discrete morph factor sequence $\{\alpha_{i}\}^{N}_{i=1}$ to produce a morphed series $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ with constant $\Delta p$, we achieve three typical morphing methods as follows.

Static morphing. To achieve controllable static morphing, we control the target SPDP point $p$,
which represents how the desired output perceptually intermediate between $x^{(0)}$ and $x^{(1)}$. We find the corresponding $\alpha$ value by the binary search with the target $p$ and further obtain a morphed result ${x^{(\alpha)}}$. Pseudocodes for static morphing are in Algorithm 3.

Cyclostationary morphing. To produce $N$ perceptually uniform hybrid sounds between $x^{(0)}$ and $x^{(1)}$, we first obtain $N$ uniform interpolated SPDP points $\{p_{i}\}^{N}_{i=1}$.
Then we find corresponding morph factors $\{\alpha_{i}\}_{i=1}^{N}$ and further obtain $N$ morphed results $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ as in Algorithm 4.

Dynamic morphing.
Dynamic morphing performs sound morphing over time, but one challenge is that if the morphing path fails to ensure perceptual intermediateness and content correspondence, the resulting sounds may exhibit perceptual discontinuities or unnatural intermediate stages. As in Algorithm 5, we obtain $N$ interpolated target SPDP points $\{p_{i}\}^{N}_{i=1}$ with $\Delta p$. The corresponding morph factors $\{\alpha_{i}\}_{i=1}^{N}$ are determined by binary search with the target SPDP points. Each morphed result $x^{(\alpha_{i})}$ contributes a segment of duration $\frac{T}{N}$, producing an audio segment $\tilde{x}^{(\alpha_{i})}$ according to index $i$. The final audio signal is obtained by concatenating these morphed segments, resulting in

$$ $[x_{0},x_{1},...,x_{T}]=\text{concat}(\tilde{x}^{(0)},\tilde{x}^{(\alpha_{1})} ,...,\tilde{x}^{(1)})$ (8) $$

**Table 1: Timbral morphing for musical instruments compared to baseline on different instruments.**
| Group | Method | FAD $\downarrow$ | FD $\downarrow$ | CDPAMT $\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | $\mathcal{L}_{2}^{timbre}$ $\downarrow$ | CDPAM${}_{E}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Piano $\leftrightarrow$ Guitar | SMT | 24.73 | 102.57 | 1.170 | 0.116 $\pm$ 0.074 | 1.263 | 0.122 |
| Ours | 5.21 | 41.11 | 0.404 | 0.044 $\pm$ 0.020 | 0.466 | 0.132 |  |
| Harp $\leftrightarrow$ Kalimaba | SMT | 13.46 | 88.89 | 1.495 | 0.150 $\pm$ 0.117 | 1.355 | 0.182 |
| Ours | 4.67 | 37.92 | 0.768 | 0.076 $\pm$ 0.089 | 0.462 | 0.159 |  |
| Taiko $\leftrightarrow$ Hihat | SMT | 8.51 | 131.57 | 2.339 | 0.234 $\pm$ 0.332 | 1.584 | 0.732 |
| Ours | 3.32 | 47.59 | 1.314 | 0.131 $\pm$ 0.058 | 0.359 | 0.102 |  |
| Piano $\leftrightarrow$ Violin | SMT | 21.38 | 90.63 | 1.902 | 0.190 $\pm$ 0.069 | 0.558 | 0.217 |
| Ours | 3.42 | 20.14 | 0.782 | 0.078 $\pm$ 0.020 | 0.415 | 0.085 |  |
| Piano $\leftrightarrow$ Organ | SMT | 21.36 | 63.26 | 1.291 | 0.129 $\pm$ 0.074 | 1.106 | 0.097 |
| Ours | 3.29 | 19.73 | 0.233 | 0.023 $\pm$ 0.010 | 0.423 | 0.097 |  |

## 5 Experiment

In this section, we showcase three applications of SoundMorpher in real-world scenarios: Timbral morphing for musical instruments, Environmental sound morphing, and Music morphing.

### 5.1 Evaluation Metric

We verify SoundMorpher according to the criteria mentioned in Section 3.1.
Correspondence. We design a metric that computes absolute error for the mid-point MFCCs proportion, namely MFCCsE, for description correspondence (see Appendix 8.2 for detail). We use Fréchet audio distance (FAD) (Kilgour et al., ) and Fréchet distance (FD) (Eiter & Mannila, ) between morphed audios and sourced audios to verify semantic similarity and morphed audio quality (see Appendix 8.3 for detail).
Intermediateness.
We use total CDPAM (Manocha et al., ) by CDPAM${}_{T}=\sum^{N-1}_{i=1}\text{CDPAM}(x^{(\alpha_{i})},x^{(\alpha_{i+1})})$ for morph sequence to reflect direct perceptual intermediateness. A smaller CDPAMT indicates the morphing sequence exhibits higher perceptual intermediate similarity between consecutive sounds, suggesting intermediate consistency.
Smoothness. We calculate the mean and standard deviation of CDPAM along with the morphing path to validate smoothness, as CDPAMmean±std = CDPAM${}_{mean}\pm\text{CDPAM}_{std}$, where $\text{CDPAM}_{mean}=\frac{1}{N-1}\sum^{N-1}_{i=1}\text{CDPAM}(x^{(\alpha_{i})}
,x^{(\alpha_{i+1})})$, and $\text{CDPAM}_{std}=\sqrt{\frac{1}{N-1}\sum^{N-1}_{i=1}(\text{CDPAM}(x^{(\alpha
_{i})},x^{(\alpha_{i+1})})-\text{CDPAM}_{mean})^{2}}$.
In timbre space study, we define timbral distance by $\mathcal{L}_{2}^{timbre}=\frac{1}{N-1}\sum_{i=1}^{N-1}||q^{(\alpha_{i+1})}-q^{
(\alpha_{i})}||_{2}$, where $q^{(\alpha_{i})}$ represents the corresponding timber point of $x^{(\alpha_{i})}$ in timbre space (McAdams et al., ). Lastly, to verify reconstruction perceptual correspondence, we denote CDPAME that calculate CDPAM between $\{x^{(0)},x^{(1)}\}$ and $\{\hat{x}^{(0)},\hat{x}^{(1)}\}$, where $\hat{x}$ represents resynthesized end points when $\alpha=0$ and $\alpha=1$.

### 5.2 Timbral morphing for musical instruments

Sound morphing can allow timbral morphing between the sound of two known musical instruments, creating sounds from unknown parts of the timbre space (McAdams, ; McAdams & Goodchild, ). Timbral morphing for musical instruments involves transitioning between timbres of two different musical instruments to create a new sound. This new sound could possess characteristics of both original timbres as well as new timbral qualities between them, which usually applied to creative arts.
In this experiment, we perform timbral morphing for isolated musical instruments given two recordings of the same musical composition played by different music instruments.

Dataset.
To ensure high quality of paired composition musical instrument on timbral study, we selected 22 paired musical instrument composition samples from demonstration pages of musical timbre transform projects, MusicMagus (Zhang et al., ) and Timbrer (Kemppinen P., ). The paired samples have durations varies from 5s to 10s, with 16.0kHz and 44.01kHz, which involve 5 groups of instrument pairs: 2 paired samples of piano-violin; 10 paired samples of piano-guitar; 1 paired sample of taiko-hihat; 1 paired sample of piano-organ, and 8 paired samples of harp-kalimaba.

Figure: (a) SMT
Refer to caption: extracted/5897536/Figure/piano-organ-smt.png

Baseline.
We compare our method with Sound Morphing Toolbox (SMT) (Caetano, ), which is a set of Matlab functions targeting on musical instrument morphing that implement a sound morphing algorithm for isolated musical instrument sounds. Since SMT does not offer guidance for selecting perceptually uniform morph factors, we uniformly interpolate 11 morph factors in $[0,1]$.

Results and analysis.
We set $N=11$ for SoundMorpher with an initial prompt ‘a music composition by {instrument}’. The comparison of our method and the baseline on timbral morphing is in Table 1. Overall, SoundMorpher demonstrates superior morphing quality compared to STM across various metrics, including audio quality, intermediateness, smoothness, and resynthesis quality
(^2^22Since we perform timbral morphing within the same music composition, MFCCsE may not a suitable metric under the same musical content. In contrast, we focus on evaluating smoothness and intermediateness.), when applied to different types of musical instrument timbre morphing. Notably, STM fails in Taiko-Hihat timbral morphing due to significant high reconstruction perceptual error. In contrast, SoundMorpher maintains robustness across different types of musical instruments, making it a more flexible and efficient solution for timbral morphing applications on different types of musical instruments.
Figure. 1 provides a visualization of normalized timbre space, illustrating morphing trajectories generated by SMT and SoundMorpher. The timbre space is defined by three important timbral features: Log-Attack Time, Spectral Centroid, and Spectral Flux (McAdams et al., ; McAdams, ). The SMT trajectory shows distinct steps, indicating that the transitions between each intermediate sound are relatively abrupt. The spacing between the blue points suggests that each step represents a significant change in timbre, which may result in a less smooth perceptual transition between two musical instruments. In contrast, the trajectory produced by SoundMopher demonstrates a smoother curve. The points are more closely spaced, indicating more gradual changes between each intermediate timbre. This suggests that SoundMopher achieves a more continuous and natural-sounding morphing process, with each step being a smaller, more refined adjustment compared to SMT. Figure 7 and Figure 8 in the appendix provides additional visualization for this experiment.

### 5.3 Environmental Sound Morphing

Environmental sounds are used in video game production to provide a sense of presence within a scene. For example, in video, AR and VR games, sound morphing could enhance user immersion by adapting audio cues to specific visual and interactive contexts. This means that it could be useful to morph between sonic locations, e.g., a city and a park, or between sound effects, e.g., different animal sounds to represent fantasy creatures. In this experiment, we perform cyclostationary morphing with $N=5$ by SoundMorpher across various types of environmental sounds.

Dataset. We use ESC50 (Piczak, ) which consists of 5-second recordings organized into 50 semantic classes which loosely arranged into 5 major categories. We randomly select 4 major categories of scenarios to verify our method, including (1) Dog-Cat (animals voices), (2) Laughing-Crying baby (human sounds), (3) Church bells-Clock alarm (urban noise-interior sound), (4) Door knock-clapping (interior sounds-human sounds). Each category of scenarios contains 25 randomly selected audio pairs, thereby, 100 randomly paired samples in total.

**Table 2: Environmental sound morphing with different types of environmental sounds.**
| Category | FAD${}_{\text{category}}$ | MFCCsE$\downarrow$ | FAD$\downarrow$ | FD$\downarrow$ | CDPAMT$\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | CDPAME$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Dog $\leftrightarrow$ Cat | 26.08 | 0.081 | 17.77 | 73.92 | 1.293 | 0.323 $\pm$ 0.160 | 0.236 |
| Laughing $\leftrightarrow$ Crying baby | 10.39 | 0.044 | 9.35 | 65.98 | 0.855 | 0.214 $\pm$ 0.077 | 0.289 |
| Church bells $\leftrightarrow$ Clock alarm | 68.29 | 0.058 | 22.89 | 75.77 | 2.205 | 0.551 $\pm$ 0.299 | 0.312 |
| Door knock $\leftrightarrow$ Clapping | 21.36 | 0.083 | 10.85 | 76.35 | 1.594 | 0.428 $\pm$ 0.220 | 0.321 |

Results and analysis.
In this experiment, we use initial text prompt as ‘a sound clip of {sound class}’.
Table 2 presents the results of applying SoundMorpher to various categories of environmental sounds. To quantify the semantic gap between sound scence classes, we calculate FAD between them as FAD${}_{\text{category}}$. The results demonstrate SoundMorpher is capable of effectively morphing a wide range of environmental sounds. However, environmental sounds with a large semantic gap between categories can negatively impact the morphing quality. Additionally, we observe that the quantitative metrics for morphing quality and reconstruction perceptual errors in this experiment are higher than those for the timbre morphing task. One reason is the inherent complexity of environmental sounds, which often involve intricate physical events with significant temporal structure differences and background noises, making them more challenging to morph compared to musical data.
Figure 9 in appendix provides spectrogram visualizations on environmental sound morphing.

### 5.4 Music Morphing

Film or game post-production often requires blending or fading between music tracks to seamlessly transition background music in between scenes. Music morphing transitions between two music compositions without cross fading, that is, each moment of the morphed music would be a single composition with elements that are perceptually in between both source and target music, rather than simply blending the two together. Different from timbral morphing, music morphing could ideally be accomplished with compositions from different genres and mixed musical instruments. In this experiment, we use SoundMorpher to perform dynamic morphing on music with $N=15$.

Dataset. In this experiment, we randomly selected 50 sample pairs from 20 musical samples available on AudioLDM2 (Liu et al., ) demonstration page. These 10-second music compositions that span different genres and feature both single or mixed musical instrument arrangements.

Figure: Figure 2: Visualization of dynamic morphed music with $N=15$, source music and target music.
Refer to caption: extracted/5897536/Figure/dynamic_morph_new2.png

Results and analysis.
In this experiment, we select an initial text prompt as ‘a sound clip of music composition’ to perform dynamic morphing. Even though this experiment contains morph complex music compositions with different music genres and music instruments, Table 3 shows our method still superiors on perceptual smoothly transiting source music to the target music and ensures correspondence, intermediateness and smoothness.
Figure 2 provides strong visual evidence that the dynamic morphing method effectively transitions from the source to the target music while maintaining perceptual smoothness, correspondence, and intermediate transformations. The spectrogram illustrates that the morphed music transitions gradually, maintaining smooth spectral changes over time, which suggests the method successfully morphs the source into the target music.

**Table 3: Music morphing experimental results & ablation study for sound perceptual features, where N represents the number of components of PCA for reducing dimension of Mel-spectrogram.**
| Feature | MFCCsE $\downarrow$ | FAD $\downarrow$ | FD $\downarrow$ | CDPAMT $\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | CDPAME $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Reduced Mel-Spec. (N=2) | 0.187 | 10.31 | 58.57 | 0.793 | 0.056 $\pm$ 0.075 | 0.182 |
| Reduced Mel-Spec. (N=3) | 0.151 | 10.76 | 59.39 | 0.779 | 0.055 $\pm$ 0.079 | 0.151 |
| Mel-Spec. | 0.056 | 9.85 | 56.09 | 0.847 | 0.068 $\pm$ 0.045 | 0.178 |
| MFCCs | 0.053 | 10.11 | 57.38 | 0.987 | 0.071 $\pm$ 0.050 | 0.156 |
| Spectral contras | 0.066 | 10.54 | 58.44 | 0.863 | 0.061 $\pm$ 0.071 | 0.155 |

**Table 4: Mean opinion score study on environmental sound morphing and music morphing task.**
| Task | Correspondence$\uparrow$ | Intermediateness$\uparrow$ | Smoothness$\uparrow$ | Overall $\uparrow$ |
| --- | --- | --- | --- | --- |
| Environmental sound morphing | 3.78 $\pm$ 0.31 | 3.67 $\pm$ 0.40 | 3.57 $\pm$ 0.47 | 3.67 $\pm$ 0.39 |
| Music morphing | 3.81 $\pm$ 0.39 | 3.55 $\pm$ 0.51 | 3.49 $\pm$ 0.48 | 3.62 $\pm$ 0.46 |

### 5.5 Discussion

Mean opinion score (MOS) study.
We conducted a MOS study as a subjective evaluation for the correspondence, intermediateness, and smoothness of morphed results from SoundMorpher. The study involved 21 volunteers, and detailed methodology is in Appendix 9. As shown in Table 4, the results suggest that SoundMorpher is versatile, performing similarly well across both music and environmental sound morphing tasks, with no significant differences observed in the overall MOS. This consistency in scores indicates that SoundMorpher effectively handles the unique challenges posed by the distinct characteristics of music and environmental sounds, such as the continuous nature of music compared to the more segmented structure of environmental sounds. Despite the objective metric results showing clear differences between the two tasks, the human evaluation suggests that SoundMorpher remains robust across different sound types. One possible interpretation is that our objective metrics are more sensitive to variations in the measured aspects than participants.

**Table 5: Comparison with MorphFader**
| Method | CDPAMT | CDPAMmean±std | MFCCsE |
| --- | --- | --- | --- |
| MorphFader | 0.972 | 0.243 $\pm$ 0.139 | 0.065 |
| SoundMorpher | 0.935 | 0.226 $\pm$ 0.162 | 0.065 |

Model comparison. We compare SoundMorpher with a concurrent work, MorphFader (Kamath et al., ), based on the criteria outlined in Section 3.1. MorphFader relies on a pre-trained AudioLDM (Liu et al., ) and perform sound morphing by text instructions. We compare with 7 examples provided on the demonstration of MorphFader, that MorphFader uniformly samples $5$ morph factors in $[0,1]$, resulting in a morph path with $[0,0.25,0.5,0.75,1]$. In contrast, SoundMorpher finds $\alpha$ values according to constant $\Delta p$ with $5$ uniformly interpolate $p$ points by binary search. As in Table 5, SoundMorpher produces a smoother and perceptual intermediates morphing than MorphFader. See Appendix 10 for details.

Ablation study on sound perceptual features. We verified the perceptual feature in SPDP in music morphing task. We select alternative music information retrieval features (MIR) including MFCCs with 13 coefficients (Logan et al., ), and spectral contrast (Jiang et al., ). We use principal component analysis (PCA) to reduce the dimensionality of Mel-spectrogram to further capture variation of spectral content over time, which is referred to as reduced Mel-Spec. (Stevens et al., ; Casey et al., ; Jiang et al., ).
Table 3 shows performance comparisons of SoundMorpher with different features, SoundMorpher with Mel-spectrogram achieves better morphing quality in terms of correspondence and smoothness variation with smaller FAD, FD and CDPAMstd. While Mel-spectrogram yields higher CDPAMmean, CDPAMT and MFCCsE compared to reduced Mel-Spec. and MFCCs, the differences in metric values are not significant. However, the overall morph quality with Mel-spectrogram is consistently better than other features. This suggests Mel-spectrogram, as a pseudo-3D representation, provide more perceptual and semantic information, which contributes to improve morph quality compared to higher-level features.

**Table 6: Experiment and ablation study results on music morphing.**
| Init. text | T = 20 | T =100 | MFCCsE $\downarrow$ | FAD $\downarrow$ | FD $\downarrow$ | CDPAMT $\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | CDPAME $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Informative | $\checkmark$ |  | 0.047 | 10.21 | 56.62 | 1.213 | 0.086 $\pm$ 0.069 | 0.166 |
| Informative |  | $\checkmark$ | 0.044 | 10.21 | 56.13 | 1.077 | 0.084 $\pm$ 0.066 | 0.155 |
| Uninformative | $\checkmark$ |  | 0.057 | 10.37 | 55.89 | 1.036 | 0.074 $\pm$ 0.049 | 0.211 |
| Uninformative |  | $\checkmark$ | 0.056 | 9.85 | 56.09 | 0.847 | 0.068 $\pm$ 0.045 | 0.178 |

Uninformative v.s. informative initial text prompt. Complex audio usually cannot easily yield precise information to users. For example, it is a challenge for non-professional users to describe the genre of a music. We conduct an ablation study for initial text prompt on music morphing to verify effectiveness of text-guided conditional embedding optimization. We use a general initial text prompt, ‘a sound clip of music composition.’, as an uninformative initial prompt. And we use the given text prompts in AudioLDM2 (^3^33[https://audioldm.github.io/audioldm2/](https://audioldm.github.io/audioldm2/)) as informative inital prompts. As in Table 6, informative initial text prompts may help with resynthesis quality and further improves morph correspondence. Despite the improved resynthesis quality with informative initial text prompts, the results show a decline in morphing intermediateness and smoothness. One possible reason is the better resynthesis quality makes the resynthesis endpoints more distinct (i.e., larger semantic gap), which could lead to slight decline in intermediateness and smoothness. However, the performance difference on initial text prompts is not significant which illustrates effectiveness of conditional embedding optimization.

Inference steps.
In our experiment, we follow the configuration of Zhang et al. () and set DDIM steps to 100. To verify whether DDIM steps affect SoundMorpher performances, we compare with 20 DDIM steps in Table 6. Larger inference step seems to help for reconstruction quality and slightly imporves morph quality, however, performance differences between inference steps are not significant. This indicates SoundMorpher is robust for inference steps, and we extend this ablation study on environmental sound morphing task in Appendix 11.1. Thus, we suggest selecting a suitable DDIM step to trade-off overall binary search algorithm time-consuming and morph quality.

Limitations. The current implementation of SoundMorpher based on AudioLDM2 with 16.0kHz sampling rate, which may limit output audio quality. The conditional embeddings optimization only applies to sounds that can be produced by AudioLDM2. Sound examples that close to white noise, such as pure wind blowing used in MorphGAN (Gupta et al., ) are not easily generated by AudioLDM2, which makes the conditional embedding optimization produce low quality resynthesis sounds. We also observed that input sounds with a large semantic gap (e.g., Church bells-Clock alarm in Table 2) result in lower morphing quality. Furthermore, we observed when two audios exhibit significant temporal structure differences, such as environmental sounds, SoundMorpher may produce abrupt transitions, see Appendix 11.4 and Figure 6 for further details.

## 6 Conclusion

We propose SoundMorpher, a sound morphing method base on a pretrained diffusion model that produces perceptually uniform morphing trajectories. Unlike existing methods, we refined the sound morphing problem and explored an explicit connection between morph factor and perceptual stimuli of morphed results which offers better flexibility and higher morphing quality, making it adaptable to various morphing methods and real-world scenarios. We validate SoundMorpher by a series of objective quantitative metrics as well as mean opinion score study following criteria proposed by Caetano & Osaka (). These quantitative objective metrics may help to formalize future studies on sound morphing evaluation. Furthermore, we demonstrated that SoundMorpher can be applied to wide range of real-world applications in our experiments and conducted in-depth discussions. SoundMorpher also has the potential to achieve voice morphing, as its foundational model AudioLDM2 supports speech generation; however, we leave this exploration for future work.

## Appendix

## 7 SoundMorper implementation details

This section provides further details on SoundMorpher pipeline and implementations.

### 7.1 Overall pipeline of SoundMorpher

The overall pseudo pipeline for SoundMorpher is provided in Algorithm 1, where the overall pipeline of SoundMorpher contains three parts: (1) Conditional embedding optimization. (2) Model adaptation. (3) perceptually uniform binary search with constant SPDP increment.

Figure: Algorithm 1 Pipeline of SoundMorpher

### 7.2 Model adaptation with LoRA

In the task of image morphing, Yang et al. () indicate adapting the model to the input pair helps to limit the degree of morphed variation by suppressing high-density regions that are not related to the given images. Compared to vanilla fine-tuning approaches, LoRA has advantages in training efficiency with injecting limited trainable parameters. The model adaptation can be defined as

$$ $\text{min}_{\Delta\theta^{\prime}}\mathcal{L}_{DPM}(z_{0}^{(0)},E^{(0)};\theta +\Delta\theta^{\prime})+\text{min}_{\Delta\theta^{\prime}}\mathcal{L}_{DPM}(z_ {0}^{(1)},E^{(1)};\theta+\Delta\theta^{\prime})\text{ s.t. rank}(\Delta\theta^ {\prime})=r$ (9) $$

where $r$ represents LoRA rank.

Unconditional Bias Correction.
To achieve high text alignment during inference time, we use additional LoRA parameters $\Delta\theta_{0}$ with a small rank $r_{0}$ to perform bias correction, as

$$ $\text{min}_{\Delta\theta_{0}}\mathcal{L}_{DPM}(z_{0}^{(0)},\varnothing;\theta+ \Delta\theta_{0})+\text{min}_{\Delta\theta_{0}}\mathcal{L}_{DPM}(z_{0}^{(1)}, \varnothing;\theta+\Delta\theta_{0})\text{ s.t. rank}(\Delta\theta_{0})=r_{0}$ (10) $$

During inference, with $\theta^{\prime}=\theta+\Delta\theta^{\prime}$ and $\theta_{0}=\theta+\Delta\theta_{0}$, the noise prediction becomes

$$ $\hat{\epsilon}_{\theta}(z_{t},t,E):=w\epsilon_{\theta^{\prime}}(z_{t},t,E)+(1- w)\epsilon_{\theta_{0}}(z_{t},t,\varnothing).$ (11) $$

In our experiment, we set $r=4$ and $r_{0}=2$. Although Yang et al. () provide a heuristic suggestion for setting the LoRA rank value for image morphing task, however, we further investigate the relationship between LoRA rank $r$ and method performance in Table 8 in sound morphing task and discussion in Appendix 11.2.

### 7.3 perceptually uniform binary search with constant SPDP increment

Algorithm 2 provides detail pesudocodes for how to implement perceptually uniform binary search with constant SPDP increment. This pseudo algorithm includes two steps, firstly, compute constant SPDP increment according to interpolte point number $N$ and obtain $N$ target SPDP points as $\{p_{i}\}_{i=1}^{N}$. Secondly, perform binary search to find correponding morph factor $\alpha$ series $\{\alpha_{i}\}_{i=1}^{N}$ according to $\{p_{i}\}_{i=1}^{N}$.

Figure: Algorithm 2 Pseudo algorithm for perceptually uniform binary search with constant SPDP increment

Figure: Algorithm 3 Pseudo algorithm for static morphing

Figure: Algorithm 4 Pseudo algorithm for cyclostationary morphing

Figure: Algorithm 5 Pseudo algorithm for dynamic morphing

### 7.4 Sound Morphing with Discrete α 𝛼 \alpha italic_α Series

This section provide detail pseudo algorithm to perform different types of sound morphing methods:

- 1.
Static morphing: see Algorithm 3;
- 2.
Cyclostationary morphing: see Algorithm 4;
- 3.
Dynamic morphing: see Algorithm 5;

However, SoundMorpher is not restricted to the aforementioned sound morphing methods; it can be extended to other approaches, such as warped dynamic morphing, by concatenating the original dynamic morphing result with its reversed counterpart. We leave this exploration for future work.

### 7.5 Convex CFG scheduling for quality boosting

Background for Classifier-free Guidance (CFG).
Controllable generation can be achieved by using guidance at each sampling step in diffusion model. When a conditional and unconditional diffusion models are jointly trained, samples can be obtained by CFG (Ho & Salimans, ). In AudioLDM2 (Liu et al., ), the the conditional and unconditional noise esitimation becomes

$$ $\hat{\epsilon}_{\theta}(z_{t},t,E):=w\epsilon_{\theta}(z_{t},t,E)+(1-w) \epsilon_{\theta}(z_{t},t,\varnothing)$ (12) $$

where $w$ determines the guidance scale.

Convex CFG scheduling.
Following Yang et al. (), we involve a convex CFG scheduling in SoundMorpher to boost morphing quality which is defined as

$$ $w_{\alpha}=w_{max}-2(w_{max}-w_{min})|\alpha-0.5|$ (13) $$

where $w$ is the guidance scale, $\alpha$ is the morph factor.$w_{max}$ and $w_{min}$ are predefined maximum and minimum guidance scales. We discussed the impact of guidance scales in SoundMorpher in Appendix 11.3.

## 8 Experiment setup and implementation details

### 8.1 Experiment setup

We perform our experiment on one NVIDIA GeForce RTX 3090 with GPU 24GB memory. Following configuration in (Yang et al., ), we use AdamW optimizer (Loshchilov & Hutter, ) with learning rate 0.002 and 2500 steps to perform conditional embedding optimization. We use LoRA (Hu et al., ) with $r=4$ and $r_{0}=2$ to perform model adaptation, the LoRA is trained by Adam optimizer with 0.001 learning rate. We trained 150 steps for the LoRA injected trainable paramters for model adaptation and 15 steps for LoRA injected trainable parameters for unconditional bias correction. For convex CFG scheduling, we set $w_{max}=3.5$ and $w_{min}=1.5$ for timbral morphing and environmental sound morphing task, and $w_{max}=4$ and $w_{min}=1.5$ for music morphing task.

### 8.2 Implementation details for MFCCs E calculation

The goal of designing MFCCsE feature is to verify the correspondence of morphed samples as an objective metric.
Let $N$ be an odd integer, We define a series of perceptually uniform morphing results $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ by the proposed method with source and target audio $x^{(0)}$ and $x^{(1)}$, where $i$ in the range of $1$ to $N$ and the source audio $x^{(0)}$ and the target audio $x^{(1)}$ has different contents. Each element $x^{(\alpha_{i})}$ in the series represents a morphed result corresponding to a specific morphing parameter $\alpha_{i}$, thus the set can be written as

$$ $\{x^{(\alpha_{i})}\}_{i=1}^{N}=\{x^{(\alpha_{1})},x^{(\alpha_{2})},...,x^{( \alpha_{N})}\}$ (14) $$

The MFCCsE is computed by

$$ $\text{MFCCs}_{\mathcal{E}}=\text{abs}(\frac{||m^{(\frac{N+1}{2})}-m^{(0)}||_{2 }}{||m^{(\frac{N+1}{2})}-m^{(0)}||_{2}+||m^{(\frac{N+1}{2})}-m^{(1)}||_{2}}-0.5)$ (15) $$

where $m^{(i)}$ represents the extracted MFCCs feature of the $i^{th}$ morphed results in the series $x^{(\alpha_{i})}$, $m^{(0)}$ and $m^{(1)}$ represents MFCCs feature of $x^{(0)}$ and $x^{(1)}$. This metric aims to evaluate spectrogram content of the perceptual midpoint result $x^{(\frac{N+1}{2})}$ between two end points $x^{(0)}$ and $x^{(1)}$. Ideally, we wish the midpoint morphed result contains half-and-half content on two end points. The larger $\text{MFCCs}_{error}$ indicates the content consistency is far away than the midpoint (i.e., 0.5). We extract MFCCs feature with 13 coefficients to compute MFCCsE.

### 8.3 Implementation details for computing FAD and FD

FAD and FD are commonly used in audio generation tasks to measure the quality of synthesized audio. Given two branches of audio groups, synthesized audios and real audios, these metrics offer a comprehensive assessment of the global quality by evaluating how closely the synthesized audio matches the distribution of real audio.

In our experiment, we calculate FAD and FD between morphed audios $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ and sourced audios to reflect correspondence of morphing and audio quality of morphed results. Smaller the FAD and FD values indicate the morphed audios has smaller semantic distribution gap between real sourced audios, suggesting that the morphed audios are more natural and exhibit semantic consistency.

Specifically, in the timbral morphing task, we categorize source audios based on groups of musical instruments, such as piano-guitar, harp-kalimba, etc. We then compute FAD and FD values between a consentive morphed path $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ and the corresponding instrument group to which the endpoint audios belong.
Similarity, in the case of environmental sound morphing task, we classify source audios according to sound scene categories and compute FAD and FD values between a consentive morphed path $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ and the corresponding sounds to which the endpoint audios belong to the categories. And in music morphing task, we calculate FAD and FD values between morphed audios $\{x^{(\alpha_{i})}\}_{i=1}^{N}$ with 20 samples of sourced music.

### 8.4 Implementation details for timbral space calculation

In timbral morphing task, we calculate timbral distance as an additional metric for evaluating smoothness of morphing. Refering to (McAdams, ; McAdams et al., ), we compute log attack time, spectral centroid and spectral flux from audio signal as the first, second and third dimension for plotting the timbre space. To plot the timbre space as in Figure 1, we normalized the value of each point $q$ into values between [-1,1].

## 9 Further information of mean opinion socre study

Figure: Figure 3: Example questions for a group of environmental sound morphing sample in MOS study.
Refer to caption: extracted/5897536/Figure/example_Q1.png

This section provide further implementation deatils for mean opinion score study. Therefore, we designed a 30 minues survey with 30 groups of evaluation questions. We randomly select 16 groups of cyclostationary morphing results (4 samples for each category) from the task of environmental sound morphing, and 14 groups of dynamic morphing results from the task music morphing, resulting in 30 groups of morphed examples in total. Each group has around 30 seconds audio durations, thereby, the quesionnaire takes around 30 minutes to complete (including time for reading the participants information sheet and response the questions).

For each group, we designed three questions for participants to give their opinion score regarding to correspondence, intermediateness, and smoothness. Details are

- 1.
Content Consistency: How consistent is the content of the morphed sound with the content of both the source and target sounds?
- 2.
Perceptual Consistency: How much does the morphed sound seem to be in between the source and target sounds?
- 3.
Smoothness of Transition: How smoothly does the transition occur in the morphed sound from the source sound to target sound?

Participants give score according to following scale:

- •
1 - Bad
- •
2 - Poor
- •
3 - Fair
- •
4 - Good
- •
5 - Excellent

Figure 3 and Figure 4 provide example questions in case of environmental sound morphing sample and music dynamic morphing sample in our MOS study.

We distributed our questionnaire link to some facebook groups and collected 21 completed responses from volunteer participants in this MOS study. During this study, only the opinion score participants provided are collected, we didn’t collect any participants’ personal information.

Figure: Figure 4: Example questions for a group of music dynamic morphing sample in MOS study.
Refer to caption: extracted/5897536/Figure/example_Q2.png

## 10 Further information of model comparison with MorphFader

Figure: Figure 5: Visualization of spectrogram for morphed results compred with MorphFader.
Refer to caption: extracted/5897536/Figure/model_compare_vis2.png

### 10.1 Experimental details

Due to MorphFader hasn’t open source their method, we make comparison with 7 pairs of morphing examples on its demonstration page (^4^44[https://pkamath2.github.io/audio-morphing-with-text/webpage/audio-morphing.html](https://pkamath2.github.io/audio-morphing-with-text/webpage/audio-morphing.html)). To have a fair comparison, We set DDIM inversion step as $T=100$ and match number of interpolation $N=5$ as in MorphFader. Due to we don’t have original source and target audios for their demonstrations, we cannot compute CDPAME, FAD and FID in this comparison. In this model comparison, we aim to validate how assumption that there exists a non-linear relationship between morph factor $\alpha$ and perceptual stimuli of the morphed result $p$. Therefore, simply uniformly increasing morph factor $\alpha$ cannot achieve truely perceptually smooth transitions. Fruthermore, this comparison showcases SoundMorpher’s superior on producing sound morphing sequences with constant perceptual stimuli increment, which cannot achieves by simply controlling morph factors.

In this experiment, we use demonstrated audios with $\alpha=0.0$ and $\alpha=1.0$ as our input target and source audio. We set the source and target text prompt provided in MorphFader demonstration page as the initial text prompt for LOA feature optimization. We set scale${}_{max}=4$ and scale${}_{min}=1.5$.

### 10.2 Further anaylsis

Figure 5 provides a visualization comparison of a pair of audio samples between MorphFader and SoundMorpher. We analysis Figure 5 according to two aspects: correspondence and smoothness.

Frequency Band Stability. Yellow rectangles in Figure 5 highlight frequency band intensity cross morphing results of MorphFader and SoundMorpher. The intensity of the frequency bands within the yellow rectangle changes more abruptly for MorphFader, which could suggest that the morphing process introduces inconsistencies in the spectral content. In contrast, yellow rectangles in SoundMorpher are more stable and consistent across time. The transitions between different frequency bands appear smoother, with fewer abrupt changes in intensity. This suggests that SoundMorpher maintains better spectral consistency during the morphing process, with smoother transitions between different timbral characteristics.

Transition Smoothness. As red rectangles indicate, MorphFader introduces more abrupt changes at the end of the morphing sequence. The spectral lines do not gradually transition; instead, there is a noticeable shift in the pattern, indicating less smoothness on transition. In contrast, SoundMorpher shows a more gradual and consistent transition within the red rectangles. The spectral patterns remain more stable and exhibit smoother transitions towards the end of the morphing sequence.

Overall, SoundMorpher appears to provide a more seamless and stable morphing process. The transitions are smoother, and the spectral content is more consistent across the morphing stages.

**Table 7: Ablation study on inference steps for environmental sound morphing with different types of environmental sounds.**
|  | T$=20$ | T$=100$ | FAD$\downarrow$ | FD$\downarrow$ | CDPAMT$\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | MFCCsE$\downarrow$ | CDPAME$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dog $\leftrightarrow$ Cat |  | $\checkmark$ | 17.77 | 73.92 | 1.293 | 0.323 $\pm$ 0.160 | 0.081 | 0.236 |
| $\checkmark$ |  | 18.27 | 81.60 | 1.172 | 0.293 $\pm$ 0.149 | 0.052 | 0.242 |  |
| Laughing $\leftrightarrow$ Crying baby |  | $\checkmark$ | 9.35 | 65.98 | 0.855 | 0.214 $\pm$ 0.077 | 0.044 | 0.289 |
| $\checkmark$ |  | 7.82 | 68.17 | 0.832 | 0.208 $\pm$ 0.115 | 0.078 | 0.250 |  |
| Church bells $\leftrightarrow$ Clock alarm |  | $\checkmark$ | 22.89 | 75.77 | 2.205 | 0.551 $\pm$ 0.299 | 0.058 | 0.312 |
| $\checkmark$ |  | 25.23 | 77.84 | 2.205 | 0.551 $\pm$ 0.304 | 0.056 | 0.352 |  |
| Door knock $\leftrightarrow$ Clapping |  | $\checkmark$ | 10.85 | 76.35 | 1.594 | 0.428 $\pm$ 0.220 | 0.083 | 0.321 |
| $\checkmark$ |  | 13.11 | 80.58 | 1.734 | 0.433 $\pm$ 0.281 | 0.118 | 0.281 |  |

**Table 8: Ablation study on model adaptation with different LoRA rank $r$ on music morphing. Rank with $-$ represents results without LoRA model adaptation.**
| Rank | MFCCsE$\downarrow$ | FAD$\downarrow$ | FD$\downarrow$ | CDPAMT$\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | CDPAME$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| - | 0.073 | 10.38 | 56.02 | 1.052 | 0.085 $\pm$ 0.054 | 0.198 |
| 4 | 0.056 | 9.85 | 56.09 | 0.847 | 0.068 $\pm$ 0.045 | 0.178 |
| 8 | 0.059 | 10.01 | 56.35 | 1.035 | 0.073 $\pm$ 0.051 | 0.180 |
| 16 | 0.059 | 9.95 | 56.14 | 1.058 | 0.075 $\pm$ 0.052 | 0.169 |
| 32 | 0.130 | 10.77 | 59.06 | 0.818 | 0.058 $\pm$ 0.082 | 0.158 |

## 11 Further discussion

### 11.1 Ablation study on inference steps for environmental sound morphing task

This section provides a supplementary ablation study on the inference steps for the environmental sound morphing task. Table 7 presents the results of this ablation. Similar to the findings in the music morphing task, we observe no significant performance difference between using larger and smaller inference steps. While larger inference steps appear to slightly improve reconstruction error in the music morphing task, however, in the case of the Laughing-Crying baby sound and Door knock-Clapping sound, the smaller DDIM steps result in a lower CDPAME score. Thus, we cannot conclusively establish a strong relationship between inference steps and perceptual resynthesis perceptual error.
One possible reason for results in music morphing task is the input audio music are synthesised by AudioLDM2 with 200 inference steps, therefore, larger inference steps helps for improving reconstruction quality in that case.
Overall, larger inference steps indicates a slight improvement on morph quality cross the four sound groups in this experiment. However, larger inference steps require longer time consumption on binary serach with SPDP algorithm. Therefore, we suggest a trade-off between overall algorithm time-comsumming and morphing quality when setting the DDIM inference steps for SoundMorpher.

### 11.2 Ablation study on model adaptation

In this experiment, we conduct an ablation study on model adaptation with LoRA on task of music morphing. We test SoundMorpher with different LoRA rank as well as SoundMorpher without model adaptation. Following Yang et al. (), we train LoRA parameters for 150 steps with 1e-3 learning rate. We also set unconditional bias correction with $r_{0}=2$ for 15 steps with 1e-3 learning rate.
Table 8 shows the results of SoundMorpher with different rank size on model adaptation settings. According to Table 8, SoundMorher without model adaptation has obvious performance drop on morphing correspondence compares results with LoRA model adaptation. Even though higher LoRA rank has a slight improvement on perceptual reconstruction quality, however, SoundMorpher with $r=32$ indicates poor correspondence with large MFCCsE and large smoothness variance CDPAMstd. This result indicates that SoundMorpher with higher LoRA rank not lead to a better morphing quality. When $r=4$, SoundMorpher achieves best performance on smoothness, and correspondence compared to $r=8$, $r=16$ and $r=32$. Therefore, we suggest LoRA rank for model adaptation in SoundMorpher shouldn’t be a large value such as $r=32$.

In image morphing task by Yang et al. (), they observed that higher LoRA rank on model adaptation leading to more diverse image morping path. However, our results indicate different observation. One possible interpretation is, different from image morphing, diverse audio morphing path may lead to a large semantic gap, which resulting a higher FAD, FD and MFCCsE (i.e., poor correspondence). This phenomena leads to a future study on how LoRA rank affects SoundMorpher performance in different morphing scenarios.

### 11.3 Ablation study on classifier-free guidance (CFG) scales

In this experiment, we explore impacts of CFG scales on SoundMorpher, we conduct an ablation study on music morphing task with $N=15$ on different sets of max-min CFG scales in Table 9. According to our experimental results, maximum scale controls correspondence quality and smoothness quality of morphed results, whereas higher maximum scale leads to a lower MFCCsE and higher CDPAM${}_{mean}\pm$ CDPAstd. In contrast, minimum scale controls intermediate quality of morphed results, where higher minimum scale leads to higher CDPAMT.

Figure: Figure 6: Failure cases for SoundMorpher with $N=5$. The source and target sounds that have significant semantic difference in contents, this leads SoundMorpher to produce abrupt transitions.
Refer to caption: extracted/5897536/Figure/abrupts.png

### 11.4 Failure case

Although SoundMorpher produces high-quality sound morphing results, abrupt transitions can occur when the source and target sounds have significant temporal structure differences. A clear example of this is attempting to morph continuous environmental sounds with sounds that contain more silence. One obviouse example is to morph continuous environmental sounds and sounds contains more slience as Figure 6 shows.

Environmental sounds often consist of discrete and temporally separated events, such as a dog barking or a cat meowing, which have distinct and abrupt characteristics. These are inherently different from the more continuous and harmonically structured nature of music, where elements blend more fluidly over time. As a result, creating smooth transitions between such disjointed environmental sounds can be more challenging, leading to the perception of more abrupt or less natural transitions in the morphing process.

**Table 9: Ablation study on classifier-free guidance (CFG) scales on music morphing task**
| CFG scale | MFCCsE$\downarrow$ | FAD$\downarrow$ | FD$\downarrow$ | CDPAMT$\downarrow$ | CDPAM${}_{mean\pm std}\downarrow$ | CDPAME$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| min: 1.5 - max: 3 | 0.150 | 10.95 | 59.20 | 0.808 | 0.058 $\pm$ 0.088 | 0.157 |
| min: 1.5 - max: 4 | 0.056 | 9.85 | 56.11 | 0.847 | 0.068 $\pm$ 0.045 | 0.178 |
| min: 1.5 - max: 5 | 0.055 | 9.81 | 56.01 | 1.101 | 0.078 $\pm$ 0.056 | 0.189 |
| min: 2 - max: 4 | 0.064 | 9.91 | 56.68 | 1.043 | 0.074 $\pm$ 0.053 | 0.173 |
| min: 3 - max: 5 | 0.068 | 9.96 | 57.46 | 1.074 | 0.076 $\pm$ 0.055 | 0.174 |

## 12 More visualization examples

This section provides more visualization examples for our experiment.
Figure 7 provides additional visualization of timbre morphing compared with SMT under a paired piano-guitar music composition sample. Compare to SMT, SoundMorpher produces a smoother morphing that continuously connects target and source timbre points in the timbre space with closely spaced transition.

Figure 8 displays three examples of timbre morph with different musical instruments. SoundMorpher produces high-quality and smooth morphing with $N=11$ perceptual-uniform intermediate morphed results.

Figure: (a) SMT
Refer to caption: extracted/5897536/Figure/piano-guitar-smt-1.png

Figure: Figure 8: Visualization of timbre morphing for musical instruments with $N=11$.
Refer to caption: extracted/5897536/Figure/timbre_morph_vis.png

Figure 9 demonstrates visualization for environmental sound morphing experiment. This shows how SoundMorpher transitions between various environmental sounds, offering insights into the smoothness, quality, and intermediate stages of the morphing process.

Additionally, we randomly select audio samples from AudioCaps (Kim et al., ) and use SoundMorpher with $N=10$ to perform complex sound scene morphing. Compared to the ESC50 dataset, the audio samples in AudioCaps often contain sound scenes involving multiple
complex physical events. Visualizations of as shown in Figure 10.

These visualizations demonstrate that SoundMorpher effectively produces high-quality morphing across diverse audio types, including complex environmental sounds, music, and various musical instrument timbres. This highlights the flexibility and efficiency of SoundMorpher, showcasing its potential applicability in multiple real-world scenarios.

Figure: Figure 9: Visualization of environmental sound morphing with $N=5$, from top to bottom: (1) church bells $\leftrightarrow$ clock alarm (2) crying baby $\leftrightarrow$ laughing (3) crying baby $\leftrightarrow$ laughing (4) cat $\leftrightarrow$ dog (5) clapping $\leftrightarrow$ wood door knocking
Refer to caption: extracted/5897536/Figure/environmental_sound_vis.png

Figure: Figure 10: Visualization of complex sound scenes from AudioCaps by SoundMorper with $N=10$.
Refer to caption: extracted/5897536/Figure/Audiocap.png

## 13 Data source

This section contains details of the open-sourced data we used in our experiments.

### 13.1 Timbral morphing for musical instruments

- •
8 pairs of piano-guitar and 8 pairs of harp-kalimaba audios: [https://harskish.github.io/Timbrer/index.html](https://harskish.github.io/Timbrer/index.html).
- •
6 pairs of timbral transfer audios with isolated musical instruments:
[https://wry-neighbor-173.notion.site/MusicMagus-Zero-Shot-Text-to-Music-Editing-via-Diffusion-Models
-8f55a82f34944eb9a4028ca56c546d9d](https://wry-neighbor-173.notion.site/MusicMagus-Zero-Shot-Text-to-Music-Editing-via-Diffusion-Models%5C%5C%0A-8f55a82f34944eb9a4028ca56c546d9d).

### 13.2 Sound morphing

- •
100 pairs of randomly selected environmental sound effects from ESC50 dataset: [https://github.com/karolpiczak/ESC-50](https://github.com/karolpiczak/ESC-50)

### 13.3 Music morphing

- •
21 musical compositions with different instruments and genres: [https://audioldm.github.io/audioldm2/](https://audioldm.github.io/audioldm2/).

### 13.4 Model comparison with MorphFader

- •
7 pairs of sound examples compared with MorphFader (Kamath et al., ): [https://pkamath2.github.io/audio-morphing-with-text/webpage/audio-morphing.html](https://pkamath2.github.io/audio-morphing-with-text/webpage/audio-morphing.html)

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The recordings given in the supplementary material are either midi or short isolated environmental sounds which raises concerns about applicability of the method to real data. Did you try to apply your method to real music recordings?
soundness: 2
strengths: - The proposed method is very reasonable and has already shown good results in image processing;
- Mostly, the quantitative comparisons are solid and show the effectiveness of the method in various audio domains;
- The paper proposes metrics to measure the morphing quality in terms of correspondence, intermediateness and smoothness which is important for further development of this research topic.
summary: This paper proposes a new method for sound morphing, defined as the generation of a sequence of perceptually intermediate sound samples for a given pair of source and target audio. As this problem is not very common in the audio processing community, the authors introduce different formulations of the problem (e.g., static vs. dynamic morphing) and discuss the criteria for a good morphing path, including correspondence, intermediateness, and smoothness. The authors propose an original method of sound morphing based on interpolating latents within AudioLDM2, demonstrating impressive results compared to the baseline. They also introduce several heuristics to make the transitions between consecutive samples along the morphing path more uniform and stable, including LoRA fine-tuning of AudioLDM2 and finding an optimal sequence of interpolation coefficients using binary search. Finally, the authors describe various experiments that demonstrate the strong performance of their model across different scenarios and audio domains.
weaknesses: - The contribution of the paper is relatively incremental as the paper borrows most of its ideas from (Yang et al. 2023). All essential features of the proposed method such as latents interpolation inside a Latent Diffusion Model, LoRA adaptation, finding the optimal trajectory with binary search on a sequence of values of an auxiliary metric, and even the introduction of 3 metrics for model evaluation, were proposed in (Yang et al. 2023). The paper is basically an attempt to adapt the approach introduced in (Yang et al. 2023) to audio domain. The main challenge the authors face in this study is related to tuning AudioLDM 2 and designing perceptual metrics as in (Yang et al. 2023).
- Comparison with only one baseline on a music domain given in Table 1 seems insufficient to provide full understanding of the performance of the method. Although objective metrics and the recordings given in the supplementary material demonstrate huge improvement over the baseline, it more likely suggests that the baseline may be too weak or poorly tuned. 
- The MOS scores given in Table 4 don’t have any grounding such as GT intermediate samples or a baseline which makes them uninterpretable. Moreover, because of a small number of assessors, the variance of the score is too large.
- Table 5 shows no improvement over the baseline in 2 metrics of 3. Intuition of the qualitative difference corresponding to the improvement of 0.04 in the first metric is not provided. Some illustrations of spectra and/or audio samples in the supplementary material corresponding to different levels of CDPAM may help here.
- Metric MFCC_Se introduced in Appendix 8 doesn’t evaluate the portion of the content from x_{source} and x_{target} but similarity of the spectra. The metric is easily minimized when we average x_s and x_t. 
- Minor issues: “Spectral contras” p.9:450; “Mean opinion socre” p.20:1034

REFERENCES
Zhaoyuan Yang et al. IMPUS: IMAGE MORPHING WITH PERCEPTUALLY UNIFORM SAMPLING USING DIFFUSION MODELS, ICLR 2024

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: - Where do the "Static morphing; Cyclostationary morphing & Dynamic morphing" terms come from?
- Why not using CDPAM as the perceptual metric? 
- The choice of the interpolating path is pretty arbitrary, and the presented technique could be applied over any path. Is the perceptual metric always monotonic over these trajectories?
soundness: 3
strengths: The paper is well-written and easy to follow, with examples provided in the supplementary material.
There is an extensive experimental part with a user study, but mainly compares with SMT.
summary: The authors present a method to morph sounds in an intended way.

They rely on a pretrained AudioLDM2 model (a latent diffusion model for sounds and music) and propose to generate morphed sounds by interpolating between two sounds. 

"Textual inversion" is performed in the condition embedding level for each sound to obtain embeddings $E$.
Latent representations $z_T$ are obtained using the probability flow ODE.
Morphed sounds are obtained by generating using the probability flow ODE from $(z_T^\alpha, E^\alpha)$ where the $z_T^\alpha$ is a slerp interpolation of the $z_T$'s and $E^\alpha$ a linear interpolation of the $E$'s.

Binary search is then performed on this trajectory parameterized by $\alpha$ to obtain samples equally-spaced according to some perceptual measure, relative L2 norm over mel spectrograms in the present case.
weaknesses: - The paper lacks novelty. Most of the elements presented in the paper do come from 
Yang et al.2023 IMPUS: IMAGE MORPHING WITH PERCEPTUALLYUNIFORM SAMPLING USING DIFFUSION MODELS.
The present paper can thus be seen as a straightforward adaptation of IMPUS to the audio domain.
- If LPIPS was chosen in IMPUS as the perceptual metric, here the choice of L2 over mel-spectrograms may be less appropriate.
- Concatenating audio segments in Eq. 8 in x-space seems to produce abrupt transitions. Please not that with most of the architectures used for diffusion models, it would be possible to have this chunk-based generation done directly in z-space.

- There are some missing references concerning controlled interpolations using audio diffusion models.
- Provided audio examples are not particularly convincing and transitions sound abrupt, especially with as few points as five.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Questions and Suggestions for Improvement

From the comments above here are some questions/suggestions for the authors:

Sound Morphing Preliminary:

o	Correspondence: few words about subjectivness of the semantic meaning would help here I think.

o	Intermediateness: The example I gave in the comments above, the cat-to-dog morph in the demo files sometimes produces an "in-between" sound that contains both the cat and dog sounds, which might be realistic but isn’t necessarily an intermediate. Could the authors clarify what qualifies as intermediate? Are both interpretations valid?

o	Smoothness: Could the authors explain what they mean by "linear change"? Although numbers are used (Mel spectrograms in this case), it’s still not fully clear. For elements like pitch and loudness, does this “linear change” involve logarithmic scaling in metrics? Is it based on human perceptual linearity, and if so, what exactly does that mean, and how does it translate to audio metrics? A more explicit explanation would be helpful. In lines 258–265, the authors state that the Mel spectrogram is a good approximation of human perception, but I think a few more words on the logarithmic nature of audio perception and how the Mel scale assists here would strengthen the explanation.

________________________________________
Evaluation Concerns:

2.	Evaluation Metrics:

o	Please provide exact formula of how FAD and FD are calculated or please direct me to where it is presented.

o	Could you provide justification for why CDPAM is relevant metric?

3.	Timbral Morphing:

o	The choice of baseline model (Sound Morphing Toolbox from Caetano, 2019) seems outdated and not representative of current ML methods. Could the authors consider more recent ML-based baselines? This would provide a fairer comparison, especially as the SMT-generated audio files seem quite low quality.

4.	Environmental Sound Morphing:

o	No baseline model is used here, and the results are presented only with the authors' own metrics. Could the authors incorporate a baseline to add context to their results?

5.	Music Morphing:

o	Similarly, no baseline model is used here. Without a baseline, it is difficult to interpret the significance of the reported results. Could the authors add a comparison baseline to strengthen the evaluation?

6.	Subjective Evaluation:

o	In the subjective (human) evaluation, no comparison is made with other methods. Why did the authors choose to evaluate only the proposed model, especially considering the number of concurrent works mentioned in the related work section? Adding more baseline comparisons would enhance the evaluation's rigor.

7.	Model Comparison:

o	In the "Model Comparison" section (line 472), would it be possible to expand this comparison across all scenarios, including metrics from MorphFader or other models, for a fairer and more comprehensive comparison? Additionally, the authors report only CDPAMT, CDPAMmean±std, and MFCCsE. Why were these metrics chosen, and why were other metrics not included?
soundness: 3
strengths: This is a fairly well-written paper. I found it interesting and a pleasure to read. Even though the method it presents is not a groundbreaking novelty and is only an inference-time "trick" to achieve good sound morphing, I found it a clever way of handling and extracting desired results from a pre-trained generative model. The related work section is quite comprehensive (at least, I couldn’t recall any paper that hasn't been mentioned here), the objectives of the work are clear, and the method is fairly well explained. The proposed metrics do make sense (I’ll come back to this in the weaknesses section), and the algorithms (show in the appendix) and descriptions of the methodology are quite comprehensive and help understanding.
summary: The paper presents a MusicLDM2-based inference-time method (requiring no model training or fine-tuning) for morphing various types of music and sounds across scenarios such as timbral morphing of musical instruments, environmental sound morphing, and music morphing. The authors claim their approach differs from traditional sound morphing methods, which assume a straightforward, linear relationship between the morph factor and perceived sound qualities. Traditional methods oversimplify the complexity of sound perception and limits morph quality. In contrast, they propose SoundMorpher, which establishes a more accurate, proportional mapping between the morph factor and the perceived characteristics of morphed sounds using Mel-spectrograms. SoundMorpher employs the Sound Perceptual Distance Proportion (SPDP) ensuring that changes in the morph factor correspond uniformly to perceptual changes. This approach enables smoother sound transitions and maintains consistent quality across different sound-morphing tasks.

The paper also introduces metrics to evaluate sound morphing systems based on three key criteria: correspondence, perceptual intermediateness, and smoothness. Extensive tests demonstrate SoundMorpher's effectiveness and adaptability, with potential applications in music creation, film post-production, and interactive audio technologies. 

Demo audio files are provided in the supplementary material.
weaknesses: Here is the list of my concerns following the order of the sections.

Sound morphing preliminary:

In the paragraph of line – 141 -149, authors define 3 criterias Correspondence, Intermediateness, and Smoothness. I think there are lots of critiques one can brung up to these criterias.

1.	Correspondence: This criterion requires that the morph captures semantic-level transitions. However, perception of “semantic” qualities in sound can be subjective and context-dependent. If listeners interpret these qualities differently, then it may be difficult to ensure correspondence consistently across diverse types of sounds, especially if the morphing involves highly variable sources, like environmental sounds versus musical instruments.

2.	Intermediateness: This criterion expects listeners to perceive intermediate morphs as “between” the source and target sounds. However, the perceptual “in-betweenness” may not always be straightforward in audio and can be (again) subjective. If sounds have distinct timbral qualities, they might not blend smoothly. In such cases, perception could be binary or categorical(?), with certain morphing points perceived as “closer” to one sound or the other rather than as true intermediates.
For example, in the audio demo files, there is a cat-to-dog morph. Sometimes, the cat's meow gradually changes into a dog’s bark, but at other times, there is an intermediate sound containing both the cat and dog sounds (like an audio scene where both animals are present, which is quite realistic and could be perceived as intermediate). So, which of these is truly intermediate? Or are both? I don’t see a clear description addressing this. Please correct me if I’m missing something.

3.	Smoothness: This criterion assumes that a linear change in the morph factor will correspond to a consistent, linear perceptual transition. I have a major question here: why is a linear change necessarily good? As we know, sound perception does not always follow linear increments; auditory characteristics like volume and pitch are often perceived on logarithmic or exponential scales. This means that, depending on where we are in the spectrum, even small changes in the morph factor might lead to abrupt perceptual shifts, while large changes may go unnoticed by human listeners. This would violate the criterion of smoothness, correct?
For example, if a bass instrument is morphing into a high-pitched violin and pitch is increasing by a fixed step (say 100 Hz), that change at a low frequency would be perceived as quite large, while at higher frequencies (around 2000 Hz), the same 100 Hz step wouldn’t be perceived as a significant change. So, what exactly do the authors mean by "linear change"? when they say linear in context of perceptual doe this mean logarithmic in metrics? I didn’t find a clear explanation on this in the paper but this is a major question, I think, and need to be mentioned and clearly explained. 

Evaluation:


My main concern with this paper lies in the experiments and evaluation section. The issue I see is that the work develops its own metrics and then uses these motly for just reporting the number of the model. For the comparison with baseline(s) same metrics are used, rather than relying on established metrics (at least thiose used in the prevous work for fair comarison). This approach makes comparisons with existing work somewhat shaky. 

Here are the details.

1.	Evaluation metrics

Lines 1021 and 1023-1024: Did you mean “consecutive” instead of “consentive”? 
The calculation of FAD and FD is unclear. Are these values averaged? The paragraph starting at line 1019 is not entirely clear. I see there are instrument groups, but it’s not fully explained how FAD and FD are calculated along the morphed path. Sounds in the middle of this path would presumably be intermediate between the two instruments. Are we calculating FAD and FD across the entire sequence and then averaging, or only at the midpoint? If it's the midpoint, what are we comparing it to—the source or the target? I’m unsure what FAD and FD are meant to reveal in this context. Could you provide a more detailed explanation, maybe including a formula for how final FAD is calculated in the tables (lines 270, 380, and 447)?

Lines 316-320: use of CDMAP seems not fully justified.  CDMAP is typically used for speech. Is it suitable for measuring instrument and environmental sounds? Could you provide some justification for why this metric is relevant here?

2.	TIMBRAL MORPHING

The baseline model chose here is weird the Sound Morphing Toolbox (SMT) from Caetano, 2019. This is very old and not a ML model. Why don’t authors use some more recent work for the comparison. In the demo files the files generated by SMT sound terrible! I don’t think SMT is even worth considering as a baseline. I’m sure much better quality of audios can be achived by usieng any other concurent ML based baseline.

3.	ENVIRONMENTAL SOUND MORPHING

No baseline, Just showing the numbers for their own metrics.

4.	MUSIC MORPHING

Same concern here, no baseline! I don't undestand what are are these numbers say without a baseline...

5.	Subjective Evaluation:

Again, for some reason, the subjective (human) evaluation does not compare the proposed method with other methods. 

Generally, in the paper, baseline models are not used in the evaluation. The paper only presents results for the proposed model, with the only comparison being in timbral morphing for musical instruments—and even that is with an older signal processing method, the Sound Morphing Toolbox (SMT) from Caetano, 2019. The related work section references numerous concurrent works in the field, so I don’t understand why the authors did not adopt any of these as baselines.

6.	Model comparison

In the "Model Comparison" subsection at line 472, the authors compare their method to a concurrent work, MorphFader (Kamath et al., 2024), but they only use the 7 examples provided on the demonstration page of that work. I don’t believe that 7 examples are sufficient for a robust comparison. I would have liked to see a comparison with this model (or other concurrent models) across all scenarios with all metrics and bigger test dataset, including those used by the baseline models, for a fair assessment. Additionally, they report only CDPAMT, CDPAMmean±std, and MFCCsE—why limit the metrics?




There are some small typos and grammatical errors that I spoted:

In the line 013: if I understand what authors say correctly, "models" should be "model" to match the plural subject "methods." 

Line 243 there is a typo: adaption->adaptation.

It would be nice to have table 1 on the page where it is dsuicussed.

Line 444-445, in the table 3 N is number of components of PCA? This is misleading because N was used for N a number of sounds in morphed sequence between sores and target lines (190—193).

The experiment section is structured a little weird. I don’t understand why in experiments (where we also have results presented by scenarios) we have subsection “discussion”. And this subsection presents Mean opinion score, which, in my opinion, is the most valuable part of results section.

Line 1207 – instead of “achieves” must be “be achieved”, I suppose.