MEDIC: Zero-shot Music Editing with Disentangled Inversion Control

## Abstract

Abstract Text-guided diffusion models make a paradigm shift in audio generation, facilitating the adaptability of source audio to conform to specific textual prompts. Recent works introduce inversion techniques, like DDIM inversion, to zero-shot editing, exploiting pretrained diffusion models for audio modification. Nonetheless, our investigation exposes that DDIM inversion suffers from an accumulation of errors across each diffusion step, undermining its efficacy. Moreover, existing editing methods fail to achieve effective complex non-rigid music editing while maintaining essential content preservation and high editing fidelity.
To counteract these issues, we introduce the Disentangled Inversion technique to disentangle the diffusion process into triple branches, rectifying the deviated path of the source branch caused by DDIM inversion.
In addition, we propose the Harmonized Attention Control framework, which unifies the mutual self-attention control and cross-attention control with an intermediate Harmonic Branch to progressively achieve the desired harmonic and melodic information in the target music. Collectively, these innovations comprise the Disentangled Inversion Control (DIC) framework, enabling accurate music editing while safeguarding content integrity. To benchmark audio editing efficacy, we introduce ZoME-Bench , a comprehensive music editing benchmark hosting 1,100 samples spread across ten distinct editing categories. This facilitates both zero-shot and instruction-based music editing tasks. Our method achieves unparalleled performance in edit fidelity and essential content preservation, outperforming contemporary state-of-the-art inversion techniques. 1 1 1 The project webpage is available at https://MEDIC-Edit.github.io/ .

## 1 Introduction

Text-guided diffusion models (Song et al., ; Song & Ermon, ; Peebles & Xie, ) have made great progress in audio generation (Evans et al., , ), leveraging their impressive capability for realistic and varied outputs.
In particular, these models (Liu et al., ; Huang et al., ; Liu et al., ) provide the foundation for prompt-based audio editing, offering new opportunities to modify audio landscapes for specific textual prompts.
Early audio editing strategies rely on training models from scratch (Copet et al., ; Agostinelli et al., ) or test-time optimization (Paissan et al., ; Plitsis et al., ), hampered by intensive computational demands.
Recent works (Manor & Michaeli, ; Zhang et al., )
in zero-shot audio editing have been made through Denoising Diffusion Implicit Models (DDIM) (Song et al., ) and Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., ) inversion techniques, but challenges remain.

Key among these challenges are maintaining
fidelity of editing - ensuring the editing aligns with the provided instructions - and essential content preservation, ensuring that the unaltered musical attributes in the target prompts remain unchanged.
However, balancing these objectives involves a careful exchange of information between the source and target branch in diffusion processes, with existing inversion methods like DDIM proving sub-optimal for conditional diffusion models (Mokady et al., ).
Enhanced versions of edit-friendly DDPM inversion (Huberman-Spiegelglas et al., ) make strides in preservation by imprinting the source onto the noise space. However, this comes at the expense of reduced modification capabilities due to noise reduction.

In this work, we rigorously examine the shortcomings of the DDIM inversion approach.
Our comprehensive analysis indicates that while techniques like DDIM inversion provide an editable foundation for audio synthesis, they lack precision and may compromise the integrity of the original audio. The primary issue stems from the assumption of perfect reversibility in the ordinary differential equation (ODE) process, which is frequently not met during text-conditional editing. Consequently, this leads to distortions during the inversion. And the implementation of Classifier-free Guidance (CFG) (Ho & Salimans, ) aims to improve text adherence. However, it inadvertently amplifies the accumulated errors from the inversion process.
Moreover, attention control (Cao et al., ; Hertz et al., ) has shown promise in achieving high fidelity and essential content preservation. For instance, MusicMagus (Zhang et al., ) introduces Cross-Attention Control for fine-grained music manipulation of rigid tasks.
Nevertheless, these methods fail to resolve the issues of accumulated errors and struggle to achieve accurate editing for both rigid and non-rigid tasks, as illustrated in Figure 1.
We introduce an innovative Disentangled Inversion Control technique to bridge this gap.
This technique has two main components: Harmonic Attention Control and Disentangled Inversion.
Cross-attention control (Hertz et al., ) and mutual self-attention control (Cao et al., ) have demonstrated robust editing capabilities for rigid and non-rigid image editing tasks, respectively. However, blindly combining these two approaches sequentially for music editing can result in sub-optimal outcomes, particularly in the original dual-branch setup, where it struggles with global attention refinement. To address this challenge, we introduce an intermediate branch called the Harmonic Branch, designed to modify both rigid and non-rigid attributes in music progressively.
Furthermore, we disentangle the diffusion process into triple branches, correcting the deviation path caused by CFG in the source branch, which affects the essential content preservation. The other branches remain unchanged to ensure the highest possible edit fidelity.

Due to a lack of standardized benchmarks in audio editing, we introduce ZoME-Bench.
The first music editing benchmark consists of 1,100 audio clips, distributing them into 10 rigorously curated editing categories across rigid and non-rigid tasks.
Each entry is carefully assembled, comprising a source prompt, a target prompt, human instruction, and blended words intended for editing.
Experimental results on ZoME-Bench indicate that MEDIC outperforms baselines, achieving significant improvements in essential content preservation and editing fidelity. Additionally, MEDIC demonstrates state-of-the-art performance in the variable-length music editing settings of the MusicDelta dataset.

Our contributions can be summarized as follows. 1) We introduce a novel, training-free methodology called Disentangled Inversion Control (DIC), designed to facilitate consistent manipulations of musical elements and intricate non-rigid editing tasks.
2) An Harmonized Attention Control framework is introduced to unify cross-attention and mutual self-attention control, which enables both rigid and non-rigid editing.
3) We present Disentangled Inversion Technique to achieve superior results with negligible inversion error by branch disentanglement and correction, aiding in accurately editing the music while preserving the content information.
4) We build a new benchmark for music editing, named ZoME-Bench, which supports both zero-shot and instruction-based music editing.

## 2 Related Works

Text-based Audio editing.
The objective of text-based audio editing (Paissan et al., ; Plitsis et al., ; Han et al., ) is to utilize diffusion models to manipulate audio content based on the provided target prompt.
Existing methodologies for addressing these intricate challenges typically follow one of three paths. The first involves attempts to develop end-to-end editing models (Copet et al., ; Agostinelli et al., ; Chen et al., )that employ diffusion processes.
However, these efforts are often hampered by indirect training strategies or a lack of comprehensive datasets. The second path involves test-time optimization strategies that utilize large pre-trained models for editing (Paissan et al., ; Plitsis et al., ). Despite their versatility, these methods are often burdened by the significant computational demands of fine-tuning diffusion models or optimizing text-embeddings for signal reconstruction. Some methods may choose to employ both strategies (Kawar et al., ), further increasing the computational load. The final path involves inversion techniques, which typically use DDPM (Huberman-Spiegelglas et al., ; Wu & De la Torre, )/DDIM (Song et al., ; Zhang et al., ) inversion strategies to extract diffusion noise vectors that match the source signal.
Given its rapid and intuitive zero-shot editing capabilities, we have chosen inversion techniques as our primary research framework.
In this work, we propose a new inversion technique named Disentangled Inversion Control. This technique aims to achieve accurate editing while preserving structural information.

Inversion Techniques.
The field of image inversion techniques has experienced significant progress in recent years (Brooks et al., ; Kim et al., ; Parmar et al., ; Dhariwal & Nichol, ).
While the DDIM inversion proves effective for unconditional diffusion models (Song et al., ; Wallace et al., ), its limitations become apparent when applied to text-guided diffusion models, particularly when classifier-free guidance is necessary for meaningful editing.
A range of solutions (Mokady et al., ; Tumanyan et al., ) have been proposed to address these challenges.
For example, Negative-Prompt Inversion strategically assigns conditioned text embeddings to Null-Text embeddings, effectively reducing potential deviation during editing.
Conversely, Edit Friendly DDPM provides an alternative latent noise space via modified DDPM sample distributions, promoting the successful reconstruction of the desired image (Huberman-Spiegelglas et al., ).
Optimization-based inversion methods using specific latent variables have recently gained popularity (Ju et al., ; Kawar et al., ).
These are designed to minimize accumulated errors stemming from the DDIM inversion.
Techniques such as Null-Text Inversion (Mokady et al., ) are promising, but they introduce complexity and instability into the optimization process, making it relatively time-consuming.
Differently, we introduce a plug-in-plus method called Disentangled Inversion Control to separate branches, achieving superior performance with considerably fewer computational resources.

Figure: Figure 1: Comparisons of our method with two-branch inversion techniques, including DDIM Inversion and MusicMagus. (a) Framework of DDIM Inversion, showing configurations with and without classifier-free guidance. (b) Framework of MusicMagus, which incorporates cross-attention control. (c) Framework of our method, featuring disentangled inversion control. (d) An illustration comparing the output of the two-branch techniques with the progressive output of our triple-branch approach.
Refer to caption: x1.png

## 3 Disentangled Inversion Control

### 3.1 Problem Definition and BenchMark Construction

Despite significant work in text-to-audio generation models, particularly with the emergence of latent diffusion models (LDM), research on zero-shot music editing remains limited. Zero-shot music editing seeks to leverage the capabilities of text-to-music generation models to synthesize the desired music, denoted as ${\bm{x}}^{tgt}_{0}$. This synthesized music should align with the target edited text prompt ${\mathcal{P}}^{*}$, which is directly modified from the source music ${\bm{x}}^{src}_{0}$ and its corresponding text prompt ${\mathcal{P}}$. We compress source audio signals into latent ${\bm{z}}^{src}_{0}$ for inversion.

To systematically validate our proposed method as a plug-and-play strategy for editing models, compare our method with existing zero-shot music editing methods, and compensate for the absence of standardized performance criteria for inversion and editing techniques, we construct a benchmark dataset named ZoME-Bench (Zero-shot Music Editing Benchmark).
ZoME-Bench comprises 1,100 audio samples which are selected from MusicCaps (Doh et al., ), spanning ten different editing types that include both rigid and non-rigid modifications.
Each sample is accompanied by its corresponding source prompt, target prompt, human instruction, and source audio.

Additionally, we include annotations relevant to attention control, such as blended words, to facilitate detailed evaluations. Further details about our benchmark can be found in Appendix B.

### 3.2 Motivation

Figure 1 and Preliminaries in Appendix A reveal that while techniques like DDIM inversion offer an editable base, they fall short of precision. This potentially compromising the essential content preservation. The implementation of Classifier-free Guidance (CFG) further amplifies the accumulated errors.

In the landscape of prompt-based editing (Dong et al., ; Kim et al., ; Feng et al., ), the ability to grasp the subtleties of linguistics and enable more granular cross-modal interactions stands as a formidable challenge.  Hertz et al. () acknowledges that in image editing, the fusion between text and visual modalities happens within the parameterized noise prediction network $\epsilon_{\theta}$. This leads to the development of various attention control techniques that guide the target denoiser network $\hat{\epsilon}_{\theta}$ in the image domain to better align with target prompts. Yet, analogous control mechanisms for non-rigid music editing are noticeably limited.

Taking these insights forward, we introduce Disentangled Inversion Control (DIC), a novel approach to achieve both rigid and non-rigid music editing.
DIC strategically disentangles the diffusion process as triple branches, allowing each branch to optimize its functionality.
At the same time, the strategy leverages harmonized attention control to facilitate targeted editing, thus aligning with the dual objectives of preserving the original audio essence and ensuring edit relevance.
We will first introduce Harmonized Attention Control in Section 3.3 and discuss Disentangled Inversion in Section 3.4.

### 3.3 Harmonized Attention Control Framework

The denoising architecture denoted as $\epsilon_{\theta}$, is structured as a sequence of fundamental blocks, each comprising a residual block (He et al., ) followed by self-attention and cross-attention modules (Vaswani et al., ; Dosovitskiy et al., ; Liu et al., ).
At the denoising step $t$, the output of the $(l-1)$-th block is passed through self-attention and then aligned with textual cues from prompt $P$ within the cross-attention layer.
The attention mechanism is formalized as:

$$ $\text{Attention}(Q,K,V)=MV=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V$ (1) $$

where $Q$ denotes the query features derived from the audio features, $K$ and $V$ represent the key and value features, and $M$ is the attention map.
We explore varying semantic transformations of audio content through harmonized attention control strategies — cross-attention control for rigid changes and mutual self-attention control for non-rigid adjustments.
Finally, we introduce an intermediate branch to host the desired harmonic and melodic information in the target music. The framework of harmonized attention control is depicted in Figure 2.

Figure: Figure 2: The framework of Harmonized Attention Control (HAC). HAC unifies cross-attention control and mutual self-attention control with an additional branch named Harmonic Branch to host the desired composition and structural information in the target music.
Refer to caption: x2.png

#### 3.3.1 Cross-Attention Control

Cross-attention Control (CAC) aims to inject the attention maps that are
obtained from the generation with the original prompt ${\mathcal{P}}$, into a second generation with the target prompt ${\mathcal{P}}^{*}$. We follow the practice of Prompt-to-Prompt (Hertz et al., ) and define CAC as Global Attention Refinement and Local Attention Blend.

##### Global Attention Refinement

At a given time step $t$, the attention map $M_{t}$ for both the origin and target branches is computed, averaging over all layers with respect to the noised latent $\mathbf{z}_{t}$. We employ an alignment function $A$ that maps each token index from the target prompt $\mathcal{P}^{*}$ to its equivalent in $\mathcal{P}$ or to None for non-aligning tokens. The refinement action is thus:

$$ $\text{Refine}(M^{src}_{t},M^{tgt}_{t},t)=\begin{cases}(M^{tgt}_{t})_{i,j}& \text{if }A(j)=\text{None},\\ (M_{t})_{i,j}&\text{otherwise}.\end{cases}$ (2) $$

##### Local Attention Blends

Beyond global attention, we incorporate a blending mechanism as suggested by  Hertz et al. () and  Mokady et al. (). This method selectively integrates and maintains certain semantics by using target blend words $w^{tgt}$ for semantic additions and source blend words $w^{src}$ for semantic preservation. At each denoising step $t$, the mechanism operates on the target latent ${\bm{z}}^{tgt}_{t}$ as follows:

$$ $\displaystyle m_{\text{tgt}}$ $\displaystyle=\text{Threshold}\left[M^{tgt}_{t},w_{\text{tgt}},k_{\text{tgt}} \right],$ (3) $\displaystyle m_{\text{src}}$ $\displaystyle=\text{Threshold}\left[M^{src}_{t},w_{\text{src}},k_{\text{src}} \right],$ (4) $\displaystyle{\bm{z}}^{tgt}_{t}$ $\displaystyle=(1-m^{tgt}+m^{src})\odot{\bm{z}}^{src}_{t}+(m^{tgt}-m^{src}) \odot{\bm{z}}^{tgt}_{t}$ (5) $$

where $m_{\text{tgt}}$ and $m_{\text{src}}$ are binary masks and
threshold function is as delineated below:

$$ $\text{Threshold}(M,k)=\begin{cases}1&\text{if }M_{i,j}\geq k,\\ 0&\text{if }M_{i,j}<k.\end{cases}$ (6) $$

For simplicity, we define the process of local editing in Equation 5 as:

$$ ${\bm{z}}^{tgt}_{t}=\text{LocalEdit}({\bm{z}}^{src}_{t},{\bm{z}}^{tgt}_{t},M^{ src}_{t},M^{tgt}_{t},w_{\text{src}},w_{\text{tgt}})$ (7) $$

##### Scheduling Cross-Attention Control

Applying cross-attention control only at early stages ensures creative flexibility while maintaining structure.
Acting on insights from  Hertz et al. (), we limit cross-attention to the initial phases up to a cutoff point $\tau_{c}$. This moderation allows us to capture the nuances and intended changes in musical compositions effectively. The approach is defined as:

$$ $\text{CrossEdit}(M^{src},M^{tgt},t)=\begin{cases}\text{Refine}(M^{src}_{t},M^{ tgt}_{t})&\text{if}\ t\geq\tau_{c},\\ M^{tgt}_{t}&\text{if}\ t<\tau_{c}.\end{cases}$ (8) $$

#### 3.3.2 Mutual Self-Attention Control

We diverge from the conventional use of cross-attention mechanisms and instead draw inspiration from the MasaCtrl (Cao et al., ) technique to refine music structure through self-attention queries.
These queries adeptly navigate through non-rigid musical transformations, aligning with the designated musical theme or instrument (target prompt).
The process begins by sketching the foundational musical theme using the target’s self-attention components—$Q^{tgt}$, $K^{tgt}$, and $V^{tgt}$. This is followed by enriching this theme with elements resembling the thematic content from the source ($K^{src}$, $V^{src}$), steered by $Q^{tgt}$.
However, applying this attentive modulation uniformly over all processing layers and through every denoising step might result in a composition excessively mirroring the source. Consequently, echoing the ethos of MasaCtrl, our proposed solution selectively employs mutual self-attention in the decoder portion of our music editing U-Net, initiated after a set number of denoising iterations.

##### Scheduling Mutual Self-Attention Control

The application of mutual self-attention is meticulously planned, beginning at a specific denoising step $S$ and extending beyond a designated layer $L$. The strength and influence of this control mechanism are designed as follows:

$$ $\text{SelfEdit}({Q^{src},K^{src},V^{src}},{Q^{tgt},K^{tgt},V^{tgt}},t)=\begin{ cases}{Q^{src},K^{src},V^{src}}&\text{if }t\geq S\text{ and }l\geq L,\\ {Q^{tgt},K^{src},V^{src}}&\text{otherwise}\\ \end{cases}$ (9) $$

In this framework, $S$ signifies the denoising step from which the mutual self-attention control commences, serving as a temporal threshold. Similarly, $L$ distinguishes the layer index below which this nuanced control strategy becomes operational, tailoring the musical output towards the intended artistic direction.

#### 3.3.3 Harmonic Branch Integretion

The naive combination of cross-attention control and mutual self-attention control sequentially would lead to sub-optimal results in the original dual-branch setup, especially failing the global attention refinement.
To address this issue, we introduce an additional latent harmonic branch, which serves as an intermediate to host the desired composition and structural information in the target music.

Our unified framework is detailed in Algorithm 2. During each forward step of the diffusion process, we start with mutual self-attention control on $z^{src}$ and $z^{tgt}$ and assign the output to the harmonic branch latent $z^{har}$. This lays the formal structure of the target music. Following this, cross-attention control is applied on $M^{har}$ and $M^{tgt}$ to refine the semantic information for $M^{tgt}$. As illustrated in Figure 2, the harmonic branch output $z^{har}_{0}$ reflects the requested non-rigid changes (e.g.,
“violin"), while preserving the rigid content semantics (e.g., “with noise"). The target branch output $z^{tgt}_{0}$ builds upon the structural layout of the $z^{har}$ while
reflecting the requested rigid changes (e.g., “with noise").

Figure: Algorithm 1 Harmonized Attention Control in one DDIM Forward Process

### 3.4 Disentangled Inversion Technique

Recognizing the limitations of using DDIM inversion without classifier-free guidance, we observe that it yields an easily modifiable but imprecise approximation of the original audio signal. Increasing the guidance scale enhances editability, but sacrifices reconstruction accuracy due to latent code deviation during editing.

In order to address this issue, our methodology, which we have termed the Disentangled Inversion Technique disentangles into three branches: the source, the harmonic, and the target branch. This decoupling is designed to unleash the capabilities of each branch separately. For the source branch, we implement a targeted correction mechanism. By reintegrating the distance $z^{*}_{t}-z^{src}_{t}$ into $z^{src}_{t}$, we directly mitigate the deviation of the pathway. This straightforward adjustment effectively rectifies the path and minimizes the accumulated errors introduced by both DDIM inversion and classifier-free guidance, thereby enhancing consistency in the reconstructed audio. On the other hand, the target branch and harmonic branch are left unmodified to fully leverage the innate capabilities of diffusion models in generating the desired target audio. The branches’ untouched state ensures that the model’s potential is utilized to its fullest extent, thereby ensuring the fidelity and integrity of the generated audio. We will further discuss this in the Section 4.2.2. The algorithm of Disentangled Inversion Technique has been outlined in Algorithm 2.

Typical diffusion-based editing (Han et al., ; Miyake et al., ) involves two parts: an inversion process to get the diffusion space of the audio, and a forward process to perform editing on the diffusion space. Disentangled Inversion can be plug-and-played into the forward process and rectifies the deviation path step by step. Specifically, Disentangled Inversion first computes the difference between ${\bm{z}}^{*}_{t-1}$ and ${\bm{z}}^{src}_{t-1}$, then adds the difference back to ${\bm{z}}^{src}_{t-1}$ in DDIM forward. We only add the difference of the source prompt in latent space and update ${\bm{z}}^{src}_{t-1}$, which is the key to retaining the editability of the target prompt’s latent space.

Figure: Algorithm 2 Disentangled Inversion Technique

## 4 Experiments

Implementation Details. We infer different editing methods using the pre-trained AudioLDM 2 (Liu et al., ) models with 200 inference steps. The setting of baselines is followed by Manor & Michaeli () and is all evaluated in NVIDIA A800 GPU for a fair comparison. We evaluate all methods in ZoMo-Bench across all editing types with fixed length. And we use the MusicDelta subset of the MelodyDB (Bittner et al., ) dataset for variable length comparisons, comprised of 34 musical excerpts in varying styles and in lengths ranging from 20 seconds to 5 minutes. Details can be found in Appendix C.

Evaluation Metrics. Our models employ a comprehensive evaluation using both objective and subjective metrics to assess essential content preservation, text-audio alignment fidelity, and audio quality. Objective metrics include Structure Distance (SD) (Ju et al., ), CLAP (Contrastive Language-Audio Pretraining) Score (Elizalde et al., ), LPAPS (Learned Perceptual Audio Patch Similarity) (Iashin & Rahtu, ; Paissan et al., ), and FAD (Fréchet Audio Distance) (Kilgour et al., ). The CLAP Score evaluates how well the edited audio aligns with the target prompt, while Structure Distance, LPAPS, and FAD, which have been adapted from image domain metrics, measure the similarity between the edited and source audio. Specifically, LPAPS quantifies the consistency of the edited audio in relation to the source audio, and the FAD metric assesses the distance between two distributions of audio signals. The Structure Distance measures the structural similarity between the edited and source audio. For the subjective evaluation, we conduct crowd-sourced human assessments using the Mean Opinion Score (MOS) to evaluate both editing fidelity (MOS-Q) and content preservation (MOS-P).
We attach the details of all metrics in Appendix C.1.

### 4.1 Zero-shot Music Editing Results

**Table 1: Comparison with baselines on ZoME-Bench with fixed length about structure, content preservation, and CLAP similarity.**
| Method | Objective Metrics | Subjective Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\text{SD}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | CLAP Score$\uparrow$ | MOS-Q$\uparrow$ | MOS-P$\uparrow$ |  |
| AudioLDM 2 | 23.86 | 0.21 | 10.36 | 0.58 | 73.48 | 70.12 |
| MusicGen | 23.39 | 0.21 | 6.63 | 0.59 | 75.46 | 71.28 |
| SDEdit | 25.87 | 0.22 | 12.18 | 0.40 | 69.38 | 66.23 |
| DDIM Inversion | 22.52 | 0.21 | 9.51 | 0.49 | 73.10 | 3.38 |
| MusicMagus | 16.23 | 0.19 | 5.15 | 0.55 | 75.12 | 74.34 |
| DDPM-Friendly | 18.30 | 0.19 | 5.16 | 0.54 | 75.27 | 73.86 |
| MEDIC | 11.97 | 0.15 | 2.49 | 0.61 | 79.81 | 77.29 |

We present a comparative study of our Disentangled Inversion Control (DIC) against several established music generation and editing baselines, including AudioLDM 2 (Liu et al., ), MusicGen (Copet et al., ), SDEdit (Liu et al., ), DDIM Inversion Ho et al. (), MusicMagus (Zhang et al., ), and DDPM-Friendly (Manor & Michaeli, ). Utilizing the ZoMo-Bench test set we developed, we assess the quality of the generated audio samples, focusing on two key aspects: Content Preservation and Editing Fidelity. The results, summarized in Table 1, lead to the following conclusions: (1) Our method, MEDIC, significantly outperforms both generation-based and inversion-based models in terms of editing fidelity and audio similarity, demonstrating its effectiveness in addressing complex editing tasks.
(2) Although DDPM-Friendly and MusicMagus show improvements in essential content preservation, they struggle to maintain high text-audio alignment compared to generation models. MEDIC effectively addresses accumulated errors, achieving high audio similarity and an impressive CLAP score, reflecting its superior editing fidelity.
(3) MEDIC also excels in subjective metrics, achieving the best performance in audio quality and content preservation.

Variable Length Comparisons
We further evaluate MEDIC against baseline methods in a variable length setting, with the results presented in Table 2. Our analysis leads to the following conclusions: (1) MEDIC outperforms all baselines across all metrics, demonstrating the effectiveness of our approach in variable-length scenarios. (2) Inversion-based baselines experience a significant degradation in CLAP score, while MEDIC maintains the highest CLAP score of 0.51 on the MusicDelta dataset.

**Table 2: Comparison with baselines in variable length setting.**
| Method | Objective Metrics | Subjective Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $\text{SD}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | CLAP Score$\uparrow$ | MOS-Q$\uparrow$ | MOS-P$\uparrow$ |  |
| AudioLDM 2 | 24.40 | 0.22 | 7.07 | 0.44 | 66.37 | 64.28 |
| MusicGen | 27.71 | 0.23 | 7.70 | 0.46 | 67.41 | 63.76 |
| SDEdit | 28.12 | 0.24 | 13.21 | 0.24 | 62.80 | 62.18 |
| DDIM Inversion | 23.5 | 0.21 | 10.12 | 0.27 | 65.94 | 65.73 |
| MusicMagus | 25.6 | 0.22 | 7.13 | 0.43 | 67.45 | 67.12 |
| DDPM-Friendly | 21.53 | 0.23 | 6.68 | 0.30 | 66.34 | 67.28 |
| MEDIC | 19.5 | 0.20 | 6.58 | 0.51 | 71.62 | 70.18 |

Figure: (a) Spider chart for CLAP score comparisons .
Refer to caption: x3.png

Fine-grained Comparisons on ZoME-Bench
To substantiate the robustness of our method, we delve into fine-grained comparisons across different types of editing. The FAD and CLAP scores are depicted in Figure 3. The insights gleaned from this analysis are as follows:
(1) Across all editing types, our Disentangled Inversion Control surpasses other methods, demonstrating its prowess in handling both rigid and non-rigid editing tasks.
(2) While baselines exhibit capabilities in handling certain rigid editing scenarios, they fall short in executing non-rigid manipulations, as reflected in their inferior performance, especially in the “Change Genre" and “Change Melody" non-rigid editing types.

### 4.2 Ablation Study

The ablation studies presented in this section aim to validate the contributions of the Harmonized Attention Control and Disentangled Inversion to our framework’s overall performance. Additionally, we examine the impact of the classifier-free guidance scale on editing outcomes, with a comprehensive analysis included in Appendix D.

#### 4.2.1 Ablation on Attention Control Methods

**Table 3: Ablation results about different control methods. MSA: Mutual Self-Attention, CA: Cross Attention.**
| Method | $\text{Structure Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | CLAP Score$\uparrow$ |
| --- | --- | --- | --- | --- |
| HAC | 11.97 | 0.15 | 2.49 | 0.61 |
| w/o MSA Control | 14.13 | 0.19 | 2.75 | 0.56 |
| w/o CA Control | 13.78 | 0.18 | 2.50 | 0.58 |
| w/o Harmonic Branch | 12.75 | 0.16 | 2.59 | 0.59 |

To validate the impact of our attention control mechanisms, we perform ablations on the following configurations: Remove Mutual Self Attention Control (w/o MSA Control), Remove Cross Attention Control (w/o CA Control), and without Harmonic Branch (w/o Harmonic Branch). The findings, detailed in Table 3, reveal that: (1) Both cross-attention and mutual self-attention controls individually enhance editing performance. (2) Although the naive combination of mutual self-attention control and cross-attention control improves the preservation and fidelity, it still yields sub-optimal outcomes due to the lack of a progressive harmonic branch. This demonstrates the effectiveness of harmonized attention control.

#### 4.2.2 Ablation on Disentangled Inversion Technique

To demonstrate the soundness of Algorithm 2 and the effective disentanglement of triple branches, we can draw conclusions from Table 4: 1) Incorporating source distance into the target latent and harmonic branch, this leads to a decline in both preservation metrics and CLAP similarity score. 2) Incorporating target distance into the target branch and harmonic branch improves structural integrity and content preservation. However, this modification leads to a significant decrease in the CLAP score, indicating that while structural quality may improve, the alignment with the target prompt suffers. 3) Adding harmonic distance to the harmonic branch leads to a noticeable decline in performance. This finding showcases that introducing excessive deviations in the harmonic branch may adversely affect the overall audio quality and coherence.

**Table 4: Ablation results from the disentangled inversion technique. $[\cdot,\cdot,\cdot]$ denotes adding the distance (line 6 in Algorithm 2). MSE means the mean square error loss between the edited audio features and source audio features.**
| Distance | $\text{Structure Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | $\text{MSE}_{\times 10^{5}}\downarrow$ | CLAP Score$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| $[{\bm{d}}_{src},{\bm{d}}_{src},{\bm{0}}]$ | 14.28 | 0.17 | 2.64 | 4.67 | 0.57 |
| $[{\bm{d}}_{src},{\bm{0}},{\bm{d}}_{src}]$ | 13.17 | 0.17 | 2.55 | 4.59 | 0.58 |
| $[{\bm{d}}_{src},{\bm{d}}_{tgt},{\bm{0}}]$ | 11.24 | 0.15 | 2.44 | 4.32 | 0.56 |
| $[{\bm{d}}_{src},{\bm{0}},{\bm{d}}_{tgt}]$ | 11.12 | 0.14 | 2.41 | 4.29 | 0.56 |
| $[{\bm{d}}_{src},{\bm{d}}_{har},{\bm{0}}]$ | 37.51 | 0.27 | 27.2 | 14.23 | 0.28 |
| $[{\bm{d}}_{src},{\bm{0}},{\bm{0}}]$ | 11.97 | 0.15 | 2.49 | 4.54 | 0.61 |

### 4.3 Qualitative Results

Figure: Figure 4: Visualizations of the source audio’s mel and edited mel-spectrograms by different editing methods.
Refer to caption: x5.png

To complement our quantitative findings, we present a qualitative comparison in Figure 4. Methods such as SDEdit and inversion-based techniques often struggle to balance high editability with preserving melodic content and harmonic structure. In contrast, our Disentangled Inversion Control performs better in precise music editing while preserving structural integrity. We provide additional qualitative results in the Appendix G for further examination.

## 5 Conclusion

In this paper, we explored the burgeoning realm of text-guided diffusion models for audio generation, recognizing their potential to reshape source audio in alignment with specific textual prompts.
We proposed the Disentangled Inversion Control to support both rigid and non-rigid editing tasks. Instead of a two-branch setting, we add an intermediate branch named the harmonic branch to progressively integrate harmonic and melodic information in music by cross-attention control and mutual self-attention control. To counteract the accumulated errors caused by DDIM inversion and CFG, we introduced a simple but effective method named disentangled inversion to separate the diffusion process into triple branches and eliminate the latent discrepancy distance in the source branch.
Our comprehensive evaluations, conducted on the ZoME-Bench—a robust benchmark for music editing comprising 1,100 samples across 10 varied editing categories—attested to the superiority of our methods. And the experiment results on variable length test sets and ablation studies further validated the effectiveness and robustness of our method. We envisage that our work could serve as a basis for future zero-shot music editing studies.

## Appendix A Preliminaries And Analyses

This section introduces the foundational concepts of DDIM sampling and classifier-free guidance as applied to diffusion models for text-guided audio synthesis. It further explores the challenges associated with these methods.

### A.1 Diffusion Models

Text-guided diffusion models aim to map a random noise vector ${\bm{z}}_{t}$ and textual condition ${\bm{c}}$ to an output audio ${\bm{z}}_{0}$, corresponding to the given conditioning prompt. We train a denoiser network $\epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})$ to predict the Gaussian noise $\epsilon\in{\mathcal{N}}(0,{\mathbf{I}})$ following the objective:

$$ $\underset{\theta}{min}{\mathbf{E}}_{{\bm{z}}_{0},\epsilon\in{\mathcal{N}}(0,{ \mathbf{I}}),t\in Uniform(1,T)}||\epsilon-\epsilon_{\theta}({\bm{z}}_{t},t,{ \bm{c}})||^{2}$ (10) $$

Where noise is added to the sampled data ${\bm{z}}_{0}$ according to timestamp $t$. At inference, given a noise vector ${\bm{z}}_{T}$, the noise $i$ is gradually removed by sequentially predicting it using a pre-trained network for $T$ steps.
To generate audios from given ${\bm{z}}_{T}$, we employ the deterministic DDIM sampling:

$$ ${\bm{z}}_{t-1}=\frac{\sqrt{\alpha_{t-1}}}{\sqrt{\alpha_{t}}}{\bm{z}}_{t}+\sqrt {\alpha_{t-1}}(\sqrt{\frac{1}{\alpha_{t-1}}-1}-\sqrt{\frac{1}{\alpha_{t}}-1}) \epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})$ (11) $$

### A.2 DDIM Inversion

While diffusion models have superior characteristics in the feature space that can support various down-stream tasks, it is hard to apply them to audios in the absence of natural diffusion feature space for non-generated audios. Thus, a simple inversion technique known as DDIM inversion is commonly used for unconditional diffusion models, predicated on the presumption that the ODE process can be reversed in the limit of infinitesimally small steps:

$$ ${\bm{z}}^{*}_{t}=\frac{\sqrt{\alpha_{t}}}{\sqrt{\alpha_{t-1}}}{\bm{z}}^{*}_{t- 1}+\sqrt{\alpha_{t}}(\sqrt{\frac{1}{\alpha_{t}}-1}-\sqrt{\frac{1}{\alpha_{t-1} }-1})\epsilon_{\theta}({\bm{z}}^{*}_{t-1},t-1)$ (12) $$

However, in most text-based diffusion models, this presumption cannot be guaranteed, resulting in a perturbation from ${\bm{z}}_{t}$ to ${\bm{z}}^{*}_{t}$.
Consequently, an additional perturbation from ${\bm{z}}^{*}_{t}$ to ${\bm{z}}^{src}_{t}$ arises when sampling an audio from ${\bm{z}}^{*}_{T}$ where $\alpha$ is hyper-parameter:

$$ ${\bm{z}}_{t}=\sqrt{\alpha_{t}}{\bm{z}}_{0}+\sqrt{1-\alpha_{t}}\epsilon$ (13) $$

### A.3 Classifier-free Guidance

Classifier-free Guidance (CFG) (Ho & Salimans, ) is proposed to overcome the limitation of text-conditioned models, where text adherence could be weak. However, a higher guidance scale, $\omega$, which is intended to strengthen the model’s fidelity to the text prompt, inadvertently magnifies the accumulated inversion error. This becomes problematic in editing scenarios where precise control over the audio synthesis is desired. The modified noise estimation in CFG can be expressed as:

$$ $\hat{\epsilon}_{\theta}({\bm{z}}_{t},t,{\bm{c}},\varnothing)=\omega\cdot \epsilon_{\theta}({\bm{z}}_{t},t,{\bm{c}})+(1-\omega)\cdot\epsilon_{\theta}({ \bm{z}}_{t},t,\varnothing)$ (14) $$

where $\varnothing$ = (“”) is the embedding of a null text. This further leads to another perturbation from ${\bm{z}}^{{}^{\prime}}_{t}$ to ${\bm{z}}^{{}^{\prime\prime}}$ due to the destruction of the DDIM process and causes error augmentation as demonstrated in Figure 1.

## Appendix B Benchmark construction

### B.1 General information

Here are the details of our ZoME-Bench dataset (Zero-shot Music Editing Benchmark). This dataset contains 1,000 audio samples, selected from MusicCaps, with each sample being 10 seconds long and having a sample rate of 16k.

We refactor the original captions to express specific edits and divide them into 10 parts, each representing a different type of editing. A sample and details are shown in the following table 5.

**Table 5: Information of ZoME-Bench dataset**
| Editing<br>type id | Editing type | size | origin prompt | editing prompt | editing instruction |
| --- | --- | --- | --- | --- | --- |
| 0 | change instrument | 131 | ambient acoustic [guitar] music | ambient acoustic [violin] music | change the instrument from guitar to violin |
| 1 | add instrument | 139 | metal audio with a distortion guitar [and drums] | metal audio with a distortion guitar | add drums to the piece |
| 2 | delete instrument | 133 | an eerie tense instrumental featuring electronic drums [and synth keyboard] | an eerie tense instrumental featuring electronic drums | remove the synth keyboard |
| 3 | change genre | 134 | a recording of a solo electric guitar playing [blues] licks | a recording of a solo electric guitar playing [rocks] licks | change the genre from blues to rock |
| 4 | change mood | 100 | a recording featuring electric bass with an [upbeat] vibe | a recording featuring electric bass with an [melancholic] vibe | turn upbeat mood into melancholic mood |
| 5 | change rhythm | 69 | a live ukulele performance featuring [fast] strumming and emotional melodies | a live ukulele performance featuring [slow] strumming and emotional melodies | change fast rhythm into slow one |
| 6 | change background | 95 | female voices in unison with [acoustic] guitar | female voices in unison with [electric] guitar | switch acoustic guitar to electric guitar |
| 7 | change melody | 121 | this instrumental song features a [relaxing] melody | this instrumental song features a [cheerful] melody | change relaxing melody into cheerful melody |
| 8 | extract instrument | 111 | a reggae rhythm recording with bongos [djembe congas acoustic drums and electric guitar] | a reggae rhythm recording with bongos | extract bongos from the recording |
| 9 | random | 67 | / | / | / |

### B.2 Annotation Process

We rebuild our caption from captions for Musiccaps offered by  (Agostinelli et al., ). With the help of ChatGPT-4 (OpenAI, ), we rebuild the caption with prompt as follows(take type “change melody" as examples):

Description: “There is a description of a Piece of music, Please judge whether the description has information of melody. If not, just answer “Flase", else change its melody properly into the opposite one, just change the adjective and don’t replace any instrument!
",“blended_word" is [origin melody, changed melody],
“emphasize" is [changed melody],
“blended_word" and “emphasize" are tuples.
Question: (A mellow, passionate melody from a noisy electric guitar)
Answer:(“source_prompt": “A mellow, [passionate] melody from a noisy electric guitar",
“editing_prompt": “A mellow, [soft] melody from a noisy electric guitar",
“blended_word": [“passionate melody", “soft melody"],
“emphasize": [“soft melody"])

Question: (A recording of solo harp music with a dreamy, relaxing melody.)
Answer: (“source_prompt": “A recording of solo harp music with a dreamy, [relaxing] melody.",
“editing_prompt": “A recording of solo harp music with a dreamy, [nervous] melody.",
“blended_word": [“relaxing melody",“nervous melody"],
“emphasize" :[“nervous melody"])
Question: (“A vintage, emotional song with mellow harmonized flute melody and soft wooden percussions")
Answer: (“source_prompt": “A vintage, emotional song with [passionate] flute melody and soft wooden percussions.",
“editing_prompt": “A vintage, emotional song with [harmonized] flute melody and soft wooden percussions.",
“blended_word": [“harmonized flute melody",“passionate flute melody"]),
“emphasize" :[“passionate flute melody"])
Now we have
Question:({origin caption}), Answer(?)"

In the same way, instructions are appended by prompt as follows (take type “change melody" as examples):

Description: “There are two descriptions of different pieces of music divided by &, Please describe the difference you need to give me the results in the following format: Question: this instrumental song features a [relaxing] melody with a country feel accompanied by a guitar piano simple percussion and bass in a slow tempo & this instrumental song features a [cheerful] melody with a country feel accompanied by a guitar piano simple percussion and bass in a slow tempo
Answer: change relaxing melody into cheerful melody
Question: this song features acapella harmonies with a [high pitched] melody complemented by both high pitched female whistle tones and male low pitch tones & this song features acapella harmonies with a [smooth] melody complemented by both high pitched female whistle tones and male low pitch tones
Answer: turn a high pitched melody into smooth melody
Question: a traditional and hopeful song with a harmonizing throaty male vocal and [dissonant] background melody from strings albeit presented in low quality & a traditional and hopeful song with a harmonizing throaty male vocal and [harmonic] background melody from strings albeit presented in low quality
Answer: change dissonant melody into harmonic melody
Now we have Question: [‘source prompt’] & [‘editing prompt’], Answer(?)"

Through this method, supplemented by rounds of manual review, we ensure the quality of this benchmark.

### B.3 Data Format

Taking the first piece as an example, we express our data in JSON format with six keys

“Editing_prompt" refers to the edited caption, while “source_prompt" denotes the original caption. "Blended_word" indicates the subject to be edited, and “Emphasize" represents the word that should be highlighted. “Editing_instruction" provides a description of the editing process. Additionally, in the editing type “delete instrument," we introduce another key, “neg_prompt", which helps reduce the likelihood of deleted instruments reappearing.

## Appendix C Implementation Details

For our evaluation, we utilize the pre-trained AudioLDM 2-Music model (Liu et al., ). Our assessment employs a comprehensive set of metrics, namely CLAP, LPAPS, Structure Distance, and FAD. These metrics are calculated using the CLAP models available in the AudioLDM_eval package, which is accessible at [https://github.com/haoheliu/audioldm_eval](https://github.com/haoheliu/audioldm_eval). In line with the methodology described by  Manor & Michaeli (), we apply a forward guidance of 3 and a reverse guidance scale of 12 for DDPM inversion. For the DDIM inversion, the guidance scale is set to 5, while for SDEdit, we employ a guidance scale of 12. The forward guidance of MEDIC is 1 while the reverse scale is 5. We chose these values by exploring different guidance scales, as discussed in Appendix D. We conduct all experiments in NVIDIA 4090.
For our evaluation, we have selected the public pre-trained AudioLDM 2-Music model (Liu et al., ). To ensure a thorough and multidimensional assessment, we measure performance using a suite of metrics that includes CLAP, LPAPS, Structure Distance, and FAD and conduct on an NVIDIA 4090. The computation of these metrics is facilitated by the CLAP models provided within the AudioLDM_eval package, which is publicly available at [https://github.com/haoheliu/audioldm_eval](https://github.com/haoheliu/audioldm_eval).

Our methodology is aligned with the protocol established by  Manor & Michaeli (), where we have adopted a forward guidance scale of 3 and a reverse guidance scale of 12 for DDPM inversion. In contrast, the DDIM inversion employs a guidance scale of 5, and SDEdit utilizes a guidance scale of 12. For Disentangled Inversion Control, we have determined the forward guidance to be 1 and the reverse scale to be 5. These specific guidance scale values are selected after extensive experimental exploration, the details of which are discussed in Appendix D.

### C.1 Metrics

Objective Metrics
There are details about four metrics to evaluate the performance of our novel Disentangled Inversion Control framework:
(1) CLAP Score (Elizalde et al., ): This criterion evaluates the degree to which the output conforms to the specified target prompt.
(2) Struture Distance (Ju et al., ): Leveraging self-similarity of audio features to measure the structure distance between the source and edited audio.
(3) LPAPS (Iashin & Rahtu, ; Paissan et al., ): An audio adaptation of the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ), this measure evaluates the consistency of the edited audio with the source audio.
(4) FAD (Fréchet Audio Distance) (Kilgour et al., ): Analogous to the FID used in image analysis, this metric calculates the distance between two distributions of audio signals.

Subjective Metrics
To directly reflect the quality of the audio generated, we carry out MOS (Mean Opinion Score) tests. These tests involve scoring two aspects: MOS-Q, which assesses the edited quality of the audio, and MOS-P, which measures the content preservation of edited audio.

For assessing editing fidelity, the evaluators were specifically directed to “Does the natural language description align with the audio?” They were provided with both the audio and its corresponding caption. They were then asked to give their subjective rating (MOS-Q) on a 20-100 Likert scale.

To assess essential content preservation, human evaluators were presented with source audio, target audio, source prompt, and target prompt. They were then asked to answer the question, “To what extent does the target audio retain the essential features of the source audio, such as melody, instrumentation, and overall style?” The raters had to select one of the options: “completely,” “mostly,” or “somewhat,” using a 20-100 Likert scale for their response.

Our crowd-sourced subjective evaluation tests were conducted via Amazon Mechanical Turk where participants were paid $8 hourly.

## Appendix D Quantitative Results

Analyses on Different CFG Scale
The lack of systematic experiments that determine the optimal combination of guidance scales for achieving the
best editing performance, and analysis of how these guidance scales affect the final consequence in both reconstruction and editing, we conduct this experiment to find the best scales.

**Table 6: Ablation Studies on Different Guidance Scale**
| Guidance Scale | Structure | Background Preservation | CLIP Similariy |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Inverse | Forward | $\textbf{Distance}_{\times 10^{3}}\downarrow$ | LPAPS$\downarrow$ | FAD$\downarrow$ | $\text{MSE}_{\times 10^{5}}\downarrow$ | CLAP Score $\uparrow$ |
| 1 | 1 | 8.56 | 0.12 | 1.17 | 3.25 | 0.51 |
| 1 | 2.5 | 11.97 | 0.15 | 2.49 | 4.54 | 0.56 |
| 1 | 5 | 15.99 | 0.17 | 4.22 | 6.07 | 0.61 |
| 1 | 7.5 | 15.99 | 0.17 | 4.22 | 6.07 | 0.59 |
| 2.5 | 1 | 22.80 | 0.20 | 6.39 | 8.65 | 0.30 |
| 2.5 | 2.5 | 14.24 | 0.16 | 2.50 | 5.40 | 0.46 |
| 2.5 | 5 | 14.46 | 0.16 | 3.31 | 5.49 | 0.53 |
| 2.5 | 7.5 | 15.51 | 0.17 | 3.94 | 5.89 | 0.53 |
| 5 | 1 | 29.94 | 0.24 | 9.81 | 11.36 | 0.20 |
| 5 | 2.5 | 29.16 | 0.24 | 9.11 | 11.07 | 0.22 |
| 5 | 5 | 22.15 | 0.20 | 5.59 | 8.40 | 0.36 |
| 5 | 7.5 | 17.57 | 0.18 | 5.57 | 6.67 | 0.48 |
| 7.5 | 1 | 31.41 | 0.25 | 10.62 | 11.92 | 0.20 |
| 7.5 | 2.5 | 31.05 | 0.25 | 10.14 | 11.78 | 0.20 |
| 7.5 | 5 | 29.20 | 0.24 | 9.32 | 11.08 | 0.24 |
| 7.5 | 7.5 | 24.16 | 0.22 | 7.33 | 9.17 | 0.34 |

Inference Time
We compare the inference time of our method with baselines, and the results are compiled in Table 7. MEDIC achieves the comparative inference time with generation models and inversion techniques. We will make an attempt to reduce the inference time of zero-shot music editing in our future work.

**Table 7: Inference Time across different methods.**
| Method | Inference Time |
| --- | --- |
| AudioLDM 2 | 42.5s |
| MusicGen | 83.3s |
| SDEdit | 44.3s |
| DDIM Inversion | 81.6s |
| MusicMagus | 89.0s |
| DDPM-Friendly | 33.3s |
| MEDIC | 92.0s |

## Appendix E Potential Negative Societal Impacts

MEDIC may also lead to potential negative societal impacts that are worthy of consideration. If the data sample of the training model is not diverse enough or biased, the AI-generated music may be overly biased toward one style or element, limiting the diversity of the music and causing discrimination.MEDIC could be used to create fake audio content, such as faking someone’s voice or creating fake musical compositions, posing the risk of fraud and impersonation. Hopefully, all these issues could be taken into consideration when taking the model for real use to avoid ethical issues.

## Appendix F Limitations

In spite of the remarkable outcome of our method, due to the limitation of the generation model we used, we are incapable of instigating a profound change.

Due to the numerous steps it requires (T=200), the duration of computing distance is quite long. Thus, we will implement a more powerful text-to-music generation model to support better editing, while trying to use a consistency model or flow-matching model to achieve high-quality and fast music generation in future work. We will make an attempt to edit more interesting and complex music tasks in the future.

## Appendix G Qualitative Results

For each type in ZoME-Bench, We provide samples to observe the capability of MEDIC intuitively.

### G.1 Change Instrument

In Figure 5, we show the capability of MEDIC to change the instrument. Here we edit the ground truth music piece with the source prompt “a live recording of ambient acoustic [guitar] music" and editing prompt “a live recording of ambient acoustic [violin] music". The difference in instruments can be observed in the Mel-spectrum.

Figure: Figure 5: Editing Type 0 :Change Instrument
Refer to caption: extracted/5930335/figs/mels/t0p0gt.png

### G.2 Add Instrument

In Figure 6, we show the capability of MEDIC to add more instruments. Here we edit the ground truth music piece with the source prompt “a heavy metal instructional audio with a distortion guitar" and editing prompt “a heavy metal instructional audio with a distortion guitar [and drums]". The appearance of the new instrument can be observed in the Mel-spectrum which presents a drum sound of high frequency.

Figure: Figure 6: Editing Type 1 Add Instrument
Refer to caption: extracted/5930335/figs/mels/t1p0gt.png

### G.3 Delete Instrument

In Figure 7, we show the capability of MEDIC to delete instruments. Here we edit the ground truth music piece with the source prompt “a lively ska instrumental featuring keyboard trumpets bass [and percussion] with a groovy mood" and the editing prompt “a lively ska instrumental featuring keyboard trumpets and bass with a groovy mood". The vanishing of the instrument can be observed in the Mel-spectrum.

Figure: Figure 7: Editing Type 2 Delete Instrument
Refer to caption: extracted/5930335/figs/mels/t2p0gt.png

### G.4 Change Genre

In Figure 8, we show the capability of MEDIC to change the genre of a music piece. Here we edit the ground truth music piece with the source prompt “a recording of a solo electric guitar playing [blues] licks" and the editing prompt “a recording of a solo electric guitar playing [rock] licks". The obvious difference in genre can be observed in the Mel-spectrum.

Figure: Figure 8: Editing Type 3 Change Genre
Refer to caption: extracted/5930335/figs/mels/t3p0gt.png

### G.5 Change Mood

Mood is an important attribute of music. In Figure 9, we show the capability of MEDIC to change the mood of a music piece. Here we edit the ground truth music piece with the source prompt “a recording of [aggressive] electronic and video game music with synthesizer arrangements" and editing prompt “a recording of [peaceful] electronic and video game music with synthesizer arrangements". The change of mood can be observed in the Mel-spectrum.

Figure: Figure 9: Editing Type 4 Change Mood
Refer to caption: extracted/5930335/figs/mels/t4p0gt.png

### G.6 Change Rhythm

Rhythm represents the speed of the music. In Figure 10, we show the capability of MEDIC to change the Rhythm of a music piece. Here we edit the ground truth music piece with the source prompt “a [slow] tempo ukelele tuning recording with static" and the editing prompt “a [fast] tempo ukelele tuning recording with static". The change of Rhythm can be observed in the Mel-spectrum. The edited Mel-spectrum is much more intensive.

Figure: Figure 10: Editing Type 5 Change Rhythm
Refer to caption: extracted/5930335/figs/mels/t5p0gt.png

### G.7 Change Background

In Figure 11, we show the capability of MEDIC to change the background of the instrument of a music piece. Here we edit the ground truth music piece with the source prompt “an amateur ukulele recording with a [medium to uptempo] pace" and editing prompt “an amateur ukulele recording with a [steady and rhythmic] pace". The change of instrument background can be observed in the Mel-spectrum.

Figure: Figure 11: Editing Type 6 Change Background
Refer to caption: extracted/5930335/figs/mels/t6p1gt.png

### G.8 Change Melody

In Figure 12, we show the capability of MEDIC to change the melody of a music piece. Here we edit the ground truth music piece with the source prompt “this instrumental song features a [relaxing] melody with a country feel accompanied by a guitar piano simple percussion and bass in a slow tempo" and editing prompt “this instrumental song features a [cheerful] melody with a country feel accompanied by a guitar piano simple percussion and bass in a slow tempo". The change of Melody can be observed in the Mel-spectrum.

Figure: Figure 12: Editing Type 7 Change Melody
Refer to caption: extracted/5930335/figs/mels/t8p0gt.png

### G.9 Extract Instrument

In Figure 13, we show the capability of MEDIC to extract one certain instrument of a music piece. Here we edit the ground truth music piece with the source prompt “a reggae rhythm recording with bongos [djembe congas acoustic drums and electric guitar]" and editing prompt “a reggae rhythm recording with bongos". The change of instruments can be observed in the Mel-spectrum.

Figure: Figure 13: Editing Type 8 Extract Instrument
Refer to caption: extracted/5930335/figs/mels/t9p0gt.png

## Appendix H safeguards

In the processing of the data and models involved in this study, we fully considered the potential risks. We ensure that all data sources are rigorously screened and vetted, and the model we used is absolutely trained from the safe dataset to minimize the security risks of being misused.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: The introduction and methodology has some places that are unclear to me.

1. What is rigid and non-rigid editing in the context of music editing?
2. In 3.3.1 Global Attention Refinement, equation 2, should the second case be $(M_t)\_{i,A(j)}$ instead of $(M_t)\_{i,j}$?
3. In 3.3.1 Local Attention Blends, the definition of $\textrm{Threshold}(\cdot,k)$ does not match the format in equation 3. Also, what are the choices of $k$ and how will different $k$ affect the results?
4. In 3.3.1 Scheduling Cross-Attention Control, the usage of $\textrm{Refine}(\cdot,\cdot)$ does not match the definition in equation 2.
5. In 3.3.1 Scheduling Cross-Attention Control, what is the choice of $\tau_c$ and how will it affect the results?
6. In figure 2, the harmonic branch outputs something that looks like a "rapid guitar music." Is it an observable phenomenon in experiments, or is it just an assumption? Does the upper part handle non-rigid editing and the lower part handle rigid editing only?
7. In table 4, why are there no results for [0, 0, 0]?
soundness: 3
strengths: 1. Good zero-shot editing performance compared to previous STOA. The demo page shows the effective controllability of some music concepts that previous models failed to control.
2. The benchmark is very useful for future researchers on music editing.
3. The methodology of Harmonized Attention Control and Disentangled Inversion Technique is novel, which could help zero-shot editing of other domains.
summary: This paper propose a new approach to do zero-shot music editing by Disentangled Inversion Control, which integrates multiple methods to inject the diffusion process. A novel benchmark is also proposed.
weaknesses: 1. While the experiments are about music editing, the evaluation only uses metrics for general audio editing. Music content-related metrics like chroma distance [1] are missing.
2. The paper does not seem to be clear enough. See questions.
3. The values and effects of the hyperparameters in the paper are unclear, like $k, \tau_c, L$ and $S$. Ablation study or case study by changing these hyperparameters would be helpful to understand the model.
4. While the methodology seems to be general-purposed, the experiments only focus on music editing. This is okay but limits the impact of the paper a bit.
5. Typos and formatting errors. Algorithm 1: Inconsistency use of fonts; formatting error of \hat in $\hat{M}^{\textrm{tgt}}$; $\epsilon_{c_{\textrm{tgt}}}$ should be $\epsilon_{\textrm{tgt}}$. Figure 2: inconsistenct notation $M_{\textrm{tgt}}$ vs. notation in text $M^{\textrm{tgt}}$. Equation 9: $l$ is not defined. Table 6: not referenced in the appendix text.

[1] Zhang, Y., Ikemiya, Y., Xia, G., Murata, N., Martínez-Ramírez, M. A., Liao, W. H., ... & Dixon, S. (2024). Musicmagus: Zero-shot text-to-music editing via diffusion models. arXiv preprint arXiv:2402.06178.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - The benchmark dataset proposed in the paper is a good idea, but upon reviewing it, I found that it only includes a single audio file. Could the authors further clarify what constitutes the ground truth in this context?

- Finally, I am very curious about the computational efficiency of this method. Does it require more time and resources compared to baseline methods?
soundness: 3
strengths: - The main idea of this paper—incorporating mutual self-attention, cross-attention control, and harmonic control—is sensible, even though each module is not entirely novel. The combination of these mechanisms appears effective, as results indicate that combining them enhances model performance in music editing tasks, providing useful insights.

- The paper is thorough in its experimental design, including both subjective evaluations and a variety of objective experiments. The results effectively demonstrate the validity of the chosen methods for the model.

- The discussion of related work is comprehensive.
summary: This paper primarily discusses a method for enhancing the performance of zero-shot music audio editing tasks through multiple control mechanisms, referred to by the authors as Disentangled Inversion Control. Additionally, the paper contributes a benchmarking dataset based on MusicCaps, aimed at evaluating the performance of zero-shot music editing models.
weaknesses: Although this paper is a strong empirically-driven study, there are certain hypothesis-related issues that could be improved.

- First, the paper needs to clarify what is meant by “rigid” and “non-rigid” tasks. These terms appear throughout the paper, but after re-reading the entire text, I still found no clear explanation of what these tasks entail, which left me quite confused.

- The paper actually addresses a text-guided music audio editing task. However, the language and context in the main body do not consistently maintain this focus. Given the current context, I suggest aligning terms in the main text to match the title, shifting from “audio editing” to “music editing.”

- While the proposed multiple control method indeed focuses on different aspects through each control mechanism, whether this approach achieves “disentangled” control is debatable. To demonstrate that the controls are disentangled, the paper should include experiments showing that one control does not interfere with another. While these controls focus on different levels conceptually, they do not intuitively seem orthogonal, making the term “disentangled” potentially misleading. I suggest either adding experiments to confirm this or revising the terminology.

- The paper includes a subjective evaluation, which is commendable. However, the description of this evaluation is incomplete. Typically, subjective evaluations should also describe the gender, age, music background, and musical training distribution of the subjects, which helps with the interpretation of the results. Unlike data annotation, where these factors might be less crucial, they are important here due to potential biases introduced by AMT, and these underlying biases should be considered.

- In addition, hypothesis tests should be conducted for all reported results.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: **Questions**:

1. Objective Metrics

First and foremost, since evaluating audio/music editing tasks is not trivial, I think it is crucial to carefully select the evaluation metrics and design the evaluation protocol. On top of this, I have several questions about them.

- Regarding the FAD, LPAPS, and CLAP Score, could you specify which checkpoints were used to calculate each metric?
- For FAD, if it was calculated using VGG-ish, recent literature (such as in [1]) indicates that this model may not be appropriate for evaluating music samples. To support the effectiveness of the proposed method, I recommend using alternative pretrained models as suggested in [1].
   - For example, based on the correlation between MOS and FAD discussed in [1], it would be more appropriate to use FAD with LAION-CLAP to evaluate musical quality, and to use FAD with DAC/EnCodec embeddings to assess acoustic quality (please see more detail in [1]). 
- For LPAPS, if the authors used the pretrained network from this repository [2], the network was trained on the VGGSound dataset [3] (also see its data filtering process). This raises some concerns regarding its validity for numerical evaluation in music editing. Additionally, the checkpoint provided in that repository [2] has other issues. As noted in this issue [4], the authors of this repository acknowledge problems in the training procedure of the LPAPS itself, and thus, they do not recommend using the LPAPS for at least training purposes. 
    - It would be appropriate to calculate the L2 distance using other audio encoders trained properly. For instance, as in [1], I recommend calculating the L2 distance based on embeddings from audio encoders like LAION-CLAP, DAC, or EnCodec.
- Besides, could you provide at least an intuitive explanation, if not a theoretical one, supporting why LPAPS is suitable for evaluating "consistency"?
- CLAP model: Appendix C refers to this repository [5], but I couldn’t find the CLAP model there. Could you clarify this in more detail?


2. In cases where a source prompt $\mathcal{P}$ is provided, is there a benefit to using the inversion process in L3 in Algorithm 2? It seems that just using the proposed attention control technique in Section 3.3 during reverse sampling alone might be sufficient. From an ODE perspective, the score function at a given timestep should be almost the same in both forward and backward directions in terms of conditioning. The difference between them would be the accumulated errors from $z_{0}$ and $z_{T}$. If L3 were removed, it would no longer be 'inversion'. It would be 'text-guided music editing by attention map control' such as in MusicMagus.

3. The definitions of "rigid" and "non-rigid" tasks mentioned in the Introduction are unclear in the paper, leaving some doubts about the validity of claims regarding the proposed method’s effectiveness. Even the example provided around L321 in Section 3.3.3 does not seem intuitive enough. Could you elaborate more?

4. In Section 2, L143, the authors state, "Differently, we introduce a plug-in-plus method called Disentangled Inversion Control to separate branches, achieving superior performance with considerably fewer computational resources." Was this claim tested thoroughly in the paper? From Table 7, the computational cost of the proposed method appears to be higher than that of the baseline methods (also, it seems that Null-Text Inversion is not included as a baseline).

**Comments**:
- In diffusion model literature, the terms ‘forward process’ and ‘backward process’ typically refer to the process from $z_{0}$ to $z_{T}$ and $z_{T}$ to $z_{0}$, respectively, even when dealing with inversions [6][7]. To minimize unnecessary confusion for readers, I recommend revising the current manuscript to maintain consistency with prior work in fundamental aspects. (In fact, in Appendix C, the authors use the term “forward guidance” naturally.)
- In Figure 1, citations to MusicMagus should be included (for a self-contained perspective). There are instances of subjective terms like “tiny distance,” “small distance,” and “large distance” without clarification on what these distances pertain to. While the intent becomes clearer upon multiple readings, I suggest revisions to improve clarity, allowing readers to grasp the meaning on the first read-through. Additionally, the term "two-branch inversion techniques" does not appear to be a widely recognized term in inversion, I feel.
- Missing explanations for indices $i, j, k$ in Section 3.3.1. Also, Eq (6) is not consistent with Eq (3), (4).
- The values of hyperparameters such as $S, L, \tau$ are not explained in the experiment section.
- Section 3.4, L356–L358: Citing only image-editing literature while discussing audio/music editing seems wired.
- In Appendix C, content from L881 is repeated.

[1] Gui, A., Gamper, H., Braun, S. and Emmanouilidou, D., 2024, April. Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1331-1335). IEEE.

[2] https://github.com/v-iashin/SpecVQGAN

[3] Chen, H., Xie, W., Vedaldi, A. and Zisserman, A., 2020, May. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 721-725). IEEE.

[4] https://github.com/v-iashin/SpecVQGAN/issues/13

[5] https://github.com/haoheliu/audioldm_eval

[6] Song, J., Meng, C. and Ermon, S., 2020. Denoising diffusion implicit models. ICLR 2021

[7] Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J. and Zhu, J.Y., 2023, July. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings (pp. 1-11).
soundness: 2
strengths: To improve the music editing performance of DDIM inversion, the authors did not simply combine Cross-attention control and Mutual self-attention control; they introduced an additional Harmonic Branch to integrate these techniques. Furthermore, they proposed the Disentangled Inversion Technique. By leveraging these methods, they surpass existing music-editing methods in both objective and subjective metrics.

Originality/Contribution:
- Introduction of the Harmonic Branch and Disentangled Inversion Technique for DDIM inversion
- Proposal of ZoME-Bench
summary: The authors propose MEDIC, a training-free music editing method utilizing pretrained diffusion models. MEDIC extends DDIM inversion to enable better music editing. Specifically, it achieves this by first obtaining the noise latent $z_{T}$ through standard DDIM forward sampling. During reverse sampling, MEDIC incorporates Cross-attention control, as proposed in Prompt-to-prompt, and Mutual Self-attention control, as proposed in MasaCtrl, while introducing "Harmonic Branch" for integrating Cross-attention control and Mutual Self-attention control.

Additionally, authors propose Disentangled Inversion Technique. This approach focuses on the difference between the latent $z^{*}_{t}$ obtained during DDIM forward sampling, and the source latent $z^{src}$, to guide the reverse sampling.

Alongside the MEDI, authors also introduce a new benchmark dataset, ZoME-Bench, designed specifically for music editing evaluation.
weaknesses: **Overall**:

The following points represent the overall weaknesses in the current manuscript. Please refer to the detailed explanations in the latter part of the Weaknesses and Questions sections.
1. Insufficient or unclear validation of the effectiveness of the proposed method, which is directly related to the originality of this work. (For more details, see A. in Weaknesses and 1. in Questions.)
2. Unclear motivation for incorporating the inversion process (L3 in Algorithm 2) within the problem setup (where a source prompt $\mathcal{P}$ is provided). (See more details at 2. in Questions.)
3. The contribution of ZoME-Bench to the music-editing field seems somewhat limited. (For further details, see B in Weaknesses.)


**Details**:

A. The validity of the objective metrics used for evaluation remains unclear. For more details, please refer to Question 3.
- Given the ambiguity of these objective metrics, the experimental justification for the advantages of using the Harmonic Branch and introducing the Disentangled Inversion Technique seems insufficient.
- On the other hand, the benefits of combining Prompt-to-prompt and MasaCtrl appear to be adequately validated in subjective evaluation. However, this aspect alone may not be sufficient to fully support the originality and strengths of this work.

B. While I agree with the importance of introducing standardized benchmarks in audio/music editing and appreciate the effort to create a dataset, ZoME-Bench still has some limitations. ZoME-Bench includes original audio samples, original prompts, editing prompts/types/instructions, etc., but it lacks edited audio samples that define how the source samples are supposed to be edited in a certain sense. In this respect, although ZoME-Bench contributes to standardizing editing instructions, it leaves unresolved the larger issue of verifying the edited results, which remains a significant challenge in audio/music editing evaluation. Therefore, while ZoME-Bench contributes to the audio/music-editing benchmark to some extent, its impact is limited.
(I understand how difficult it is to construct such edited audio samples. I mention this point to assess and clarify the degree of the contribution and originality of the ZoME-Bench proposal.)

C. To improve the paper's presentation quality, I recommend revising the current manuscript. (See Comments in Questions.)

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Most of my questions are brought up throughout the previous section. In general, there are a number of what I think are typos (but I may be misunderstanding things), as well as my questions regarding the definitions of "rigid" and "non-rigid", which acronym-like terms refer to what, and questions regarding baseline reproduction and comparison.
soundness: 2
strengths: - The proposed method overall seems reasonably novel and well-motivated. Much space is given to explaining the facets of their method, and graphical comparison to existing methods like MusicMagus is very appreciated.
- Ablations of proposed method are solid and thorough, and shows clear strengths to the design choices the authors made.
summary: This paper presents MEDIC, a novel method for training-free editing of musical audio with freeform text using pre-trained text-to-audio diffusion models. The paper also presents ZoME-Bench, a new editing benchmark for musical audio editing.
weaknesses: Overall, while the proposed method is solidly novel and seems to perform better than current SOTA training-free editing approaches, issues in the overall clarity of the paper, evaluation suite, and in particular the proposed benchmark overweigh the contributions and thus I recommend rejection.



# Overall Clarity
---
The paper contains a number of grammatical errors, incorrect names of things, and incorrect citations.

- The “Branch” term (line 070) is introduced without explanation.
- "rigid" and "non-rigid" edits are introduced without explanation. These terms are not elucidated by figure 1, as there seems to be no difference between the input prompt and the "non-rigid" prompt.
- (Line 143) What is “plug-in-plus”? Is this a typo for plug-and-play?
- (Line 369) What is ZoMo-bench? Is this a typo from ZoME-Bench?
- It should be “MedleyDB” not “MelodyDB” (line 370)
- Appendix C seems to contain the same information duplicated twice (874-880 and 881-892)
- MusicCaps has the incorrect citation on line (161), which should point to Agostinelli et al.

While the above issues are minor, more importantly there is a distinct lack of clarity in the methodological contributions and what are contributions from the authors.
- In particular, it feels like the paper suffers from overuse of acronym-like terms. Between Disentangled Inversion Control, Harmonized Attention Control, and Disentangled Inversion Technique, it is hard to tell what is a subset of what.
- In section 3.3.1. even more acronym-like terms are introduced (Global Attention Refinement and Local Attention Blend), and it is unclear both A) why are these getting special names if they are from existing work where these terms are not used and B) if these are novel contributions, this is not made explicit.
- In Equation 2, there is a lone $M_t$, and it is unclear which attention map this refers to
- In equation 3/4, the threshold functions takes $w_{src/tgt}$ as argument but in equation 6 it is only a function of the mask and the threshold
- Both 3.3.1. and 3.3.2 are realtively hard to follow without an intimate knowledge of past (image-domain) works. These sections as a whole could be made more clear by drawing specific examples to editing tasks.
- 3.3.3 (and algorithm 2), it is a bit unclear whether “forward” refers to the forward diffusion process (i.e. data $\rightarrow$ noise) or the “forward” pass of the model. I think it is the latter, and if so I think this nomenclature should be fixed to the “backward” step of the diffusion process to bring it in line with standard diffusion terminology.
- Paragraph 2 continuously refers to some caption include “with noise”, but this does not exist anywhere
- What is MEDIC? It is sometimes used as (what I can infer) to be the main method, but this is never stated and it is unclear what this refers to specifically.

# Evaluation Suite
---
 - A key hyperparameter in all these editing-through-inversion methods is the $T_{start}$ parameter, which should in theory determine overall edit strength. It is unclear how this hyperparameter is chosen for the proposed method (and admittedly it may be ignored with using all 200 steps), but it is unclear how it was chosen for the baseline comparisons (such as DDPM-friendly).
- As DIC is a text-based editing method, it is unclear how the MusicDelta section of MedleyDB was used for the task. If the authors used the prompts generated from Manor and Michaeli, this should be explicitly stated. In general, it is unclear what sorts of edits this dataset even contains other than that the samples are longer.
- LPAPS results seem odd when comparing to Manor and Michaeli, as the values for table 2 are in theory on the same dataset, yet are all considerably lower (by about 10x) than the values reported in Manor and Michaeli. Similar inconsistencies hold for FAD, and in general it is unclear why the results for DDPM-friendly are different from the original paper (as supposedly this is the same data).
- Standard error bars should be reported for the subjective listening results, as all the values (with the exception of SDEdit) are quite close, and it is thus unclear whether the differences are statistically significant. In particular, it is also not stated how many subjects they used for the listening test and how many samples of each editing task were tested, as statistical significance should be calculated on average scores for each stimulus aggregated across users to assess whether such results would extrapolate to additional results from the proposed method.
- FAD is a rather odd metric to be using in this context with paired samples, both because A) it is an unpaired, distributional metric that ignores the structural relevance between pairs of samples and B) FAD has documented clear instability in small sample sizes for estimating the covariance matrix of each Gaussian (with generally needing 2.5k-5k+ samples to estimate this reliably [1]), thus making results somewhat unstable given ZoME-Bench’s size of only 1100 samples. Other metrics such as CLAP Maximum Mean Discrepancy (MMD) could be better suited, but in general it would make more sense to compare the FAD of generated samples to some existing reference set (such as Song Describer), as it is really only a measure of audio realism than anything on similarity to reference signals (which the text should reflect).
- The argument in the “Qualitative Results” section is reasonably heuristic. From visually inspecting the spectrograms, it is not clear of any structural relevance to the original source audio, and simply saying “the change of ___ can be seen in the Mel-spectrum” is insufficient to point to anything meaningful (though admittedly, the utility of spectrograms as a visualization here is not great). However, I think the overall success of the proposed method is somewhat overstated, as most of the examples provided in the **audio demo samples** do not actually perform the target edit and preserve the non edited content at the same time, with most samples sacrificing one of these two facets (which the authors identify as the most important parts of the editing process).

# Proposed Benchmark
---
My biggest issue with the paper is the proposed benchmark dataset of ZoME-Bench, as it seems to contain a number of logical flaws that severely limit its utility as a public benchmark.
-  It is odd that as the first “editing benchmark”, there is no actual ground truth information about the correctness of the edit itself. If it is only being assessed by CLAP score, this implicitly assumes that 1-2 word changes in the CLAP text prompt return meaningful changes in the input embedding in order to assess this change, which is assumed without support here. One could imagine here that in a truly suboptimal sense, a model could simply prioritize an edit that does absolutely nothing to the source audio but is able to change the CLAP embedding of the output, which would theoretically achieve perfect results on the benchmark. As the benchmark is a core contribution of the present work, the authors should either have ground truth target audio samples for each source audio (which would be easily doable for some of the instrument tasks if one had source-separated tracks), and/or at least followed the growing standard practice [2] in editing evaluation and use pretrained discriminative models to help assess the edit fidelity of more fine-grained tasks, which is fully doable in this case (such as using instrument tagging models for edits 0/1/2/8 or genre classifiers for edits 3/4).
- Many of the tasks are similar to previous work (AUDIT / InstructME) in being source separation tasks (0/1/2/8), and thus a much more natural choice for this benchmark would include source separated tracks in order to actually assess these edits that have real ground truth answers. While it is still unclear where the text captions came from for the MedleyDB subset (i.e. if they came from the Manor and Michaeli), it is odd that the MedleyDB subset was not used for creating the benchmark, as it seems readily available and has separated tracks for the instrument-based tasks, thus giving a possible avenue for ground truth targets.
-  The paper in particular is missing a rigorous definition of what “rigid” and “non-rigid” mean in the context of text-based audio editing, and why they have deemed certain editing tasks in one category or another. For the rest of my point here, I assume rigid means “content-based” and non-rigid means “style-based,” as that is what the paper seems to imply and is inline with past image domain works. For example, it is unclear why “instrument change” is referred to as a non-rigid task, given that in theory the change of an instrument should preserve all harmonic and melodic content and only reflect timbral changes (as it would be if a guitar player was changed to a violin but played the same part). Unlike in image domain tasks (where edits can mostly be grouped into those than edit a particular masked / bounding box region of the source image vs. ones that make global edits), this notion of region-concept cooccurrence does not exist in audio and thus porting over the definitions of “rigid” and “non-rigid” is not applicable out of the box.
- In general, I think a number of the tasks proposed do not make for an informative benchmark.  Tasks 3/4/5/6/7 (genre/mood/rhythm/background/melody) are ill-defined, as these conceptual buckets do not disentangle content vs. stylistic changes in the audio, and seem to be rather divorced from how actual musicians talk about semantic changes in music. As examples:
  - For genre, if “blues” changes to “rocks” then what changes? Is this a reflection of content changing (such as chords being simplified and melodic lines using fewer blues scales) or of stylistic changes (micro-timing, ornamentation, guitar fx)?
  - If “fast” is changed to “slow”, should the entire content be slowed down (thus reflecting a content-based change that can only be seen as “stylistic” if realignment is assessed between the original and now slower edit) or is this just a measure of decreased density of perceptual onsets?
  - If a “relaxing” melody is changed to a “cheerful” one, does this reflect changes in the pitch values, the rhythmic interpretation, both, or neither?
- While this is somewhat up to subjectivity, none of these semantic tasks seem non-rigid to me (as they all involve some amount of content preservation with stylistic changes). If these are meant to allow content-based changes, this should be explicitly stated, and in general, I’m hesitant to even phrase such hypothetical tasks as “edits” in the first place (as something has to stay the same for it to be considered an edit).


Between the issues with the non-rigid tasks, lack of ground truth for the rigid tasks, and over-reliance on CLAP similarity as a measure of edit accuracy, the overall use of this benchmark for standardizing and improving music editing is quite limited. To improve the paper, I think that either completely focusing on the MedleyDB subset and mostly dropping the proposed benchmark from the paper (as the methodological contributions stand on their own) or performing the significant work to improve the benchmark (and/or justifying why CLAP similarity can be used as a ground truth signal so heavily) would both be valid options, as the present version of the benchmark is my main concern.

[1] Jayasumana, Sadeep et al. “Rethinking FID: Towards a Better Evaluation Metric for Image Generation.” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 9307-9315.

[2] Basu, Samyadeep et al. “EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods.” ArXiv abs/2310.02426 (2023): n. pag.