High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching

## Abstract

Abstract We introduce MelodyFlow , an efficient text-controllable high-fidelity music generation and editing model.
It operates on continuous latent representations from a low frame rate 48 kHz stereo variational auto encoder codec.
Based on a diffusion transformer architecture trained on a flow-matching objective the model can edit diverse high quality stereo samples of variable duration, with simple text descriptions.
We adapt the ReNoise latent inversion method to flow matching and compare it with the original implementation and naive denoising diffusion implicit model (DDIM) inversion on a variety of music editing prompts.
Our results indicate that our latent inversion outperforms both ReNoise and DDIM for zero-shot test-time text-guided editing on several objective metrics.
Subjective evaluations exhibit a substantial improvement over previous state of the art for music editing. Code and model weights will be publicly made available. Samples are available at https://melodyflow.github.io .

## 1 Introduction

Text-conditioned music generation has made tremendous progress in the past two years (Schneider et al., ; Huang et al., ; Agostinelli et al., ; Copet et al., ; Ziv et al., ; Liu et al., ; Li et al., ; Prajwal et al., ).
The prevailing method for audio representation involves compressing the waveform into a series of discrete or continuous tokens, and then training a generative model on top of those.
Two dominant generative model architectures have emerged, one based on autoregressive Language Models (LMs) (Agostinelli et al., ; Copet et al., ), the other on diffusion (Schneider et al., ; Huang et al., ; Liu et al., ; Li et al., ; Prajwal et al., ).
A third method sometimes referred to as discrete diffusion relies on non-autoregressive masked token prediction (Ziv et al., ; Garcia et al., ).
The target level of audio fidelity depends on the models and some have already successfully generated 44.1 kHz or high stereo signals (Schneider et al., ; Li et al., ; Evans et al., ).

The increasing popularity of diffusion models in computer vision has led to the emergence of a new area of research focused on text-controlled audio editing (Wang et al., ; Lin et al., ; Garcia et al., ; Wu et al., ; Novack et al., ; Zhang et al., ; Manor & Michaeli, ).
The sound design process often involves multiple iterations, and using efficient editing methods is a key approach to achieving this effectively.
Music editing encompasses a wide range of tasks, including but not limited to: inpainting/outpainting, looping, instrument or genre swapping, vocals removal, lyrics editing, tempo control, and recording conditions modification (e.g. from studio quality to a concert setting).
Recent works have addressed some of these tasks using specialized models (Wang et al., ; Garcia et al., ; Lin et al., ; Wu et al., ; Copet et al., ) or zero-shot editing methods from the computer vision domain, which are exclusive to diffusion models (Novack et al., ; Zhang et al., ; Manor & Michaeli, ).
Despite recent efforts, no approach has yet shown the ability to perform high-fidelity generic style transfer across various music editing tasks. This limitation can be attributed to several factors, including insufficient high-quality data, inadequate foundational music generation models, and design choices that fail to generalize effectively to diverse editing tasks.
Inference speed is crucial for creatives, and the music domain presents a unique challenge due to the high-fidelity (48 kHz stereo) requirement in the sound design process.
Recently Lipman et al. () proposed the Flow Matching (FM) generative modeling formulation, which involves constructing optimal transport paths between data and noise samples.
Flow Matching (FM) offers a more robust and stable approach to training diffusion models, with the added benefit of faster inference.
This method has been successfully applied to train foundational speech (Le et al., ) and audio (Vyas et al., ) generative models.
For the music domain Prajwal et al. () utilized a two-stage FM model for text-guided music generation, where the first stage generates semantic features and the second stage generates acoustic features.

In this work we present MelodyFlow, a single-stage text-conditioned FM model designed for instrumental music generation and editing.
The model operates on continuous representations of a low frame rate Variational Audio Encoder (VAE) codec. Additionally, thanks to the versatility of FM, MelodyFlow is compatible with any zero-shot test-time editing method such as DDIM inversion (Song et al., ) or ReNoise (Garibi et al., ).
We enhance the editability of the FM inversion by adapting the latent inversion of Garibi et al. () to the FM formulation.
Both our objective and subjective evaluations on music editing indicate that MelodyFlow can support a diversity of editing tasks on real songs without any finetuning, achieving fast music editing with remarkable consistency, text-adherence and minimal quality loss compared with original samples.
In addition we conduct an ablation study on the importance of the key design choices on the overall model quality/efficiency trade off.

Our contributions: (i) We introduce the first of its kind single-stage text-to-music FM model to generate and edit 48 kHz stereo samples of up to 30 seconds, with enhancements in both the audio latent representation and generative model, striking a better balance between quality and efficiency. (ii) We explore a novel regularized FM inversion method capable of performing faithful zero-shot test-time text-guided editing on various axes while maintaining coherence with the original sample. (iii) We publicly release the code and model weights to foster research on music editing.

## 2 Method

MelodyFlow combines a continuous audio codec, a text-conditioned Diffusion Transformer (DiT) FM model and a regularized latent inversion method.
The model can perform text-guided editing of real or generated audio samples.
The overall editing process is depicted in the Figure 1.

Figure: Figure 1: Overview of the MelodyFlow editing process. A waveform is encoded into $\mathbf{x}_{src}$ before being fed to the ODE solver. Step-by-step, the DiT predicts the velocity $\delta$ from data to noise, while being regularized against the prediction of an artificially constructed $\mathbf{\tilde{z}}_{t}$ so as to enhance editability. Once the target inversion flow step $T_{edit}$ has been reached, the model is used in the classic generation setting (bottom of the Figure, from right to left), except that the starting latent $\mathbf{z}_{t_{edit}}$ has been estimated so as to achieve better editability and consistency with the source waveform.
Refer to caption: x1.png

### 2.1 Latent audio representation

Our codec derives from EnCodec (Défossez et al., ) with additional features from the Descript Audio Codec (DAC) (Kumar et al., ) (snake activations, band-wise STFT discriminators) and Evans et al. () (KL-regularized bottleneck, perceptual weighting).
A convolutional auto-encoder encodes the waveform into a sequence of latent bottleneck representations, its frame rate function of the convolution strides.
Audio fidelity is enforced by multi-scale STFT reconstruction losses complemented by the sum and difference STFT loss for stereo support (Steinmetz et al., ).

### 2.2 Conditional flow matching model

Given an audio sample $\mathbf{a}\in\mathbb{R}^{D\times f_{s}}$, a sequence $\mathbf{x}\in\mathbb{R}^{L\times d}$ of latent representations is extracted by the neural codec.
FM models the optimal transport paths that map a sequence $\mathbf{\epsilon}\in\mathbb{R}^{L\times d}\sim\mathcal{N}(0,I)$ to $\mathbf{x}$ trough a linear transformation - function of the flow step $t$ - following equation 2.2.

$$ $\mathbf{z}_{t}=t\mathbf{x}+(1-t)\mathbf{\epsilon},t\in[0,1]$ $$

During training, $t$ is randomly sampled and the DiT $\Theta$ is trained to estimate $d\mathbf{z}_{t}/dt$ conditioned on $t$ and a text description $c$.

$$ $d\mathbf{z}_{t}/dt=v_{\Theta}(\mathbf{z}_{t},t,c)=\mathbf{x}-\mathbf{\epsilon}$ $$

By design, after training, the model can be used with any ODE solver to estimate $\mathbf{x}=\mathbf{z_{1}}$ given $\mathbf{\epsilon}=\mathbf{z_{0}}$ (and vice versa), and a text description.
The text-to-music inference happens as such: starting from a random noise vector $\mathbf{\epsilon}\in\mathbb{R}^{L\times d}\sim\mathcal{N}(0,I)$ and a text description $c$ of the expected audio the ODE solver is run from $t=0$ to $t=1$ to estimate the most likely sequence of latents $\mathbf{x}_{generated}$.

$$ $\mathbf{x}_{generated}=\textbf{ODE}_{0\xrightarrow{}1}(\epsilon,c)$ $$

After the latents have been estimated they are fed to the codec decoder to materialize the waveform.
Kingma & Gao () show that the flow step sampling density during training plays an important role in model performance.
In our implementation $t$ is sampled from a logit-normal distribution (Karras et al., ; Esser et al., ).

### 2.3 Text-guided editing through latent inversion

Due to the bijective nature of the FM formulation (where given a text condition, each latent sequence is mapped to a single noise vector), the model is compatible with existing latent inversion methods such as DDIM inversion (Song et al., ).
Given the latent representation $\mathbf{x}_{src}$ of an existing audio with an optional accompanying caption $c\in\{\varnothing,c_{src}\}$, the model can estimate its corresponding noise (or intermediate) representation $\mathbf{z}_{t_{edit}}=\textbf{ODE}_{t_{edit}\xleftarrow{}1}(\mathbf{x}_{src},c)$ by running the ODE solver in the backward direction until an intermediary time step $t_{edit}$ (top of the Figure 1).
Given the intermediary representation $\mathbf{z}_{t_{edit}}$, the ODE forward process can be conditioned on a new text description $c_{edit}$ that materialises the editing prompt: $\mathbf{x}_{edit}=\textbf{ODE}_{t_{edit}\xrightarrow{}1}(\mathbf{z}_{t_{edit}}
,c_{edit})$.
A good inversion process should accurately reconstruct the input when $c_{edit}=c_{src}$, as shown in equation 2.3.

$$ $\mathbf{x}_{edit}=\textbf{ODE}_{t_{edit}\xrightarrow{}1}(\textbf{ODE}_{t_{edit }\xleftarrow{}1}(\mathbf{x}_{src},c\in\{\varnothing,c_{src}\}),c_{src})\approx \mathbf{x}_{src}$ $$

In such case when swapping $c_{src}$ for $c_{edit}$ in the $t_{edit}\xrightarrow{}1$ forward direction, the expectation is for the generated audio to preserve some consistency with the source while being faithful to the prompt.
However in practice it was observed by Mokady et al. () that DDIM inversion suffers from poor editability due to the classifier free guidance.

### 2.4 Regularized latent inversion

Even though FM consists in estimating straight trajectories, in practice those are never completely straight and the edited samples do not preserve enough consistency with the source.

- 1.
The distribution of predicted velocities tends to shift away from that of training due to the classifier free guidance (Mokady et al., ), which can lead to divergence of the inversion trajectory. This was observed by Parmar et al. () with $\epsilon$-prediction, which they address by adding an autocorrelation regularization during inversion to preserve the statistical properties of the predictions.
- 2.
Any pair of successive $(\mathbf{z}_{t},\mathbf{z}_{t-\Delta t})$ along the inversion path usually has estimated velocities $v_{\Theta}(\mathbf{z}_{t},t,c)\neq v_{\Theta}(\mathbf{z}_{t-\Delta t},t-\Delta
t
,c)$, which affects reversibility (hence the consistency with the source sample).
Building a fully reversible inversion path requires estimating $\mathbf{z^{\prime}}_{t-\Delta t}$ such that $v_{\Theta}(\mathbf{z}_{t},t,c)\approx v_{\Theta}(\mathbf{z^{\prime}}_{t-\Delta
t
},t-\Delta t,c)$, for example following Garibi et al. ().

ReNoise (Garibi et al., ) addresses those two problems by combining both $\epsilon$-prediction regularization and reversible inversion trajectory estimation.
Applying ReNoise to FM requires either (1) reformulating FM as $\epsilon$-prediction or (2) adapting the regularization mechanism.
Indeed since our FM model predicts the velocity $v_{\Theta}(\mathbf{z}_{t},t,c)=\mathbf{x}-\mathbf{\epsilon}$ and ReNoise operates on noise predictions, applying ReNoise to FM (1) requires subtracting the source latent $\mathbf{x}_{src}$ from $v_{\Theta}$ to try and isolate and regularize $\mathbf{\epsilon}$ directly.
However in such setting the inversion diverges when conditioning on text ($c_{src}$) and using CFG (appendix A.2.4), likely due to $v_{\Theta}(\mathbf{z}_{t},t,c)-\mathbf{x}_{src}$ not properly removing the signal component of $\mathbf{x}-\epsilon$ at lower flow steps.
To prevent this behavior we propose to (2) directly regularize the FM prediction using only the KL regularization from Garibi et al. ().
An thorough comparison between the considered approaches can be found in the sections 4.3.1, 4.3.2 and 4.5.2.

The Algorithm 1 details our proposed inversion. Each iteration consists in estimating a reversible inversion point $\mathbf{z}_{t-\Delta t}$ from a source point $\mathbf{z}_{t}$ such that $v_{\Theta}(\mathbf{z}_{t-\Delta t},t-\Delta t,c)\approx v_{\Theta}(\mathbf{z}_
{t},t,c)$. In such case the jump from $\mathbf{z}_{t}$ to $\mathbf{z}_{t-\Delta t}$ is considered reversible. This is done iteratively in $K$ steps following the convergence property of Garibi et al. ().
During each of those steps, the model prediction is regularized against the prediction of an artifically constructed $\mathbf{\tilde{z}}_{t-\Delta{t}}$ (also shown in the Figure 1).

Figure: Algorithm 1 Proposed regularized FM inversion

### 2.5 Improving Flow Matching for text-to-music generation

#### 2.5.1 Codec bottleneck

Recently Prajwal et al. () trained a two-stage music FM model on continuous latent representations, but both the semantic and acoustic latent representations where trained with a discretization objective (HuBERT semantic features and RVQ-regularized codec).
The concurrent work of Evans et al. () demonstrated long form music generation capabilities by using a KL-regularized bottleneck in their codec with a temporal downsampling as low as 21.5 Hz.
However none of these works have carefully investigated the influence of the bottleneck regulariser on both music reconstruction and generation performance, all other things being equal.
Indeed Rombach et al. () - a seminal work on VAE for image generation - note that LDMs trained in VQ-regularized latent spaces achieve better sample quality than KL-regularized ones.
Our ablation in section 4.4 leads to a different conclusion.
Using a KL-regularizer achieves indeed better music reconstruction and generation performance for a much lower frame rate, which is key for faster inference.

#### 2.5.2 Minibatch coupling

Tong et al. () and Pooladian et al. () expanded over prior work on FM modeling by sampling pairs $(\mathbf{x},\mathbf{\epsilon})$ from the joint distribution given the by the optimal transport plan between the data $\mathbf{X}=\{{\mathbf{x}^{(i)}}\}_{i=1}^{B}$ and noise $\mathbf{E}=\{{\mathbf{\epsilon}^{(i)}}\}_{i=1}^{B}$ samples within a batch of size $B$. Essentially this translates into running the Hungarian algorithm so as to find the permutation matrix $\mathbf{P}$ that minimizes $||\mathbf{X}-\mathbf{P}\mathbf{E}||_{2}^{2}$.
They demonstrate it results in straighter optimal transport paths during inference (that are closer to the theoretical linear mapping assumption between noise and data samples) and consequently offers better quality-efficiency trade offs. We shed light on the importance of mini-batch coupling in sections 4.5.1 and 4.5.2 where we underline the overall benefit of our FM model design choices on both music generation and editing.

## 3 Experimental setup

### 3.1 Model

MelodyFlow uses a DiT of sizes 400M (small) and 1B (medium) parameters with U-shaped skip connections Bao et al. ().
The model is conditioned via cross attention on a T5 representation (Raffel et al., ) computed from the text description of the music.
The model integrates a specific L-shaped self-attention mask meant to better generalize to different segment lengths during inference (appendix A.2.2).
The flow step is injected following Hatamizadeh et al. (). Minibatch coupling is computed with torch-linear-assignement(^1^11[https://github.com/ivan-chai/torch-linear-assignment](https://github.com/ivan-chai/torch-linear-assignment)).
MelodyFlow-small (resp. MelodyFlow-medium) is trained on latent representation sequences of 32 kHz mono (resp. 48 kHz stereo) segments of 10 (resp. 30) seconds, encoded at 20 Hz frame rate (resp. 25 Hz).
From the codec perspective the only difference between encoding mono or stereo waveform is the number of input (resp. output) channels for the first (resp. last) convolution of the encoder (resp. decoder): 1 for mono and 2 for stereo.
The appendix A.2.1 specifically investigates the impact of encoding stereo instead of mono signals on both reconstruction and generation performance.
More details regarding audio representation and FM model implementation and training are provided in the appendix A.1.

### 3.2 Generation and editing

For text-to-music generation we use the midpoint ODE solver from torchdiffeq with a step size of $0.03125$.
A classifier free guidance (CFG) of $4.0$ is chosen after grid search (appendix A.2.4).
For music editing we use the same configuration for DDIM inversion. For ReNoise and MelodyFlow we use a longer step size of 0.04 to account for the additional forward passes induced by the reversible trajectory estimation.
For DDIM inversion this gives a total of 64 inversion and 64 generation steps (e.g. forward passes through the DiT).
For ReNoise and MelodyFlow the inversion takes 25 steps (each of them requires 4 iterations for the reversibility estimation) and 25 forward steps, for a total of 125.
In summary MelodyFlow’s inversion is run with $S=25,K=4,w_{0}=w_{1}=0,w_{2}=2,w_{3}=3$ and $\lambda_{KL}=0.2$.

### 3.3 Datasets

##### Training

Our training dataset is made of 10K high-quality internal music tracks and the ShutterStock and Pond5 music collections with respectively 25K and 365K instrument-only music tracks, totalling into 20k hours.
All datasets consist of full-length music sampled at 48 kHz stereo with metadata composed of a textual description sometimes containing the genre, BPM and key.
Descriptions are curated by removing frequent patterns that are unrelated to the music (such as URLs).
For 32 kHz mono models the waveform is downsampled and the stereo channels are averaged.

##### Evaluation

For the main text-to-music generation results we evaluate MelodyFlow and prior work on the MusicCaps dataset (Agostinelli et al., ).
We compute objective metrics for MelodyFlow and report those from previous literature.
Subjective evaluations are conducted on a subset of 198 examples from the genre-balanced set.
For ablations we rely on an in-domain held out evaluation set different from that of Copet et al. (), made of 8377 tracks.
The same in-domain tracks are used for objective editing evaluations.
Subjective evaluations of edits are run on a subset of 181 higher fidelity samples from our in-domain test set with LLM-assisted designed prompts (more details in appendix A.1.3).

### 3.4 Metrics

We evaluate MelodyFlow using both objective and subjective metrics following the evaluation protocol of Kreuk et al. () and Copet et al. () for generation.
Reported objective metrics are the Fréchet Audio Distance (FAD) (Roblek et al., ) with VGGish embeddings (Hershey et al., ), the Kullback–Leibler divergence (KLD) with PASST audio encoder (Koutini et al., ) and CLAP(^2^22[https://github.com/LAION-AI/CLAP](https://github.com/LAION-AI/CLAP)) cosine similarity (Elizalde et al., ).
For music editing evaluations we compute the average L2 distance between the original and edited latent sequences (LPAPS (Iashin & Rahtu, )), FADedit between the distribution of source and edited samples and CLAPedit between the edited audio and the editing prompt.
Subjective evaluations relate to (i) overall quality (OVL), and (ii) relevance to the text input
(REL), both using a Likert scale (from 1 to 5).
Additionally for music editing evaluations we report (iii) editing consistency (CON).
Raters were recruited using the Amazon Mechanical Turk platform and all samples were normalized to -14dB LUFS (Series, ).
For stereo samples objective evaluation the signal is down mixed into mono prior to metrics computation.
For subjective ratings we keep the original audio format generated by each model.
A screenshot of the evaluation form is presented in appendix A.1.4.

## 4 Results

### 4.1 Text-guided music editing

**Table 1: Comparison to baselines on text-guided high fidelity music editing of samples from the in-domain test set, using LLM-assisted editing prompts.**
| Model | Method | Ovl. $\uparrow$ | Rel. $\uparrow$ | Con. $\uparrow$ | Avg. $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| AudioLDM 2-music | DDPM inv. | 2.48$\pm$0.07 | 2.36$\pm$0.08 | 2.72$\pm$0.09 | 2.52 |
| MusicGen-melody | Chroma cond. | 2.57$\pm$0.08 | 2.46$\pm$0.09 | 2.14$\pm$0.07 | 2.39 |
| MelodyFlow-medium | Reg. inv. | 2.72$\pm$0.08 | 2.72$\pm$0.07 | 2.61$\pm$0.10 | 2.68 |

We compare MelodyFlow-medium with existing open source music editing implementations, namely MusicGen-melody
and AudioLDM 2 with DDPM inversion (following Manor & Michaeli ()).
The Table 1 presents the main music editing subjective evaluation results.
MelodyFlow outperforms both baselines on the quality and text-fidelity axes.
MusicGen-melody specifically underperforms consistency-wise while AudioLDM 2 suffers from lower text adherence.
Indeed during our listening tests we observe that AudioLDM 2 with DDPM inversion sometimes only generates a distorted version of the original track, hence does not take into account the editing prompt and keeps a strong similarity with the original.
This also explains why consistency-wise MelodyFlow lags slightly behind AudioLDM 2.
Averaging on the three axes MelodyFlow sets a new baseline for zero-shot music editing at test-time.

### 4.2 Text-to-music generation

Text-to-music generation performance is reported in the Table 2.
For text-to-music qualitative evaluations we compare MelodyFlow to three baselines that also support both generation and editing: MusicGen, AudioLDM 2, Stable-Audio.
For MusicGen and AudioLDM 2 we use the available open source implementations and for Stable-Audio we use the public API (as of Wed. May 14 2024, AudioSpark 2.0 model version).
MelodyFlow achieves comparable performance with MusicGen, both lagging slightly behind Stable-Audio in terms of human preference.
We do not report objective metrics on Stable-Audio as none were reported on the full MusicCaps benchmark Evans et al. ().
We do not run any subjective evaluation against MAGNeT but report their objective metrics and latency values.
MelodyFlow achieves remarkable efficiency with only 64 inference steps.

**Table 2: Comparison to text-to-music baselines. We report the original objective metrics for AudioLDM 2 and MusicGen. For subjective evaluations we report mean and CI95.**
| Model | FAD${}_{\text{vgg}}\downarrow$ | Kl $\downarrow$ | CLAP${}_{\text{sim}}\uparrow$ | Ovl. $\uparrow$ | Rel. $\uparrow$ | # Steps | Latency (s) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Reference | - | - | - | 3.67$\pm$0.10 | 4.04$\pm$0.10 | - | - |
| AudioLDM 2 | 3.1 | $1.20$ | $0.31$ | 2.79$\pm$0.08 | 3.40$\pm$0.08 | 208 | 18.1 |
| MusicGen-small | 3.1 | $1.29$ | $0.31$ | - | - | 1500 | 17.6 |
| MusicGen-medium | 3.4 | $1.23$ | $0.32$ | 3.40$\pm$0.08 | 3.79$\pm$0.07 | 1500 | 41.3 |
| Stable-Audio | - | - | - | 3.67$\pm$0.08 | 3.89$\pm$0.07 | 100 | 8.0 |
| MAGNeT-small | 3.3 | $1.12$ | $0.31$ | - | - | 180 | 4.0 |
| MAGNeT-large | 4.0 | $1.15$ | $0.29$ | - | - | 180 | 12.6 |
| MelodyFlow-small | 2.8 | 1.27 | 0.33 | 3.27$\pm$0.08 | 3.83$\pm$0.08 | 64 | 1.8 |
| MelodyFlow-medium | 3.5 | 1.30 | 0.31 | 3.41$\pm$0.08 | 3.77$\pm$0.07 | 64 | 2.3 |

### 4.3 Latent Inversion

#### 4.3.1 Inversion methods

Figure: (a) FADedit as a function of $\lambda_{KL}$.
Refer to caption: x2.png

We compare MelodyFlow with DDIM and ReNoise in the Figures 2(a), 2(b) and 2(c), as a function of the divergence loss weight $\lambda_{KL}$.
During the inversion we use a classifier-free-guidance (CFG) of 0 and employ a CFG of 4 during the regeneration.
The choice of zero CFG is meant to prevent divergence during inversion (see A.2.4). For ReNoise and MelodyFlow the predictions are regularized by the weighted KL patch-wise divergence loss $\mathcal{L}_{patchKL}$ of Algorithm 1 and ReNoise additionally uses an autocorrelation loss with $\lambda_{pair}=10$ (Garibi et al., ).
Both also employ the reversible inversion trajectory estimation while DDIM does not.
The Figures show that both MelodyFlow and ReNoise outperform DDIM inversion by a large margin on the three evaluated axes.
an optimum can be achieved around $\lambda_{KL}=0.2$ for velocity prediction and around $0.1$ for noise prediction.
Overall the quality is better (lower FADedit in the Figure 2(a)) when directly regularizing the velocity prediction.
In both cases we observe a higher CLAPedit in the Figure 2(b) when the original text description $c_{orig}$ conditions the inversion process, confirming better text-adherence.
This happens at the expense of a higher FADedit compared with unconditional inversion.

#### 4.3.2 Target inversion flow step

In the Figures 3(a), 3(b) and 3(c) we report music editing objective metrics as a function of $T_{edit}$, comparing DDIM inversion with MelodyFlow.
The consistency with the source sample is higher (lower LPAPS) with our method than DDIM inversion.
The S-shaped FAD curves of the Figure 3(a) indicate an inversion optimum around $T_{edit}=0.06$, correlating with the peak in CLAPedit score.

Figure: (a) Editing FADedit vs. $T_{edit}$.
Refer to caption: x5.png

### 4.4 Codec bottleneck regularizer

All other things being equal, we ablate on the bottleneck regularizer for a fixed frame rate of 50 Hz by comparing RVQ- (using 4 codebooks of size 2048 each), KL-regularizer (Evans et al., ) and no regularizer at all in the Table 3.
Our results indicate optimal reconstruction performance with no regularizer, closely followed by KL. RVQ stands much further away, likely due to the high level of compression enforced by the discretization (despite the significant dictionary size of $2048^{4}=1.7\times 10^{13}$).
The same ranking applies for SI-SDR (Le Roux et al., ).
Regarding text-to-music generation performance, the KL-regularizer outperforms the other options.
Overall this shows the KL-regularizer offers the best trade off between reconstruction and generation performance.

Ablating on the codec frame rate with the KL regularizer shows that 5 Hz achieves comparable performance with the 50 Hz codecs trained with RVQ or no regularizer, a 10$\times$ improvement factor.
We chose to work with the 20 Hz KL-regularized codec for the 32 kHz mono MelodyFlow-small, as it offers a good trade off between quality and speed.
Accounting for the additional information to compress when scaling to 48 kHz stereo, we chose a frame rate of 25 Hz for MelodyFlow-medium.

**Table 3: Codec bottleneck and framerate ablation for 32 kHz mono audio. Both compression and generative model performances are reported on the in-domain test set.**
| Regularizer | Frame rate (Hz) | STFT${}_{loss}\downarrow$ | SI-SDR$\uparrow$ | FAD${}_{vgg}\downarrow$ |
| --- | --- | --- | --- | --- |
| $\varnothing$ | 50 | 0.35 | 18.5 | 0.68 |
| RVQ | 50 | 0.55 | 4.4 | 0.55 |
|  | 50 | 0.34 | 18.1 | 0.48 |
| KL | 20 | 0.44 | 12.9 | 0.47 |
|  | 5 | 0.53 | 3.5 | 0.67 |

### 4.5 FM design

#### 4.5.1 Model training

We compare our FM model design with the baseline implementation of Le et al. (), both being trained on the same music latents.
The most notable changes are the removal of the infilling objective during training, the change in flow step sampling and the introduction of mini-batch coupling.
Table 4 presents the impact of those choices on the last FM model validation MSEloss of the EMA checkpoint, and on the in-domain test FAD (in the text-to-music generation setting).
No loss value is reported for the baseline as the infilling objective facilitates the task (hence values are not fairly comparable), and for validation we sample flow steps uniformly regardless of the training sampling scheme.
Such infilling objective in Le et al. ()’s FM model was designed to handle variable length sequences that are inherent to the speech domain.
In our experiments it showed to be detrimental for the model performance, and we know diffusion models can support infilling/outfilling without additional tweaks (Liu et al., ).
With all methods combined the in-domain FAD is reduced to 0.39 from 0.53 and consistent with the observed loss decrease, which validates our design.

**Table 4: FM model design ablation. FAD (resp. MSE) is reported on the in-domain test (resp. validation) set. Baseline is adapted from (Le et al., ) but retrained on our music latents.**
| Ablation | Heads | Layers | Infill | Sampling | OT-FM | MSEloss $\downarrow$ | FADvgg $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| baseline | 16 | 24 | ✓ | uniform | ✗ | - | .53 |
| $-$ infilling | 16 | 24 | ✗ | uniform | ✗ | .8596 | .50 |
| $+$ sampling | 16 | 24 | ✗ | logit-normal | ✗ | .8484 | .44 |
| $+$ batch coupling | 16 | 24 | ✗ | logit-normal | ✓ | .8322 | .42 |
| $+$ wider model | 18 | 18 | ✗ | logit-normal | ✓ | .8310 | .39 |

Figure: (a) FADedit vs. NFE.
Refer to caption: x8.png

#### 4.5.2 Music editing quality/efficiency

We compare DDIM with MelodyFlow’s inversion using a target inversion step of $T_{edit}=0$. FADedit (Figure 4(a)), CLAPedit (Figure 4(b)) and LPAPS (Figure 4(c)) are plotted as a function of the total NFE count (inversion + regeneration included).
Quality-wise, the combination of our FM and inversion designs outperform the baseline.
Regardless of the FM design choice, DDIM inversion requires as few as 32 NFEs to achieve an acceptable FAD.
Our inversion only outperforms after 125 NFEs.
On the text-adherence axis, the FM model design alone does not translate in better performance when combined with DDIM inversion.
Swapping DDIM with our method shows a different trend, highlighting the benefit of combining FM and inversion methods.
Analyzing the consistency with the original sample, again we observe that the regularized inversion plays a more important role than the FM model design: the baseline FM model actually outperforms ours when used in conjunction with DDIM inversion.
Overall our method consistently outperform the baseline for 125 NFEs.

## 5 Related work

### 5.1 Audio representation

Recent advancements in neural codecs have seen the application of VQ-VAE on raw waveforms, incorporating a RVQ bottleneck as demonstrated in Zeghidour et al. (); Défossez et al. (), later refined as per Kumar et al. (). Evans et al. () proposed a modification to this approach by replacing the RVQ with a VAE bottleneck to enhance the modeling of continuous representations.
In addition, several recent audio generative models have adopted Mel-Spectrogram latent representations, coupled with a vocoder for reconstruction, as shown in the works of (Ghosal et al., ; Liu et al., ; Le et al., ).

### 5.2 Text-to-music generation

Models that operate on discrete representation are presented in the works of (Agostinelli et al., ; Copet et al., ; Ziv et al., ). Agostinelli et al. () proposed a representation of music using multiple streams of tokens, which are modeled by a cascade of transformer decoders conditioned on a joint textual-music representation (Huang et al., ).
Copet et al. () introduced a single-stage language model that operates on streams of discrete audio representations, supporting both 32 kHz mono and stereo. Ziv et al. () replaced the language model with a masked generative single-stage non-autoregressive transformer.
Schneider et al. (); Huang et al. (); Liu et al. () use diffusion models. Schneider et al. () utilized diffusion for both the generation model and the audio representation auto-encoder. Liu et al. () trained a foundational audio generation model that supports music with latent diffusion, conditioned on autoregressively generated AudioMAE features (Huang et al., ).
Evans et al. (, ) proposed an efficient long-form stereo audio generation model based on the latent diffusion of VAE latent representations. This model introduced timing embeddings conditioning to better control the content and length of the generated music.

### 5.3 Music editing

Lin et al. () proposed a parameter-efficient fine-tuning method for autoregressive language models to support music inpainting tasks.
Garcia et al. () developed a masked acoustic modeling approach for music inpainting, outpainting, continuation and vamping.
Wu et al. () fine-tuned a diffusion-based music generation model with melody, dynamics and rhythm conditioning.
Novack et al. () is a fine-tuning free framework for controlling pre-trained text-to-music diffusion models at inference-time via initial noise latent optimization. Zhang et al. () investigated zero-shot text-guided music editing with conditional latent space and cross attention maps manipulation.
Manor & Michaeli () employs DDPM inversion (Huberman-Spiegelglas et al., ) for zero-shot unsupervised and text-guided audio editing.

## 6 Discussion

##### Limitations

The proposed model specifically focuses on text-guided audio editing with the quality/efficiency trade off in mind, hence we do to not aim nor claim to outperform previous state of the art text-to-music generation models.
Under our current setup text-guided music editing prompts are not instructions.
They describe what the edited sample should sound like given an original music sample and description, but the model is not designed to understand direct editing instructions like *replace instrument A by instrument B*.
While we observed that MelodyFlow performs convincing editing tasks for several axes (genre or instrument swap, tempo modification, key transposition, inpainting/outpainting), more research work is required to accurately evaluate each of those axes.
Music editing human listening tests are conducted for a fixed $T_{edit}$, but eventually it should depend on the sound designer’s preference on the creativity-consistency axis.
Finally the reported objective metrics are mostly used as a proxy for subjective evaluations but they have their limitations.
As an example we observe that optimizing FAD for MusicCaps is usually achieved by overfitting on our training dataset, which negatively correlates with perceived quality.
Overall subjective evaluations remain the best source of truth until a model that mimics human ratings is developed.

##### Conclusion

In this work we presented MelodyFlow, the first non-autoregressive model tailored for zero-shot test-time text-guided editing of high-fidelity stereo music.
In the text-to-music setting the model offers competitive performance thanks to a low frame rate VAE codec and FM model featuring logit-normal flow step sampling, optimal-transport minibatch coupling and L-shaped attention mask.
Combined with our proposed regularized latent inversion method, MelodyFlow outperforms previous zero-shot test-time methods by a large margin.
The model achieves remarkable efficiency that is key for the sound design creative process and supports variable duration samples.
Our extensive evaluation, that includes objective metrics and human studies, highlights MelodyFlow promise for efficient music editing with remarkable consistency, text-adherence and minimal quality degradation compared with the original, while remaining competitive on the task of text-to-music generation.
For future work we intend to explore how to accurately evaluate specific editing axes and how such a model could help design metrics that better correlate with human preference.

## Appendix A Appendix

### A.1 Experimental setup

#### A.1.1 Audio latent representation

Our compression model implementation is that of Copet et al. ()(^3^33[https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/encodec.py](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/encodec.py)) enhanced by band-wise discriminators and snake activations from Kumar et al. (), perceptual weighting (Wright & Välimäki, ), VAE bottleneck and multi resolution STFT reconstruction loss from Evans et al. ().
We train a mono 32 kHz codec at 20 Hz frame rate and another one supporting stereo 48 kHz audio at 25 Hz. The bottleneck dimension is of 128. Both are trained on one-second random audio crops for 200K steps, with a constant learning rate of 0.0003, AdamW optimizer and loss balancer of (Défossez et al., ).
Stereo codecs are trained with sum and difference loss (Steinmetz et al., ).
The bottleneck layer statistics are tracked during training (dimension-wise) for normalization prior to FM model training.

#### A.1.2 Flow matching model

MelodyFlow’s DiT follows Esser et al. () configurations where each head dimension is of 64 and the model has the same number of heads and layers (either 18 or 24).
Model implementation is that of audiocraft(^4^44[https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/modules/transformer.py](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/modules/transformer.py)) but adapted for FM following Vyas et al. (): U-shaped skip connections are added along with linear projections applied after concatenation with each transformer block output Bao et al. ().
The model is conditioned via cross attention on a T5 representation (Raffel et al., ) computed from the text description of the audio, using 20% dropout rate during training in anticipation for the classifier free guidance applied at inference.
Cross attention masking is used to properly adapt to the text conditioning sequence length of each sample within a batch and we use zero attention for the model to handle unconditional generation transparently.
No prepossessing is applied on the text data and we only rely on the descriptions (additional annotations tags such as musical key, tempo, type of instruments, etc. are discarded, although they also sometimes appear in the text description).
The flow timestep is injected following Hatamizadeh et al. (). Minibatch coupling is computed with torch-linear-assignement(^5^55[https://github.com/ivan-chai/torch-linear-assignment](https://github.com/ivan-chai/torch-linear-assignment)).
MelodyFlow-small (resp. MelodyFlow-medium) is trained on latent representation sequences of 32 kHz mono (resp. 48 kHz stereo) segments of 10 (resp. 30) seconds, encoded at 20 Hz frame rate (resp. 25 Hz).
MelodyFlow-small (resp. MelodyFlow-medium) is trained for 240k (resp. 120k) steps with AdamW optimizer ($\beta_{1}=0.9$, $\beta_{2}=0.95$, weight decay of 0.1 and gradient clipping at 0.2), a batch size of 576 and a cosine learning rate schedule with 4000 warmup steps.
Additionally, we update an exponential moving average of the model weights ever 10 steps with a decay of 0.99.
Each model is trained on 8 H100 96GB GPUs with bfloat16 mixed precision and FSDP (Zhao et al., ). MelodyFlow-small requires 3 days and MelodyFlow-medium 6 days of training.

#### A.1.3 LLM-assisted editing prompt generation

For editing prompts design we prompted the LLama-3 large language model Dubey et al. () to modify the original descriptions by targeting genre swapping.
Edited descriptions were then manually verified to ensure their plausibility and coherence.
As an example, given the original description *This is a lush indie-folk song featuring soaring harmony interplay and haunting reverb-y harmonica*, the resulting editing prompt is *This is a lush Indian classical-inspired song featuring soaring harmony interplay and haunting reverb-y bansuri flute*.

#### A.1.4 Subjective evaluation form

A screenshot of the music subjective evaluation form is shown in the Figure 5.

Figure: Figure 5: Music editing subjective evaluation form. Given the original song A, raters are asked to evaluate three different edits of A, on the three following axes: quality, text adherence, consistency.
Refer to caption: extracted/5931873/figs/amt_3.png

### A.2 Additional experiments

#### A.2.1 Stereo codec

The Table 5 reports the impact of scaling from mono to stereo with the same MelodyFlow-medium model size (1B parameters) trained on 30 second segments.
Two codecs are trained on 48 kHz audio, using the same 25 Hz latent frame rate: one mono and one stereo.
The in-domain FAD is reported for 10s and 30s generated segments.
Moving from mono to stereo marginally affects the generative model performance.

#### A.2.2 Length generalization

One drawback of training the FM model of a fixed segment duration is that the inference can only be run for the same duration, otherwise the quality will degrade (this can be seen in the FAD10s column of the Table 5, when comparing the first two rows). This can be handled by using padded segments and specific conditioning (Evans et al., ), but does not save any resource when targeting shorter segments.
Another solution is to train on variable length segments but then the model does not generalize well for full length segments, and will better learn for the uttermost left positions of the sequence that appear more often.
We propose to simulate training on variable length segments, while keeping the model learning for the full length scenario.
This is done by applying a L-shaped attention mask during model.
For each sequence of length L, we randomly select a segment boundary within the range $[0,L]$. Positions before the boundary can only attend to themselves in the DiT’s self-attention, while positions after it attend to the entire sequence.

Comparing the first two rows of the Table 5 indicate that the L-shaped mask helps supporting versatile duration with no penalty on full-length segments, unlocking faster inference for segments shorter than 30 seconds.
This method does not generalize to segments longer than 30 seconds, which should be specifically handled with a sliding window/outpainting approach.

**Table 5: Ablation on L-shaped attention mask and stereo for MelodyFlow-large. Each variant is trained on 30s audio segments encoded with a 25 Hz frame rate codec trained on 48 kHz audio.**
| Channels | STFT${}_{loss}\downarrow$ | SI-SDR$\uparrow$ | L-mask | FAD${}_{10s}\downarrow$ | FAD${}_{30s}\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| 2 | 0.40 | 12.48 | ✓ | 0.59 | 0.65 |
| ✗ | 1.48 | 0.65 |  |  |  |
| 1 | 0.39 | 13.34 | ✓ | 0.49 | 0.59 |

Figure: (a) Generation CFG.
Refer to caption: x11.png

#### A.2.3 Text-to-music generation efficiency

In the Figure 6(b) we report the text-to-music generation test FAD as a function of the number of DiT forward passes (NFEs) for both the baseline FM architecture (Le et al. ()) and final version of MelodyFlow. Not only does MelodyFlow achieve better performance, but with 16 times fewer NFEs (e.g. where the baseline required 256 NFEs to reach 0.53 FAD, MelodyFlow only requires 16 NFEs to score 0.50).

Figure: (a) Editing FAD vs. CFG.
Refer to caption: x13.png

Figure: (a) Editing FAD vs. CFG.
Refer to caption: x16.png

#### A.2.4 Classifier-free guidance

In the Figure 6(a) we report the in-domain test FAD as a function of the classifier-free guidance factor in the text-to-music generation setting.
We use a classifier-free guidance factor of $4.0$ for the text-to-music generation inference.

In the Figures 7(a), 7(b), 7(c) we plot our objective metrics for text-guided music editing, as a function of the CFG.
The performance is bad whatever the considered inversion method, showing that using the CFG during inversion is detrimental.
Above a CFG of 0 ReNoise completely diverges (the LPAPS skyrockets), while MelodyFlow achieves the best robustness.
This explains why we consider that ReNoise is not directly compatible with the FM formulation, even after converting to $\epsilon$-prediction, and that FM requires an adapted regularized latent inversion method.
After keeping the CFG to zero during latent inversion to stabilize the process, the results are presented in the Figures 8(a), 8(b), 8(c).
We end up using the same classifier-free guidance factor of $4.0$ for our editing experiments.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - Performance in Table 2: Why does the medium model in Table 2 perform worse than the small model across most metrics?
-  Figure 2: I found Figure 2 is difficult to interpret. What do the four lines correspond to? Additionally, the three subplots report different metrics (FAD, CLAP, and LPAPS), making it impossible to directly compare the three methods under consideration.
- What KL regularization factor was used when training the Codec?
soundness: 4
strengths: 1- The authors demonstrated good music generation results

2- The authors promised the release of the code and model checkpoints which can be useful for the community 

3- The paper is well written.
summary: The authors proposed a method for the music editing task that adapts the ReNoise regularization approach to flow matching. The paper also offer insights on training audio flow matching models, such as Codec bottleneck size and the use of minibatch coupling, which could benefit the community. While the authors provide extensive experimental insights, the overall results and contributions of the paper remain questionable.
weaknesses: - Questionable quantitative results: In Table 1, the proposed method shows marginal improvement in relevance to the editing prompt, but it falls behind in terms of consistency. Additionally, a better overall sound quality does not necessarily indicate improved editing capability. Overall, the metrics presented regarding the editting are not convincing.

- Poor qualitative results: The provided samples in the supp. do not demonstrate fine-grained editing capabilities, as they fail to showcase cases where only one or two words are changed in the prompts. This makes it difficult to assess the quality of the editing. Personally, I found the samples to lack consistency with the source music clips.

- Limited novelty: Although the authors offer valuable insights into training audio FM models, the proposed editing method is primarily an adaptation of the ReNoise technique, which limits the work’s technical novelty.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Questions (see also Weakness): 
1. Which checkpoint was used for the CLAP score? There are several checkpoints available at https://github.com/LAION-AI/CLAP.
2. Regarding the subjective evaluation for the music editing task in Table 1, the methods are rated on a 1–5 scale, with none of the methods achieving a rating of 3 or higher. Does this imply that none of the methods achieved a "fair" level of quality according to the standard MOS scale?
3. In Section 4.3.2, L376, the authors claim that LPAPS is being used to measure 'consistency' between source and edited samples. Could you provide at least an intuitive explanation, if not a theoretical one, supporting why LPAPS is suitable for evaluating "consistency"? (Also, could you provide an intuitive definition of "consistency" in this context for better understanding?)

Minor comments:
- Abstract, L16: The term "RENOISE" is not defined.
- Equations 2.2 and 2.3: The equations are missing reference numbers, such as (2.2) and (2.3).
soundness: 2
strengths: - Music generation: The authors experimentally demonstrate that their FM-based text-to-music generative model can produce 48kHz, stereo-format waveforms. To achieve this, as discussed in Section 2.5, Appendix A.2, and Table 4, they conduct a detailed exploration of architectural improvements, including audio compression parts and enhanced training techniques such as Minibatch Coupling.
- Music editing: They reformulate techniques of improved DDIM inversion, as proposed in Pix2pix-zero [1] and ReNoise, to make them applicable to FM-based music generative models. A key point in the originality of this work lies in not merely applying DDIM inversion or ReNoise directly but specifically refining the regularization technique in ReNoise.

[1] Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J. and Zhu, J.Y., 2023, July. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings (pp. 1-11).
summary: The authors propose MelodyFlow, a flow-matching-based (FM-based) text-to-music generative model capable of producing 48kHz, stereo-format music. Additionally, they introduce a music editing method built on the pretrained MelodyFlow. Specifically, they adapt diffusion model-based inversion methods, such as DDIM inversion and ReNoise, to work with FM-based models, enabling editing through FM-based inversion. They experimentally demonstrate MelodyFlow's capabilities in both text-to-music generation and music editing.
weaknesses: **Overall**:
1. Music generation part: 
- The applicability of FM-based generative models for audio data is already explored in some prior work (for example, Text-to-Music Generation [1] and Text-to-Audio [2]). Therefore, simply applying FM-based model to text-to-music generation task (even 'single-stage') does not contribute significantly to new insights for the audio/music community.
    - Therefore, additional novel technical contributions, such as sophisticated ideas to surpass the sample quality of existing text-to-music models (regardless of whether the FM-based or diffusion-based models) or techniques that enable more efficient generation while maintaining sample quality, along with sufficient experimental evaluation to demonstrate their effectiveness, should be explored.

- Considering of above, the authors' efforts to improve generation performance through architectural and training enhancements (as discussed in Section 2.5, Appendix A.2, and Table 4) could still provide some valuable insights. However, even from this perspective, both objective and subjective metrics in Table 2 suggest that, despite incorporating various techniques, the main advantage of the FM-based models lies in halving the number of sampling steps compared to diffusion-based models like Stable Audio. Thus, the contributions offered from this angle remain limited.

2. Music-editing part:
- From the perspective of ODE-based generative models, score-based and FM-based generative models share a broadly similar framework. Thus, the fundamental contribution of reformulating DDIM inversion or its extensions, such as Pix2pix-zero/ReNoise, to apply to FM-based models is not particularly significant. (Of course, I acknowledge the practical differences, so please note that I am not evaluating the proposed editing method solely from this perspective.)
- Regarding the proposed inversion technique, the experimental results do not seem to fully support its effectiveness. For example, in Table 1, considering the inherent challenges of subjective evaluation in music editing tasks and the CI95, the advantage over DDPM inversion in terms of 'AVG.' does not appear sufficiently clear. (To clarify, I am indicating that the results are not clear enough; I am not suggesting that the proposed method is ineffective. For more details on objective metrics, please refer to Details below and Questions 1~3.)

**Details**:

The objective metrics used for music editing and generation leave some uncertainty regarding their adequacy in validating the effectiveness of the proposed methods.

- Recent studies, such as [3], suggest that using FAD with VGG-ish to evaluate music samples is inappropriate. To better support the authors' claims in this work, I recommend using alternative pretrained models as introduced in [3] (for both editing and generation tasks).
    - For example, based on the correlation between MOS and FAD discussed in [3], it would be more appropriate to use FAD with LAION-CLAP to evaluate musical quality, and to use FAD with DAC/EnCodec embeddings to assess acoustic quality (please see more detail in [3]). 
- For LPAPS, if the authors used the pretrained network from this repository [4], it was trained on the VGGSound dataset [5] (also, please review its data filtering process). This raises concerns about its validity for numerical evaluation in music editing. Additionally, the checkpoint provided in that repository [4] has other issues. As noted in this issue [6], the authors of this repository acknowledge a problem in the LPAPS training procedure itself and do not recommend using LPAPS for at least training purposes.
    - It would be appropriate to calculate the L2 distance using other audio encoders trained properly. For instance, as in [3], I recommend calculating the L2 distance based on embeddings from audio encoders like LAION-CLAP, DAC, or EnCodec. 
- Considering both these points and the subjective evaluation results, the current experimental results do not seem sufficient to demonstrate the effectiveness of the proposed latent KL regularization, which contributes to the originality of this method. I recommend revisiting the evaluation protocol to provide stronger support for contributions of this paper. (I do acknowledge that the authors have demonstrated that FM-based music generative models can achieve music editing through inversion techniques to some extent.)
   -  For instance, conducting additional evaluations using the objective metrics mentioned above, as well as a subjective evaluation for an ablation study on the necessity of latent KL regularization in MelodyFlow’s inversion ($\lambda_{KL}=0$ or not), would support the contribution much more in terms of the proposed editing method if their effectiveness is sufficiently demonstrated.

[1] Tal, O., Ziv, A., Gat, I., Kreuk, F., Adi, Y., 2024 Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation. The 25th International Society for Music Information Retrieval (ISMIR) Conference.

[2] Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W. and Wang, J., 2023. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821.

[3] Gui, A., Gamper, H., Braun, S. and Emmanouilidou, D., 2024, April. Adapting frechet audio distance for generative music evaluation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1331-1335). IEEE.
Here is toolkit https://github.com/microsoft/fadtk.

[4] https://github.com/v-iashin/SpecVQGAN

[5] Chen, H., Xie, W., Vedaldi, A. and Zisserman, A., 2020, May. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 721-725). IEEE.

[6] https://github.com/v-iashin/SpecVQGAN/issues/13

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - It would be great to know how what's the composition of the in-domain evaluation set. For example. does it curated to have balanced genres like the set used in subjective tests? An example of such breakdown can be found in the Appendix A of [1].
[1] MusicLM: Generating Music From Text, Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, Christian Frank, 
URL: https://arxiv.org/pdf/2301.11325

- Will the best value for lambda_KL dependent on specific network architecture or even dependent with network size? 

- The problem setup of text-guided editing in this work has an implicit assumption: The coherence to x_src can be achieved with ODESolve(0->t_edit), and the adherence of the editing can be achieved with ODESolve(t_edit->1). I understand this is an assumption existed from previous works, but it lacks evidence that it can always hold for most editing scenarios. Could it be possible to categorize the text prompts into those categories mentioned in L042-043 and L514-515 and show the categorical-wise editing performance?

Heuristically, during the forward diffusion, timbre information is more likely to be erased before the note information. Therefore, an interesting case to test about this assumption would be find a way to promp the model to change the melody(e.g. raise the melody with 3 semitones) but maintaining the timbre on an audio that has predominant melody trajectory. Conventional MIR algorithms/metrics can be used to evaluate the result.

- I am not sure if I am over cautious, but since the model checkpoint will be released, it would be nice to ensure if all the licenses of training sets are compatible with this.
soundness: 3
strengths: - This work is adapts ReNoise with flow matching on music modality plus its own modification, which demonstrates the feasibility of such methodology on music editing. The objective and subjective results also shown that the proposed model have better performance on editing tasks compared to other baselines.

- In terms of generation, proposed model is capable of generating samples which have quality on par with baseline models in a much shorter inference latency. 

- The proposed model and method are well presented. Design choices in autoencoder and training techniques are well-described. Moreover, their usefulness are justified with ablation studies. 

- Authors plan to release model weights and code, which will benefit the community for sure.

- The limitation part clearly mentioned several issues and limitations of the proposed work, which is also insightful for readers.
summary: This work proposes a text-to-music model based on a DiT(diffusion transformer) established on the continuous latent of VAE, in the setup of flow matching. This model not only supports usual generation conditioned by text, but also music editing by depicting music editing as the combination of latent inversion plus re-generation with a new prompt. During the latent inversion, a KL regularization method derived from ReNoise is introduced to enhance the quality and coherence to the original sample. 

This work incorporated several techniques for quality improvements and shown their usefulness in ablation studies (Table 3, 4). Other ablations shown the trend of changing NFEs, lambda_KL and T_edit (Fig. 2,3,4). Furthermore, the ablations and experiments show that, when proper hyperparameters are chosen, the regularized latent inversion can outperform DDIM and ReNoise. The quality on generation and text-guided editing of proposed work is evaluated by both subjective tests and objective metrics.
weaknesses: - Components of this work is a combination from existing works. Its novelty lies within the modification to the regularization method of ReNoise, but it lacks theoretical support/analysis on the effectiveness of such modification. It should be worth to have more discussion and analysis on, for example, why removing L_pair improves the result, whether there's a sweetspot for lambda_pair, and what's the side effect of removing L_pair. Empirically, it might be helpful to have a grid search based on both lambda_pair and lambda_KL, and measure the KL-divergence between the latents from different approaches. Visualizing the distribution of the latents with low dimensional projection such as t-SNE or UMAP could also be interesting.

- The quality evaluation could not show whether the proposed model is qualified to be called as "high-fidelity". Although the the FAD score is the lowest for the proposed model, but FAD_vgg cannot evaluate the content beyond 16k due to the limitation of the VGG model. The experiment included MusicCaps as the reference set in the subjective evaluation, but the audio of MusicCaps is a subset of AudioSet, which is obtained from Youtube video, thus unlikely to be in high-fidelity format(uncompressed 44.1 or 48 khz). Moreover, the subjective test result on overall quality does not favor proposed work (around the same as MusicGen-medium, but not at the same level of StableAudio). One way to justify this is to compare the generated samples of the proposed model with another reference set which is natively 48khz. A held out set of the internal 48khz dataset could serve this purpose.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - “MELODYFLOW uses a DiT of sizes 400M (small) and 1B (medium) parameters with U-shaped skip connections Bao et al. (2023).” Is this not just a U-Net then? As many TTA/TTM UNets utilize internal attention layers, it’s unclear whether this is an actual DiT or some DiT-UNet hybrid (though I may be misunderstanding something here).
- For figure 3, what happens after $T_{edit} > .25$? Especially in the case of FAD, it seems like another optimum may arise if $T_{edit}$ is increased further.
soundness: 3
strengths: - Overall writing style is clear
- Introduction of FM objective is streamlined and easy to follow.
- The breadth of ablations are much appreciated, as the authors go to reasonable lengths to understand the design space and limitations of their method.
summary: This paper presents MelodyFlow, a novel Flow-Matching based TTM model capable of generating 30 seconds of 48 kHz audio, and in parallel introduce a regularized inversion method to allow for inference-time editing.
weaknesses: - Line (046-047) “editing methods from the computer vision domain, which are exclusive to diffusion models (Novack et al., 2024; Zhang et al., 2024; Manor & Michaeli, 2024)” is not wholly true. Optimization methods like Novack et al., 2024 are agnostic to the sampling process (and in fact, the flow-matching equivalent has already been explored [1]), and guidance methods like Zhang et al., 2024 are also agnostic to sampler (as nothing specific about their method *requires* DDIM inversion as the inversion method).
- In general, Section 2.4 is not clear enough. It is hard to tell which parts of the the algorithm from ReNoise are being kept and which are novel, and in general as there is very little textual explanation of the method (and only consistent referrals back to the ReNoise paper), it is hard to tell where the novelty lies. As this is a claimed **core contribution** of the paper, more time should be spent in section 2.4 to make the differences between the baseline and their method explicitly clear, and to explain the algorithm in depth textually. Along with this, Figure 1 is very hard to parse, and a more semantic diagram (rather than ground truth control flow) may be more helpful.
- 2.5.2 is also very much understated, and is hard to parse without an intimate knowledge of the references. As the paper uses this method to show its increased performance, it should be made explicit in the paper how this works and what the goal of the method is (e.g. is this a training or inference time change? Why do we want to run the Hungarian algorithm?)
- I wonder at a high level that the overall framing (and title) of the paper is somewhat misplaced. On line 047, it is stated that “Despite recent efforts, no approach has yet shown the ability to perform high-fidelity generic style transfer across various music editing tasks,” which I think is the right framing for the inversion task used here, as the example samples show much more strengths in “stylistic” changes rather than grounded content changes (i.e. they very much sound “inspired” by the original samples rather than editing them). This is a fine approach for the paper’s framing, but this is the *only* mention of style in the entire paper, and thus much of the paper sounds like its talking about some other, more grounded form of editing.
- While previous works have used it, it is publicly documented that the VGGish backbone for FAD is not correlated with human perception [2], and thus some other backbone (like CLAP) should be used. This shouldn’t be too much of an issue, as all the models tested here are open-source (with the exception of Stable Audio, but Stable Audio Open exists) to run directly.
- Overall, the technical results are somewhat weak. While it is mentioned that in the limitations the goal is music editing and *not* to outperform other TTM models (line 511), this is at odds with their first main stated contribution of making an improved TTM generation model *in general* (line 097). The results against existing baselines show no clear strengths other than moderate latency speed ups.  It also seems like between the baseline ReNoise and the proposed method, the differences are small.

As a whole, the paper seems to be caught somewhere between being a modeling paper (in its proposal of ways to improve training of FM-based TTM models) and a music editing paper (through its FM inversion method), weakening it in both respects. The strongest modeling results are only with respect to an internal non-TTM baseline and do not seem to get comparable results with current SOTA methods, and a full ablation of FM TTM modeling is given to the space dedicated to the editing results. And conversely, without the modeling contributions, it seems like the core contribution then rests solely on the FM inversion technique, in which its novelty is hard to assess. I do not think the present version of the paper is ready for acceptance, though I think changes to spending more time on the editing results (as those seem comparatively stronger than the modeling changes) as the main contribution and overall framing would improve the paper substantially.

[1] Ben-Hamu, Heli et al. “D-Flow: Differentiating through Flows for Controlled Generation.” ArXiv abs/2402.14017 (2024): n. pag.

[2] Gui, Azalea et al. “Adapting Frechet Audio Distance for Generative Music Evaluation.” ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2023): 1331-1335.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: See questions in weakness.
soundness: 2
strengths: 1. This paper introduces a novel single-stage flow matching model for text-to-music generation, capable of generating and editing audio samples at 48 kHz stereo quality.
2. The authors exploit a regularized flow matching inversion method to facilitate text-based music editing and conduct ablation studies to validate its effectiveness.
3. The experimental results demonstrate that the proposed approach outperforms all baseline methods across all objective metrics.
summary: This paper leverages the diffusion transformer architecture with flow matching objective for text-to-music generation and editing at 48kHz stereo quality. The model adopts a variational autoencoder to compress the raw stereo waveforms to latent representations. Subsequently, a flow matching model is trained for text-based conditional generation. For the editing component, the authors adapt an optimization-based inversion method from ReNoise, integrating it into the flow matching framework. The experimental results demonstrate that the proposed text-based editing approach surpasses both DDIM Inversion and ReNoise on objective metrics.
weaknesses: 1. Although the paper claims to introduce the flow matching model for text-to-music generation, it is apparent that its performance on subjective metrics does not match that of Stable Audio. Furthermore, the authors have not provided an evaluation of Stable Audio's objective metrics.

2. Need a more comprehensive literature review. Important editing methods like DITTO or MEDIC are not in the literature review. Also need to compare with MusicMagus in zero-shot editing.

3. This work proposes a new regularized inversion method; however, the primary distinction from ReNoise lies in the alteration of noise prediction to velocity prediction. While the results achieved are commendable, the contribution lacks sufficient novelty.

4. The authors assert that their method retains similarity to the source audio; however, it obtains a lower score on the CON metric in subjective evaluations compared to AudioLDM2-Music. Moreover, the demonstrated demo reveals inadequate preservation of essential content, raising concerns about the validity of their claim regarding the maintenance of the original audio's critical elements.

5. The paper claims capability of handling 30 seconds of music, yet it predominantly conducts comparative experiments using MusicCaps, which features a duration of only 10 seconds. It is advisable for the authors to include comparisons with longer music datasets, such as the Song Describer Dataset.

6. The authors have not yet made their code and model publicly available, which should not be presented as a contribution point in the paper.