SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
Authors: Michael Ungersböck
Florian Grötschla
Luca A Lanzendörfer
June Young Yi
Changho Choi
Roger Wattenhofer
Venue: NeurIPS 2025 poster ()
Keywords: audio editing
text-to-audio generation
generative audio
Primary area: applications

Abstract: Generative models have made significant progress in synthesizing high-fidelity audio from short textual descriptions. However, editing existing audio using natural language has remained largely underexplored. Current approaches either require the complete description of the edited audio or are constrained to predefined edit instructions that lack flexibility. In this work, we introduce SAO-Instruct, a model based on Stable Audio Open capable of editing audio clips using any free-form natural language instruction. To train our model, we create a dataset of audio editing triplets (input audio, edit instruction, output audio) using Prompt-to-Prompt, DDPM inversion, and a manual editing pipeline. Although partially trained on synthetic data, our model generalizes well to real in-the-wild audio clips and unseen edit instructions. We demonstrate that SAO-Instruct achieves competitive performance on objective metrics and outperforms other audio editing approaches in a subjective listening study. To encourage future research, we release our code and model weights.

SAO-Instruct: Free-form Audio Editing using Natural Language Instructions
Michael Ungershöck ETH Zurich mungersboeck0ethzz.co
Florian Grötschla ETH Zurich fgroetschla@ethz.com ch
Luca A. Lanzendörfer ETH Zurich
lanzendoerfer@ethh..co ch
June Young Yi Scoul National University julianyi Milianyi@@es 10snu. ac kr
Changho Choi Korea University
Roger Wattenhofer
ETH Zurich
changho88@korea..a ac
wattenhoferretthecc ch
Abstract
Generative models have made significant progress in synthesizing high-fidelity audio from short textual descriptions. However, editing existing audio using natural language has remained largely underexplored. Current approaches either require the complete description of the edited audio or are constrained to predefined edit instructions that lack flexibility. In this work, we introduce IAO-Instruct, model based on Stable Audio Open capable of editing audio clips using any free-form natural language instruction. To train our model, we create dataset of audio editing triplets (input audio, edit instruction, output audio) using Prompt-to-Prowet Prommp,, DDPM inversion, and a manual editing pipeline. Although partially trained on synthetic data, our model generalizes well to real in-the-wild audio clips and unseen edit instructions. We demonstrate that SAO-Instruct achieves competitive performance objective metrics and outperforms other audio editing approaches in a subjective listening study. To encourage future research, we release our code and model weights.
Introduction
Generative audio models have become increasingly popular, allowing users to generate high-fidelity long-form audio within seconds. While they have been successfully applied in various domains, including music {8. I0 [ 5). speech [29] 26 21 4. and general audio [30 24
9 13, the area of audio editing still remains largely unexplored. These models, especially when given short or ambiguous prompts, have freedom and flexibility in their outputs. While this enables diverse and creative generations, it may lead to outputs that deviate from the user's original intent. similar issue exists in recorded audio, which often contains imperfections and typically requires manual editing before being suitable for real- world use. These modifications can range from subtle adjustments to stylistic and spatial effects, such "the birds should chirp louder". "make it sound muffled". or "add reverb to the fireworks." In addition to the challenges faced in generative audio, such as modeling the high-dimensional long-term temporal dependencies, audio editing introduces further complexities. Edits must selectively modify specific aspects the provided audio while preserving the overall background context. User-specified edits can also vary dramatically in scope and typically do not follow rigid structures.
Recent methods have made progress towards addressing some of these challenges. Zero-shot inversion approaches [33, 20 modify existing audio by conditioning on a full target description. However, describing audio clip with its unique sound characteristics in concise, unambiguous text challenging and requires significant effort []. more intuitive alternative is instruction. based
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

door an alarm goee
door closes a phone riging."
<1) ..*
SAO- SAO-Instruct
door closing
slarm going off
change the alarm phone ringing
door closing
phone ringing
Given an input audio clip and an edit instruction, SAO-Instruct outputs the edited audio while keeping the overall audio context intact.
Figure I:
editing, where users specify only the intended change in natural language rather than the desired outcome. The AUDIT 44 model is one of the first to explore instruction-based editing, but only supports predefined set of editing tasks. Additionally, user instructions are often diverse and underspecified, which prevents them from aligning neatly with such fixed operations. We argue that enabling fully free-form instruction-based editing would allow more expressive transformations, significantly simplify user interaction, and broaden the applicability of generative audio models.
In this work, we introduce SAO-Instruct, the first model for free-form instruction-based editing in the audio domain. We use Prompt-to-Promp [16], DDPM inversion [3 and a manual editing pipeline to generate data triplets of input audio, edit instruction, and the corresponding edited audio. As illustrated in Fig. I we show that the model learns to modify audio given a free-form edit instruction. While partially fine-tuned on synthetic data, our experiments show that the model able generalize well to real in-the-wild audio and is able to follow diverse editing operations.
Our key contributions be summarized follows:
We present SAO-Instruct, the first fully free-form instruction-baser audio editing model based on Stable Audio Open. Our model generalizes well to real in-the-wild audio clips and achieves competitive performance compared to zero-shot approaches that rely on full audio descriptions.
We design a novel pipeline to create dataset of audio editing triplets (input audio, edit instruction, output audio), combining LLM- prompt generation, Bayesian Optimization, and filtering mechanism that addresses the limitations of current generative audio models.
We create diverse dataset using three complementary approaches: fully synthetic generation via Prompt-to-Promph semi-synthetic samples via DDPM inversion, and wollwwrld manual edits. We demonstrate their contribution to the performance of our editing model an ablation study.
Samples are available online: httpps://th disco .ww.ubittt...........o i0 sao- instruct
2 Related Work
2.1
Text-to-Audio Generation
The area of generative audio has gained significant popularity in recent years. Applications range from music generation [8] 10 [5 and speech synthesis (29 26 21 to the broader general audio domain 30 24 9 13.. The latter category encompasses everything from short, distinct sound effects ("the sound of a whip") ") to more complex scenes ("frogs croaking at a pond") and environmental ambient noise (" ("light ruin with distanl ihunder"). Several architectures have been explored for general audio synthesis. Apart from autoregressive approaches 244 recent advancements in latent diffusion models have achieved state-of-the-arr results. AudioL.DM [30] uses a latent diffusion architecture conditioned on Contrastive Language- Audio Pretraining (CLAP) 45 embeddings to learn the latent space of mel spectrograms. While such approaches require a vocoder to reconstruct the waveform, which can introduce artifacts due missing phase information, recent advances such as ViAN (28) have significantly improved reconstruction fidelity. To remove the need for vocoders. Stable Audio Open 9 uses a variational autoencoder operating at a 21.5 Hz latent framerate to encode stereo audio into a continuous latent representation. A diffusion transformer [38] then operates on this latent space, conditioned on the text prompt, timing information, and the current diffusion timestep. Finally, the output is decoded back into 44.1 1 kHz stereo audio of up to 47 seconds in length. By conditioning on timing information, users can specify the start and end points of the generated audio, while the model is trained to fill the remaining segments with silence.

---

Prompt Generation
Sample Goneration
ino-tuunin
Inference
+ 4) tr
Stable Audio Open
DDPM Inversion
Insut Caption
Stack Aud Ocen
LLM
Edit Instrutiio
Triplcts
Prompt-to-P Pooompppp uaing Stile Aud Upan
Instruatoo
E:it Instruaiio
Speniics
LLM
Eoi intruction
Manual Edits
Audio Cantion
SAO-Instruct
Figure 2: Overview of our proposed method. Green indicates synthetic data. Audio datasets are used as the starting point for prompt generation. DDPM inversion and Coompttoo-Prmpp use the input caption and generated output caption to create a partial and fully synthetic dataset, respectively. For manual edits, predefined edit operation is sampled. In the fine-tuning stage, Stable Audio Open is trained on the combined generated samples and edit instructions. During inference, SAO-Instruct receives an audio clip and free-form edit instruction and produces the edited output.
2.2 Generative Editing
Recent progress in generative models has enabled new capabilities in editing content across various domains. Approaches to generative editing be broadly categorized into zero-shot methods, which adapt existing models without training, and supervised methods, which fine-tune generative models for specific editing tasks.
Zero-shot Methods. A common zero-shot editing strategy for diffusion models involves partially noising an input sample and then guiding the denoising process with modified description (36). Other approaches [37 19. 33 (00] use inversion techniques which first estimate the latent representation given input sample and its description, and then generate the edited output by guiding the denoising process with a modified prompt. However, these inversion approaches typically require explicit access to detailed input and output descriptions, which are often unavailable in real-world settings. The quality of results is also highly dependent on how these descriptions are phrased, making such approaches less intuitive for practical use.
Supervised Methods. Another line of work trains generative models specifically for editing tasks. Diffusion models with an infilling objective learn to reconstruct masked audio regions based on the surrounding context and a target text description [3). more intuitive alternative is instruction- based editing, where users specify only the intended change in natural language rather than the desired outcome. In the image domain, InstructPix2Pix [3] fine-tunes a diffusion model on synthetic data triplets (input image, edit instruction, output image), generated using an LLM and Prompt- to-Prompt [16), which enables image edits by selectively injecting attention maps of the original prompt during the generation of the modified prompt. In the general audio domain, AUDIT 4 was among the first to enable instruction- based editing. It supports the following five core operations: addition, removal, replacement, inpainting, and super-resolution of audio. InstructME [4] introduced a related approach for music editing that also considers the harmonic consistency while executing edits. Fugatto [42 further explores free-form audio generation and transformation through specialized multitasks datasets. While such models demonstrate broader instruction- nnstructionfoolloving capabilities, they are not specifically tailored for
audio editing using natural free-form instructions.
3 Method
Free-form audio editing with supervised learning approach requires a dataset of triplets consisting of an input audio clip, an edit instruction, and the corresponding edited audio. Since no such dataset is readily available, we construct one using combination of LLM -bsed prompt generation and
.. As illustrated in Fig. 2 our approach first prepares a dataset of prompts that contain an input caption, an edit instruction, and an output caption. The corresponding paired audio samples are created using three complementary
partially synthetic audio, inspired by recent work in the image domain 3
methods: fully synthetic audio generated via Prompt-to- tromt [16), semi-synthetic audio via DDPM
3

---

inversion [33] and manually edited real audio clips. These generated edit instructions and audio pairs are then combined form the full training set. Finally, Stable Audio Open (SAO) [ is fine-tuned on this dataset to obtain SAO-Instruct, which can edit a given audio clip based on a free-form natural language instruction.
3.1 Prompt Generation
To train our model for free-form editing, we need samples of triplets (input caption, edit instruction, and output caption). For the imput captions, we use the captioning datasets AudioCaps [22 and WavCaps B31 AudioCaps contains 50k human-written captions paired with audio clips sourced from AudioSet [1]]. WayCaps consists of 400k audio-caption pairs collected from multiple sources, including FreeSound AudioSet-SL. [15] and the BBC Sound Effects library. We use GPT-4o to generate a synthetic dataset of structured prompts. Given an input caption, the LLM is prompted to generate a fitting edit instruction and a corresponding output caption. For example, starting from the caption "Birds chirping and water flowing". " it may generate the instruction "Remove the water flowing" " and the output caption "Birds chirping". with the output caption derived by applying the edit instruction to the input caption. The LLM also generates additional metadata, including negative prompts and a count of distinct audible elements, which we use to improve sample quality for audio synthesis and to enable downstream filtering. More details can be found in Appendix A
3.2 Prompt-to-Promp
significant challenge audio editing applying targeted modifications while preserving the overall audio context, including background sounds and overall atmosphere. Since there is no such dataset available, we adapt the Prompt-to-Pronee to the audio domain. Prompt-to- Prompt enables localized edits of synthesized audio by injecting attention maps from the input prompt into the generation process of the edited prompt. We use Stable Audio Open [9] as the underlying generative model, due to its high-fidelity 44.1 kHz stereo output and flexibility of generating audio up to 47 seconds in length. As Stable Audio Open sometimes omits sounds present in captions with connectors, such as "Helicopter taking off with wind blowing and dogs barking", ", we fine- men-tuneii it on AudioCaps to improve prompt adherence on general audio. While fine-tuning improved the alignment between prompts and outputs, as detailed in Appendix [ achieving more consistent high-quality results also requires selecting a suitable combination of seed and Classifier- Free Guidance (CFG) [18] values. To this end, our approach consists of two stages, as shown in Fig. 3 First, our method explores different combinations of seed and CFG values to identify a configuration that produces satisfactory results for given input/output caption pair. Next, Coompttoo-rompp is applied to generate the edited audio pair.
[16] method, originally developed for image editing,
toonpt-to-Promt
Candidate Search. The process for finding a suitable candidate configuration (seed and CFG value) for each prompt pair is illustrated in Fig- 3 section (a) Candidate Search. We use Stable Audio Open to generate seven audio pairs, each with different configuration. Specifically, each pair is generated using 50 denoising steps, a randomly chosen seed, and CFG value sampled between 3 and 9. To assess the similarity between the generated audio and corresponding caption, we use the Gemini 2.0 Flash API. We instruct Gemini to perform perceptual quality check by analyzing all audible elements in the provided audio and assigning a score between and 10, based on how closely the generated audio resembles the provided caption. Samples that score above a threshold of proceed to the next stage, in which the CLAP similarity is calculated. Finally, we select the configuration that passes the Gemini filter and has the highest mean CLAP similarity across input and output.
Sample Generation. After finding suitable configuration for prompt, we move on to the sample generation as outlined in section (b) of Fig. B There are various parameters that can be configured in Prompt-to toopptto-Prroppt. The injection fraction Atse configures the extent to which attention maps from the input audio influence the generation of the output audio. fraction of 0 means that no attention maps from the input audio are injected. As a result, the output is not directly influenced by the input, apart from shared initial noise and sampling noise. In contrast, an injection fraction of enforces strong similarity such that the output audio closely resembles the input audio. Intermediate values allow
hmhh
httpa: org/ https://soundeeffece bberewind. co. .

---

Candidate Search
"npput"; "A fewate choir s singing", "instruction" "it should be nale choirt, "output": "A nale choir singingg
female choir singing Bood: 1. Text Gu danecc
A female choir singing Scod: 2: Toxt danedi
female choir singing Sood: S. Text Guida noc.
SAO
6- IM
male choir singing Bood: 1: danioi
A male choir singing Suedi Tuat danioi
A male choir singing Soudi
Gemini Fitter
Gemiri (Nio, Yee)
Gamini Filter: (Mio, Yoo)
Gemiri Filter (Miss, Noj
singing
Judge the simiirit..
CLAP: 12.30, 0.26) -.228
CLAP: (0.20, 0.26)=+0..2
7
Accepted
Rejected tue tolouee CLAP
Rejected dua to Ounara Hilor
male choir
Beed 1; lust duaturne
Bayesian Optim. trai 1
Injection Praction: 0.1 Altamiio Pluee lling 1.5
Hi d---ee +ja
Obj Trial 1: 0.63
SAO with
famale choir singing
A male choir singlng
Valuu
Boyesian Optimn. 2
Injection Dalay: 0.34
Value
0.53
Altartion Plaweo Lingg 1.2
female choir singing
male choir singing
2:
Sample Generation
Selected sample poir
Figure 3: Pipeline for ComppttooProopg audio generation. In (a), various seeds and CFG value combinations tested and filtered using Gemini and CLAP to identify suitable configurations for prompts. In (b) Stable Audio Open (SAO) with Prompt-to-Promp generates audio pairs using the seed and CFG configuration found in (a). A Bayesian Optimization process suggests Prompt-to- Prompt parameters and resulting samples are evaluated using an objective function. For clarity, only 3 candidate pairs and 2 Bayesian Optimization trials are shown.
for a balance between flexibility and audio similarity during generation. The injection delay Adllay controls at which point the attention injection begins, expressed as fraction of the total number of attention maps. A value of 0 indicates that the injection starts from the first attention map, while higher values delay the injection. For example, a value of 0.3 skips the first 30% of attention maps before the injection begins. As a constraint, it follows that Amme + attn < 1. Attention reweighting delay
weight specifies multiplier that increases attention to tokens that differ between the input and output captions. These tokens are identified by comparing the tokenized versions of both captions, and their corresponding cross-altention maps are scaled by the given factor. For example, value of 1.5 increases the attention on changed tokens or newly added tokens by 50%, which helps the model pay closer attention to proposed edits.
Using a fixed trompttto-Prompt configuration often leads to suboptimal results. While some in- structions involve only subtle changes (e.g., "add an echo effect"), others require more extensive modifications (e.g., "remove the people talking"). To handle this variability, we assign a unique parameter configuration
each prompt pair. Since manual tuning is infeasible at scale, on Bayesian Optimization [2] to automatically select suitable parameters. This setup requires an objective function that evaluates both the quality of the generated audio pair and the edit accuracy.
rely
Objective Function. We define an objective function that evaluates the quality of the generated audio pair and how well the edit was executed, using combination of multiple metrics. Building on insights from prior work, particularly InstructPix2Pix [3], as well as our own experiments, we identified the following metrics suitable. The output CLAP similarity Mout measures the cosine similarity between the CLAP embeddings of the generated output audio and the output caption. MaA measures the direction of change in the embedded CLAP space, initially introduced for the image domain П. It calculates the cosine similarity between the difference of the CLAP embedded audio pair and the CLAP embedded prompt pair. The CLAP audio similarity Ma measures the cosine similarity between the CLAP embedding of the input and output audio. The mel spectrogram audio similarity Maim calculates the multi-scale mel spectrogram loss (2 between the generated audio pair. The objective is computed as weighted sum and is maximized during Bayesian Optimization. Note that the multi-scale mel spectrogram loss is subtracted as lower values indicate better perceptual alignment.
Losj = w1 MCLL + Wy MM + wl MM W. Maim
(I)
Since not all metrics contribute equally. we conducted a small-scale listening study to determine effective weightings. Using our pipeline, we generated around 100 audio pairs with different weightings used for the objective function. Listeners were presented with an input audio clip, the edit
5

---

instruction, and two output audio samples selected by the objective functions initialized with different weightings. An ELO rating system was used rank the different weighting configurations based on their effectiveness in selecting high-quality, relevant edits. We found the following weightings to be effective: w1 = 8, W2 = 14, w3 = 0.5, W4 1.5.
Optimization. We run the Bayesian Optimization process for 10 trials for each caption pair. Specifically, we optimize Afru [0.3, 0.9), Auue [0.0, 0.6], and X weight E [1.0, 1.8). Stable Audio Open is configured to 50 denoising steps during each trial, while the final selected audio pair is generated using 100 denoising steps. Using fewer denoising steps during optimization significantly reduces runtime without affecting the relative ranking of generated samples, while the final 100 steps maximize audio fidelity once the optimal parameters are found.
3.3 DDPM Inversion
In addition to the fully synthetic Prompt-to-Prompt dataset, we create a semi-synthetic dataset of audio pairs using DDPM inversion to increase data diversity. The main benefit of this approach is that the input audio is real audio clip, while only the output audio is generated. We use the zero-shot text-based audio (ZETA) 33] implementation of DDPM inversion with Stable Audio Open as the underlying generative model. We provide the model with an existing input audio and caption, with the output caption generated as outlined in Section 3.1
Sample Generation. To use ZETA in an automated manner, several parameters need to be config- ured per sample. The CFGorc and CFGrrr parameters control the CFG strength of the input and output prompt, respectively. Unlike traditional inversion techniques, which are applied to the full denoising process, ZETA uses the parameter T stat to specify up to which timestep the inversion performed. Lower values for Taat result in high consistency with the input audio, while higher values enable more editing ficxibility. As the required edit strength varies across instructions, a fixed value would lead to under- over-editing. We therefore apply Bayesian Optimization with the objective function as defined Section 3.2
Optimization. For each caption pair, we run Bayesian Optimization for 7 trials. We search for optimal values within the following ranges: CFG E [1,3), CFG E [3, 10), and Taan E [18, 65]. Throughout optimization, Stable Audio Open is configured to use 70 denoising steps. Compared to Prompt-to-Pree trrmmpp we use fewer optimization trials and denoising steps to balance quality and runtime, given the higher computational cost of DDPM inversion.
3.4 Manual Edits
We further extend our dataset using a suite of twelve manually implemented operations, inspired by the approach from AUDIT We refer to these operations as manual edits, as each edit is implemented using standard, deterministic, and interpretable audio effects. The twelve editing tasks are ADD, REPLACE, DROP, SWAP, LOOP, PITCH, SPEED, LOw_PASS HIGH_PASS, INPAINT, SUPER_RES DENOISE. Each task has a set number of inputs, certain constraints on those inputs, and an optional controllable parameter. The input audio clips may be constrained by the number of unique audio elements, as analyzed Section 3.1 or by their duration. Some tasks accept parameters that introduce variability in their edits. For instance, the PITCH task allows control over the amount of pitch shifting, while the LOOP task accepts how many times the input audio should get repeated. The final output of each operation is audio pair consisting of the input and edited audio clip. All tasks, input constraints, and parameters are detailed Appendix B
and
Sample Generation. To create an audio editing triplet, we first sample one of the twelve tasks with equal probability. Based on the task's constraints, input audio clips are filtered before suitable candidates are randomly selected. After applying the edit operation, we synthesize the natural language instruction by passing the captions of the input audio clips, alongside the optional parameter value, to a task-specific generation process is provided with a description of the task and few-shot examples, allowing the model to produce tailored task-specific instructions. To increase instruction diversity and better reflect how real-worke users might phrase requests, we apply two post- post-procsssing steps using custom stages based on o4-mini. Each
--aaeddd instruction generation stage built on GPT-4.1 mini. Each
step is executed with 50% probability, independently. In the

---

Table 1: An example three-stage instruction generation pipeline for the ADD task, where the selected base audio "People talking in a roadside cafe" " is mixed with the target audio "A chirping bird". First, the initial stage combines the base and target audio captions to produce an edit instruction. Then, the variation stage rewrites with altemnate phrasing to increase diversity. Finally, the minimization stage compresses it into its most concise form. The last two stages are applied with 50% probability.
Generation Stage
Applied
Instruction
1. Initial
100%
"Add the sound of a bird chirping to the people talking in roadside cafe"
Variation
50%
" "Add some chirping birds to the chatter in roudside cafe"
3. Minimization
50%
"Add bird sounds"
variation stage, the instruction is rewritten without altering its meaning. In the minimization stage, the instruction is compressed into its most concise form. An example of this three-stage pipeline for the ADD task is shown in Table
4 Experimental Setup
4.1 SAO-Instruct
Dataset. To generate the dataset of audio editing triplets we use the three approaches outlined in Section 3 For Prompt-to-Prompt, input captions are taken from the AudioCaps [22 training split and a random subset of FreeSound from WavCaps [35.. For DDPM inversion, the input audio and caption are sourced from a random subset of AudioSet-SL [15]. Edit instruction and output captions are generated as described in Section 3.1] For the manual edits, we choose different subset of FreeSound from WavCaps to avoid overlaps with Prompt-to- to--roppt...... The final audio samples are stored as 44.1 1 kHz stereo WAV files. The computational resources used for the dataset generation are listed in Appendix C
Fine-tuning and Inference. We start from the open-source weights of Stable Audio Open and fine-tune the model on our created dataset of triplets: input audio, output audio, and edit instruction. SAO-Instruct uses three types of conditioning: (1) the text prompt is replaced with a free-form edit instruction, (2) the timing condition is set the length of the provided input audio, (3) an additional audio condition for the input audio is concatenated to the model's 's input channels. During training, the model learns to modify the provided input audio based on the edit instruction, such that the resulting output matches the reference output audio. During inference, we encode the input audio into the latent space of the diffusion model and add Gaussian noise. This noised latent is used as the initial starting point for the denoising process. Unless specified otherwise, we use 100 denoising steps and a CFG value of 5. More fine-tuning and inference details are found Appendix D and
4.2 Evaluation Metries
Objective Metrics. We use several metries for objective evaluation that capture both distributional similarity and perceptual quality. To measure how closely the edited samples match the distribution of real audio, we compute the Fréchet Distance (FD) 17,, the log spectral distance (LSDD), and the Kullback-Leibler (KL) divergence. To evaluate perceptual quality, we use the Inception Score (IS) (39. The FD, KL, and IS metrics utilize the PANNs (23 classifier. The LSD calculates the distance between spectrograms of original and target audio clips. To compute these metrics, use the evaluation pipeline provided by AudioLDM [30]. Additionally, we evaluate how well edits were performed using the CLAP 45 score, which measures the cosine similarity between the generated audio and the target caption in the CLAP embedding space. As CLAP is also used in our data generation pipeline to filter samples, it can bias results towards its embedding space. We mitigate this by not training SAO-Instruct with a CLAP loss and by reporting multiple objective and subjective metrics.
Subjective Metrics. For the subjective evaluation, we conducted listening study with 13 par- ticipants. Each participant was presented with 10 randomly selected 10-second audio clips from the AudioCaps test subset, including the original caption, edit instruction, and the corresponding
7

---

Table 2: Ablation study comparing the influence of different generated datasets on edit relevance and audio quality. FD, LSD, and KL are shown for both original and regenerated audio. Metrics reported with + indicate mean and standard deviation across evaluation samples. Outputs from SAO-Instruct fine-tuned on the combined dataset balance the strengths of the individual approaches, achieving both accurate edits and faithfulness to the input audio.
Original Ref.
Regenerated Ref.
Training Dataset
Samples
FD
| LSD 4 KL 4
FD
LSD
KL
IS f
CLAP f
Prompt-to- Prompt
50k
18.71
1.50
1.32
18.29
2.68
1.77
0.94 0.38 14
DDPM Inversion
50k
20.50
1.34
0.86
20.72
2.75
1.87
6.82+1.73 0.34
Manual Edits
50k
14.60
1.42
0.58
21.21
2.76
1.89
0.50
0.35++.115
Combined
50k
19.11
1,41
1.02
19.24
2.72
1.74
7.69
0.38
Combined-LLagge
150k
18.38
1.36
0.93
18.97
2.72
1.76
7.59:1.000 0.114
edited audio clips generated by the models. Model names were hidden from participants and the order of outputs was randomized on a per-sample basis. We use similar metrics to AudioEditor (0], where participants were tasked to rate each clip using the Mean Opinion Score (MOS) on discrete scale between and 5. Ratings were collected for three categories: Quality, the perceptual quality of the edited audio compared to the original input audio, Relevance, how well the edit was performed, and Faithfulness, the similarity between the input audio and the edited audio. The input audio refers to the original, unedited clip provided to the model for editing. The final MOS for each category was computed by averaging ratings across all participants and samples. Appendix Fiswws the evaluation interface and the instructions given participants.
5 Evaluation
We evaluate the performance of SAO-Instruct in an ablation study that measures the impact of the different methods for dataset generation, and compare it with audio editing baselines. We evaluate Ik 10-second samples from the AudioCaps test subset, where edit instructions and output captions generated using the approach described in Section 3.1 For the FD, LSD, and KL metrics two types of references: the original AudioCaps clips and synthetic audio generated from the target captions using Stable Audio Open. The original reference tries capture realism with natural audio, while the regenerated reference serves as a proxy to measure how well the edits were performed. Notably, lower FD, LSD, and KL scores do necessarily reflect better editing performance, as these metrics primarily measure the similarity to the reference audio. For instance, edited audio that is perceptually close the original audio clips may score well, even if the edit instruction ignored or only partially followed. By including both types of references, we aim to capture both naturalness and edit accuracy of generated outputs.
5.1 Ablation
To identify the effectiveness of the different dataset generation techniques, we fine-tune Stable Audio Open separately on datasets generated using Prompttto-rroppt, DDPM inversion, and manual edits, each consisting of 50k samples. Additionally, we evaluate two combined variants: with 50k samples with equal contribution from each of the three generation methods, and larger version containing 150k samples with 50k samples from each method.
The results of this ablation study are shown in Table Edits from SAO-Instruct fine-tuned on Prompt-to-Prontt Prompt showease high similarity with the regenerated reference. This indicates that the performed edits may be more accurate, while being less faithful to the original qualities of the input audio. In contrast, fine-tuning on DDPM inversion and manual edits, which are built on partial or fully real audio, may produce edits that are less precise while better preserving the characteristics and quality of the input audio. Overall, the combined approaches perform robustly across all metrics, balancing the advantages from the individual datasets.
Results.
8

---

Table 3: Comparison with zero-shot audio editing baselines, Results (mean + standard deviation) are shown for applicable objective and all subjective metrics. Inf. denotes the inference time per sample in seconds, measured on single NVIDIA A6000 GPU with batch size of 1.
Original Ref.
Regenerated Ref.
Subjective Metrics
Models
Inf. (8) 4 FD 4 LSD 4 KL., FD4 LSD 4. KL.
IS
CLAP
Quality
Relevance 1 Faithfulness
AndioEditor
79.49
17.21
1.73
1,40
25.97
2.32
1,46
10.01:40.60
0.48=00.12 3.22=1.01
3.33=1.35
2.75:+0.99
ZETA
15.31
24.65
2.27
1.64
17.01
2.55
1.26
9.06+0067
0.3 3.56=0883
3.25=1.25
2.95=
ZE5
17.78
27,91
2.69
1.86
18.88
2.59
1.36
9.07+0.72
0.36#0.13 3.28=1.00
3.04=1.24
2.75+1.12
SAO-Instruct
9.94
18.38
1,36
0.93
18.97
2.72
1.76
7,59++100
0.38 3,54
3.83+1.00 1.00
0.99
5.2
Comparison with Baselines
We compare SAO-Instruct with prior zero-shot audio editing baselines. The ZETA baseline 33 uses Stable Audio Open with 100 denoising steps as the underlying generative model, which was trained on FreeSound and the Free Music Archive (FMA) [6]. We evaluate two variants by adjusting the edit strength via the saat parameter, set to 50 and 75. ZETA has access to the full original and target audio description. We also compare our model to AudioEditor (20), which uses Auffusion [46] as its underlying model trained several audio datasets, including AudioCaps (22), WavCaps (35), and MACS [34]. Depending on the edit type, AudioEditor conditions on either the original or target description, along with the indices of words that changed between the descriptions. SAO-Instruct trained on a combined dataset of I 150k audio editing triplets, consisting of 50k samples each from Prompt-to- torpptt DDPM inversion, and manual edits. Unlike the baselines, access to the edit instruction, conveying significantly less information than full audio descriptions.
only has
The results of the comparison with baselines are shown in Table 3 SAO-Instruct has the fastest inference time, editing audio in just under 10 seconds, making it nearly 8x faster than AudioEditor. On the FD, LSD, and KL metrics, the models perform similarly, with no model dominating across both reference proxies. AudioEditor has the highest CLAP and Inception scores by significant margin, likely due to its additional information available the full target audio caption and longer inference time. However, this also leads to more aggressive edits that reduce the similarity with the original audio clip, reflected in its lower faithfulness score. SAO-Instruct shows strong performance in the subjective listening test and outperforms the other models in both edit relevance and faithfulness to the original audio, while maintaining high audio quality. Using only free-form instruction, InO-Intt demonstrates competitive performance and, in several cases, exceeds the results of models conditioned on full audio descriptions.
Results.
5.3 Limitations
While SAO-Instruct shows promising results, some limitations remain. The generation process for our Prompt-to-Promp and DDPM inversion datasets computationally expensive and constrained by the capabilities of the underlying generative model. Examples of failure cases are shown in Appendix G.3 While our work focuses on editing general audio, our approach could be applied to music editing, provided appropriate generative music models exist and triplets are constructed from music datasets. Future work could explore handling multi-step edits and extending support to languages other than English. See Appendix for a discussion on broader impacts.
6 Conclusion
We introduce SAO-Instruct, the first fully free-form instruction- audio editing model. SAO- Instruct can perform a wide-range of edit instructions, while preserving the overall context and coherence of the provided input audio. We propose a novel data generation pipeline utilizing Prompt- to-Prompt [16], DDPM inversion (33) and manual edits 44 to create diverse dataset of audio editing triplets. Our evaluations show that SAO-Instruct outperforms existing zero-shot editing baselines in subjective listening tests and achieves competitive performance on objective metrics. While prior baselines require information from both the input and target audio captions to guide the editing process, SAO-Instruct requires only single free-form edit instruction. We believe free-form instruction-basse audio editing unlocks new possibilities for creative audio workflows and hope that our released model weights and code will encourage further research in this area.

---

References
[1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, et al. MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325 2023. URL httph::ttaatttttt http:/iww.rrg/bb/////00 11325
[2] Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: next-generation hyper- parameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
[3] T. Brooks, A. Holynski, and A. A. Efros, InstructPix2Pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pages 13392-18402. 2023.
[4] Chen, Liu, L. Zhou, Liu, X. Tan, J. Li, S. Zhao, Y. Qian, and F. Wei. VALL-E 2: neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05377 00600530, 2024.
[5] Copet, F. Kreuk, Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Defossez. Simple and controllable music generation. In Advances in Neural Information Processing Systems, volume 36, pages 47704 47720. Curran Associates, Inc., 2023.
[6] M. Defferrard, K. Benzi, Vandergheynst, and Bresson. FMA: dataset for music analysis. In International Society for Music Information Retrieval Conference. 2017.
[7] A. Défossez, N. Usunier, L. Bottou, and F. Bach. Demucs: Deep extractor for music sources with extra unlabeled data remixed. arXiv preprint 11X9v:1909.011 2019.
[8] Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and Sutskever. Jukebox: A generative model for music. arXiv preprint orXiv:2005.00341 2020.
[9] Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons. Stable Audio Open, 2024. URL httts::////ttiiii .rgg
[10] S. Forsgren and H. Martiros. Riffusion stable diffusion for real-tine music generation. 2022.
URL htt.ttttt/tfffuiiiiiii com/ about
[11] R. Gal, O. Patashnik, H. Maron, A. H. Bermano, G. Chechik, and D. Cohen-Or. StyleGAN- NADA: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), (1 2022.
[12] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. AudioSet: An ontology and human-labelee dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[13] J. Hai, Y. Xu, H. Zhang, C. Li, H. Wang. M. Elhilali, and D. Yu. EzAudio: Enhancing text-to- audio generation with efficient diffusion transformer, 2024. URL httpt::/t//aat org/ abs/ 2409. 10819)
[14] B. Han, J. Dai, W. Hao, X. He, D. Guo, J. Chen, Y. Wang, Y. Qian, and X. Song. InstructME: An instruction guided music edit and remix framework with latent diffusion models. arXiv preprint arXiv:2308. 14360, 2023.
[15] S. Hershey, D. P. Ellis, E. Fonseca, A. Jansen, Liu, R. C. Moore, and M. Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 366-370. IEEE, 2021.
[16] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-or. Prompt-to- Prompt image editing with cross- attention control. The Eleventh International Conference on Learning Representations, 2023.
[17] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
10

---

[18] J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint 2022.
[19] Huberman- -Spegelgllll V. Kulikov, and Michaeli. An edit friendly DDPM noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12469-12478, 2024.
[20] Y. Jia, Chen, J. Zhao, Zhao, Zeng, Y. Chen, and Y. Qin. AudioEditor: A training-free diffusion-based audio editing framework. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2025.
[21] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang. Liu, Y. Leng, Song, S. Tang, 7. Wu, T. Qin, X.-Y. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and S. Zhao. NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models, 2024. URL https
//arxiv wwgg/tg///////////ee//../eaaeea/ /200//223.... 04030001000
[22] C. D. Kim. B. Kim, H. Lee, and G. Kim. AudioCaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
pages 119-132, 2019.
[23] Q. Kong, Y. Cao, T. Iqbal. Y. Wang. W. Wang, and M. D. Plumbley. PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions Audio, Speech, and Language Processing, 28:2880-2894, 2020.
[24] F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. Défossez, J. Copet, D. Parikh, Y. Taigman, and Y. Adi. AudioGen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023.
[25] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. High-fidelity audio compression with improved RVQGAN. Advances in Neural Information Processing Systems, 36:27980- 27993, 2023.
[26] M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V. Manohar, Y. Adi, J. Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation scale. Advances in neural information processing systems, 36, 2024.
[27] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey. haf-half-baked or well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 626-630, IEEE, 2019.
[28] S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon. BigVGAN: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.0466 04658, 2022.
[29] Y. A. Li, C. Han, V. Raghavan, G. Mischler, and N. Mesgarani. StyleTTS towards human- level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36: 16:19594-11 11994-196.. 19621, 2023.
[30] H. Liu, Z. Chen, Y. Yuan, X. Mei, X. Liu, D. Mandic, W. Wang, and M. D. Plumbley. Au- dioLDM: Text-to-audii generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learn- ing Research, pages 214744. PMLR, 2023.
[31] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05101, 2017.
[32] Y. Luo and N. Mesgarani. TasNet: time-domain audio separation network for real-time, single- channel speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 696-700. IEEF, 2018
[33] H. Manor and Michaeli. Zero-shot unsupervised and text-based audio editing using DDPM inversion. In Proceedings of the 41st International Conference on Machine Leurning, volume 235 of Proceedings of Machine Learning Research, pages 34603-34629, PMLR, 21-27 Jul 2024.
11

---

[34] I.
and A. Mesaros. What is the ground truth? reliability of anntiiannotttorr data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pages 76-80. IEFE, 2021.
[35] X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y. Zou, and W. Wang. WavCaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2024.
[36] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
[37] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6038-608 00476047,4 2023.
[38] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195-4205, October 2023.
[39] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung. A. Radford, and X. Chen. Improved techniques for training GANs. Advances in neural information processing systems, 29, 2016.
[40] U.-H. Shin, decoder for speech separation. Advances in Neural Information Processing Systems, 37: 52215-52240, 2024.
S. Lee, T. Kim, and H.-M. Park. Separate and reconstruct: Asymmetric encoder-
[41] A. Tjandra, Y.-C. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139 2025.
[42] R. Valle, R. Badlani, Z. Kong, S.-g. Lee, A. Goel, S. Kim, J. F. Santos, S. Dai, S. Gururani, A. Aljafari, al. Fugatto 1: Foundational generative audio transformer opus 1. In The Thirteenth International Conference
Learning Representations, 2025.
[43] A. Vyas, B. Shi, M. Le, A. Tjandra, Y-C.. Wu, B. Guo, J. Zhang, X. Zhang, R. Adkins, W. Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv::312. 15821, 2023.
[44] Y. Wang, Z. Ju, X. Tan, L. He, Wu, Bian, and Zhao. AUDIT: Audio editing by following instructions with latent diffusion models. In Advances in Neural Information Processing Systems, volume 36, pages 91340-71351 Curran Associates, Inc., 2023.
[45] Y. Wu, K. Chen, T. Zhang. Y. Hui, T. Kirg-Kirkaatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[46] J. Xue, Y. Deng, Y. Gao, and Y. Li. Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[47] R. Yamamoto, E. Song, and J.-M. Kim. Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199 0209. IEEE, 2020.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are substantiated in the main text of the paper. Specifically, our approach is outlined in Section 3 and our model is compared with baselines in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are outlined in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
Our work does not contain theoretical results or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our dataset generation approach is outlined in Section while our specific experimental setup is discussed Section and Appendix D Additionally, we provide all our code and model weights online: https: /e///gittuuu com heth disco sao- instruct.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code and model weights with documentation reproduce dataset and results: https:///gthh:ii com/co//////o//oo http:/thh-dicccooo instruct
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
coblic//uuides/Codeeuubiis
watassiinoooylyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
wowpdeee/oooSSSmiissssdseee www.ssssiicggsooooo for more details. / //nips. wttps/cccppblcccuui.eeeeeecccaea
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details for the dataset generation are discussed in Section Experimental setup is outlined in Section 4 with additional information in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We show the standard deviation for applicable objective metrics and for subjective metrics in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources used to generate the dataset and fine-tune the
models are described in Appendix C Cland Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification:
Our research conforms to the NeurlPS code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe safeguards in Appendix
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited all existing assets that used in our work. Additional licenses of dataset and models that we build upon are outlined in Appendix!!
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our approach generate the dataset of audio editing triplets is outlined
in Section 3
We describe SAO-Instruct and used datasets in Section 4.1 Additional
documentation to generate the dataset and use IAO-Instruct is released alongside our code at htttps://githu: com/ eth acc-ddan wwl-ddisssssso instruct
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The evaluation interface and instructions given to participants for our listening study are outlined in Appendix
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
Our listening study did not pose any risk to participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We describe our LLM usage for prompt generation in Section 3.1 Appendix and Section 3.4 manual edits.
A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hutt::://eerripsccc/oooofereeere /c/////neeeeeecc/////////1 for what should or should not be described.
19

---

A Prompt Generation
Caption Datasel
System Prompt
Generate syntheiic dalasel
"Input"; "Birds chirping and water flowing". "reasoning"; "There are two nain elenents "instruction": the flowing". "output": "Birds chirping". "input_neg" "output_neg": "water flowing".
Dogs barking and squealing
Birds chirping and water flowing
GPT-4o
Dogs barking squealing
"Cors crive...
Figure 4: Pipeline for prompt generation. A caption is taken from dataset and passed to an LLM, which produces an edit instruction and corresponding output caption. Additional metadata is generated for downstream filtering and for improving sample quality for synthetic audio generation.
GPT-4o to create dataset of structured prompts. Before generating outputs, the LLM reasons about the provided input caption to identify all audible elements and suitable edit instruction. The LLM also generates additional metadata, including a count of distinct audio elements of the input caption. Captions vary in complexity, with some containing only single audible element ("A meows") "), while others may have multiple distinet elements, such as "A man speaks during heavy traffic with emergency sirens going by". Current generative audio models often struggle accurately synthesize these complex captions, which can result in incomplete generations, with parts of the caption missing, or outputs with substantial audible artifacts. To address this, we filter prompts that contain more than two distinct audio elements.
Generation and Filtering. As illustrated in Fig. 4
Table 4: Examples of prompt generation. The input caption is taken from an existing captioning dataset, while the rest is generated using LLM. The reasoning step is not shown for brevity.
Input Caption
Instruction
Output Caption
Elem.
Neg. Input
Neg. Output
A man speaks as birds chirp
"Remove the birds chirping"
A man speaks
2
birds chirp
Thunder and rain
"add distant wind"
Thunder and rain with distant wind
2
distant wind
accelerates
"Change it matorcycle"
motoreycle accelerates
1
motorcycle
A plane takes off
"it should be further away"
A plane takes off in the distance
1
Negative Prompts. Depending on the edit instruction, the LLM also generates optional negative captions, which contain descriptions of audio elements that should be excluded from the input and output. These negative prompts help improve sample quality during synthetic audio generation by ensuring that elements are properly removed or added. For example, given the input caption "A man speaks as birds chirp" the model generates the edit instruction "Remove the birds chirping" " (cf. Table Since this instruction removes a sound, the negative input caption is empty, while the negative output caption contains "birds chirp". If the edit instruction instead adds negative input caption would contain that new element, while the negative output caption remains empty. Both negative input and output prompts are used during synthetic sample generation with Prompt-to-Pree to-Prompt. For DDPM inversion, only the negative output prompt is used, since the input audio generated.
sound, the
B Manual Edit Tasks
We define twelve editing tasks in total: ADD, REPLACE, DROP, SWAP, LOOP, PITCH, SPEED, ROW_PASS. HIGH JASS, INPAINT, SUPER_RES RES, and DENOISE, These tasks differ the number of inputs they require, the constraints those inputs, and whether they accept controllable parameters.
20

---

Table 5: The four editing tasks that accept parameters. The function len(-) returns the audio length in seconds.
Task/Inputs
Parameter
Description and Constraints
Example Instruction
ADD/2
Position
Mixes a larget audio into audio.
"Add the sound of barking dog to the beginning of the street ambience."
len(target) I T. len(base)
E {start, middle, end} [0, L
LOOP/1
Count
Repeats an audio multiple times.
"Repeat five times."
Z>o len(result) I 47s
PITCH/1
Semitones p
Shifts pitch up or down.
"Make the voice sound deeper by three notes."
p E [-12, 12]
SPEED/1
Factor
Changes speed without affecting pitch.
"Slow this clip down by about 30 percens."
~ LogUnif (1/3,
len(result) 2 17s
Input Constraints. While some tasks operate on multiple audio clips, others only require one. For example, ADD overlays two audio clips, while HIGH_ PAASS processes single clip by removing its low-frequency region. In addition to the number of inputs, each task may also impose constraints based the relative length of the inputs the number of distinct audio elements analyzed in Section 3.1 These constraints ensure that the edited audio does not exceed the 47 second limitation of Stable Audio Open and avoids producing samples that contain excessive number of distinct audio elements. For instance, DROP is configured to only remove a single element from a composite audio.
Parameters. Tasks can be separated into two groups based on whether they accept controllable parameters. The tasks ADD, LOOP, PITCH, and SPEED each accept one controllable parameter as detailed in Table The values of these parameters are all numerical, except for ADD, which accepts the insertion location either as a floating point timestamp or as one of the keywords "start", "middle", or "end", which are automatically converted into their corresponding timestamp values. These parameters also come with some restrictions. For example, the insertion location of the ADD tasks cannot insert the target audio before the start or beyond the length of the base audio, while the pitch shift of the PITCH tasks restricted to a full octave either direction. For tasks such as LOOP that change the length of given audio clip, the number of loops is constrained to ensure that the final audio does not exceed 47 seconds. The parameter values are sampled uniformly, unless stated otherwise, within the allowed values. The REPLACE, DROP, SWAP. LOW_ PASS, HIGH_ PASS, INP INNINT, SUPER_RES and DENOISE tasks do not accept parameters and are outlined in Table 6 For example, HISH simply removes the low-frequency region of a given audio.
C Dataset Generation
We outline the computational resources used for generating the dataset of audio editing triplets via the three approaches introduced in Section 3 While the dataset was generated using various GPUs, we report the average inference times based on a single NVIDIA A6000 GPU for consistency. For the Prompt-to-Promp dataset, we generate seven candidate pairs to find suitable seed and CFG values and filter them using Gemini and CLAP. For each prompt pair, this step takes on average 2.2 minutes. After finding suitable candidate, we perform the Prompt-to- -or-mtpp sample generation using Bayesian Optimization with 10 trials, which takes approximately 3.4 minutes per audio pair. For DDPM inversion, we use 7 Bayesian Optimization trials to find the optimal inversion parameters, taking 1.8 minutes per audio pair. generate 100k samples (50k from each method), we used total of 6.2k GPU hours. In contrast, generating manual edits is comparatively lightweight, no GPUs are required. The full pipeline, including audio manipulation and generating the edit instruction, takes approximately 5.1 seconds per sample single CPU core.
21

---

Table 6: The eight editing tasks that take no parameters. The function len(-) returns the audio length seconds and elem(-) counts the number of audible elements as generated in Section 3.1
Task/Inputs
Description and Constraints
Example Instruction
REPLACE/3
Swaps element (target) in composite
"Replace the engine hum with the sound of propeller plane."
audio (hase target) with another (replace).
elem(target) = clem(base)
len(target), len(replace) I len(base)
DROP/2
Drops sound (target) from composite audio (base + target).
"Remove the rain sounds from this outdoor recording.
elem(target) =
Ien(target) & len(base)
SWAP/2
Reorders two audio clips.
"Swap the order of these two sounds."
elem(first) = elem(second)
len(first) + len(sccond) < 47s
LOW_PASS/1
Removes high-frequency region.
"Filter the high-pitched noise from the recording."
cutoff at 8000 Hz
HIGH_PASS/1
Removes low-frequency region.
"Remove the bass rumble from the audio."
cutoff 1000 Hz
INPAINT/1
Fills in silent region.
"Restore the missing audio in the middle of this clip."
a ~ U(0, 95)
randomly blank out a% of input
SUPER_RES/1
Reconstruct high- seggffreyeeccy region.
"Enhance the quality of this low- frequency audio."
input resampled to 1 1/4th of its sample rate
DENOISE/1
Removes noise from the signal.
Remove the background hiss from this audio."
gaussian noise (0, 0.01) added to input
Table Performance of Stable Audio Open (SAO) and its fine-tuned variant on AudioCaps. The AudioCaps test subset was used for evaluation.
Models
FD 4 KI.
IS I
CLAP
SAO
41.64
2.19
8.5610.47 0.26:10.14
SAO + AudioCaps 20.85 1.42
10.05:0.51 0.46:10.11
D Fine-tuning
To fine-tune Stable Audio Open on both AudioCaps and our generated audio editing triplets, follow the guidelines in the official repository Fl and adopt the default optimizer and inverse learning rate scheduler with exponential warmup. Specifically, the AdamW (31 optimizer configured with a learning rate of 5e 5, (2), (0.9, 0.999), and weight decay of le 3. In both cases, we keep the autoencoder frozen and only train the diffusion transformer.
Stable Audio Open on AudioCaps. To improve prompt adherence of Stable Audio Open, we fine-tune the model on both the training and validation splits of AudioCaps (22) using a total of 47k samples. The model is trained for 15 epochs with batch size of 64 on two NVIDIA A100 GPUs for 30 hours. For evaluation, we provide the models with captions from the test subset of AudioCaps and generate audio using 100 denoising steps and CFG of 6. As shown Table 7 the fine-tuned version followed prompts more closely.
SAO-Instruct We train the diffusion transformer on two NVIDIA A6000 GPUs with batch size of 16 for epochs. Models trained on the individual datasets (50k samples each) were trained for 30 hours, while the final model on the large combined dataset (150k samples) was trained for 80 hours.
http::/://ggghhoooo///aaaaaaaaat aoti tools
22

---

Table 8: Comparing two inference configurations of SAO-Instruct. Starting from an encoded input audio with added Gaussian noise preserves more input audio characteristics while still providing enough editing flexibility.
Original Ref.
Regenerated Ref.
Initial Latent
FD
LSD 4 KL 4 FD 4 LSD L KL + IS 1 CLAP 1
Pure Noise
29.73
2.23
2.59
19.17
2.61
2.27 8.10 0.32=0.15
Audio + Noise 18.38
1.36
0.93
18.97
2.72
1.76
7.59:1..00 0.18
Table 9: Comparison with zero-shot audio editing baselines on Production Quality (PQ) and Produc- tion Complexity (PC) from AudioBox Aesthetics 41.
Model
PQ f
PC 1
AudioEditor
5.3810.91
3.02:10.72
ZETATi 550
6.06110.4
2.76+:0.6
ZETA 75
6.0410.91
2.7310.66
SAO-Instruct
5.61
3.23
E
Inference
We compare two inference configurations for SAO-Instruct in Table 8 sampling from pure noise and sampling from the latent substantially improves metrics computed against the original audio clips, which indicates better preservation of the input audio's characteristics. also achieves a higher CLAP score and lower FD/KL relative to the target cendion-conditioned regenerated reference, showing accurate instruction following. Overall, sampling from the noised latent preserves more input audio characteristics while still providing enough flexibility to perform the required edits.
latent of the encoded input audio. Starting from the noised
F Listening Study
The evaluation interface for the subjective listening study is shown in Fig. 5 The 13 participants were volunteers and received no compensation. They were given the following instructions to rate the model outputs:
Quality: How good is the sound quality of the edited audio compared to the input?
= Poor quality with strong artifacts, 5 Same quality as the input audio
Relevance: How well does the edited audio match the given instruction?
= Completely irrelevant to instruction, = Perfectly follows instruction
Faithfulness: How similar does the edited audio sound to the input audio?
= Completely different from the input audio, 5 Same as input audio
G
Results
G.1
Comparison with Baselines
To compare our model with zero-shot editing baselines, we also evaluate on objective metrics designed for automatic quality assessment of audio. Production Quality (PQ) measures technical aspects of audio quality, while Production Complexity (PC) focuses on the complexity of the audio scene [41] The results are shown in Table 9 We observe that SAO-Instruct slightly underperforms ZETA in production quality, while slightly outperforming ZETA in production complexity. These results reflect the findings from subjective listening study.
23

---

Audio Editing Evaluation
This wowk moskuation campares the pe performene difieent wudio wditing modebs based on sample edits. comparison based the lowling input?
Usten input comparison audio
Quality:
the eched
Relevance: How well does the edited audio match the given instructio??
Compacutyy inclosens so instruction 5 Pheflctly to instruction
Fulthiflnees: sinilar
audio
audio?
Sw between companieon clpp buttoon heyboard shortoctts (1-4))
Plyyppuuse comparson
by pressing
Sample
Input: People laughing
Instruction: Add clapping
Cond
Cond.
Gond.
Cond.
Qualty: 3
Quality
Qualit::
Quality:
Relevance:
Relevanoe:
Falttrunnes:
Falthfunnss:
Falthfulnes::
Fahthulnnes:
Back
Nisc
Figure 5: Evaluation interface for the subjective listening study comparing SAO-Instruct with audio editing baselines.
G.2 Deterministic Editing
We compare BAO-Instruch with the baselines on deterministic manual editing tasks using 100 samples per edit type drawn from the AudioCaps test set. As evaluation metrics, we provide the STFT loss, the Multi-Resolltionn STFT loss (MR-STFT) 47 the Multi-Resolution Mel-Spectrogram loss (MR-MEL) 251, the Scale-Invariant -einal-to-Dssootton Ratio (SI-SDR) [27]. and the Scale- invariant Signal-to-Noise Ratio (SI-SNR) 27 32) Together, these metrics provide perceptual- and signal-level view of the performance on these tasks. The input caption, edit instruction, and target captions generated using LLM based on the selected edit task, its optional parameter, and the captions from the selected input audio clips from AudioCaps. The edit types ADD and REPLACE are evaluated as their target audio ambiguous and not deterministic. For all other edit types the input audio and target audio can be created similarly as during dataset generation (cf. Section 3.4) and evaluated using common time- and frequency based metrics [7] 40 25 summary of the results
averaged over all deterministic manual editing tasks is given in Table 10 while the full per-task results are shown in TI
G.3 Qualitative Results
Fig. 6 demonstrates the editing capabilities of SAO-Instruct across a range of instruction types. In the first example, ambient noise is introduced in speech recording without distorting the primary signal. The second example shows global transformation where the volume of rain is reduced. The

---

Table 10: Average performance across all deterministic manual editing tasks.
Model
STFT 4
MR-STFT
MR-MEL + SI-SDR
SI-SNR t
AudioEditor
6.06:44.16
6.02+44.0
7.53+2254
-54.55+11.74
15.550+11.57
ZETATa==50
4.73#3.43
4.65+3..3
5.61#2.12
-28.75=13.68
-28.89#13..5
ZETATa.=77
5.21+:4.41
5.13+4.51
5.91+2.19
-30.65+14.85
-30.67+14483
SAO-Instruce 2.91#1.82 2.82+1183
3.47+1.33
-21.62+12281
-21.79+12.65
third example shows more localized operation, in which the sound of crumpling paper is replaced with typing, while the original speech remains intact. Notably, the model performs all edits based solely on the input audio and free-form edit instruction, with no access to either the original or target audio captions.
Woman speaking
heavy rain dies down and begins again
Crampling pager with
add amhient office noise &
mabe the
replace the crumpling poper with typing
A womnn speaking with ambieut office noise A quiet rain dies down and begins again
Typing noise with speech
Figure 6: Edits performed by SAO-Instruct. The model has only access to the input audio and the edit instruction. It is able perform global operations and local operations while keeping the overall background context intact.
Failure Cases While the performance of SAO-Instruct can be further improved by per-sample adjustments, such tuing the CFG scale the amount of noise applied to the initial encoded audio, some limitations remain. In Fig. 7 we observe that the phrasing of edit instruction can influence the edit quality and accuracy of the model. The model also occasionally struggles reconstruct coherent speech and may produce edits with audible artifacts. In Fig. 8 when adding elements, the newly added sounds sometimes fail to naturally blend with the background and instead appear overlaid on existing sound elements. Additionally. if clip contains many distinct elements, the model may be unable to alter sounds confuses them, leading unintended edits. These limitations primarily stem from insufficient data diversity and could be mitigated by training on larger and more diverse datasets.
H Broader Impacts
Our work introduces SAO-Instruct, model that enables free-form instruction-basee audio editing. It enables an intuitive and accessible way for users edit audio using natural language instructions. However, it also introduces potential risks of misuse, such as the manipulation of audio clips for deceptive or harmful purposes. mitigate these concerns, we not scrape data from arbitrary sources and only use well-established datasets that have been widely adopted in prior research. We encourage future work to further explore responsible deployment practices for instruction-base audio models and methods for detecting synthetically edited audio.
25

---

Table 11: Performance on all deterministic manual editing tasks.
Task
Model
STFT
MR-STFT MR-MEL 4 SI-SDR
SI-SNR f
AudioEditor
6.59+6..3
6.58+6.69
6.37+2.67
-51.24499226
-51.20++9.28
ZETAT
5.58:17222
5.5747.63
5.51#2.17
-20.21116.19
16.388
DROP
ZETAT T 75
6.27-18.88
6.2949941
5.93+2.31
-23.28+18.21
-3..06+
SAO-Instruce
6.12:t:1.63
6.06=111.68
4.4433088
133.77+16.86
-33.69:16.75
AudioEditor
1.12
6.07
11.7842.71
-60.00113.89
13.77
ZETA 55
4.39:t1.34
4.29+1.311
6.7042.30
-57.52+:00.8
-0.201
SWAP
ZETA TuE 75
4.56
4.4611.56
6.45
-56.4519.86
-56.5319.65
SAO-Instruct
1.08
4.25
7.32.2.17
-54.19:11.355
54.30
AudioEditor
6.34++2.23
6.21+2.19
10.3943.90
-55.31+12.119
-55.19
TETAT 550
1.70:t1.85
1.55
8.01
-18.13412.79
12.75
LOOP
ZETAT 75
5.11#2.29 4.95+2.27
8.2643334
-19.61+14.10
19.58=14.28
SAO-Instruct
2.01:10.58
1.94
2.3610093
-11.91:11.78
11.89913.76
AudioEditor
5.73:L4.04
5.7314.20
5.79
-51.73411.31
10.940
ZETAT, 50
5.44+:7.75
5.42+8832
4.78+
46.78+11.71
16.79+13.79
PITCH
ZETAT, Ju
5.95:18.82
5.9411944
5.08
148.66114.11
48.59+14.21
SAO-Instruct
2.63-1-0.70
2.49
3.11
41.39
41.31+14.42
AudioEditor
6.49:14.85
6.45
7.0243.03
-50.96:13..5
15.560112.52
ZETAT 4.9412.82
4.83+2.83
5.8412.44
45.93113.81
46.75=11.66
SPEED
ZETA 75
5.33:12.97 5.23
6.2242.54
47.288112.97
17.39
SAO-Instruct
3.53:1122
3.4311122
5.13#1137
-45.12113.79
46.15511.000
AudioEditor
4.94:12.117
4.9242.16
5.92
-53.40+10.43
-53.99+
ZETAT TETAT...:5 550
3.41+1.27
3.32+1.27
1.87+1.76
-16.07=13.30 13.30
LOW_PASS
ZETAT
3.79:11.54 3.70#1148
5.28
-17.76=15.49 15.47
IAO-Inst
1.63:10.45
1.56+0.47
1.8940.59
-2.26
10.95
AudioEditor
6.9517.75
6.9417.74
6.52
-62.28+12008
-62.25
ZETA Tu
4.8913342
4.8213.43
5.15
-31.05
13..0
HIGH_PASS
ZETA T 75
5.82++5.53
5.7145.46
5.65
-34.44+1585
14.52
SAO-Inttr
1.75:10.38
1.68:10.00
2.08:0.59
-28.48111.66
12.66
AudioEditor
6.50++3.80
6.43
9.8144.40
-5..40+12.39
11.971+1..92
ZETA
4.61::3.45
4.5413..4
5.33+2.13
-22.34+17.05
-2..51
INPAINT
ZETA
75
5.62:6.32 5.53
5.13
-23.49116.82
12.221
InO-Iust
2.49+11.0
2.42
3.31+2224
-13.88+13.20
14.61111285
AudioEditor
5.56++2.66
5.63
6.66+1.75
-52.83+12.23
12.33
ZETAT, 50
4.511:3.02
4.43133.2
4.68
-14.19111.74 11.71
SUPER_RES RES
ZETAT
4.75:t3.447
4.67+33.47
4.96+1.70
-16.97+14.71
-16.96+14.74
SAO-Instruct
2.16.1.0.1
2.10
2.36
-1.6819.70
-1.67.19.6
AudioEditor 5.30-16.88
5.2916.38
5.00
15.165+10.11
152.79+10.24
ZETAT 4.794.2.1
4.70+2.17
5.2842.02
-15.31+14.27
-15.16=14.28
DENOISE
ZETAT, 1E 4.9112.70
4.8312.70
6.1042.59
-18.37116.37
118.34116.62
SAO-Instruct
2.36.10.50
2.30
2.7411..6
3.39:11..3
-3.50=11.02
26

---

An alarm beeps while woman speaks
An alarm beeps while a woman speaks
remone the alarm
the alarm should be silent! 4
A speaks
A speaks
Figure An example failure case where the phrasing of an instruction impacts edit quality and accuracy. While "remove the alarm" fails to suppress the alarm, "the alarm should be silent!" is more successful.
A cat meowing
Drums, footsteps, frogs, and crickets are heard
1 add dog howling 1
I replace the drums with claps
A cat meowing and dog howling Claps, footsteps, frogs, and crickets are heard
Figure 8: Examples of failure cases where newly added sounds fail to blend naturally with the background and where edits in complex audio scenes lead to unintended edits.
I Licenses
We provide the licenses of datasets and models that we build upon in our work below. We ensure that use of these assets fully complies with their license and terms of use. AudioCaps [22] released under the MIT License. AudioSet [12] and AudioSet-SL. [15] are available under the 35 is available for academic
and includes audio data from multiple sources. We follow the licensing terms for the BBC Sound Effects library and respect the licenses associated with each audio clip from FreeSound F
CC BY 4.0 license. WavCaps
We comply with the Stability AI Community License Agreement [ which allows the use of Stable Audio Open O for research purposes. CLAP 45) is licensed under the Creative Commons CCO 1.0 Universal license, which allows unrestricted use and distribution.
http::::///uundefffttte.tteee www.:tttt..bbbbeindddde co. otg.e//ccccc...eeeoooaaaooooaaao
27


Reviews:
[{"review_id": "O445Dhk9FH", "reviewer": "Reviewer_t9mM", "summary": "This paper presents **SAO-Instruct**, a model fine-tuned from the Stable Audio Open model on triplets (audio, text instruction, target audio) to enable free-form, instruction-based audio editing. The approach uses a supervised training pipeline and proposes a scalable method for generating synthetic instruction-editing data using a combination of prompt-to-prompt, DDPM inversion, and manual editing methods. For prompt-to-prompt and DDPM inversion, an LLM is used to generate paired instructions and output captions, which are then filtered using a CLAP-based objective function. For manual editing, a set of twelve editing tasks inspired by AUDIT is used. Experiments on datasets constructed from AudioSet, Freesound, and AudioSet-SL (50K for ablations and 150K for final training) demonstrate the effectiveness of SAO-Instruct, which significantly outperforms baselines in subjective evaluations.", "questions": "- The authors mention fine-tuning SAO on AudioCaps to improve prompt adherence for general audio. It may also be useful to report metrics such as KLD, IS, and FAD, which are commonly used in the literature.\n- Table 3 suggests that objective scores do not align with subjective evaluations. Have you considered using directional CLAP, similar to your filtering setup, to better capture relevance? For quality, have the authors looked at perceptual metrics like those proposed in [3,4]?\n- Lines 259–260: \"During inference, we encode the input audio into the latent space of the diffusion model and add Gaussian noise...\"\nWhy is the noised latent used during inference, this seems to be different from training time?  Did the authors compare this with sampling from noise directly? Adding those results will be useful.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- The paper is well written and generally easy to follow.\n- SAO-Instruct significantly extends the coverage of text instructions for audio editing and outperforms baselines, especially on relevance and faithfulness. The inference speedups are also quite substantial.\n- The proposed data generation pipeline for instruction-based audio editing is scalable, and the released code for sample generation and training would help with reproducibility.\n\n**Weaknesses**:\n- The dataset ablations are not very convincing, particularly because in Table 3 subjective and objective metrics do not align well. It would be more useful if other metrics like in [3, 4] were included that align better with subjective judgments or evaluations\n- The reliability of the objective metrics is unclear. The authors suggest using both the original and generated audio as references; however, for complex edits, measuring KL or LSD against the original audio may not be meaningful, as the event distribution can change significantly.\n- Several choices in the data generation pipeline are not well-motivated, and discussion is missing:\n    - Prompt-to-prompt sample generation uses 100 steps (line 200) and DDPM uses 70 (line 218).  A discussion of this design choice would help.\n    - Lines 247-250 describe the data composition for Prompt-to-prompt, DDPM, and manual edits without much discussion around the choices or sampling strategy if any.\n    - Line 286: \"We evaluate on 1k 10-second samples from the AudioCaps test subset...\" — the composition of the test set is unclear. Given that manual and prompt-to-prompt edits may differ in distribution, this could influence conclusions. Would the authors consider releasing the test set?\n\n*Minor Comments / Suggestions**\n- Lines 68-72: \"One downside of using mel spectrograms ... continuous latent representation\"\n  While it is true that Mel-Spectrogram based vocoders can suffer from missing phase information,  VAE-based continuous representations can also introduce artifacts. BigVGAN-based vocoders perform well in practice, so this statement could be rephrased to present a more balanced view.\n- Besides the prompt-to-prompt and zero-shot editing approaches discussed in Section 2.2, diffusion models [1, 2] trained with masked in-filling support editing via masked audio + target text. These models could be used to generate instruction fine-tuning data for datasets like AudioSet-Strongly Labelled, which provides event timestamps. Including this perspective may be useful for a general audience.\n\n**References**:\n[1] Unified Audio Generation with Natural Language Prompts. Arxiv 2023\n\n[2] Fugatto 1: Foundational generative audio transformer opus 1. ICLR 2025\n\n[3] Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sounda. Arxiv 2023 \n\n[4] https://github.com/wavlab-speech/versa", "was_revised": true, "final_justification": "I have read the review and rebuttals and decided to keep my original rating", "rebuttal": "We thank the reviewer for their valuable insights and address their feedback in the following:\n\n> The reliability of the objective metrics is unclear. The authors suggest using both the original and generated audio as references; however, for complex edits, measuring KL or LSD against the original audio may not be meaningful, as the event distribution can change significantly.\n\nWe thank the reviewer for this remark. The main challenge we face during evaluation is the lack of ground-truth edited general audio pairs. Following AudioEditor [1], we use two complementary reference proxies: (i) scores against the original audio clips to measure quality and content preservation to guard against over-editing, and (ii) scores against a regenerated sample conditioned on the target caption to approximate edit accuracy. These scores reflect similarity to their respective references rather than absolute correctness and should be evaluated jointly to assess the model’s performance. As future work, datasets exploring edited general audio pairs would enable a more direct evaluation of edit accuracy.\n\n> Prompt-to-prompt sample generation uses 100 steps (line 200) and DDPM uses 70 (line 218).\n\nWe use different step counts because these methods have different goals and compute profiles. For Prompt-to-Prompt, we found that generating the final selected pair with 100 denoising steps results in high-quality synthetic samples. The DDPM inversion approach using ZETA consists of two stages: (i) the input audio is inverted to an initial latent up to T_start, (ii) this latent is conditioned on the target caption and denoised for 70 steps. As this two-stage process results in a longer inference time compared to Prompt-to-Prompt, 70 denoising steps gave a good quality-compute trade-off for ZETA. We clarified these choices in the revised manuscript. \n\n> Lines 247-250 describe the data composition for Prompt-to-prompt, DDPM, and manual edits without much discussion around the choices or sampling strategy if any.\n\nWe use a random subset of AudioSet‑SL and Freesound, plus the full AudioCaps train split. We then compute the number of audible elements per caption using the procedure in Section 3.1 (Prompt Generation) and discard any sample with >2 elements, as mentioned in Appendix A.\n\n> \"We evaluate on 1k 10-second samples from the AudioCaps test subset...\" — the composition of the test set is unclear. Given that manual and prompt-to-prompt edits may differ in distribution, this could influence conclusions. Would the authors consider releasing the test set?\n\nWe thank the reviewer for raising this point. We are considering releasing the prompt test set for reproducibility which contains input captions from AudioCaps. However, AudioCaps states that their dataset may only be used for academic purposes.\n\n> Minor Comments / Suggestions\n\nIn regards to the minor comments and suggestions: We agree that mel-spectrogram based vocoders have achieved great results and have revised Section 2.1 to provide a more balanced view. We also expanded Section 2.2 to mention current work in masked in-filling editing, including Fugatto.\n\n> The authors mention fine-tuning SAO on AudioCaps to improve prompt adherence for general audio. It may also be useful to report metrics such as KLD, IS, and FAD, which are commonly used in the literature.\n\nWe thank the reviewer for this suggestion. In the current manuscript, we compared SAO with its version fine-tuned on the AudioCaps training subset using CLAP similarity. We now also report FD, KL, and IS to better motivate our choice to fine-tune SAO, with all metrics evaluated on the same AudioCaps test subset for consistency. We report FD computed with PANNs features rather than FAD with VGGish, as the PANNs classifier provided more reliable results in our experiments and in prior work [2].\n\n|  Model                                  | FD $\\downarrow$ |  KL $\\downarrow$ |       IS $\\uparrow$ |  CLAP $\\uparrow$ |\n| -------------------------------- | --------------------: | --------------------: | --------------------: | --------------------: |\n| SAO                                      |                 $41.64$ |                    $2.19$ |   $8.56 \\pm 0.47$ |    $0.26 \\pm 0.14$ |\n| SAO AudioCaps Fine-tuned |                 $20.85$ |                    $1.42$ | $10.05 \\pm 0.51$ |    $0.46 \\pm 0.11$ |\n\n> Lines 259–260: \"During inference, we encode the input audio into the latent space of the diffusion model and add Gaussian noise...\" Why is the noised latent used during inference, this seems to be different from training time? Did the authors compare this with sampling from noise directly? Adding those results will be useful.\n\nWe have added an ablation table comparing two inference modes for SAO-Instruct: sampling from pure-noise versus sampling from the Gaussian-noised latent of the encoded input audio. Metrics with * are computed against the regenerated reference, while the others use the original audio. Starting from the noised latent substantially improves metrics computed against the original audio clips, which indicates better preservation of the input audio's characteristics. It also achieves a higher CLAP score and lower FD/KL relative to the target caption-conditioned regenerated reference, showing accurate instruction following. Overall, sampling from the noised latent preserves more input audio characteristics while still providing enough flexibility to perform the required edits. We will include these results in the Appendix of the revised manuscript.\n\n| Reference                                             | FD $\\downarrow$ | LSD $\\downarrow$ | KL $\\downarrow$ | FD* $\\downarrow$ | LSD* $\\downarrow$ | KL* $\\downarrow$ | IS $\\uparrow$ | CLAP $\\uparrow$ |\n| ------------------------------------------- | :--------- | :-------- | :------ | :--------- | :------- | :------- | :----------------- | :------------------ |\n| SAO‑Instruct (without encoded audio) | $29.73$ | $2.23$ | $2.59$ | $19.17$ | $2.61$ | $2.27$ | $8.10 \\pm 0.69$ | $0.32 \\pm 0.15$ |\n| SAO‑Instruct (with encoded audio)      | $18.38$ | $1.36$ | $0.93$ | $18.97$ | $2.72$ | $1.76$ | $7.59 \\pm 1.00$ | $0.38 \\pm 0.14$ |\n\n\n[1] Jia, Yuhang, et al. \"AudioEditor: A training-free diffusion-based audio editing framework.\" ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.\n\n[2] Liu, Haohe, et al. \"AudioLDM: Text-to-audio generation with latent diffusion models.\" Proceedings of the 40th International Conference on Machine Learning. 2023."}, {"review_id": "4Hrrqo0yLr", "reviewer": "Reviewer_HFny", "summary": "The paper presents SAO-Instruct, which fine-tunes the previous stable audio open into an audio editing model. The main efforts of this are in the data synthesis, which aims to generate the (input audio, instruction, output audio) triplets. The author presents 3 different data synthesis methods, aka, prompt-to-prompt, DDPM inversion, and manual edition. \n\nThe evaluation is on AudioCaps, with subjective and objective evaluation results.\nSAO-Instruct t shows similar generation quality to previous baseline models, while claiming better relevance and faithfulness.", "questions": "As above", "limitations": "Some recent peer-reviewed works are missing from the related work. Especially, Fuggato has claimed the free-form instruction-following and editing.\n\nhttps://openreview.net/forum?id=B2Fqu7Y2cd\nhttps://arxiv.org/abs/2412.19351", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n(1) The work is well-motivated, with clear writing. Although most techniques are inherited from image generation/editing, it has been well adapted in the audio generation scenarios. \n\n(2) The authors present 3 different synthesis techniques, which cover fully-synthetic, partially-synthetic, and manual edits, which makes the exploration comprehensive. \n\n(3) The evaluation contains both subjective and objective aspects, with multiple aspects and metrics.\n\nWeakness:\n(1) As the main efforts of this work should be considered in data synthesis, I would recommend the authors to provide more insights into this process. Specifically, for each of the three data synthesis methods, how do you monitor the intermediate results quality, what are the statistical numbers along this way, and how is data quality control achieved and impact the overall results. Since the model is only fine-tuned on 50k samples, I would expect the authors to run more ablation studies to dig out what factors during the data synthesis would have the most impact on the final performance.\n\n(2) The author claims their model can follow the free-form instructions. However, when using the manual edition method alone (note this is implemented with a closed set of operations), it shows close performance with the other two methods. That's to say, under the evaluation paradigm provided by the authors, the ability to follow free-form instruction is not well justified.\n\n(3) When claiming \"achieving both accurate edits and faithfulness\" in Table 2, do the authors want to mention the MOS scores like Table 3?\n\n(4) This is optional, but is recommended: can the authors provide some generated samples for demonstration?", "was_revised": true, "final_justification": "The initial manuscript is already good enough, which provides a good implementation for free-form audio generation. My concern/questions are mostly based on the data and implementation details, which have already been well addressed during rebuttal stage. I have no more concerns.", "rebuttal": "We thank the reviewer for their valuable insights and address their feedback in the following:\n\n> (1) As the main efforts of this work should be considered in data synthesis, I would recommend the authors to provide more insights into this process. Specifically, for each of the three data synthesis methods, how do you monitor the intermediate results quality, what are the statistical numbers along this way, and how is data quality control achieved and impact the overall results.\n\nWe agree that the quality of our generated data is critical for our approach. This is why we use a two-stage pipeline for the fully synthetic sample generation using Prompt-to-Prompt. Candidates (seed/CFG values) are first filtered using a perceptual Gemini check and CLAP similarity before the final sample is generated. Because intermediate statistics can be misleading across very different free-form edits (e.g., a subtle “add a light echo” vs. a structural “remove people talking”), we relied on subjective listening during development to set suitable ranges for the parameters we optimize (e.g., attention injection strength). To have a proxy for human judgement of generated samples at scale, we defined an objective function used during Bayesian Optimization in Prompt-to-Prompt and DDPM Inversion that combines several metrics. We calibrated the metric weights in a small scale listening study where listeners compared samples picked by the objective function initialized with different weightings. Finally, we selected the top-ranked weight initialization based on an ELO rating as described in Section 3.2 (Objective Function). We clarified our reliance on subjective listening for design decisions and the selection of parameter ranges in the revised manuscript.\n\n> Since the model is only fine-tuned on 50k samples, I would expect the authors to run more ablation studies to dig out what factors during the data synthesis would have the most impact on the final performance.\n\nWe report an ablation study isolating the contribution of each proposed data generation method (Prompt-to-Prompt, DDPM inversion, and manual edits) and their combination, showing complementary gains (quality and edit accuracy) when all are used together. During development we conducted internal listening tests that informed our in-method design decisions (e.g., the perceptual Gemini filter during Prompt-to-Prompt). Exhaustive ablations for each data generation method, especially for the fully or semi-synthetic data generation methods, would require a significantly higher compute budget. Therefore, our ablation study focuses on comparing the three methods and their combination.\n\n> (2) The author claims their model can follow the free-form instructions. However, when using the manual edition method alone (note this is implemented with a closed set of operations), it shows close performance with the other two methods. That's to say, under the evaluation paradigm provided by the authors, the ability to follow free-form instruction is not well justified.\n\nWe thank the reviewer for this observation. The model fine-tuned on manual edits scores well on metrics computed against the original audio indicating high content preservation and quality. However, it underperforms on metrics computed against the caption-conditioned regenerated reference, which indicates weaker edit instruction adherence. This trade-off is also evident in the MOS for the ablation study that we provide in this response. While the manual-only model can still follow some free-form edit instructions, mainly due to the Stable Audio Open pretraining and our diverse edit instruction generation, it cannot follow the same broad range of free-form edits covered by the Prompt-to-Prompt and DDPM pipelines (e.g., “make the car drive on gravel instead”).\n\n> (3) When claiming \"achieving both accurate edits and faithfulness\" in Table 2, do the authors want to mention the MOS scores like Table 3?\n\nWe provide the MOS for the ablation in the following table:\n\n| Training Dataset     | Quality $\\uparrow$ | Relevance $\\uparrow$ | Faithfulness $\\uparrow$ |\n| --------------------- | ---------------------: | --------------------------: | --------------------------: |\n| Prompt-to-Prompt | $3.32 \\pm 0.79$     | $3.92 \\pm 0.80$           | $3.48 \\pm 0.83$             |\n| DDPM Inversion     | $3.42 \\pm 0.84$     | $2.80 \\pm 1.16$           | $3.90 \\pm 0.68$             |\n| Manual Edits          | $3.78 \\pm 0.95$     | $2.00 \\pm 1.01$           | $4.48 \\pm 0.61$             |\n| Combined              | $3.52 \\pm 0.81$     | $3.76 \\pm 1.10$           | $3.56 \\pm 0.76$             |\n\nFine-tuning on the Prompt-to-Prompt dataset achieves the highest edit relevance but at a slight cost in quality and faithfulness to the input audio (it can over-edit and introduce perceptual artifacts). DDPM inversion improves quality and faithfulness, but tends to follow instructions less precisely. Using only manual edits achieves the best faithfulness and quality but suffers from low edit relevance due to the closed operation set. The combined setting balances the advantages of these methods and achieves high relevance while maintaining strong quality and faithfulness to the input audio characteristics. \n\n> (4) This is optional, but is recommended: can the authors provide some generated samples for demonstration?\n\nThe manuscript on page 2 (Line 57) includes a link to our online demo page with generated samples. We will make it more prominent in the revised manuscript.\n\n> Some recent peer-reviewed works are missing from the related work. Especially, Fuggato has claimed the free-form instruction-following and editing.\n\nWe thank the reviewer for noting this. We will expand the related work section in our manuscript to include these recent works and briefly clarify how our setting and contributions differ. \n\nBased on our feedback and proposed improvements, would the reviewer be willing to reconsider their score?"}, {"review_id": "34AQNq6RuO", "reviewer": "Reviewer_AzCo", "summary": "SAO-Instruct is an instruction-based audio editing framework that fine-tunes Stable Audio Open’s diffusion transformer on a dataset of editing triplets generated via Prompt-to-Prompt, DDPM inversion, and manual DSP edits, enabling users to modify existing clips with plain-English instructions.", "questions": "See weaknesses mentioned above.", "limitations": "In the limitations sections, the authors mentioned their approach “can naturally extend to music”. The training data is drawn from event‐focused caption datasets (AudioCaps, WavCaps, AudioSet) and manually edited effects. None of these guarantee coverage of the complex patterns found in music. For example, you are not able to simply construct a regenerated reference of the instruction “add a guitar track to this song”. The regenerated reference pipeline ignores the source audio. It doesn’t know the melody, key, or ambience of your particular song, so it will produce a new guitar-backed track rather than a faithful edit of your input.\nThus, I am strongly against to the statement that their approach “can naturally extend to music”.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Free-form, instruction-based editing. \n- Combines three complementary data-generation pipelines, fully synthetic Prompt-to-Prompt, DDPM inversion, and real manual DSP edits.\n- Fast inference. At ≈9.94 s per 10 s clip on an NVIDIA A6000 GPU, SAO-Instruct is nearly 8× faster than AudioEditor.\n\nWeaknesses:\n- The “regenerated reference” is produced solely by sampling from Stable Audio Open using the target caption, with no conditioning on the original input clip. This isn’t a true “ground-truth” edit of the input audio, but rather a brand-new, synthetic sample that may lose all source-specific content.\n- In Section 4.2 the authors define FD/LSD/KL (and IS, CLAP) as their objective metrics, but these are always computed and reported as a single number averaged over all instructions. I want to point out that using the same set of evaluation criteria for different editing operations is not reasonable. For operations like LOOP with instructions as “repeat the audio 5 times”, we want the model to repeat the exact same audio 5 times. I would expect the edited audio to be exactly the same as the ground truth to be considered as a successful edit. However, none of the metrics in Table 2 and 3 are able to provide information like this because the performance values are averaged over different operations. While LOOP is defined as “repeat the audio l times” in Table 5 (with the constraint len(result) ≤ 47 s), there is no mechanism in the evaluation to verify how many loops were actually produced. A perfect DSP-based loop would trivially get an exact-match score, but the paper’s FD/LSD/KL cannot distinguish “five perfect repeats” from “four repeats plus a cut-off,” nor do they introduce any operation-specific metric (e.g. loop-count accuracy). I don’t think simply relying on subjective evaluation for these kinds of requirements are suitable as they are not precise enough.\n- The subjective evaluation is unreliable due to these two reasons:\n1. The scores are based on a 1 to 5 rating, while only 1 and 5 are provided with a criteria. 2, 3 and 4 are undefined. Human annotators do not have a clear criteria for the ratings. Please refer to AudioBox on how to define criteria for ratings of 1 to 5.\n2. The design of Faithfulness in Section 4.2 (Subjective Metrics) is not reasonable and very confusing. Faithfulness is defined as “the similarity between the reference and the edited audio” in Section 4.2, while mentioned in Appendix, it is defined as “How similar does the edited audio sound to the input audio?” By simply reading through these statements, one would consider the term “reference” referring to the ground truth edited audio, while “input audio” is the original audio intended to be edited. Even if there is no confusion in the term “reference”, setting the criteria of faithfulness as “How similar does the edited audio sound to the input audio?” makes no sense because isn’t the goal of audio editing intended to modify the input audio into something different? Why do we need a rating of 5 indicating that the edited audio is “same as input audio”, which means that a rating of 5 indicates that the model did not do edits, while this metric is the higher the better?\n- Instructions are generated via a templated LLM pipeline and evaluated only in English. The paper does not explore robustness to paraphrases beyond its few-shot LLM variations, nor to multi-step or compound edits (for example “speed up then repeat”).", "was_revised": true, "final_justification": "I will upgrade my score from 2 to 4 as the authors provided separate evaluation results for 10 types of editing operations.", "rebuttal": "We thank the reviewer for their valuable insights and address their feedback in the following:\n\n> The “regenerated reference” is produced solely by sampling from Stable Audio Open using the target caption, with no conditioning on the original input clip. This isn’t a true “ground-truth” edit of the input audio, but rather a brand-new, synthetic sample that may lose all source-specific content.\n\nWe agree that a regenerated reference based on the target caption is not a ground-truth edit. Evaluating the edit accuracy is challenging as paired ground-truth edited general audio datasets are unavailable. This is why we report two complementary proxies in accordance with previous work [1]: (i) scores against the original audio clips to measure quality and content preservation to guard against over-editing, and (ii) scores against this regenerated reference conditioned on the target caption to approximate edit accuracy. These metrics should be interpreted jointly to assess the model’s ability to perform accurate and faithful edits. To compare our model with other baselines, we additionally perform a human listening study to reduce reliance on these reference-based metrics.\n\n> In Section 4.2 the authors define FD/LSD/KL (and IS, CLAP) as their objective metrics, but these are always computed and reported as a single number averaged over all instructions. I want to point out that using the same set of evaluation criteria for different editing operations is not reasonable. For operations like LOOP with instructions as “repeat the audio 5 times”, we want the model to repeat the exact same audio 5 times. I would expect the edited audio to be exactly the same as the ground truth to be considered as a successful edit. However, none of the metrics in Table 2 and 3 are able to provide information like this because the performance values are averaged over different operations. While LOOP is defined as “repeat the audio l times” in Table 5 (with the constraint len(result) ≤ 47 s), there is no mechanism in the evaluation to verify how many loops were actually produced. A perfect DSP-based loop would trivially get an exact-match score, but the paper’s FD/LSD/KL cannot distinguish “five perfect repeats” from “four repeats plus a cut-off,” nor do they introduce any operation-specific metric (e.g. loop-count accuracy). I don’t think simply relying on subjective evaluation for these kinds of requirements are suitable as they are not precise enough.\n\nWe agree that for deterministic edits such as LOOP operation-specific evaluation measures (e.g., loop-count accuracy) are more appropriate then our chosen reference-based metrics. Our work focuses on free-form instruction-based editing, where edits vary widely in type of strength and are therefore difficult to categorize and evaluate. This motivated our choice of using reference proxies to that measure (i) similarity to original audio clips to measure quality and content preservations, and (ii) similarity to a regenerated sample conditioned on the target caption to approximate instruction relevance. To evaluate aspects not captured by these reference proxies, we additionally conducted a subjective listening study. We will clarify in the revised manuscript that aggregate metrics are intended to be interpreted jointly and will note the value of task-specific validators (e.g., loop-count checks) for future benchmarks. \n\n> The scores are based on a 1 to 5 rating, while only 1 and 5 are provided with a criteria. 2, 3 and 4 are undefined. Human annotators do not have a clear criteria for the ratings. Please refer to AudioBox on how to define criteria for ratings of 1 to 5.\n\nWe thank the reviewer for this suggestion. All participants in our listening study were recruited from our lab and briefed prior to the study. While they were familiar with such listening tests, we acknowledge that providing text anchors for every point on the 1-to-5 scale can improve consistency and reduce potential confusion. We will adopt this practice in future listening tests.\n\n> The design of Faithfulness in Section 4.2 (Subjective Metrics) is not reasonable and very confusing. Faithfulness is defined as “the similarity between the reference and the edited audio” in Section 4.2, while mentioned in Appendix, it is defined as “How similar does the edited audio sound to the input audio?” By simply reading through these statements, one would consider the term “reference” referring to the ground truth edited audio, while “input audio” is the original audio intended to be edited.\n\nWe thank the reviewer for pointing out this ambiguity. We will revise the explanation of the subjective metrics in Section 4.2 to use the term “input audio” instead of “reference”. For clarity, we will explicitly state that “input audio” refers to the original audio provided for editing, as described in the instructions given to the listening study participants.\n\n> Even if there is no confusion in the term “reference”, setting the criteria of faithfulness as “How similar does the edited audio sound to the input audio?” makes no sense because isn’t the goal of audio editing intended to modify the input audio into something different? Why do we need a rating of 5 indicating that the edited audio is “same as input audio”, which means that a rating of 5 indicates that the model did not do edits, while this metric is the higher the better?\n\nBecause evaluating Faithfulness or Relevance in isolation can be misleading, we interpret them together: Faithfulness quantifies how many acoustic characteristics of the original clip survive the edit, whereas Relevance measures how accurately the edit instruction is carried out. For example, consider the input “birds chirping with water flowing” and the instruction “add dogs barking.\"\n- If the output keeps the birds and water and adds dogs, both metrics are high.\n- If birds disappear but dogs are added, Relevance stays high while Faithfulness drops.\n- If dogs are never added, Faithfulness may remain high yet Relevance is low.\n\nAs shown in previous work [1], jointly examining these two metrics reveals whether the model both follows the instruction and preserves the essence of the source audio.\n\n> Instructions are generated via a templated LLM pipeline and evaluated only in English. The paper does not explore robustness to paraphrases beyond its few-shot LLM variations, nor to multi-step or compound edits (for example “speed up then repeat”).\n\nWe thank the reviewer for raising this point. Because our underlying model (Stable Audio Open) was mainly trained on English data, we restricted our pipeline and evaluation to English instructions. While our work focuses on single free-form edits, we agree that compound edits are an interesting direction for future work. We will note these limitations in the revised manuscript.\n\n> In the limitations sections, the authors mentioned their approach “can naturally extend to music”. The training data is drawn from event‐focused caption datasets (AudioCaps, WavCaps, AudioSet) and manually edited effects. None of these guarantee coverage of the complex patterns found in music. For example, you are not able to simply construct a regenerated reference of the instruction “add a guitar track to this song”. The regenerated reference pipeline ignores the source audio. It doesn’t know the melody, key, or ambience of your particular song, so it will produce a new guitar-backed track rather than a faithful edit of your input. Thus, I am strongly against to the statement that their approach “can naturally extend to music”.\n\nThe regenerated reference is used only as an evaluation proxy to measure edit accuracy. During training, SAO-Instruct receives the input audio by concatenating it to the model’s input channels. During inference, the input audio is additionally encoded into the latent space, with added Gaussian noise, and used as a starting point for the denoising process. The amount of Gaussian noise can be configured and controls how similar the edited audio should be to the provided input audio.\n\nRegarding applicability to music, the same semi-synthetic triplet generation (input audio, edit instruction, output audio) can be applied on music datasets using a music-specific generative model. Editing music using DDPM inversion was already demonstrated in [2]. To avoid overstating the claim, we have revised our statement in the updated manuscript to: “While our work focuses on editing general audio, our approach could be extended to music editing tasks provided appropriate generative music models exist and triplets are constructed from music datasets.”\n\nBased on our feedback and proposed improvements, would the reviewer be willing to reconsider their score?\n\n[1] Jia, Yuhang, et al. \"AudioEditor: A training-free diffusion-based audio editing framework.\" ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.\n\n[2] Manor, Hila, and Tomer Michaeli. \"Zero-shot unsupervised and text-based audio editing using DDPM inversion.\" arXiv preprint arXiv:2402.10009 (2024)."}, {"review_id": "LnmVJFSdpe", "reviewer": "Reviewer_mRyx", "summary": "This paper introduces SAO-Instruct, an audio input + text editing instruction to edited audio output framework. The model is finedtuning the Stable Audio Open (a latent diffusion model model released by Stability AI in July 2024) on a mixture composed of partial synthetic data, fully synthetic data, and real data (manual edits). The performance and latency beats the AudioEditor and ZETA baselines on multiple metrics. Compared to other methods, the proposed approach is particularly great at \"better preserving the characteristics and quality of the input audio\".", "questions": "Question 1: CLAP score is used in many places in the paper, including training data generation, Objective Function, and evaluations. Is there a risk of overfitting a specific CLAP model here? Any mitigations? \n\nQuestion 2: Manual edit - is this really manualy edit or agentic edit? It's not super clear to me how manual edit was implemented. It seems each manual edit is just from a list of predefined function calls with arguments. But how are the arguments determined? Using LLM or by human? \n\nQuestion 3: Does the authors plan to open source the generated data?\n\nQuestion 4: In Table 3, are all the models being compared here trained using completely different data? Clarification would be helpful.", "limitations": "A study of failure cases would be helpful. Also in the demo site, adding a section for failure cases or cases where the proposed model performs worse than baseline models will complete the story.\n\nA discussion regarding the \"safety\" aspects of the proposed model might be helpful. E.g. what can we do to prevent evil or inappropriate use of the model?", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThe paper is very clearly presented and is very easy to follow. I find two parts of the paper particularly interesting and inspiring to the audience: the data generation part and the sample selection/rejection part. The paper used SAO to generate two types of data: partially synthetic data (input audio is real, output audio is SAO) and fully synthetic data (both input and output audio are SAO). For sample selection, authored first use Gemini to perform perceptual quality check on the data, then used CLAP score to select.\n\nThe demos, especially the spectrogram view, are super convincing. Well done!\n\nWeaknesses:\n\n1 - Several metrics were not on-par with the AudioEditor and ZETA baselines. But given the much lower latency, I do not really see this as a big weakness.\n\n2 - Also, all the baselines compared in the paper are diffusion based model. No autoregressive models were compared.\n\n3 - The test sets are generated in the same way as the training sets for the proposed method, which creates a potentially unfair comparison with the baseline models.", "was_revised": true, "final_justification": "My questions have all been resolved. Really strong paper, with huge potentials for real world applications. My final rating is 6 Strong Accept.", "rebuttal": "We thank the reviewer for their valuable insights and address their feedback in the following:\n\n> Question 1: CLAP score is used in many places in the paper, including training data generation, Objective Function, and evaluations. Is there a risk of overfitting a specific CLAP model here? Any mitigations?\n\nWe acknowledge the risk of over‑reliance on a single CLAP model. In our pipeline, CLAP is one metric among several: Prompt-to-Prompt candidate search also uses a perceptual Gemini filter, the Bayesian Optimization objective includes a multi-scale mel-spectrogram distance, and evaluation includes reference-based metrics in addition to a human listening study. In our experiments we used different public CLAP models and checkpoints (e.g., LAION and Microsoft), and while absolute scores were slightly different, the produced samples and rankings were overall stable. While we did not observe overfitting to the chosen CLAP checkpoint, future work could explore aggregating scores across diverse CLAP models and checkpoints to further reduce model-specific bias.\n\n> Question 2: Manual edit - is this really manualy edit or agentic edit? It's not super clear to me how manual edit was implemented. It seems each manual edit is just from a list of predefined function calls with arguments. But how are the arguments determined? Using LLM or by human?\n\nYes, edits are randomly sampled from a predefined set of supported edit operations (e.g., ADD, PITCH, ...). Some of these editing tasks support additional parameters as outlined in Table 5. For example, the PITCH tasks allows to change the amount of pitch shift. Unless stated otherwise, parameter values are sampled uniformly from a fixed range of supported values and the edit is deterministically applied to the audio. The LLM does not choose the operation or its arguments but is responsible for creating a fitting edit instruction for the sampled editing task and its optional parameter. This is done in multiple stages: first, the initial edit instruction is generated, and then optionally rephrased and minimized to make it more natural. In the revised manuscript, we clarified the role of the LLM to avoid confusion.\n\n> Question 3: Does the authors plan to open source the generated data?\n\nWe will open source the model and our full codebase. We are also exploring releasing the generated datasets. For the semi-synthetic DDPM inversion and the fully real manual edits we will first need to verify potential licensing issues as they contain audio from FreeSound and AudioSet-SL. \n\n> Question 4: In Table 3, are all the models being compared here trained using completely different data? Clarification would be helpful.\n\nWe thank the reviewer for raising this point. AudioEditor uses Auffusion [1] as its underlying model, fine-tuned on several general audio datasets including AudioCaps, WavCaps, ESC50, and MACS. The ZETA DDPM inversion approach uses Stable Audio Open as the base model, trained on Freesound and FMA. We have updated Section 5.2 to make these training data differences more explicit.\n\n> A study of failure cases would be helpful. Also in the demo site, adding a section for failure cases or cases where the proposed model performs worse than baseline models will complete the story.\n\nWe thank the reviewer for this suggestion. We will update the demo page with relevant failure cases after the review period ends (NeurIPS rebuttal policy).\n\n> A discussion regarding the \"safety\" aspects of the proposed model might be helpful. E.g. what can we do to prevent evil or inappropriate use of the model?\n\nWe discuss broader impacts in Appendix G, including that we do not scrape data from arbitrary sources and rely on well-established datasets which have been widely adopted in prior research. According with the reviewer’s feedback, our revised manuscript will expand on this section to discuss possible safeguards against inappropriate use of the model. For example, embedding inaudible watermarks in the model’s output to help detect AI-generated content, similar to existing watermarking approaches [2, 3, 4]. Additionally, for our current model checkpoints, we will consider a more restricted release of the model weights under a research license, to further reduce the risk of misuse.\n\n\n[1] Xue, Jinlong, et al. \"Auffusion: Leveraging the power of diffusion and large language models for text-to-audio generation.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024).\n\n[2] San Roman, Robin, et al. \"Latent watermarking of audio generative models.\" ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025.\n\n[3] Wen, Yuxin, et al. \"Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust.\" arXiv preprint arXiv:2305.20030 (2023).\n\n[4] Guo, Yiyang, et al. \"FreqMark: Invisible Image Watermarking via Frequency Based Optimization in Latent Space.\" Advances in Neural Information Processing Systems 37 (2024): 112237-112261."}]

Author rebuttal:

Meta-review: Audio editing is a relatively underexplored area where most focus has been on image editing. Although some of the techniques are derived from image-based generative models, the reviewers still appreciated the effort in translating those to the audio domain. There are positive results compared to sensitive baselines. Most of the questions by reviewers were aimed at the data generation process as the paper also claims to provide a valuable data resource that is generated partially through synthetic data generation. Reviewers were mostly satisfied and provided positive scores overall. The AC reviewed the paper and agrees with the reviewer consensus.
Decision: Accept (poster)