SAO-Instruct: Free-form Audio Editing using Natural Language
  Instructions
Authors: name: Michael Ungersböck, name: Florian Grötschla, name: Luca A. Lanzendörfer, name: June Young Yi, name: Changho Choi, name: Roger Wattenhofer
Organization: DISCO

Generative models have made significant progress in synthesizing
high-fidelity audio from short textual descriptions. However, editing existing
audio using natural language has remained largely underexplored. Current
approaches either require the complete description of the edited audio or are
constrained to predefined edit instructions that lack flexibility. In this
work, we introduce SAO-Instruct, a model based on Stable Audio Open capable of
editing audio clips using any free-form natural language instruction. To train
our model, we create a dataset of audio editing triplets (input audio, edit
instruction, output audio) using Prompt-to-Prompt, DDPM inversion, and a manual
editing pipeline. Although partially trained on synthetic data, our model
generalizes well to real in-the-wild audio clips and unseen edit instructions.
We demonstrate that SAO-Instruct achieves competitive performance on objective
metrics and outperforms other audio editing approaches in a subjective
listening study. To encourage future research, we release our code and model
weights.