AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
Authors: Yuancheng Wang
Zeqian Ju
Xu Tan
Lei He
Zhizheng Wu
Jiang Bian
sheng zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: audio editing
text-to-audio generation
diffusion models
TL;DR: We proposed a latent diffusion model for instruction-guided audio editing.

Abstract: Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, \textbf{AUDIT} has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demopage.github.io/.

AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
Yuancheng Wang 2 Zeqian Ju', Xu Tan', Lei He', Zhizheng Wu', Jiang Bian', Sheng Zhao'
Microsoft, 2The Chinese University of Hong Kong, Shenzhen
{v- (v-yuuancwang, v- v-zeqianju, xuta, helei,, halei,jjabia,, "yanchhngwang@llink cuhk, edu. cn, wwwhizzenn@@cukk.eee
wrosoft.....m.mmmmmmmmmmmooooo.. com
Abstract
Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-brit based methods achieved zero-shot audio editing by using diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-tha art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution. Demo samples are available at /tttt::// //w//tt/uttttt/e demopage github. io/
AUDIT
Adding
T5 Encoder
Abll Lakguer
Dropping
Replocement
VAE Encoder
VAE Decoder
Diffusion
Tapies ming srzurattio duan
Inpainting
Super-Resolution
6- (7(0.)
Figure 1: AUDIT consists of VAE, T5 text encoder, and diffusion network, and accepts the mel-spectrogram the input audio and the edit instructions as conditional inputs and generates the edited audio output.
'Work done as an intern at Microsoft.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Audio editing is crucial step in producing high-quality audio video content, which involves tasks like adding background sound effects, replacing background music, repairing incomplete audio, and enhancing low-sampled audio. Typically, these tasks require professional software and manual operations. However, if audio editing tools can be empowered to follow human instructions, it could significantly reduce the need for manual intervention and benefit content creators. The utilization of natural language instructions such as "add jazz music in the background" could enable more adaptable editing in line with human expectations. In this work, we concentrate on designing audio editing models that can follow human instructions.
Previous works on audio editing usually leverage traditional signal processing technologies. Later, GAN-based methods [33 TO, 1 have achieved success in some audio editing tasks such as audio inpainting and super-resolution. However, these methods are usually designed for specific editing task. Recently, the most advanced deep generative models, diffusion models, have achieved great success in image editing. Some methods [18 27 have attempted apply diffusion models the field of audio editing. These diffusion-based audio editing methods are primarily based on a pre-trained sext-to-audis generation model. However, these methods still suffer from several issues: 1) they are mainly based on a pre-trained model to noise and denoise an input audio conditioned on a target text description, and in most cases, they offer no guarantee to achieve good editing effect since they are not trained directly on audio editing tasks; 2) they can lead to erroneous modifications some audio segments that do not need to be edited, as is difficult to restrict the editing area; 3) they require a complete description of the output audio, which is not always available or necessary in real-world editing scenarios. Instead, it is desirable to provide natural language editing instructions such as "add man whistling in the background" or "drop the sound of the guitar" for audio editing.
To solve the above issues, we propose AUDIT, to the best of our knowledge, the first audio editing model based on human instructions. As shown in Figure I AUDIT is latent diffusion model which takes the audio be edited and the editing instruction as conditions and generates the edited audio as output. The core designs of AUDIT can be summarized in three points: 1) we generate triplet data (instruction, input audio, output audio) for different audio editing tasks, and train an audio editing diffusion model in a supervised manner; 2) we directly use input audio as conditional input to train the model, forcing it to automatically learn to ensure that the audio segments that do not need to be edited remain consistent before and after editing; 3) our method uses editing instructions directly as text guidance, without the need for a complete description of the output audio, making it more flexible and suitable for real-world scenarios. Our editing model achieves state-of-the-art results in objective and subjective metrics.
Our contributions can be summarized as follows:
We propose AUDIT, which demonstrates the first attempt to train latent diffusion model condi- tioned on human text instructions for audio editing.
To train AUDIT in supervised manner, we design novel data construction framework for each editing task; AUDIT can maximize the preservation of audio segments that do not need to be edited; AUDIT only requires simple instructions as text guidance, without the need for a complete description of the editing target.
AUDIT achieves state-of-the-art results in both objective and subjective metries for several audio editing tasks.
2 Related Works
2.1
Audio/Speech Editing
Previous work related to audio editing has primarily focused on human speech or music. 16 5 35 55 explored the task of speech/music inpainting based on the corresponding text or music score. Another category of editing work is focused on style transfer for speech or music. Speech voice conversion
2 35 44 43 and singing voice conversion 7 28 aim to modify the timbre without changing
the speech or singing content. Music style transfer [51 9 30 40 aims to change the instrument being played without modifying the music score. In addition, some methods using GANs [33] I0 have achieved success in some specific audio editing tasks, such as audio inpainting and audio

---

super-resolution. However, there is still lack of research on general audio editing following human instructions.
2.2 Diffusion-based Editing
Diffusion-basee editing tasks have received similar attention as diffusion- based generation tasks. We summarize the common diffusion- based editing work into two types.
14] use diffusion- based generate models to achieve zero-shot editing. SDEdit [34] uses a pre-trained diffusion model add noise an input image and then denoise the image with new target text. These methods have difficulty in editing specific regions and have poor controllability. To edit specific regions, [4] uses an additional loss gradient to guide the sampling process, and 4 3 replace unmodified parts with the noisy input image in each sampling step, equivalent to only generating in the masked part. However, these methods are only for inpainting tasks and also need complete description of the editing target.
Zero-Shot Editing Some methods 34
37 handle editing tasks in a supervised manner. 47 train an image-to-image diffusion model which takes the image be edited
Supervised Training
Another type of methods [47 6 54
condition. uses paired image data generated by 14] to train a text-based image editing model. ControlNet 54 and T21-Adapter [37] add an adapter model
frozen pre-trained generation diffusion model and train the adapter model to achieve image editing. Our method follows the supervised training paradigm, using the generated triplet data (instruction, input audio, output audio) to train an audio editing model conditioned on the instruction and the audio to be edited, and can handle different audio editing tasks by following human instructions.
2.3 Text-Guided Audio Generation
Currently, most diffusion-based audio editing methods 18 27 are based on text-to-audio diffusion generative models. Text-to-audio generation aims to synthesize general audio that matches the text description. Diff'Sound [52] is the first attempt to build text-to-audio system based on discrete diffusion model [13. AudioGen (26) another text-to-audio generation model based on a Transformer-bas decoder that uses an autoregressive structure. AudioGen directly predicts discrete tokens obtained by compressing from the waveform (3)] Recently, Make-an-Audin [88 and
AudioLDM [27] have attempted to build text-guided audio generation systems in continuous space, which use an autoencoder to convert mel-speetrograms into latent representation, build a diffusion model that works on the latent space, and reconstruct the predicted results of the latent diffusion model as mel-spectrograms using the autoencoder. Furthermore, MusicLM (3 and Muise2Music [17] generate music that matches the semantic information in the input text. In our work, we focus on editing the existing audio with human instructions.
Method
In this section, we present our method for implementing audio editing based on human instructions. We provide an overview of our system architecture and training objectives in Section 3.1 and show the details about how we generate triplet data (instruction, input audio, output audio) for each editing Lastly, we discuss the advantages of our method in Section 3.3
task in Section 3.2
3.1 System Overview
Our system consists of an autoencoder that projects the input mel-spectrograms an efficient, low-dimensional latent space and reconstructs it back to the mel-spectrograms, a text encoder that encodes the input text instructions, diffusion network for editing in the latent space, and vocoder for reconstructing waveforms. The overview of the system is shown in Figure
The autoencoder contains an encoder and a decoder G. The encoder E transforms the mel-spectrogram into latent representation and the decoder G reconstructs å from the latent space. our work, we employ a variational autoencoder (VAE) 23 model as the autoencoder. We train our autoencoder with the following loss: 1) The L1 and Le reconstruction loss. 2) The
Autoencoder
3

---

Zin
VAE Encoder
xT
Latent Diffusion
in
Zt-1
2
Encoder
Encoder
Za
Ze
Vocoder
"Drop the sound of duck quacking"
ont
Figure 2: A high-level overview of our system.
Kullback- loss Cx to regularize the latent representation z. We only assign small weight AKL to the KI. loss following 45 3) The GAN loss LGAN. We employ patch-based discriminator and reconstructed mel-spectrograms, The total training loss of the autoencoder can be expressed as AV = X1 + X2 + AKL + AGAN X1, X2. AKL. and AGAN are the weights L1, L2, LKL, and LGAN respectively.
D 19 45) to distinguish between
Text Encoder We pre-trained language model T5 [2]] text encoder, which is used to convert text input into embeddings that contain rich semantic information. The parameters of the text encoder frozen in the training stage.
Latent Diffusion Model Our text-guided audio editing model can be seen as conditional latent diffusion model that takes the latent representation Zin of the input (to be edited) mel-spectrogram Zin and text embeddings as conditions. The model aims to learn ((Zout|Zin- Rin, Ctent). where zout is the latent representation the output (edited) mel-spectrogram Fout, of the editing instruction. Given the latent representation Zin and the text instruction, select time step and noise e generate noisy version 2, of the latent representation Zout by using noise schedule and then use the diffusion network co (2 f, Zin Creat) to predict the sampled E E0 Zins Ctezt) -E|/2-
Ctezt is the embedding randomly
noise. The training loss LLDM = E (sun.Bous,,ee Ne Ex
we use U-Net 466 with the cross- attention mechanism [50] as the diffusion network. Our editing model takes Zin condition by directly concatenating 2, and zin at the channel level. Therefore, the input channel number of the first layer of the U-Net twice the output channel number of the last layer.
Similarly to the original standard diffusion model 15
49
Vocoder A vocoder model is required to convert the output mel-spectrogram into audio. In this work, we use HiFi-GAN 24 as the vocoder, which one of the most widely used vocoders in field of speech synthesis. HiFi-GAN uses multiple small uub-iiscrimintooss to handle different cycle patterns. Compared with some autoregressive [38] 20 or flow-based [42 vocoders, takes into account both generation quality and inference efficiency.
3.2 Generating Triplet Training Data for Each Editing Task
We use generated triplet data (instruction, input audio, output audio) to train our text-guided audio editing model. In this work, focus five different editing tasks, including adding, dropping, replacement, inpainting, and super-resolution. Note that we train all the editing tasks in a single editing model. Figure provides an overview of how our data generation workflow for different editing tasks works.
Adding We randomly select two audio clips and from the text-audio datasets, then combine A and B to get new audio clip C. We use A as the input audio and C' as the output audio, and we fill in the caption (or label) B into the instruction template to get the instruction. For example, shown in Figure the instruction template is "Add {1 in the background" and the caption of B

---

Adding
Dropping
Replacement
Inpainting
Super-resolution
Template: 'Add in the background'
Template: "Drop (I"
Template: "Raplace with
Template: "Inpaint"
Tamplate: "Incress resolution"
car engine
Belling.
accolerating.
The sound of guitar.
input audio I
audio
output audio
output
output audio
meewing.
A bird singing.
randamiy meuk
dosn somple
1 M
-
outpur
input
imput sudio
Baby crying.
Dog barking.
Somoone clapping.
input audio
Input audio
Instruction: "Add baby crying the background"
Instruction: "Drop dog barking
Instruction: "Replace clapping with guitar"
Instruction: "Inpaint"
Instruction: "Increase resolution'
Figure 3:
Examples about how generate triplet training data for different audio editing tasks (adding, dropping, replacement, inpainting, and super-resolution)
is "Baby crying", so the instruction is "Add baby crying in the background". Note that we design multiple templates for different editing tasks.
Dropping We randomly select two audio clips A and B from the text-audio datasets, then combine A and B to get new audio clip C. We use as the input audio and A as the output audio, and we fill in the caption (or label) of B into the instruction template to get the instruction. For example, the instruction template is "Drop ()" and the caption of is "Dog barking", so the instruction is "Drop Dog barking".
For the replacement task, select three audio clips A, B, and C' from the datasets, then insert and C separately into A (in roughly the same area) to obtain two new audio clips D and E. We use D as the input audio and E the output audio, and we fill in the captions (or labels) of B and C' into the instruction template to get the instruction. For example, the instruction template is "Replace () with ()", the caption of B is "Someone clapping", ", and the caption of C is "The sound of guitar", so the instruction is "Replace clapping with guitar".
Replacement
For the inpainting tasks, we select an audio clip as the output audio and randomly mask some parts of the audio to get new audio clip as the input audio. We can use instructions like "Inpaint" or "Inpaint the audio" directly use the instructions iike"Inpintt cat meowing". For the super-resolution tasks, we use down-sampled audio as the input audio. The instructions are as "Increase resolution" " or "Increase resolution: bird singing".
Inpainting and Super-Resolution
Extending Instructions with ChatGPT We design multiple instruction templates for each editing task. In order to further expand the diversity of editing instructions, we use ChatGPT [39 to generate more editing instruction templates for us. See more details in Appendix
3.3 Advantages of AUDIT
We analyze the advantages of our methods over previous works on three key points. 1) We generated triplet data (instruction, input data, output data) to train text-guided audio editing model instead of performing zero-shot editing, which can ensure good edit quality. 2) We directly use the input audio as a condition for supervised training of our diffusion model, allowing the model automatically learn to preserve the parts that do not need to be edited before and after editing. Specifically, we concatenate the latent representations zin. of the input audio and z, on the channel dimension and input them into the latent diffusion model, so that the model can "see" Zin. (rather than its noisy version) during both training and inference. 3) Instead of using a full description of the output audio, we use human instructions the text input, which is more in line with real-world applications.
5

---

Experimental Settings
4.1 Dataset
The datasets used in our work consist of AudioCaps 222. AudioSet [12), FSD50K M., and ESC50 41] AudioSet is the largest audio-label pairs dataset; however, each audio clip in AudioSet only has a corresponding label and no description. Because of the presence of large number of human speech and some audio clips that are only weakly related to their labels in AudioSet, of AudioSet (AudioSet96) that includes approximately 149K pairs with 96 classes. AudioCaps is a dataset of around 44K semantic information. The length of each audio clip in AudioSet and AudioCaps is around 10 seconds. FSD50K includes around 40K audio clips and 200 audio-label pairs of variable lengths ranging from 0.3 to 30 seconds. We split FSD50K training set into two datasets, FSD5OK-L LL (19K) and FSD50K-S (22K), comprising audio clips of length less than and greater than seconds, respectively. ESC50 is smaller dataset with 50 classes, each containing four audio clips of around 5 seconds, for total of 2000 audio clips.
subset
pairs, where each audio clip corresponds to caption with rich
We use AudioCaps, AudioSet96, FSD50K-S, and ESC50 to generate triplet training data for five audio editing tasks. We total of about 0.6M triplet data to train our audio editing model. More details about datasets and data processing are shown in Appendix A
4.2 Baseline Systems
Text-to-Audio Generation Since we compare with generative model-based audio editing baseline methods, use AudioCaps, AudioSet96, FSD50K, and ESC50 train a text-to-audio latent diffusion model and use the test set of AudioCaps to evaluate the text-to-audio model. To demonstrate the performance of our generative model, compare with some state-of-the-a text-to-audio
generative models 52 26 27 18.
Adding, Dropping, and Replacement We use methods like SDEdit 34 as baselines. SDEdit uses pre-trained text-to-image diffusion model to noise and then denoise an input image with the editing target description. In work, we directly use SDEdit in the adding task, dropping task, and replacement task. We use our own trained text-to-audio latent diffusion model. We test different total denoising steps N for N 3/4T, , N 1/2T,, and N 1/4T, where T is the total step in the forward diffusion process. The three baselines are called: 1) SDEdit-3/4T; 2) SDEdit- 3) SDEdit-I/4T.
Inpainting For the inpainting task, we designed four baselines derived from SDEdit but with some differences between each other. 1) SDEdit. We directly use SDEdit, we first add noise to the input audio, then denoise the audio with the description (caption or label) of the output audio as text guidance. 2) SDEdit-Rough and 3) SDEdit-Precii masked) needs to be edited in the inpainting task, we call the part that does not need to be edited the "observable" part, and we call the masked part the "unobservable" part. In each step of the denoising process, we can replace the "observable" part with the ground truth in the latent space. The difference between SDEdit- Rough and SiDEdit-Precise is that in SDEdit-Rough the "unobservable" part is a rough region, while in SDEdit-Preciss the "unobservable" part is a precise region. 4) SDodit-wo-Tex. EDEdit-wo-Text is similar to SDEdit-Precise, however, it has no text input. An example of the difference of the "unobservable" part between SDEdit- Rough and SDEdit-Preche is shown in Appendix [F
Only a part of the input audio (the part that is
We also compared methods with two task- specifie methods GACELA [32) and DAS [T). GACELA is a generative adversarial network for restoring missing musical audio data. The publicly available checkpoint trained solely music data; hence, trained the model using our own dataset. DAS is another GAN-based audio editing model, we also trained it on our dataset.
Super-Resolution Super-resolution can be seen as inpainting in the frequency domain. We use three baselines similar to the inpainting task: 1) SDEdir, 2) SDEdit-Precise; 3) SDEdit-wo-Texr. We also conducted a comparison with the NU-Wave2 (51) model, which is a task-specific model designed for super-resolution. We utilized the provided open-source checkpoint for NU-Wave2; however, note that NU-Wave2 was originally trained on speech data. To adapt to our specific task, we trained on own dataset.

---

4.3 Evaluation Metries
Objective Metrics We use log spectral distance (LSD), frechet distance (FD), and kullback-leibler divergence (KL) to evaluate our text-guided audio editing model and use inception score (IS), FD, and KI. evaluate our text-to-audio generation model following [22 26 27 18). LSD measures the distance between frequency spectrograms of output samples and target samples. FD measures the fidelity between generated samples and target samples. measures the quality and diversity of generated samples. KL measures the correlation between output samples and target samples. FD, IS, and KL are based on the state-of-the-an audio classification model PANNs [25]. We use the evaluation pipeline Prpelineeprovieee provided by 27 for objective evaluation for fairer comparisons.
Subjective Metrics We use overall quality (Quality) to measure the sound quality and naturalness of the output audio compared the input audio and use relevance to the editing instruction (Relevance) to measure how well the output audio matches the input human instruction. Each sample will be scored from 100 based on Quality and Relevance. See more details about the evaluation in Appendix D
4.4 Model Configurations
Our autoencoder compresses a mel-spectrograa size x x W into a latent representation of size 4 x H W Our models are trained on 8 NVIDIA V100 GPUs for 500K steps with batch
x 4
size of 2 on each device. We use the weights of our pre-trained text-to-audio model initialize our audio editing model. Our HiFi-GAN vocoder is trained on AudioSet96 and AudioCaps datasets using 8 NVIDIA V100 GPU for 200 epochs. More details about model configurations are shown in Appendix B
5
Results
5.1
Objective Evaluation
Adding Table ssoow the objective evaluation results of the adding task. Our method achieves the best performance, with LSD of 1.35, KL of 0.92, and FD of 21.80. Compared with the best baseline method, our method reduces FD by 6.45 and KI. by 0.38.
Table Objective evaluation results of the adding task.
Model
Text
LSD(L)
KL(4)
FD(I)
SDEdit-3/4T
caption
1.54
1.68
28.87
SDEdit-1/2T
caption
1.43
1.38
28.75
SDEdit-1/47
caption
1.38
1.30
28.25
AUDIT
instruction
1.35
0.92
21.80
Dropping 2 the objective evaluation results of the dropping task. Our method achieves the best performance, with LSD of 1.37, KL of 0.95, and FD of 22.40. Compared with the best baseline method, our method reduces FD by 5.79 and KL. by 0.10.
Table 2: Objective evaluation results of the dropping task.
Model
Text
LSD())
KL(4)
FD(L)
SDEdit-3/4T SDEdit- 1/2T
caption
1.54
1.14
29.66
caption
1.43
1.05
28.19
SDEdit- 1/4T
caption
1.40
1.30
31.31
AUDIT
instruction
1.37
0.95
22.40

---

Replacement shows the objective evaluation results of the replacement task. Our method achieves the best performance, with LSD of 1.37, KL 0.84, and FD of 21.65. Compared with the best baseline method, our method reduces FD by 5.07 and KI. by 0.31.
Table 3: Objective evaluation results of the replacement task.
Model
Text
LSD(L)
KL(4)
FD(J)
SDEdit-3/4T 3DEdit-1/2T
caption
1.63
1.58
28.78
caption
1.52
1.27
27.71
SDEdit- 1/4T
caption
1.46
1.15
26.72
AUDIT
instruction
1.37
0.84
21.65
Inpainting Table 4 shows the objective evaluation results the inpainting task. Our method achieves the best performance, with LSD of 1.32, KL of 0.75, and FD of 18.17. We find that for baseline methods, not providing a text input (the caption the audio) as guidance leads to a large performance drop, SDEdit-wo-Tee Text gets the worst performance in terms of KL and FD among the baseline methods. However, AUDIT-wo- Text which only uses instructions like "inpaint" achieves performance close to AUDIT which uses instructions like "inpaint caption". We also discovered that AUDIT's performance on inpainting tasks can surpass that of task-specific baseline methods. One possible explanation for this that during training, GACELA and DAS do not have access any textual information, whereas our model requires explicit instructions to edit the audio. As result, model is better able learn the semantic information of the audio, which likely contributes its improved performance in inpainting tasks.
Table 4: Objective evaluation results of the inpainting task.
Model
Text
LSD())
KL(
FD(()
SDEdit
caption caption
2.91
1.47
25.42
SDEdit- SDEdit-Precise SDEdit Text GACELA DAS
1.64
0.98
21.99
caption
1.54
0.94
21.07
1.55
1.63
27.63
1,41
0.78
20.49
1.57
0.89
21.97
AUDIT-wo- Text AUDIT
instruction
1.37
0.81
19.03
instruction caption
1.32
0.75
18.17
shows the objective evaluation results of the super-resolution task. Our method achieves the best performance, with LSD of 1.48, KL of 0.73, and FD of 1 14. Similar gets the worst performance in terms of KL and FD among
Super-Resolution Table shows
to the inpainting task, SDEdit-wo-Dii the baseline methods. Our method can achieve significantly better results than baselines using only simple instructions like "Increase resolution" ", which shows that our method can learn sufficient semantic information from oow-frequeccy information. The results also indicate that our AUDIT model outperforms the NU-Wave2 model trained on speech and is comparable to or even better than the NU-Wave2 model trained our dataset.
Text-to-Audio Generation We also present the comparison results of our text-to-audio latent diffusion model with other text-to-audio models in Table Our model achieves the best performance, with FD of 20.19, KI 1.32, and IS of 9.23, outperforming AudioLDM--LLe with FD of 23.31, 1, KI. of 1.59, and IS of 13. This demonstrates that our generation model can serve strong baseline model for generation- based editing methods.
5.2 Subjective Evaluation
The results of the subjective evaluation are shown in Table We choose the best results the baseline to report. Our method clearly outperforms the baseline methods in both Quality and Relevance across

---

Table 5: Objective evaluation results of the super-resolution task.
Model
Text
LSD())
KI(J)
FD(J)
SDEdit SDEdit-Precise
caption caption
3.14
1.50
25.31
1.75
1.17
27.81
SDEdit-wo-Text
1.78
1.56
32.66
NU-Wave2 (origin)
1.66
0.89
22.61
NU-Wave?
1.42
1.78
19.57
AUDIT-wo- Text AUDIT
instruction
1.53
0.92
21.97
instruction + caption
1.48
0.73
18.14
Table 6: The comparison between our text-to-audio generative model and other baseline models on the AudioCaps test set.
Model
Dataset
FD())
IS(1)
KL(L)
DiftSound
AudioSetAAudioCaps
47.68
4.01
2.52
AudioGen
26
AudioSet+AudioCaps
2.09
Make-an- Aueeion-Auioo
AudioSet-AudioCapp--.. others
2.79
AudioLDM-L 27
AudioCaps
27.12
7.51
1.86
AudioLDM-L--ull 27
AudioSet+AAudioCaps-22 others
23.31
8.13
1.59
AUDIT
AudioSet96--AudicCap== others
20.19
9.23
1.32
all five tasks. the dropping task, our method achieves the highest scores with Quality of 78.1 and Relevance of 81.0. We find that compared with the baseline methods, our method improves more significantly on the adding, dropping, and replacement tasks. The possible reason is that compared with the inpainting and super-resolution tasks, which have explicit positioning of the editing area area), the adding, dropping, and replacement tasks need to
(the masked part and the first locate the region to be edited, and also need to ensure that other regions cannot be modified, which is more difficult for the baseline method. Our model has learned this ability through supervised learning.
Table 7: Subjective evaluation. For each audio editing task, we report the results of the best baseline method and the results of our method.
Method
Task
Quality(t)
Relevance(1)
Baseline
60.2
56.7
Adding
AUDIT
75.5
77.3
Baseline AUDIT
54.4
48.2
Dropping
78.1
81.0
Baseline
57.7
47.6
Replacement
AUDIT
72.5
74.5
Baseline AUDIT
65.8
66.3
Inpainting
75.2
78.7
Baseline
61.8
59.9
Super-Resolution
AUDIT
74.6
76.3
5.3 Case Study
In order to more specifically show the difference between the performance of our model and the baseline methods, we also give some studies in Appendix G

---

Conclusions
In this work, we propose an audio editing model called AUDIT, which can perform different editing tasks (e.g.. adding, dropping. replacement, inpainting, super-resolution) based on human text instructions. Specifically, we train a text-guided latent diffusion model using our generated triplet training data (instruction, input audio, output audio), which only requires simple human instructions as guidance without the need for the description of the output audio and performs audio editing accurately without modifying audio segments that do not need to be edited. AUDIT achieves state-of-the-ar performance on both objective and subjective metries for five different audio editing tasks. For future work, we will explore more editing audio tasks with our framework, and achieve more precise control for audio editing. We also discuss some limitations and broader impacts of our work in Appendix H
10

---

References
[1] Sherif Abdulatif, Ruizhe Cao, and Bin Yang. Cmgan: Conformer- metric-gan for monaural speech enhancement. arXiv preprint arXiv:2209.11112, 2022.
[2] Masanobu Abe, Satoshi Nakamura, Kiyohiro Shikano, and Hisao Kuwabara. Voice conversion through vector quantization. Journal of the Acoustical Society of Japan (E), 11(2):11-66, 1990.
[3] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et Musiclm: Generating music from text. arXiv preprint arXiv:2301. 1325, 2023.
[4] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208-18215 18218, 2022.
[5] Zalán Borsos, Matt Sharifi, and Marco Tagliasacchi. Speechpainter: Text-conditioned speech inpainting. arXiv preprint arXiv:2202.07273, 2022.
[6] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint S.Ii:222110098000 2022.
[7] Xin Chen, Wei Chu, Jinxi Guo, and Ning Xu. Singing voice conversion with non -or-palllll data. In 2019 IEEE Conference Multimedia Information Processing and Retrieval (MIPR), pages 292-296. IEEE, 2019.
[8] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilv:: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02936 2021.
[9] Ondiej Cifka, Umut Şimşekli, and Gaël Richard. Groove2groove: one-shot music style transfer with supervision from synthetic data. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 11-2638-26500 2020.
[10] Pirmin P Ebner and Amr Eltelt. Audio inpainting with generative adversarial network. arXiv preprint arXiv:2003.07704 2020.
[11] Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829-000 852, 2021.
[12] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, IEEE, 2017.
[13] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706. 2022.
[14] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or trompttto-prompt image editing with cross attention control. arXiv preprint 0211v:2208.01626. 2022.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 6851, 2020.
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
[17] Qingqing Huang, Daniel S Park, Tao Wang, Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, al. moise2musie: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.0391 2023.
11

---

[18] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren. Luping Liu, Mingze Li. Zhenhui Ye. Jinglin Liu. Xiang Yin, and Zhou Zhao. lake-an-audivn Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301. 12661, 2023.
[19] Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1125-1 11341134, 2017.
[20] Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pages 2410-2419. PMLR, 2018.
[21] Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005. 10433, 2020.
[22] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 119-132, 2019.
[23] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 131266 14, 2013.
[24] Jungil Kong. Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 53:17022-17033 2020.
[25] Qiugiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.
[26] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv: 15352, 2022.
[27] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley, Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. /2503, 2023.
[28] Songxiang Liu, Yuewen Cao, Dan Su, and Helen Meng. Diffsve: A diffusion probabilistic model for singing voice conversion. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 741-748. IEEE, 2021.
[29] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05100 2017.
[30] Chien Yu Lu, Min-Xin Xue, Chia-Che Chang, Che-Rung Lee, and Li Su. Play as you like: Timbre-enhanced multi-modal music style transfer. In Proceedings of the uaai conference on artificial intelligence, volume 33, pages 1061-1068, 2019.
[31] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pages 11461-11471, 2022.
[32] Andrés Marafioti, Piotr Majdak, Nicki Holighaus, and Nathanaël Perraudin. A generative adversarial context encoder for long audio inpainting. 2020.
[33] Andrés Marafioti, Piotr Majdak, Nicki Holighaus, and Nathanaël Perraudin. Gacela: A genera- tive adversarial context encoder for long audio inpainting of music. IEEE Journal of Selected Topics in Signal Processing, 15(1):120-131 2021.
12

---

[34] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
[35] Seyed Hamidreza Mohammadi and Alexander Kain. An overview of voice conversion systems. Speech Communication, 88:65-82, 2017.
[36] Ondřej Mokrý and Pavel Rajmic. Audio inpainting: Revisited and reweighted. IEEE/ACM Trunsuctions on Audio, Speech, and Language Processing, 28:2906-2918, 2020.
[37] Chong Mou, Xintao Wang, Liangin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. arXiv preprint 030.v22302.08453, 2023.
[38] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint (0X9vv660900344 2016.
[39] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[40] Marco Pasini. Melgan-ve: Voice conversion and audio style transfer on arbitrarily long samples using spectrograms. arXiv preprint 1vXiv:1910.03713, 2019.
[41] Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference Multimedia, pages 1015-1018, 2015.
[42] Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. In International Conference on Machine Learning, pages 7706-7716, PMLR, 2020.
[43] Adam Polyak, Lior Wolf, Yossi Adi, and Yaniv Taigman. Unsupervised cross-domain singing voice conversion. arXiv preprint arXiv:2008 02830, 2020.
[44] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. arXiv preprint WvXiv:2109.13821 2021.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister InterventionMIICA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[47] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-imal diffusion models. ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[48] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.
[49] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:20/1. 13456, 2020.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
13

---

[51] Shih-Lun Wu and Yi-Hsuan Yang. Musemorphose: Full-song and fin-grained music style transfer with one transformer vae. arXiv preprint Stt::2105.040990 2021.
[52] Donghhao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.09983 2022.
[53] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507, 2021.
[54] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[55] Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang. Vision-infused deep audio inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 283-292, 2019.
14

---

A Datasets and Data Processing
Table presents some details about the datasets we used and generated. All audio data have a sampling rate of 16KHz. For AudioSet, AudioCaps, and AudioSet96, we pad or truncate all audio to 10s. For FSD50K, we pad audio shorter than 5s to 5s, and truncate or pad audio longer than 5s to 10s. All generated audio data have length of 10 seconds. We convert 10s of audio into -ee-sppetrsgraass with a size of 80 x 624, using a hop size 256, a window size 1024, and mel-bins size 80, covering frequencies from 0 to 8000. The autoencoder compresses the mel-spectrograms to latent representation of size x 10 x 78.
Table 8: Details about audio-text datasets we use and our generated audio editing datasets
Dataset
Hours
Number
Text
AudioSet
5800
2M
label
A
414
149K
label
AudioCaps
122
44K
caption
FSD50K
108
51K
label
FSD50K-S
31
22K
label
FSD50K-I.
53
19K
label
ESC50
3
2K
label
Task
Datasets
Number
Text
Generation
AudioCaps, AudioSet96, FSDS0K, ESC50
243K
label or caption
Adding
AudioCaps, AudioSer96, SSDSOK-SSESCCE
71K
Instruction
Dropping
AudioCaps, AudioSer96, FSDSOK-S,ESCO0
71K
Instruction
Replacement
AudioSer96,, FSD5OK-S, ESC50
50K
Instruction
Inpainting
AudioSet96, AudioCaps AudioSet96, AudioCaps
193K
Instruction
Super-resolution
193K
Instruction
B
Model Details
Table | shows more details about our audio editing and audio generative models. We train our autoencoder model with batch size of 32 (8 per device) on 8 NVIDIA V100 GPUs for total of 50000 steps with a learning rate of 7.5e 5. For both audio editing and U-Net audio generative diffusion, we train with batch size of on 8 NVIDIA V100 GPUs for total of 500000 steps with learning rate of 5e - 5. Both the autoencoder and diffusion models use Adam W[29] as the optimizer with (8,, (2)) (0.9, 0.999) and weight decay of le We follow the official repository to train the HiFi-GAN vocoder.
C Classifier-free Guidance
16 proposed using classifier-fiet frre guidance to trade off diversity and sample quality. The classifier- free guidance strategy has been widely used in conditional diffusion models. Our model has two additional conditions, Ctext and Zin- However, during training, we only consider helping the model learn the marginal distribution of p(z, ((z,(zin) am) (without explicitly learning ((2e]Ctext)) Therefore, during training, mask the text with certain probability (replacing it with empty text () to learn the no-text condition score (z I, zm.). Then, according to Bayes' formula ((ex)))e xx p(ss|ereatuiin p(zia #t."teztt)'
can derive the score relationship in Equation | which corresponds to the classifier- free guidance Equation 2 Here, the parameter is the guidance coefficient used to balance the diversity and quality of the samples.
Vz, log p(zin Iz, Ctezt) = Ve, ((eeeeeen Zin) Vz, log.((z(/zz))
(I)
ēø(ze, Zin, Crewt) c0(2,, Zin. 0) (Ee( t, Zin, Crewe) c0( 0))
(2)
15

---

Table 9: Details about our audio editing and audio generative models
Model
Configuration
Number of Parameters
83M
In/Out Channels
Autoencoder
Latent Channels
4
Number of Down/Up Blocks
4
Block Out Channels
(128, 256, 512. 512)
Activate Function
SiLU
Number of Parameters
109M
T5 Text Encoder
Output Channels
768
Max Length
300
Number of Parameters
859M
In Channels
8
Out Channels
4
Editing Diffusion U-Net
Number of Down/Up Blocks
4
Block Out Channels
(320, 640, 1280, 1280)
Attention Heads
8
Cross Attention Dimension
768
Activate Function
SiLU
Number of Parameters
859M
In Channels
Out Channels
4
Generative Diffusion U-Net
Number of Down/Up Blocks
Block Out Channels
(320, 640, 1280, 1280)
Attention Heads
Cross Attention Dimension
768
Activate Function
SiLU
Sampling Rate
16000
Number of Mels
80
HiFi-GAN
Hop Size
256
Window Size
1024
D Human Evaluation
For each audio editing task, our human evaluation set comprises ten samples randomly selected from the set. Ten raters score each sample according to metrics, Quality and Relevance, using scale of 1 to 100.
E Extending Instructions with ChatGPT
Some examples of instruction templates designed by ourselves:
"Add {} in the beginning" "
"Add 0 at the beginning"
"Add in the end"
"Add O in the middle" "
"Add O in the background"
"Drop (}"
"Remove ()"
"Replace () to ()"
"Replace 0 with 0
16

---

"Inpaint"
"Inpainting"
"Inpaint O
"Inpaint: {
"Increase resolution"
"Increase resolution: 0
Perform super-resolution" "
PPerform super-resolution: O
We ChatGPT to extend editing templates, specifically, we submit the editing instruction templates we designed to ChatGPT and let it generate more instruction templates with the same semantic information. Below some examples.
"Mix I into the background"
"Blend { with existing audio"
"Incorporate (
new element the end of the audio" "
"Place O in the foreground"
"Erase { from the track"
"Subtract {} from the audio"
"Take out { from the foreground"
"Exchange O} for O in the mix"
"Use O to replace O in the audio" "Interchange () and O in the track"
"Replace missing audio with synthesized sound"
"Fill in the gaps in track 0
"Upscale audio higher resolution"
"Apply super-resolution the audio improve clarity"
Baseline Methods
For the inpainting task, only part of the input audio (the part that is masked) needs to be edited, call the part that does not need be edited the "observable" part, and call the masked part the "unobservable" part. In each step of the denoising process, we can replace the "observable" part with the ground truth in the latent space. The difference between SDEdit-Rouge and Preccie is that in Rough, the "unobservable" part is rough region, while in EDEdit-Precisse the "unobservable" part precise region. Figure gives an example.
The "unobservable" part SDEdit-Precise.
The "unobservable" part SDEdit-Rough.
Figure 4: The difference between SDEdit-Rough and SDEdit-Preciss
17

---

G
Case Study
We show case studies. Figure shows case for the adding task. The caption of the input audio is "The sound of machine gun", " and the editing target is adding a bell ringing in the beginning. AUDIT performs audio editing accurately in the correct region without modifying audio segments that do not need to be edited. Figure[Jshore bird whistles continuously, while duck quacking in water in the background", and the editing target is dropping the sound of the duck quacking. Our method successfully removes the background sound and preserves the sound the bird whistling. but the sound of the bird whistling in the SDEdit editing result is incorrectly modified. It shows that method can better ensure that the audio segments that do not need to be edited are not modified. Figure shows a case for the inpainting task. The caption of the input audio is "A person is typing computer", While both AUDIT and the baseline method generate semantically correct results, the result generated by AUDIT more natural and contextual.
case for the dropping task. The caption of the input audio is "A
Input
Generate by AUDIT
Generate by SDEdit
Figure 5: A case for the adding task. The caption of the input audio is "The sound of machine gun", and the editing target adding bell ringing in the beginning.
Input
Generate by AUDIT
Generate SDEdit
Figure 6: A case for the dropping task. The caption of the input audio is "A bird whistles continuously. while duck quacking in water in the background", and the editing target is dropping the sound of the duck quacking.
Input
Generate by AUDIT
Generate by ADEdit-Precise
Figure 7: A case for the inpainting task. the caption of the input audio is "A person is typing computer".
H Limitations and Broader Impacts
Our work still has limitations. For example, the sampling efficiency is low since model is diffusion-barr model. In the future, we will try to improve the generation efficiency of our model using efficient strategies like consistency models [48] In addition, will also explore the of more data and more diverse editing instructions to achieve more kinds of editing tasks. AUDIT can edit existing audio based on natural language instructions, which may be used inappropriately, such as synthesizing fake audio for fraud. Therefore, we urge everyone not to abuse this technology and develop synthetic audio detection tools.
18


Reviews:
[{"review_id": "RHFBy5AY39", "reviewer": "Reviewer_XrSg", "summary": "This paper proposes AUDIT, an instruction-guided audio editing model based on latent diffusion models. The model is trained with triplet training data (instruction, input audio, output audio) for\ndifferent audio editing tasks and is based on a diffusion model using instruction and input\n(to be edited) audio as conditions and generating output (edited) audio. The proposed model outperforms several baselines in the editing tasks", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The AUDIT model is able to perform audio editing conditioned on text instructions, which have rarely been studied by prior work on audio generation. The data construction strategy of using paired text-audio data and refining text instructions based on templates and LLM is also useful for making instruction datasets. In terms of performance, it is better than several baselines (SDEdit and its variants).", "weaknesses": "The main weakness lies in the evaluation. The model is mostly compared to SDEdit and its variants, which is zero-shot audio editing model. It is unclear how it compares to task-specific models such as NU-Wave2 [1] for audio super-resolution. Some of the editing tasks considered in the paper (e.g., Adding) can also be addressed via text-to-audio generation, while proposed method has not been compared to the methods of this category. There also lacks analysis on the gain of the proposed method over SDEdit and its variants. For example, how robust the model is to the category of the dropped sound in Dropping task? As both the training and evaluation data are synthesized, it is unclear how the model will perform on real data.\n\n[1]. Han et al., NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments! We will explain your concerns point by point.\n\n>**1. It is unclear how it compares to task-specific models such as NU-Wave2 [1] for audio super-resolution.**\n\nThanks for your valuable suggestion! Since our approach is based on a diffusion model, our primary comparisons are conducted with zero-shot editing methods that also rely on diffusion models. We have incorporated comparative experiments with task-specific models, focusing on the tasks of super-resolution and inpainting.\n\nSuper-resolution:\n\nFor the super-resolution task, we conducted a comparison with the NU-Wave2 [1] model, which is a task-specific model designed for super-resolution. We utilized the provided open-source checkpoint for NU-Wave2; however, note that NU-Wave2 was originally trained on speech data. To adapt it to our specific task, we trained it on our own dataset.\n\n| Mehtod | LSD($\\downarrow$) | KL($\\downarrow$) |FD($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n|SDEdit (best)|1.75|1.17|27.81|\n| AUDIT | 1.48| **0.73**| **18.14**|\n| NU-Wave2 (origin) | 1.66| 0.89| 22.61|\n| NU-Wave2 (trained with our data)| **1.42**| 0.78| 19.57|\n| | | | |\n\nThe results indicate that our AUDIT model outperforms the NU-Wave2 model trained on speech and is comparable to or even better than the NU-Wave2 model trained on our dataset.\n\nInpainting:\n\nFor the audio inpainting task, we conducted comparative experiments with GACELA [2] and DAS [3]. GACELA is a generative adversarial network for restoring missing musical audio data. The publicly available checkpoint was trained solely on music data; hence, we trained the model using our own dataset. DAS is another GAN-based audio editing model, we also trained it on our dataset.\n\n| Mehtod | LSD($\\downarrow$) | KL($\\downarrow$) |FD($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n|SDEdit (best)|1.54|0.94|21.07|\n| AUDIT | **1.32**| **0.75**| **18.17**|\n| GACELA (trained with our data) | 1.41| 0.78| 20.49|\n| DAS (trained with our data) | 1.57 | 0.89| 21.97|\n| | | | |\n\nWe discovered that AUDIT's performance on inpainting tasks can surpass that of task-specific baseline methods. One possible explanation for this is that during training, GACELA and DAS do not have access to any textual information, whereas our model requires explicit instructions to edit the audio. As a result, our model is better able to learn the semantic information of the audio, which likely contributes to its improved performance in inpainting tasks.\n\n> **2. Some of the editing tasks considered in the paper (e.g., Adding) can also be addressed via text-to-audio generation.**\n\nIndeed, tasks such as adding, dropping, and replacement editing can be achieved through text-to-audio generation. However, directly using a text-to-audio generation model might struggle to retain the parts that don't require editing. For example, if we want to add \"adding a train is passing by\" to the audio of \"a man is speaking,\" using a text-to-generation model would require inputting \"a man is speaking while a train is passing by.\" But in the resulting new audio generated by the model, the \"a man is speaking\" part would changed. \n\nIn fact, we have actually compared it with the text-to-generation model. We employ SDEdit as the baseline for our editing tasks. We add noise and then use a pre-trained text-to-audio diffusion model to denoise input audio with the editing target description. When SDEdit uses a large number of noising steps, denoted as $N$ (such as when $N$ equals the total steps $T$ in the forward diffusion process used to train the text-to-audio generation model), it can be viewed as directly generating new audio from the introduced noise. In our paper, we present the results of the baseline method (SDEdit) at different values of $N$, we observed that the editing performance of these variations of SDEdit consistently outperforms the direct generation approach using our trained text-to-audio model (or equivalently, setting $N$ equal to $T$ within SDEdit).\n\n> **3. There also lacks analysis on the gain of the proposed method over SDEdit and its variants.**\n\nIn Section 3.3 of the paper, we analyze the advantages of the AUDIT model. In contrast to the zero-shot SDEdit based on a text-to-audio generative diffusion model, our method directly use the input audio as a condition for supervised training of our diffusion model, allowing the model to automatically learn to preserve the parts that do not need to be edited before and after editing.  In contrast, SDEdit requires editing to commence from the noised input audio based on new textual descriptions, lacking visibility into the origin audio. Furthermore, SDEdit can't engage in editing directly based on commands, unlike AUDIT (which can operate based on concise instructions), it needs a complete description of the target. In the revised version, we will provide a general summary encompassing the aforementioned aspects.\n\n> **4. How robust the model is on real data.**\n\nWe tested our model on some real data and discovered that as long as the semantic information of the audio events requiring editing (e.g., labels) is present in our training data, our model performs well in executing editing tasks. For instance, it excels at super-resolution on real speech data or inpainting on music data. In the case of adding, dropping, and replacement tasks, our model demonstrates robust recognition capabilities for most audio events and can successfully carry out these edits on real-world data.\n\nWe also discussed the model's generalization at the task level. You can check our response to reviewer LhWX's question 2 for more details.\n\n[1] Han et al., NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates\n\n[2] Marafioti A, Majdak P, Holighaus N, et al. A generative adversarial context encoder for long audio inpainting[J]. 2020.\n\n[3] Ebner P P, Eltelt A. Audio inpainting with generative adversarial network[J]. arXiv preprint arXiv:2003.07704, 2020."}, {"review_id": "9eZ3Wbsc7Y", "reviewer": "Reviewer_LhWX", "summary": "This paper proposes AUDIT, an instruction-guided audio editing model based on latent diffusion models. AUDIT is trained on triplet data consisting of an instruction, an input audio, and an output audio. The instruction is used to guide the diffusion model to only modify the audio segments that need to be edited. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks, such as adding, dropping, replacement, inpainting, and super-resolution.", "questions": "1. Please add ablation analysis for each component.\n2. Can you train it on one task and try to zero shot in other task?\n3. The model compare to SDEdit. please add more comparison to other method (classic / deep learning based).", "limitations": ".", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "AUDIT is a novel audio editing model that can be trained using human text instructions. AUDIT is trained on a dataset of triplet data, which consists of an audio clip, a text instruction, and the edited audio clip. AUDIT can maximize the preservation of audio segments that do not need to be edited, and it only requires simple instructions as text guidance. AUDIT has achieved state-of-the-art results in both objective and subjective metrics for several audio editing tasks.\n\n", "weaknesses": ".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments! We will explain your concerns point by point.\n\n> **1. Please add ablation analysis for each component.**\n\nWe have supplemented the impact of the text encoder on the performance of the text-to-audio generation model.\n\n| Text Encoder | FD($\\downarrow$) | IS($\\uparrow$) |KL($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n| T5-base | **20.19** | **9.23** | **1.32** |\n| CLAP | 21.62| 8.25| 1.51|\n| BERT-base | 20.97| 9.12| 1.37|\n| | | | |\n\nWe have observed that language models pre-trained solely on textual data as text encoders achieve better performance compared to the CLAP model. Although the CLAP model is trained on text-audio pairs, its representation of audio collapses into a single embedding, providing a relatively coarse audio semantic information. As a result, it might struggle to effectively represent complex audio containing multiple events, including the relationships between different audio events, such as sequential order. For instance, consider the example: \"A man is speaking, after that, a train is passing by.\"\n\nWe also conducted a comparison of our vocoder's impact on model objective metrics, contrasting it with AudioLDM. Our vocoder exhibited superior performance, which contributed to the enhancement of our model's objective metrics in the realm of text-to-audio generation.\n\n| Vocoder | FD($\\downarrow$) | IS($\\uparrow$) |KL($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n| AudioLDM | 8.76 | 10.71 | 0.23 |\n| Ours | **7.11** | **12.02** | **0.18** |\n| | | | |\n\nFor our audio editing model, we compared the differences between instruction prompts generated using ChatGPT and instructions generated directly through rules. We found that instruction prompts generated by ChatGPT tend to yield better results compared to instructions generated solely by rules. This is primarily because instruction data generated by ChatGPT exhibits a richer distribution, while instructions directly generated by rules tend to have a more limited distribution.\n\n> **2. Can you train it on one task and try to zero shot in other task?**\n\nA crucial issue! In our work, we primarily consider five different types of editing tasks, while in reality, there are many specific types of editing tasks. Can we build a general audio editing model that can handle almost all editing tasks, similar to the current large language models that can process a variety of tasks based on provided prompts? This is an important direction for our future work. For the current version of AUDIT, our model demonstrates a certain level of generalization at the task level:\n\n- Generalization from Single Tasks to Composite Tasks. Our training data consists of five distinct editing tasks, with each data point focusing on one of the five tasks. Our model demonstrates a certain degree of generalization to composite tasks. For instance, when presented with a segment of low-resolution audio along with a portion masked, we can simultaneously apply super-resolution and completion to the audio.\n\n- Generalization of Editing Audio Event Count. A significant portion of our dataset involves editing a single audio event, such as \"add a dog barking sound to the audio.\" Our model also exhibits generalization to editing multiple audio events simultaneously. For example, given the instruction \"add a dog barking sound and a baby crying,\" our model can simultaneously incorporate both a dog barking sound and a baby crying sound.\n\n> **3. Please add more comparison to other method (classic / deep learning based).**\n\nwe have add comparison to task-specific methods for super-resolution and inpainting.\n\nSuper-resolution:\n\nFor the super-resolution task, we conducted a comparison with the NU-Wave2 [1] model, which is a task-specific model designed for super-resolution. We utilized the provided open-source checkpoint for NU-Wave2; however, note that NU-Wave2 was originally trained on speech data. To adapt it to our specific task, we trained it on our own dataset.\n\n| Mehtod | LSD($\\downarrow$) | KL($\\downarrow$) |FD($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n|SDEdit (best)|1.75|1.17|27.81|\n| AUDIT | 1.48| **0.73**| **18.14**|\n| NU-Wave2 (origin) | 1.66| 0.89| 22.61|\n| NU-Wave2 (trained with our data)| **1.42**| 0.78| 19.57|\n| | | | |\n\nThe results indicate that our AUDIT model outperforms the NU-Wave2 model trained on speech and is comparable to or even better than the NU-Wave2 model trained on our dataset.\n\nInpainting:\n\nFor the audio inpainting task, we conducted comparative experiments with GACELA [2] and DAS [3]. GACELA is a generative adversarial network for restoring missing musical audio data. The publicly available checkpoint was trained solely on music data; hence, we trained the model using our own dataset. DAS is another GAN-based audio editing model, we also trained it on our dataset.\n\n| Mehtod | LSD($\\downarrow$) | KL($\\downarrow$) |FD($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n|SDEdit (best)|1.54|0.94|21.07|\n| AUDIT | **1.32**| **0.75**| **18.17**|\n| GACELA (trained with our data) | 1.41| 0.78| 20.49|\n| DAS (trained with our data) | 1.57 | 0.89| 21.97|\n| | | | |\n\nWe discovered that AUDIT's performance on inpainting tasks can surpass that of task-specific baseline methods. One possible explanation for this is that during training, GACELA and DAS do not have access to any textual information, whereas our model requires explicit instructions to edit the audio. As a result, our model is better able to learn the semantic information of the audio, which likely contributes to its improved performance in inpainting tasks.\n\nWe will add more experiments in the revised version!\n\n[1] Han et al., NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates\n\n[2] Marafioti A, Majdak P, Holighaus N, et al. A generative adversarial context encoder for long audio inpainting[J]. 2020.\n\n[3] Ebner P P, Eltelt A. Audio inpainting with generative adversarial network[J]. arXiv preprint arXiv:2003.07704, 2020."}, {"review_id": "ZQGKEjEI7f", "reviewer": "Reviewer_ptM3", "summary": "This work proposes a new editing technique to audio data.", "questions": "**Technical questions:** \n- Which diffusion process was used? \n- How do you pass the condition to the model?\n\nI couldn't find answers to those questions explicitly in the paper.", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors introduce a new task and a base method to tackling it. ", "weaknesses": "**Conditioning:** The authors concatenate the input audio to the sampled noise, why did you pick this option on top of sampling a Gaussian around the input sample of giving it as part of the cross attention (which will probably allow for modifications in the generation length)? Can the authors elaborate on this point?\n\n**FAD** The authors do not show the FAD of audiogen and make-an-audio. Why? I noted that the classifier is different (PANN vs. VGGISH), but it is hard to understand what are the differences in performance. \n\n**Comparison to prior work:** I am having hard time understanding what is the cause for the model's performance - is it the data or the modeling. All the baselines present results when trained on different (mostly open) datasets; To do an appropriate comparison, it is expected to at least train one-two of those models on the same data the authors used.\n\n**Subjective study** I find the experimental study not reproducible (even after reading Appendix D). Did the authors also check metrics compared to ground truth data? At least for dropping and super resolution, you have a hard gt data.\n\n**Error bars** Some of the metric differences are small - can the authors report error bars?\n\nOverall: I think that the task is interesting. However, since it is a new task and the paper doesn't introduce a new modeling approach, I would expect extensive experiments, which I was unable to find.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments! We will explain your concerns point by point.\n\n> **1. conditioning.**\n\nThank you found the alternative conditioning approaches helpful. Using Gaussian sampling around the input sample as the starting point for diffusion denoising (replacing the conventional method of directly sampling noise from a standard normal distribution) is indeed a viable approach, as demonstrated in works like GradTTS. However, it's worth noting that this approach might not provide a completely origin input audio due to the presence of variance, So, this approach might not be the optimal choice.\n\nUsing the input audio as cross-attention input would indeed introduce additional computational complexity. On the other hand, directly concatenating the latent representation of the input audio along the channel dimension is a simpler and more straightforward approach. This design allows the model to potentially \"copy\" the content of the input audio in regions where no editing is required. This simplicity can also aid in the preservation of unedited areas.\n\nThis approach is also present in some other works, such as the image editing model Palette [1]. We believe that this method represents one of the most straightforward and simplistic ways to ensure that the model can comprehensively \"see\" the audio that needs to be edited during both training and inference.\n\n> **2. FAD.**\n\nOur subjective evaluation criteria follow AudioLDM. AudioLDM points out that the difference between the FD and FAD metrics lies in the use of a more advanced audio classification model, PANNs, for FD. Therefore, FD could potentially be a better evaluation metric. As a result, we have chosen FD as the main evaluation metric in our paper. We will supplement the results of AUDIT on FAD in the revised version of the paper. You can refer to the table below question 3 for more details!\n\n> **3. Comparison to prior work.**\n\nCompared to previous work, our text-to-audio model has a larger parameter count. Our diffusion U-Net has 859M parameters (while AudioLDM-L has 739M). Additionally, we observed a significant impact of the vocoder on objective metrics. A vocoder trained with larger datasets and longer training times can improve objective metrics.\n\nIn essence, our paper primarily focuses on audio editing by following instructions. We trained the text-to-audio diffusion model to provide a strong baseline (the SDEdit method based on the text-to-audio diffusion model) for our editing model. Therefore, we aimed to achieve high performance with our generation model. Indeed, our text-to-audio model has reached state-of-the-art levels, ensuring its effectiveness as a baseline for the editing model.\n\nTo conduct a fair comparison for the text-to-audio generation model, we also included results from models trained on AudioCaps and AudioSet datasets. However, it's important to note that previous work using the AudioSet dataset also involved using only a subset of AudioSet (e.g., excluding speech, music, etc.). Due to the lack of specific data segmentation details, achieving complete data parity remains challenging. You can find more details in the table below.\n\n| Model | Dataset | FD($\\downarrow$) | IS($\\uparrow$) |KL($\\downarrow$) | FAD($\\downarrow$) |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| DiffSound | AudioSet+AudioCaps | 47.68 | 4.01 | 2.52 | 7.75|\n| AudioGen |  AudioSet+AudioCaps | - | - | 2.09| 3.13|\n| Make-an-Audio | AudioSet+AudioCaps+13 others | - | - | 2.79| 4.61|\n| AudioLDM-L |  AudioCaps | 27.12| 7.51| 1.86| 2.08|\n| AudioLDM-L-Full | AudioSet+AudioCaps+2 others | 23.31| 8.13| 1.59| **1.96**|\n| AUDIT | AudioSet+AudioCaps+2 others| **20.19**| **9.23**| **1.32**| 2.01|\n| AUDIT | AudioSet + AudioCaps| 21.24| 8.47| 1.39| 1.98|\n| | | | | | |\n\n> **4. Subjective study**\n\nTo evaluate audio quality, we explicitly instruct the raters to \"focus on examining the audio quality\", to evaluate text-audio alignment, raters are shown an audio and a text and asked \"Does the natural language description align with the audio\". For the super-resolution and inpainting tasks, we also evaluated the results using ground truth mel spectrograms (Inpainting: Quality 89.74, Relevance 91.92; Super-resolution: Quality 90.31, Relevance 92.03). We will add the result in the revised version paper.\n\n> **5. Error bars** \n\nWhen comparing to prior work, we've seen substantial gains in objective metrics, You can check the table below question 3 to get more details.\n\n> **6. Which diffusion process was used?**\n\nWe employ a DDPM process for the forward diffusion process, where the total number of forward diffusion steps is set to 1000 (with $\\beta_0=0.0001, \\beta_{1000}=0.02$, and a linear beta schedule). In the reverse process, we utilize DDIM for sampling. The default number of sampling steps for DDIM is set to 100.\n\n> **7. How do you pass the condition to the model?**\n\nFor the text-to-audio generation model, where the condition is text, we begin by encoding the textual content into a text embedding using a T5 encoder model. This text embedding is then incorporated into the model using cross-attention, the text embedding providing the keys (K) and values (V) for this cross-attention mechanism.\n\nRegarding the audio editing model, where the conditions are text and input audio, the text condition is introduced in the same manner as in the text-to-audio model. As for the input audio, we directly concatenate the latent representation of the input audio with the model's input, which consists of the latent representation of the target audio after introducing noise. This concatenation is performed along the channel dimension, ensuring that the model has access to information from the input audio during both training and inference stages. This design allows the model to \"see\" the input audio and effectively utilize it for the editing process.\n\n[1] Saharia C, Chan W, Chang H, et al. Palette: Image-to-image diffusion models."}, {"review_id": "cv6sNidnKj", "reviewer": "Reviewer_ujZe", "summary": "This paper proposes a zero-shot generative editing model for audio signals. The text-based user edits include adding, dropping, replacing, inpainting, and perceptually upsampling audio signals. The audio is not limited to any particular audio domain (e.g., speech or music). Comparisons to recent text-to-audio and zero-shot editing models demonstrate a clear improvement in the state-of-the-art.", "questions": "Why regenerate the context conditioning during inpainting? Don’t you already have access to the ground truth?\n\nIt seems speech is excluded throughout the paper—including in the audio samples. Why exclude speech?\n\nHave you evaluated performance formally or informally on out-of-domain data? What sort of results do you get? How fragile is the model outside the training distribution?", "limitations": "Listening to the audio samples, the pitch reconstruction seems noticeably off—enough that the current system would not yet be amenable to fine-grained music editing. As well, the reconstruction of high frequencies could be improved and the context conditioning degrades noticeably in inpainting. The paper could benefit from a reconstruction task (i.e., inpaint with no mask) to better understand why context conditioning is degrading when the ground truth is provided to the model. Otherwise, it's not clear whether quality/accuracy errors can be attributed to the audio generator or the editing process.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a notable advance in text-to-audio generation as well as audio editing. This paper outlines a framework for general purpose text-based neural audio editing that I could foresee becoming a blueprint for how this type of editing is performed. I imagine many possible subsequent works focusing on quality improvements and further editing capabilities—which there are many.\n\nOne of the challenges of producing such an editing system is that each editing capability requires its own evaluation. That is well handled here.", "weaknesses": "This is not “the first audio editing model based on human instructions” (line 43-44) as any editing model is technically following the user’s instructions. Consider “the first model to perform semantic audio editing from text-based prompts” or something similar. Also, you could have a machine generate the instructions, so they don’t have to be human users.\n\nThe evaluation of super-resolution would benefit from a comparison with a recent task-specific model (e.g., [1, 2]). I’m not yet convinced that performing super-resolution via text prompt would be preferable over having a dedicated super-resolution model.\n\nIt is not clear what each of the baseline models is in Table 7.\n\n[1] Su, Jiaqi, Zeyu Jin, and Adam Finkelstein. \"HiFi-GAN-2: Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features.\" 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021.\n\n[2] Andreev, Pavel, et al. \"Hifi++: a unified framework for bandwidth extension and speech enhancement.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments! We will explain your concerns point by point.\n\n> **1. This is not “the first audio editing model based on human instructions” (line 43-44) as any editing model is technically following the user’s instructions. Consider “the first model to perform semantic audio editing from text-based prompts” or something similar. Also, you could have a machine generate the instructions, so they don’t have to be human users.**\n\nThank you very much for your valuable suggestions! We will revise the wording in the revised version of the paper to make it more accurate.\n\n> **2. The evaluation of super-resolution would benefit from a comparison with a recent task-specific model. I’m not yet convinced that performing super-resolution via text prompt would be preferable over having a dedicated super-resolution model.**\n\nThanks for your valuable suggestion! Since our approach is based on a diffusion model, our primary comparisons are conducted with zero-shot editing methods that also rely on diffusion models. We have incorporated comparative experiments with task-specific models, focusing on the tasks of super-resolution and inpainting. Please check our general rebuttal or our rebuttal for Reviewer XrSg (Question 1).\n\n> **3. It is not clear what each of the baseline models is in Table 7.**\n\nWe have opted to conduct a subjective evaluation on the best-performing baseline methods from Table 1 to Table 5. In the revised version of the paper, we will amend the description in Table 7 to specify the names of the baseline methods that were selected for the subjective evaluation.\n\n> **4. Why regenerate the context conditioning during inpainting? Don’t you already have access to the ground truth?**\n\nWe regenerate the context conditioning during inpainting to better assess our model's ability to retain the unedited portions. In fact, for improved results, it is possible to replace non-masked areas with ground truth information (this approach is adopted by our baseline methods, SDEdit-Precise and SDEdit-Rough). For the sake of maintaining overall framework consistency, we omitted this step in the final system.\n\n> **5. It seems speech is excluded throughout the paper—including in the audio samples. Why exclude speech?**\n\nIt's a truly thought-provoking question! In reality, our training data includes audio data containing human speech. Our showcase samples also encompass instances like \"Drop the sound of a woman talking.\" However, due to the intricate nature of human speech and the absence of finer supervisory signals (such as phonemes), generating or editing high-quality speech data can be challenging. For instance, even though my training data comprises audio segments of people speaking, when attempting to generate audio of a person speaking, like \"A man is speaking and a dog is barking,\" we can produce speech-like sounds but are unable to discern the specific speech content, making it appear as if it's gibberish.\n\nOur paper primarily concentrates on the broader aspects of general audio generation and editing. In the future, we aspire to seamlessly integrate speech as well, yet I believe that achieving this goal will undoubtedly require supplying more granular supervisory signals.\n\n> **6. Have you evaluated performance formally or informally on out-of-domain data? What sort of results do you get? How fragile is the model outside the training distribution?**\n\nWe tested our model on some real data and discovered that as long as the semantic information of the audio events requiring editing (e.g., labels) is present in our training data, our model performs well in executing editing tasks. For instance, it excels at super-resolution on real speech data or inpainting on music data. In the case of adding, dropping, and replacement tasks, our model demonstrates robust recognition capabilities for most audio events and can successfully carry out these edits on real-world data.\n\nWe also discussed the model's generalization at the task level. For example, generalization from single tasks to composite tasks. You can check our response to reviewer LhWX's question 2 for more details.\n\n> **7. Limitations.**\n\nWe encountered a similar issue where the audio quality slightly degraded after editing. This problem primarily stems from the vocoder and VAE reconstruction. We computed objective metrics for audio generated using ground truth mel spectrograms plus vocoder, as well as VAE reconstruction plus vocoder. Of course, we believe that introducing a complete reconstruction task (directly copying the input) in our training regimen can further aid the model in retaining unedited portions. This suggestion is highly valuable!\n\n| Setting | FD($\\downarrow$) | IS($\\uparrow$) |KL($\\downarrow$)\n| :---: | :---: | :---: | :---: |\n| GT mel + vocoder | 7.11 | 12.02 | 0.18 |\n| VAE reconstruction + vocoder | 9.89 | 14.37 | 0.35 |\n| | | | |"}]

Author rebuttal:
Thank you very much for the valuable suggestions and insightful questions you've provided! We have addressed each reviewer's concerns point by point.

Furthermore, we would like to reiterate some common points of discussion (of course, we have also responded individually to each reviewer's specific questions).

**Regarding the comparison with task-specific models**, we have expanded our experiments to include audio super-resolution and audio inpainting tasks.

For the super-resolution task, we conducted a comparison with the NU-Wave2 [1] model, which is a task-specific model designed for super-resolution. We utilized the provided open-source checkpoint for NU-Wave2; however, it's important to note that NU-Wave2 was originally trained on speech data. To adapt it to our specific task, we performed additional training on our own dataset.

| Mehtod | LSD($\downarrow$) | KL($\downarrow$) |FD($\downarrow$)
| :---: | :---: | :---: | :---: |
|SDEdit (best)|1.75|1.17|27.81|
| AUDIT | 1.48| **0.73**| **18.14**|
| NU-Wave2 (origin) | 1.66| 0.89| 22.61|
| NU-Wave2 (trained with our data)| **1.42**| 0.78| 19.57|
| | | | |

The results indicate that our AUDIT model outperforms the NU-Wave2 model trained on speech and is comparable to or even better than the NU-Wave2 model trained on our dataset.

For the audio inpainting task, we conducted comparative experiments with GACELA [2] and DAS [3]. GACELA is a generative adversarial network for restoring missing musical audio data. The publicly available checkpoint was trained solely on music data; hence, we trained the model using our own dataset. DAS is another GAN-based audio editing model, we also trained it on our dataset.

| Mehtod | LSD($\downarrow$) | KL($\downarrow$) |FD($\downarrow$)
| :---: | :---: | :---: | :---: |
|SDEdit (best)|1.54|0.94|21.07|
| AUDIT | **1.32**| **0.75**| **18.17**|
| GACELA (trained with our data) | 1.41| 0.78| 20.49|
| DAS (trained with our data) | 1.57 | 0.89| 21.97|
| | | | |


We discovered that AUDIT's performance on inpainting tasks can surpass that of task-specific baseline methods. One possible explanation for this is that during training, GACELA and DAS do not have access to any textual information, whereas our model requires explicit instructions to edit the audio. As a result, our model is better able to learn the semantic information of the audio, which likely contributes to its improved performance in inpainting tasks.

**Regarding the generalization of our model**, we have primarily discussed two key aspects. Firstly, we have examined its generalization to real data or out-of-domain scenarios. Secondly, we have explored its generalization capabilities in various editing tasks.

We tested our model on some real data and discovered that as long as the semantic information of the audio events requiring editing (e.g., labels) is present in our training data, our model performs well in executing editing tasks. For instance, it excels at super-resolution on real speech data or inpainting on music data. In the case of adding, dropping, and replacement tasks, our model demonstrates robust recognition capabilities for most audio events and can successfully carry out these edits on real-world data.

In our work, we primarily consider five different types of editing tasks, while in reality, there are many specific types of editing tasks. Can we build a general audio editing model that can handle almost all editing tasks, similar to the current large language models that can process a variety of tasks based on provided prompts? This is an important direction for our future work. For the current version of AUDIT, our model demonstrates a certain level of generalization at the task level:

- Generalization from Single Tasks to Composite Tasks. Our training data consists of five distinct editing tasks, with each data point focusing on one of the five tasks. Our model demonstrates a certain degree of generalization to composite tasks. For instance, when presented with a segment of low-resolution audio along with a portion masked, we can simultaneously apply super-resolution and completion to the audio.

- Generalization of the Number of Audio Events to be Edited. A significant portion of our dataset involves editing a single audio event, such as "add a dog barking sound to the audio." Our model also exhibits generalization to editing multiple audio events simultaneously. For example, given the instruction "add a dog barking sound and a baby crying," our model can simultaneously incorporate both a dog barking sound and a baby crying sound.

The primary innovations of our work lie in introducing a novel task format, performing semantic audio editing from text-based prompts, and establishing a pipeline that leverages large language models to construct training datasets. In the future, our objective is to unify a broader spectrum of audio generation and editing tasks under a prompt-guided framework. We hope that our efforts can serve as an initial exploration for subsequent research in the community.

If you have any further questions or concerns, please do not hesitate to let us know. We are more than happy to engage in further discussions and exchanges with all of you!

Meta-review: All reviewers agree that the results presented by the authors are impressive and the proposed method is interesting. The authors engaged with reviewers during the rebuttal phase and provided additional results and clarifications which makes this submission stronger. 
The main concern raised by the reviewers is model evaluation using real-world data. I do encourage the authors to include the additional details and results provided during the rebuttal period in their final version, including results on real-world data. Due to all of the above, I recommend acceptance.
Decision: Accept (poster)