Title: AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: audio editing

text-to-audio generation

diffusion models
Abstract: Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, \textbf{AUDIT} has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demopage.github.io/.

AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

Abstract

Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT , an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-theart results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demopage.github.io/ .

VAE

Encoder

VAE

Decoder

'

Add

horse hooves in the background'

'

the sound of duck quacking'

'

Replace

string instrument to drum kit'

'

Inpaint

'

'

Increase resolution

'

Diffusion

𝝐 ~ 𝓝(𝟎, 𝑰)

AUDIT

T5 Encoder

Adding

Dropping

Replacement

Inpainting

Super-Resolution

1 Introduction

Audio editing is a crucial step in producing high-quality audio or video content, which involves tasks like adding background sound effects, replacing background music, repairing incomplete audio, and enhancing low-sampled audio. Typically, these tasks require professional software and manual operations. However, if audio editing tools can be empowered to follow human instructions, it could significantly reduce the need for manual intervention and benefit content creators. The utilization of natural language instructions such as "add jazz music in the background" could enable more adaptable editing in line with human expectations. In this work, we concentrate on designing audio editing models that can follow human instructions.

Previous works on audio editing usually leverage traditional signal processing technologies. Later, GAN-based methods [33, 10, 1] have achieved success in some audio editing tasks such as audio inpainting and super-resolution. However, these methods are usually designed for a specific editing task. Recently, the most advanced deep generative models, diffusion models, have achieved great success in image editing. Some methods [18, 27] have attempted to apply diffusion models to the field of audio editing. These diffusion-based audio editing methods are primarily based on a pre-trained text-to-audio generation model. However, these methods still suffer from several issues: 1) they are mainly based on a pre-trained model to noise and denoise an input audio conditioned on a target text description, and in most cases, they offer no guarantee to achieve good editing effect since they are not trained directly on audio editing tasks; 2) they can lead to erroneous modifications on some audio segments that do not need to be edited, as it is difficult to restrict the editing area; 3) they require a complete description of the output audio, which is not always available or necessary in real-world editing scenarios. Instead, it is desirable to provide natural language editing instructions such as 'add a man whistling in the background' or 'drop the sound of the guitar' for audio editing.

To solve the above issues, we propose AUDIT, to the best of our knowledge, the first audio editing model based on human instructions. As shown in Figure 1, AUDIT is a latent diffusion model which takes the audio to be edited and the editing instruction as conditions and generates the edited audio as output. The core designs of AUDIT can be summarized in three points: 1) we generate triplet data (instruction, input audio, output audio) for different audio editing tasks, and train an audio editing diffusion model in a supervised manner; 2) we directly use input audio as conditional input to train the model, forcing it to automatically learn to ensure that the audio segments that do not need to be edited remain consistent before and after editing; 3) our method uses editing instructions directly as text guidance, without the need for a complete description of the output audio, making it more flexible and suitable for real-world scenarios. Our editing model achieves state-of-the-art results in objective and subjective metrics.

Our contributions can be summarized as follows:

We propose AUDIT, which demonstrates the first attempt to train a latent diffusion model conditioned on human text instructions for audio editing.

To train AUDIT in a supervised manner, we design a novel data construction framework for each editing task; AUDIT can maximize the preservation of audio segments that do not need to be edited; AUDIT only requires simple instructions as text guidance, without the need for a complete description of the editing target.

AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks.

2 Related Works

2.1 Audio/Speech Editing

Previous work related to audio editing has primarily focused on human speech or music. [36, 5, 33, 55] explored the task of speech/music inpainting based on the corresponding text or music score. Another category of editing work is focused on style transfer for speech or music. Speech voice conversion [2, 35, 44, 43] and singing voice conversion [7, 28] aim to modify the timbre without changing the speech or singing content. Music style transfer [51, 9, 30, 40] aims to change the instrument being played without modifying the music score. In addition, some methods using GANs [33, 10, 1] have achieved success in some specific audio editing tasks, such as audio inpainting and audio

super-resolution. However, there is still a lack of research on general audio editing following human instructions.

2.2 Diffusion-based Editing

Diffusion-based editing tasks have received similar attention as diffusion-based generation tasks. We summarize the common diffusion-based editing work into two types.

Zero-Shot Editing Some methods [34, 8, 4, 31, 14] use diffusion-based generate models to achieve zero-shot editing. SDEdit [34] uses a pre-trained diffusion model to add noise to an input image and then denoise the image with a new target text. These methods have difficulty in editing specific regions and have poor controllability. To edit specific regions, [4] uses an additional loss gradient to guide the sampling process, and [4, 31] replace unmodified parts with the noisy input image in each sampling step, equivalent to only generating in the masked part. However, these methods are only for inpainting tasks and also need a complete description of the editing target.

Supervised Training Another type of methods [47, 6, 54, 37] handle editing tasks in a supervised manner. [47] train an image-to-image diffusion model which takes the image to be edited as a condition. [47] uses paired image data generated by [14] to train a text-based image editing model. ControlNet [54] and T2I-Adapter [37] add an adapter model to a frozen pre-trained generation diffusion model and train the adapter model to achieve image editing. Our method follows the supervised training paradigm, using the generated triplet data (instruction, input audio, output audio) to train an audio editing model conditioned on the instruction and the audio to be edited, and can handle different audio editing tasks by following human instructions.

2.3 Text-Guided Audio Generation

Currently, most diffusion-based audio editing methods [18, 27] are based on text-to-audio diffusion generative models. Text-to-audio generation aims to synthesize general audio that matches the text description. DiffSound [52] is the first attempt to build a text-to-audio system based on a discrete diffusion model [13]. AudioGen [26] is another text-to-audio generation model based on a Transformer-based decoder that uses an autoregressive structure. AudioGen directly predicts discrete tokens obtained by compressing from the waveform [53]. Recently, Make-an-Audio [18] and AudioLDM [27] have attempted to build text-guided audio generation systems in continuous space, which use an autoencoder to convert mel-spectrograms into latent representation, build a diffusion model that works on the latent space, and reconstruct the predicted results of the latent diffusion model as mel-spectrograms using the autoencoder. Furthermore, MusicLM [3] and Noise2Music [17] generate music that matches the semantic information in the input text. In our work, we focus on editing the existing audio with human instructions.

3 Method

In this section, we present our method for implementing audio editing based on human instructions. We provide an overview of our system architecture and training objectives in Section 3.1 and show the details about how we generate triplet data (instruction, input audio, output audio) for each editing task in Section 3.2. Lastly, we discuss the advantages of our method in Section 3.3.

3.1 System Overview

Our system consists of an autoencoder that projects the input mel-spectrograms to an efficient, low-dimensional latent space and reconstructs it back to the mel-spectrograms, a text encoder that encodes the input text instructions, a diffusion network for editing in the latent space, and a vocoder for reconstructing waveforms. The overview of the system is shown in Figure 2.

Autoencoder The autoencoder contains an encoder E and a decoder G . The encoder E transforms the mel-spectrogram x into a latent representation z , and the decoder G reconstructs ˆ x from the latent space. In our work, we employ a variational autoencoder (V AE) [23] model as the autoencoder. We train our autoencoder with the following loss: 1) The L 1 and L 2 reconstruction loss. 2) The

Latent Diffusion

VAE Decoder

Vocoder

T5 Encoder

𝒙

𝒊𝒏

𝒙

𝒐𝒖𝒕

'Drop the sound of a duck quacking'

VAE Encoder

VAE Encoder

𝒛

𝟎

𝒛

𝒕

𝒛

𝒊𝒏

𝒛

𝒕'𝟏

% 𝒛

𝟎

Kullback-Leibler loss L KL to regularize the latent representation z . We only assign a small weight λ KL to the KL loss following [45]; 3) The GAN loss L GAN . We employ a patch-based discriminator D [19, 45] to distinguish between real and reconstructed mel-spectrograms. The total training loss of the autoencoder can be expressed as L V AE = λ 1 L 1 + λ 2 L 2 + λ KL L KL + λ GAN L GAN . λ 1 , λ 2 , λ KL , and λ GAN are the weights of L 1 , L 2 , L KL , and L GAN respectively.

Text Encoder We use a pre-trained language model T5 [21] as our text encoder, which is used to convert text input into embeddings that contain rich semantic information. The parameters of the text encoder are frozen in the training stage.

Latent Diffusion Model Our text-guided audio editing model can be seen as a conditional latent diffusion model that takes the latent representation z in of the input (to be edited) mel-spectrogram x in and text embeddings as conditions. The model aims to learn p ( z out | z in , c text ) , where z out is the latent representation of the output (edited) mel-spectrogram x out , and c text is the embedding of the editing instruction. Given the latent representation z in and the text instruction, we randomly select a time step t and a noise ϵ to generate a noisy version z t of the latent representation z out by using a noise schedule and then use the diffusion network ϵ θ ( z t , t, z in , c text ) to predict the sampled noise. The training loss is L LDM = E ( z in , z out ,text ) E ϵ ∼N (0 , I ) E t || ϵ θ ( z t , t, z in , c text ) -ϵ || 2 .

Similarly to the original standard diffusion model [15, 49], we use a U-Net [46] with the crossattention mechanism [50] as the diffusion network. Our editing model takes z in as a condition by directly concatenating z t and z in at the channel level. Therefore, the input channel number of the first layer of the U-Net is twice the output channel number of the last layer.

Vocoder A vocoder model is required to convert the output mel-spectrogram into audio. In this work, we use HiFi-GAN [24] as the vocoder, which is one of the most widely used vocoders in the field of speech synthesis. HiFi-GAN uses multiple small sub-discriminators to handle different cycle patterns. Compared with some autoregressive [38, 20] or flow-based [42] vocoders, it takes into account both generation quality and inference efficiency.

3.2 Generating Triplet Training Data for Each Editing Task

We use generated triplet data (instruction, input audio, output audio) to train our text-guided audio editing model. In this work, we focus on five different editing tasks, including adding, dropping, replacement, inpainting, and super-resolution. Note that we train all the editing tasks in a single editing model. Figure 3 provides an overview of how our data generation workflow for different editing tasks works.

Adding We randomly select two audio clips A and B from the text-audio datasets, then combine A and B to get a new audio clip C . We use A as the input audio and C as the output audio, and we fill in the caption (or label) of B into the instruction template to get the instruction. For example, as shown in Figure 3, the instruction template is 'Add {} in the background' and the caption of B

Template: 'Inpaint'

A cat meowing.

input audio

randomly mask

Instruction: 'Inpaint'

output audio

Template: 'Increase resolution'

A bird singing.

input audio

down sample

Instruction: 'Increase

resolution'

output audio

Template: 'Add {} in the

background'

Belling.

Baby crying.

Instruction: 'Add baby crying

in the background'

output audio

input audio

Template: 'Drop {}'

A car engine

accelerating.

Dog barking.

Instruction: 'Drop dog barking'

input audio

output audio

Template:

'Replace

{}

with

{}'

Instruction: 'Replace clapping

with guitar'

input audio

output audio

Someone clapping

.

The sound of guitar.

Adding

Dropping

Replacement

Inpainting

Super-resolution

→

is 'Baby crying' , so the instruction is 'Add baby crying in the background' . Note that we design multiple templates for different editing tasks.

Dropping We randomly select two audio clips A and B from the text-audio datasets, then combine A and B to get a new audio clip C . We use C as the input audio and A as the output audio, and we fill in the caption (or label) of B into the instruction template to get the instruction. For example, the instruction template is 'Drop {}' and the caption of B is 'Dog barking' , so the instruction is 'Drop Dog barking' .

Replacement For the replacement task, we select three audio clips A , B , and C from the datasets, then insert B and C separately into A (in roughly the same area) to obtain two new audio clips D and E . We use D as the input audio and E as the output audio, and we fill in the captions (or labels) of B and C into the instruction template to get the instruction. For example, the instruction template is 'Replace {} with {}' , the caption of B is 'Someone clapping' , and the caption of C is 'The sound of guitar' , so the instruction is 'Replace clapping with guitar' .

Inpainting and Super-Resolution For the inpainting tasks, we select an audio clip as the output audio and randomly mask some parts of the audio to get a new audio clip as the input audio. We can use instructions like 'Inpaint' or 'Inpaint the audio' directly or use the instructions like 'Inpaint: a cat meowing' . For the super-resolution tasks, we use down-sampled audio as the input audio. The instructions are as 'Increase resolution' or 'Increase resolution: a bird singing' .

Extending Instructions with ChatGPT We design multiple instruction templates for each editing task. In order to further expand the diversity of editing instructions, we use ChatGPT [39] to generate more editing instruction templates for us. See more details in Appendix E.

3.3 Advantages of AUDIT

We analyze the advantages of our methods over previous works on three key points. 1) We generated triplet data (instruction, input data, output data) to train a text-guided audio editing model instead of performing zero-shot editing, which can ensure good edit quality. 2) We directly use the input audio as a condition for supervised training of our diffusion model, allowing the model to automatically learn to preserve the parts that do not need to be edited before and after editing. Specifically, we concatenate the latent representations z in of the input audio and z t on the channel dimension and input them into the latent diffusion model, so that the model can "see" z in (rather than its noisy version) during both training and inference. 3) Instead of using a full description of the output audio, we use human instructions as the text input, which is more in line with real-world applications.

4 Experimental Settings

4.1 Dataset

The datasets used in our work consist of AudioCaps [22], AudioSet [12], FSD50K [11], and ESC50 [41]. AudioSet is the largest audio-label pairs dataset; however, each audio clip in AudioSet only has a corresponding label and no description. Because of the presence of a large number of human speech and some audio clips that are only weakly related to their labels in AudioSet, we use a subset of AudioSet (AudioSet96) that includes approximately 149K pairs with 96 classes. AudioCaps is a dataset of around 44K audio-caption pairs, where each audio clip corresponds to a caption with rich semantic information. The length of each audio clip in AudioSet and AudioCaps is around 10 seconds. FSD50K includes around 40K audio clips and 200 audio-label pairs of variable lengths ranging from 0.3 to 30 seconds. We split FSD50K training set into two datasets, FSD50K-L (19K) and FSD50K-S (22K), comprising audio clips of length less than and greater than 5 seconds, respectively. ESC50 is a smaller dataset with 50 classes, each containing four audio clips of around 5 seconds, for a total of 2000 audio clips.

We use AudioCaps, AudioSet96, FSD50K-S, and ESC50 to generate triplet training data for five audio editing tasks. We use a total of about 0.6M triplet data to train our audio editing model. More details about datasets and data processing are shown in Appendix A.

4.2 Baseline Systems

Text-to-Audio Generation Since we compare with generative model-based audio editing baseline methods, we use AudioCaps, AudioSet96, FSD50K, and ESC50 to train a text-to-audio latent diffusion model and use the test set of AudioCaps to evaluate the text-to-audio model. To demonstrate the performance of our generative model, we compare it with some state-of-the-art text-to-audio generative models [52, 26, 27, 18].

Adding, Dropping, and Replacement We use methods like SDEdit[34] as baselines. SDEdit uses a pre-trained text-to-image diffusion model to noise and then denoise an input image with the editing target description. In our work, we directly use SDEdit in the adding task, dropping task, and replacement task. We use our own trained text-to-audio latent diffusion model. We test different total denoising steps N for N = 3 / 4 T , N = 1 / 2 T , and N = 1 / 4 T , where T is the total step in the forward diffusion process. The three baselines are called: 1) SDEdit-3/4T ; 2) SDEdit-1/2T ; 3) SDEdit-1/4T .

Inpainting For the inpainting task, we designed four baselines derived from SDEdit but with some differences between each other. 1) SDEdit . We directly use SDEdit, we first add noise to the input audio, then denoise the audio with the description (caption or label) of the output audio as text guidance. 2) SDEdit-Rough and 3) SDEdit-Precise . Only a part of the input audio (the part that is masked) needs to be edited in the inpainting task, we call the part that does not need to be edited the 'observable' part, and we call the masked part the 'unobservable' part. In each step of the denoising process, we can replace the 'observable' part with the ground truth in the latent space. The difference between SDEdit-Rough and SDEdit-Precise is that in SDEdit-Rough, the 'unobservable' part is a rough region, while in SDEdit-Precise, the 'unobservable' part is a precise region. 4) SDEdit-wo-Text . SDEdit-wo-Text is similar to SDEdit-Precise, however, it has no text input. An example of the difference of the 'unobservable' part between SDEdit-Rough and SDEdit-Precise is shown in Appendix F. We also compared our methods with two task-specific methods GACELA [32] and DAS [10]. GACELA is a generative adversarial network for restoring missing musical audio data. The publicly available checkpoint was trained solely on music data; hence, we trained the model using our own dataset. DAS is another GAN-based audio editing model, we also trained it on our dataset.

Super-Resolution Super-resolution can be seen as inpainting in the frequency domain. We use three baselines similar to the inpainting task: 1) SDEdit ; 2) SDEdit-Precise ; 3) SDEdit-wo-Text . We also conducted a comparison with the NU-Wave2 [51] model, which is a task-specific model designed for super-resolution. We utilized the provided open-source checkpoint for NU-Wave2; however, note that NU-Wave2 was originally trained on speech data. To adapt it to our specific task, we trained it on our own dataset.

4.3 Evaluation Metrics

Objective Metrics We use log spectral distance (LSD), frechet distance (FD), and kullback-leibler divergence (KL) to evaluate our text-guided audio editing model and use inception score (IS), FD, and KL to evaluate our text-to-audio generation model following [52, 26, 27, 18]. LSD measures the distance between frequency spectrograms of output samples and target samples. FD measures the fidelity between generated samples and target samples. IS measures the quality and diversity of generated samples. KL measures the correlation between output samples and target samples. FD, IS, and KL are based on the state-of-the-art audio classification model PANNs [25]. We use the evaluation pipeline 2 provided by [27] for objective evaluation for fairer comparisons.

Subjective Metrics Weuse overall quality (Quality) to measure the sound quality and naturalness of the output audio compared to the input audio and use relevance to the editing instruction (Relevance) to measure how well the output audio matches the input human instruction . Each sample will be scored from 1 to 100 based on Quality and Relevance. See more details about the evaluation in Appendix D.

4.4 Model Configurations

Our autoencoder compresses a mel-spectrogram of size 1 × H × W into a latent representation of size 4 × H 4 × W 4 . Our models are trained on 8 NVIDIA V100 GPUs for 500K steps with a batch size of 2 on each device. We use the weights of our pre-trained text-to-audio model to initialize our audio editing model. Our HiFi-GAN vocoder is trained on AudioSet96 and AudioCaps datasets using 8 NVIDIA V100 GPU for 200 epochs. More details about model configurations are shown in Appendix B.

5 Results

5.1 Objective Evaluation

Adding Table 1 shows the objective evaluation results of the adding task. Our method achieves the best performance, with LSD of 1.35, KL of 0.92, and FD of 21.80. Compared with the best baseline method, our method reduces FD by 6.45 and KL by 0.38.

Dropping Table 2 shows the objective evaluation results of the dropping task. Our method achieves the best performance, with LSD of 1.37, KL of 0.95, and FD of 22.40. Compared with the best baseline method, our method reduces FD by 5.79 and KL by 0.10.

Replacement Table 3 shows the objective evaluation results of the replacement task. Our method achieves the best performance, with LSD of 1.37, KL of 0.84, and FD of 21.65. Compared with the best baseline method, our method reduces FD by 5.07 and KL by 0.31.

Inpainting Table 4 shows the objective evaluation results of the inpainting task. Our method achieves the best performance, with LSD of 1.32, KL of 0.75, and FD of 18.17. We find that for baseline methods, not providing a text input (the caption of the audio) as guidance leads to a large performance drop, SDEdit-wo-Text gets the worst performance in terms of KL and FD among the baseline methods. However, AUDIT-wo-Text which only uses instructions like 'inpaint' achieves performance close to AUDIT which uses instructions like 'inpaint + caption' . We also discovered that AUDIT's performance on inpainting tasks can surpass that of task-specific baseline methods. One possible explanation for this is that during training, GACELA and DAS do not have access to any textual information, whereas our model requires explicit instructions to edit the audio. As a result, our model is better able to learn the semantic information of the audio, which likely contributes to its improved performance in inpainting tasks.

Super-Resolution Table 5 shows the objective evaluation results of the super-resolution task. Our method achieves the best performance, with LSD of 1.48, KL of 0.73, and FD of 18.14. Similar to the inpainting task, SDEdit-wo-Text gets the worst performance in terms of KL and FD among the baseline methods. Our method can achieve significantly better results than baselines using only simple instructions like 'Increase resolution' , which shows that our method can learn sufficient semantic information from low-frequency information. The results also indicate that our AUDIT model outperforms the NU-Wave2 model trained on speech and is comparable to or even better than the NU-Wave2 model trained on our dataset.

Text-to-Audio Generation We also present the comparison results of our text-to-audio latent diffusion model with other text-to-audio models in Table 6. Our model achieves the best performance, with FD of 20.19, KL of 1.32, and IS of 9.23, outperforming AudioLDM-L-Full with FD of 23.31, KL of 1.59, and IS of 8.13. This demonstrates that our generation model can serve as a strong baseline model for generation-based editing methods.

5.2 Subjective Evaluation

The results of the subjective evaluation are shown in Table 7. We choose the best results in the baseline to report. Our method clearly outperforms the baseline methods in both Quality and Relevance across

all five tasks. In the dropping task, our method achieves the highest scores with Quality of 78.1 and Relevance of 81.0. We find that compared with the baseline methods, our method improves more significantly on the adding, dropping, and replacement tasks. The possible reason is that compared with the inpainting and super-resolution tasks, which have explicit positioning of the editing area (the masked part and the high-frequency area), the adding, dropping, and replacement tasks need to first locate the region to be edited, and also need to ensure that other regions cannot be modified, which is more difficult for the baseline method. Our model has learned this ability through supervised learning.

5.3 Case Study

In order to more specifically show the difference between the performance of our model and the baseline methods, we also give some case studies in Appendix G.

6 Conclusions

In this work, we propose an audio editing model called AUDIT, which can perform different editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution) based on human text instructions. Specifically, we train a text-guided latent diffusion model using our generated triplet training data (instruction, input audio, output audio), which only requires simple human instructions as guidance without the need for the description of the output audio and performs audio editing accurately without modifying audio segments that do not need to be edited. AUDIT achieves state-of-the-art performance on both objective and subjective metrics for five different audio editing tasks. For future work, we will explore more editing audio tasks with our framework, and achieve more precise control for audio editing. We also discuss some limitations and broader impacts of our work in Appendix H.

References

Sherif Abdulatif, Ruizhe Cao, and Bin Yang. Cmgan: Conformer-based metric-gan for monaural speech enhancement. arXiv preprint arXiv:2209.11112 , 2022.

Masanobu Abe, Satoshi Nakamura, Kiyohiro Shikano, and Hisao Kuwabara. Voice conversion through vector quantization. Journal of the Acoustical Society of Japan (E) , 11(2):71-76, 1990.

Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325 , 2023.

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208-18218, 2022.

Zalán Borsos, Matt Sharifi, and Marco Tagliasacchi. Speechpainter: Text-conditioned speech inpainting. arXiv preprint arXiv:2202.07273 , 2022.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

Xin Chen, Wei Chu, Jinxi Guo, and Ning Xu. Singing voice conversion with non-parallel data. In 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) , pages 292-296. IEEE, 2019.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 , 2021.

Ondˇ rej Cífka, Umut ¸ Sim¸ sekli, and Gaël Richard. Groove2groove: one-shot music style transfer with supervision from synthetic data. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:2638-2650, 2020.

Pirmin P Ebner and Amr Eltelt. Audio inpainting with generative adversarial network. arXiv preprint arXiv:2003.07704 , 2020.

Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:829-852, 2021.

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 776-780. IEEE, 2017.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917 , 2023.

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661 , 2023.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1125-1134, 2017.

Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning , pages 2410-2419. PMLR, 2018.

Mihir Kale and Abhinav Rastogi. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.10433 , 2020.

Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 119-132, 2019.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems , 33:17022-17033, 2020.

Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:2880-2894, 2020.

Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352 , 2022.

Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503 , 2023.

Songxiang Liu, Yuewen Cao, Dan Su, and Helen Meng. Diffsvc: A diffusion probabilistic model for singing voice conversion. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pages 741-748. IEEE, 2021.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Chien-Yu Lu, Min-Xin Xue, Chia-Che Chang, Che-Rung Lee, and Li Su. Play as you like: Timbre-enhanced multi-modal music style transfer. In Proceedings of the aaai conference on artificial intelligence , volume 33, pages 1061-1068, 2019.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022.

Andrés Marafioti, Piotr Majdak, Nicki Holighaus, and Nathanaël Perraudin. A generative adversarial context encoder for long audio inpainting. 2020.

Andrés Marafioti, Piotr Majdak, Nicki Holighaus, and Nathanaël Perraudin. Gacela: A generative adversarial context encoder for long audio inpainting of music. IEEE Journal of Selected Topics in Signal Processing , 15(1):120-131, 2021.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Seyed Hamidreza Mohammadi and Alexander Kain. An overview of voice conversion systems. Speech Communication , 88:65-82, 2017.

Ondˇ rej Mokr` y and Pavel Rajmic. Audio inpainting: Revisited and reweighted. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:2906-2918, 2020.

Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 , 2016.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Marco Pasini. Melgan-vc: Voice conversion and audio style transfer on arbitrarily long samples using spectrograms. arXiv preprint arXiv:1910.03713 , 2019.

Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia , pages 1015-1018, 2015.

Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. In International Conference on Machine Learning , pages 7706-7716. PMLR, 2020.

Adam Polyak, Lior Wolf, Yossi Adi, and Yaniv Taigman. Unsupervised cross-domain singing voice conversion. arXiv preprint arXiv:2008.02830 , 2020.

Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. arXiv preprint arXiv:2109.13821 , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACMSIGGRAPH 2022 Conference Proceedings , pages 1-10, 2022.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Shih-Lun Wu and Yi-Hsuan Yang. Musemorphose: Full-song and fine-grained music style transfer with one transformer vae. arXiv preprint arXiv:2105.04090 , 2021.

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.09983 , 2022.

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:495-507, 2021.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang. Vision-infused deep audio inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 283-292, 2019.

A Datasets and Data Processing

Table 8 presents some details about the datasets we used and generated. All audio data have a sampling rate of 16KHz. For AudioSet, AudioCaps, and AudioSet96, we pad or truncate all audio to 10s. For FSD50K, we pad audio shorter than 5s to 5s, and truncate or pad audio longer than 5s to 10s. All generated audio data have a length of 10 seconds. We convert 10s of audio into mel-spectrograms with a size of 80 × 624 , using a hop size of 256, a window size of 1024, and mel-bins of size 80, covering frequencies from 0 to 8000. The autoencoder compresses the mel-spectrograms to a latent representation of size 4 × 10 × 78 .

B Model Details

Table 9 shows more details about our audio editing and audio generative models. We train our autoencoder model with a batch size of 32 (8 per device) on 8 NVIDIA V100 GPUs for a total of 50000 steps with a learning rate of 7 . 5 e -5 . For both audio editing and U-Net audio generative diffusion, we train with a batch size of 8 on 8 NVIDIA V100 GPUs for a total of 500000 steps with a learning rate of 5 e -5 . Both the autoencoder and diffusion models use AdamW[29] as the optimizer with ( β 1 , β 2 ) = (0 . 9 , 0 . 999) and weight decay of 1 e -2 . We follow the official repository to train the HiFi-GAN vocoder.

C Classifier-free Guidance

[16] proposed using classifier-free guidance to trade off diversity and sample quality. The classifierfree guidance strategy has been widely used in conditional diffusion models. Our model has two additional conditions, c text and z in . However, during training, we only consider helping the model learn the marginal distribution of p ( z t | z in ) (without explicitly learning p ( z t | c text ) ). Therefore, during training, we mask the text with a certain probability (replacing it with an empty text ∅ ) to learn the no-text condition score ϵ θ ( z t , t, z in ) . Then, according to Bayes' formula p ( z t | c text ) ∝ p ( z t | c text , z in ) p ( z in | z t ,c text ) , we can derive the score relationship in Equation 1, which corresponds to the classifierfree guidance Equation 2. Here, the parameter s ≥ 1 is the guidance coefficient used to balance the diversity and quality of the samples.

D Human Evaluation

For each audio editing task, our human evaluation set comprises ten samples randomly selected from the test set. Ten raters score each sample according to two metrics, Quality and Relevance, using a scale of 1 to 100.

E Extending Instructions with ChatGPT

Some examples of instruction templates designed by ourselves:

'Add {} in the beginning'

'Add {} at the beginning'

'Add {} in the end'

'Add {} in the middle'

'Add {} in the background'

'Drop {} '

'Remove {} '

'Replace {} to {} '

'Replace {} with {} '

'Inpaint'

'Inpainting'

'Inpaint {} '

'Inpaint: {} '

'Increase resolution'

'Increase resolution: {} '

'Perform super-resolution'

'Perform super-resolution: {} '

We use ChatGPT to extend editing templates, specifically, we submit the editing instruction templates we designed to ChatGPT and let it generate more instruction templates with the same semantic information. Below are some examples.

'Mix {} into the background'

'Blend {} with existing audio'

'Incorporate {} as a new element at the end of the audio'

'Place {} in the foreground'

'Erase {} from the track'

'Subtract {} from the audio'

'Take out {} from the foreground'

'Exchange {} for {} in the mix'

'Use {} to replace {} in the audio'

'Interchange {} and {} in the track'

'Replace missing audio with synthesized sound'

'Fill in the gaps in track {} '

'Upscale audio to higher resolution'

'Apply super-resolution to the audio to improve clarity'

F Baseline Methods

For the inpainting task, only a part of the input audio (the part that is masked) needs to be edited, we call the part that does not need to be edited the 'observable' part, and we call the masked part the 'unobservable' part. In each step of the denoising process, we can replace the 'observable' part with the ground truth in the latent space. The difference between SDEdit-Rough and SDEdit-Precise is that in SDEdit-Rough, the 'unobservable' part is a rough region, while in SDEdit-Precise, the 'unobservable' part is a precise region. Figure 4 gives an example.

The 'unobservable' part in SDEdit-Precise.

G Case Study

We show case studies. Figure 5 shows a case for the adding task. The caption of the input audio is 'The sound of machine gun' , and the editing target is adding a bell ringing in the beginning. AUDIT performs audio editing accurately in the correct region without modifying audio segments that do not need to be edited. Figure 6 shows a case for the dropping task. The caption of the input audio is 'A bird whistles continuously, while a duck quacking in water in the background' , and the editing target is dropping the sound of the duck quacking. Our method successfully removes the background sound and preserves the sound of the bird whistling, but the sound of the bird whistling in the SDEdit editing result is incorrectly modified. It shows that our method can better ensure that the audio segments that do not need to be edited are not modified. Figure 7 shows a case for the inpainting task. The caption of the input audio is 'A person is typing computer' . While both AUDIT and the baseline method generate semantically correct results, the result generated by AUDIT is more natural and contextual.

Input

Generate by AUDIT

Generate by SDEdit

Input

Generate by AUDIT

Generate by SDEdit

Input

Generate by AUDIT

Generate by SDEdit-Precise

H Limitations and Broader Impacts

Our work still has some limitations. For example, the sampling efficiency is low since our model is a diffusion-based model. In the future, we will try to improve the generation efficiency of our model using efficient strategies like consistency models [48]. In addition, we will also explore the use of more data and more diverse editing instructions to achieve more kinds of editing tasks. AUDIT can edit existing audio based on natural language instructions, which may be used inappropriately, such as synthesizing fake audio for fraud. Therefore, we urge everyone not to abuse this technology and develop synthetic audio detection tools.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a zero-shot generative editing model for audio signals. The text-based user edits include adding, dropping, replacing, inpainting, and perceptually upsampling audio signals. The audio is not limited to any particular audio domain (e.g., speech or music). Comparisons to recent text-to-audio and zero-shot editing models demonstrate a clear improvement in the state-of-the-art.\n\nSTRENGTHS:\nThis is a notable advance in text-to-audio generation as well as audio editing. This paper outlines a framework for general purpose text-based neural audio editing that I could foresee becoming a blueprint for how this type of editing is performed. I imagine many possible subsequent works focusing on quality improvements and further editing capabilities—which there are many.\n\nOne of the challenges of producing such an editing system is that each editing capability requires its own evaluation. That is well handled here.\n\nWEAKNESSES:\nThis is not “the first audio editing model based on human instructions” (line 43-44) as any editing model is technically following the user’s instructions. Consider “the first model to perform semantic audio editing from text-based prompts” or something similar. Also, you could have a machine generate the instructions, so they don’t have to be human users.\n\nThe evaluation of super-resolution would benefit from a comparison with a recent task-specific model (e.g., [1, 2]). I’m not yet convinced that performing super-resolution via text prompt would be preferable over having a dedicated super-resolution model.\n\nIt is not clear what each of the baseline models is in Table 7.\n\n[1] Su, Jiaqi, Zeyu Jin, and Adam Finkelstein. \"HiFi-GAN-2: Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features.\" 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2021.\n\n[2] Andreev, Pavel, et al. \"Hifi++: a unified framework for bandwidth extension and speech enhancement.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\nQUESTIONS:\nWhy regenerate the context conditioning during inpainting? Don’t you already have access to the ground truth?\n\nIt seems speech is excluded throughout the paper—including in the audio samples. Why exclude speech?\n\nHave you evaluated performance formally or informally on out-of-domain data? What sort of results do you get? How fragile is the model outside the training distribution?\n\nLIMITATIONS:\nListening to the audio samples, the pitch reconstruction seems noticeably off—enough that the current system would not yet be amenable to fine-grained music editing. As well, the reconstruction of high frequencies could be improved and the context conditioning degrades noticeably in inpainting. The paper could benefit from a reconstruction task (i.e., inpaint with no mask) to better understand why context conditioning is degrading when the ground truth is provided to the model. Otherwise, it's not clear whether quality/accuracy errors can be attributed to the audio generator or the editing process.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

comments: SUMMARY:
This work proposes a new editing technique to audio data.

STRENGTHS:
The authors introduce a new task and a base method to tackling it.

WEAKNESSES:
**Conditioning:** The authors concatenate the input audio to the sampled noise, why did you pick this option on top of sampling a Gaussian around the input sample of giving it as part of the cross attention (which will probably allow for modifications in the generation length)? Can the authors elaborate on this point?

**FAD** The authors do not show the FAD of audiogen and make-an-audio. Why? I noted that the classifier is different (PANN vs. VGGISH), but it is hard to understand what are the differences in performance. 

**Comparison to prior work:** I am having hard time understanding what is the cause for the model's performance - is it the data or the modeling. All the baselines present results when trained on different (mostly open) datasets; To do an appropriate comparison, it is expected to at least train one-two of those models on the same data the authors used.

**Subjective study** I find the experimental study not reproducible (even after reading Appendix D). Did the authors also check metrics compared to ground truth data? At least for dropping and super resolution, you have a hard gt data.

**Error bars** Some of the metric differences are small - can the authors report error bars?

Overall: I think that the task is interesting. However, since it is a new task and the paper doesn't introduce a new modeling approach, I would expect extensive experiments, which I was unable to find.

QUESTIONS:
**Technical questions:** 
- Which diffusion process was used? 
- How do you pass the condition to the model?

I couldn't find answers to those questions explicitly in the paper.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes AUDIT, an instruction-guided audio editing model based on latent diffusion models. AUDIT is trained on triplet data consisting of an instruction, an input audio, and an output audio. The instruction is used to guide the diffusion model to only modify the audio segments that need to be edited. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks, such as adding, dropping, replacement, inpainting, and super-resolution.

STRENGTHS:
AUDIT is a novel audio editing model that can be trained using human text instructions. AUDIT is trained on a dataset of triplet data, which consists of an audio clip, a text instruction, and the edited audio clip. AUDIT can maximize the preservation of audio segments that do not need to be edited, and it only requires simple instructions as text guidance. AUDIT has achieved state-of-the-art results in both objective and subjective metrics for several audio editing tasks.

WEAKNESSES:
.

QUESTIONS:
1. Please add ablation analysis for each component.
2. Can you train it on one task and try to zero shot in other task?
3. The model compare to SDEdit. please add more comparison to other method (classic / deep learning based).

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes AUDIT, an instruction-guided audio editing model based on latent diffusion models. The model is trained with triplet training data (instruction, input audio, output audio) for
different audio editing tasks and is based on a diffusion model using instruction and input
(to be edited) audio as conditions and generating output (edited) audio. The proposed model outperforms several baselines in the editing tasks

STRENGTHS:
The AUDIT model is able to perform audio editing conditioned on text instructions, which have rarely been studied by prior work on audio generation. The data construction strategy of using paired text-audio data and refining text instructions based on templates and LLM is also useful for making instruction datasets. In terms of performance, it is better than several baselines (SDEdit and its variants).

WEAKNESSES:
The main weakness lies in the evaluation. The model is mostly compared to SDEdit and its variants, which is zero-shot audio editing model. It is unclear how it compares to task-specific models such as NU-Wave2 [1] for audio super-resolution. Some of the editing tasks considered in the paper (e.g., Adding) can also be addressed via text-to-audio generation, while proposed method has not been compared to the methods of this category. There also lacks analysis on the gain of the proposed method over SDEdit and its variants. For example, how robust the model is to the category of the dropped sound in Dropping task? As both the training and evaluation data are synthesized, it is unclear how the model will perform on real data.

[1]. Han et al., NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07