SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dropping Table 2 shows the objective evaluation results of the dropping task. Our method achieves the best performance, with LSD of 1.37, KL of 0.95, and FD of 22.40. Compared with the best baseline method, our method reduces FD by 5.79 and KL by 0.10.
Table 2: Objective evaluation results of the dropping task.
Model	Text	LSD(↓)	KL(↓)	FD(↓)
SDEdit-3/4T	caption	1.54	1.14	29.66
SDEdit-1/2T	caption	1.43	1.05	28.19
SDEdit-1/4T	caption	1.40	1.30	31.31
AUDIT	instruction	1.37	0.95	22.40
$^{2}$ https://github.com/haoheliu/audioldm_eval
Replacement Table 3 shows the objective evaluation results of the replacement task. Our method achieves the best performance, with LSD of 1.37, KL of 0.84, and FD of 21.65. Compared with the best baseline method, our method reduces FD by 5.07 and KL by 0.31.
Table 3: Objective evaluation results of the replacement task.
Model	Text	LSD(↓)	KL(↓)	FD(↓)
SDEdit-3/4T	caption	1.63	1.58	28.78
SDEdit-1/2T	caption	1.52	1.27	27.71
SDEdit-1/4T	caption	1.46	1.15	26.72
AUDIT	instruction	1.37	0.84	21.65
** inpainting** Table 4 shows the objective evaluation results of the inpainting task. Our method achieves the best performance, with LSD of 1.32, KL of 0.75, and FD of 18.17. We find that for baseline methods, not providing a text input (the caption of the audio) as guidance leads to a large performance drop, SDEdit-wo-Text gets the worst performance in terms of KL and FD among the baseline methods. However, AUDIT-wo-Text which only uses instructions like "inpaint" achieves performance close to AUDIT which uses instructions like "inpaint + caption". We also discovered that AUDIT's performance on inpainting tasks can surpass that of task-specific baseline methods. One possible explanation for this is that during training, GACELA and DAS do not have access to any textual information, whereas our model requires explicit instructions to edit the audio. As a result, our model is better able to learn the semantic information of the audio, which likely contributes to its improved performance in inpainting tasks.
Table 4: Objective evaluation results of the inpainting task.
Model	Text	LSD(↓)	KL(↓)	FD(↓)
SDEdit	caption	2.91	1.47	25.42
SDEdit-Rough	caption	1.64	0.98	21.99
SDEdit-Precise	caption	1.54	0.94	21.07
SDEdit-wo-Text	-	1.55	1.63	27.63
GACELA	-	1.41	0.78	20.49
DAS	-	1.57	0.89	21.97
AUDIT-wo-Text	instruction	1.37	0.81	19.03
AUDIT	instruction + caption	1.32	0.75	18.17
Super-Resolution Table 5 shows the objective evaluation results of the super-resolution task. Our method achieves the best performance, with LSD of 1.48, KL of 0.73, and FD of 18.14. Similar to the inpainting task, SDEdit-wo-Text gets the worst performance in terms of KL and FD among the baseline methods. Our method can achieve significantly better results than baselines using only simple instructions like "Increase resolution", which shows that our method can learn sufficient semantic information from low-frequency information. The results also indicate that our AUDIT model outperforms the NU-Wave2 model trained on speech and is comparable to or even better than the NU-Wave2 model trained on our dataset.
Text-to-Audio Generation We also present the comparison results of our text-to-audio latent diffusion model with other text-to-audio models in Table 6. Our model achieves the best performance, with FD of 20.19, KL of 1.32, and IS of 9.23, outperforming AudioLDM-L-Full with FD of 23.31, KL of 1.59, and IS of 8.13. This demonstrates that our generation model can serve as a strong baseline model for generation-based editing methods.
5.2 Subjective Evaluation
The results of the subjective evaluation are shown in Table 7. We choose the best results in the baseline to report. Our method clearly outperforms the baseline methods in both Quality and Relevance across
Table 5: Objective evaluation results of the super-resolution task.
Model	Text	LSD(↓)	KL(↓)	FD(↓)
SDEdit	caption	3.14	1.50	25.31
SDEdit-Precise	caption	1.75	1.17	27.81
SDEdit-wo-Text	-	1.78	1.56	32.66
NU-Wave2 (origin)	-	1.66	0.89	22.61
NU-Wave2	-	1.42	1.78	19.57
AUDIT-wo-Text	instruction	1.53	0.92	21.97
AUDIT	instruction + caption	1.48	0.73	18.14
Table 6: The comparison between our text-to-audio generative model and other baseline models on the AudioCaps test set.
Model	Dataset	FD(↓)	IS(↑)	KL(↓)
DiffSound [52]	AudioSet+AudioCaps	47.68	4.01	2.52
AudioGen [26]	AudioSet+AudioCaps	-	-	2.09
Make-an-Audio[18]	AudioSet+AudioCaps+13 others	-	-	2.79
AudioLDM-L [27]	AudioCaps	27.12	7.51	1.86
AudioLDM-L-Full [27]	AudioSet+AudioCaps+2 others	23.31	8.13	1.59
AUDIT	AudioSet96+AudioCaps+2 others	20.19	9.23	1.32
all five tasks. In the dropping task, our method achieves the highest scores with Quality of 78.1 and Relevance of 81.0. We find that compared with the baseline methods, our method improves more significantly on the adding, dropping, and replacement tasks. The possible reason is that compared with the inpainting and super-resolution tasks, which have explicit positioning of the editing area (the masked part and the high-frequency area), the adding, dropping, and replacement tasks need to first locate the region to be edited, and also need to ensure that other regions cannot be modified, which is more difficult for the baseline method. Our model has learned this ability through supervised learning.
Table 7: Subjective evaluation. For each audio editing task, we report the results of the best baseline method and the results of our method.
Method	Task	Quality(↑)	Relevance(↑)
Baseline AUDIT	Adding	60.2	56.7
75.5	77.3
Baseline AUDIT	Dropping	54.4	48.2
78.1	81.0
Baseline AUDIT	Replacement	57.7	47.6
72.5	74.5
Baseline AUDIT	Inpainting	65.8	66.3
75.2	78.7
Baseline AUDIT	Super-Resolution	61.8	59.9
74.6	76.3
5.3 Case Study
In order to more specifically show the difference between the performance of our model and the baseline methods, we also give some case studies in Appendix G.
In this work, we propose an audio editing model called AUDIT, which can perform different editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution) based on human text instructions. Specifically, we train a text-guided latent diffusion model using our generated triplet training data (instruction, input audio, output audio), which only requires simple human instructions as guidance without the need for the description of the output audio and performs audio editing accurately without modifying audio segments that do not need to be edited. AUDIT achieves state-of-the-art performance on both objective and subjective metrics for five different audio editing tasks. For future work, we will explore more editing audio tasks with our framework, and achieve more precise control for audio editing. We also discuss some limitations and broader impacts of our work in Appendix H.
[51] Shih-Lun Wu and Yi-Hsuan Yang. Museomorphose: Full-song and fine-grained music style transfer with one transformer vae. arXiv preprint arXiv:2105.04090, 2021.
[52] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. arXiv preprint arXiv:2207.09983, 2022.
[53] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[54] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[55] Hang Zhou, Ziwei Liu, Xudong Xu, Ping Luo, and Xiaogang Wang. Vision-infused deep audio inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 283-292, 2019.
A Datasets and Data Processing
Table 8 presents some details about the datasets we used and generated. All audio data have a sampling rate of  $16\mathrm{kHz}$ . For AudioSet, AudioCaps, and AudioSet96, we pad or truncate all audio to 10s. For FSD50K, we pad audio shorter than 5s to 5s, and truncate or pad audio longer than 5s to 10s. All generated audio data have a length of 10 seconds. We convert 10s of audio into mel-spectrograms with a size of  $80 \times 624$ , using a hop size of 256, a window size of 1024, and mel-bins of size 80, covering frequencies from 0 to 8000. The autoencoder compresses the mel-spectrograms to a latent representation of size  $4 \times 10 \times 78$ .
Table 8: Details about audio-text datasets we use and our generated audio editing datasets
Dataset	Hours	Number	Text
AudioSet	5800	2M	label
AudioSet96	414	149K	label
AudioCaps	122	44K	caption
FSD50K	108	51K	label
FSD50K-S	31	22K	label
FSD50K-L	53	19K	label
ESC50	3	2K	label
Task	Datasets	Number	Text
Generation	AudioCaps, AudioSet96, FSD50K, ESC50	243K	label or caption
Adding	AudioCaps, AudioSet96, FSD50K-S,ESC50	71K	Instruction
Dropping	AudioCaps, AudioSet96, FSD50K-S,ESC50	71K	Instruction
Replacement	AudioSet96, FSD50K-S, ESC50	50K	Instruction
Inpainting	AudioSet96, AudioCaps	193K	Instruction
Super-resolution	AudioSet96, AudioCaps	193K	Instruction
B Model Details
Table 9 shows more details about our audio editing and audio generative models. We train our autoencoder model with a batch size of 32 (8 per device) on 8 NVIDIA V100 GPUs for a total of 50000 steps with a learning rate of  $7.5e - 5$ . For both audio editing and U-Net audio generative diffusion, we train with a batch size of 8 on 8 NVIDIA V100 GPUs for a total of 500000 steps with a learning rate of  $5e - 5$ . Both the autoencoder and diffusion models use AdamW[29] as the optimizer with  $(\beta_{1},\beta_{2}) = (0.9,0.999)$  and weight decay of  $1e - 2$ . We follow the official repository to train the HiFi-GAN vocoder.
C Classifier-free Guidance
[16] proposed using classifier-free guidance to trade off diversity and sample quality. The classifier-free guidance strategy has been widely used in conditional diffusion models. Our model has two additional conditions,  $c_{text}$  and  $z_{in}$ . However, during training, we only consider helping the model learn the marginal distribution of  $p(\pmb{z}_t|\pmb{z}_{in})$  (without explicitly learning  $p(\pmb{z}_t|c_{text})$ ). Therefore, during training, we mask the text with a certain probability (replacing it with an empty text  $\emptyset$ ) to learn the no-text condition score  $\epsilon_{\theta}(\pmb{z}_t,t,\pmb{z}_{in})$ . Then, according to Bayes' formula  $p(\pmb{z}_t|c_{text}) \propto \frac{p(\pmb{z}_t|c_{text},\pmb{z}_{in})}{p(\pmb{z}_{in}|\pmb{z}_t,c_{text})}$ , we can derive the score relationship in Equation 1, which corresponds to the classifier-free guidance Equation 2. Here, the parameter  $s \geq 1$  is the guidance coefficient used to balance the diversity and quality of the samples.
$$
\nabla_ {\boldsymbol {z} _ {t}} \log p (\boldsymbol {z} _ {i n} | \boldsymbol {z} _ {t}, c _ {t e x t}) = \nabla_ {\boldsymbol {z} _ {t}} \log p (\boldsymbol {z} _ {t} | c _ {t e x t}, \boldsymbol {z} _ {i n}) - \nabla_ {\boldsymbol {z} _ {t}} \log p (\boldsymbol {z} _ {t} | \boldsymbol {z} _ {i n}) \tag {1}
$$
$$
\tilde {\epsilon} _ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t, \boldsymbol {z} _ {i n}, c _ {\text {t e x t}}\right) = \epsilon_ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t, \boldsymbol {z} _ {i n}, \emptyset\right) + s \cdot \left(\epsilon_ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t, \boldsymbol {z} _ {i n}, c _ {\text {t e x t}}\right) - \epsilon_ {\boldsymbol {\theta}} \left(\boldsymbol {z} _ {t}, t, \boldsymbol {z} _ {i n}, \emptyset\right)\right) \tag {2}
$$
Table 9: Details about our audio editing and audio generative models
Model	Configuration
Autoencoder	Number of Parameters	83M
In/Out Channels	1
Latent Channels	4
Number of Down/Up Blocks	4
Block Out Channels	(128, 256, 512, 512)
Activate Function	SiLU
T5 Text Encoder	Number of Parameters	109M
Output Channels	768
Max Length	300
Editing Diffusion U-Net	Number of Parameters	859M
In Channels	8
Out Channels	4
Number of Down/Up Blocks	4
Block Out Channels	(320, 640, 1280, 1280)
Attention Heads	8
Cross Attention Dimension	768
Activate Function	SiLU
Generative Diffusion U-Net	Number of Parameters	859M
In Channels	4
Out Channels	4
Number of Down/Up Blocks	4
Block Out Channels	(320, 640, 1280, 1280)
Attention Heads	8
Cross Attention Dimension	768
Activate Function	SiLU
HiFi-GAN	Sampling Rate	16000
Number of Mels	80
Hop Size	256
Window Size	1024
D Human Evaluation
For each audio editing task, our human evaluation set comprises ten samples randomly selected from the test set. Ten raters score each sample according to two metrics, Quality and Relevance, using a scale of 1 to 100.
E Extending Instructions with ChatGPT
Some examples of instruction templates designed by ourselves:
"Add {} in the beginning"
"Add {} at the beginning"
"Add {} in the end"
"Add {} in the middle"
"Add {} in the background"
"Drop {}
"Remove {}
"Replace  $\{\}$  to  $\{\}$ "
"Replace {} with {}
"Inpaint"
"Inpainting"
"Inpaint {}
"Inpaint: {}"
"Increase resolution"
"Increase resolution: {}
"Perform super-resolution"
"Perform super-resolution: {}
We use ChatGPT to extend editing templates, specifically, we submit the editing instruction templates we designed to ChatGPT and let it generate more instruction templates with the same semantic information. Below are some examples.
"Mix {} into the background"
"Blend {} with existing audio"
"Incorporate {} as a new element at the end of the audio"
"Place {} in the foreground"
"Erase {} from the track"
"Subtract {} from the audio"
"Take out {} from the foreground"
"Exchange  $\{\}$  for  $\{\}$  in the mix
"Use \{\} to replace \{\} in the audio"
"Interchange {} and {}. in the track"
"Replace missing audio with synthesized sound"
"Fill in the gaps in track {}
"Upscale audio to higher resolution"
"Apply super-resolution to the audio to improve clarity"
F Baseline Methods
For the inpainting task, only a part of the input audio (the part that is masked) needs to be edited, we call the part that does not need to be edited the "observable" part, and we call the masked part the "unobservable" part. In each step of the denoising process, we can replace the "observable" part with the ground truth in the latent space. The difference between SDEdit-Rough and SDEdit-Precise is that in SDEdit-Rough, the "unobservable" part is a rough region, while in SDEdit-Precise, the "unobservable" part is a precise region. Figure 4 gives an example.
The "unobservable" part in SDEdit-Precise.
The "unobservable" part in SDEdit-Rough.
Figure 4: The difference between SDEdit-Rough and SDEdit-Precise.
G Case Study
We show case studies. Figure 5 shows a case for the adding task. The caption of the input audio is "The sound of machine gun", and the editing target is adding a bell ringing in the beginning. AUDIT performs audio editing accurately in the correct region without modifying audio segments that do not need to be edited. Figure 6 shows a case for the dropping task. The caption of the input audio is "A bird whistles continuously, while a duck quacking in water in the background", and the editing target is dropping the sound of the duck quacking. Our method successfully removes the background sound and preserves the sound of the bird whistling, but the sound of the bird whistling in the SDEdit editing result is incorrectly modified. It shows that our method can better ensure that the audio segments that do not need to be edited are not modified. Figure 7 shows a case for the inpainting task. The caption of the input audio is "A person is typing computer". While both AUDIT and the baseline method generate semantically correct results, the result generated by AUDIT is more natural and contextual.
Generate by AUDIT
Generate by SDEdit
Figure 5: A case for the adding task. The caption of the input audio is "The sound of machine gun", and the editing target is adding a bell ringing in the beginning.
Input
Figure 6: A case for the dropping task. The caption of the input audio is "A bird whistles continuously, while a duck quacking in water in the background", and the editing target is dropping the sound of the duck quacking.
Input
Figure 7: A case for the inpainting task. the caption of the input audio is "A person is typing computer".
Generate by SDEdit-Precise
H Limitations and Broader Impacts
Our work still has some limitations. For example, the sampling efficiency is low since our model is a diffusion-based model. In the future, we will try to improve the generation efficiency of our model using efficient strategies like consistency models [48]. In addition, we will also explore the use of more data and more diverse editing instructions to achieve more kinds of editing tasks. AUDIT can edit existing audio based on natural language instructions, which may be used inappropriately, such as synthesizing fake audio for fraud. Therefore, we urge everyone not to abuse this technology and develop synthetic audio detection tools.