SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DIFF-FOLEY: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
Simian Luo $^{1,2}$  Chuanhao Yan $^{1}$  Chenxu Hu $^{1}$  Hang Zhao $^{1,2*}$
$^{1}$ IIIS, Tsinghua University  $^{2}$ Shanghai Qi Zhi Institute
{luosm22, yanch21, hu-cx21}@mails.tsinghua.edu.cn
hangzhao@mail.tsinghua.edu.cn
The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present DIFF-FOLEY, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with 'double guidance'. DIFF-FOLEY achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning. Project Page: https://diff-foley.github.io/
Figure 1: Traditional Foley v.s Neural Foley: Traditional Foley, a time-consuming and labor-intensive process involving skilled artists manipulating objects to create hours of physical sounds for sound recording [31]. In contrast, Neural Foley presents an appealing alternative utilizing neural networks to synthesize high-quality synchronized audio, accelerating video production and alleviating human workload.
Recent advances in diffusion models [42; 45; 16] have accelerated the development of AI-generated contents, e.g. Text-to-Image (T2I) generation [37; 39; 38], Text-to-Audio (T2A) generation [48; 27], and Text-to-Video (T2V) generation [15]. This paper focuses on Video-to-Audio (V2A) generation, which has practical applications in video/film production. During live shooting, due to the presence of excessive background noise and challenges in the audio collection in complex scenarios, the majority
of the sounds recorded in the film need to be recreated during post-production. This process of adding synchronized and realistic sound effects to videos is known as Foley [1]. We compare traditional Foley performed in the studio and our neural Foley in Figure 1. Traditional Foley is a laborious and time-consuming process, that involves skilled artists manipulating objects to create hours of authentic physical sounds. In contrast, Neural Foley offers an appealing alternative. High-quality synchronized audio generated by AI can greatly accelerate video production and alleviate the human workload. Different from Neural Dubber [19], which synthesizes speech from text scripts and video frames, Neural Foley focus on generating a broad range of audio solely based on the video content, a task that poses a considerably higher level of difficulty.
Video-based audio generation offers two natural advantages over text-based generation while performing Foley. First, T2A requires lots of hard-to-collect text-audio pairs for training, in contrast, audio-video pairs are readily available on the Internet, e.g. millions of new videos are uploaded to YouTube daily. Second, along with the semantics of the generated audio, V2A can further control the temporal synchronization between the Foley audio and video.
Semantic content matching and temporal synchronization are two major goals in V2A. While some progress [35; 50; 3; 21; 41] have been made recently on V2A, most methods of audio generation focus only on the content relevance, neglecting crucial aspect of audio-visual synchronization. For example, given a video of playing drums, existing methods can only generate drums sound, but cannot ensure the sounds match exactly with what's happening in the video, i.e. hitting the snare drum or the crash cymbal at the right time.
RegNet [3] uses a pretrained (RGB+Flow) network as conditional inputs to GAN for synthesizing sounds. Meanwhile, SpecVQGAN [21] uses a Transformer [47] autoregressive model conditioned on pretrained ResNet50 [14] or (RGB+Flow) visual features for better sample quality. These methods have limitations in generating audio that is both synchronized and relevant to video content as pretrained image and flow features cannot capture the nuanced correlation between audio and video.
We present DIFF-FOLEY, a novel Neural Foley framework based on Latent Diffusion Model (LDM) [38] that synthesizes realistic and synchronized audio with strong audio-visual relevance. DIFF-FOLEY first learns temporally and semantically aligned features via CAVP. By maximizing the similarity of visual and audio features in the same video, it captures subtle audio-visual connections. Then, an LDM conditioned on CAVP visual features is trained on the spectral latent space. CAVP-aligned visual features help LDM in capturing audio-visual relationships. To further improve sample quality, we propose 'double guidance', using classifier-free and alignment classifier guidance jointly to guide the reverse process. DIFF-FOLEY achieves state-of-the-art performance on large-scale V2A dataset VGGSound [2] with IS of 62.37, outperforming SpecVQGAN [21] baseline (IS of 30.01) by a large margin. We also demonstrate DIFF-FOLEY practical applicability and adaptability via customized downstream finetuning, validating the potential of V2A pretrained generative models.
Video-to-Audio Generation Generating audio from silent videos has potential in video production, which can greatly improve post-production efficiency. Despite recent progress, open-domain V2A remains a challenge. FoleyGAN [11] uses a class-conditioned GAN for synchronized audio synthesis, but it requires a specific action class label and is tested on a limited dataset. RegNet [3] extracts video features by using a pretrained (RGB+Flow) network and serves as a conditional input for GAN to synthesize sounds. SpecVQGAN [21] uses more powerful Transformer-based autoregressive models for sound synthesis with extracted ResNet50 or RGB+Flow features. Im2Wav [41], current state-of-the-art, adopts a two-transformer model, using CLIP [36] features as the condition, yet suffers from slow inference speed, requiring thousands of inference steps. Existing V2A methods struggle with synchronization as they lack audio-related information in pretrained visual features, limiting the capture of intricate audio-visual correlations. A notable subfield, video-to-speech, has gained attention with works like vid2speech [7], Neural Dubber [19]. However, the foley task aims to generate complex and synchronized audio from video, presenting much higher challenge.
Contrastive Pretraining Contrastive pretraining [34; 13; 4] has shown potential in various generative tasks. CLIP aligns text-image representations using contrastive pretraining. For Text-to-Image, CLIP is integrated with Stable Diffusion [38] to extract text prompt features for realistic image generation. For Text-to-Audio, CLAP [6] aligns text and audio representations, and the resulting audio features
Figure 2: Overview of DIFF-FOLEY: First, it learns more semantically and temporally aligned audio-visual features by CAVP, capturing the subtle connection between audio-visual modality. Second, a LDM conditioned on the aligned CAVP visual features is trained on the spectrogram latent space. DIFF-FOLEY can synthesize highly synchronized audio with strong audio-visual relevance.  $\mathcal{P}(\cdot)$  denotes temporal pooling layer.
are used in AudioLDM [27] for audio generation. Recognizing the importance of modality alignment in multimodal generation, we propose contrastive audio-visual pretraining (CAVP) to learn temporally and semantically aligned features initially and use them for subsequent generation tasks.
Latent Diffusion Model Diffusion Models [43; 16] have achieved remarkable success in various generative tasks including image generation [43; 16; 32; 37; 38], audio generation [25; 48; 27], and video generation [18; 15; 8]. Latent diffusion models, like Stable Diffusion (SD) [38] perform forward and reverse processes in data latent space, resulting in efficient computation and accelerated inference. SD is capable of running inference on personal laptop GPUs, making it a practical choice for a wide range of users.
DIFF-FOLEY consists of two stages: Stage1 CAVP and Stage2 LDM training. Our model overview is shown in Figure 2. Audio and visual components in a video exhibit strong correlation and complementarity. Unfortunately, existing image or optical flow backbones (ResNet, CLIP etc.) struggle to reflect the strong alignment relationship between audio and visual. To address this, we propose Contrastive Audio-Visual Pretraining (CAVP) to align audio-visual features at the outset. Then, an LDM conditioned on CAVP-aligned visual features is trained on the spectral latent space.
3.1 Contrastive Audio-Visual Pretraining
Given a audio-video pair  $(x_{a}, x_{v})$ , where  $x_{a} \in \mathbb{R}^{T' \times M}$  is a Mel-Spec. with  $M$  mel basis and  $T'$  is the time axis.  $x_{v} \in \mathbb{R}^{T'' \times 3 \times H \times W}$  is a video clip with  $T''$  frames. An audio encoder  $f_{A}(\cdot)$  and a video encoder  $f_{V}(\cdot)$  are used to extract audio feature  $E_{a} \in \mathbb{R}^{T \times C}$  and video feature  $E_{v} \in \mathbb{R}^{T \times C}$  with same temporal dim  $T$ . We adopt the design of the audio encoder from PANNs [24], and SlowOnly [9] architecture for the video encoder. Using temporal pooling layer  $P(\cdot)$ , we obtain temporal-pooled audio/video features,  $\bar{E}_{a} = P(E_{a}) \in \mathbb{R}^{C}, \bar{E}_{v} = P(E_{v}) \in \mathbb{R}^{C}$ . We then use a similar contrastive objective in CLIP [36] to contrast  $\bar{E}_{a}$  and  $\bar{E}_{v}$ . To improve the semantic and temporal alignment of audio-video features, we use two objectives: Semantic contrast  $\mathcal{L}_{S}$  and Temporal contrast  $\mathcal{L}_{\mathcal{T}}$ .
For  $\mathcal{L}_S$ , we maximize the similarity of audio-visual pairs from the same video and minimize the similarity of audio-visual pairs from different videos. It encourages learning semantic alignment for audio-visual pairs across different videos. In specific, we extract audio-visual features pairs from different videos,  $\mathcal{B}_S = \{(\bar{E}_a^i,\bar{E}_v^i)\}_{i = 1}^{N_S}$ , where  $N_{S}$  is the number of different videos. We define the per-sample pair semantic contrast objective:  $\mathcal{L}_S^{(i,j)}$ , where  $sim(\cdot)$  is the cosine similarity.
$$
\mathcal {L} _ {S} ^ {(i, j)} = - \frac {1}{2} \log \frac {\exp \left(s i m \left(\bar {E} _ {a} ^ {i} , \bar {E} _ {v} ^ {j}\right) / \tau\right)}{\sum_ {k = 1} ^ {N _ {S}} \exp \left(s i m \left(\bar {E} _ {a} ^ {i} , \bar {E} _ {v} ^ {k}\right) / \tau\right)} - \frac {1}{2} \log \frac {\exp \left(s i m \left(\bar {E} _ {a} ^ {i} , \bar {E} _ {v} ^ {j}\right) / \tau\right)}{\sum_ {k = 1} ^ {N _ {S}} \exp \left(s i m \left(\bar {E} _ {a} ^ {k} , \bar {E} _ {v} ^ {j}\right) / \tau\right)} \tag {1}
$$
For  $\mathcal{L}_{\mathcal{T}}$ , we sample video clips at different times within the same video. It aims to maximize the similarity of audio-visual pairs from the same time segment and minimize the similarity of audio-visual pairs across different time segments. In details, we sample different time segments in the
same video to extract audio-visual feature pairs.  $\mathcal{B}_{\mathcal{T}} = \{(\bar{E}_a^i,\bar{E}_v^i)\}_{i = 1}^{N_T}$ , where  $N_{T}$  is the number of sampled video clip within the same video. We define the per-sample pair temporal contrast objective:
$$
\mathcal {L} _ {\tau} ^ {(i, j)} = - \frac {1}{2} \log \frac {\exp \left(s i m \left(\bar {E} _ {a} ^ {i} , \bar {E} _ {v} ^ {j}\right) / \tau\right)}{\sum_ {k = 1} ^ {N _ {T}} \exp \left(s i m \left(\bar {E} _ {a} ^ {i} , \bar {E} _ {v} ^ {k}\right) / \tau\right)} - \frac {1}{2} \log \frac {\exp \left(s i m \left(\bar {E} _ {a} ^ {i} , \bar {E} _ {v} ^ {j}\right) / \tau\right)}{\sum_ {k = 1} ^ {N _ {T}} \exp \left(s i m \left(\bar {E} _ {a} ^ {k} , \bar {E} _ {v} ^ {j}\right) / \tau\right)} \tag {2}
$$
The final objective is the weighted sum of semantic and temporal objective:  $\mathcal{L} = \mathcal{L}_S + \lambda \mathcal{L}_T$ , where  $\lambda = 1$ . After training, CAVP encodes an audio-video pair into embedding pair:  $(x_{a}, x_{v}) \to (E_{a}, E_{v})$  where  $(E_{a}, E_{v})$  are highly aligned, with the visual features  $E_{v}$  containing rich information for audio. The aligned and strongly correlated features  $E_{v}$  and  $E_{a}$  facilitate subsequent audio generation.
3.2 LDM with Aligned Visual Representation
LDMs [38] are probabilistic models that fit the data distribution  $p(x)$  by denoising on data latent space. LDMs first encode the origin high-dim data  $x$  into low-dim latent  $z = \mathcal{E}(x)$  for efficient training. The forward and reverse process are performed in the compressed latent space. In V2A generation, our goal is to generate synchronized audio  $x_{a}$  given video clip  $x_{v}$ . Using similar latent encoder  $\mathcal{E}_{\theta}$  in [38], we compress Mel-Spec  $x_{a}$  into a low-dim latent  $z_{0} = \mathcal{E}_{\theta}(x_{a}) \in \mathbb{R}^{C^{\prime} \times \frac{T^{\prime}}{r} \times \frac{M}{r}}$ , where  $r$  is the compress rate. With the pretrained CAVP model to align audio-visual features, the visual features  $E_{v}$  contain rich audio-related information. This enables synthesis of highly synchronized and relevant audio using LDMs conditioned on  $E_{v}$ . We adopt a projection and positional encoding layer  $\tau_{\theta}$  to project  $E_{v}$  to the appropriate dim. In the forward process, origin data distribution transforms into standard Gaussian distribution by adding noise gradually with a fixed schedule  $\alpha_{1}, \ldots, \alpha_{T}$ , where  $T$  is the total timesteps, and  $\bar{\alpha}_{t} = \prod_{i=1}^{t} \alpha_{i}$ .
$$
q \left(z _ {t} \mid z _ {t - 1}\right) = \mathcal {N} \left(z _ {t}; \sqrt {\bar {\alpha} _ {t}} z _ {t - 1}, (1 - \alpha_ {t}) \mathbf {I}\right), \quad q \left(z _ {t} \mid z _ {0}\right) = \mathcal {N} \left(z _ {t}; \sqrt {\bar {\alpha} _ {t}} z _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right) \tag {3}
$$
The goal of LDM is to mirror score matching by optimizing the denoising objective [46; 16; 44]:
$$
\mathcal {L} _ {L D M} = \mathbb {E} _ {z _ {0}, t, \epsilon} \| \epsilon - \epsilon_ {\theta} (z _ {t}, t, E _ {v}) \| _ {2} ^ {2} \tag {4}
$$
After LDM is trained, we generate audio latent by sampling through the reverse process with  $z_{T} \sim \mathcal{N}(0,\mathbf{I})$ , conditioned on the given visual-features  $E_{v}$ , with the following reverse dynamics:
$$
p _ {\theta} \left(z _ {t - 1} \mid z _ {t}\right) = \mathcal {N} \left(z _ {t - 1}; \mu_ {\theta} \left(z _ {t}, t, E _ {v}\right), \sigma_ {t} ^ {2} \mathbf {I}\right) \tag {5}
$$
$$
\mu_ {\theta} (z _ {t}, t, E _ {v}) = \frac {1}{\sqrt {\alpha_ {t}}} \left(z _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} (z _ {t}, t, E _ {v})\right), \quad \sigma_ {t} ^ {2} = \frac {1 - \bar {\alpha} _ {t - 1}}{1 - \bar {\alpha} _ {t}} (1 - \alpha_ {t}) \qquad (6)
$$
Finally, the Mel-Spec.  $\hat{x}_a$  is obtained by decoding the generated latent  $z_0$  with a decoder  $\mathcal{D}$ ,  $\hat{x}_a = \mathcal{D}(z_0)$ . In the case of DIFF-FOLEY, it generates audio samples with a duration of 8 seconds.
3.3 Temporal Split & Merge Augmentation
Using large-scale text-image pairs datasets like LAION-5B [40] is crucial for the success of current T2I models [38]. However, for V2A generation task, large-scale and high-quality datasets are still lacking. Further, we expect V2A model to generate highly synchronized audio based on visual content, such temporal audio-visual correspondence requires a large amount of audio-visual pairs for training. To overcome this limitation, we propose using Temporal Split & Merge Augmentation, a MixUp [49] like augmentation strategy to facilitate model training by incorporating prior knowledge for temporal alignment into the training process. During training, we randomly extract video clips of different time lengths from two videos (Split), denoted as  $(x_{a}^{1}, x_{v}^{1})$ ,  $(x_{a}^{2}, x_{v}^{2})$ , and extract visual features  $E_{v}^{1}$ ,  $E_{v}^{2}$  with pretrained CAVP model. We then create a new audio-visual feature pair for LDM training with:
$$
z _ {a} ^ {\text {n e w}} = \mathcal {E} _ {\theta} ([ x _ {a} ^ {1}; x _ {a} ^ {2} ]) \quad , \quad E _ {v} ^ {\text {n e w}} = [ E _ {v} ^ {1}; E _ {v} ^ {2} ], \tag {7}
$$
where  $[\cdot ;\cdot ]$  represent temporal concatenation (Merge). Split and merge augmentation greatly increase the number of audio-visual pairs, preventing overfitting and facilitating LDM to learn temporal correspondence. We validate the effectiveness of this augmentation method in Sec 4.1.2.
MODEL	VISUAL FEATURE FPS	GUIDANCE	METRICS	INFER. TIME↓
IS ↑	FID ↓	KL ↓	ACC (%) ↑
SpecVQGAN [21]	RGB + Flow	21.5	X	30.01	8.93	6.93	52.94	5.47s
SpecVQGAN [21]	ResNet50	21.5	X	30.80	9.70	7.03	49.19	5.47s
Im2Wav [41]	CLIP	30	CFG (✓)	39.30	11.44	5.20	67.40	6.41s
DIFF-FOLEY (Ours)	CAVP	4	CFG (✓)	53.34	11.22	6.36	92.67	0.38s
DIFF-FOLEY (Ours)	CAVP	4	Double (✓✓)	62.37	9.87	6.43	94.05	0.38s
Table 1: Video-to-Audio generation evaluation results with CFG scale  $\omega = 4.5$ , CG scale  $\gamma = 50$ , using DPM-Solver [30] Sampler with 25 inference steps. DIFF-FOLEY achieves impressive temporal synchronization and audio-visual relevance (Align Acc) with only using 4 FPS video, compared with baseline method using 21.5/30 FPS. CFG represents Classifier-Free Guidance, Double represents Double Guidance, and ACC refers to Align Acc. INFER. TIME denotes the average inference time per sample when generating 64 samples in a batch.
MODEL	VISUAL FEATURE	FPS	GUIDANCE	HUMAN EVAL. METRICS
Content Relevance	Synchronization
SpecVQGAN [21]	ResNet50	21.5	x	46.20	45.20
Im2wav [41]	CLIP	30	CFG (✔)	62.13	57.73
Diff-Foley (Ours)	CAVP	4	CFG (✔)	71.73	71.00
Diff-Foley (Ours)	CAVP	4	Double (✔✔)	74.53	74.93
Ground Truth	84.80	84.20
Table 2: Video-to-Audio generation human evaluation results with CFG scale  $\omega  = {4.5}$  ,CG scale  $\gamma  = {50}$  . Raters are required to score the generated audio based on content relevance and synchronization. DIFF-FOLEY with double guidance shows superiority performance compared with baseline methods.
3.4 Double Guidance
Guidance techniques is widely used in diffusion model reverse process for controllable generation. There are currently two main types of guidance techniques: classifier guidance [5] (CG), and classifier-free guidance [17] (CFG). For CG, it additionally trains a classifier (e.g class-label classifier) to guide the reverse process at each timestep with gradient of class label loglikelihood  $\nabla_{x_t}\log p_\phi (y|x_t)$ . For CFG, it does not require an additional classifier, instead it guides the reverse process by using linear combination of the conditional and unconditional score estimates [17], where the  $c$  is the condition and  $\omega$  is the guidance scale. In stable diffusion [38], the CFG is implemented as:
$$
\tilde {\epsilon} _ {\theta} \left(z _ {t}, t, c\right) \leftarrow \omega \epsilon_ {\theta} \left(z _ {t}, t, c\right) + (1 - \omega) \epsilon_ {\theta} \left(z _ {t}, t, \varnothing\right). \tag {8}
$$
When  $\omega = 1$ , CFG degenerates to conditional score estimates. Although CFG is currently the mainstream approach used in diffusion models, the CG method offers the advantage of being able to guide any desired property of the generated samples given true label. In V2A setting, the desired property refers to semantic and temporal alignment. Moreover, we discover that these two methods are not mutually exclusive. We propose a double guidance technique that leverages the advantages of both CFG and CG methods by using them simultaneously at each timestep in the reverse process. In specific, for CG we train an alignment classifier  $P_{\phi}(y|z_t,t,E_v)$  that predicts whether an audiovisual pair is a real pair in terms of semantic and temporal alignment. For CFG, during training, we randomly drop condition  $E_{v}$  with prob.  $20\%$  , to train conditional and unconditional likelihood  $\epsilon_{\theta}(z_t,t,E_v),\epsilon_{\theta}(z_t,t,\varnothing)$  . Then double guidance is achieved by improved noise estimation:
$$
\hat {\epsilon} _ {\theta} \left(z _ {t}, t, E _ {v}\right) \leftarrow \omega \epsilon_ {\theta} \left(z _ {t}, t, E _ {v}\right) + (1 - \omega) \epsilon_ {\theta} \left(z _ {t}, t, \varnothing\right) - \gamma \sqrt {1 - \bar {\alpha} _ {t}} \nabla_ {z _ {t}} \log P _ {\phi} \left(y | z _ {t}, t, E _ {v}\right), \tag {9}
$$
where  $\omega, \gamma$  refer to CFG, CG guidance scale respectively. We provide further analysis of the intuition and mechanism behind double guidance in Appendix C.
Datasets We use two datasets VGGSound [2] and AudioSet [10]. VGGSound consists of  $\sim 200\mathrm{K}$  10-second videos. We follow the original VGGSound train/test splits. AudioSet comprises 2.1M videos with 527 sound classes, but it is highly imbalanced, with most of the videos labeled as Music and Speech. Since generating meaningful speech directly from video is not expected in V2A tasks (not necessary either), we download a subset of the Music tag data and all other tags except Speech, resulting in a new dataset named AudioSet-V2A with about  $80\mathrm{K}$  music-tagged videos and  $310\mathrm{K}$
Input Silent Video: Playing Golf
Figure 3: Video-to-Audio generation results on VGGSound: Given a silent playing golf video, only the DIFF-FOLEY successfully generates the corresponding hitting sound at the 4th second timestamp, showcasing its remarkable ability to generate synchronized audio based on video content.
other tagged videos. We use VGGSound and AudioSet-V2A for Stage1 CAVP, while for Stage2 LDM training and evaluation, we only use VGGSound, which is consistent with the baseline.
Data Pre-Processing For DIFF-FOLEY training, videos in VGGSound and AudioSet-V2A are sampled at 4 FPS, which already yields significantly better alignment results compared to the baseline method [21; 41] using 21.5 or 30 FPS. Each 10-second video sample consists of 40 frames and resized to  $224 \times 224$ . For audio data, the original audio is sampled at  $16\mathrm{kHz}$  and transformed into Mel-spectrograms (Mel Basis  $M = 128$ ). For Stage1 Pretraining, we use hop size 250 for better audio-visual data temporal dimension alignment, while we use hop size 256 for Stage2 LDM training.
Model Configuration and Inference For CAVP, we use a pretrained audio encoder from PANNs [24] and a pretrained SlowOnly [9] based video encoder. For training, we randomly extract 4-second audio-video frames pairs from 10-second samples, resulting in  $x_{a} \in \mathbb{R}^{256 \times 128}$  and  $x_{v} \in \mathbb{R}^{16 \times 3 \times 224 \times 224}$ . We use temporal contrast  $\mathcal{L}_{\mathcal{T}}$  with  $N_{T} = 3$  and a minimum time difference of 2 seconds between each pair. For LDM training, we utilize the pretrained Stable Diffusion-V1.4 (SD-V1.4) [38] as a powerful denoising prior model. Leveraging the pretrained SD-V1.4 greatly reduces training time and improves generation quality (discussed in Sec 4.3). Frozen pretrained latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  from SD-V1.4 are used. Interestingly, despite being trained on image datasets (LAION-5B), we found  $\mathcal{E}, \mathcal{D}$  can encode and decode Mel-Spec well. For inference, we use DPM-Solver [30] sampler with 25 sampling steps, unless otherwise stated. More training details for each model are in Appendix. A.