SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alignment Classifier for Align Acc Metric. To evaluate the generated audio synchronization and audio-visual relevance, we introduce Alignment Accuracy (Align Acc.) as a new metric in Sec. 4. In specific, we train an alignment classifier  $F_{\theta}$  to predict real audio-visual pairs. To train the classifier, we use three types of audio-visual pairs. (1).  $50\%$  of the pairs are the real audio-visual pairs from the same video and the same time (true pair), labeled as 1; (2).  $25\%$  are audio-visual pairs from the same video but temporally shifted, resulting in an out-of-sync audio-video pair (temporal shift pair), labeled as 0. (3). The remaining  $25\%$  are audio-visual pairs from different videos, resulting in completely mismatched audio-visual pairs (wrong pair), labeled as 0. This dataset comprises of  $50\%$  aligned audio-visual pairs and  $50\%$  unaligned audio-visual pairs (temporal and semantic). By training a classifier on this audio-visual pairs dataset, we can evaluate both audio-visual synchronization and audio-visual relevance using the alignment probability predicted by the alignment classifier  $F_{\theta}$ . We use the VGGSound [2] datasets, and our alignment classifier  $F_{\theta}$  achieves  $88.31\%$  accuracy on test set. The detailed metrics of our alignment classifier is Recall:  $84.92\%$ , Precision:  $91.32\%$ , and Accuracy:  $88.31\%$ . For the model architecture of the alignment classifier  $F_{\theta}^{sync}$ , we adopt the lightweight U-Net encoder block design in SD-V1.4. The basic channel number is set to  $c = 128$ , and the channel dimensions of the encoder blocks are  $[c, 2c, 2c]$ . Cross-attention is applied to latents with down-sampling rates of [2, 4] and the model has  $\sim 12M$  parameters. The alignment classifier  $F_{\theta}^{sync}$  takes two input. The encoded spectrogram latent  $z_0$ , and the visual aligned features  $E_v$  from Stage1 CAVP. The predicted alignment label  $\hat{y}$  is computed as follows:  $\hat{y} = F_{\theta}(z_0, E_v)$ .
Here, we emphasize that the alignment classifier  $F_{\theta}^{sync}$  for align acc metric is different from the sync classifier  $F_{\phi}^{DG}$  used in double guidance techniques. When utilizing the double guidance techniques, we did not access the alignment classifier that used for accuracy measurement. In specific, the classifier  $F_{\phi}^{DG}$  used in double guidance is a noisy classifier, and it takes the noisy latent  $z_{t}$ , time embedding  $t$ , and visual aligned features  $E_{v}$  as input, the predicted alignment label  $\hat{y}$  is computed as follows:  $\hat{y} = F_{\phi}^{DG}(z_{t}, t, E_{v})$ . While the sync classifier  $F_{\theta}^{sync}$  for accuracy measurement, it only takes two input, latent  $z_{0}$  and visual aligned features  $E_{v}$  to predict the synchronization label with  $\hat{y} = F_{\theta}^{sync}(z_{0}, E_{v})$ .
A.2 Data Processing
In this section, we provide additional information on data processing for reproducibility.
Video-to-Audio Generation. To train Stage1 CAVP, we use the VGGSound [2] and AudioSet-V2A [10] datasets. The videos are sampled at 4 FPS, resulting in 40 frames for each 10-second video sample. The frames are then resized to  $224 \times 224$ . For audio data, the audios are sampled at  $16\mathrm{kHz}$  and then transformed into Mel-spectrograms, using FFT Num 1024, mel basis Num 128 and hop size 250. This results in a 10-second spectrogram with the size of  $640 \times 128$  ( $T' \times M$ ). We use a hop size of
250 in CAVP for better audio-visual data temporal dimension alignment. To train CAVP, we randomly extract 4-second audio-video frame pairs from the 10-second samples, resulting in  $x_{a} \in \mathbb{R}^{256 \times 128}$  and  $x_{v} \in \mathbb{R}^{16 \times 3 \times 224 \times 224}$ . We then pass the audio-visual pairs through the audio encoder  $f_{A}(\cdot)$  and video encoder  $f_{V}(\cdot)$  respectively. This results in  $E_{a} \in \mathbb{R}^{16 \times 512}$  and  $E_{v} \in \mathbb{R}^{16 \times 512}$ . In Stage2 LDM training, we only use the VGGSound datasets, which is consistent with the baseline setting. We use  $\sim 8$  second audio and video pairs for training LDM. In specific, we use FFT Num 1024, mel basis Num 128, and hop size 256 to transform the  $\sim 8$  seconds audios into Mel-spectrograms with the size  $512 \times 128$  ( $T' \times M$ ), resulting in  $x_{a} \in \mathbb{R}^{512 \times 128}$  and  $x_{v} \in \mathbb{R}^{32 \times 3 \times 224 \times 224}$ . The LDMs take the encoded spectrogram noisy latent  $z_{t} \in \mathbb{R}^{4 \times 16 \times 64}$ , time embedding  $t$ , and CAVP-aligned visual features  $E_{v} \in \mathbb{R}^{32 \times 512}$  as input:  $\epsilon_{\theta}(z_{t}, t, E_{v})$ .
Downstream Finetuning. For downstream finetuning, we utilized the high-quality egocentric video dataset, EPIC-Kitchens [20], which contains approximately 100 hours of daily activities and object interactions in kitchens with minimal sound noise. The sound categories in EPIC-Kitchens are significantly different from those in VGGSound [2]. For dataset processing, we randomly divided the original video dataset into training and testing sets, with  $90\%$  of the total duration assigned to the training set and  $10\%$  to the testing set. We split the original videos into 10-second segments for each dataset, resulting in about 26k training samples and 2.9k testing samples. It's important to note that the video clips in the training and testing sets were sourced from different videos and do not overlap. The processing steps for the datasets were the same as those described in Sec. A.2. Notably, we did not finetune the Stage1 CAVP models, but instead directly adopted the pretrained CAVP model to extract aligned visual features for EPIC-Kitchens videos. Subsequently, we finetune the Stage2 LDM, which was originally trained on VGGSound, using pairs of extracted CAVP visual features and audio from EPIC-Kitchens datasets.
A.3 Training Details
CAVP Training We train the CAVP model for 1.4M steps on 8 A100 GPUs, with a total batch size of 720 using automatic mixed-precision training (AMP). We used the AdamW [29] optimizer with a learning rate of 8e-4 and 200 steps of warmup. The total training time for the CAVP model was approximately 80 hours.
LDM Training We initialize the LDM model with pretrained weights from SD-V1.4 [38] and train LDM for 24.4K steps on 8 A100 GPUs with a total batch size of 1760. We used the AdamW [29] optimizer with a learning rate of 1e-4 and 1000 steps of warmup. The total training time for DIFF-FOLEY-L is approximately 60 hours.
Downstream Finetuning For downstream finetuning, we finetune the DIFF-FOLEY-L model (trained on VGGSound) on EPIC-Kitchens for 3.2K steps on 8 A100 GPUs with a total batchsize 1760. We use the AdamW [29] optimizer with the learning rate 1e-4 and 1000 steps of warmup. The total downstream finetuning time is around 9 hours.
For metric evaluation, we use Inception Score (IS), Frechet Distance (FID) and Mean Kullback-Leibler Divergence (MKL) from [21]. For comparison, we use two current state-of-the-art V2A models: SpecVQGAN [21] and Im2Wav [41]. We use the pre-trained models provided by the authors, and these models are both trained on VGGSound [2] datasets. SpecVQGAN [21] utilizes videos with 21.5 FPS to generate 10-second audio samples conditioned on 10-second video features, while Im2Wav [41] utilizes videos with 30 FPS to generate 4-second audio samples conditioned on 4-second video features. In contrast, our DIFF-FOLEY generates 8-second audio conditioned on 8-second video features. To ensure a fair comparison, for SpecVQGAN, we generate 10s audio samples and truncate them to the first 8 seconds. For Im2Wav, we generate the first 4 seconds of audio based on the features extracted from the first 4 seconds of the video, followed by generating the (4s-8s) audio conditioned on the (4s-8s) video features. We then merge the two audio segments to obtain the final 8 seconds audio output. Lastly, we compare the generated 8-second audio samples with the ground truth Mel-spectrogram. Since our generated Mel-spectrogram has a Mel basis num of 128, while the baseline ground truth Mel-spectrogram has a Mel basis num of 80, we transform our generated Mel-spectrogram into 80 Mel basis num for a fair comparison. For each video sample in the test set, we generate 10 audio samples for evaluation, resulting in around  $145\mathrm{K}$  generated audio
samples. For human evaluation, we conducted a human evaluation by randomly selecting 60 videos from the VGGSound test set and having different models to generate corresponding audio samples. The output and ground truth audios were anonymized and rated by 30 people unfamiliar with the project. Each sample received scores from at least 5 raters, ranging from 1 (bad) to 5 (excellent) for content relevance and synchronization. The scores were then scaled by a factor of 20.
B More Generation Results on Video-to-Audio Generation
B.1 V2A Generation Results on VGGSound.
We present additional generation results for Video-to-Audio generation on VGGSound dataset in Figure 7. We observe that DIFF-FOLEY effectively generates the corresponding sounds (e.g., gunshots and underwater bubbling) at the appropriate time, which matches the ground truth audio. In contrast, other methods fail to generate highly synchronized audio that aligns with the video content. This supports the superior performance of our approach. For more details, please visit our demo websites.
B.2 Downstream Finetuning Results on EPIC-Kitchens.
We have included additional downstream finetuning results on the EPIC-Kitchens dataset in Figure 8. Our observations indicate that DIFF-FOLEY generates highly synchronized audio that effectively captures the timing of sounds with the original video. In the first video, DIFF-FOLEY produces the corresponding sound of opening drawer at both the 1st, 3rd seconds. In the second video, it successfully generates the sound of plates clinking and placing at both the 1st, 4th seconds. For more details, please visit our demo websites. These results demonstrate the potential of DIFF-FOLEY to adapt to other video datasets, making it easier and more promising for future users to generate specific types of audio data through downstream finetuning.
C Guidance Techniques
C.1 Guidance Techniques Overview
Guidance techniques are commonly utilized in the reverse process of the diffusion model to achieve controllable generation. Currently, there are two main types of guidance techniques: Classifier Guidance [5] (CG) and Classifier-Free Guidance [17] (CFG). For CG, it additionally trains a classifier  $p_{\phi}$  (e.g. class-label  $y$  classifier) to guide the reverse process at each timestep using the gradient of the class label loglikelihood  $\nabla_{x_t} \log p_{\phi}(y|x_t)$ , resulting in the conditional score estimates [5]:
$$
C G: \quad \tilde {\epsilon} _ {\theta} \left(x _ {t}, t\right) \leftarrow \epsilon_ {\theta} \left(x _ {t}, t\right) - \sqrt {1 - \bar {\alpha} _ {t}} \gamma \nabla_ {x _ {t}} \log p _ {\phi} (y | x _ {t}) \tag {10}
$$
, where  $\gamma$  is the CG scale. For CFG, it does not require an additional classifier, instead it guides the reverse process by using linear combination of the conditional and unconditional score estimates [17], where the  $c$  is the condition and  $\omega$  is the CFG scale.
$$
C F G: \quad \tilde {\epsilon} _ {\theta} \left(x _ {t}, t, c\right) \leftarrow \omega \epsilon_ {\theta} \left(x _ {t}, t, c\right) + (1 - \omega) \epsilon_ {\theta} \left(x _ {t}, t, \varnothing\right) \tag {11}
$$
When  $\omega = 1$ , CFG degenerates to conditional score estimate. Although CFG is currently the mainstream approach used in diffusion models, the CG method offers the advantage of being able to guide any desired property of the generated samples given true label  $y$ . For instance, if we want to generate images with specific attributes (e.g. a girl with yellow hair) from a pretrained image diffusion model, we can simply train a 'girl with yellow hairs' classifier to guide the generation process. In V2A setting, the desired property refers to audio semantic and temporal alignment.
In Sec. 3.4, we discover that the CG and CFG methods are not mutually exclusive. We then propose a double guidance (DG) technique that leverages the advantages of both CFG and CG methods by using them simultaneously at each timestep in the reverse process. In specific, for CG we train an noisy alignment classifier  $P_{\phi}(y|z_t,t,E_v)$  that predict whether an audio-visual pair is a real pair in terms of semantic and temporal alignment. The noisy alignment classifier  $P_{\phi}(y|z_t,t,E_v)$  takes noisy latent  $z_{t}$ , time embedding  $t$ , and the CAVP-aligned visual features  $E_{v}$  as input. We follow similar training process in Sec. A.1 (see Align Acc.) to train the noisy alignment classifier  $P_{\phi}(y|z_t,t,E_v)$ . For CFG, during training, we randomly drop condition  $E_{v}$  with probability  $20\%$  to estimate the conditional
Input Silent Video: Gun Shooting
Input Silent Video: Underwater Bubbling
Figure 7: More V2A results on VGGSound: In the first gun shooting video, we see that only the DIFF-FOLEY generate the corresponding gun shooting sound at the appropriate time given the video content (see the 4th, 6th second.), while other methods fail to generate the corresponding gun shooting sound. In the second underwater bubbling video, DIFF-FOLEY produces highly realistic audio that captures the transition from above to underwater (see the 4th second), while other methods fail to capture this transition. For more details, please visit our demo websites.
score  $\epsilon_{\theta}(z_t,t,E_v)$  and unconditional score  $\epsilon_{\theta}(z_t,t,\varnothing)$ . Then double guidance is achieved by the improved noise estimation:
$$
\hat {\epsilon} _ {\theta} \left(z _ {t}, t, E _ {v}\right) \leftarrow \omega \epsilon_ {\theta} \left(z _ {t}, t, E _ {v}\right) + (1 - \omega) \epsilon_ {\theta} \left(z _ {t}, t, \varnothing\right) - \gamma \sqrt {1 - \bar {\alpha} _ {t}} \nabla_ {z _ {t}} \log P _ {\phi} \left(y | z _ {t}, t, E _ {v}\right) \tag {12}
$$
where  $\omega, \gamma$  refer to CFG, CG guidance scale respectively.
OpenDrawer
Get Plate from the Cupboard
Figure 8: More downstream finetuning results on EPIC-Kitchens: DIFF-FOLEY accurately generates the sound of opening drawers at the 1st, 3rd seconds in the first video, and produces the sound of plates clinking and placing at the 1st, 4th seconds in the second video, showing the potential of DIFF-FOLEY on downstream finetuning. For more details, please visit our demo websites.
C.2 Theoretical Perspective of Double Guidance
Although the approach of double guidance may appear heuristic, we provide a theoretical perspective for explaining the underlying mechanism behind this techniques. The goal of guidance techniques is to generate sample from the conditional distribution:  $p(z_{t}|y)$ , where  $z_{t}$  is the generated noisy sample at each reverse process timesteps and the  $y$  is the condition or specific attribute. In diffusion models, the  $\epsilon_{\theta}(z_{t},y,t)$  estimate the conditional score function, which is  $\nabla_{z_t}\log p(z_t|y)$ .
For CG, we first derive the mechanism behind CG [5] method to show how CG help in controllable generation. With the bayes's formula, we have:
$$
\begin{array}{l} p (z _ {t} | y) = \frac {p (y | z _ {t}) \cdot p (z _ {t})}{p (y)} \\ \Rightarrow \quad \log p \left(z _ {t} \mid y\right) = \log p \left(y \mid z _ {t}\right) + \log p \left(z _ {t}\right) - \log p (y) \tag {13} \\ \Rightarrow \quad \nabla_ {z _ {t}} \log p (z _ {t} | y) = \nabla_ {z _ {t}} \log p (y | z _ {t}) + \nabla_ {z _ {t}} \log p (z _ {t}) \\ \end{array}
$$
, where the  $\nabla_{z_t}\log p(y|z_t)$  is actually the classifier guidance (CG) term. To enhance the conditional singal  $y$ , we scale the  $\nabla_{z_t}\log p(y|z_t)$  term by a factor  $\gamma$ , also known as CG scale, then we obtain the CG improved score estimation in Equation 10.
$$
\begin{array}{l} \nabla_ {z _ {t}} \log p _ {\gamma} (z _ {t} | y) = \nabla_ {z _ {t}} \log p (z _ {t}) + \gamma \nabla_ {z _ {t}} \log p (y | z _ {t}) (14) \\ \Longrightarrow \quad (C G): \quad \tilde {\epsilon} _ {\theta} (z _ {t}, y, t) \leftarrow \epsilon_ {\theta} (z _ {t}, t) - \sqrt {1 - \bar {\alpha} _ {t}} \gamma \nabla_ {z _ {t}} \log p _ {\phi} (y | z _ {t}) (15) \\ \end{array}
$$
, where  $\nabla_{z_t}\log p(z_t) = -\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\epsilon_\theta (z_t,t)$  [16].
For CFG, we also derive the mechanism behind CFG [17] method. To improve the conditional signal  $c$ , CFG use the mixture of conditional score  $\nabla_{z_t}\log p(z_t|c)$  and unconditional score  $\nabla_{z_t}\log p(z_t)$  to enhance the term  $\nabla_{z_t}\log p(c|z_t)$ . Using the bayes formula, we have:
$$
p (c | z _ {t}) = \frac {p (z _ {t} | c) \cdot p (c)}{p (z _ {t})}
$$
$$
\begin{array}{l} \Rightarrow \quad \log p (c | z _ {t}) = \log p \left(z _ {t} \mid c\right) + \log p (c) - \log p \left(z _ {t}\right) \tag {16} \\ \Longrightarrow \quad \nabla_ {z _ {t}} \log p (c | z _ {t}) = \nabla_ {z _ {t}} \log p (z _ {t} | c) - \nabla_ {z _ {t}} \log p (z _ {t}) \\ \end{array}
$$
, then we scale the  $\nabla_{z_t}\log p(c|z_t)$  with a factor of  $\omega$  (CFG scale), and substitute to Equation 14, then we obtain the CFG improved score estimation in Equation 11.
$$
\begin{array}{l} \nabla_ {z _ {t}} \log p _ {\omega} (z _ {t} | c) = \nabla_ {z _ {t}} \log p (z _ {t}) + \omega (\nabla_ {z _ {t}} \log p (z _ {t} | c) - \nabla_ {z _ {t}} \log p (z _ {t})) (17) \\ \Longrightarrow \quad (C F G): \quad \tilde {\epsilon} _ {\theta} \left(z _ {t}, t, c\right) \leftarrow \omega \epsilon_ {\theta} \left(z _ {t}, t, c\right) + (1 - \omega) \epsilon_ {\theta} \left(z _ {t}, t, \varnothing\right) (18) \\ \end{array}
$$
, where  $\nabla_{z_t}\log p(z_t) = -\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\epsilon_\theta (z_t,t,\emptyset)$  and  $\nabla_{z_t}\log p(z_t|c) = -\frac{1}{\sqrt{1 - \bar{\alpha}_t}}\epsilon_\theta (z_t,t,c)$ .
For double guidance (DG), we construct a more fine-grained and controllable conditional distribution  $p(z_{t}|c,y)$  that takes into account two conditions,  $c$  and  $y$ , where  $c$  is the general condition and  $y$  is another data attribute label. Specifically, in DIFF-FOLEY,  $c$  refers to the CAVP-aligned visual features  $E_{v}$  (general condition), while  $y$  refers to a more fine-grained and desired property, such as semantic and temporal alignment (data attribute). DIFF-FOLEY aims to generate high-quality synchronized audio with strong audio-visual relevance from this distribution  $z_{t} \sim p(z_{t}|c,y)$ . To simplify the derivation, we assume that  $p(y|z_{t})$  and  $p(c|z_{t})$  are independent conditioned on  $z_{t}$ . Using the Bayes formula, we have:
$$
p (z _ {t} | c, y) = \frac {p (c , y | z _ {t}) \cdot p (z _ {t})}{p (c , y)} = \frac {p (c | z _ {t}) \cdot p (y | z _ {t}) \cdot p (z _ {t})}{p (c , y)}
$$
$$
\begin{array}{l} \Rightarrow \quad \log p \left(z _ {t} \mid c, y\right) = \log p \left(c \mid z _ {t}\right) + \log p \left(y \mid z _ {t}\right) + \log p \left(z _ {t}\right) - \log p (c, y) \tag {19} \\ \Longrightarrow \quad \nabla_ {z _ {t}} \log p (z _ {t} | c, y) = \nabla_ {z _ {t}} \log p (c | z _ {t}) + \nabla_ {z _ {t}} \log p (y | z _ {t}) + \nabla_ {z _ {t}} \log p (z _ {t}) \\ \end{array}
$$
, we then use CG scale  $\gamma$  and CFG scale  $\omega$  to scale the CG term  $\nabla_{z_t}\log p(y|z_t)$  and CFG term  $\nabla_{z_t}\log p(c|z_t)$  respectively:
$$
\begin{array}{l} \nabla_ {z _ {t}} \log p _ {\omega , \gamma} (z _ {t} | c, y) = \omega \nabla_ {z _ {t}} \log p (c | z _ {t}) + \gamma \nabla_ {z _ {t}} \log p (y | z _ {t}) + \nabla_ {z _ {t}} \log p (z _ {t}) \\ = \omega (\nabla_ {z _ {t}} \log p (z _ {t} | c) - \nabla_ {z _ {t}} \log p (z _ {t})) + \gamma \nabla_ {z _ {t}} \log p (y | z _ {t}) + \nabla_ {z _ {t}} \log p (z _ {t}) \\ = \omega \nabla_ {z _ {t}} \log p (z _ {t} | c) + (1 - \omega) \nabla_ {z _ {t}} \log p (z _ {t}) + \gamma \nabla_ {z _ {t}} \log p (y | z _ {t}) \tag {20} \\ \end{array}
$$
Finally, we obtain the double guidance (DG) improved score estimation in Equation 12.
$$
(D G): \quad \hat {\epsilon} _ {\theta} \left(z _ {t}, t, E _ {v}\right) \leftarrow \omega \epsilon_ {\theta} \left(z _ {t}, t, E _ {v}\right) + (1 - \omega) \epsilon_ {\theta} \left(z _ {t}, t, \emptyset\right) - \gamma \sqrt {1 - \bar {\alpha} _ {t}} \nabla_ {z _ {t}} \log P _ {\phi} \left(y | z _ {t}, t, E _ {v}\right) \tag {21}
$$
, where  $c$  is the CAVP-aligned visual features  $E_{v}$ ,  $y$  is the semantic and temporal alignment label,  $\omega$  is CFG scale and  $\gamma$  is the CG scale.
C.3 Intuition Perspective of Double Guidance
We offer another intuitive perspective to explain the technique of double guidance. In practice, the CFG [17] method tends to outperform the CG [5] method. This is because the classifier in the CG method may learn shortcuts for classification, leading to good classification results, but when its loglikelihood gradient is used to guide the reverse process, it can generate incorrect samples. In contrast, the double guidance technique addresses this issue by leveraging both the CFG and CG terms. The CFG term provides a robust and reliable noise direction for each timestep in the reverse process, while the CG term refines the CFG term's direction towards the desired data attribute, resulting in better sample quality. The effectiveness of the double guidance technique is validated by the evaluation results in main paper Table. 1, 6.
C.4 The Effect of Guidance Techniques
We present the generated results using double guidance techniques in Figure 9. The first row showcases the ground truth Mel-spectrogram and video frames, depicting a man smashing a window. Subsequent rows show the generated Mel-spectrogram results, arranged from left to right with progressively larger CFG scales  $\omega$  and from top to bottom with increasingly larger CG scales  $\gamma$ . Moving from left to right, we observed a significant improvement in audio quality and synchronization with increasing CFG scale  $\omega$ . Additionally, with increasing CG scale  $\gamma$ , the generation results of the original CFG scales were progressively refined, leading to higher levels of audio-visual alignment in the generated results.
Figure 9: Results of double guidance techniques: The first rows show the Ground Truth Mel-spectrogram and video frames. The subsequent rows exhibit generated results with progressively larger CFG scales  $\omega$  and CG scales  $\gamma$ .
Model	Reconstruction Metrics
FID ↓	KL ↓	MSE ↓
DIFF-FOLEY (Encoder E & Decoder D)	9.20	1.44	0.00264
Table 7: Reconstruction results of SD-V1.4 frozen latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$ . MSE represents the mean square error of the reconstructed Mel-spectrogram. The frozen latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  exhibit strong capability in reconstructing the Mel-spectrogram.
Figure 10: Visualization results of Mel-spectrogram reconstruction. The frozen latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  from SD-V1.4 [38] demonstrate exceptional capability in reconstructing the Mel-spectrogram.
D Discussion of Stable Diffusion Encoder and Decoder