SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Reconstruction Capability of SD Encoder & Decoder In DIFF-FOLEY, we directly use the frozen pretrained latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  in SD-V1.4 [38] to encode the Mel-spectrogram to a low-dimensional latent (mentioned in Sec. A.1). The reason we choose not to retrain encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  on the Mel-spectrogram data is that using the pretrained SD-V1.4 weight for parameter initialization can significantly reduce model training time and improve generation quality (Already discussed in main paper Table. 4). However, it is necessary to use the corresponding frozen latent encoders  $\mathcal{E}$  and decoders  $\mathcal{D}$  when using the pre-trained SD-V1.4 model. Interestingly, despite being trained on large scale image datasets (LAION-5B [40]), we found that latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  demonstrate good reconstruction capability on Mel-spectrogram data. We report the reconstruction metrics results of SD latent encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  in Table. 7. It can be noticed that the latent autoencoder  $(\mathcal{E},\mathcal{D})$  has relatively good reconstruction capability, as demonstrated by the KL and MSE metrics in Table. 7. However, the FID score is slightly higher, which may explain why DIFF-FOLEY does not exhibit a significant advantage over other baselines in terms of FID, which is likely due to the limitation in the frozen pretrained autoencoder reconstruction capabilities. We also visualize the reconstructed Mel-spectrogram results in Figure 10, showing that frozen pretrained  $\mathcal{E}$  and  $\mathcal{D}$  have excellent reconstruction capability on Mel-spectrogram data.