Sukhvansh/audio-scene-synthesis

Audio Scene Synthesis — trained checkpoints

Checkpoints for Sukhvansh2004/Audio-Scene-Synthesis, a refined Audio-LDM for text-to-audio generation that replaces AudioLDM's conditioning stack with Microsoft CLAP (HTSAT audio encoder + GPT-2 text encoder), plus several side experiments.

Almost everything here is weights we trained ourselves, plus a handful of generated samples. The one exception is third-party/clap_htsat_tiny.pt, which is mirrored because no upstream release ships it. The other third-party components — the pretrained mel VAE, HiFi-GAN vocoders, AudioMAE, the official AudioLDM checkpoints — are not mirrored here; the repository's scripts/setup_data.sh fetches those from their original sources.

Files

audioldm/ — the main model

File	Size	What it is
checkpoint-fad-133.00-global_step=69999.ckpt	5.5 GB	The refined model. MS-CLAP (HTSAT + GPT-2) conditioning, FiLM over a 1024-d embedding, trained from scratch for 70k steps on AudioCaps. Config: audioldm_train/config/2023_08_23_reproduce_audioldm/audioldm_custom.yaml
checkpoint-fad-133.00-global_step=499999.ckpt	4.6 GB	Our AudioLDM baseline for comparison, 500k steps. Config: audioldm_original.yaml

The refined model reached a lower training loss at 70k steps than the baseline did at 500k.

dcase2023-task7/ — Foley sound synthesis experiment

Our fork of the DCASE 2023 Task 7 baseline, in experiments/dcase2023_task7/.

File	Size	What it is
vqvae_046.pt	8.7 MB	Multi-scale VQ-VAE, epoch 46 — the discrete time-frequency representation
pixelsnail_bottom_050.pt	97 MB	Class-conditional PixelSNAIL prior over the VQ-VAE codes, epoch 50
checkpoint_cvae_improved_no_fl_500_6000.pt	28 MB	Conditional VAE (cVAE_improved.py) at 6000 epochs, our continuous alternative to the discrete prior

pixelsnail-melspec/ — autoregressive spectrogram model

File	Size	What it is
checkpoint_41_model2.pt	17 MB	PixelSNAIL trained directly on mel spectrograms rendered as images, epoch 41. See experiments/pixelsnail/

clap-autoencoder/

File	Size	What it is
encoder_model.pth	269 MB	Autoencoder over CLAP text embeddings, loaded by msclap/models/clap2.py and used by the custom_audioldm.yaml config

third-party/ — mirrored, not ours

File	Size	What it is
clap_htsat_tiny.pt	1.7 GB	LAION CLAP HTSAT-tiny weights, needed by the audioldm_original.yaml baseline config. Mirrored here only because upstream's checkpoints.tar does not ship it; the original project is LAION-AI/CLAP.

samples/ — generated audio, for a quick listen

Ten 16 kHz mono clips, 10.24 s each, so you can hear what the refined model produces without downloading a 5.9 GB checkpoint. samples/index.csv records the prompt, source split and provenance of each one.

• samples/curated/ (4) — a hand-picked demo set. The exact checkpoint that produced these was not recorded.
• samples/audiocaps-test/ (6) — from a validation pass at global_step 70005, the same training run as the 69,999 checkpoint above, on held-out AudioCaps test captions. Filenames are the prompts.

Usage

The repository's setup script places everything for you:

git clone https://github.com/Sukhvansh2004/Audio-Scene-Synthesis.git
cd Audio-Scene-Synthesis
./scripts/setup_data.sh --minimal      # pretrained components + the refined model
./scripts/setup_data.sh --experiments  # the DCASE and PixelSNAIL weights

Or fetch a single file directly:

from huggingface_hub import hf_hub_download

ckpt = hf_hub_download(
    "Sukhvansh/audio-scene-synthesis",
    "audioldm/checkpoint-fad-133.00-global_step=69999.ckpt",
)

Then run inference with the matching config:

python3 audioldm_train/infer.py \
  --config_yaml audioldm_train/config/2023_08_23_reproduce_audioldm/audioldm_custom.yaml \
  --list_inference tests/captionlist/inference_test.lst \
  --reload_from_ckpt "$ckpt"

Note the known CUDA driver issue with inference documented in the repository README.

Licence and attribution

Released under CC-BY-NC-4.0, matching the licence of the pretrained AudioLDM checkpoints these models were built against — non-commercial use only.

The work builds on:

• AudioLDM and AudioLDM-training-finetuning (MIT) — Liu et al., ICML 2023
• Microsoft CLAP (MIT)
• LAION CLAP, HiFi-GAN
• DCASE 2023 Task 7 baseline and liuxubo717/sound_generation

Training data: AudioCaps (Kim et al., NAACL 2019) over AudioSet audio. The DCASE models were trained on the DCASE 2023 Task 7 development set (Zenodo 8091972), part of which is BBC Sound Effects material provided for that challenge under research-use-only terms.

@article{liu2023audioldm,
  title={{AudioLDM}: Text-to-Audio Generation with Latent Diffusion Models},
  author={Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo
          and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D},
  journal={Proceedings of the International Conference on Machine Learning},
  year={2023}
}