Package: laion-clap
CLAPThis repository provides representations of audios and texts via Contrastive Language-Audio Pretraining (CLAP)With CLAP, you can extract a latent representation of any given audio and text for your own model, or for different downstream tasks.All codes are comming officially with the following paper, accepted by IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023:Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption AugmentationNew Update: we release new CLAP pretrained checkpoints pretrained on music and speech data collecstions fromour dataset collection repo.About this projectThis project is a project inLAIONthat aims at learning better audio understanding and getting more audio data.
This is an opensource project. We adopt the codebase ofopen_clipfor this project.many thanks to@cfoster0for allowing us to use his repo name.ArchitectureContrastive Language-Audio Pretraining, known as CLAP. Referring to the CLIP (Contrastive Language-Image Pretraining) architecture, the CLAP architecture is as follows.Quick StartWe provide the PyPI library for our CLAP model:pipinstalllaion-clapThen you can follow the below usage or refer tounit_test.py.For the documentation of the API, please refer tohook.py.importnumpyasnpimportlibrosaimporttorchimportlaion_clap# quantizationdefint16_to_float32(x):return(x/32767.0).astype(np.float32)deffloat32_to_int16(x):x=np.clip(x,a_min=-1.,a_max=1.)return(x*32767.).astype(np.int16)model=laion_clap.CLAP_Module(enable_fusion=False)model.load_ckpt()# download the default pretrained checkpoint.# Directly get audio embeddings from audio filesaudio_file=['/home/data/test_clap_short.wav','/home/data/test_clap_long.wav']audio_embed=model.get_audio_embedding_from_filelist(x=audio_file,use_tensor=False)print(audio_embed[:,-20:])print(audio_embed.shape)# Get audio embeddings from audio dataaudio_data,_=librosa.load('/home/data/test_clap_short.wav',sr=48000)# sample rate should be 48000audio_data=audio_data.reshape(1,-1)# Make it (1,T) or (N,T)audio_embed=model.get_audio_embedding_from_data(x=audio_data,use_tensor=False)print(audio_embed[:,-20:])print(audio_embed.shape)# Directly get audio embeddings from audio files, but return torch tensoraudio_file=['/home/data/test_clap_short.wav','/home/data/test_clap_long.wav']audio_embed=model.get_audio_embedding_from_filelist(x=audio_file,use_tensor=True)print(audio_embed[:,-20:])print(audio_embed.shape)# Get audio embeddings from audio dataaudio_data,_=librosa.load('/home/data/test_clap_short.wav',sr=48000)# sample rate should be 48000audio_data=audio_data.reshape(1,-1)# Make it (1,T) or (N,T)audio_data=torch.from_numpy(int16_to_float32(float32_to_int16(audio_data))).float()# quantize before send it in to the modelaudio_embed=model.get_audio_embedding_from_data(x=audio_data,use_tensor=True)print(audio_embed[:,-20:])print(audio_embed.shape)# Get text embedings from texts:text_data=["I love the contrastive learning","I love the pretrain model"]text_embed=model.get_text_embedding(text_data)print(text_embed)print(text_embed.shape)# Get text embedings from texts, but return torch tensor:text_data=["I love the contrastive learning","I love the pretrain model"]text_embed=model.get_text_embedding(text_data,use_tensor=True)print(text_embed)print(text_embed.shape)Pretrained ModelsThe pretrained checkpoints can be found inhere.
Please refer to the previous section for how to load and run the checkpoints.
For the PyPI library,630k-audioset-best.ptand630k-audioset-fusion-best.ptare our default models (non-fusion and fusion)We further provide below pretrained models according to your usages:For general audio less than 10-sec:630k-audioset-best.ptor630k-best.ptFor general audio with variable-length:630k-audioset-fusion-best.ptor630k-fusion-best.ptFor music:music_audioset_epoch_15_esc_90.14.ptFor music and speech:music_speech_epoch_15_esc_89.25.ptFor speech, music and general audio:music_speech_audioset_epoch_15_esc_89.98.ptThe checkpoints list here for each model setting is the one with the highest average mAP score in training.
The average mAP score is calculated by averaging 4 scores: A-->T mAP@10 on AudioCaps, and T-->A mAP@10 on AudioCaps, A-->T mAP@10 on Clotho, and T-->A mAP@10 on Clotho.To use above pretrained models, you need to load the ckpt by yourself, as:Update 2023.4.7: we have released 3 larger CLAP models trained on music, speech dataset in addition to LAION-Audio-630k. Here are descriptions of the model and their performance:music_speech_audioset_epoch_15_esc_89.98.pt: trained on music + speech + Audioset + LAION-Audio-630k. The zeroshot ESC50 performance is 89.98%, the GTZAN performance is 73%.music_audioset_epoch_15_esc_90.14.pt: trained on music + Audioset + LAION-Audio-630k. The zeroshot ESC50 performance is 90.14%, the GTZAN performance is 64%.music_speech_epoch_15_esc_89.25.pt: trained on music + speech + LAION-Audio-630k. The zeroshot ESC50 performance is 89.25%, the GTZAN performance is 73%.The model uses a larger audio encoder. To load the model using the pip API:importlaion_clapmodel=laion_clap.CLAP_Module(enable_fusion=False,amodel='HTSAT-base')model.load_ckpt('checkpoint_path/checkpoint_name.pt')Please note that this is a temporary release for people who are working on larger-scale down-stream task.
We will release a more comprehensive version of the model with detailed experiments in the future.
Please take your own risk when using this model.All the new checkpoints did not trained with fusion. The training dataset size formusic_speech_audioset_epoch_15_esc_89.98.ptis around 4M samples. The zeroshot GTZAN score is evaluated using the promptThis audio is a <genre> song.We provide the CLAP's performance on audio classification tasks under the zero-shot setting or the supervised setting. More results can be found at our paper.Environment InstallationIf you want to check and reuse our model into your project instead of directly using the pip library, you need to install the same environment as we use, please run the following command:condacreateenv-nclappython=3.10
condaactivateclap
gitclonehttps://github.com/LAION-AI/CLAP.gitcdCLAP# you can also install pytorch by following the official instruction (https://pytorch.org/get-started/locally/)pipinstalltorch==1.11.0+cu113torchvision==0.12.0+cu113torchaudio==0.11.0+cu113-fhttps://download.pytorch.org/whl/torch_stable.html
pipinstall-rrequirements.txtDataset formatWe use training data in webdataset format. For details of our dataset please seehttps://github.com/LAION-AI/audio-dataset.Due to copyright reasons, we cannot release the dataset we train this model on. However, we releasedLAION-audio-630K, the data source we used to compose the dataset with link to each audio and their caption. Please refer toLAION-audio-630Kfor more details. You could download the dataset, preprocess it on your own and train it locally. To train on the local dataset, please change the--remotedatain training scripts (seeexperiment_scriptsfolder) with--datasetpath <your dir to datasets>.You can find an example of our dataset format inhere.
It contains the full ESC50 dataset, split according to the first 5-fold split.Training, Fine-tuning and EvaluationPlease find the script of training, fine-tuning and evaluation (zero-shot and retrieval) in theexperiment_scriptsfolder.
The scripts included there are the one we used to train our model on a SLURM cluster.
You need to change the script to fit your own environment.
For example, in a single machine multi-GPU setting, you might want to usetorchruninstead ofsrunto run the script.
To train on a single GPU machine, useCUDA_VISIBLE_DEVICES=0 python -m ...instead ofsrun.
We useWeights and Biasesfor experiment logging. You need to configure the weights and biases in your environment.
To train on local dataset, please change the--remotedatain training scripts (seeexperiment_scriptsfolder) with--datasetpath <your dir to datasets>.Core CodePlease refer tomain.py,train.py,data.py,andmodel.pyto quicly get familiar with our model.ReproducibilityAn example of the preprocessed Clotho dataset in webdataset format can be downloadhere(by downloading, you will be agreeing the license described in theClotho dataset). The audio encoder pretrained with 48kHz AudioSet can be foundhere, whereHTSAT-fullset-imagenet-map=0.467.ckptis the checkpoint used to initalize our HTSAT audio encoder. You should get similar result by loading from the audio encoder checkpoint and training on same dataset.The script to train the model on Clotho dataset is includedhere. You need to replace thedatasetpathandpretrained-audioto pointing to your own directory. You could check thereportof the training script on a single A100 GPU for reference.Because most of the dataset has copyright restriction, unfortunatly we cannot directly share other preprocessed datasets. The caption generated by keyword-to-caption model for Audioset can be foundhereZeroshot Classification with ESC50 official splitHere is an example code to run the zeroshot classification onfirstESC50 official split with the pip API:importlaion_clapimportglobimportjsonimporttorchimportnumpyasnpdevice=torch.device('cuda:0')# download https://drive.google.com/drive/folders/1scyH43eQAcrBz-5fAw44C6RNBhC3ejvX?usp=sharing and extract ./ESC50_1/test/0.tar to ./ESC50_1/test/esc50_test_dir='./ESC50_1/test/*/'class_index_dict_path='./class_labels/ESC50_class_labels_indices_space.json'# Load the modelmodel=laion_clap.CLAP_Module(enable_fusion=False,device=device)model.load_ckpt()# Get the class index dictclass_index_dict={v:kforv,kinjson.load(open(class_index_dict_path)).items()}# Get all the dataaudio_files=sorted(glob.glob(esc50_test_dir+'**/*.flac',recursive=True))json_files=sorted(glob.glob(esc50_test_dir+'**/*.json',recursive=True))ground_truth_idx=[class_index_dict[json.load(open(jf))['tag'][0]]forjfinjson_files]withtorch.no_grad():ground_truth=torch.tensor(ground_truth_idx).view(-1,1)# Get text featuresall_texts=["This is a sound of "+tfortinclass_index_dict.keys()]text_embed=model.get_text_embedding(all_texts)audio_embed=model.get_audio_embedding_from_filelist(x=audio_files)ranking=torch.argsort(torch.tensor(audio_embed)@torch.tensor(text_embed).t(),descending=True)preds=torch.where(ranking==ground_truth)[1]preds=preds.cpu().numpy()metrics={}metrics[f"mean_rank"]=preds.mean()+1metrics[f"median_rank"]=np.floor(np.median(preds))+1forkin[1,5,10]:metrics[f"R@{k}"]=np.mean(preds<k)# map@10metrics[f"mAP@10"]=np.mean(np.where(preds<10,1/(preds+1),0.0))print(f"Zeroshot Classification Results: "+"\t".join([f"{k}:{round(v,4):.4f}"fork,vinmetrics.items()]))For ESC50 dataset, you could either download our processed ESC50 in webdataset format
fromhere, and extract the./test/0.tarto./test/. Or you could download the original ESC50 dataset and
preprocess the label to the format ofclass_labels/ESC50_class_labels_indices_space.jsonby yourself (replace_with space).The result should be the same as the following:Formodel = laion_clap.CLAP_Module(enable_fusion=True, device=device):mean_rank: 1.2425	median_rank: 1.0000	R@1: 0.9050	R@5: 0.9900	R@10: 0.9925	mAP@10: 0.9407Formodel = laion_clap.CLAP_Module(enable_fusion=False, device=device):mean_rank: 1.1450	median_rank: 1.0000	R@1: 0.9275	R@5: 0.9975	R@10: 1.0000	mAP@10: 0.9556Note that the results is slightly higher than the reported results in the paper, because we use the train + test data of ESC50 and removing the data overlap in other training datasets (mainly freesound).CitationIf you find this project and the LAION-Audio-630K dataset useful, please cite our paper:@inproceedings{laionclap2023,
  title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
  author = {Wu*, Yusong and Chen*, Ke and Zhang*, Tianyu and Hui*, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP},
  year = {2023}
}
@inproceedings{htsatke2022,
  author = {Ke Chen and Xingjian Du and Bilei Zhu and Zejun Ma and Taylor Berg-Kirkpatrick and Shlomo Dubnov},
  title = {HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP},
  year = {2022}
}AcknowledgementsThis project is working in progress, thus the codebase and model might not be perfect or bug-free.
We will very much appreciate any kind of contribution or and issue raised.
If you find a bug or have any suggestion, please feel free to open an issue or contact us.
If you would actively contribute to this project, please join the discord of LAION.