Package: conette
CoNeTTE model for Audio CaptioningCoNeTTE is an audio captioning system, which generate a short textual description of the sound events in any audio file. The architecture and training are explained in the correspondingpaper. The model has been developped by me (Étienne Labbé) during my PhD. A simple interface to test CoNeTTE is available onHuggingFace website.InferenceInstallationpython-mpipinstallconetteUsage with pythonfromconetteimportCoNeTTEConfig,CoNeTTEModelconfig=CoNeTTEConfig.from_pretrained("Labbeti/conette")model=CoNeTTEModel.from_pretrained("Labbeti/conette",config=config)path="/your/path/to/audio.wav"outputs=model(path)candidate=outputs["cands"][0]print(candidate)The model can also accept several audio files at the same time (list[str]), or a list of pre-loaded audio files (list[Tensor]). In this second case you also need to provide the sampling rate of this files:importtorchaudiopath_1="/your/path/to/audio_1.wav"path_2="/your/path/to/audio_2.wav"audio_1,sr_1=torchaudio.load(path_1)audio_2,sr_2=torchaudio.load(path_2)outputs=model([audio_1,audio_2],sr=[sr_1,sr_2])candidates=outputs["cands"]print(candidates)The model can also produces different captions using a Task Embedding input which indicates the dataset caption style. The default task is "clotho".outputs=model(path,task="clotho")candidate=outputs["cands"][0]print(candidate)outputs=model(path,task="audiocaps")candidate=outputs["cands"][0]print(candidate)Usage with command lineSimply use the commandconette-predictwith--audio PATH1 PATH2 ...option. You can also export results to a CSV file using--csv_export PATH.conette-predict--audio"/your/path/to/audio.wav"PerformanceThe model has been trained on AudioCaps (AC), Clotho (CL), MACS (MA) and WavCaps (WC). The performance on the test subsets are :Test dataSPIDEr (%)SPIDEr-FL (%)FENSE (%)VocabOutputsScoresAC-test44.1443.9860.81309LinkLinkCL-eval30.9730.8751.72636LinkLinkThis model checkpoint has been trained with focus on the Clotho dataset, but it can also reach a good performance on AudioCaps with the "audiocaps" task.LimitationsThe model expected audio sampled at32 kHz. The model automatically resample up or down the input audio files. However, it might give worse results, especially when using audio with lower sampling rates.The model has been trained on audio lasting from1 to 30 seconds. It can handle longer audio files, but it might require more memory and give worse results.Train a modelRequirementsIntended for Ubuntu 20.04 only. Requiresjava< 1.13,ffmpeg,yt-dlp, andzipcommands.Recommanded GPU: NVIDIA V100 with 32GB VRAM.WavCaps dataset might requires more than 2 TB of disk storage. Other datasets requires less than 50 GB.InstallationBy default,only the inference requirements are installed for conette. To install training requirements you need to use the following command:python-mpipinstallconette[train]If you already installed conette for inference, it ishighly recommanded to create another environmentbefore installing conette for training.Download external models and dataThese steps might take a while (few hours to download and prepare everything depending on your CPU, GPU and SSD/HDD).First, download the ConvNeXt, NLTK and spacy models :conette-preparedata=nonedefault=truepack_to_hdf=falsecsum_in_hdf_name=falsepann=falseThen download the 4 datasets used to train CoNeTTE :cnext_bl_path="$HOME/.cache/torch/hub/checkpoints/convnext_tiny_465mAP_BL_AC.pth"common_args="data.download=true pack_to_hdf=true audio_t=resample_mean_convnext audio_t.pretrain_path=${cnext_bl_path}post_hdf_name=bl pretag=cnext_bl"conette-preparedata=audiocapsaudio_t.src_sr=32000${common_args}conette-preparedata=clothoaudio_t.src_sr=44100${common_args}conette-preparedata=macsaudio_t.src_sr=48000${common_args}conette-preparedata=wavcapsaudio_t.src_sr=32000${common_args}datafilter.min_audio_size=0.1datafilter.max_audio_size=30.0datafilter.sr=32000Train a modelCNext-trans (baseline) on CL only (~3 hours on 1 GPU V100-32G)conette-trainexpt=[clotho_cnext_bl]pl=baselineCoNeTTE on AC+CL+MA+WC, specialized for CL (~4 hours on 1 GPU V100-32G)conette-trainexpt=[camw_cnext_bl_for_c,task_ds_src_camw]pl=conetteCoNeTTE on AC+CL+MA+WC, specialized for AC (~3 hours on 1 GPU V100-32G)conette-trainexpt=[camw_cnext_bl_for_a,task_ds_src_camw]pl=conetteAbout reproducibility: any training with AC data cannot be reproduced because a part of this data is deleted from the YouTube source, and I cannot share my own audio files.CitationThe preprint version of the paper describing CoNeTTE is available on arxiv:https://arxiv.org/pdf/2309.00454.pdf@misc{labbé2023conette,title={CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding},author={Étienne Labbé and Thomas Pellegrini and Julien Pinquier},year=2023,journal={arXiv preprint arXiv:2309.00454},url={https://arxiv.org/pdf/2309.00454.pdf},eprint={2309.00454},archiveprefix={arXiv},primaryclass={cs.SD}}Additional informationCoNeTTE stands forConvNeXt-Transformer withTaskEmbedding.Model weights are available on HuggingFace:https://huggingface.co/Labbeti/conetteThe weights of the encoder part of the architecture is based on a ConvNeXt model for audio classification, available here:https://zenodo.org/record/8020843under the filename "convnext_tiny_465mAP_BL_AC_70kit.pth".ContactMaintainer:Etienne Labbé "Labbeti":labbeti.pub@gmail.com