Package: hear21passt
PaSST package for HEAR 2021 NeurIPS Challenge Holistic Evaluation of Audio RepresentationsThis is an implementation forEfficient Training of Audio Transformers with Patchoutfor HEAR 2021 NeurIPS Challenge
Holistic Evaluation of Audio RepresentationsCUDA versionThis is an implementation is tested with CUDA version 11.1, and torch installed:pip3installtorch==1.8.1+cu111torchaudio==0.8.1-fhttps://download.pytorch.org/whl/lts/1.8/torch_lts.htmlbut should work on newer versions of CUDA and torch.InstallationInstall the latest version of this repo:pipinstallhear21passtThe models follow thecommon APIof HEAR 21
:hear-validator--modelhear21passt.base.pthear21passt.base
hear-validator--modelnoweights.txthear21passt.base2levelF
hear-validator--modelnoweights.txthear21passt.base2levelmelThere are three modules availablehear21passt.base,hear21passt.base2level,hear21passt.base2levelmel:importtorchfromhear21passt.baseimportload_model,get_scene_embeddings,get_timestamp_embeddingsmodel=load_model().cuda()seconds=15audio=torch.ones((3,32000*seconds))*0.5embed,time_stamps=get_timestamp_embeddings(audio,model)print(embed.shape)embed=get_scene_embeddings(audio,model)print(embed.shape)Getting the Logits/Class LabelsYou can get the logits (before the sigmoid activation) for the 527 classes of audioset:fromhear21passt.baseimportload_modelmodel=load_model(mode="logits").cuda()logits=model(wave_signal)The class labels indices can be foundhereYou can also use different pre-trained models, for example, the model trained with KDpasst_s_kd_p16_128_ap486:fromhear21passt.baseimportget_basic_modelmodel=get_basic_model(mode="logits",arch="passt_s_kd_p16_128_ap486")logits=model(wave_signal)Supporting longer clipsIn case of an input longer than 10 seconds, theget_scene_embeddingsmethod compute the average of the embedding of a 10-second overlapping windows.
Depending on the application, it may be useful to use a pre-trained that can extract embeddings from 20 or 30 seconds without averaging. These variant has pre-trained time positional encoding or 20/30 seconds:# from version 0.0.18, it's possible to use:fromhear21passt.base20secimportload_model# up to 20 seconds of audio.# orfromhear21passt.base30secimportload_model# up to 30 seconds of audio.model=load_model(mode="logits").cuda()logits=model(wave_signal)Loading other pre-trained models for logits or fine-tuningEach pre-trained model has a specific frequency/time positional encoding, it's necessary to select the correct input shape to be able to load the models. The important variables for loading areinput_tdim,fstrideandtstrideto specify the spectrograms time frames, the patches stride over frequency, and patches stride over time, respectively.importtorchfromhear21passt.baseimportget_basic_model,get_model_passtmodel=get_basic_model(mode="logits")logits=model(some_wave_signal)# Examples of other pre-trained models using the same spectrograms# pre-traind on openMIC-18model.net=get_model_passt(arch="openmic",n_classes=20)# pre-traind on FSD-50kmodel.net=get_model_passt(arch="fsd50k",n_classes=200)# pre-traind on FSD-50k without patch-overlap (faster)model.net=get_model_passt(arch="fsd50k-n",n_classes=200,fstride=16,tstride=16)# models are trained on 10 seconds audios from Audioset, but accept longer audios (20s, or 30s)# These models are trained by sampling a 10-second time-pos-encodings sequencemodel.net=get_model_passt("passt_20sec",input_tdim=2000)model.net=get_model_passt("passt_30sec",input_tdim=3000)If you provide the wrong spectrograms, the model may fail silently, by generating low-quality embeddings and logits. Make sure you have the correct spectrograms' config for the selected pre-trained models.
Models with higher spectrogram resolutions, need to specify the correct spectrogram config:fromhear21passt.models.preprocessimportAugmentMelSTFT# high-res pre-trained on Audiosetmodel.net=get_model_passt("stfthop160",input_tdim=2000)# hopsize=160 for this pretrained modelmodel.mel=AugmentMelSTFT(n_mels=128,sr=32000,win_length=800,hopsize=160,n_fft=1024,freqm=48,timem=192,htk=False,fmin=0.0,fmax=None,norm=1,fmin_aug_range=10,fmax_aug_range=2000)# higher-res pre-trained on Audiosetmodel.net=get_model_passt("stfthop100",input_tdim=3200)# hopsize=100 for this pretrained modelmodel.mel=AugmentMelSTFT(n_mels=128,sr=32000,win_length=800,hopsize=100,n_fft=1024,freqm=48,timem=192,htk=False,fmin=0.0,fmax=None,norm=1,fmin_aug_range=10,fmax_aug_range=2000)