Package: nkululeko
OverviewConfusion matrixEpoch progressionFeature importanceFeature distributiont-SNE plotsData distributionBias checkingDocumentationInstallationUsageini-file valuesHello World exampleFeaturesLicenseOverviewA project to detect speaker characteristics by machine learning experiments with a high-level interface.The idea is to have a framework (based on e.g. sklearn and torch) that can be used to rapidly and automatically analyse audio data and explore machine learning models based on that data.NEW: Nkululeko now automatically generates PDF reportssample for EmoDBThe latest features can be seen inthe ini-fileoptions that are used to control NkululekoBelow is aHello World examplethat should set you up fastly, also onGoogle Colab, andwith KaggleHere's a blog post on how to set up nkululeko on your computer.Here is a slack channel to discuss issues related to nkululeko. Please click the link if interested in contributing.Here's a slide presentation about nkululekoHere's a video presentation about nkululekoHere's the 2022 LREC article on nkululekoHere are some examples of typical output:Confusion matrixPer default, Nkululeko displays results as a confusion matrix using binning with regression.Epoch progressionThe point when overfitting starts can sometimes be seen by looking at the results per epoch:Feature importanceUsing theexploreinterface, Nkululeko analyses the importance of acoustic features:Feature distributionAnd can show the distribution of specific features per category:t-SNE plotsA t-SNE plot can give you an estimate wether your acoustic features are useful at all:Data distributionSometimes you only want to take a look at your data:Bias checkingIn cases you might wonder if there's bias in your data. You can try to detect this with automatically estimated speech properties, by visualizing the correlation of target label and predicted labels.DocumentationThe documentation, along with extensions of installation, usage, INI file format, and examples, can be foundnkululeko.readthedocs.io.InstallationCreate and activate a virtual Python environment and simply runpip install nkululekoWe excluded some packages from the automatic installation because they might depend on your computer and some of them are only needed in special cases. So if the errormodule x not foundappears, please trypip install xFor many packages you will need the missing torch package.
If you don't have a GPU (which is probably true if you don't know what that is), please usepip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpuelse, you can use the default:pip install torch torchvision torchaudioSome functionalities require extra packages to be installed, which we didn't include automatically:the SQUIM model needs a special torch version:pip uninstall -y torch torchvision torchaudio
pip install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cputhe spotlight adapter needs spotlight:pip install renumics-spotlight sliceguardSome examples forini-files (which you use to control nkululeko) are in thetests folder.Usageini-file valuesBasically, you specify your experiment in an"ini" file(e.g.experiment.ini) and then call one of the Nkululeko interfaces to run the experiment like this:python -m nkululeko.nkululeko --config experiment.iniA basic configuration looks like this:[EXP]
root = ./
name = exp_emodb
[DATA]
databases = ['emodb']
emodb = ./emodb/
emodb.split_strategy = speaker_split
target = emotion
labels = ['anger', 'boredom', 'disgust', 'fear']
[FEATS]
type = ['praat']
[MODEL]
type = svm
[EXPL]
model = tree
plot_tree = True
[PLOT]
combine_per_speaker = modeRead theHello World examplefor initial usage with Emo-DB dataset.Here is an overview of the interfaces:nkululeko.nkululeko: do machine learning experiments combining features and learnersnkululeko.multidb: do multiple experiments, comparing several databases cross and in itselfnkululeko.demo: demo the current best model on the command linenkululeko.test: predict a series of files with the current best modelnkululeko.explore: perform data explorationnkululeko.augment: augment the current training datankululeko.aug_train: augment the current training data and do a training including this datankululeko.predict: predict features like SNR, MOS, arousal/valence, age/gender, with DNN modelsnkululeko.segment: segment a database based on VAD (voice activity detection)nkululeko.resample: check on all sampling rates and change to 16kHzThere's myblogwith tutorials:IntroductionNkulueko FAQHow to set up your first nkululeko projectSetting up a base nkululeko experimentHow to import a databaseComparing classifiers and featuresUse Praat featuresCombine feature setsClassifying continuous variablesTry out / demo a trained modelPerform cross database experimentsMeta parameter optimizationHow to set up wav2vec embeddingHow to soft-label a databaseRe-generate the progressing confusion matrix animation wit a different framerateHow to limit/filter a datasetSpecifying database disk locationAdd dropout with MLP modelsDo cross-validationCombine predictions per speakerRun multiple experiments in one goCompare several MLP layer layouts with each otherImport features from outside the softwareExplore feature importancePlot distributions for feature valuesShow feature importanceAugment the training setVisualize clusters of acoustic featuresVisualize your data distributionCheck your datasetSegmenting a databasePredict new labels for your data from public models and check biasResampleGet some statistics on correlation and effect-sizeGenerate a latex / pdf reportInspect your data with SpotlightAutomatically stratify your split setsre-name data column namesOversample the training setCompare several databasesHello World exampleNEW:Here's a Google colab that runs this example out-of-the-box, and here is the samewith KaggleI made a video to show you how to do this on WindowsSet up Python on your computer, version >= 3.8Open a terminal/commandline/console windowTest python by typingpython, python should start with version >3 (NOT 2!). You can leave the Python Interpreter by typingexit()Create a folder on your computer for this example, let's call itnkulu_workGet a copy of theBerlin emodb in audformatand unpack inside the folder you just created (nkulu_work)Make sure the folder is called "emodb" and does contain the database files directly (not box-in-a-box)Also, in thenkulu_workfolder:Create a Python environmentpython -m venv venvThen, activate it:under Linux / macsource venv/bin/activateunder Windowsvenv\Scripts\activate.batif that worked, you should see a(venv)in front of your promptInstall the required packages in your environmentpip install nkululekoRepeat until all error messages vanished (or fix them, or try to ignore them)...Now you should have two folders in yournkulu_workfolder:emodbandvenvDownload a copy of the fileexp_emodb.inito the current working directory (nkulu_work)Run the demopython -m nkululeko.nkululeko --config exp_emodb.iniFind the results in the newly created folder exp_emodbInspectexp_emodb/images/run_0/emodb_xgb_os_0_000_cnf.pngThis is the main result of you experiment: a confusion matrix for the emodb emotional categoriesInspect and play around with thedemo configuration filethat defined your experiment, then re-run.There are many ways to experiment with different classifiers and acoustic features sets,all described hereFeaturesThe framework is targeted at the speech domain and supports experiments where different classifiers are combined with different feature extractors.Classifiers: Naive Bayes, KNN, Tree, XGBoost, SVM, MLPFeature extractors: Praat, Opensmile, openXBOW BoAW, TRILL embeddings, Wav2vec2 embeddings, audModel embeddings, ...Feature scalingLabel encodingBinning (continuous to categorical)Online demo interface for trained modelsHere's a rough UML-like sketch of the framework (andhere's the real one done with pyreverse).Currently, the following linear classifiers are implemented (integrated from sklearn):SVM, SVR, XGB, XGR, Tree, Tree_regressor, KNN, KNN_regressor, NaiveBayes, GMM
and the following ANNs (artificial neural networks)MLP (multi-layer perceptron), CNN (convolutional neural network)Here'san animation that shows the progress of classification done with nkululekoLicenseNkululeko can be used under theMIT licenseIf you use it, please mention the Nkululeko paperF. Burkhardt, Johannes Wagner, Hagen Wierstorf, Florian Eyben and Björn Schuller: Nkululeko: A Tool For Rapid Speaker Characteristics Detection, Proc. Proc. LREC, 2022@inproceedings{Burkhardt:lrec2022,
   title = {Nkululeko: A Tool For Rapid Speaker Characteristics Detection},
   author = {Felix Burkhardt and Johannes Wagner and Hagen Wierstorf and Florian Eyben and Björn Schuller},
   isbn = {9791095546726},
   journal = {2022 Language Resources and Evaluation Conference, LREC 2022},
   keywords = {machine learning,speaker characteristics,tools},
   pages = {1925-1932},
   publisher = {European Language Resources Association (ELRA)},
   year = {2022},
}ChangelogVersion 0.79.2fixed bug and report number of epochs for early stoppingVersion 0.79.1root directory does not have to end with /Version 0.79.0added extra_train for multidb experimentVersion 0.78.2added transformer layer selection for wav2vec2removed best_model and epoch progression for non-DL modelsVersion 0.78.1added evaluation lossVersion 0.78.0added 3-d scatter plotsremoved epoch-plots if epoch_num=1Version 0.77.14fixed bug preventing bin scaling to workVersion 0.77.13added bins scalerVersion 0.77.12fixed bug with scatter plots for numeric targetsmade type of numeric target distributions selectable, default "hist"Version 0.77.11added simple target distribution plotsVersion 0.77.10show the best and not the last result for multidbVersion 0.77.9added results text for multidbVersion 0.77.8added caption to multidb heatmaprenamed datasets to databases in multidbVersion 0.77.7added multidb moduleVersion 0.77.6added functions to call modules with config file path directlyVersion 0.77.5fixed augmentation bug for python version 10Version 0.77.4made traditional augmentations (audiomentation module) configurableVersion 0.77.3added augment and train interfaceVersion 0.77.2added models for features importance computationVersion 0.77.1added permutation algorithm to compute feature importanceshifted util.py to utilsVersion 0.77.0added more latex report outputgot splitutils from a packageVersion 0.76.0added possibility to aggregate feature importance modelsVersion 0.75.0added max val for reversingadded xgb for feature importanceVersion 0.74.6added standard Wav2vec2 modelVersion 0.74.5added praat feature extractor for one sampleVersion 0.74.4fixed bug combining augmentationsVersion 0.74.3audiomentations interface changedVersion 0.74.2combined augmentation methodsVersion 0.74.1fixed various bugs with augmentationVersion 0.74.0added patience (early stopping)added MAE loss and measureVersion 0.73.0added reverse and scale arguments to target variablealso, the data store can now be csvVersion 0.72.0worked over explore value counts sectionadded bin_reals for all columnsVersion 0.71.4automatic epoch reset if not ANNscatter plots now show a regression lineVersion 0.71.3enabled scatter plots for all variablesVersion 0.71.2enabled scatter plots for continuous labelsVersion 0.71.1made a wav2vec defaultrenamed praat features, ommiting spacesfixed plot distribution bugsadded feature plots for continuous targetsVersion 0.71.0added explore visuals.all columns from databases should now be usableVersion 0.70.0added imb_learn balancing of training setVersion 0.69.0added CNN model and melspec extractorVersion 0.68.4bugfix: got_gender was uncorrectly setVersion 0.68.3Feinberg Praat scripts ignore error and log filenameVersion 0.68.2column names in datasets are now configurableVersion 0.68.1added error message on file to praat extractionVersion 0.68.0added stratification framework for split balancingVersion 0.67.0added first version of spotlight integrationVersion 0.66.13small changes related to github workerVersion 0.66.12fixed bug that prevented Praat features to be selectedVersion 0.66.11removed torch from automatic install. depends on cpu/gpu machineVersion 0.66.10Removed print statements from feats_wav2vec2Version 0.66.9Version that should install without requiring opensmile which seems not to be supported by all Apple processors (arm CPU (Apple M1))Version 0.66.8forgotinit.py in reporting moduleVersion 0.66.7minor changes to experiment classVersion 0.66.6minor cosmeticsVersion 0.66.5Latex report now with imagesVersion 0.66.4Pypi version mixupVersion 0.66.3made path to PDF output relative to experiment rootVersion 0.66.2enabled data-pathes with quotesenabled missing category labelsused tgdm for progress displayVersion 0.66.1start on the latex report frameworkVersion 0.66.0added speechbrain speakerID embeddingsVersion 0.65.9added a filter that ensures that the labels have the same size as the featuresVersion 0.65.8changed default behaviour of resampler to "keep original files"Version 0.65.7more databases and force wav while resamplingVersion 0.65.6minor catch for seaborn in plotsVersion 0.65.5added fill_na in plot effect sizeVersion 0.65.4added datasets to distributionchanges in wav2vec2Version 0.65.3various bugfixesVersion 0.65.2fixed bug in dataset.csv that prevented correct paths for relative filesfixed bug in export module concerning new file directoryVersion 0.65.1small enhancements with transformer featuresVersion 0.65.0introduced export moduleVersion 0.64.4added num_speakers for reloaded datare-formatted all with blackVersion 0.64.3added number of speakers shown after data loadVersion 0.64.2addedinit.py for submodulesVersion 0.64.1fix error on csvVersion 0.64.0added bin_realsadded statistics for effect size and correlation to plotsVersion 0.63.4fixed bug in split selectionVersion 0.63.3Introduced data.audio_pathVersion 0.63.2re-introduced min and max_length for silero segmenatationVersion 0.63.1fixed bug in resampleVersion 0.63.0added wavlm modeladded error on filename for modelsVersion 0.62.1added min and max_length for silero segmenatationVersion 0.62.0fixed segment silero bugadded all Wav2vec2 modelsadded resampler moduleadded error on file for embeddingsVersion 0.61.0added HUBERT embeddingsVersion 0.60.0some bugfixesnew package structurefixed wav2vec2 bugsremoved "cross_data" strategyVersion 0.59.1bugfix, after fresh install, it seems some libraries have changedadded no_warningschanged print() to util.debug()added progress to opensmile extractVersion 0.59.0introduced SQUIM featuresadded SDR predictadded STOI predictVersion 0.58.0added dominance predictadded MOS predictadded PESQ predictVersion 0.57.0renamed autopredict predictadded arousal autopredictadded valence autopredictVersion 0.56.0added autopredict moduleadded snr as feature extractoradded gender autopredictadded age autopredictadded snr autopredictVersion 0.55.1changed error message in plot classVersion 0.55.0added segmentation moduleVersion 0.54.0added audeering public age and gender model embeddings and age and gender predictionsVersion 0.53.0added file checks: size in bytes and voice activity detection with sileroVersion 0.52.1bugfix: min/max duration_of_sample was not workingVersion 0.52.0added flexible value distribution plotsVersion 0.51.0added datafilterVersion 0.50.1added caller information for debug and error messages in UtilVersion 0.50.0removed loso and added pre-selected logo (leave-one-group-out), aka foldsVersion 0.49.1bugfix: samples selection for augmentation didn't workVersion 0.49.0added random-splicingVersion 0.48.1bugfix: database object was not loaded when dataframe was reusedVersion 0.48.0enabled specific feature selection for praat and opensmile featuresVersion 0.47.1enabled feature storage format csv for opensmile featuresVersion 0.47.0added praat speech rate featuresVersion 0.46.0added warnings for non-existent parametersadded sample selection for scatter plottingVersion 0.45.4added version attribute to setup.cfgVersion 0.45.4addedversionattributeVersion 0.44.1bugfixing: feature importance:https://github.com/felixbur/nkululeko/issues/23bugfixing: loading csv database with filewise indexhttps://github.com/felixbur/nkululeko/issues/24Version 0.45.2bugfix: sample_selection in EXPL was required wronglyVersion 0.45.2added sample_selection for sample distribution plotsVersion 0.45.1fixed dataframe.append bugVersion 0.45.0added auddim as featuresadded FEATS store_formatadded device use to feat_audmodelVersion 0.44.1bugfixesVersion 0.44.0added scatter functions: tsne, pca, umapVersion 0.43.7added clap featuresVersion 0.43.6small bugsVersion 0.43.5because of difficulties with numba and audiomentations importing audiomentations only when augmentingVersion 0.43.4added error when experiment type and predictor don't matchVersion 0.43.3fixed further bugs and added augmentation to the test runsVersion 0.43.2fixed a bug when running continuous variable as classification problemVersion 0.43.1fixed test_runsVersion 0.43.0added augmentation module based on audiomentationVersion 0.42.0age labels should now be detected in databasesVersion 0.41.0added feature tree plotVersion 0.40.1fixed a bug: additional test database was not label encodedVersion 0.40.0added EXPL section and first functionalityadded test module (for test databases)Version 0.39.0added feature distribution plotsadded  plot formatVersion 0.38.3added demo mode with list argumentVersion 0.38.2fixed a bug concerned with "no_reuse" evaluationVersion 0.38.1demo mode with file argumentVersion 0.38.0fixed demo modeVersion 0.37.2mainly replaced pd.append with pd.concatVersion 0.37.1fixed bug preventing praat feature extraction to workVersion 0.37.0fixed bug cvs import not detecting multiindexVersion 0.36.3published as a pypi moduleVersion 0.36.0added entry nkululeko.py scriptVersion 0.35.0fixed bug that prevented scaling (normalization)Version 0.34.2smaller bug fixed concerning the loss_stringVersion 0.34.1smaller bug fixes and tried Soft_f1 lossVersion 0.34.0smaller bug fixes and debug ouputsVersion 0.33.0added GMM as a model typeVersion 0.32.0added audmodel embeddings as featuresVersion 0.31.0added models: tree and tree_regVersion 0.30.0added models: bayes, knn and knn_regVersion 0.29.2fixed hello world exampleVersion 0.29.1bug fix for 0.29Version 0.29.0added a new FeatureExtractor class to import external dataVersion 0.28.2removed some Pandas warningsadded no_reuse function to database.load()Version 0.28.1with database.value_counts show only the data that is actually usedVersion 0.28.0made "label_data" configuration automatic and added "label_result"Version 0.27.0added "label_data" configuration to label data with trained model (so now there can be train, dev and test set)Version 0.26.1Fixed some bugs caused by the multitude of feature setsAdded possibilty to distinguish between absolut or relative pathes in csv datasetsVersion 0.26.0added the rename_speakers funcionality to prevent identical speaker names in datasetsVersion 0.25.1fixed bug that no features were chosen if not selectedVersion 0.25.0made selectable features universal for feature setsVersion 0.24.0added multiple feature sets (will simply be concatenated)Version 0.23.0added selectable features for Praat interfaceVersion 0.22.0added David R. Feinberg's Praat features, praise also to parselmouthVersion 0.21.0Revoked 0.20.0Added support for only_test = True, to enable later testing of trained models with new test dataVersion 0.20.0implemented reuse of trained and saved modelsVersion 0.19.0added "max_duration_of_sample" for datasetsVersion 0.18.6added support for learning and dropout rate as argumentVersion 0.18.5added support for epoch number as argumentVersion 0.18.4added support for ANN layers as argumentsVersion 0.18.3added reuse of test and train file setsadded parameter to scale continous target values: target_divide_byVersion 0.18.2added preference of local dataset specs to global onesVersion 0.18.1added regression value display for confusion matricesVersion 0.18.0added leave one speaker group outVersion 0.17.2fixed scaler, added robustVersion 0.17.0Added minimum duration for test samplesVersion 0.16.4Added possibility to combine predictions per speaker (with mean or mode function)Version 0.16.3Added minimal sample length for databasesVersion 0.16.2Added k-fold-cross-validation for linear classifiersVersion 0.16.1Added leave-one-speaker-out for linear classifiersVersion 0.16.0Added random sample splits