Package: heareval
hear-eval-kitEvaluation kit for theHEAR Benchmarkusing tasks fromhear-preprocessand audio embedding models that follow theHEAR API.Downstream evaluation on each task involves two
steps:computing audio embeddingslearning a shallow fully-connected predictorThe first step's speed depends upon a variety of factors.
The second step's speed is relatively similar between models.If you have any questions or comments:File anissue.Email us (deep at neuralaudio dot ai).RequirementsTested with Python 3.7 and 3.8. Python 3.9 is not officially supported
because pip3 installs are very finicky, but it might work.We officially support Torch 1.9 and Tensorflor 2.6.0, as well as
Tensorflow 2.4.2 using the hack described in theDockerfile
README. We use CUDA 11.2. Other versions are
possible, please contact us.We test on 16GB GCP GPUs.QuickstartHere is a simple example to evaluate thehearbaseline wav2vec2audio embedding model on theMridingam Tonictask, which is a classification
task using sounds from a pitched percussion instrument called a Mridingam.This example shows how to compute embeddings on a pre-processed version of the data, and
then learn a shallow prediction model on the embeddings for evaluation.InstallationThere are 3 ways to runheareval:Locally, through pip3 install (or conda)Using DockerOn the cloudYou are welcome to contact us if you have any questions or issues.Local installationpip3 install hearevalDockerWe have docker images containing thehearevalenvironment.turian/heareval:stablecontains the latest stable image with all
dependencies bundled in.Cloud GPUsThe easiest way to do evaluation is to launch a Spotty GCP instance.
You can easily adapt Spotty also for AWS GPU instances.Prepare aspotty.yamlfile with the provided template file:cp spotty.yaml.tmpl spotty.yamlChange the instance name in the copied file. Specifically, change"USERNAME"suffix ininstances: nameto allow for multiple users in the same project
to make separate gcp instances and volumes to avoid conflicts within the project.Run spotty:spotty start
spotty shThis requires the heareval Docker image, which is pre-built and
published on Dockerhub for your convenience.Please refer toREADME.spottyfor more details.Downloading TasksWe've released pre-processed versions of all HEAR 2021 open and secret tasks on Zenodo,
you can access those here:https://doi.org/10.5281/zenodo.5885750Note on Sample Rate:All the tasks hosted on Zenodo have been pre-processed to
48kHz. If the embedding model that you are using requires a different sample rate,
then you will need to resample the audio to that rate before running.For other sampling rates (16000, 22050, 32000, 44100), please download
files (requester pays) from Google Storagegs://hear2021-archive/tasks/Alternatively, you
can generate the pre-processed datasets usinghear-preprocess.Compute embeddingstime python3 -m heareval.embeddings.runner MODULE_NAME --model WEIGHTS_FILE --tasks-dir hear-2021.0.3/tasks/whereMODULE_NAMEis your embedding model name.This will create directoriesembeddings/MODULE_NAME/TASK/with
your embeddings. If you run the above command multiple times, it
will skip tasks it has already performed embedding on. You can
delete directories if you want to recompute embeddings.There is an advanced option--model-optionswhereby you can pass
a JSON string of parameters to the model. This is useful for
experimenting with model hyperparameters. These options appear in
the embeddings output directory name, so you can run several different
model variations at once.Evaluation over embeddingsYou can then run final downstream evaluation on these embeddings as follows:python3 -m heareval.predictions.runner embeddings/{MODULE_NAME}/*This will run on a particular module, over all tasks, with determinism
and the default number of grid points. Embeddings will be loaded
into CPU memory, for speed of training.
Logs will be sent to stdout and concise logs will be inlogs/.
If you run this multiple times, it should be deterministic, but will
always start from scratch.Ignore warnings aboutLeaking Caffe2 thread-pool after fork, this
is a known torch bug.More advanced flags allow different downstream training regimesFinal test scores are logged to stdout and also to{EMBEDDINGS_DIR}/{MODULE_NAME}/{TASK_NAME}/test.predicted-scores.json.Note on SpeedModels with larger embeddings scale sub-linearly in training time
(because of GPU optimizations) and linearly in hop-size (for
event-based prediction tasks). The main hyperparameters controlling
downstream training time are the maximum number of epochs and number
of grid points for grid search.DevelopmentIf you are developing this repo, clone repo:git clone https://github.com/hearbenchmark/hear-eval-kit
cd hear-eval-kitInstall in development mode:pip3 install -e ".[dev]"Make sure you have pre-commit hooks installed:pre-commit installRunning tests:python3 -m pytestNOTE: Please make sure the workflows for each of the open task (./gihub/workflows/task-{task_name}.yml) is using the correct version of preprocessed tasks from thePreprocessed Downsampled HEAR Open
TasksRepo
for Continuous Integration.Current hearpropress version used for Continuous Integration -2021.0.6Please keep the version in sync with hearpreprocess