Package: zerospeech-benchmarks
Zero Resource Challenge Benchmark ToolkitThis repository contains a toolbox assisting in running and handling all things related to the zerospeech benchmarks.
For more information on theZero Resource Challenge you can visit our website.This toolbox allows to download all resources linked with the benchmarks (datasets, model checkpoints, samples, etc..),
to run the various benchmarks on your own submissions, and to upload the results to our website for them to be included in our leaderboards.The available resources can be also found directly on our repositorydownload.zerospeech.comInstallationThe zerospeech benchmark toolbox is a python package, so you require a version of python installed on your system
before you can start.You can useminicondaa
lightweight version of anaconda or any other way of installing python you prefer.Note that the package has been tested on python 3.8+, other versions are not recommended.Once python is installed you can install the package using :pip install "zerospeech-benchmarks[all]"If you are a conda user you can use our prepackaged environment  :conda env create coml/zrc-toolkitTo verify that the toolbox is installed correctly you can tryzrc versionwhich should print
the version information. If this is not the case you can open an issue with your errors on directly on ourgithubToolbox UsageDownloadsInstallation location can be specified using the environment variableAPP_DIR, in linux & macOS this can
be done in a terminal:$ export APP_DIR=/location/to/dataBy default, all data is saved in$HOME/zr-dataAll temporary files are saved in/tmpthis can be changed using the environment variableTMP_DIRDownload benchmark datasetsYou can start by listing the available datasets using the commandzrc datasetsthen you can download the
dataset you want using the commandzrc datasets:pull [dataset-name].When listing datasets theInstalledcolumn specifies whether the dataset has been downloaded.Datasets are installed in the$APP_DIR/datasetsfolder.
To delete a dataset you can use the commandzrc dataset:rm [dataset-name]Download model checkpointsThe commandzrc checkpointsallows you to list available checkpoints.You can then download each set by typingzrc checkpoints:pull [name]Checkpoints are installed in the$APP_DIR/checkpointsfolder.To delete the checkpoints you can use the command :zrc checkpoints:rm [name]Download samplesThe commandzrc samplesallows you to list the available samples.You can then download each sample by typingzrc samples:pull [name].
Samples are installed in the$APP_DIR/samplesfolder.To delete a sample from your system you can usezrc samples:rm [name]BenchmarksYou can list available benchmarks by typing thezrc benchmarkscommand.To create a submission you have to follow the instructions on each of our task pagesTask1,Task2,Task3,Task4Some older benchmarks may not be available straight away, but they will be added as soon as possible.Once the submission has been created you can run the benchmark on it
with the following command :zrc benchmarks:run [name] [/path/to/submission] [...args]Some benchmarks are split into sub-tasks you can run partial tasks by using the following syntax:zrc benchmarks:run sLM21 [/path/to/submission]  -t lexical syntacticWith this syntax we run the sLM21 benchmark our submission but only for the lexical and syntactic task and we omit the semantic.In the same way we can also only run on the dev set (or the test) :zrc benchmarks:run sLM21 [/path/to/submission] -s dev -t lexical syntacticWe run the same tasks as previously but only on the dev set of the benchmark.For information on each benchmark you can run thezrc benchmarks:info [name]command or visit the corresponding section on our websitezerospeech.comSubmission FormatEach benchmark has a specific format that a submission has to follow, you can initialize a
submission directory by using the following syntax :zrc submission:init [name] [/path/to/desired-location-for-submission], this will
create a set of folders in the architecture corresponding to the benchmark name selected.
For more detailed information on each benchmark you can see each Task page respectively.Once all submission files have been created your can validate your submission to see if everything is working properly.
To do so use the following syntax :zrc submission:verify [name] [/path/to/submission]this will verify that all files
are set up correctly, or show informative errors if not.Note:During benchmark evaluation the default behavior is to run validation on your submission, you can deactivate this by adding
the option--skip-verification.SubmitThe submit functionality allows uploading scores to our platform,
this helps us keep track of new models or new publications that happen using the benchmarks also we compile all the
scores into our leaderboards to be able to compare them.The submit functionality is a Work in progress, it will be available soon.