Package: undr
Unified Neuromorphic Datasets RepositoryUnified Neuromorphic Datasets RepositoryGetting StartedInstall the undr moduleGenerate a default configuration fileDownload the datasetsGenerate a BibTex filePython moduleConceptsPython APIsDataset format specificationDataset mirrorsExample configurationApacheNginxUpload a datasetDownload with existing CLIBuild documentationcompilationGetting StartedInstall the undr modulepip3installundrDocumentation:https://neuromorphicsystems.github.io/undr/Generate a default configuration filepython3-mundrinitThe generatedundr.tomlfile is written in TOML (https://github.com/toml-lang/toml). It lists the datasets that will be downloaded or streamed, hence it needs to be ajusted to your needs.The linedirectory = 'datasets'specifies the directory where downloaded files are stored (relatively to the configuration file). All the files generated byundr(directory indexes, downloaded data, temporary files...) are stored in this directory.Datasets are listed as[[datasets]]entries with three mandatory properties:name,urlandmode. The optionalserver_typeproperty is used internally to speed up the download process. To discard a dataset, you can either remove it from the configuration file or comment all its lines with#signs.modechanges the download strategy on a per-dataset basis, with three possible values:'remote'only downloads the dataset's file index. TheundrPython package can be used to process the dataset files as if they were on your hard drive by streaming them from the server. This option is particularly useful for large datasets that do not fit on your disk but requires a fast internet connection since files are re-downloaded every time they are processed.'local'downloads all the dataset files locally but does not decompress them (most datasets are stored asBrotliarchives). TheundrPython library transparently decompresses files in memory when you read them, making this option a good trade-off between disk usage and processing speed.'local-decompressed'downloads all the dataset files locally and decompresses them. Decompressed files use a relatively inefficient plain binary file format so this option requires vast amounts of disk space (3 to 5 times as much as the Brotli archives). On the other hand, the plain binary format facilitates processing with other languages such as Matlab or C++.undralso supports hybrid configurations where only part of a dataset is downloaded or decompressed. You may also use local directories without a server. See [NOT DOCUMENTED YET] for details.Download the datasetspython3-mundrinstallThis command downloads the datasets file indexes. If themodeis'compressed'or'decompress', it also downloads the dataset files (and possibly decompresses them).This command can be interrupted at any time with CTRL + C. Re-running it will resume download where it left off.Generate a BibTex filepython3-mundrbibtex--outputdatasets.bibThe UNDR project does not claim authorship of the datasets. Please use this file to cite the origiinal articles.Python modulepip3installundrConceptspriority-index.jsonfilesother_filesPython APIsAPI nameComplexityConfigurabilityParallel processingProgress displayloopsimplehighnonomapsimplelowyesyestaskcomplexhighyesyesAll three approches support progress persistence. Progress persistence sightly increases the code complexity but makes it possible to resume processing after a network or power failure.Dataset format specification-index.jsonrationaleprepend a special character to make sure the index file is ordered first in ASCIIuse an unreserved URL character to avoid escaping problemsout of the unreserved URL characters (-/./_/~, seehttps://www.rfc-editor.org/rfc/rfc3986#section-2):.would result in a hidden file on UNIX systems_comes after alpha-numerical characters in ASCII~is a shortcut for the user's home directory in many shellsMany command-line programs treat-index.jsonas a flag, hence a command such ascat -index.jsonreturns an error. Prepending./to the filename avoids the problem:cat ./-index.json.Dataset mirrorsExample configurationApache<VirtualHost*:80>Alias//path/to/local/directory/<Directory"/path/to/local/directory/">RequireallgrantedOptions+Indexes</Directory></VirtualHost>To use another port, remember to edit/etc/apache2/ports.confas well.Nginxserver{listen80;location/{alias/path/to/local/directory/;autoindexon;sendfileon;tcp_nopushon;sendfile_max_chunk1m;}}Upload a datasetpython3 -m undr check-conformity /path/to/datasetCaveat: An UNDR server can provide multiple compressed files (different formats) for each resource. The Python UNDR library always picks the best compression (smallest encoded size).check-conformityonly checks the best compression and will not report errors for other compressions.for macOS users (.DS_Store)Add to~/.zshrc:# rmdsstore removes .DS_Store files recursivelyrmdsstore(){if[$#-eq0];thenprintf'usage: rmdsstore directory\n'>&2returnfifind"$1"-name".DS_Store"-delete-print}Runrmdsstore /path/to/datasetbefore runningpython3 -m undr check-conformity /path/to/dataset.Download with existing CLIwget--no-parent--recursive--level=infhttp://localhost:5432/dvs09/
find.-iname'\*.br'|whilereadfilename;dobrotli-d-j"$filename";done;Build documentation# setuppython3-mvenv.venvsource.venv/bin/activate
pythondocumentation/build.pypip install --upgrade pip setuptools
pip install -r documentation/requirements.txtcompilationcd python
pip install .
cd ..
sphinx-build documentation documentation/_build