Package: bicycle-bell-seds-cli
Bicycle Bell Sound Event Detection SystemAuthor: Clemens KubachThis repository is one of three for my bachelor thesis on "Development of an Embedded System
for Detecting Acoustic Alert Signals of Cyclists Using Neural Networks".It contains the software as an easy-to-use and customizable CLI for the project.
Only this part has to be installed on the target device and can be used and
developed independently of the other components.A trained saved model can be selected, which is then converted to an inference format
(TFLite or TF-TRT), allowing real-time predictions to be made to a single sound event for
live-streamed audio via connected sound devices.The other related repositories are:bicycle-bell-sed-modelsbicycle-bell-sed-pipelineGetting StartedThe software is based on thePortAudiolibrary for audio I/O.
Therefore, this must be installed on the system.
For more detailed installation instructions on an embedded device like theJetson Nano, see the corresponding chapter.apt-getupdate
apt-getinstallportaudio19-dev

pipinstall--upgradepip
pipinstallbicycle-bell-seds-cli

seds-cli--help
seds-clirun--tfmodel='!crnn'productionThere are generally 4 main functionalities that are displayed withseds-cli --help.conversioncan convert recordings of a pre-executed run command with appropriate
parameterization for sound recording to a wave file.devicescan be used for testing the available devices by doing a sound check.resourcescan be used for find the location of resource files like log-files or recordings.runis the main functionality of the software.
This command is used to start a sound event detection.General InformationGenerally, two versions of the CLI are installed:jn-seds-cliandseds-cli.
The first one is based on the second one and only contains simplifications and specifications for
the execution of the bicycle bell detection on the Jetson Nano.
With the right choice of parameters, however,
both CLIs can be used on all devices without any problems.
Details about the differences can be found viajn-seds-cli run --help. In the following, thejn-seds-cliversion will be used for an easier copy-and-paste usage on the Jetson Nano as
target device.Please use--helpfor detailed explanations of the individual software
functionalities and parameters.
With this you can get help for each level, i.e.:jn-seds-cli --help,jn-seds-cli run --help,jn-seds-cli run evaluation --help.Usage ExamplesShow the location of the resources' folder:jn-seds-cliresourceswhereMake a sound check of the audio devices:jn-seds-clidevicessoundcheckStart a sound event detection with saving the logs to a file in the resources' folder and
record the first minute of the received audio input stream:jn-seds-clirun--tfmodel='!crnn'production--save_log=True--save_records=True--storage_length=60Convert the recorded file of the previous run into a wave file:jn-seds-cliconversionrecord_to_wav--path_storage_pickle="/abs/path/to/seds_cli/res/records/record-xx.pickle"--target_wav_path="./target_filepath/filename.wav"Run CommandThere are two different modes with the run command: production and evaluation.
The production mode is the main mode and receives live the current sound of the environment
through the selected microphone device.
The evaluation mode can play a recorded wave file with a corresponding annotation csv file and
displays the ground-truth value as well as the prediction for the live microphone recordings.Most parameters for the run command are available for both modes.
Mode specific parameters can be found via--helpfor the selected mode.
The following flags are used for the production mode, but are available for the evaluation mode too.There are three models pre-defined and pre-trained for direct usage for
detecting bicycle bell sounds. They can be chosen by using--tfmodel="!model-name"without any
further specifications ofsaved_modeltype or the absolute path to the saved model resource
intfmodel. Available are!crnn,yamnet_baseandyamnet_extended.Select the predefined CRNN model via!crnn, and run the production mode without displaying
the probability value of the predictions. The logs will be saved to a file:jn-seds-clirun--tfmodel='!crnn'production--save_log=True--prob_logging=FalseVia specifying an integer forinput_device, notNone, a specific (not default) sound device
can be selected.
Use the extended YAMNet model in a production run and define that the selected (default)
input device only has one input channel:jn-seds-clirun--tfmodel='!yamnet_extended'production--channels=1Use the base YAMNet model in a production run with a lower threshold as default and activate the
logging of the probabilities to see that every prediction from a value of 0.3 will True:jn-seds-clirun--tfmodel='!yamnet_base'production--threshold=0.3--prob_logging=TrueMode specific flag-usage examples...For the evaluation mode, use the first option for a random test example or specify an own test:jn-seds-clirun--tfmodel='!crnn'evaluation--save_log=True--silent=False# Orjn-seds-clirun--tfmodel='!crnn'evaluation--save_log=True--wav_file="/path/to/wave.wav"--annotation_file="/path/to/annotations.csv"--silent=FalseFor playback:jn-seds-clirun--tfmodel='!crnn'production--save_log=True--use_output=TrueAdvanced UsagePlease note that in order to use the gpu, an appropriately compatible TensorFlow build must be
installed and used with--gpu=True.
In addition, the inference model type must be set to use
TensorFlow-TenorRT via--infer_model=tftrt, depending on the specific machine.
In some cases, also TFLite can be used with gpu support.
Unfortunately, TF-TRT could not yet be tested thoroughly because compatible devices or
software dependencies were not available.
For further information, readrun --helpunder the related parameters.New trained models can be used via--tfmodel="/path/to/tf-savedmodel-dir savedmodel=crnn.
The best way, is to modify the source code in the filesaved_models.pyand create a new child class of BaseSavedModel or Mono16kWaveInputSavedModel.
Then create an entry for this class in selectors.py, thus custom preprocessing and postprocessing for the model can be defined.
An easier way without modifying code is to use a saved model with the currently support interface of mono 16 kHz waveform input.
If so, it can easily be used via--tfmodel="/path/to/tf-savedmodel-dir --saved_model=MONO_16K_IN.
Please note that this feature has not yet been thoroughly tested.Jetson Nano SetupThe following explanation based on the latest stable version of the JetPack OS (4.6.1) for the
Jetson Nano at the time of writing.
The use of the future version JetPack 5.0 is expected to resolve the installation issues with
Tensorflow_io and thus possibly allow support of the GPU and Tensorflow-TRT on the Jetson Nano.
However, without development intentions, the here documented version 4.6.1 of JetPack
should be used for now.Make sure that JetPack 4.6.1 has been installed!apt-getupdate
apt-getinstallportaudio19-dev

sudoapt-getupdate
sudoapt-getinstalllibhdf5-serial-devhdf5-toolslibhdf5-devzlib1g-devziplibjpeg8-devliblapack-devlibblas-devgfortran
sudoapt-getinstallpython3-pip

mkdir./venv
python3-mvenv./venvsourcevenv/bin/activate

pipinstall-Upiptestresourcessetuptools==49.6.0pipinstall-U--no-depsnumpy==1.19.4future==0.18.2mock==3.0.5keras_preprocessing==1.1.2keras_applications==1.0.8gast==0.4.0protobufpybind11cythonpkgconfig
envH5PY_SETUP_REQUIRES=0pipinstall-U--no-build-isolationh5py==3.1.0
pipinstall--pre--extra-index-urlhttps://developer.download.nvidia.com/compute/redist/jp/v461'tensorflow>=2'python-c"from tensorflow.python.client import device_lib; device_lib.list_local_devices()"PIP_EXTRA_INDEX_URL=https://snapshots.linaro.org/ldcg/python-cachepipinstalltensorflow_io
python-c"from tensorflow.python.client import device_lib; device_lib.list_local_devices()"pipinstallbicycle-bell-seds-cli

jn-seds-cli--help
jn-seds-clirun--tfmodel='!crnn'productionIt is expected that aftertensorflow_ioinstallation no gpu will be detected.
This is because this build oftensorflow_iobrings a specific build oftensorflow(2.6) that does not
support gpus'.
With JetPack 5.0 and thus higher Python version (>3.6), more recent versions oftensorflow_iocan
be installed directly, for which there are also pre-build wheels for aarch64.Most of the installation steps for TensorFlow on the Jetson Nano are from
"Prerequisites and Dependencies"in corresponding the Nvidia Docs.Using Docker can save some on setup steps, but can also add some others.
If Docker should be used on the Jetson Nano:sudodockerrun-it--rm--runtimenvidia--networkhost-v/home/jetson:/home/jetson--device/dev/sndnvcr.io/nvidia/l4t-tensorflow:r32.7.1-tf2.7-py3DevelopmentFeel free to report bugs as issues and also contribute to the project.
Please contact me for this.
Especially the integration of new models and the full and tested integration of TF-TRT are still
outstanding points of improvement.
In addition, the SEDS-CLI will be offered completely separate from the bicycle bell sound event
in a further step and repository.Use the following steps to install directly from the GitHub repository and
do not forget to callgit lfs pullbefore running.
This will download the model data.apt-getupdate
apt-getinstallportaudio19-devgitgit-lfs

gitclonehttps://github.com/ClemensKubach/bicycle-bell-sed-software.gitcdbicycle-bell-sed-software
gitlfspull
pipinstall-e.