Package: resurfemg
ReSurfEMGis an open source collaborative python library for analysis
of respiratory electromyography (EMG). On the same site as
the repository for this library we keeprelated resources.ReSurfEMG includes amain code librarywhere the user can access the code to change various filter and analysis settings directly and/or in ourresearcher interface notebooks.
In addition, ReSurfEMG has adashboard interfacewhich contains default settings for preprocessing and analysis which can be changed through a graphical (no code) interface. We have some functionality available through acommand line interfaceas well.The library was initially
built for surface EMG, however many functions will also work for
invasively measured respiratory EMG.  This library
supports the ongoing research at University of Twente on respiratory
EMG.Program filesThe core functions of ReSurfEMG are in the folderresurfemg:cli:Scripts for the command line interfaceconfig:Configure all paths for data analysisdata_connector:Converter functions for various hardware/software and the TMSisdk lite modulehelper_functions:General functions to support the functions in this repositorymachine_learning:Run machine learning algorithms on arrayspost-processing:Calculate features from the respiratory data:entropyarea under curvetime under curveslopepeak in breathvariabilitySampEn adapted from nolds packagepre_processing:Process the raw EMG signalecg-removal: ICA and gatingenvelope: RMS and smoothersfiltering: cutters, low-, high- and bandpass, notchfilter, computer power lossvisualization:Show powerspectrumFolders and NotebooksOurguide to notebooksis under construction. To look around keep in mind the following distinction on folders:researcher_interface:These are a growing series of interactive notebooks that allow
researchers to investigate questions about their own EMG data⚡ Important: in almost all data there will be a time
difference between EMG signals and ventilator signals. You can
pre-process to correct for this lead or lag with the notebook
called lead_lag_mismatch_upsample.open_work:This folder contains experimental work by core members of the ReSurfEMG
team (Dr. Eline Mos-Oppersma, Rob Warnaar, Dr. Walter Baccinelli and Dr. Candace Makeda Moore)Data setsThe notebooks are configured to run on various datasets.  Contact
Dr. Eline Mos-Oppersma( 📫e.mos-oppersma@utwente.nl) to discuss any
questions on data configuration for your datasets.If you want to use a standardized dataset for any purpose we reccomend
the data in the ReSurfEMG/synthetic_data repositoryData there can be used with any respiratory EMG algorithms in any program. Thus that data can function as a benchmarking set to compare algorithms across different programs.Configuring (to work with your data)In order to preprocess and/or to train  models the code needs to be
able to locate the raw data you want it to find.There are several ways to specify the location of the following
directories:root_emg_directory:Special directory.  The rest of the directory layout can
be derived from its location.preprocessed:The directory that will be used by preprocessing
code to output to.models:The directory to output trained models to.You can store this information persistently in several locations.In the same directory where you run the script (or the notebook).
e.g../config.json.In home directory, e.g.~/.resurfemg/config.json.In global directory, e.g./etc/resurfemg/config.json.However, we highly recommend you use the home directory.
This file can have this or similar contents:{
    "root_emg_directory": "/mnt/data",
    "preprocessed": "/mnt/data/preprocessed",
    "models": "/mnt/data/models",
    "output": "/mnt/data/output",
}The file is read as follows: if the files specifiesroot_emg_directorydirectory, then the missing entries are assumed to be relative to
the root.  You don't need to specify all entries.Test dataYou can get test data by extracting it from the Docker image like
this:mkdir-pnot_pushedcd./not_pushed
dockercreate--nametest-datacrabbone/resurfemg-poly5-test-files:latest
dockercptest-data:/ReSurfEMG/tests/not_pushed/..
dockerrm-ftest-dataSupported PlatformsReSurfEMG is a pure Python package. Below is the list of
platforms that should work. Other platforms may work, but have had less extensive testing.
Please note that where
python.org Python or Anaconda Python stated as supported, it means
that versions 3.8 or 3.9 (depending on the release) are supported.AMD64 (x86)LinuxWinOSXSupportedUnknownUnknownSupportedSupportedSupportedInstallation for all supported platformsInstallation with Anaconda/conda and/or mamba are the preffered methods.
They are covered inthe "Getting Started" section.
If you wish to install with pip:Create and activate a virtual environment (see developer setup section for more details)Install ResurfEMG package by runningpip install resurfemg.Getting Startedwith the reccomended Conda setupHow to get the notebooks running?  Assuming the raw data set and
metadata is available. Note for non-conda installations see next sections.Assuming you are using conda for package management:Make sure you are in no environment:condadeactivate(optional repeat if you are in the base environment)You can build on your
base environment if you want, or if you want to not use option A, you can go below it (no environment)Option A: Fastest option:
In a base-like environment with mamba installed, you can install all Python packages required, usingmambaand theenvironment.ymlfile.If you do not have mamba installed you can follow instructionshereThe command for Windows/Anaconda/Mamba users can be something like:mambaenvcreate-fenvironment.ymlOption B: To work with the most current versions with the possibility for development:
Install all Python packages required, usingcondaand theenvironment.ymlfile.The command for Windows/Anaconda users can be something like:condaenvcreate-fenvironment.ymlLinux users can create their own environment by hand (use
install_dev as in setup).Make sure to enter your newly created environment.Option C: In theory if you want to work, but never develop (i.e. add code), as a conda user
with a stable (released) version create an empty environment, and install
there:Create a blank environment with python pinned to 3.8 (assuming version < 0.1.0):condacreate-nblankpython=3.8Install within the blank environmentcondaactivateblankcondainstall-cconda-forge-cresurfemgresurfemgjupyteripymplOpen a notebook (we useJupyter
notebooks)
in researcher_interface folder and interactively run the cells.
You can use the commandjupyter notebookto open a browser window
on the folders of notebooks.  Note, if you run with an installed
library import appropriately. Thebasic_emg_analysisnotebook can
be used to understand how to use the package.Advanced contributor's setup / "Developer's setup"We distinguish between people who want to use this library in
their own code and/or analysis and people who also want to develop this library who we call developers, be it as
members of our team or independent contributors.  People who
simply want to use our library need to install the packaged version
from one of the package indexes to which we publish released versions
(eg. PyPI).  This section of the readme is for advanced developers who want to
modify the library code (and possibly contribute their changes back or eventually publish thier own modified fork). NB: you can accomplish modifications of the code, submit PRs and soforth without
a 'developer's setup' but we feel this setup will make advanced contributions easier.We have transitioned to a fully Python 3.9 environment.  The
instructions below are for our newer versions above 0.1.0:
(For older instructions withvenvplease see versions below 0.1.0, and
adapt them if using Windows and/or a different Python version than
Python.org Python e.g. you may need to use.venv/Scripts/activatein
place of.venv/bin/activate.  This will create a distributable
package from the source code, then install it in the currently active
environment.  This will also install development tools we use
s.a.pytestandcodestyleand will also install tools we use for
working with the library, s.a.jupyter.)After checking out the source code, create virtual environment.  Bothcondaandvenvenvironments are supported, however, if you are on
Windows, we reccomend usingconda.Using Anaconda Pythoncondacreate-nresurfemgpython=3.9
condaactivateresurfemg
pythonsetup.pyanaconda_gen_meta
pythonsetup.pyinstall_devNote, you will need to runanaconda_gen_meta.  This generatesmeta.yamlwhich is necessary to createcondapackage.  In the
future this will probably be called automatically byinstall_dev.Using PyPI Pythonpython3.9-mvenv.venv3.9# On Linux:..venv3.9/bin/activate# On Windows:.venv3.9/Scripts/activate
pythonsetup.pyinstall_devNow you should have everything necessary to start working on the
source code.  Whenever you make any changes, re-runinstall_devto
see them applied in your environment.  It's possible to find a
shortcut sometimes to running the entire cycle (install_devtakes a
long time to run).  Look at the source ofbdist_condaandsdist_condacommands insetup.pyfor more info.Generating documentationOnline documentation can be found athttps://resurfemg.github.io/ReSurfEMG/or onhttps://readthedocs.org/by searching for ReSurfEMG.
Up to date documentation can be generated in command-line as follows
(in bash terminal):python3-mvenv.venv
../.venv/bin/activate
pipinstallwheelsphinx
./setup.pyinstall
./setup.pyapidoc
./setup.pybuild_sphinxIf you are working in a VScode command line interface (terminal cmd)
should be more or less something like the following:This is given withcmd.exein mind:python3-mvenv.venv
.venv/bin/activate
pipinstallwheelsphinx
pythonsetup.pyinstall
pythonsetup.pyapidoc
pythonsetup.pybuild_sphinxAutomationThe project comes with several modifications to the typical
defaultsetup.py.At the moment, the support for Anaconda Python is lacking.  The
instructions and the commands listed below will work in Anaconda
installation but due to the difference in management of installed
packages, the project will be installed into base environment
regardless of the currently active one.  We are planning to integrate
with Anaconda in near future.The project has a sub-project of a related dashboard.  Dashboard is a GUI that
exposes some of the project's functionality. In the past we kept a a legacy dashboard
in the same repository with ReSurfEMG code but we have deleted it. The
current version of the dashboard into it's own repository:https://github.com/ReSurfEMG/ReSurfEMG-dashboardNew commandsCommands that perform repeating tasks have a--fastoption.  Use
this if you ransetup.py install_dev, and you are sure the
dependencies are up to date.  Otherwise, these commands will create a
new virtual environment and install necessary dependencies there
before execution.  This is primarily intended for use in CI to create
controlled environment.Please note that documentation is built usingsphinxcommand
forsetuptools:setup.py build_sphinx, butsphinxis not
installed as part of development dependencies, rather it is declared
as a dependency ofsetup.pyitself.  There are cases whensetup.pywill not install its own dependencies.  You are advised to install
them manually.setup.py lintchecks that the source code is formatted according to
PEP-8 recommendations.setup.py isortchecks that the imports are properly formatted and
sorted.setup.py apidocgenerates RST outlines necessary to generate
documentation.setup.py install_devinstalls dependencies necessary for
development.Modified commandssetup.py testthe original command is overloaded with a new one.
Similarly to most of the new commands, this command takes--fastas an option with the same meaning.  Unlike its predecessor, this
command will create a clean environment and install all necessary
dependencies before running the test (the original did install
dependencies, but ran the test in the source directory, this made it
impossible to test code that relied on being installed).Installing from sourceThe traditional way to install from source is to runsetup.py installorsetup.py develop.  Both of these will
work... sort of.  This is because of the default behavior inherited fromsetuptools.  The problem here is that instead of creating a
distributable package and installing that,setuptoolsdoes it
in the other order: it installs the package in order to create a distributable
one.As was already mentioned earlier, Anaconda Python will need better
support, at which point,setup.py installwill have to change to
enable that.We are not planning on patchingsetup.py developas we don't believe
it is a good practice to use this command.  It is not removed,
however, and should work in the same way it would work with a typicalsetuptoolsproject.Note thatpip install -e .andpip install -e '.[dev]'are
discouraged by association (since that is just a wrapper aroundsetup.py develop.)  Similarly tosetup.py developthey might work,
but you have to be careful with interpreting the results.  If those
don't work, it's on you.TestingThe project doesn't include testing data.  It was packaged into a Docker
image which can be found atcrabbone/resurfemg-poly5-test-files:latest.
This test data was created by taking a signal from equipment, not a human,
and certainly not a patient.It is possible to run tests in container created from this image.
Alternatively, you may download the image and extract directory/ReSurfEMG/tests/not_pushedintonot_pushedin the root of the
project and run:pythonsetup.pytestBelow is a snippet that may help you to run the tests in a container:dockerrun--rm-v$(pwd):/ci\--cap-add=SYS_ADMIN\--privileged=true\crabbone/resurfemg-poly5-test-files\sh-c'set -xecd /cimkdir -p ./not_pushed/mount --bind /ReSurfEMG/tests/not_pushed/ ./not_pushed/python setup.py test'Command-Line InterfaceYou will be able to preprocess, train and use models using command-line interface.
You can also, in some cases, create files in the correct format for our Dashboard
in a per folder batch process.Below is an example of how to do that:This will pre-process (with the alternative_a_pipeline_multi algorithm) the
Poly5 files in the/mnt/data/originalsdirectory, and output leads 1 and 2 preprocessed.
(Note the \ symbol is simply a line-break and not meant to be included.)python -m resurfemg acquire \
       --input /mnt/data/originals \
       --lead 1 --lead 2 \
       --output /mnt/data/preprocessed \
       --preprocessing alternative_a_pipeline_multi \The following will run an ML model over all files:python -m resurfemg ml |
        --input /mnt/data/preprocessed \
        --output /mnt/data/ml_output \
        --model  ml_models/finalized_lr_model_in_111.sav \You can also make synthetic data. To explore this start withpython -m resurfemg synth --helpYou can also make from horizontally formated csv files
that can be read by the dashboard. To explore this start withpython -m resurfemg save_np --helpThe help commandis also available for ml and acquire.All long options have short aliases.✨Copyright 2022 Netherlands eScience Center and U. Twente
Licensed under the Apache License, version 2.0. See LICENSE for details.✨