Package: heartfelt-tools
MIMIC-III Database ToolsFor extracting and cleaning ppg and abp data from the MIMIC-III Waveforms Database.Table of ContentsIntroductionGetting StartedPrerequisitesUsagePoetryGet Valid RecordsBuild DatabaseEvaluate DatasetGenerate RecordsRead RecordsLicenseIntroductionThis repo contains a set of tools for extracting and cleaning photoplethysmography (ppg) and artial blood pressure (abp) waveforms from theMIMIC-III Waveforms Databasefor the purpose of blood pressure estimation via deep learning.(back to top)Getting StartedThis sections details the requirements to start using this library. Links are for Ubuntu installation.PrerequisitesPythonsudoaptinstallpython3.8-y
sudoaptinstallpython3.8-devpython3.8-venv-yecho'export PATH="$PATH:/home/ubuntu/.local/bin"'>>~/.bashrcsource~/.bashrc

curl-sShttps://bootstrap.pypa.io/get-pip.py|python3.8
python3.8-mpipinstallvirtualenv
python3.8-mvenv.venv/base-envecho'alias base-env="source ~/.venv/base-env/bin/activate"'>>~/.bashrc
base-env

python3.8-mpipinstall--upgradepipPoetrycurl-sSLhttps://install.python-poetry.org|python3-echo'export PATH="$PATH:$HOME/.local/bin"'>>~/.bashrcsource~/.bashrc# Verify installationpoetry--version(back to top)UsagePoetryThe commands below can be used to install the poetry environment, build the project, and activate the environment.cddatabase-tools
poetrylock
poetryinstall
poetrybuild
poetryshellCreate Data DirectoryThe functions in this library rely on a data folder named with the conventiondata-YYYY-MM-DD. This directory contains two additional folders,mimic3/andfigures/. Themimic3/lines/folder is intended to hold the jsonlines files the data will initially saved to. Themimic3/records/folder will hold the TFRecords files generated from these jsonlines files. This will be discussed in greater depth in theGenerate Recordssection.Get Valid RecordsThe class DataLocator (located indatabase_tools/tools/) is specifically written to find valid data files in the MIMIC-III Waveforms subset and create a csv of the html links for these data files. Performing this task prior to downloading is done to improve runtime and the usability of this workflow. Valid records refers to data files that contain both PPG and ABP recordings and are at least 10 minutes in length. Currently this code is only intended for the MIMIC-III Waveforms subset but will likely be adapated to allow for valid segments to be identified in the MIMIC-III Matched Subset (records are linked to clinical data). To perform an extraction the filescripts/get-valid-segs.pycan be run (data directory and repository path must be configured manually). This function will output a csv calledvalid-segments.csvto the data directory provided. The figure below shows how these signals are located.Add mimic3 valid segs logic figure.Build DatabaseThe classBuildDatabase(located indatabase_tools/tools/) downloads data fromvalid-segments.csv, extracts PPG and ABP data, and then processed it by leveraging theSignalProcessorclass (located indatabase_tools/preprocessing/). A database can be build by runningscripts/build_database.py(be sure to configure the paths). BuildDatabase takes a few important parameters which modify how signals are excluded and how the signals are divided prior to processing. Thewin_lenparameter controls the length of each window,fsis the sampling rate of the data (125 Hz in the case of MIMIC-III), whilesamples_per_file,samples_per_patient, andmax_samplescontrol the size of the dataset (how many files the data is spread across, how many samples a patient can contribute, and the total number of samples in the dataset. The final parameterconfigcontrols the various constants of the SignalProcessor that determine the quality threshold for accepting signals. The SignalProcessor filters signals according to the figure chart below. The functions used for this filtering can be found indatabase_tools/preprocessing/. Data exctracted with this script is saved directly to themimic3/lines/folder in the data directory. A file namedmimic3_stats.csvcontaining the stats of every processed waveform (not just the valid ones) will also be saved to the data directory.Add data preprocessing figure.Evaluate DatasetThe classDataEvaluator(located indatabase_tools/tools/) reads themimic3_stats.csvfile from the provided data directory and outputs figures to visualize the statistics. These figures are saved directly to thefigures/folder in the data directory in addition to be output such that they can be viewed in a Jupyter notebook. The 3D histogram are generated using the fuctionhistogram3dlocated indatabase_tools/plotting/.Generate RecordsOnce data has been extracted TFRecords can be generated for training a Tensorflow model. The classRecordsHandlercontains the methodGenerateRecordswhich is used to create the TFRecords. This can be done usingscripts/generate_records.py(paths must be configured). When callingGenerateRecordsthe size of the train, validation, and test splits, as well as the max number of samples per file and a boolean to control whether or not the data is standardized must be specified (usingsklearn.preprocessing.StandardScaler().Read RecordsThe classRecordsHandleralso contains the functionReadRecordswhich can be used to read the TFRecords into a TensorflowTFRecordsDatasetobject. This function can be used to inspect the integrity of the dataset or for loading the dataset for model training. The number of cores and a TensorFlowAUTOTUNEobject must be provided.(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)