Package: wisconsinsc-cleaner
WisconsinSC_cleanerA tool to clean and uniform annotation files from Wisconsin Sleep Cohort (WSC), distributed by NSRRhere.MotivationThe Wisconsin Sleep Cohort is a great resource for studing longitudinal history of sleep disorders in a large population.However, this long time span comes at the price of some issues and inconsistencies in the annotation of events, making the life of researchers more difficult than necessary.
The WSC dataset uses also two different annotation formats (Twin and Gamma) that are slightly incompatible between each other.
For example, obstructive apneas may be formatted as: 'Obs Apnea', 'OBS Apnea', ' Obs Apnea ', 'Obst Apnea', 'OA', 'Apnea', 'Obst. Apnea'
Other issues relates to missing information in the annotations' columns, inconsistent time formats (most are 24h, some am/pm) and other dirty bits that are not necessary and complicate automatic parsing.How to use this scriptClone the repo or download thewsc_clean.pyandmappings.txtfile and run it from command line as:python wsc_clean.py <your_dataset_polysomnography_folder>The package can be also installed from PyPI:pip install wisconsinsc_cleanerand executed as:wsc_clean <your_dataset_polysomnography_folder>Content of this repoA single python script (no installation needed) parses all the annotation files and produce another set of annotation files with the suffix.uniform.txt.
The mapping of annotations is available in themappings.txtfile in the formA|B|C(see [https://zzz.bwh.harvard.edu/luna/ref/annotations/#remap] for details), meaning that every instance ofBorCwill be mapped asA. If a mapping does not exist, the original value is returned with a prefixmisc:.An extra text file includes all lines that were not mapped.If a recording uses the Twin format (allscore.txt files) the output is kept as one file.
If it uses the Gamma format (log.txt files) sleep stages and event scoring are merged together with the log.The code does not remove any existing annotation nor modify original files. However, some redundant information is ignored in Gamma logs (SeeKnown Issuesfile.)The script is entirely built on Python standard library and tested on Python v3.8.
The script is not optimized for efficiency and it parse recordings sequentially. Parsing 2570 recordings takes less than 10 minutes.Format of the outputThe.uniform.txtfile will have a columnar format (comma separated values) with a header:TimestampEventKeyDurationParam1Param2Param3hh:mm:ss.msStringsecondssee belowsee belowsee belowThe Duration and Param[1-3] depend on the type of eventsSleep stages, position and miscellaneaThey don't need extra information other than the event itself. Duration set to -1, Params to 0.
The duration is defined by the next event of the same typeSensor gain in Gamma filesgain:sensor_affected, -1, new gain value, channel affected, 0Respiratory eventsevent_key, Duration of the event in seconds, SpO2 minimum of the event [%], 0, 0Oxygen desaturationsevent_key, Duration of the event in seconds, SpO2 minimum of the event [%], SpO2 drop [%], 0Leg movements, arousals, ekg events, snore and any other without additional parametersevent_key, Duration of the event in seconds, 0, 0, 0Known issuesSeeKnown Issuesfile.ContributingPlease let us know if you encountered issues or bugs through github issues tracker.If you feel generous and this library helped your project: