Package: utime
U-Time & U-SleepOfficial implementation ofTheU-Time[1]model for general-purpose time-series segmentation.TheU-Sleep[2]model for resilient high-frequency sleep staging.This repository may be used to train both the original U-Time and newer U-Sleep models.
However, the repository has been significantly extended since[1]and may gradually
diverge from the version described in[2]. Earlier versions may be found at:U-Sleep paper version.Latest U-Time only version.U-Time paper version.ContentsOverviewSystem RequirementsInstallation GuideU-Sleep DemoFull Reproduction of U-SleepU-Time ExampleU-Time and U-Sleep ReferencesOverviewThis document describes the official software package developed for and used to create the free and public sleep staging systemU-Sleep[2].
U-Sleep is a fully convolutional deep neural network for automated sleep staging. A single instance of the model may be trained to perform accurate and resilient sleep staging
across a wide range of clinical populations and polysomnography (PSG) acquisition protocols.This software allows simultaneous training of U-Sleep across any number of PSG datasets using on-the-fly random selection of input channel configurations. It features a command-line interface for initializing, training and evaluating models without needing to modify the underlying codebase.Pre-trained U-SleepIn the following we will introduce the software behind U-Sleep in greater detail. Please note that:If you are interested to re-implement, extend, or train U-Sleep yourself e.g. on other datasets, you are at the right place!If you are looking to use our pre-trained U-Sleep model for automated sleep staging, please refer tohttps://sleep.ai.ku.dkand follow the displayed guide. See alsothis repositoryfor Python bindings to the webserver API.U-Time and U-Sleep - What's the Difference?This repository stores code for training and evaluating theU-Sleepsleep staging model. It builds upon and significantly extends ourU-Timerepository, published at NeurIPS 2019[1]. In the following, we will use the termU-Sleepto denote the resilient high frequency sleep staging model[2], andU-Timeto denote this repository of code used to train and evaluate the U-Sleep model.You can still use this repository to train the older U-Time model, seeU-Time Examplebelow.System RequirementsMinimal Hardware RequirementsUsing an already trained U-Sleep model for sleep staging may typically be done on any modern laptop (subject to the software requirements listed below).For training U-Sleep models from scratch, however, we highly recommend using a Linux based computer with at least the following hardware specifications:4+ CPU cores8+ GiB RAMSignificant physical storage space*1 CUDA enabled GPU (please refer tohttps://developer.nvidia.com/cuda-gpusfor a detailed list).It is possible to train the model on smaller machines, and without GPUs, but doing so may take considerable time. Likewise, more resources will speed up training. If the considered dataset exceeds the system memory (e.g. the 8 GiB of RAM suggested above), data must be preprocessed and streamed from disk as demonstrated in theDemosection below. On larger machines, one may benefit from maintaining a larger pool of data loaded in memory. For instance, we trained U-Sleep[2]using 8 CPU cores, 1 GPU and 40 GiB of RAM, please refer to theFull Reproduction of U-Sleepsection below.*The required hard-disk space depends on the number and sizes of datasets considered. For aFull Reproduction of U-Sleepapproximately 4 TiB of available storage is needed.Software Requirements:A computer with a Linux operating system installed. We have developed and tested the software for Red Hat Enterprise (v7.8) and Ubuntu (v18.04) servers, but any modern distribution should work. The software has also been tested on MacOS Catalina (10.15.6) for CPU-based training and prediction only.AnacondaorMiniconda, v4.5 or higher, 64-bit.If you are going to train a U-Sleep model yourself from scratch, we highly recommend doing so on a GPU. In order to use theU-Timepackage with a GPU, thetensorflow(v2.8.0) library is required. For this, the following additional software is required on your system:NVIDIA GPU driversv450.x or higher.CUDA Toolkitv11.2.cuDNN SDKv8.1.Please refer tohttps://www.tensorflow.org/install/gpufor additional details. You do not need to install TensorFlow yourself (seeInstallation Guidebelow), but the above software must be installed before proceeding.Installation GuideOn a Linux machine with at least 1 CUDA enabled GPU available andanacondaorminicondainstalled, run the following commands to download the software, create a conda environment namedu-sleepand setup the latest U-Time software package and its dependencies:git clone https://github.com/perslev/U-Time.git
conda env create --file U-Time/environment.yaml
conda activate u-sleep
pip install U-Time/Alternatively, you may install the package fromPyPi(may be updated less frequently):pip install utimeDemoIn this following we will demonstrate how to launch a short training session of U-Sleep on a significantly limited subset of the datasets used in[2].RequirementsCompletion of the steps outlined in theInstallation Guide.This demo takes approximately30 minutesto complete on a typical computer and network connection. The majority of this time is spend downloading the required data from public databases. This step may take significantly longer depending on current database traffic.Approximately11 GiBof available disk-space on your computer.Preparing a project directoryFirst, we create a project directory that will store all of our data for this demo. Theut initcommand will create a folder and populate it with a set of default hyperparameter values:ut init --name demo --model usleep_demoEntering the newly created project directory we will find a folder storing hyperparameters:cd demo
ls
> hyperparametersDownload public PSG dataWe will download 6 PSG studies from the public sleep databasesSleep-EDFandDCSMusing theut fetchcommand.
You will need approximately10 GiBof free hard-disk space to store the downloaded files. Depending on your internet speed and the current load on each of the two servers, downloading may take anywhere from 5 minutes to multiple hours:ut fetch --dataset sedf_sc --out_dir data/sedf_sc --N_first 6
ut fetch --dataset dcsm --out_dir data/dcsm --N_first 6The raw data that we will consider in this demo has now been downloaded.
Ourdemoproject folder now has roughly the following structure:└─ demo
   ├─ hyperparameters
   └─ data
      ├─ dcsm
      │  ├─ tp005f7e68_a084_46bb_9f0a_b6a084155a1c
      │  │  ├─ hypnogram.ids
      │  │  └─ psg.h5
      │  ├─ ...
      └─ sedf_sc
         ├─ SC4001E0
         │  ├─ SC4001E0-PSG.edf
         │  └─ SC4001EC-Hypnogram.edf
         ├─ ...Dataset splittingBefore proceeding to train the U-Sleep model we split each dataset into fixed train/validation/test splits using theut cv_splitcommand.
The command must be invoked twice each with a unique set of parameters specifying the naming conventions of dataset:# Split dataset SEDF-SC
ut cv_split --data_dir data/sedf_sc/ \
            --subject_dir_pattern 'SC*' \
            --CV 1 \
            --validation_fraction 0.10 \
            --test_fraction 0.25 \
            --subject_matching_regex 'SC4(\d{2}).*' \
            --seed 123
            
# Split dataset DCSM
ut cv_split --data_dir data/dcsm/ \
            --subject_dir_pattern 'tp*' \
            --CV 1 \
            --validation_fraction 0.10 \
            --test_fraction 0.25 \
            --seed 123Note that the splitting ofSEDF-SCis performed on a per-subject basis. All PSG records from the same subject will be placed into the same dataset split.
This is not needed forDCSMas all recordings are of unique subjects.Please be aware that if you modify any of the above commands to e.g. use different output directory names, you will need to modify paths in dataset hyperparameter files stored underhyperparameters/dataset_configurationsas appropriate before proceding with the next steps.Data pre-processingRun the following command to prepare the data for training:ut preprocess --out_path data/processed_data.h5 --dataset_splits train_data val_dataTheut preprocessscript loads and processes all datasets as described by the parameters set inhyperparameters/hparams.yamland all dataset-specific files in the folderhyperparameters/dataset_configurations.
Specifically, it loads the needed channels (ignoring the rest), re-samples, scales and clips the data, maps hypnogram stages to interger representations used internally during training and finally saves the processed data to an HDF5 archive.
When training, data may be streamed directly from this archive to significantly reduce the required system memory.It is also possible to skip this step all together and either1)load all data needed for training up front, or2)stream and apply preprocessing on-the-fly during training as shown in theFull Reproduction of U-Sleepsection below.Training the modelWe may now start training by invoking theut traincommand. A default set of optimization hyperparameters have been pre-specified and are located in thehyperparameters/hparams.yamlfile of your project directory. In this demo, we are going to run only a very short training session, but feel free to modify any parameters in thehparams.yamlfile as you see fit.
Run the following command:ut train --num_gpus=1 --preprocessed --seed 123You may replace the--num_gpus=1parameter in the above command with--num_gpus=0if you do not have a GPU available, and wish to train on the CPU. Training on CPU may take up to 30 minutes.Following training, a set of candidate models will be available in the foldermodel. Using the best one observed (highest validation mean F1 score),
we may predict on the testing sets of bothSEDF-SCandDCSMusing all channel combinations as well as compute majority votes by invoking the followingut predictcommand:Predicting and evaluating on the test setsut predict --num_gpus=1 \
           --data_split test_data \
           --strip_func strip_to_match \
           --one_shot \
           --save_true \
           --majority \
           --out_dir predictionsThe predicted hypnograms are now available under directorypredictions/test_data.
Finally, let us print a global confusion matrix (computed across all subjects) for datasetsedf_sc(replacesedf_sc->dcsmfor DCSM evaluation):ut cm --true 'predictions/test_data/sedf_sc/*TRUE.npy' \
      --pred 'predictions/test_data/sedf_sc/majority/*PRED.npy' \
      --ignore 5 \
      --round 2 \
      --wake_trim_min 30

>>>  Looking for files...
>>>  Loading 2 pairs...
>>>  OBS: Wake trimming of 30 minutes (period length 30 sec)
>>>  OBS: Ignoring class(es): [5]
>>>  
>>>  Raw Confusion Matrix:
>>>  
>>>          Pred 0  Pred 1  Pred 2  Pred 3  Pred 4
>>>  True 0       0       0      17     234       0
>>>  True 1       0       0     132     146       0
>>>  True 2       0       0     790     157       0
>>>  True 3       0       0      25     189       0
>>>  True 4       0       0     243      99       0
>>>  
>>>  Raw Metrics:
>>>  
>>>             F1  Precision  Recall/Sens.
>>>  Class 0  0.00       0.00          0.00
>>>  Class 1  0.00       0.00          0.00
>>>  Class 2  0.73       0.65          0.83
>>>  Class 3  0.36       0.23          0.88
>>>  Class 4  0.00       0.00          0.00
>>>  mean     0.22       0.18          0.34If you received an outputsimilarto the above, congratulations! You have successfully installed, configured, trained and evaluated a U-Sleep model on two different datasets.Please note that:If you ran the above code on a GPU, you may not obtain the exact same numbers listed here, even if you specified the --seed arguments. This is because some computations used during the training of U-Sleep are fundamentally non-deterministic when evaluated on a GPU.
However, predicting using a trained U-Sleep model will give deterministic outputs.The performance of the obtained demo model is very low and not suitable for actual sleep staging. The reason is that we trained U-Sleep on a very limited set of data and for a very limited number of epochs.
Please refer to theFull Reproduction of U-Sleepsection for details on how to prepare and train a complete version of U-Sleep.Full Reproduction of U-SleepTheDemoabove in principle describes all steps needed to reproduce U-Sleep as reported in[2].
The main - and significant - difference is that in order to reproduce the full model, you will need to 1) be able to access 2) download and 3) preprocess all the required datasets. You may also need a computer with more resources as described inSystem Requirements.Prepare the datasetsWe did our best to make this process as easy as possible. You should take the following steps:Carefully read (at least) theMethodsand supplementaryDatasetssections of our paper[2]to familiarize yourself with the datasets, preprocessing, training pipeline and more.Download all datasets from theNational Sleep Research Resource,PhysioNetor other sleep repositories as described and referenced in the Supplementary Material'sDatasetsection. For some datasets you must apply to gain access, while others are publicly available. Some datasets may be easily downloaded using theut fetchcommand. Please invokeut fetch --helpto see an up-to-date list of which datasets may be downloaded this way.Place all downloaded data into a single folder[LOCAL_PATH]with 1 sub-folder for each dataset.Runut extract,ut extract_hypno, andut cv_spliton all datasets as specified for each dataset separately in files under the folderresources/usleep_dataset_predof this repository (also foundhere). These commands will extract and place data into a folder-structure and format that U-Time accepts, as well as split the data into subsets.(optional) Theut extractcommand will select the relevant channels, re-sample them to 128 Hz and store the data in HDF5 archives. The original data will not be deleted by default. If you have limited hard-drive space, consider removing the old files before processing the next dataset.Initialize a U-Sleep project:ut init --name u-sleep --model usleep.For each dataset configuration file inu-sleep/hyperparameters/dataset_configurations/replace the string [LOCAL_PATH] with the[LOCAL_PATH]of your data.Train the modelIf you have 40+ GiB system memory available, train U-Sleep using the following command:ut train --num_gpus 1 --max_loaded_per_dataset 40 --num_access_before_reload 32 --train_queue_type limitation --val_queue_type lazy --max_train_samples_per_epoch 1000000On systems with less memory, you may either 1) reduce the--max_loaded_per_datasetparameter from the current40to a lower value (this will keep fewer PSG records in the active memory pool, which will reduce randomness when selecting records), or 2) preprocess the data and stream data during training (as demonstrated in the Demo above) by invoking the following two commands (replacing [LOCAL_PATH] as applicable):ut preprocess --out_path '[LOCAL_PATH]/processed_data.h5' --dataset_splits train_data val_data
ut train --num_gpus 1 --preprocessed --max_train_samples_per_epoch 1000000This will apply all preprocessing, create a data archive suitable for streaming, and train U-Sleep using samples loaded on-the-fly from disk.Due to the vast size of the dataset considered, training U-Sleep with the default parameters may take very long.
We suggest increasing the learning rate (from the current1e-7to e.g.1e-6) unless you are looking to re-create U-Sleep under the exact conditions considered in[2].U-Time ExampleYou can still use this repository to train the older U-Time model.
In the following we show an end-to-end example. The commands listed below prepares a project folder, downloads the sleep-edf-153 dataset, fits and evaluates
a U-Time model in a fixed train/val/test dataset split setup. Please note that the below code does not reproduce the sleep-edf-153 experiment of[1]as 10-fold CV was used.
To run a CV experiment, please refer to theut cv_split --helpandut cv_experiment --helpcommands.# Obtain a public sleep staging datasetut fetch --dataset sedf_sc --out_dir datasets/sedf_sc# Prepare a fixed-split experimentut cv_split --data_dir 'datasets/sedf_sc' \
            --subject_dir_pattern 'SC*' \
            --CV 1 \
            --validation_fraction 0.20 \
            --test_fraction 0.20 \
            --subject_matching_regex 'SC4(\d{2}).*' \
            --file_list# Initialize a U-Time projectut init --name my_utime_project \
        --model utime \
        --data_dir datasets/sedf_sc/views/fixed_split# Start trainingcd my_utime_project
ut train --num_gpus=1 --channels 'EEG Fpz-Cz'# Predict and evaluateut evaluate --out_dir eval --one_shot# Print a confusion matrixut cm --true 'eval/test_data/dataset_1/files/*/true.npz' \
      --pred 'eval/test_data/dataset_1/files/*/pred.npz'# Print per-subject summary statisticsut summary --csv_pattern 'eval/test_data/*/evaluation_dice.csv' \
           --print_all# Output sleep stages for every 3 seconds of 128 Hz signal# Here, the 'folder_regex' matches 2 files in the datasetut predict --folder_regex '../datasets/sedf_sc/SC400[1-2]E0' \
           --out_dir high_res_pred \
           --data_per_prediction 384 \
           --one_shotReferences[1] U-Time@incollection{NIPS2019_8692,
	title = {U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging},
	author = {Perslev, Mathias and Jensen, Michael and Darkner, Sune and Jennum, Poul J\o rgen and Igel, Christian},
	booktitle = {Advances in Neural Information Processing Systems 32},
	editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\textquotesingle Alch\'{e}-Buc and E. Fox and R. Garnett},
	pages = {4415--4426},
	year = {2019},
	publisher = {Curran Associates, Inc.},
	url = {http://papers.nips.cc/paper/8692-u-time-a-fully-convolutional-network-for-time-series-segmentation-applied-to-sleep-staging.pdf}
}[2] U-SleepU-Sleep: Resilient High-Frequency Sleep Staging
Mathias Perslev (1), Sune Darkner (1), Lykke Kempfner (2), Miki Nikolic (2), Poul Jørgen Jennum (2) & Christian Igel (1)
npj Digital Medicine, 4, 72 (2021)
https://doi.org/10.1038/s41746-021-00440-5

(1) Department of Computer Science, University of Copenhagen, Denmark
(2) Danish Center for Sleep Medicine, Rigshospitalet, Glostrup, DenmarkHistory1.0.0 (2022-04-11)This version includes a major refactoring of the utime package in several areas:No longer relies on code from the mpunet package (https://github.com/perslev/MultiPlanarUNet).All code related to loading/processing/saving of PSG data has been separated into the 'psg_utils' package (https://github.com/perslev/psg-utils).Hyperparameter file loading/manipulation/saving has been separated into the 'yamlhparams' package (https://github.com/perslev/yamlhparams).Logging has been overhauled and standardized across all utime scripts (invoked with ut <script> command / in utime/bin).Many other smaller changes to both functions and classesSome hyperparameter configuration attributes have been renamed or changed/removed entirely.In addition, this update includes various bug fixes (e.g., issue 43 - non changing validation loss).
Some rarely used loss functions and metrics have been removed (these are still available in the mpunet package).All in all, this version includes many API changes that are incompatible with earlier versions. However, all utime
scripts (ut <script> commands) may - except for a few command line argument naming changes - be used as in earlier versions
and with earlier versions of utime projects. Old hyperparameter files with incompatible attribute names will to some extend
be attempted automatically converted. When not possible, the user will need to re-initialize the project with 'ut init'
to create a fresh set of hyperparameters.0.0.1 (2019-01-11)Project created and packaged