Package: imsy-htc
Hyperspectral Tissue ClassificationThis package is a framework for automated tissue classification and segmentation on medical hyperspectral imaging (HSI) data. It contains:The implementation of deep learning models to solve supervised classification and segmentation problems for a variety of different input spatial granularities (pixels, superpixels, patches and entire images, cf. figure below) and modalities (RGB data, raw and processed HSI data) from our paper“Robust deep learning-based semantic organ segmentation in hyperspectral images”. It is based onPyTorchandPyTorch Lightning.Corresponding pretrained models.A pipeline to efficiently load and process HSI data, to aggregate deep learning results and to validate and visualize findings.Presentation of several solutions to speed up the data loading process (seePytorch Conference 2023 poster detailsbelow).This framework is designed to work on HSI data from theTivitacameras but you can adapt it to different HSI datasets as well. Potential applications include:Use our data loading and processing pipeline to easily access image and meta data for any work utilizing Tivita datasets.This repository is tightly coupled to work with the publicHeiPorSPECTRALdataset. If you already downloaded the data, you only need to perform the setup steps and then you can directly use thehtcframework to work on the data (cf.our tutorials).Train your own networks and benefit from a pipeline offering e.g. efficient data loading, correct hierarchical aggregation of results and a set of helpful visualizations.Apply deep learning models for different spatial granularities and modalities on your own semantically annotated dataset.Use our pretrained models to initialize the weights for your own training.Use our pretrained models to generate predictions for your own data.If you use thehtcframework, please consider citing thecorresponding papers. You can also cite this repository directly via:Cite via BibTeX@software{sellner_htc_2023,author={Sellner, Jan and Seidlitz, Silvia},publisher={Zenodo},url={https://github.com/IMSY-DKFZ/htc},date={2024-02-05},doi={10.5281/zenodo.6577614},title={Hyperspectral Tissue Classification},version={v0.0.15},}SetupPackage InstallationThis package can be installed via pip:pipinstallimsy-htcThis installs all the required dependencies defined inrequirements.txt. The requirements includePyTorch, so you may want to install it manually before installing the package in case you have specific needs (e.g. CUDA version).⚠️ This framework was developed and tested using the Ubuntu 20.04+ Linux distribution. Despite we do provide wheels for Windows and macOS as well, they are not tested.⚠️ Network training and inference was conducted using an RTX 3090 GPU with 24 GiB of memory. It should also work with GPUs which have less memory but you may have to adjust some settings (e.g. the batch size).PyTorch CompatibilityWe cannot provide wheels for all PyTorch versions. Hence, a version ofimsy-htcmay not work with all versions of PyTorch due to changes in the ABI. In the following table, we list the PyTorch versions which are compatible with the respectiveimsy-htcversion.imsy-htctorch0.0.91.130.0.101.130.0.112.00.0.122.00.0.132.10.0.142.1However, we do not make explicit version constraints in the dependencies of theimsy-htcpackage because a future version of PyTorch may still work and we don't want to break the installation if it is not necessary.💡 Please note that it is always possible to build theimsy-htcpackage with your installed PyTorch version yourself (cf. Developer Installation).Optional Dependencies (imsy-htc[extra])Some requirements are considered optional (e.g. if they are only needed by certain scripts) and you will get an error message if they are needed but unavailable. You can install them viapipinstall--extra-index-urlhttps://read_package:CnzBrgDfKMWS4cxf-r31@git.dkfz.de/api/v4/projects/15/packages/pypi/simpleimsy-htc[extra]or by adding the following lines to yourrequirements.txt--extra-index-url https://read_package:CnzBrgDfKMWS4cxf-r31@git.dkfz.de/api/v4/projects/15/packages/pypi/simple
imsy-htc[extra]This installs the optional dependencies defined inrequirements-extra.txt, including for example our Python wrapper for thechallengeR toolkit.DockerWe also provide a Docker setup for testing. As a prerequisite:Clone this repositoryInstallDockerand theNVIDIA Container ToolkitInstall the required dependencies to run the Docker startup script:pipinstallpython-dotenvMake sure that your environment variables are available and then bash into the containerexportPATH_Tivita_HeiPorSPECTRAL="/path/to/the/dataset"pythonrun_docker.pybashYou can now run any commands you like. All datasets you provided via an environment variable that starts withPATH_Tivitawill be accessible in your container (you can also check the generateddocker-compose.override.ymlfile for details). Please note that the Docker container is meant for small testing only and not for development. This is also reflected by the fact that per default all results are stored inside the container and hence will also be deleted after exiting the container. If you want to keep your results, let the environment variablePATH_HTC_DOCKER_RESULTSpoint to the directory where you want to store the results.Developer InstallationIf you want to make changes to the package code (which is highly welcome 😉), we recommend to install thehtcpackage in editable mode in a separate conda environment:# Set up the conda environment# Note: By adding conda-forge to your default channels, you will get the latest patch releases for Python:#   conda config --add channels conda-forgecondacreate--yes--namehtcpython=3.11
condaactivatehtc# Install the htc package and its requirementspipinstall-rrequirements-dev.txt
pipinstall--no-use-pep517-e.Before commiting any files, please run the static code checks locally:gitadd.
pre-commitrun--all-filesEnvironment VariablesThis framework can be configured via environment variables. Most importantly, we need to know where your data is located (e.g.PATH_Tivita_HeiPorSPECTRAL) and where results should be stored (e.g.PATH_HTC_RESULTS). For a full list of possible environment variables, please have a look at the documentation of theSettingsclass.💡 If you set an environment variable for a dataset path, it is important that the variable name matches the folder name (e.g. the variable namePATH_Tivita_HeiPorSPECTRALmatches the dataset pathmy/path/HeiPorSPECTRALwith its folder nameHeiPorSPECTRAL, whereas the variable namePATH_Tivita_some_other_namedoes not match). Furthermore, the dataset path needs to point to a directory which contains adataand anintermediatessubfolder.There are several options to set the environment variables. For example:You can specify a variable as part of your bash startup script~/.bashrcor before running each command:PATH_HTC_RESULTS="~/htc/results"htctraining--modelimage--config"models/image/configs/default"However, this might get cumbersome or might not give you the flexibility you need.Recommended if you cloned this repository (in contrast to simply installing it via pip): You can create a.envfile in the repository root and add your variables, for example:exportPATH_Tivita_HeiPorSPECTRAL=/mnt/nvme_4tb/HeiPorSPECTRALexportPATH_HTC_RESULTS=~/htc/resultsRecommended if you installed the package via pip: You can create user settings for this application. The location is OS-specific. For Linux the location might be at~/.config/htc/variables.env. Please runhtc infoupon package installation to retrieve the exact location on your system. The content of the file is of the same format as of the.envabove.After setting your environment variables, it is recommended to runhtc infoto check that your variables are correctly registered in the framework.TutorialsA series oftutorialscan help you get started on thehtcframework by guiding you through different usage scenarios.💡 The tutorials make use of our public HSI datasetHeiPorSPECTRAL. If you want to directly run them, please download the dataset first and make it accessible via the environment variablePATH_Tivita_HeiPorSPECTRALas described above.As a start, we recommend to take a look at thisgeneral notebookwhich showcases the basic functionalities of thehtcframework. Namely, it demonstrates the usage of theDataPathclass which is the entry point to load and process HSI data. For example, you will learn how to read HSI cubes, segmentation masks and meta data. Among others, you can use this information to calculate the median spectrum of an organ.If you want to use our framework with your own dataset, it might be necessary to write a customDataPathclass so that you can load and process your images and annotations. Wecollected some tipson how this can be achieved.You have some HSI data at hand and want to use one of our pretrained models to generate predictions? Then ourprediction notebookhas got you covered.You want to use our pretrained models to initialize the weights for your own training? See the section aboutpretrained modelsbelow for details.You want to use our framework to train a network? Thenetwork training notebookwill show you how to achieve this on the example of a heart and lung segmentation network.If you are interested in our technical validation (e.g. because you want to compare your colorchecker images with ours) and need to create a mask to detect the different colorchecker fields, you might find our automaticcolorchecker mask creation pipelineuseful.We do not have a separate documentation website for our framework yet. However, most of the functions and classes are documented so feel free to explore the source code or use your favorite IDE to display the documentation. If something does not become clear from the documentation, feel free to open an issue!Pretrained ModelsThis framework gives you access to a variety of pretrained segmentation and classification models. The models will be automatically downloaded, provided you specify the model type (e.g.image) and the run folder (e.g.2022-02-03_22-58-44_generated_default_model_comparison). It can then be used for example tocreate predictionson some data or as a baseline for your own training (see example below).The following table lists all the models you can get:model typemodalityclassrun folderimagehsiModelImage2023-02-08_14-48-02_organ_transplantation_0.8imagergbModelImage2023-01-29_11-31-04_organ_transplantation_0.8_rgbimagehsiModelImage2022-02-03_22-58-44_generated_default_model_comparisonimageparamModelImage2022-02-03_22-58-44_generated_default_parameters_model_comparisonimagergbModelImage2022-02-03_22-58-44_generated_default_rgb_model_comparisonpatchhsiModelImage2022-02-03_22-58-44_generated_default_64_model_comparisonpatchparamModelImage2022-02-03_22-58-44_generated_default_64_parameters_model_comparisonpatchrgbModelImage2022-02-03_22-58-44_generated_default_64_rgb_model_comparisonpatchhsiModelImage2022-02-03_22-58-44_generated_default_model_comparisonpatchparamModelImage2022-02-03_22-58-44_generated_default_parameters_model_comparisonpatchrgbModelImage2022-02-03_22-58-44_generated_default_rgb_model_comparisonsuperpixel_classificationhsiModelSuperpixelClassification2022-02-03_22-58-44_generated_default_model_comparisonsuperpixel_classificationparamModelSuperpixelClassification2022-02-03_22-58-44_generated_default_parameters_model_comparisonsuperpixel_classificationrgbModelSuperpixelClassification2022-02-03_22-58-44_generated_default_rgb_model_comparisonpixelhsiModelPixel2022-02-03_22-58-44_generated_default_model_comparisonpixelparamModelPixelRGB2022-02-03_22-58-44_generated_default_parameters_model_comparisonpixelrgbModelPixelRGB2022-02-03_22-58-44_generated_default_rgb_model_comparison💡 The modalityparamrefers to stacked tissue parameter images (named TPI in our paper“Robust deep learning-based semantic organ segmentation in hyperspectral images”). For the model typepatch, pretrained models are available for the patch sizes 64 x 64 and 32 x 32 pixels. The modality and patch size is not specified when loading a model as it is already characterized by specifying a certain run folder.After successful installation of thehtcpackage, you can use any of the pretrained models listed in the table. There are several ways to use them but the general principle is that models are always specified via theirmodelandrun_folder.Option 1: Use the models in your own training pipelineEvery model class listed in the table has a static methodpretrained_model()which you can use to create a model instance and initialize it with the pretrained weights. The model object will be an instance oftorch.nn.Module. The function has examples for all the different model types but as a teaser consider the following example which loads the pretrained image HSI network:importtorchfromhtcimportModelImage,Normalizationrun_folder="2022-02-03_22-58-44_generated_default_model_comparison"# HSI modelmodel=ModelImage.pretrained_model(model="image",run_folder=run_folder,n_channels=100,n_classes=19)input_data=torch.randn(1,100,480,640)# NCHWinput_data=Normalization(channel_dim=1)(input_data)# Model expects L1 normalized inputmodel(input_data).shape# torch.Size([1, 19, 480, 640])💡 Please note that when initializing the weights as in this example, the segmentation head is initialized randomly. Meaningful predictions on your own data can thus not be expected out of the box, but you will have to train the model on your data first.Option 2: Use the models to create predictions for your dataThe models can be used to predict segmentation masks for your data. The segmentation models automatically sample from your input image according to the selected model spatial granularity (e.g. by creating patches) and the output is always a segmentation mask for an entire image. The set of output classes is determined by the training configuration, e.g. 18 organ classes + background for our semantic models. TheCreatingPredictionsnotebook shows how to create predictions and how to map the network output to meaningful label names.Option 3: Use the models to train a network with thehtcpackageIf you are using thehtcframework totrain your networks, you only need to define the model in your configuration:{"model":{"pretrained_model":{"model":"image","run_folder":"2022-02-03_22-58-44_generated_default_model_comparison",}}}This is very similar to option 1 but may be more convenient if you train with thehtcframework.CLIThere is a common command line interface for many scripts in this repository. More precisely, every script which is prefixed withrun_NAME.pycan also be run viahtc NAMEfrom any directory. For more details, just typehtc.PapersThis repository contains code to reproduce our publications listed below:📝Robust deep learning-based semantic organ segmentation in hyperspectral imagesIn this paper, we tackled fully automatic organ segmentation and compared deep learning models on different spatial granularities (e.g. patch vs. image) and modalities (e.g. HSI vs. RGB). Furthermore, we studied the required amount of training data and the generalization capabilities of our models across subjects. The pretrained networks are related to this paper. You can find the notebooks to generate the paper figures inpaper/MIA2021(the folder also includes areproducibility document) and the models inhtc/models. For each model, there are three configuration files, namelydefault,default_rgbanddefault_parameters, which correspond to the HSI, RGB and TPI modality, respectively. You can also download theNSD thresholdswhich we used for the NSD metric (cf. Fig. 12).📂 The dataset for this paper is not publicly available.Cite via BibTeX@article{seidlitz_seg_2022,author={Seidlitz, Silvia and Sellner, Jan and Odenthal, Jan and Özdemir, Berkin and Studier-Fischer, Alexander and Knödler, Samuel and Ayala, Leonardo and Adler, Tim J. and Kenngott, Hannes G. and Tizabi, Minu and Wagner, Martin and Nickel, Felix and Müller-Stich, Beat P. and Maier-Hein, Lena},date={2022-08},doi={10.1016/j.media.2022.102488},issn={1361-8415},journaltitle={Medical Image Analysis},keywords={Hyperspectral imaging,Surgical data science,Deep learning,Open surgery,Organ segmentation,Semantic scene segmentation},pages={102488},title={Robust deep learning-based semantic organ segmentation in hyperspectral images},volume={80},}📝Semantic segmentation of surgical hyperspectral images under geometric domain shiftsThis MICCAI2023 paper is the direct successor of our MIA2021 paper. We analyzed how well our networks perform under geometrical domain shifts which commonly occur in real-world open surgeries (e.g. situs occlusions). The effect is drastic (drop of Dice similarity coefficient by 45 %) but the good news is that performance on par with in-distribution data can be achieved with our simple, model-independent solution (augmentation method). You can find all the code inhtc/contextand paper figures as well asreproducibility instructionsinpaper/MICCAI2023. Pretrained models are available for our organ transplantation networks with HSI and RGB modalities.💡 If you are only interested in our data augmentation method, you can also head over toKorniawhere this augmentation is implemented for generic use cases (including 2D and 3D data). You will find it under the nameRandomTransplantation.📂 The dataset for this paper is not publicly available.Cite via BibTeX@inproceedings{sellner_context_2023,author={Sellner, Jan and Seidlitz, Silvia and Studier-Fischer, Alexander and Motta, Alessandro and Özdemir, Berkin and Müller-Stich, Beat Peter and Nickel, Felix and Maier-Hein, Lena},editor={Greenspan, Hayit and Madabhushi, Anant and Mousavi, Parvin and Salcudean, Septimiu and Duncan, James and Syeda-Mahmood, Tanveer and Taylor, Russell},location={Cham},publisher={Springer Nature Switzerland},booktitle={Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023},date={2023},doi={10.1007/978-3-031-43996-4_59},isbn={978-3-031-43996-4},pages={618--627},title={Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts},}📝Dealing with I/O bottlenecks in high-throughput model trainingThe poster was presented at the PyTorch Conference 2023 and presents several solutions to improve data loading for faster network training. This originated from our MICCAI2023 paper, where we load huge amount of data while using a relatively small network resulting in GPU idle times when the GPU has to wait for the CPU to deliver new data. This requested the need for fast data loading strategies so that the CPU delivers data in-time for the GPU. The solutions include (1) efficient data storage viaBlosccompression, (2) appropriate precision settings, (3) GPU instead of CPU augmentations using theKornialibrary and (4) a fixed shared pinned memory buffer for efficient data transfer to the GPU. For the last part, you will find the relevant code to create the buffer in this repository as part of theSharedMemoryDatasetMixinclass (_add_tensor_shared()method).You can find the code to generate the results figures of the poster inpaper/PyTorchConf2023includingreproducibility instructions.📂 The dataset for this poster is not publicly available.Cite via BibTeX@misc{sellner_benchmarking_2023,author={Sellner, Jan and Seidlitz, Silvia and Maier-Hein, Lena},language={en},url={https://e130-hyperspectal-tissue-classification.s3.dkfz.de/figures/PyTorchConference_Poster.pdf},date={2023-10-16},title={Dealing with I/O bottlenecks in high-throughput model training},}📝Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine modelIn this paper, we trained a classification model based on median spectra from HSI data. You can find the model code inhtc/tissue_atlasand the confusion matrix figure of the paper inpaper/NatureReports2021(including a reproducibility document).📂 The dataset for this paper is not fully publicly available, but a subset of the data is available through the publicHeiPorSPECTRALdataset.Cite via BibTeX@article{studierfischer_atlas_2022,author={Studier-Fischer, Alexander and Seidlitz, Silvia and Sellner, Jan and Özdemir, Berkin and Wiesenfarth, Manuel and Ayala, Leonardo and Odenthal, Jan and Knödler, Samuel and Kowalewski, Karl Friedrich and Haney, Caelan Max and Camplisson, Isabella and Dietrich, Maximilian and Schmidt, Karsten and Salg, Gabriel Alexander and Kenngott, Hannes Götz and Adler, Tim Julian and Schreck, Nicholas and Kopp-Schneider, Annette and Maier-Hein, Klaus and Maier-Hein, Lena and Müller-Stich, Beat Peter and Nickel, Felix},date={2022-06-30},doi={10.1038/s41598-022-15040-w},issn={2045-2322},journaltitle={Scientific Reports},number={1},pages={11028},title={Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine model},volume={12},}📝HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological OrgansThis paper introduces theHeiPorSPECTRALdataset containing 5756 hyperspectral images from 11 subjects. We are using these images in our tutorials. You can find the visualization notebook for the paper figures inpaper/NatureData2023(the folder also includes areproducibility document) and the remaining code inhtc/tissue_atlas_open.If you want to learn more about theHeiPorSPECTRALdataset (e.g. the underlying data structure) or you stumbled upon a file and want to know how to read it, you might find thisnotebook with low-level detailshelpful.📂 The dataset for this paper is publicly available.Cite via BibTeX@article{studierfischer_open_2023,author={Studier-Fischer, Alexander and Seidlitz, Silvia and Sellner, Jan and Bressan, Marc and Özdemir, Berkin and Ayala, Leonardo and Odenthal, Jan and Knoedler, Samuel and Kowalewski, Karl-Friedrich and Haney, Caelan Max and Salg, Gabriel and Dietrich, Maximilian and Kenngott, Hannes and Gockel, Ines and Hackert, Thilo and Müller-Stich, Beat Peter and Maier-Hein, Lena and Nickel, Felix},url={https://heiporspectral.org/},date={2023-06-24},doi={10.1038/s41597-023-02315-8},issn={2052-4463},journaltitle={Scientific Data},number={1},pages={414},title={HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological Organs},volume={10},}📝Künstliche Intelligenz und hyperspektrale Bildgebung zur bildgestützten Assistenz in der minimal-invasiven ChirurgieThis paper presents several applications of intraoperative HSI, including our organsegmentationandclassificationwork. You can find the code generating our figure for this paper atpaper/Chirurg2022.📂 The sample image used here is contained in the dataset from our paper“Robust deep learning-based semantic organ segmentation in hyperspectral images”and hence not publicly available.Cite via BibTeX@article{chalopin_chirurgie_2022,author={Chalopin, Claire and Nickel, Felix and Pfahl, Annekatrin and Köhler, Hannes and Maktabi, Marianne and Thieme, René and Sucher, Robert and Jansen-Winkeln, Boris and Studier-Fischer, Alexander and Seidlitz, Silvia and Maier-Hein, Lena and Neumuth, Thomas and Melzer, Andreas and Müller-Stich, Beat Peter and Gockel, Ines},date={2022-10-01},doi={10.1007/s00104-022-01677-w},issn={2731-698X},journaltitle={Die Chirurgie},number={10},pages={940--947},title={Künstliche Intelligenz und hyperspektrale Bildgebung zur bildgestützten Assistenz in der minimal-invasiven Chirurgie},volume={93},}FundingThis project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (NEURAL SPICING, grant agreement No. 101002198) and was supported by the German Cancer Research Center (DKFZ) and the Helmholtz Association under the joint research school HIDSS4Health (Helmholtz Information and Data Science School for Health). It further received funding from the Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg.