Package: delia
DELIA facilitates data extraction from DICOM files to support large-scale image analysis workflows.Notable featuresBulk extraction of images and segmentations from multiple patients' DICOM files. Segmentations must be inDICOM-SEGorRTStructformat.Creation of anHDF5database containing multiple patients' medical images as well as binary label maps (segmentations). The database is then easier to use than DICOMs to perform tasks on medical data, such as training deep neural networks.Bulk extraction of radiomics features from multiple patients' DICOM files usingpyradiomics. Note thatpyradiomicsis not a dependency ofdeliaand must be installed separately.InstallationLatest stable version :pip install deliaLatest (possibly unstable) version :pip install git+https://github.com/MaxenceLarose/deliaQuick usage previewExtract images asnumpyarraysfromdelia.extractorsimportPatientsDataExtractorpatients_data_extractor=PatientsDataExtractor(path_to_patients_folder="patients")forpatientinpatients_data_extractor:forimage_datainpatient.data:array=image_data.image.numpy_array"""Perform any tasks on images on-the-fly."""print(array)Create patients database (HDF5)fromdelia.databasesimportPatientsDatabasefromdelia.extractorsimportPatientsDataExtractorpatients_data_extractor=PatientsDataExtractor(path_to_patients_folder="patients")database=PatientsDatabase(path_to_database="patients_database.h5")database.create(patients_data_extractor=patients_data_extractor)Create radiomics dataset (CSV)fromdelia.extractorsimportPatientsDataExtractorfromdelia.radiomicsimportRadiomicsDataset,RadiomicsFeatureExtractorpatients_data_extractor=PatientsDataExtractor(path_to_patients_folder="patients")radiomics_dataset=RadiomicsDataset(path_to_dataset="radiomics.csv")radiomics_dataset.extractor=RadiomicsFeatureExtractor(path_to_params="features_extractor_params.yaml")radiomics_dataset.create(patients_data_extractor=patients_data_extractor,organ="Heart",image_modality="CT")Note thatpyradiomicsis not a dependency ofdeliaand must be installed separately.MotivationDigital Imaging and Communications in Medicine(DICOM) istheinternational standard for medical images and related information. The working groupDICOM WG-23on Artificial Intelligence / Application Hosting is currently working to identify or develop the DICOM mechanisms to support AI workflows, concentrating on the clinical context. Moreover, their futureroadmap and objectivesincludes working on the concern that current DICOM mechanisms might not be adequate to cover some use cases, particularly bulk analysis of large repository data, e.g. for training deep learning neural networks.However, no tool has been developed to achieve this goal at present.The mainpurposeof this module is therefore to provide the necessary tools to facilitate the use of medical images in an AI workflow.  This goal is accomplished by using theHDF file formatto create a database containing patients' medical images as well as binary label maps obtained from the segmentation of these images.How it worksMain conceptsThere are 4 main concepts :PatientDataModel: It is the primarydeliadata structure. It is a named tuple gathering the image and segmentation data available in a patient record.PatientsDataExtractor: A PythonGeneratorthat allows to iterate over several patients and create aPatientDataModelobject for each of them. A sequence ofdeliaand/ormonaitransformations to apply to specific images or segmentations can be specified (seeMONAI).PatientsDatabase: An object that is used to create/interact with an HDF5 file (a database!) containing all patients information (images + label maps). ThePatientsDataExtractoris used to populate this database.RadiomicsDataset: An object that is used to create/interact with a csv file (a dataset!) containing radiomics features extracted from images. ThePatientsDataExtractoris used to populate this dataset.Organize your dataSince this module requires the use of data, it is important to properly configure the data-related elements before using it.File formatImages files must be in standardDICOMformat and segmentation files must be inDICOM-SEGorRTStructformat.If your segmentation files are in a research file format (.nrrd,.nii, etc.), you need to convert them into the standardizedDICOM-SEGorRTStructformat. You can use thepydicom-seglibrary to create the DICOM-SEG files OR use theitkimage2dicomSEGpython module, which provide a complete pipeline to perform this conversion. Also, you can use theRT-Utilslibrary to create the RTStruct files.Tag values (Optional)This dictionary isnotmandatory for the code to work and therefore its default value isNone. Note that if notag_valuesdictionary is given, i.e.tag_values = None, then all images will be added to the database.The tag values are specified as adictionarythat contains the values for the tag specified when creating the PatientsDataExtractor object of the images that need to be extracted from the patients' files. Keys are arbitrary names given to the images we want to add and values are lists of values for the desired tag. The images associated with these tag values do not need to have a corresponding segmentation volume. If none of the descriptions match the series in a patient's files, a warning is raised and the patient is added to the list of patients for whom the pipeline has failed.Note that the tag values can be specified as a python dictionary or as a path to ajson filethat contains the desired values. Both methods are presented below.Using a json fileCreate a json file containing only the dictionary of the names given to the images we want to add (keys) and lists of tag values (values). Place this file in your data folder.Here is an example of a json file configured as expected :{"PT":["PT WB CORR (AC)","PT WB XL CORR (AC)"],"CT":["CT 2.5 WB","AC CT 2.5 WB"]}Using a Python dictionaryCreate the organs dictionary in your main.py python file.Here is an example of a python dictionary instanced as expected :tag_values={"PT":["PT WB CORR (AC)","PT WB XL CORR (AC)"],"CT":["CT 2.5 WB","AC CT 2.5 WB"]}Structure your patients directoryIt is important to configure the directory structure correctly to ensure that the module interacts correctly with the data files. Thepatientsfolder, must be structured as follows.Note that all DICOM files in the patients' folder will be read.|_📂 Project directory/
  |_📄 main.py
  |_📂 data/
    |_📄 tag_values.json
    |_📂 patients/
      |_📂 patient1/
       	|_📄 ...
       	|_📂 ...
      |_📂 patient2/
       	|_📄 ...
       	|_📂 ...
      |_📂 ...Import the packageThe easiest way to import the package is to use :importdeliaYou can explicitly use the objects sub-modules :fromdelia.databasesimportPatientsDatabasefromdelia.extractorsimportPatientsDataExtractorfromdelia.radiomicsimportRadiomicsDataset,RadiomicsFeatureExtractorUse the packageExample using thePatientsDatabaseclassThis file can then be executed to obtain an hdf5 database.fromdelia.databasesimportPatientsDatabasefromdelia.extractorsimportPatientsDataExtractorfromdelia.transformsimport(PETtoSUVD,ResampleD)frommonai.transformsimport(CenterSpatialCropD,Compose,ScaleIntensityD,ThresholdIntensityD)patients_data_extractor=PatientsDataExtractor(path_to_patients_folder="data/patients",tag_values="data/tag_values.json",transforms=Compose([ResampleD(keys=["CT_THORAX","PT","Heart"],out_spacing=(1.5,1.5,1.5)),CenterSpatialCropD(keys=["CT_THORAX","PT","Heart"],roi_size=(1000,160,160)),ThresholdIntensityD(keys=["CT_THORAX"],threshold=-250,above=True,cval=-250),ThresholdIntensityD(keys=["CT_THORAX"],threshold=500,above=False,cval=500),ScaleIntensityD(keys=["CT_THORAX"],minv=0,maxv=1),PETtoSUVD(keys=["PT"])]))database=PatientsDatabase(path_to_database="data/patients_database.h5")database.create(patients_data_extractor=patients_data_extractor,tags_to_use_as_attributes=[(0x0008,0x103E),(0x0020,0x000E),(0x0008,0x0060)],overwrite_database=True)The created HDF5 database will then look something like :Example using thePatientsDataExtractorclassThis file can then be executed to perform on-the-fly tasks on images.fromdelia.extractorsimportPatientsDataExtractorfromdelia.transformsimportCompose,CopySegmentationsD,PETtoSUVD,ResampleDimportSimpleITKassitkpatients_data_extractor=PatientsDataExtractor(path_to_patients_folder="data/patients",tag_values="data/tag_values.json",transforms=Compose([ResampleD(keys=["CT_THORAX","Heart"],out_spacing=(1.5,1.5,1.5)),PETtoSUVD(keys=["PT"]),CopySegmentationsD(segmented_image_key="CT_THORAX",unsegmented_image_key="PT")]))forpatient_datasetinpatients_data_extractor:print(f"Patient ID:{patient_dataset.patient_id}")forpatient_image_datainpatient_dataset.data:dicom_header=patient_image_data.image.dicom_headersimple_itk_image=patient_image_data.image.simple_itk_imagenumpy_array_image=sitk.GetArrayFromImage(simple_itk_image)"""Perform any tasks on images on-the-fly."""print(numpy_array_image.shape)Need more examples?You can find more in theexamples folder.A deeper look into thePatientsDataExtractorobjectThePatientsDataExtractorhas 4 important variables:  apath_to_patients_folder(which dictates the path to the folder that contains all patient records), atag(which specifies which tag to use when choosing which images need to be extracted, if no value is given, this defaults toSeriesDescription), atag_values(which dictates the images that needs to be extracted from the patient records) andtransformsthat defines a sequence of transformations to apply on images or segmentations. For each patient/folder available in thepath_to_patients_folder, all DICOM files in their folder are read. If the specified tag's value of a certain volume match one of the descriptions present in the giventag_valuesdictionary, this volume and its segmentation (if available) are automatically added to thePatientDataModel. Note that if notag_valuesdictionary is given (tag_values = None), then all images (and associated segmentations) will be added to the database.ThePatientsDataExtractorcan therefore be used to iteratively perform tasks on each of the patients, such as displaying certain images, transforming images into numpy arrays, or creating an HDF5 database using thePatientsDatabase. It is this last task that is highlighted in this package, but it must be understood that the data extraction is performed in a very general manner by thePatientsDataExtractorand is therefore not limited to this single application. For example, someone could easily develop aNumpydatabasewhose creation would be ensured by thePatientsDataExtractor, similar to the currentPatientsDatabasebased on the HDF5 format.TODOGeneralize the use of arbitrary tags to choose images to extract. At the moment, the only tag available isseries_descriptions.Find which DICOM tags act in unusual ways such as having data associated with .repval and .value which differ by more than just their type. For now, both are used to obtain the same value in delia/readers/patient_data/factories/patient_data_factories.py (152-155) and delia/readers/image/dicom_reader.py (113-116).Allow for more comprehensive use of tags such as using AND, OR and NOT between tags and their values.Loosen monai version requirements (monai==1.0.1) to allow for more recent versions. This needs a redefinition of the way transforms are applied.LicenseThis code is provided under theApache License 2.0.Citation@misc{delia,
  title={DELIA: DICOM Extraction for Large-scale Image Analysis},
  author={Maxence Larose},
  year={2022},
  publisher={Université Laval},
  url={https://github.com/MedPhysUL/delia},
}ContactMaxence Larose, B. Ing.,maxence.larose.1@ulaval.ca