Package: wsidicom
wsidicomwsidicomis a Python package for readingDICOM WSI. The aims with the project are:Easy to use interface for reading and writing WSI DICOM images and annotations either from file or through DICOMWeb.Support the latest and upcoming DICOM standards.Platform independent installation via PyPI.Installingwsidicomwsidicomis available on PyPI:pip install wsidicomAnd through conda:conda install -c conda-forge wsidicomImportant notePlease note that this is an early release and the API is not frozen yet. Function names and functionality is prone to change.Requirementswsidicomuses pydicom, numpy, Pillow, marshmallow, fsspec, universal-pathlib, and dicomweb-client. Imagecodecs, pylibjpeg-rle, pyjpegls, and pylibjpeg-openjpeg can be installed as optionals to support additional transfer syntaxes.LimitationsLevels are required to have (close to) 2 factor scale and same tile size.Only 8 bits per sample is supported for color images, and 8 and 16 bits for grayscale images.Without optional dependencies, the following transfer syntaxes are supported:JPEGBaseline8BitJPEG2000JPEG2000LosslessHTJPEG2000HTJPEG2000LosslessHTJPEG2000RPCLLosslessImplicitVRLittleEndianExplicitVRLittleEndianExplicitVRBigEndianWith imagecodecs, the following transfer syntaxes are additionally supported:JPEGExtended12BitJPEGLosslessP14JPEGLosslessSV1JPEGLSLosslessJPEGLSNearLosslessRLELosslessWith pylibjpeg-rle RLELossless is additionally supported.With pyjpegls JPEGLSLossless and JPEGLSNearLossless is additionally supported.Optical path identifiers needs to be unique across instances.Only one pyramid (i.e. offset from slide corner) per frame of reference is supported.Basic usageLoad a WSI dataset from files in folder.fromwsidicomimportWsiDicomslide=WsiDicom.open("path_to_folder")Thefilesargument accepts either a path to a folder with DICOM WSI-files or a sequence of paths to DICOM WSI-files.Load a WSI dataset from remote url usingfsspec.fromwsidicomimportWsiDicomslide=WsiDicom.open("s3://bucket/key",file_options={"s3":"anon":True})Or load a WSI dataset from opened streams.fromwsidicomimportWsiDicomslide=WsiDicom.open_streams([file_stream_1,file_stream_2,...])Or load a WSI dataset from a DICOMDIR.fromwsidicomimportWsiDicomslide=WsiDicom.open_dicomdir("path_to_dicom_dir")Or load a WSI dataset from DICOMWeb.fromwsidicomimportWsiDicom,WsiDicomWebClientfromrequests.authimportHTTPBasicAuthauth=HTTPBasicAuth('username','password')client=WsiDicomWebClient.create_client('dicom_web_hostname','/qido','/wado,auth)slide=WsiDicom.open_web(client,"study uid to open","series uid to open"or["series uid 1 to open","series uid 2 to open"])Alternatively, if you have already created an instance ofdicomweb_client.DICOMwebClient, that may be used to create theWsiDicomWebClientlike so:dicomweb_client=DICOMwebClient("url")client=WsiDicomWebClient(dicomweb_client)Then proceed to callWsiDicom.open_web()with this as in the first example.Use as a context manager.fromwsidicomimportWsiDicomwithWsiDicom.open("path_to_folder")asslide:...Read a 200x200 px region starting from px 1000, 1000 at level 6.region=slide.read_region((1000,1000),6,(200,200))Read a 2000x2000 px region starting from px 1000, 1000 at level 4 using 4 threads.region=slide.read_region((1000,1000),6,(200,200),threads=4)Read 3x3 mm region starting at 0, 0 mm at level 6.region_mm=slide.read_region_mm((0,0),6,(3,3))Read 3x3 mm region starting at 0, 0 mm with pixel spacing 0.01 mm/px.region_mpp=slide.read_region_mpp((0,0),0.01,(3,3))Read a thumbnail of the whole slide with maximum dimensions 200x200 px.thumbnail=slide.read_thumbnail((200,200))Read an overview image (if available).overview=slide.read_overview()Read a label image (if available).label=slide.read_label()Read (decoded) tile from position 1, 1 in level 6.tile=slide.read_tile(6,(1,1))Read (encoded) tile from position 1, 1 in level 6.tile_bytes=slide.read_encoded_tile(6,(1,1))Close filesslide.close()API differences between WsiDicom and OpenSlideThe WsiDicom API is similar to OpenSlide, but with some important differences:In WsiDicom, theopen-method (i.e.WsiDicom.open()) is used to open a folder with DICOM WSI files, while in OpenSlide a file is opened with the__init__-method (e.g.OpenSlide()).In WsiDicom thelocationparameter inread_regionis relative to the specifiedlevel, while in OpenSlide it is relative to the base level.In WsiDicom thelevelparameter inread_regionis the pyramid index, i.e. level 2 always the level with quarter the size of the base level. In OpenSlide it is the index in the list of available levels, and if pyramid levels are missing these will not correspond to pyramid indices.Conversion between OpenSlidelocationandlevelparameters to WsiDicom can be performed:withWsiDicom.open("path_to_folder")aswsi:level=wsi.levels[openslide_level_index]x=openslide_x//2**(level.level)y=openslide_y//2**(level.level)MetadataWsiDicom parses the DICOM metadata in the opened image into easy-to-use dataclasses, seewsidicom\metadata.withWsiDicom.open("path_to_folder")aswsi:metadata=wsi.metadataThe obtainedWsiMetadatahas child dataclass properties the resembelse the DICOM WSI modules (compare
with theVL Whole Slide Microscopy Image CIOD):study: The study the slide is part of (study identifiers, study date and time, etc.).series: The series the slide is part of.patient: Patient information (name, identifier, etc.).equipment: Scanner information information.optical_paths: List of optical path descriptions used for imaging the slide.slide: Slide information, including slide identifier, stainings done on the slide, and samples placed on the slide, see details inSlide informationlabel: Slide label information, such as label text.image: Image information, including acquisition datetime, pixel spacing, focus method, etc.frame_of_reference_uid: The unique identifier for the frame of reference for the image.dimension_organization_uids: List of dimension organization uids.Note that not all DICOM attributes are represented in the defined metadata model. Instead the full ´pydicom´ Datasets can be accessed per level, for example:withWsiDicom.open("path_to_folder")aswsi:wsi.levels.base_level.datasets[0]If you encounter that some important and/or useful attribute is missing from the model, please make an issue (seeContributing).Slide informationTheSlideinformation model models theSpecimenmodule has the following properties:identifier: Identifier for the slide.stainings: List of stainings done on the slide. Note that the model assumes that the same stainings have been done on all the samples on the slide.samples: List of samples placed on the slide.Note that that while the parsing of slide information is designed to be as flexible and permissive as possible, some datasets contains non-standard compliantSpecimenmodules that are (at least currently) not possible to parse. In such cases thestainingsandsamplesproperty will be set toNone. If you have a dataset with aSpecimenmodule that you think should be parsable, please make an issue (seeContributing).SlideSampleEach sample is model with theSlideSampledataclass, which represents an item in the DICOMSpecimen Description Sequenceidentifier: Identifier of the sample.anatomical_sites: List of codes describing the primary anatomic structures of interest in the sample.sampled_from: The sampling (of another specimen) that was done to produce the sample (if known). If the sampled specimen also was produced through sampling, this property will give access to the full hierarchy of (known) specimens.uid: Unique identifier for the sample.localization: Description of the placement of the sample on the slide. Should be present if more than one sample is placed on the slide.steps: List of preparation steps performed on the sample.short_description: Short description of the sample (should not exceed 64 characters).detailed_description: Unlimited description of the sample.SamplingsThe optionalsampled_fromproperty can either be aSamplingor aUnknownSampling. Both of these specify a sampledspecimen, with the difference that theUnknownSamplingis used when the sampling conditions are not fully know. ASamplingis more detailed, and specifies the samplingmethodand optional properties such as samplingdate_time,descriptionandlocation.SpecimensThespecimenproperty of aSamplingor aUnknownSamplinglinks to either aSpecimenor aSample. ASpecimenhas no known parents (e.g. could be the specimen extracted from a patient), while aSamplealways is produced from one or more samplings of otherSpecimens orSamples. The samplings used to produce aSampleis given by itssampled_from-property. BothSpecimenandSamplecontain additional properties describing the specimen:identifier: Identifier of the specimen.type: Optional anatomic pathology specimen type code (e.g. "tissue specimen"). Should be a specimen type defined inCID 8103.steps: List of processing steps performed on the specimen.container: Optional container type code the specimen is placed in. Should be a container type defined inCID 8101.Processing and staining stepsThe processing steps that can be performed on a sample are:Sampling: Sampling of the specimen in order to produce new specimen(s). The samplingmethodshould be a method defined inCID 8110.Collection: Collection of a specimen from a body. This can only be done on aSpecimen, i.e. not on a specimen produced by sampling. The collectionmethodshould be a method defined inCID 8109.Processing: Processing performed on the specimen. The processingmethodshould be a method defined inCID 8113.Embedding: Embedding done on the specimen. The embedding medium should be a medium defined inCID 8115.Fixation: Fixation of the specimen. The fixative should be a fixative defined inCID 8114.Receiving: Receiving of the specimen.Storage: Storage of the specimen.TheStaining(s) for aSlidecontains a list of substances used for staining. The substances used should defined inCID 8112.Every processing step (including staining) also have the optional propertiesdate_timefor when the processing was done anddescriptionfor a textual description of the processing.These steps are parsed from theSpecimenPreparationSequencefollowingTID 8004for each specimen identifier in the item sequence.Exporting to jsonThe metadata can be exported to json:fromwsidicom.metadata.schema.jsonimportWsiMetadataJsonSchemawithWsiDicom.open("path_to_folder")aswsi:metadata=wsi.metadataschema=WsiMetadataJsonSchema()metadata_json=schema.dump(metadata)SettingsThe strictness of parsing of DICOM WSI metadata can be configured using the following settings (seeSettings):strict_specimen_identifier_check: Controls how to handle matching between specimen identifiers if one of the identifiers have a issuer of identifier set and the other does not. IfTruethe identifiers are considered equal (provided that the identifier value is the same), ifFalsethe issuer of identifier must always also match. This setting is useful if for example a issuer of identifier is specified in theSpecimen Description Sequencebut steps in theSpecimen Preparation Sequencelacks the issuer of identifier. The default value isTrue.ignore_specimen_preparation_step_on_validation_error: Controls how to handle if a step in theSpecimen Preparation Sequencefails to validate. IfTrue, only steps that fails will be ignored. IfFalseall steps will be ignored. The default value isTrue.Saving filesAn opened WsiDicom instance can be saved to a new path using the save()-method. The produced files will be:Fully tiled. Any sparse tiles will be replaced with a blank tile with color depending on the photometric interpretation.Have a basic offset table (or optionally an extended offset table or no offset table).Not be concatenated.By default frames are copied as-is, i.e. without re-compression.withWsiDicom.open("path_to_folder")asslide:slide.save("path_to_output")The output folder must already exists. Be careful to specify a unique folder folder to avoid mixing files from different images.Optionally frames can be transcoded, either by a encoder setting or an encoder:fromwsidicom.codecimportJpegSettingswithWsiDicom.open("path_to_folder")asslide:slide.save("path_to_output",transcoding=JpegSettings())Settingswsidicomcan be configured with the settings variable. For example, set the parsing of files to strict:fromwsidicomimportsettingssettings.strict_uid_check=Truesettings.strict_attribute_check=TrueAnnotation usageAnnotations are structured in a hierarchy:AnnotationInstance
Represents a collection of AnnotationGroups. All the groups have the same frame of reference, i.e. annotations are from the same wsi stack.AnnotationGroup
Represents a group of annotations. All annotations in the group are of the same type (e.g. PointAnnotation), have the same label, description and category and type. The category and type are codes that are used to define the annotated feature. A good resource for working with codes is availablehere.Annotation
Represents a annotation. An Annotation has a geometry (currently Point, Polyline, Polygon) and an optional list of Measurements.Measurement
Represents a measurement for an Annotation. A Measurement consists of a type-code (e.g. "Area"), a value and a unit-code ("mm")Codes that are defined in the 222-draft can be created using the create(source, type) function of the ConceptCode-class.Load a WSI dataset from files in folder.fromwsidicomimportWsiDicomslide=WsiDicom.open("path_to_folder")Create a point annotation at x=10.0, y=20.0 mm.fromwsidicomimportAnnotation,Pointpoint_annotation=Annotation(Point(10.0,20.0))Create a point annotation with a measurement.fromwsidicomimportConceptCode,Measurement# A measurement is defined by a type code ('Area'), a value (25.0) and a unit code ('Pixels).area=ConceptCode.measurement('Area')pixels=ConceptCode.unit('Pixels')measurement=Measurement(area,25.0,pixels)point_annotation_with_measurment=Annotation(Point(10.0,20.0),[measurement])Create a group of the annotations.fromwsidicomimportPointAnnotationGroup# The 222 supplement requires groups to have a label, a category and a typegroup=PointAnnotationGroup(annotations=[point_annotation,point_annotation_with_measurment],label='group label',categorycode=ConceptCode.category('Tissue'),typecode=ConceptCode.type('Nucleus'),description='description')Create a collection of annotation groups.fromwsidicomimportAnnotationInstanceannotations=AnnotationInstance([group],'volume',slide.uids)Save the collection to file.annotations.save('path_to_dicom_dir/annotation.dcm')Reopen the slide and access the annotation instance.slide=WsiDicom.open("path_to_folder")annotations=slide.annotationsSetup environment for developmentRequires poetry installed in the virtual environment.git clone https://github.com/imi-bigpicture/wsidicom.gitpoetry installTo watch unit tests use:poetry run pytest-watch -- -m unittestThe integration tests uses test images from nema.org that's needs to be downloaded. The location of the test images can be changed from the default tests\testdata\slides using the environment variable WSIDICOM_TESTDIR. Download the images using the supplied script:python .\tests\download_test_images.pyIf the files are already downloaded the script will validate the checksums.To run integration tests:poetry run pytest -m integrationData structureA WSI DICOM pyramid is inwsidicomrepresented by a hierarchy of objects of different classes, starting from bottom:WsiDicomReader, represents a WSI DICOM file reader, used for accessing WsiDicomFileImageData and WsiDataset.WsiDicomFileImageData, represents the image data in one or several (in case of concatenation) WSI DICOM files.WsiDataset, represents the image metadata in one or several (in case of concatenation) WSI DICOM files.WsiInstance, represents image data and image metadata.Level, represents a group of instances with the same image size, i.e. of the same level.Pyramid, represents a group of levels, i.e. the pyrimidal structure.Pyramids, represents a collection of pyramids, each with different image coordate system or extended depth of field.WsiDicom, represents a collection of pyramids, labels and overviews.Labels and overviews are structured similarly to levels, but with somewhat different properties and restrictions. For DICOMWeb the WsiDicomFile* classes are replaced with WsiDicomWeb* classes.A Source is used to create WsiInstances, either from files (WsiDicomFileSource) or DICOMWeb (WsiDicomWebSource), and can be used to to Initiate aWsiDicomobject. A source is easiest created with the open() and open_web() helper functions, e.g.:slide=WsiDicom.open("path_to_folder")Code structurecodec- Encoders and decoders for image pixel data.file- Implementation for reading and writing DICOM WSI files.group- Group implementations, e.g. Level.instance- Instance implementations WsiIsntance and WsiDataset, the metaclass ImageData and ImageData implementations WsiDicomImageData and PillowImageData.metadata- Metadata models and schema for serializing and deserializing to DICOM and json.series- Series implementations Levels, Labels, and Overview.web- Implementation for reading DICOM WSI from DICOMWeb.conceptcode.py- Handling of DICOM concept codes.config.py- Handles configuration settings.errors.py- Custom errors.geometry.py- Classes for geometry handling.graphical_annotations- Handling graphical annotations.source.py- Metaclass Source for serving WsiInstances to WsiDicom.stringprinting.py- For nicer string printing of objects.tags.py- Definition of commonly used DICOM tags.threads.py- Implementation of ThreadPoolExecutor that does not use a pool when only single worker.uid.py- Handles DICOM uids.wsidicom.py- Main class with methods to open DICOM WSI objects.Adding support for other file formatsSupport for other formats (or methods to access DICOM data) can be implemented by creating a new Source implementation, that should create WsiInstances for the implemented formats. A format specific implementations of theImageDatais likely needed to access the WSI image data. Additionally a WsiDataset needs to be created that returns matching metadata for the WSI.The implemented Source can then create a instance from the implemented ImageData (and a method returning a WsiDataset):image_data=MyImageData('path_to_image_file')dataset=create_dataset_from_image_data(image_data)instance=WsiInstance(dataset,image_data)The source should arrange the created instances and return them at the level_instances, label_instances, and overview_instances properties. WsiDicom can then open the source object and arrange the instances into levels etc as described in 'Data structure'.Other DICOM python toolspydicomhighdicomwsidicomizerdicomslideopenslide-pythonContributingWe welcome any contributions to help improve this tool for the WSI DICOM community!We recommend first creating an issue before creating potential contributions to check that the contribution is in line with the goals of the project. To submit your contribution, please issue a pull request on theimi-bigpicture/wsidicom repositorywith your changes for review.Our aim is to provide constructive and positive code reviews for all submissions. The project relies on gradual typing and roughly follows PEP8. However, we are not dogmatic. Most important is that the code is easy to read and understand.Acknowledgementwsidicom: Copyright 2021 Sectra AB, licensed under Apache 2.0.This project is part of a project that has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 945358. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA. IMI website: <www.imi.europa.eu>