Package: cleanX
cleanXCleanX is an open source python library for exploring, cleaning and augmenting
large datasets of X-rays, or certain other types of radiological images.  The images
can be extracted fromDICOMfiles or used
directly. The primary authors are Candace Makeda H. Moore, Oleg Sivokon, and Andrew Murphy.DocumentationOnline documentation is athttps://drcandacemakedamoore.github.io/cleanX/.
You can also build up-to-date documentation, which will be generated in ./build/sphinx/html directory, by command as follows:python setup.py apidoc
python setup.py build_sphinxSpecial additional documentation for medical professionals with limited programming ability is availablehereon the project wiki. To get a high level overview of some of the functionality of the program you can look at the Jupyter notebooks insideworkflow_demo folder.RequirementsPython3.7, 3.8, 3.9.  Python
3.10 has not been tested yet.Ability to create virtual environments (recommended, not absolutely
necessary)tesserocr,matplotlib,pandas, andopencvOptional recommendation ofSimpleITKorpydicomfor DICOM/dcm to JPG
conversionAnaconda is now supported, but not technically necessarySupported PlatformscleanX is a pure Python package, but it has many dependencies on
native libraries.  We try to test it on as many platforms as we can to
see if dependencies can be installed there.  Below is the list of
platforms that will potentially work. Please note that where
python.org Python or Anaconda Python stated as supported, it means
that versions 3.7, 3.8 and 3.9 (but not 3.10) are supported.AMD64 (x86)LinuxWinOSXSupportedUnknownUnknownSupportedSupportedSupportedARM64Unsupported at the moment on both Linux and OSX, but it's likely that
support will be added in the future.32-bit Intell and ARMWe don't know if either one of these is supported. There's a good
chance that 32-bit Intell will work.  There's a good chance that ARM
won't. It's unlikely that the support for ARM will be added in the
future.Installationsetting up a virtual environment is desirable, but not absolutely
necessaryactivate  the environmentAnaconda Installationuse command for conda as belowcondainstall-cdoctormakeda-cconda-forgecleanxYou need to specify both channels because there are some cleanX
dependencies that exist in both Anaconda main channel and in
conda-forgepip installationuse pip as belowpipinstallcleanXThetesserocrpackage deserves a special mention.  It is not
possible to installtesseractlibrary from PyPI server.  Thetesserocris simply a binding to the library.  You will need to
install the library yourself.  For example, on Debian flavor Linux,
this might work:sudoapt-getinstalllibleptonica-dev\tesseract-ocr-all\libtesseract-devGetting StartedWe will imagine a very simple scenario, where we need to automate
normalization of the images we have.  We stored the images in
directory/images/to/clean/and they all have ajpgextension.  We
want the cleaned images to be saved in thecleaneddirectory.Normalization here means ensuring that the lowest pixel value (the
darkest part of the image) is as dark as possible and that the
lightest part of the image is as light as possible.CLI ExampleThe problem above doesn't require writing any new Python code.  We can
accomplish our task by calling thecleanXcommand like this:mkdircleaned

python-mcleanXimagesrun-pipeline\-sAcquire\-sNormalize\-s"Save(target='cleaned')"\-j\-r"/images/to/clean/*.jpg"Let's look at the command's options and arguments:python -m cleanXis the Python's command-line option for loading
thecleanXpackage.  All command-line arguments that follow this
part are interpreted bycleanX.imagessub-command is used for processing of images.run-pipelinesub-command is used to start aPipelineto process
the images.-s(repeatable) option specifiesPipelineStep.  Steps map to
their class names as found in thecleanX.image_work.stepsmodule.
If the__init__function of a step doesn't take any arguments, only
the class name is necessary.  If, however, it takes arguments, they
must be given using Python's literals, using Python's named arguments
syntax.-joption instructs to createjournalingpipeline.  Journaling
pipelines can be restarted from the point where they failed, or had
been interrupted.-rallows to specify source for the pipeline.  While, normally, we
will want to start withAcquirestep, if the pipeline was
interrupted, we need to tell it where to look for the initial
sources.Once the command finishes, we should see thecleaneddirectory filled
with images with the same names they had in the source directory.Let's consider another simple task: batch-extraction of images from
DICOM files:mkdirextracted

python-mcleanXdicomextract\-idir/path/to/dicoms/-oextractedThis callscleanXCLI in the way similar to the example above, however,
it calls thedicomsub-command withextract-imagessubcommand.-itellscleanXto look for directory named/path/to/dicoms-otellscleanXto save extracted JPGs inextracteddirectory.If you have any problems with this check#40and add
issues or discussions.Coding ExampleBelow is the equivalent code in Python:importosfromcleanX.image_workimport(Acquire,Save,GlobSource,Normalize,create_pipeline,)dst='cleaned'os.mkdir(dst)src=GlobSource('/images/to/clean/*.jpg')p=create_pipeline(steps=(Acquire(),Normalize(),Save(dst),),journal=True,)p.process(src)Let's look at what's going on here.  As before, we've created a
pipeline usingcreate_pipelinewith three steps:Acquire,NormalizeandSave.  There are several kinds of sources available
for pipelines.  We'll use theGlobSourceto match our CLI example.
We'll specifyjournal=Trueto match the-jflag in our CLI
example.And for the DICOM extraction we might use similar code:importosfromcleanX.dicom_processingimportDicomReader,DirectorySourcedst='extracted'os.mkdir(dst)reader=DicomReader()reader.rip_out_jpgs(DirectorySource('/path/to/dicoms/','file'),dst)This will look for the files withdcmextension in/path/to/dicoms/and try to extract images found in those files,
saving them inextracteddirectory.Developer's GuidePlease refer toDeveloper's Guidefor more detailed explanation.Developing Using Anaconda's PythonUse Git to check out the project's source, then, in the source
directory run:condacreate-ncleanx
condaactivatecleanx
python./setup.pygenconda
python./setup.pyinstall_devNote that the last command may result in errors related toconda-buildbeing unable to delete Microsoft's C++ runtime DLL.  This is typical
behavior ofconda-buildas can be seen here:https://github.com/conda/conda/issues/7682The workaround is to add:condaconfig--setalways_copytrueAnd re-run the last step (this will make virtual environment created withcondanoticeably bigger).You may have to do this for Python 3.7, Python 3.8 and Python 3.9 if
you need to check that your changes will work in all supported
versions.Thegencondacommand needs to run only once per checkout and version
of Python used.  At the moment, it's not possible to have multiplecondapackage configurations generated at the same time.  So, if you
are switching Python versions, you will need to rerun this command.Also note that the build will package only the changes committed to
the Git repository.  This means that if you are building with
uncommitted changes, they will not make it into the built package.
The decision to do this was motivated by the presence of symbolic
links in the working directory, which makes it impossible to build
without superuser permissions on MS Windows.  It is possible that in
the future we will add a flag tosetup.py installto allow "dirty"
builds.To run unit test and linter you may use:pythonsetup.pylintandpythonsetup.pytestrespectively.  Note that by default, these commands will try to installcleanXand its dependencies before doing any work.  This may take a
very long time, especially on MS Windows.  There is a way to skip the
installation part by running:pythonsetup.pylint--fastandpythonsetup.pytest--fastDeveloping Using python.org's PythonUse Git to check out the project's source, then in the source
directory run:python-mvenv.venv
../.venv/bin/activate
python./setup.pyinstall_devSimilar tocondabased setup, you may have to use Python versions
3.7, 3.8 and 3.9 to create three different environments to recreate
our CI process.Build up-to-date documentationDocumentation can begenerated by command. The documentation
will be generated in a./build/sphinx/htmldirectory. Documentation is generated
automatically as new functions are added.About using this libraryIf you use the library, please cite the package.
CleanX is free ONLY when used according to license.You can get in touch with me by starting adiscussionif you
have a legitimate reason to use my library without open-sourcing your
code base, or following other conditions, and I can make you,
specifically, a different license.We are adding new functions and classes all the time. Many unit tests
are available in the test folder. Test coverage is currently
partial. Some newly added functions allow for rapid automated data
augmentation (in ways that are realistic for radiological data) and some
preliminary image quality checks. Some other classes and functions are for
cleaning datasets including ones that:Get image and metadata out of dcm (DICOM) files into jpeg and csv
filesProcess datasets from csv or json or other formats to generate
reportsRun on dataframes to make sure there is no image leakageRun on a dataframe to look for demographic or other biases in
patientsCrop off excessive black frames (run this on single images) one at a
timeRun on a list to make a prototype tiny Xray others can be compared
toRun on image files which are inside a folder to check if they are
"clean"Take a dataframe with image names and return plotted(visualized)
imagesRun to make a dataframe of pics in a folder (assuming they all have
the same 'label'/diagnosis)Normalize images in terms of pixel values (multiple methods)All important functions are documented in the online documentation for
programmers. You can also check out one of our videos by clicking the
linked picture below:cleanX: video demonstration