Package: acutils-python
acutilsPython library providing a robust pipeline for data processing tasks.The acutils library is designed to facilitate data processing tasks, especially for individuals dealing with custom data preprocessing before building machine learning algorithms. It has been used in various domains, including pathology image processing, custom segmentation, and frame extractions from videos.HERE ARE THEONLINE DOCUMENTATIONAND THEDOCUMENTATION FILES.Key featuresYou only need tocodeonefunctionfor a custom treatment, nothing else.Easyrandomdistribution/split of the data intodatasets.Remember andreproduceyourdistribution/splitby saving it toJSONfiles.Made formultiprocessingand facilitate GPU usage for computation.Works withanykind ofdatafiles.If somefilesarerelated(for example: two medical images from the same patient), you candefinegroupsto ensure that those are in the same dataset (toavoidbiases).Brief exampleimportacutilsasau# Define the handler and linked it to a source directoryhandler=au.handler.DataHandler('./data',allowed_cpus=2)# Load filenames and labelshandler.load_data_fromdatapath()handler.load_labels_fromsheet('./labels.xlsx',idcol="id",labelcol="label")# Even load relations between files through groups (optional for split)handler.load_groups_fromsheet(os.path.join(DIR,'./labels.xlsx'),idcol="id",groupcol="patient")# Randomly split into datasets (dict<filename,label>) and balance themtdata,vdata=handler.split(train_percentage=0.70)# use groups if definedbal_tdata,bal_vdata=handler.balance_datasets(tdata,vdata)#TODOdefyour_custom_treatment(self,src,dstdir,arg1):au.file.tmnt_copyfile_to_dir(src,dstdir)# example# Process the data using your custom function and save the datasets:handler.make_datasets('./train_bal','./val_bal',bal_tdata,bal_vdata,func=your_custom_treatment,# custom functionarg1="very useful argument")# its argumentsInstallationIt should work using any OS, but for now, we only tested using Ubuntu 22.04.It works using any Python version >= 3.8 (maybe lowers too, but not tested yet).From pippipinstallacutils-pythonFrom this repository (still pip though :)pipinstall--upgradebuildpython3-mbuildpipinstalldist/acutils_python-0.1.1-py3-none-any.whlAdditional requirementsPillow,scikit-image,poochandopenslide-python:pathologymodule.opencv-python:image,pathologyandvideomodules.pipinstallopencv-pythonPillowscikit-imagepoochopenslide-pythonFinally, thepathologymodule requires you to installOpenslide,openslide-pythonis just a mapping of it. Maybe reinstall openslide-python after installing Openslide, but it should not be necesarry.GPU computationFor now, this is only used inpathologymodule (because the process takes a while).cupy: numpy but using GPU.cucim: includes cucim.skimage that is skimage (older version) using GPU.pipinstallcupycucimMake your CUDA install locatable from cupy:exportLD_LIBRARY_PATH=/path/to/cudnn/lib:$LD_LIBRARY_PATHChoose at least one device (if multiple, the first is taken):exportCUDA_VISIBLE_DEVICES=0ModulesUse the relevant modules from acutils based on your data processing needs:handler: High-level classes to handle data processing.file: About directories and files.gpu: GPU computation (for now, only used in pathology module).image: Computer Vision tasks on images.multiprocess: Multiprocessing and process management.pathology: Pathology data processing for segmentation and tiling.sheet: Handling pandas DataFrames.video: Computer Vision tasks on Videos.Refer to theonline documentationand codeexamplesfor detailed usage instructions.TBDProvide more code examples.Define a test pipeline to check if all features work.Ensure that acutils works on multiple OS and Python versions.LicenseApache-2.0, see theLICENSE.ContributingThis library is maintained byAcuzle's development team, lead by@ThomasPDM.We welcome and appreciate contributions from the community to enhance acutils. If you have ideas, bug fixes, or new features that can benefit others, we encourage you to contribute to the project. Just fork the project, create a new branch, do whatever you want and create a pull request.