Package: metaclean3
MetaClean3.0MetaClean3.0 cleans/removes low-quality events inFCS (flow cytometry standard)data detected during irregular flow.This method is developed for and funded byMetafora biosystem'sMetaFlowplatform.InstallationPre-requisites:Python 3PyTorchandcuML(Optional, for processing on GPU)To install Python 3 on Linux:metaclean3_public$ sudo apt install python3metaclean3_public$ sudo apt install python3-pipInstall this package:Install this package via PyPI:metaclean3_public$ pip install metaclean3OR install this package locally:Download this repository.Build and install the package; on Linux, go to the repository directory:metaclean3_public$ python -m pip install pip-toolsmetaclean3_public$ pip-compile pyproject.toml # generate requirements.txtmetaclean3_public$ pip-sync # install dependencies from generate requirements.txtmetaclean3_public$ pip install .UsageSee APIhere.MetaClean3.0 contains three classes, to of which you will directly interact with:Once you have installedmetaclean3, in Python:Load your .fcs file using thefcsparserpackage. (Optional) We recommend that you compensate and transform your file before applying MetaClean3.0 for optimal results.importfcsparserfcs_file_local='/path/to/fcs_file.fcs'meta,data=fcsparser.parse(fcs_file_local,reformat_meta=False)Instantiate an FCSfile classfthat will help format your .fcs file for MetaClean3.0. We strongly recommend users to compensate (and optionally transform) the .fcs file before applying MetaClean3.0.frommetaclean3importFCSfilefrommetaclean3.utilsimport(get_timestep,get_spillover_raw,apply_compensation_matrix)# compensate .fcs filesm=get_spillover_raw(meta=meta,dat_columns=data.columns)ifnot(smisNone):data[sm.columns]=apply_compensation_matrix(data[sm.columns],sm)# instantiate FCSfilef=FCSfile(data=data,time_step=get_timestep(meta))Run MetaClean3.0 on your filef.frommetaclean3importMetaCleanFCSd=MetaCleanFCS().apply(fcs=f)If you wish, MetaClean3.0 can save cleaning processplotsto a directory of your choice:png_dir='/path/to/save/png/plots/in'd=MetaCleanFCS(png_dir=png_dir).apply(fcs=f)The outputpandas.DataFrame,dcontains additional columns:clean_keep: The boolean column indicating which events to keep (True) or remove (False). This is the final result you are looking for!bin: The bin label for each event.outlier_keep: The boolean column indicating which events were deemed as outliers (False).val_*: The feature values extracted based on the chosen fluorescent measurement.General-purpose cleaningIf you already have your way of extracting features and binning your .fcs data and you just want to execute the cleaning portion of MetaClean3.0 on apandas.DataFrame,binned_data, you can do so directly withMetaClean:frommetaclean3importMetaCleand=MetaClean().clean(data=binned_data)# all columns will be used.Fine-tuning MetaClean3.0For most cases, we recommend using default settings. However, if there are cases when you want to fine-tune results, some common arguments you can change are listed below. See attributes in theMetaCleanFCSandMetaCleanfor more details.FCSfilearguments control how events are binnedSeeFCSfileAPIhere.min_bin_sizeandmax_bin_size(default:2000and10000): The minimum and maximum number of bins allowed. If you have a large file with more than ten million events and you want MetaClean3.0 results to be more precise, you can increasemax_bin_size. Adjust with moderation.f=FCSfile(data=data,time_step=get_timestep(meta),min_bin_size=2000,max_bin_size=10000)MetaCleanFCSarguments control feature selection and generationSeeMetaCleanFCSAPIhere.fluo_chans_no(default:4): The number of fluorescent measurements to consider.d=MetaCleanFCS(fluo_chans_no=4).apply(fcs=f)fluo_chans_clean(default:None): If you already know what fluorescent measurements you want MetaClean3.0 to refer to when determining what events to keep, you can list them here as a string list. To see what fluorescent measurements are in your file, refer to yourFCSfileinstance,f.fluo_chans.d=MetaCleanFCS(fluo_chans_clean=['FL1','FL2']).apply(fcs=f)candidate_chans_type(default:fluo): The type of channels to use for cleaning i.e.fluo(fluorescent),phys(physical morphology), orall.d=MetaCleanFCS(candidate_chans_type=fluo).apply(fcs=f)rm_outliers(default: 'all'): MetaClean3.0 detects and removes outliers so they do not skew MetaClean3.0's judgement when removing low-quality events. If, in the final results, you do not want to remove all of these outliers, you can specify to keep them by settingrm_outliersto'all'. If you want to keep some of the less outlying events, set this parameter to 'some' and if you want to remove all outliers, set this parameter to 'none'.d=MetaCleanFCS(rm_outliers='all').apply(fcs=f)n_cores(default:-1): The number of cores to use while calculating the density feature. Set to -1 to use all cores.d=MetaCleanFCS(n_cores=-1).apply(fcs=f)MetaCleanarguments control degree of leniencySeeMetaCleanAPIhere.min_ref_percent(default:0.440%; range:[0, 1]): The minimum percentage of bins they want MetaClean3.0 to keep.d=MetaCleanFCS(min_ref_percent=0.4).apply(fcs=f)min_ref_percent_to_keep(default:0.440%; range:[0, 1]): If non-reference segments contain at leastmin_ref_percent_to_keepbins, we keep it even if it differs from the longest reference segment.d=MetaCleanFCS(min_ref_percent_to_keep=0.4).apply(fcs=f)p_thres(default:0.05; range:[0, 1]): Adjacent event segments (along time) that do not have significantly different values (i.e. p-value >p_thres) are merged. Ifp_thresis small, MetaClean3.0 will remove fewer events (i.e. be more lenient).d=MetaCleanFCS(p_thres=0.05).apply(fcs=f)percent_shifts(default:[0.15, 0.2, 0.25, 0.3, 0.35, 0.4]; range:[0, 1]): Adjacent event segments (along time) that have quantile values within a specified range of each other are merged.percent_diff=0.05indicates that the quantiles to be tested are 5s% and 95%.percent_shifts=[0.15, 0.2]indicates that MetaClean3.0 will test if the segments have quantile values within a 15% and 20% range of each other, whiever yeilds results that achieve a higher silhouette score. Smallpercent_shiftvalues mean that MetaClean3.0 will remove fewer events (i.e. be more lenient).d=MetaCleanFCS(percent_shift=[0.15]).apply(fcs=f)Edge casesDuplicate rows: if there are duplicate rows in input matrixdata, the last column of these duplicate rows will be purturbed by a neglige-able value of around 1/50000. This is to prevent infinite values when calculating the density feature.If you already deal with duplicate rows outside of MetaClean3.0, you can remove this step by settingrandomize_duplicates_tftoFalseto shorten runtime:d=MetaCleanFCS(percent_shift=[0.15]).apply(fcs=f,randomize_duplicates_tf=False)API/DocumentationSee APIhere.This repository usesSphinxto generate documentation contained in thedocs. To generate documentation locally on Linux:install Sphinx for Python:metaclean3_public$ apt-get install python3-sphinxmetaclean3_public$ pip3 install -r docs/source/requirements.txtmetaclean3_public$ # sphinx-quickstart docsGenerate the HTML documentation:metaclean3_public$ sphinx-build -M html docs/source/ docs/build/Now you can open the front page in your web browser:docs/build/html/index.html.DevelopmentUnit testing: put test .fcs files intests/databefore starting!!!metaclean3_public$ python3 -m unittest discover testsBuild and compile package:metaclean3_public$ python -m pip install pip-toolsmetaclean3_public$ pip-compile --extra dev pyproject.toml # generate requirements.txtmetaclean3_public$ pip install -r requirements.txt # test installation of dependenciesmetaclean3_public$ sudo python3 -m build # builds package into a zipmetaclean3_public$ twine check dist/* # check if package description renders on PyPIUpload to TestPyPI:metaclean3_public$ twine upload -r testpypi dist/* # uploadmetaclean3_public$ python -m pip install -i https://test.pypi.org/simple metaclean3 # test installUpload package to PyPI:metaclean3_public$ twine upload dist/*metaclean3_public$ pip install metaclean3 # test installSupportSee theissuessection of this repository.CitationIf this package helped you in any way please cite MetaClean3.0's paper:(TBP)