Package: pycytominer
Data processing for image-based profilingPycytominer is a suite of common functions used to process high dimensional readouts from high-throughput cell experiments.
The tool is most often used for processing data through the following pipeline:Click here for high resolution pipeline imageImage data flow from a microscope to cell segmentation and feature extraction tools (e.g. CellProfiler or DeepProfiler).
From here, additional single cell processing tools curate the single cell readouts into a form manageable for pycytominer input.
For CellProfiler, we usecytominer-databaseorCytoTable.
For DeepProfiler, we include single cell processing tools inpycytominer.cyto_utils.From the single cell output, pycytominer performs five steps using a simple API (described below), before passing along data tocytominer-evalfor quality and perturbation strength evaluation.InstallationYou can install pycytominer via pip:pipinstallpycytomineror conda:condainstall-cconda-forgepycytominerFrameworksPycytominer is primarily built on top ofpandas, also using aspects of SQLAlchemy, sklearn, and pyarrow.Pycytominer currently supportsparquetand compressed text file (e.g..csv.gz) i/o.APIPycytominer has five major processing functions:Aggregate - Average single-cell profiles based on metadata information (most often "well").Annotate - Append metadata (most often from the platemap file) to the feature profileNormalize - Transform input feature data into consistent distributionsFeature select - Exclude non-informative or redundant featuresConsensus - Average aggregated profiles by replicates to form a "consensus signature"The API is consistent for each of these functions:# Each function takes as input a pandas DataFrame or file path# and transforms the input data based on the provided options and methodsdf=function(profiles_or_path,features,samples,method,output_file,additional_options...)Each processing function has unique arguments, see ourdocumentationfor more details.UsageThe default way to use pycytominer is within python scripts, and using pycytominer is simple and fun.# Real world exampleimportpandasaspdimportpycytominercommit="da8ae6a3bc103346095d61b4ee02f08fc85a5d98"url=f"https://media.githubusercontent.com/media/broadinstitute/lincs-cell-painting/{commit}/profiles/2016_04_01_a549_48hr_batch1/SQ00014812/SQ00014812_augmented.csv.gz"df=pd.read_csv(url)normalized_df=pycytominer.normalize(profiles=df,method="standardize",samples="Metadata_broad_sample == 'DMSO'")Pipeline orchestrationPycytominer is a collection of different functions with no explicit link between steps.
However, some options exist to use pycytominer within a pipeline framework.ProjectFormatEnvironmentpycytominer usageProfiling-recipeyamlagnosticfull pipeline supportCellProfiler-on-TerraWDLgoogle cloud / Terrasingle-cell aggregationCytoSnakesnakemakeagnosticfull pipeline supportA separate project calledAuSPICESoffers pipeline support up to image feature extraction.Other functionalityPycytominer was written with a goal of processing any high-throughput image-based profiling data.
However, the initial use case was developed for processing image-based profiling experiments specifically.
And, more specifically than that, image-based profiling readouts fromCellProfilermeasurements fromCell Paintingdata.Therefore, we have included some custom tools inpycytominer/cyto_utilsthat provides other functionality:CellProfiler CSV collationCell locations lookup table generationGenerating gct files for Morpheus visualizationNote,pycytominer.cyto_utils.cells.SingleCells()contains code to interact with single-cell SQLite files, which are output from CellProfiler.
Processing capabilities for SQLite files depends on SQLite file size and your available computational resources (for ex. memory and cores).CellProfiler CSV collationIf running your images on a cluster, unless you have a MySQL or similar large database set up then you will likely end up with lots of different folders from the different cluster runs (often one per well or one per site), each one containing anImage.csv,Nuclei.csv, etc.
In order to look at full plates, therefore, we first need to collate all of these CSVs into a single file (currently SQLite) per plate.
We currently do this with a library calledcytominer-database.If you want to perform this data collation inside pycytominer using thecyto_utilsfunctioncollate(and/or you want to be able to run the tests and have them all pass!), you will needcytominer-database==0.3.4; this will change your installation commands slightly:# Example for general case commit:pipinstall"pycytominer[collate]"# Example for specific commit:pipinstall"pycytominer[collate] @ git+https://github.com/cytomining/pycytominer@77d93a3a551a438799a97ba57d49b19de0a293ab"If usingpycytominerin a conda environment, in order to runcollate.py, you will also want to make sure to addcytominer-database=0.3.4to your list of dependencies.Creating a cell locations lookup tableTheCellLocationclass offers a convenient way to augment aLoadDatafile with X,Y locations of cells in each image.
The locations information is obtained from a single cell SQLite file.To use this functionality, you will need to modify your installation command, similar to above:# Example for general case commit:pipinstall"pycytominer[cell_locations]"Example using this functionality:metadata_input="s3://cellpainting-gallery/test-cpg0016-jump/source_4/workspace/load_data_csv/2021_08_23_Batch12/BR00126114/test_BR00126114_load_data_with_illum.parquet"single_single_cell_input="s3://cellpainting-gallery/test-cpg0016-jump/source_4/workspace/backend/2021_08_23_Batch12/BR00126114/test_BR00126114.sqlite"augmented_metadata_output="~/Desktop/load_data_with_illum_and_cell_location_subset.parquet"python\-mpycytominer.cyto_utils.cell_locations_cmd\--metadata_input${metadata_input}\--single_cell_input${single_single_cell_input}\--augmented_metadata_output${augmented_metadata_output}\add_cell_location# Check the outputpython-c"import pandas as pd; print(pd.read_parquet('${augmented_metadata_output}').head())"# It should look something like this (depends on the width of your terminal):#   Metadata_Plate Metadata_Well Metadata_Site  ...                                   PathName_OrigRNA ImageNumber                                        CellCenters# 0     BR00126114           A01             1  ...  s3://cellpainting-gallery/cpg0016-jump/source_...           1  [{'Nuclei_Location_Center_X': 943.512129380054...# 1     BR00126114           A01             2  ...  s3://cellpainting-gallery/cpg0016-jump/source_...           2  [{'Nuclei_Location_Center_X': 29.9516027655562...Generating a GCT file for morpheusThe softwaremorpheusenables profile visualization in the form of interactive heatmaps.
Pycytominer can convert profiles into a.gctfile for drag-and-drop input into morpheus.# Real world exampleimportpandasaspdimportpycytominercommit="da8ae6a3bc103346095d61b4ee02f08fc85a5d98"plate="SQ00014812"url=f"https://media.githubusercontent.com/media/broadinstitute/lincs-cell-painting/{commit}/profiles/2016_04_01_a549_48hr_batch1/{plate}/{plate}_normalized_feature_select.csv.gz"df=pd.read_csv(url)output_file=f"{plate}.gct"pycytominer.cyto_utils.write_gct(profiles=df,output_file=output_file)