Package: hts-tools
🏮 hts-toolsParsing and analysing platereader absorbance and fluorescence data.InstallationCommand-line usageFirst pipeline exampleParsing platereader exportsAdding experimental conditionsNormalization within batchesPlotting dose responsesSecond pipeline exampleConverting plate-shaped data to columnsStatistical testingOther commandsPython APIDocumentationInstallationThe easy wayInstall the pre-compiled version from PyPI:pipinstallhts-toolsFrom sourceClone the repository, thencdinto it. Then run:pipinstall-e.Command-line usagehts-toolsprovides command-line utlities to analyse and plot data
from platereaders, starting with therawexported data with no
manual conversion or copy-pasting needed. The tools complete specific tasks which
can be easily composed into analysis pipelines, because the TSV table output goes tostdoutby default so they can be piped from one tool to another.To get a list of commands (tools), dohts--helpAnd to get help for a specific command, dohts<command>--helpFor the Python API,see below.First pipeline exampleThis command takes several exported Excel files (matching the patternplate-?.xlsx) from a Biotek platereader,
adds annotations on experimental conditions, and normalizes the data based on positive and negative controls.
Finally, dose-response curves are plotted.htsparseplate-?.xlsx--data-shaperow--vendorBiotek\|htsjoin--rightlayout.xlsx\|htsnormalize--controlcompound_name\--positiveRIF--negativeDMSO\--groupingplate_id--methodnpg\|htsplot-dose-xconcentration--facetsguide_name--colorcompound_name\--outputplt-testParsing platereader exportsThe commandhts parseconverts Excel or CSV or TSV files exported from platereader software from a specified vendor into a
uniform columnar table format used by all downstream hts-tools. Thedata-shapeoption indicates whether the export was in plate
format or row-wise table format. It should usually be the first command in a pipeline.hts parseproduces a table with at least the following columns (with example entries):row_idcolumn_idwell_idplate_iddata_filenamedata_sheetmeasured_abs_ch1abs_ch1_wavelengthA1A01Plate 6plate-6.xlsxPlate6 - Sheet 10.366600nmA2A02Plate 6plate-6.xlsxPlate6 - Sheet 10.402600nmIf only fluorescence was measured, the last two columns would be calledmeasured_fluor_ch1andfluor_ch1_wavelength. If there were multiple
measurements, then they will appear as additional columns with increasingch(channel) numbers, for example,measured_fluor_ch2,fluor_ch2_wavelength.There will also be additional columns using information scraped from the input file. These will usually have headings starting withmeta_,
such asmeta_protocol_file_path,meta_date,meta_time, andmeta_reader_serial_number.Adding experimental conditionsThe output fromhtsparse is passed tohts join, which combines two tables based on values in shared columns (known as a
database join). For example, iflayout.xlsxcontains a sheet with this table:column_idplate_idcompound_name2Plate 6trimethoprim1Plate 6moxifloxacinthenhts parse plate-?.xlsx --data-shape row | hts join --right layout.xlsxwill result inrow_idcolumn_idwell_idplate_iddata_filenamedata_sheetmeasured_abs_ch1abs_ch1_wavelengthcompound_nameA1A01Plate 6plate-6.xlsxPlate6 - Sheet 10.366600nmmoxifloxacinA2A02Plate 6plate-6.xlsxPlate6 - Sheet 10.402600nmtrimethoprimSincecolumn_idandplate_idare the only column headings in common, the entries of these columns are used to match the rows of the
tables. So wherecolumn_id = 2,compound_name = trimethoprimwill be added.If you join an Excel XLSX file containing multiple sheets, these will each be joined in order. In this way, you can add experimental
conditions easily by, for example, first joining the conditions that vary by plate and row (such as compound), then by column
(such as concentration). This approach is very flexible, and you can join on any number of columns and add any new ones you like as
long as the column headings aren't repeated.Normalization within batchesThehts normalizecommand normalizes raw measured data based on controls. This can add power to
downstream statsical analyses by mitigating batch-to-batch variation.hts normalizeadds new columns for each measured column. These columns start withcalc_and end with_norm.{method}, for examplecalc_abs_ch1_norm.npgandcalc_abs_ch2_norm.npg. Two methods are offered which
optionally normalize within within groups, such as batches or plates.Normalized proportion of growth (NPG)This method scales raw data to be between $0$ and $1$ based on positiveandnegative
controls, optionally within groups (or batches) of measurements. In the example above,
the positive and negative controls are defined asRIFandDMSO, and should be found in the columncompound_name(which may have been added byhts join).The positive and negative controls are averaged within each value in the--groupingcolumn. In
the example above, they will be averaged for eachplate_id, and these will be used to normalize
the measured values of thatplate_idaccording to:$$s = \frac{m - \mu_p}{\mu_n - \mu_p}$$where $s$ is the normalized value, $m$ is the measured value, and $\mu_p$ and $\mu_n$ are the mean positive and negative controls.Proportion of negative (PON) [default]This method scales raw data relative negative controlsonly, optionally within groups (or batches) of measurements.The negative controls are averaged within each value in the--groupingcolumn. In
the example above, they would be averaged for eachplate_id, and these will be used to normalize
the measured values of thatplate_idaccording to:$$s = \frac{m}{\mu_n}$$where $s$ is the normalized value, $m$ is the measured value, and $\mu_n$ is the mean negative control.Plotting dose responsehts plot-doseis a very flexible command which takes the columnar tables as input and plots the data in almost any breakdown using a color-blind
palette. The required-xoption indicates which column to use as the x-axis (usually concentration). The y-axis will be values in all the measured
and calculated columns (hts plot-drplots them all automatically in seaparte files). The other options allow splitting the plots by file, facet
(panel) and color according to the values in columns.The example above,hts plot-dose -x concentration --facets guide_name --color compound_name, will produce plots like this:The panels each value with the sameguide_nameis in afacet(panel), and the lines are colored bycompound_name.Second pipeline exampleHere is another example showing the sequential use ofhts jointo join two tables of experimental data, and two other commands:hts pivotandhts summarize.htspivotcompounds.xlsx\--namecompound_name

htsparseplate-*.txt--data-shapeplate\|htsjoin--rightsample-sheet.csv\|htsjoin--rightpivoted-compounds.tsv\|htsnormalize--controlcompound_name--positiveRIF--negativeDMSO--groupingstrain_nameplate_id\|htssummarize--controlcompound_name--positiveRIF--negativeDMSO--groupingstrain_namecompound_name--plotsummary\>summary.tsvThese commands are explained below.Converting plate shaped data to columnsSometimes you will want to use data, such as plate layouts, which are in a plate-shaped layout instead of a column format. For example:You can convert this to column format usinghts pivot, which produces a table in the following format:row_idcolumn_idcompound_namewell_idplate_idfilenameC2RIFC02Plate 7compounds.xlsxD2LYSINED02Plate 7compounds.xlsxE2DMSOE02Plate 7compounds.xlsxF2RIFF02Plate 7compounds.xlsxIt is assumed that there is one plate per sheet for Excel files, and one plate per file for TSV and CSV files. The
plate name is taken from the sheet name (Excel) or filename (other formats).You can prepend the names of theplate_idandfilenamecolumns with the-xoption. for example,hts pivot compounds.xlsx -x compound_source --name compound_namewould have columnscompound_source_plate_idcompound_source_filename. This is helpful when usignhts joinlater where the plate and filename
columns would otherwise be shared but have different meanings and values, and you don't want to accidentally
join on them.Statistical testingGroups of values (such as replicates) can be compared against a negative control for statistical testing usinghts summarize. The--groupingoption indicates the columns whose values together indicate values which are
replicates of a particular condition of interest. For example,--grouping strain_name compound_namewould
indicate that values which have the samestrain_nameandcompound_nameare replicates.Statistical tests compare to the--negativevalues, and use all measured and normalized (calc_*_norm) columns.
Currently, the Student's t-test and Mann-whitney U-test are implemented. The t-test is best suited to Normal-distributed
data while the MWU is better for other distributions which might not have a nice bell curve distribution.Althoughhts summarizecalcualtes both tests simultaneously, it's not a good idea to look for "significant" $p$-values
in both. This is called p-hacking, and leads to false positives. Instead, decide which test is most appropriate for your
data and stick with that one.This command also calculates other summary statistics such as between-replicate mean, variance, and SSMD. If a filename
prefix is provided to the--plotoption, then volcano and flashlight plots are produced, which may be useful for identifying
hits of high throughput screens.Other commandsThere are several other commands fromhts-toolswhich take as input the output fromhts parse, joined
to an experimental data table (layout.xlsxin these examples).hts qcDo quality control checks by calculating mean, standard deviation, Z'-factor, and SSMD, and plotting them.htsparseplate-?.xlsx--data-shaperow\|htsjoin--rightlayout.xlsx\|htsqc--controlcompound_name--positiveRIF--negativeDMSO--groupingstrain_nameplate_id--plotqc-plot\>qc.tsvhts plot-hmPlot heatmaps of signal intensity arranged by plate well. This can be useful to identify unwanted within-plate variability.htsparseplate-?.xlsx--data-shaperow\|htsjoin--rightlayout.xlsx\|htsplot-hm--groupingstrain_namesample_idplate_id--outputhmHere,--groupingidentifies the columns which indicate values coming from the same plate. One file is produced per
measured and normalized (calc_*_norm) column.hts plot-repPlot two replicates against each other for each condition.htsparseplate-?.xlsx--data-shaperow\|htsjoin--rightlayout.xlsx\|htsplot-rep--controlcompound_name--positiveRIF--negativeDMSO--groupingstrain_namecompound_name--outputrepHere,--groupingidentifies the unique conditions within which values are treated as replicates. The positives and negatives
are plotted as different colors. One file is produced per measured and normalized (calc_*_norm) column.In the plots, the left column is on a linear-linear scale and the right column is on a log-log scale. There is one row
of plots per wavelength set in the dataset.hts plot-histPlot histograms of the data values.htsparseplate-?.xlsx--data-shaperow\|htsjoin--rightlayout.xlsx\|htsplot-hist--controlcompound_name--positiveRIF--negativeDMSO--outputhistThe positives and negatives are plotted as different colors. One file is produced per measured and normalized (calc_*_norm) column.In the plots, the left two columns are on a linear-linear scale and the right two columns are on a log-log scale. There is one row
of plots per wavelength set in the dataset.Python APIhts-toolscan be imported into Python to help make custom analyses.>>>importhtstoolsashtsYou can read raw exports from platereader software into a columnar Pandas dataframe.>>>hts.from_platereader("plates.xlsx",shape="plate",vendor="Biotek")Once in the columnar format, you can annotate experimental conditions.>>>importpandasaspd>>>a=pd.DataFrame(dict(column=['A','B','A','B'],...abs=[.1,.2,.23,.11]))>>>acolumnabs0A0.101B0.202A0.233B0.11>>>b=pd.DataFrame(dict(column=['B','A'],...drug=['TMP','RIF']))>>>bcolumndrug0BTMP1ARIF>>>shared_cols,data=join(a,b)>>>shared_cols('column',)>>>datacolumnabsdrug0A0.10RIF1A0.23RIF2B0.20TMP3B0.11TMPIf the conditions to annotate are in a plate-shaped format, you can melt them into a columnar format before joining.>>>importpandasaspd>>>importnumpyasnp>>>a=pd.DataFrame(index=list("ABCDEFGH"),...columns=range(1,13),...data=np.arange(1,97).reshape(8,12))>>>a123456789101112A123456789101112B131415161718192021222324C252627282930313233343536D373839404142434445464748E495051525354555657585960F616263646566676869707172G737475767778798081828384H858687888990919293949596>>>hts.pivot_plate(a,value_name="well_number")row_idcolumn_idwell_numberwell_idplate_id0A11A011B113B012C125C013D137D014E149E01.................91D1248D1292E1260E1293F1272F1294G1284G1295H1296H12[96rowsx5columns]This also works on the multi-sheet dictionary output ofpd.read_excel(..., sheet_names=None).>>>hts.pivot_plate({'sheet_1':a},value_name="well_number")row_idcolumn_idwell_numberwell_idplate_id0A11A01sheet_11B113B01sheet_12C125C01sheet_13D137D01sheet_14E149E01sheet_1.................91D1248D12sheet_192E1260E12sheet_193F1272F12sheet_194G1284G12sheet_195H1296H12sheet_1[96rowsx5columns]Replicates within condition groups can be annotated.>>>importpandasaspd>>>a=pd.DataFrame(dict(group=['g1','g1','g2','g2'],...control=['n','n','p','p'],...m_abs_ch1=[.1,.2,.9,.8],...abs_ch1_wavelength=['600nm']*4))>>>agroupcontrolm_abs_ch1abs_ch1_wavelength0g1n0.1600nm1g1n0.2600nm2g2p0.9600nm3g2p0.8600nm>>>hts.replicate_table(a,group='group')groupcontrolm_abs_ch1abs_ch1_wavelengthreplicate0g1n0.1600nm11g1n0.2600nm22g2p0.9600nm23g2p0.8600nm1If you prefer, you can get a "wide" output.>>>hts.replicate_table(a,group='group',wide='m_abs_ch1')replicaterep_1rep_2groupg10.20.1g20.80.9Values can be normalized to values between 0 and 1 relative to their positive (0%) and negative (100%) controls, optinally within groups or batches.>>>importpandasaspd>>>a=pd.DataFrame(dict(control=['n','n','','','p','p'],...m_abs_ch1=[.1,.2,.5,.4,.9,.8],...abs_ch1_wavelength=['600nm']*6))>>>acontrolm_abs_ch1abs_ch1_wavelength0n0.1600nm1n0.2600nm20.5600nm30.4600nm4p0.9600nm5p0.8600nm>>>hts.normalize(a,control_col='control',pos='p',neg='n',measurement_col='m_abs_ch1')controlm_abs_ch1abs_ch1_wavelengthm_abs_ch1_neg_meanm_abs_ch1_pos_meanm_abs_ch1_norm0n0.1600nm0.150.851.0714291n0.2600nm0.150.850.92857120.5600nm0.150.850.50000030.4600nm0.150.850.6428574p0.9600nm0.150.85-0.0714295p0.8600nm0.150.850.071429The scaling can be reversed withflip=True.>>>hts.normalize(a,control_col='control',pos='p',neg='n',measurement_col='m_abs_ch1',flip=True)controlm_abs_ch1abs_ch1_wavelengthm_abs_ch1_neg_meanm_abs_ch1_pos_meanm_abs_ch1_norm0n0.1600nm0.150.85-0.0714291n0.2600nm0.150.850.07142920.5600nm0.150.850.50000030.4600nm0.150.850.3571434p0.9600nm0.150.851.0714295p0.8600nm0.150.850.928571Summary statstics and statsitcial tests relative to the negative controls can be generated.>>>a=pd.DataFrame(dict(gene=['g1','g1','g2','g2','g1','g1','g2','g2'],...compound=['n','n','n','n','cmpd1','cmpd1','cmpd2','cmpd2'],...m_abs_ch1=[.1,.2,.9,.8,.1,.3,.5,.45],...abs_ch1_wavelength=['600nm']*8))>>>agenecompoundm_abs_ch1abs_ch1_wavelength0g1n0.10600nm1g1n0.20600nm2g2n0.90600nm3g2n0.80600nm4g1cmpd10.10600nm5g1cmpd10.30600nm6g2cmpd20.50600nm7g2cmpd20.45600nm>>>hts.summarize(a,measurement_col='m_abs_ch1',control_col='compound',neg='n',group='gene')geneabs_ch1_wavelengthm_abs_ch1_meanm_abs_ch1_std...m_abs_ch1_t.statm_abs_ch1_t.pm_abs_ch1_ssmdm_abs_ch1_log10fc0g1600nm0.17500.095743...0.3611580.7429220.2100420.0669471g2600nm0.66250.221265...-1.5443960.199787-0.807183-0.108233[2rowsx12columns]>>>hts.summarize(a,measurement_col='m_abs_ch1',control_col='compound',neg='n',group=['gene','compound'])genecompoundabs_ch1_wavelengthm_abs_ch1_mean...m_abs_ch1_t.statm_abs_ch1_t.pm_abs_ch1_ssmdm_abs_ch1_log10fc0g1n600nm0.150...0.0000001.0000000.0000000.0000001g2n600nm0.850...0.0000001.0000000.0000000.0000002g1cmpd1600nm0.200...0.4472140.7117230.3162280.1249393g2cmpd2600nm0.475...-6.7082040.044534-4.743416-0.252725[4rowsx13columns]DocumentationFull API documentation is atReadTheDocs.