BACKGROUND
micrornas  are a class of small  regulatory rnas.
they are essential regulators that act as translational repressors throughout many eukaryotic species  <cit> .
several thousand mirnas have been described in humans and other species, although in practicality only 350– <dig> are present at robust levels in mature cells and tissues  <cit> .
mirnas are known to alter their expression levels in disease, malignancy, and cell stress  <cit>  and exhibit tissue and cell-type specific patterns of expression  <cit> .
many expression platforms, originally designed to quantify mrna expression, have been adapted to globally assay mirna expression including hybridization arrays, quantitative pcr , and sequencing  <cit> .
however, each of these approaches must overcome several challenges specific to mirnas: short sequence length, low abundance of target molecules, and sequence homology between mirnas.
comparative performance assessments are crucial to understanding the strengths and limitations of each approach to mirna quantification.
a group of investigators recently systematically evaluated  <dig> available mirna platforms across  <dig> standardized control samples  <cit> .
this study, called mirqc, established metrics to assay reproducibility, sensitivity, accuracy, specificity and concordance across the different methods.
although a single platform was not found to be uniformly superior, there was substantial variability in performance across assessments.
for each of the platforms, performance depends on both the instrument and the algorithm used to convert raw measurements into expression estimates.
for example, one platform assessed in the mirqc study was rna-seq performed on an illumina gaiix instrument.
the sample prep used the truseq small rna prep kit and results were aligned to the hg <dig> reference sequence allowing one mismatch, without further delineation of the alignment method  <cit> .
a previous performance evaluation of mirna expression arrays noted a strong dependency between technology and signal processing methodology  <cit> .
more recently, we have demonstrated that different mirna rna-seq alignment algorithms produce different alignments, impacting the quality of the data  <cit> .
we surmise that many mirna expression platforms are not yet optimized to yield consistent and maximally accurate data.
another platform evaluated in the mirqc study was the life technologies taqman openarrayⓇ system.
this is a qpcr-based mirna array platform that currently has coverage for  <dig> human mirnas across two sets of primer pools.
while qpcr is considered the gold standard for low-throughput measurement of gene expression, microarray- and sequencing-based platforms are preferable for most high-throughput applications.
given the relative small number of common mirnas, it is possible to use a qpcr-based platform to measure the expression of all abundant mirnas in many tissues and cells.
the primary advantage of qpcr-based technologies is the ability to simultaneously amplify and quantify a target transcript over sequential pcr cycles.
the greater the initial amount of the target transcript present in a sample, the more rapidly the target will reach a threshold at which it can be detected by flourescence .
as such, the raw data produced by qpcr-based technologies are fluorescence signal intensities captured at the end of each amplification cycle .
analysis of these data typically begins by assigning a threshold cycle number to each amplification.
these threshold cycles can then be used to estimate target abundance, either relative or in reference to values for a standard curve.
for example, life technologies provides the expressionsuite software package, which uses the shape of the amplification curve to estimate a relative threshold cycle and corresponding expression estimate  <cit> .
while a substantial number of software tools have been developed to estimate gene expression from raw amplification data , these focused on mrna rather than mirna targets.
whether these methods perform similarly when estimating mirna expression is an area of ongoing research.
the software presented in this manuscript provides tools to assess and compare the performance of methods to transform raw amplification data into expression estimates and determine optimal quality thresholds.
while the mirqc study focused on comparing many different platforms, here we focus on a single platform but provide a much larger and more diverse data set for evaluation.
we believe that the availability of these data and corresponding software will greatly accelerate the development of improved methodology for the openarrayⓇ mirna platform.
furthermore, seamless integration with the r/bioconductor  <cit>  suite of analysis packages will enhance the value of openarrayⓇ mirna data.
therefore, we developed mircomp, an r package to assess and compare microrna expression estimation methods using a benchmark data set.
methods
experimental design
selection of tissues
two separate rna pools were prepared by blending two tissues each:  kidney and placenta and  skeletal muscle and brain .
these sources of rna were chosen based on our prior analysis of agilent v <dig> mirna array data that suggested this collection of tissues would capture a large number of micrornas, including several unique to each sample, such as mir-133a for skeletal muscle and the chromosome  <dig> mirna cluster for placenta  <cit> .
the surgical pathology archives of the department of pathology at johns hopkins hospital were used to obtain formalin fixed paraffin-embedded  tissues from four distinct tissue sources.
all tissues were verified as normal by review of tissue histology on an adjacent hematoxylin and eosin stained slide.
these anonymized human samples were used based on an exemption from the institutional review board of johns hopkins hospital.
rna extraction
we extracted rna from ffpe sections of kidney, placenta, skeletal muscle, and brain using the allprep dna/rna ffpe protocol .
xylene was chosen for deparaffinization.
extra xylene and ethanol washes were performed, and dnase digestion was done on-column.
rna quality control
concentration of eluted rna was assessed by nanodrop.
due to the low quality of longer rna molecules extracted from ffpe tissues, including the ribosomal rnas, the presence of several ubiquitous and tissue-enriched small rnas or mirnas was confirmed by stem-loop reverse transcription quantitative pcr using  <dig> ng rna per reaction.
for example, mir- <dig> and mir-133a were enriched in skeletal muscle, mir-516b was enriched in placenta, and mir-200b was enriched in kidney .
rna was stored at −80c.
reverse transcription and pre-amplification
the kidney/placenta  and skeletal muscle/brain  mixtures were made by combining equal masses of kidney and placenta or skeletal muscle and frontal cortex rna, respectively, and diluting to an equal concentration of  <dig>  ng/ul.
<dig> ng of rna was used as the input for reverse transcription using the a and b primer pools, following the life technologies openarrayⓇ protocol modification for low-concentration and ffpe rna.
separate reverse transcription and pre-amplification reactions were performed for the life technologies megaplex pools a and b primer pools, which reverse transcribe and pre-amplify specific micrornas.
following pre-amplification,  <dig> ul from the a and b reactions for both kp and mb were mixed with  <dig> ul of  <dig> x te.
further dilutions and combinations of the kp and mb mixtures were then prepared.
to keep the non-nucelic acid components equal after mixing kp and mb, we added a diluent c mix as needed .
the diluent c included the same proportions of rt buffer and pre-amp mix components as in the life technologies protocol-specified dilution of nucleic acid-containing post-pre-amp mixture.
the final concentrations were  <dig>   <dig>   <dig>   <dig> and  <dig> and  <dig>  % for each sample .
the sample numbers  are used throughout the manuscript to refer to specific mixture/dilution sample types.
fig.
<dig> two rna pools were formed by blending two tissues each:  kidney and placenta and  skeletal muscle and brain.
these pools were combined in different mixtures and dilutions to form the  <dig> sample types shown here.
gene expression in each sample type was assessed in quadruplicate  using the life technologies openarrayⓇ platform modified to perform  <dig> amplification cycles
life technologies openarrayⓇ assay
standard human taqmanⓇ openarrayⓇ human microrna panel, quantstudio tm 12k flex chips  and other necessary reagents were provided by life technologies for this experiment.
this panel contains  <dig> human mirna sequences from mirbase v <dig> which have all been previously functionally validated with mirna artificial templates.
for conversion of notation from mirbase v <dig> style to current mirna style, the webtool miriadne can be used  <cit> .
the specially prepared post-pre-amp dilution mixtures were added to the sample plates and then loaded onto the chips using the accufill robot following the standard protocols .
a modified microrna.edt file, provided by life technologies, was used to extend the cycles from the standard  <dig> to  <dig> cycles.
this was done to make sure all amplifications went to completion, as the authors noted that some microrna amplicons had not reached their maximal intensity at  <dig> cycles, causing a slight left shift to lower crt values in prior experiments.
the additional cycles do not increase the detection limit of the system.
three samples on one chip  were run using the standard microrna.edt file provided with the instrument, due to human error.
this did not have a noticeable effect on the expression estimates from any of the algorithms.
additional information on the taqmanⓇ openarrayⓇ microrna panels can be found in the technical manual .
expression estimation algorithms
there are a wide variety of algorithms available to estimate expression from qpcr amplification curves.
to facilitate comparisons between these algorithms, we have applied many of these algorithms to our benchmark data set.
the resulting expression estimates and quality scores are available as data objects in the mircomp package.
specifically, we provide expression estimates from the following methods:
lifetech expressionsuite
<dig> parameter sigmoidal model
<dig> parameter sigmoidal model
<dig> parameter log sigmoidal model
<dig> parameter log sigmoidal model
linear exponential model
additionally, the raw amplification data are available in the mircompdata package allowing researchers to easily generate expression estimates using other current or future algorithms.
statistical assessments
the primary goal of the mixture/dilution experiment described above is to provide a benchmark data set with which to assess the performance of methods that estimate mirna expression from qpcr amplification curves.
specifically, we propose assessments of accuracy, precision, data quality, titration response, limit of detection, and number of complete features.
each of these is described in detail below.
to avoid any confusion due to naming conventions , we refer to the reported values as expression estimates or simply expression.
quality scores
when estimating expression from amplification data, it is crucial for methods to provide both an expression estimate and a corresponding quality score.
these quality scores are often used to filter, flag, or down-weight poor quality expression estimates in subsequent analyses.
the qualityassessment function in the mircomp package allows one to examine the relationship between quality scores and expression estimates, the distribution of quality scores across samples, and the relationship between quality scores from two different methods.
expression comparison
when comparing two methods, a natural starting point is to compare the expression estimates produced by each method.
by examining the features and samples for which expression estimates differ substantially, one can better understand the strengths and limitations of each method.
the expressioncomp function in the mircomp package allows one to examine the relationship between expression estimates produced by two different methods.
feature/sample combinations for which the expression estimates differ by more than a given threshold are flagged for further investigation.
complete features
a measure of the amount of readily usable data produced by a method is the number of complete features .
complete features are defined as detected  and of good quality  across all samples in a given experiment.
the completefeatures function allows one to assess a single method or compare two methods.
limit of detection
the limit of detection is an estimate of the smallest signal that can be reliably measured.
we propose assessing the limit of detection in two ways:  examining the distribution of average observed expression stratified by the proportion of values within a set of replicates that are good quality, and  comparing the average observed vs expected expression in the two low input sample types .
the expected expression for both low input sample types  can be calculated based on the pure sample types  or, in the case of the  <dig> / <dig>  dilution , it can be calculated based on the expression in the  <dig> / <dig>  dilution .
visual representations of these comparisons are produced by the limitofdetection function.
the limitofdetection function also reports several potential limits of detection based on each of the following comparisons:
average observed expression in the  <dig> / <dig>  dilution samples  vs expected expression based on the pure samples .
average observed expression in the  <dig> / <dig>  dilution samples  vs expected expression based on the pure samples .
average observed expression in the  <dig> / <dig>  dilution samples  vs expected expression based on the  <dig> / <dig>  dilution samples .
for each of these comparisons, we calculate the difference between the observed and expected expression estimates.
to assess the limit of detection, we compute the expression threshold such that the median difference  of all features exceeding that threshold is equal to a predetermined tolerance.
the limitofdetection returns these potential limits of detection for each comparison and three tolerances .
titration response
the titration response is defined as the ability of a method to produce monotone increasing expression estimates in response to increasing amounts of input rna.
we consider sample types 2– <dig> and 6– <dig> as two separate titration series.
in each of these series, one mixture component is held constant at  <dig> μl and the other is doubled twice from  <dig> μl to  <dig> μl to  <dig> μl.
because this response will depend heavily on the underlying expression of a given feature in each mixture component, the titration response is stratified by the difference in expression between the component being titrated and the component being held constant.
for example, in the sample type 2– <dig> titration series, mixture component a is held constant and mixture component b is titrated.
to assess the difference in expression between mixture components a and b, we use the expression estimates in the pure sample types: sample type  <dig>  and sample type  <dig> .
accuracy
to assess accuracy, we calculate the signal detect slope, defined as the slope of the regression line of observed expression on expected expression, for the two titration series .
the ideal signal detect slope is one, representing agreement between observed and expected expression.
the signal detect slopes are stratified by pure sample expression.
a signal detect slope captures the average relationship between observed and expected expression; however, some features may perform well on average but be highly variable.
in the plots produced, features are displayed in grey if the signal detect slope is not statistically significantly different from zero .
as such, a grey point corresponding to a signal detect slope well above zero represents a particularly noisy  response.
precision
to assess precision, we calculate both the within-replicate standard deviation and coefficient of variation .
both statistics are calculated for each set of replicates  that are of acceptable quality.
for both summaries, the values are stratified by the average observed expression.
software
software implementing the assessments described in this manuscript was written in the open-source statistical language r   <cit> .
the r software package, mircomp, and the r data package, mircompdata, are available as part of the bioconductor project  <cit>  , a collaborative effort to develop software for computational biology and bioinformatics.
in addition to the primary functionality described above, the mircomp package contains many additional options for customizable use of these assessment functions.
these are described in the mircomp package vignette .
RESULTS
in the following, we compare two methods to generate expression estimates and quality scores from raw mirna qpcr amplification data.
the first method is an algorithm developed by life technologies and implemented in the expressionsuite software package.
this software package produces estimates of expression  and a measure of quality .
the second method is a four-parameter log-sigmoid curve-fitting algorithm  <cit>  implemented as the default method in the qpcr r package  <cit>  and referred to in this manuscript as simply qpcr.
this open-source r package produces expression estimates by fitting a four parameter log sigmoidal curve to the amplification data and computing the point at which the second derivative of this curve is maximized   <cit>  and a measure of quality .
four additional algorithms  were applied to the benchmark data set, and the resulting expression estimates and quality scores are available in the mircomp r package.
for clarity of presentation in this manuscript, we will focus on comparing two widely-used algorithms, the default algorithms from life technologies and the qpcr r package, in the following results.
quality assessment
given the interdependence between the expression estimates and quality scores produced by a method, we begin by examining this relationship for each method .
as one might expect, quality scores decrease as the expression estimates increase .
another feature of note is that both methods occasionally fail to produce an expression estimate .
however, while the qpcr method assigns all of these values fairly low quality , the life technologies method produces a substantial number of na expression estimates with high quality scores .
fig.
<dig> relationship between quality and expression.
for both data sets, a measure of data quality,  life technologies  ampscore or  qpcr r-squared, is plotted against the corresponding expression estimates.
the qpcr r-squared values are plotted on the complementary log-log scale to improve visibility in the region of interest.
each point represents a single expression measurement and corresponding quality measure for a unique mirna/sample combination.
two-dimensional scatter-plot smoothing is used to avoid over-plotting and convey the distribution of points across the plotting region
when comparing two methods, it is also interesting to examine the relationship between the quality scores produced by each method .
examination of this figure highlights regions of consensus high quality  and consensus low quality  as well as regions of disagreement between the methods .
while there are relatively few data points that are estimated with a high quality ampscore and low quality r <dig>  there are a substantial number of high quality r <dig> and low quality ampscore data points.
fig.
<dig> comparison of quality scores.
a direct comparison between the quality scores: life technologies  ampscore and qpcr r-squared.
the vertical dashed line represents the recommended ampscore threshold of  <dig> .
the horizontal dashed line represents a potential r-squared threshold chosen by examination of this figure.
each point represents the quality values for a unique mirna/sample combination.
two-dimensional scatter-plot smoothing is used to avoid over-plotting and convey the distribution of points across the plotting region
taken together, figs.
<dig> &  <dig> suggest quality thresholds of ampscore= <dig>  and r2= <dig> .
while we will use these thresholds throughout the remainder of this manuscript, all functions in the mircomp package allow the user to set their own quality thresholds.
furthermore, for many functions, one can compare results from a single method using two different quality thresholds to examine the effect of changing the quality threshold on each assessment.
lastly, when comparing two methods, we typically restrict the assessment to data considered to be good quality by both methods.
this provides the most direct comparison between the expression estimates produced by the two methods; however, for many of these assessments, the mircomp package allows one to perform these comparisons using each method’s own quality assessment independently.
expression comparison
when comparing two methods, it is also interesting to examine the relationship between the expression estimates produced by each method .
overall, the expression estimates produced by the life technologies and qpcr methods are quite similar; however, one feature/sample shows a substantial difference between methods .
one can investigate this difference further by examining the raw amplification data provided in the mircompdata package.
for comparison, we selected a feature from the same sample for which the two methods were approximate agreement .
the amplification curve for mir- <dig>  shows the probable cause of the discrepency between expression estimates – the amplification curve is still increasing at the maximum observed cycle.
in contrast, the amplification curve for mir-29a  has clearly reached a plateau before the maximum cycle.
this results in expression estimates that are fairly similar across methods.
fig.
<dig> comparison of expression estimates.
a expression estimates from life technologies  method and the default qpcr method are plotted against one another.
the mirna/sample combination in which the expression estimates differ most  is represented by a black dot.
an mirna/sample combination in which the expression estimates roughly agree  is represented by a red dot.
b & c the amplification curve data  for b mir- <dig> or c mir-29a in sample kw10: <dig> with expression estimates for each method denote by vertical lines.
note that the amplification curve is still increasing at cycle  <dig> for mir- <dig> whereas for mir-29a is has leveled off
complete features
given the difficulty in measuring the expression of many mirnas, we examine the number of complete features , partial features , and absent features .
examining the number of features that fall into each of these categories for each method  provides a straightforward assessment of the amount of easily usable data produced by each method.
of the  <dig> features in these data, there are  <dig> complete features using the life technologies method and  <dig> complete features using the qpcr method; however, this may simply be a result of the choice of quality threshold.
of note, there are  <dig> complete features in common between both methods and  <dig> features that are considered absent by both methods.
the latter may represent mirnas that are not expressed in any of the four tissues included in these data.
complete features are ones that are detected  and have good quality across all  <dig> samples.
partial features are those that are detected and good quality in at least one  of the  <dig> samples.
absent features are those that are not detected or of poor quality in all  <dig> samples.
these values are used to compare the performance of the life technologies  and qpcr algorithms
limit of detection
next, we examine the limit of detection for a given method.
while this is related to the previous assessment of complete features, here the focus is on determining the minimum signal that can be reliably detected.
the first assessment compares the difference between observed and expected expression vs the expected expression based on three different comparisons .
one can visually assess the limit of detection by identifying the expected expression value  at which the expression difference  begins to diverge from zero and the data are of increasingly poor quality.
a quantitative version of this approach is described in the methods section.
potential limits of detection for each comparison and each tolerance are shown in table  <dig>  in general, the three comparisons produce similar limits of detection for a given tolerance.
fig.
<dig> limit of detection ma-plots.
to assess the limit of detection for a given method , we plot the difference between the observed and expected average expression vs the expected expression for each feature in the two low input sample types .
the expected expression is calculated using either the pure sample expression  or the  <dig> / <dig>  dilution expression .
each point represents an mirna in the sample type of interest.
features for which all four data points are of good quality are shown in black; other features are shown in red with the proportion of poor quality values for each feature denoted by plotting symbol.
one can visually assess the limit of detect by the expected expression value  at which the difference between observed and expected expression begins to differ substantially from zero and the proportion of poor quality values increases
here we report three potential limits of detection based on the three assessments .
limits of detection are reported separately for the life technologies  and qpcr algorithms.
columns correspond to the different assessments, and rows correspond to the median difference between the observed and expected values.
the values in the matrix are the expected expression values such that the median absolute difference for all larger expected expression values is approximately equal to the threshold for that row.
for example, consider the life technologies method, if we focus on the  <dig> / <dig>  vs  <dig> / <dig>  comparison  and set a median average difference threshold of less than  <dig>  , our estimate of the limit of detection is approximately  <dig>
an alternative approach to assess the limit of detection is to examine the distribution of average observed expression among replicates  stratified by the proportion of poor quality .
this allows one to easily see that higher expression values result in increasing amounts of poor quality data; however, for both methods, there is substantial overlap between the distributions.
fig.
<dig> limit of detection boxplots.
to further examine the limit of detection, we examine boxplots of average observed expression stratified by the proportion of poor quality data points.
below each box, we also report the number of unique feature/sample type combinations each box contains.
results from the life technologies method are shown in panel a; results from the qpcr method are shown in panel b
titration response
we now turn to assessments of the expression estimates themselves.
for both methods, we calculated the proportion of features that show a titration response  for each titration series .
we also stratify these results by the difference in pure sample expression between the mixture component being titrated and the component being held constant .
as one might expect, if an mirna is far less abundant in the component being titrated than in the component being held constant, it is difficult to detect a titration response for this mirna.
if an mirna is far more abundant in the component being titrated than in the component being held constant, it is relatively easy to detect a titration response for this mirna.
while both methods perform fairly well when the features are expressed higher in the titrating component , the life technologies method appears to perform slightly better for features expressed higher in the component being held constant; however, neither method performs particularly well for such features.
fig.
<dig> titration response.
to examine the titration response, we plot the proportion of features that show monotone increasing expression as the amount of input rna increases stratified by the difference in expression between the sample being titrated and the sample being held constant.
here we use samples 2– <dig> and 6– <dig> as two separate titration series.
to assess the difference in expression between mixture components a and b, we use the expression estimates in the pure sample types: sample type  <dig>  and sample type  <dig> .
each point represents an average over the mirnas falling within a given bin, where bins are defined based on the observed difference in pure sample expression .
the the life technologies  values are plotted with solid circles; the qpcr values with open squares
this table displays the number of features that show a titration response  for each method, life technologies  and qpcr, and each titration series .
here a and b refer to the mixture component being titrated: placenta & kidney  or skeletal muscle & brain
accuracy
related to the titration response is the accuracy of the expression estimates.
rather than simply requiring monotone increasing expression estimates in response to increasing input rna, here we are interested in the actual magnitude of the increase in expression.
specifically, to assess accuracy, we calculate the signal detect slope, defined as the slope of the regression line of observed expression on expected expression.
we stratify the signal detect slopes into three equally sized bins to highlight the dependence on the underlying difference in expression noted in the previous subsection.
similar to the results seen in fig.
<dig>  accuracy is better for features with higher relative expression in the titrating component .
for each of the bins shown in fig.
<dig>  we can summarize the accuracy by computing robust measures of center and spread – median and median absolute deviation  – of the signal detect slopes .
the qpcr method appears to have slightly better accuracy, although this is not significant.
fig.
<dig> accuracy assessment.
to assess accuracy, we calculate the signal detect slope: the slope of the regression line of observed expression on expected expression for both the life technologies  and qpcr algorithms.
the ideal signal detect slope is one, representing agreement between observed and expected expression.
the signal detect slopes are stratified by pure sample expression.
each point represents an mirna.
points in the figures below are grey if the signal detect slope is not statistically signficantly different from zero.
as such, a grey point corresponding to a signal detect slope well above zero represents a particularly noisy  response
precision
finally, we consider a measure of precision, the within-replicate coefficient of variation.
here, replicates are defined as each unique feature/sample type combination consisting of four data points each.
the data are divided into three bins of equal size based on the average expression of each replicate .
both methods appear to have comparable precision.
fig.
<dig> precision assessment.
to assess precision, we calculate the within-replicate coefficient of variation, calculated as the within-replicate standard deviation divided by the within-replicate mean, for both the life technologies  and qpcr algorithms.
these are calculated for each set of replicates  that are of good quality and stratified by the observed expression
performance of other algorithms
to this point, we have focused on assessing two algorithms to demonstrate the functionality of the mircomp r/bioconductor package.
however, we also applied four additional algorithms  to this benchmark data set.
the resulting expression estimates and quality scores are available in the mircomp r package.
performance assessments comparing all six algorithms are shown in additional file 7: figures s2–s <dig>
the relationship between expression estimates and quality scores was fairly similar across methods with the  <dig> parameter log sigmodial method producing slightly better quality scores and the linear exponential method producing overall lower quality scores .
the limit of detection was roughly  <dig> for all six algorithms, although the limit of detection appears slightly lower for the life technologies and linear exponential methods .
the life technologies and  <dig> parameter log sigmodial methods performed somewhat better in terms of titration response, while the linear exponential method performed significantly worse .
the linear exponential and  <dig> parameter log sigmoid methods had the best accuracy on average, but they also had the most variation in accuracy .
all methods performed better when the target transcript was highly expressed.
the life technologies method had consistently the best precision and the  <dig> parameter log sigmoidal method had consistently the worst precision .
discussion
in the previous section, we focused on assessing two algorithms to demonstrate the functionality of the mircomp r/bioconductor package.
additionally, we evaluated the performance of four other algorithms.
in summary, the 4-parameter and 5-parameter sigmodial and log-sigmodial algorithms performed similarly, with the 5-parameter log-sigmoidal achieving slightly better accuracy by sacrificing some precision.
in contrast the life technologies method appears to have sacrificed a bit of accuracy to achieve better precision.
the linear exponential algorithm performed significantly worse than the other five methods.
however, the primary purpose of this work is to facilitate the development of new expression estimation algorithms by allowing researchers to assess their own method.
the benchmark data set, described in the methods section and provided in the mircompdata r/bioconductor package, provides a rich resource for novel methods development.
one can use the assessment functions in the mircomp package to examine the effect of higher or lower quality thresholds for a currently available algorithm.
we anticipate that these packages will facilitate the comparison of competing algorithms and guide the selection of those most suitable for a specific experiment.
furthermore, we anticipate that the development of open-source software to estimate expression from the raw amplification data will lead to increased integration between expression estimation procedures and subsequent statistical analyses, often performed with r/bioconductor software packages.
software to estimate expression from amplification data are implemented across a wide variety of operating systems  and programming languages   <cit> .
as there is currently no standard data structure for raw qpcr amplification data, we store the raw data in a simple table that can be easily exported for the mircompdata package and converted into formats required by other expression estimation software.
while the assessments described in this manuscript are implemented in r, the assessment functions accept simple matrices as input, which should be easy to create from the output of any other software package.
the assessment strategy proposed in this manuscript is based on a mixture/dilution experimental design.
an alternative strategy would have been to spike-in known amounts of several mirnas.
assessing accuracy for genomic technologies presents a challenge because the correct outcome for a given measurement must be known a priori.
spike-in experiments represent a natural way to accomplish this by comparing the nominal spike-in concentrations with the observed expression estimates.
this is crucial to assess the accuracy of absolute expression estimates.
however, performance assessments based on spike-in data depend strongly on the chosen spike-in concentrations  <cit> .
one must select spike-ins that span the entire dynamic range of observed expression, are unexpressed in all biological samples being analyzed, and are measured by the technology under consideration.
in a mixture/dilution experiment, the focus is on relative rather than absolute expression.
while the absolute expression of a given feature is unknown, the relative expression is known.
therefore, in the assessments proposed in this manuscript, we have focused on the ability of various methods to accurately measure relative expression.
additionally, mixture/dilution experiments are inherently more realistic with respect to the dynamic range of observed expression levels by using mixtures of two biologically distinct rna pools at varying proportions.
such a mixture/dilution approach allows one to characterize performance of assays across the full spectrum of expression, which is necessary to assess the performance of any analytic strategy.
while one is restricted to assessments of relative expression, this corresponds to the standard use of qpcr expression assays – to measure differential rather than absolute expression.
CONCLUSIONS
in this manuscript, we present a benchmark data set and software to assess the performance of methods that estimate expression from raw data on the life technologies openarrayⓇ microrna platform.
the raw data and software packages, mircomp and mircompdata, are open-source and freely available as part of the bioconductor project.
we believe that the data and assessment software will facilitate the develop of novel methods for quantification of microrna expression from the life technologies openarrayⓇ platform.
using these packages, we assessed the performance of six expression estimation algorithms.
the 5-parameter log-sigmodial algorithm had the best accuracy; the life technologies method had the best titration response and precision.
availability of data and materials
r scripts to reproduce all figures and supplementary figures are included as additional files  <dig> and  <dig>  the datasets supporting the conclusions of this article are available in the mircomp and mircompdata r/bioconductor packages, included within the article as additional files  <dig> and  <dig> and available online: http://bioconductor.org/packages/mircomp/http://bioconductor.org/packages/mircompdata/.
additional files
additional file  <dig> mircomp package vignette.
the mircomp vignette describes standard use of the package as well as additional options and functionality not presented in this manuscript.
additional file  <dig> r script to create figures.
the r script used to generate figures 2– <dig> presented in this manuscript.
additional file  <dig> r script to create supplementary figures.
the r script used to generate supplementary figures 2- <dig>
additional file  <dig> mircomp r package source.
the source code for the mircomp r package, also available at: http://bioconductor.org/packages/mircomp/.
additional file  <dig> mircompdata r package source.
the source code for the mircompdata r package, also available at: http://bioconductor.org/packages/mircompdata/.
additional file  <dig> taqman openarray microrna panels user guide.
the technical manual of the source code for the taqman openarray microrna panels, also available at: https://tools.thermofisher.com/content/sfs/manuals/cms_ <dig> pdf.
additional file  <dig> supplementary figures.
a pdf containing  <dig> supplementary figures referred to in this manuscript.
competing interests
the authors declare that they have no competing interests.
authors’ contributions
mnm, kww, asb, and mkh designed the study.
mam and rgi contributed to the data generation.
mnm, asb, ac, kww, and mkh contributed to the analysis of the data.
mnm developed the software packages.
mnm and mkh wrote the manuscript.
all authors read, gave feedback, and approved the manuscript.

 micrornas are a class of small regulatory rnas. they are essential regulators that act as translational repressors throughout many eukaryotic species. several thousand mirnas have been described in humans and other species, although in practicality only 350– are present at robust levels in mature cells and tissues. mirnas are known to alter their expression levels in disease, malignancy, and cell stress and exhibit tissue and cell-type specific patterns of expression. many expression platforms, originally designed to quantify mrna expression, have been adapted to globally assay mirna expression including hybridization arrays, quantitative pcr, and sequencing. however, each of these approaches must overcome several challenges specific to mirnas: short sequence length, low abundance of target molecules, and sequence homology between mirnas. comparative performance assessments are crucial to understanding the strengths and limitations of each approach to mirna quantification. a group of investigators recently systematically evaluated available mirna platforms across standardized control samples. this study, called mirqc, established metrics to assay reproducibility, sensitivity, accuracy, specificity and concordance across the different methods. although a single platform was not found to be uniformly superior, there was substantial variability in performance across assessments. for each of the platforms, performance depends on both the instrument and the algorithm used to convert raw measurements into expression estimates. for example, one platform assessed in the mirqc study was rna-seq performed on an illumina gaiix instrument. the sample prep used the truseq small rna prep kit and results were aligned to the hg reference sequence allowing one mismatch, without further delineation of the alignment method. a previous performance evaluation of mirna expression arrays noted a strong dependency between technology and signal processing methodology. more recently, we have demonstrated that different mirna rna-seq alignment algorithms produce different alignments, impacting the quality of the data. we surmise that many mirna expression platforms are not yet optimized to yield consistent and maximally accurate data. another platform evaluated in the mirqc study was the life technologies taqman openarrayⓇ system. this is a qpcr-based mirna array platform that currently has coverage for human mirnas across two sets of primer pools. while qpcr is considered the gold standard for low-throughput measurement of gene expression, microarray- and sequencing-based platforms are preferable for most high-throughput applications. given the relative small number of common mirnas, it is possible to use a qpcr-based platform to measure the expression of all abundant mirnas in many tissues and cells. the primary advantage of qpcr-based technologies is the ability to simultaneously amplify and quantify a target transcript over sequential pcr cycles. the greater the initial amount of the target transcript present in a sample, the more rapidly the target will reach a threshold at which it can be detected by flourescence. as such, the raw data produced by qpcr-based technologies are fluorescence signal intensities captured at the end of each amplification cycle. analysis of these data typically begins by assigning a threshold cycle number to each amplification. these threshold cycles can then be used to estimate target abundance, either relative or in reference to values for a standard curve. for example, life technologies provides the expressionsuite software package, which uses the shape of the amplification curve to estimate a relative threshold cycle and corresponding expression estimate. while a substantial number of software tools have been developed to estimate gene expression from raw amplification data, these focused on mrna rather than mirna targets. whether these methods perform similarly when estimating mirna expression is an area of ongoing research. the software presented in this manuscript provides tools to assess and compare the performance of methods to transform raw amplification data into expression estimates and determine optimal quality thresholds. while the mirqc study focused on comparing many different platforms, here we focus on a single platform but provide a much larger and more diverse data set for evaluation. we believe that the availability of these data and corresponding software will greatly accelerate the development of improved methodology for the openarrayⓇ mirna platform. furthermore, seamless integration with the r/bioconductor suite of analysis packages will enhance the value of openarrayⓇ mirna data. therefore, we developed mircomp, an r package to assess and compare microrna expression estimation methods using a benchmark data set. methods experimental design selection of tissues two separate rna pools were prepared by blending two tissues each: kidney and placenta and skeletal muscle and brain. these sources of rna were chosen based on our prior analysis of agilent v mirna array data that suggested this collection of tissues would capture a large number of micrornas, including several unique to each sample, such as mir-133a for skeletal muscle and the chromosome mirna cluster for placenta. the surgical pathology archives of the department of pathology at johns hopkins hospital were used to obtain formalin fixed paraffin-embedded tissues from four distinct tissue sources. all tissues were verified as normal by review of tissue histology on an adjacent hematoxylin and eosin stained slide. these anonymized human samples were used based on an exemption from the institutional review board of johns hopkins hospital. rna extraction we extracted rna from ffpe sections of kidney, placenta, skeletal muscle, and brain using the allprep dna/rna ffpe protocol. xylene was chosen for deparaffinization. extra xylene and ethanol washes were performed, and dnase digestion was done on-column. rna quality control concentration of eluted rna was assessed by nanodrop. due to the low quality of longer rna molecules extracted from ffpe tissues, including the ribosomal rnas, the presence of several ubiquitous and tissue-enriched small rnas or mirnas was confirmed by stem-loop reverse transcription quantitative pcr using ng rna per reaction. for example, mir- and mir-133a were enriched in skeletal muscle, mir-516b was enriched in placenta, and mir-200b was enriched in kidney. rna was stored at −80c. reverse transcription and pre-amplification the kidney/placenta and skeletal muscle/brain mixtures were made by combining equal masses of kidney and placenta or skeletal muscle and frontal cortex rna, respectively, and diluting to an equal concentration of ng/ul. ng of rna was used as the input for reverse transcription using the a and b primer pools, following the life technologies openarrayⓇ protocol modification for low-concentration and ffpe rna. separate reverse transcription and pre-amplification reactions were performed for the life technologies megaplex pools a and b primer pools, which reverse transcribe and pre-amplify specific micrornas. following pre-amplification, ul from the a and b reactions for both kp and mb were mixed with ul of x te. further dilutions and combinations of the kp and mb mixtures were then prepared. to keep the non-nucelic acid components equal after mixing kp and mb, we added a diluent c mix as needed. the diluent c included the same proportions of rt buffer and pre-amp mix components as in the life technologies protocol-specified dilution of nucleic acid-containing post-pre-amp mixture. the final concentrations were and and % for each sample. the sample numbers are used throughout the manuscript to refer to specific mixture/dilution sample types. fig. two rna pools were formed by blending two tissues each: kidney and placenta and skeletal muscle and brain. these pools were combined in different mixtures and dilutions to form the sample types shown here. gene expression in each sample type was assessed in quadruplicate using the life technologies openarrayⓇ platform modified to perform amplification cycles life technologies openarrayⓇ assay standard human taqmanⓇ openarrayⓇ human microrna panel, quantstudio tm 12k flex chips and other necessary reagents were provided by life technologies for this experiment. this panel contains human mirna sequences from mirbase v which have all been previously functionally validated with mirna artificial templates. for conversion of notation from mirbase v style to current mirna style, the webtool miriadne can be used. the specially prepared post-pre-amp dilution mixtures were added to the sample plates and then loaded onto the chips using the accufill robot following the standard protocols. a modified microrna.edt file, provided by life technologies, was used to extend the cycles from the standard to cycles. this was done to make sure all amplifications went to completion, as the authors noted that some microrna amplicons had not reached their maximal intensity at cycles, causing a slight left shift to lower crt values in prior experiments. the additional cycles do not increase the detection limit of the system. three samples on one chip were run using the standard microrna.edt file provided with the instrument, due to human error. this did not have a noticeable effect on the expression estimates from any of the algorithms. additional information on the taqmanⓇ openarrayⓇ microrna panels can be found in the technical manual. expression estimation algorithms there are a wide variety of algorithms available to estimate expression from qpcr amplification curves. to facilitate comparisons between these algorithms, we have applied many of these algorithms to our benchmark data set. the resulting expression estimates and quality scores are available as data objects in the mircomp package. specifically, we provide expression estimates from the following methods: lifetech expressionsuite parameter sigmoidal model parameter sigmoidal model parameter log sigmoidal model parameter log sigmoidal model linear exponential model additionally, the raw amplification data are available in the mircompdata package allowing researchers to easily generate expression estimates using other current or future algorithms. statistical assessments the primary goal of the mixture/dilution experiment described above is to provide a benchmark data set with which to assess the performance of methods that estimate mirna expression from qpcr amplification curves. specifically, we propose assessments of accuracy, precision, data quality, titration response, limit of detection, and number of complete features. each of these is described in detail below. to avoid any confusion due to naming conventions, we refer to the reported values as expression estimates or simply expression. quality scores when estimating expression from amplification data, it is crucial for methods to provide both an expression estimate and a corresponding quality score. these quality scores are often used to filter, flag, or down-weight poor quality expression estimates in subsequent analyses. the qualityassessment function in the mircomp package allows one to examine the relationship between quality scores and expression estimates, the distribution of quality scores across samples, and the relationship between quality scores from two different methods. expression comparison when comparing two methods, a natural starting point is to compare the expression estimates produced by each method. by examining the features and samples for which expression estimates differ substantially, one can better understand the strengths and limitations of each method. the expressioncomp function in the mircomp package allows one to examine the relationship between expression estimates produced by two different methods. feature/sample combinations for which the expression estimates differ by more than a given threshold are flagged for further investigation. complete features a measure of the amount of readily usable data produced by a method is the number of complete features. complete features are defined as detected and of good quality across all samples in a given experiment. the completefeatures function allows one to assess a single method or compare two methods. limit of detection the limit of detection is an estimate of the smallest signal that can be reliably measured. we propose assessing the limit of detection in two ways: examining the distribution of average observed expression stratified by the proportion of values within a set of replicates that are good quality, and comparing the average observed vs expected expression in the two low input sample types. the expected expression for both low input sample types can be calculated based on the pure sample types or, in the case of the / dilution, it can be calculated based on the expression in the / dilution. visual representations of these comparisons are produced by the limitofdetection function. the limitofdetection function also reports several potential limits of detection based on each of the following comparisons: average observed expression in the / dilution samples vs expected expression based on the pure samples. average observed expression in the / dilution samples vs expected expression based on the pure samples. average observed expression in the / dilution samples vs expected expression based on the / dilution samples. for each of these comparisons, we calculate the difference between the observed and expected expression estimates. to assess the limit of detection, we compute the expression threshold such that the median difference of all features exceeding that threshold is equal to a predetermined tolerance. the limitofdetection returns these potential limits of detection for each comparison and three tolerances. titration response the titration response is defined as the ability of a method to produce monotone increasing expression estimates in response to increasing amounts of input rna. we consider sample types 2– and 6– as two separate titration series. in each of these series, one mixture component is held constant at μl and the other is doubled twice from μl to μl to μl. because this response will depend heavily on the underlying expression of a given feature in each mixture component, the titration response is stratified by the difference in expression between the component being titrated and the component being held constant. for example, in the sample type 2– titration series, mixture component a is held constant and mixture component b is titrated. to assess the difference in expression between mixture components a and b, we use the expression estimates in the pure sample types: sample type and sample type. accuracy to assess accuracy, we calculate the signal detect slope, defined as the slope of the regression line of observed expression on expected expression, for the two titration series. the ideal signal detect slope is one, representing agreement between observed and expected expression. the signal detect slopes are stratified by pure sample expression. a signal detect slope captures the average relationship between observed and expected expression; however, some features may perform well on average but be highly variable. in the plots produced, features are displayed in grey if the signal detect slope is not statistically significantly different from zero. as such, a grey point corresponding to a signal detect slope well above zero represents a particularly noisy response. precision to assess precision, we calculate both the within-replicate standard deviation and coefficient of variation. both statistics are calculated for each set of replicates that are of acceptable quality. for both summaries, the values are stratified by the average observed expression. software software implementing the assessments described in this manuscript was written in the open-source statistical language r. the r software package, mircomp, and the r data package, mircompdata, are available as part of the bioconductor project, a collaborative effort to develop software for computational biology and bioinformatics. in addition to the primary functionality described above, the mircomp package contains many additional options for customizable use of these assessment functions. these are described in the mircomp package vignette.  in the following, we compare two methods to generate expression estimates and quality scores from raw mirna qpcr amplification data. the first method is an algorithm developed by life technologies and implemented in the expressionsuite software package. this software package produces estimates of expression and a measure of quality. the second method is a four-parameter log-sigmoid curve-fitting algorithm implemented as the default method in the qpcr r package and referred to in this manuscript as simply qpcr. this open-source r package produces expression estimates by fitting a four parameter log sigmoidal curve to the amplification data and computing the point at which the second derivative of this curve is maximized and a measure of quality. four additional algorithms were applied to the benchmark data set, and the resulting expression estimates and quality scores are available in the mircomp r package. for clarity of presentation in this manuscript, we will focus on comparing two widely-used algorithms, the default algorithms from life technologies and the qpcr r package, in the following results. quality assessment given the interdependence between the expression estimates and quality scores produced by a method, we begin by examining this relationship for each method. as one might expect, quality scores decrease as the expression estimates increase. another feature of note is that both methods occasionally fail to produce an expression estimate. however, while the qpcr method assigns all of these values fairly low quality, the life technologies method produces a substantial number of na expression estimates with high quality scores. fig. relationship between quality and expression. for both data sets, a measure of data quality, life technologies ampscore or qpcr r-squared, is plotted against the corresponding expression estimates. the qpcr r-squared values are plotted on the complementary log-log scale to improve visibility in the region of interest. each point represents a single expression measurement and corresponding quality measure for a unique mirna/sample combination. two-dimensional scatter-plot smoothing is used to avoid over-plotting and convey the distribution of points across the plotting region when comparing two methods, it is also interesting to examine the relationship between the quality scores produced by each method. examination of this figure highlights regions of consensus high quality and consensus low quality as well as regions of disagreement between the methods. while there are relatively few data points that are estimated with a high quality ampscore and low quality r there are a substantial number of high quality r and low quality ampscore data points. fig. comparison of quality scores. a direct comparison between the quality scores: life technologies ampscore and qpcr r-squared. the vertical dashed line represents the recommended ampscore threshold of. the horizontal dashed line represents a potential r-squared threshold chosen by examination of this figure. each point represents the quality values for a unique mirna/sample combination. two-dimensional scatter-plot smoothing is used to avoid over-plotting and convey the distribution of points across the plotting region taken together, figs. & suggest quality thresholds of ampscore= and r2=. while we will use these thresholds throughout the remainder of this manuscript, all functions in the mircomp package allow the user to set their own quality thresholds. furthermore, for many functions, one can compare results from a single method using two different quality thresholds to examine the effect of changing the quality threshold on each assessment. lastly, when comparing two methods, we typically restrict the assessment to data considered to be good quality by both methods. this provides the most direct comparison between the expression estimates produced by the two methods; however, for many of these assessments, the mircomp package allows one to perform these comparisons using each method’s own quality assessment independently. expression comparison when comparing two methods, it is also interesting to examine the relationship between the expression estimates produced by each method. overall, the expression estimates produced by the life technologies and qpcr methods are quite similar; however, one feature/sample shows a substantial difference between methods. one can investigate this difference further by examining the raw amplification data provided in the mircompdata package. for comparison, we selected a feature from the same sample for which the two methods were approximate agreement. the amplification curve for mir- shows the probable cause of the discrepency between expression estimates – the amplification curve is still increasing at the maximum observed cycle. in contrast, the amplification curve for mir-29a has clearly reached a plateau before the maximum cycle. this results in expression estimates that are fairly similar across methods. fig. comparison of expression estimates. a expression estimates from life technologies method and the default qpcr method are plotted against one another. the mirna/sample combination in which the expression estimates differ most is represented by a black dot. an mirna/sample combination in which the expression estimates roughly agree is represented by a red dot. b & c the amplification curve data for b mir- or c mir-29a in sample kw10: with expression estimates for each method denote by vertical lines. note that the amplification curve is still increasing at cycle for mir- whereas for mir-29a is has leveled off complete features given the difficulty in measuring the expression of many mirnas, we examine the number of complete features, partial features, and absent features. examining the number of features that fall into each of these categories for each method provides a straightforward assessment of the amount of easily usable data produced by each method. of the features in these data, there are complete features using the life technologies method and complete features using the qpcr method; however, this may simply be a result of the choice of quality threshold. of note, there are complete features in common between both methods and features that are considered absent by both methods. the latter may represent mirnas that are not expressed in any of the four tissues included in these data. complete features are ones that are detected and have good quality across all samples. partial features are those that are detected and good quality in at least one of the samples. absent features are those that are not detected or of poor quality in all samples. these values are used to compare the performance of the life technologies and qpcr algorithms limit of detection next, we examine the limit of detection for a given method. while this is related to the previous assessment of complete features, here the focus is on determining the minimum signal that can be reliably detected. the first assessment compares the difference between observed and expected expression vs the expected expression based on three different comparisons. one can visually assess the limit of detection by identifying the expected expression value at which the expression difference begins to diverge from zero and the data are of increasingly poor quality. a quantitative version of this approach is described in the methods section. potential limits of detection for each comparison and each tolerance are shown in table in general, the three comparisons produce similar limits of detection for a given tolerance. fig. limit of detection ma-plots. to assess the limit of detection for a given method, we plot the difference between the observed and expected average expression vs the expected expression for each feature in the two low input sample types. the expected expression is calculated using either the pure sample expression or the / dilution expression. each point represents an mirna in the sample type of interest. features for which all four data points are of good quality are shown in black; other features are shown in red with the proportion of poor quality values for each feature denoted by plotting symbol. one can visually assess the limit of detect by the expected expression value at which the difference between observed and expected expression begins to differ substantially from zero and the proportion of poor quality values increases here we report three potential limits of detection based on the three assessments. limits of detection are reported separately for the life technologies and qpcr algorithms. columns correspond to the different assessments, and rows correspond to the median difference between the observed and expected values. the values in the matrix are the expected expression values such that the median absolute difference for all larger expected expression values is approximately equal to the threshold for that row. for example, consider the life technologies method, if we focus on the / vs / comparison and set a median average difference threshold of less than, our estimate of the limit of detection is approximately an alternative approach to assess the limit of detection is to examine the distribution of average observed expression among replicates stratified by the proportion of poor quality. this allows one to easily see that higher expression values result in increasing amounts of poor quality data; however, for both methods, there is substantial overlap between the distributions. fig. limit of detection boxplots. to further examine the limit of detection, we examine boxplots of average observed expression stratified by the proportion of poor quality data points. below each box, we also report the number of unique feature/sample type combinations each box contains. results from the life technologies method are shown in panel a; results from the qpcr method are shown in panel b titration response we now turn to assessments of the expression estimates themselves. for both methods, we calculated the proportion of features that show a titration response for each titration series. we also stratify these results by the difference in pure sample expression between the mixture component being titrated and the component being held constant. as one might expect, if an mirna is far less abundant in the component being titrated than in the component being held constant, it is difficult to detect a titration response for this mirna. if an mirna is far more abundant in the component being titrated than in the component being held constant, it is relatively easy to detect a titration response for this mirna. while both methods perform fairly well when the features are expressed higher in the titrating component, the life technologies method appears to perform slightly better for features expressed higher in the component being held constant; however, neither method performs particularly well for such features. fig. titration response. to examine the titration response, we plot the proportion of features that show monotone increasing expression as the amount of input rna increases stratified by the difference in expression between the sample being titrated and the sample being held constant. here we use samples 2– and 6– as two separate titration series. to assess the difference in expression between mixture components a and b, we use the expression estimates in the pure sample types: sample type and sample type. each point represents an average over the mirnas falling within a given bin, where bins are defined based on the observed difference in pure sample expression. the the life technologies values are plotted with solid circles; the qpcr values with open squares this table displays the number of features that show a titration response for each method, life technologies and qpcr, and each titration series. here a and b refer to the mixture component being titrated: placenta & kidney or skeletal muscle & brain accuracy related to the titration response is the accuracy of the expression estimates. rather than simply requiring monotone increasing expression estimates in response to increasing input rna, here we are interested in the actual magnitude of the increase in expression. specifically, to assess accuracy, we calculate the signal detect slope, defined as the slope of the regression line of observed expression on expected expression. we stratify the signal detect slopes into three equally sized bins to highlight the dependence on the underlying difference in expression noted in the previous subsection. similar to the results seen in fig. accuracy is better for features with higher relative expression in the titrating component. for each of the bins shown in fig. we can summarize the accuracy by computing robust measures of center and spread – median and median absolute deviation – of the signal detect slopes. the qpcr method appears to have slightly better accuracy, although this is not significant. fig. accuracy assessment. to assess accuracy, we calculate the signal detect slope: the slope of the regression line of observed expression on expected expression for both the life technologies and qpcr algorithms. the ideal signal detect slope is one, representing agreement between observed and expected expression. the signal detect slopes are stratified by pure sample expression. each point represents an mirna. points in the figures below are grey if the signal detect slope is not statistically signficantly different from zero. as such, a grey point corresponding to a signal detect slope well above zero represents a particularly noisy response precision finally, we consider a measure of precision, the within-replicate coefficient of variation. here, replicates are defined as each unique feature/sample type combination consisting of four data points each. the data are divided into three bins of equal size based on the average expression of each replicate. both methods appear to have comparable precision. fig. precision assessment. to assess precision, we calculate the within-replicate coefficient of variation, calculated as the within-replicate standard deviation divided by the within-replicate mean, for both the life technologies and qpcr algorithms. these are calculated for each set of replicates that are of good quality and stratified by the observed expression performance of other algorithms to this point, we have focused on assessing two algorithms to demonstrate the functionality of the mircomp r/bioconductor package. however, we also applied four additional algorithms to this benchmark data set. the resulting expression estimates and quality scores are available in the mircomp r package. performance assessments comparing all six algorithms are shown in additional file 7: figures s2–s the relationship between expression estimates and quality scores was fairly similar across methods with the parameter log sigmodial method producing slightly better quality scores and the linear exponential method producing overall lower quality scores. the limit of detection was roughly for all six algorithms, although the limit of detection appears slightly lower for the life technologies and linear exponential methods. the life technologies and parameter log sigmodial methods performed somewhat better in terms of titration response, while the linear exponential method performed significantly worse. the linear exponential and parameter log sigmoid methods had the best accuracy on average, but they also had the most variation in accuracy. all methods performed better when the target transcript was highly expressed. the life technologies method had consistently the best precision and the parameter log sigmoidal method had consistently the worst precision. discussion in the previous section, we focused on assessing two algorithms to demonstrate the functionality of the mircomp r/bioconductor package. additionally, we evaluated the performance of four other algorithms. in summary, the 4-parameter and 5-parameter sigmodial and log-sigmodial algorithms performed similarly, with the 5-parameter log-sigmoidal achieving slightly better accuracy by sacrificing some precision. in contrast the life technologies method appears to have sacrificed a bit of accuracy to achieve better precision. the linear exponential algorithm performed significantly worse than the other five methods. however, the primary purpose of this work is to facilitate the development of new expression estimation algorithms by allowing researchers to assess their own method. the benchmark data set, described in the methods section and provided in the mircompdata r/bioconductor package, provides a rich resource for novel methods development. one can use the assessment functions in the mircomp package to examine the effect of higher or lower quality thresholds for a currently available algorithm. we anticipate that these packages will facilitate the comparison of competing algorithms and guide the selection of those most suitable for a specific experiment. furthermore, we anticipate that the development of open-source software to estimate expression from the raw amplification data will lead to increased integration between expression estimation procedures and subsequent statistical analyses, often performed with r/bioconductor software packages. software to estimate expression from amplification data are implemented across a wide variety of operating systems and programming languages. as there is currently no standard data structure for raw qpcr amplification data, we store the raw data in a simple table that can be easily exported for the mircompdata package and converted into formats required by other expression estimation software. while the assessments described in this manuscript are implemented in r, the assessment functions accept simple matrices as input, which should be easy to create from the output of any other software package. the assessment strategy proposed in this manuscript is based on a mixture/dilution experimental design. an alternative strategy would have been to spike-in known amounts of several mirnas. assessing accuracy for genomic technologies presents a challenge because the correct outcome for a given measurement must be known a priori. spike-in experiments represent a natural way to accomplish this by comparing the nominal spike-in concentrations with the observed expression estimates. this is crucial to assess the accuracy of absolute expression estimates. however, performance assessments based on spike-in data depend strongly on the chosen spike-in concentrations. one must select spike-ins that span the entire dynamic range of observed expression, are unexpressed in all biological samples being analyzed, and are measured by the technology under consideration. in a mixture/dilution experiment, the focus is on relative rather than absolute expression. while the absolute expression of a given feature is unknown, the relative expression is known. therefore, in the assessments proposed in this manuscript, we have focused on the ability of various methods to accurately measure relative expression. additionally, mixture/dilution experiments are inherently more realistic with respect to the dynamic range of observed expression levels by using mixtures of two biologically distinct rna pools at varying proportions. such a mixture/dilution approach allows one to characterize performance of assays across the full spectrum of expression, which is necessary to assess the performance of any analytic strategy. while one is restricted to assessments of relative expression, this corresponds to the standard use of qpcr expression assays – to measure differential rather than absolute expression.  in this manuscript, we present a benchmark data set and software to assess the performance of methods that estimate expression from raw data on the life technologies openarrayⓇ microrna platform. the raw data and software packages, mircomp and mircompdata, are open-source and freely available as part of the bioconductor project. we believe that the data and assessment software will facilitate the develop of novel methods for quantification of microrna expression from the life technologies openarrayⓇ platform. using these packages, we assessed the performance of six expression estimation algorithms. the 5-parameter log-sigmodial algorithm had the best accuracy; the life technologies method had the best titration response and precision. availability of data and materials r scripts to reproduce all figures and supplementary figures are included as additional files and the datasets supporting the conclusions of this article are available in the mircomp and mircompdata r/bioconductor packages, included within the article as additional files and and available online: http://bioconductor.org/packages/mircomp/http://bioconductor.org/packages/mircompdata/. additional files additional file mircomp package vignette. the mircomp vignette describes standard use of the package as well as additional options and functionality not presented in this manuscript. additional file r script to create figures. the r script used to generate figures 2– presented in this manuscript. additional file r script to create supplementary figures. the r script used to generate supplementary figures 2- additional file mircomp r package source. the source code for the mircomp r package, also available at: http://bioconductor.org/packages/mircomp/. additional file mircompdata r package source. the source code for the mircompdata r package, also available at: http://bioconductor.org/packages/mircompdata/. additional file taqman openarray microrna panels user guide. the technical manual of the source code for the taqman openarray microrna panels, also available at: https://tools.thermofisher.com/content/sfs/manuals/cms_ pdf. additional file supplementary figures. a pdf containing supplementary figures referred to in this manuscript. competing interests the authors declare that they have no competing interests. authors’ contributions mnm, kww, asb, and mkh designed the study. mam and rgi contributed to the data generation. mnm, asb, ac, kww, and mkh contributed to the analysis of the data. mnm developed the software packages. mnm and mkh wrote the manuscript. all authors read, gave feedback, and approved the manuscript. 
to assess and compare the performance of these methods, we performed a set of dilution/mixture experiments to create a benchmark data set. the benchmark data and software are freely available via two r/bioconductor packages, mircomp and mircompdata. benchmark data sets and software are crucial tools for the assessment and comparison of competing algorithms. several algorithms currently exist to estimate expression from the raw amplification data produced by qpcr-based technologies. several techniques have been tailored to the quantification of microrna expression, including hybridization arrays, quantitative pcr, and high-throughput sequencing. we also developed a suite of statistical assessments that evaluate many different aspects of performance: accuracy, precision, titration response, number of complete features, limit of detection, and data quality. each of these has certain strengths and limitations depending both on the technology itself and the algorithm used to convert raw data into expression estimates. while substantial research has been devoted to the development of methods to quantify mrna expression, relatively little effort has been spent on microrna expression. reliable quantification of microrna expression is challenging in part due to the relatively low abundance and short length of the mirnas. in this work, we focus on the life technologies taqman openarrayⓇ system, a qpcr-based platform to measure microrna expression. we believe that the mircomp and mircompdata packages will facilitate the development of new methodology for microrna expression estimation.
 several techniques have been tailored to the quantification of microrna expression, including hybridization arrays, quantitative pcr, and high-throughput sequencing. each of these has certain strengths and limitations depending both on the technology itself and the algorithm used to convert raw data into expression estimates. reliable quantification of microrna expression is challenging in part due to the relatively low abundance and short length of the mirnas. while substantial research has been devoted to the development of methods to quantify mrna expression, relatively little effort has been spent on microrna expression.  in this work, we focus on the life technologies taqman openarrayⓇ system, a qpcr-based platform to measure microrna expression. several algorithms currently exist to estimate expression from the raw amplification data produced by qpcr-based technologies. to assess and compare the performance of these methods, we performed a set of dilution/mixture experiments to create a benchmark data set. we also developed a suite of statistical assessments that evaluate many different aspects of performance: accuracy, precision, titration response, number of complete features, limit of detection, and data quality. the benchmark data and software are freely available via two r/bioconductor packages, mircomp and mircompdata. finally, we demonstrate use of our software by comparing two widely used algorithms and providing assessments for four other algorithms.  benchmark data sets and software are crucial tools for the assessment and comparison of competing algorithms. we believe that the mircomp and mircompdata packages will facilitate the development of new methodology for microrna expression estimation. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords micrornaqpcrexpressionhttp://dx.doi.org/ /100000051national human genome research institute hg006853mccall matthew n. http://dx.doi.org/ /100000968american heart association 13grnt16420015halushka marc k. issue-copyright-statement© the author 2016 