BACKGROUND
over the last decade, gene expression microarrays have become a common tool for examining gene transcript levels in hypothesis-free investigations.
microarray data is used for a wide variety of analyses, such as unsupervised clustering, classification, differential expression analysis, and expression quantitative trait loci mapping .
these studies aim to differentiate subtle changes in relevant features from other biological and technical variation.
sample preparation for gene expression microarray requires one or more transcriptional steps, followed by labelling, hybridization, and intensity measurement  <cit> .
at each step, technical variations and accompanying biases are introduced.
microarray data preprocessing and batch correction are important strategies for minimizing such confounding.
preprocessing consists of three steps: background correction, normalization, and summarization; and the choice of methods for these steps can dramatically change experiment results  <cit> .
preprocessing algorithms must contend with differing probe hybridization efficiencies that result in greater inter-probe than inter-sample variability, probe intensity variances that change with intensity levels, and inter-sample technical error.
in comparative studies with spike-in and dilution datasets , quantile normalization has performed consistently well, and thus robust multi-array average , using a global background correction, quantile normalization, and a linear model fit by median polish for probe summarization  <cit> , has become a popular method for single-color microarray preprocessing.
even after normalization, variation in processing technician, location and time can result in probe-specific batch effects .
many algorithms attempt to normalize between batches, some performing location-scale adjustments based on known batch delineations, and others using global matrix factorization under the assumption that technical effects will outweigh biological effects .
comparisons of batch effect removal methods  <cit>  have found the location-scale adjustment empirical bayesian method combat  <cit>  to be robust, particularly when the study does not include multiple platforms or tissue-types, and when cross-batch reference samples are unavailable  <cit> .
combat borrows information across genes with similar within-batch expression profiles to correct batch effects while preserving biological covariates, allowing good performance even with small sample sizes  <cit> .
these data processing methods enable comparisons of individual genes between samples, but they leave direct interpretation of the normalized values somewhat ambiguous.
this is particularly problematic for genes with low log-transformed expression values, where it can be difficult to differentiate true expression from background noise.
after quantile normalization there is no direct calculation of the real expression difference indicated by a log-fold-change in microarray expression values.
common validation technologies, such as quantitative polymerase chain reaction  and multiplex branched dna assay  <cit> , also introduce noise through transcript or signal amplification and analog detection: qpcr measures real-time changes in the level of targeted transcripts during amplification through fluorescence changes and extrapolates their relative concentrations, while branched dna assays employ a forked hybridization detection scheme, amplifying reporter fluorescence of hybridized probes for signal detection.
in addition, although many biological studies use qpcr for validation of significant findings , they examine a limited number of genes selected for specific expression characteristics and thereby fail to provide a global representation from which microarray data properties might be discerned.
previous studies with spike-in and dilution datasets  <cit>  have demonstrated compression of microarray values at high and low levels of expression.
however, a number of studies  have raised concerns that spike-in and dilution datasets create highly-controlled background noise and are therefore unlikely to reflect the differing levels of cross-hybridizing molecules within real biological samples  <cit> .
in addition, these constructed “truth” samples often contain large target gene variances that may not represent the subtle changes examined in certain experimental settings.
several studies have used samples from biological tissue to compare multiple microarray platforms with other measurement technologies, either mixing rna from two tissues at known ratios  <cit>  or harvesting rna in stimulated and unstimulated conditions  <cit> .
however, these datasets have dramatic target gene variances, the studies compare microarray measurements with those from pcr- and branched-dna-based technologies that may introduce bias in amplification or detection steps, and their analyses mainly address differential expression discovery rather than global properties of the microarray.
to our knowledge, no study has made a systematic examination of microarray signal detection accuracy and precision with diverse biological samples in reference to an amplification-free, digital rna measurement.
here we have used a count-based transcript detection technology to address some of the questions raised above, comparing measurements from the affymetrix human gene  <dig>  st microarray with those from the nanostring technologies ncounter analysis system  <cit> .
in contrast to qpcr and branched dna assays, the ncounter analysis system directly enumerates specific rna molecules by dual probe hybridization, requiring amplification of neither rna nor signal.
to avoid hybridization and fluorescence scale biases, ncounter reactions are carried out with a great excess of transcript-specific probes before purification, and measurements are made on a digital instead of analog scale.
additionally, because it does not require transcription, the ncounter system protocol never heats samples sufficiently to denature genomic dna, avoiding contamination from genomic dna hybridization to oligonucleotide probes .
finally, ncounter data has recently been shown to be highly robust to different normalization methods  <cit> , providing reassurance that our gold-standard does not suffer computational processing biases.
thus, we interpret ratios of test and control gene counts from the ncounter as true measures of the relative expression of these genes in our samples.
in this study, we compared ncounter with microarray expression measurements of an experimental dataset composed of leukocyte subset rna from healthy controls and patients with either anti-neutrophil-cytoplasmic-antibody-associated vasculitis  or inflammatory bowel disease  to exemplify the level of biological variation likely to be encountered in microarray experiments.
our analyses provide a better understanding of how preprocessed microarray results reflect rna levels in diverse biological samples, translating microarray expression value differences into molecular changes between samples, and highlighting tissue-specific noise properties.
methods
rna samples
this study was approved by the cambridgeshire  <dig> research ethics committee  and all individuals provided written informed consent.
samples were from healthy controls and individuals with active, untreated disease with the following diagnoses: ibd, specifically crohn’s disease  and ulcerative colitis ; and aav, specifically granulomatosis with polyangiitis  and microscopic polyangiitis .
whole blood was collected and separated into peripheral blood leukocyte subsets as previously described .
briefly, whole blood was passed over a histopaque  <dig>  gradient.
red blood cells from the granulocyte-red-blood-cell pellet were lysed in a buffer of 155 mm ammonium chloride, 12 mm sodium bicarbonate, and 100 mm edta, and cd16+ neutrophils were separated by magnetic bead-based positive selection .
peripheral blood mononuclear cells were split into two fractions for positive selection  of cd14+ monocytes and cd19+ b cells.
negative fractions from the cd14+ and cd19+ selections were then used in a second round of positive selection for cd4+ t cells and cd8+ t cells , respectively.
rna was extracted with the rneasy mini or allprep dna/rna mini kit , following the manufacturer’s protocols.
this study uses the cd16+ neutrophil, cd14+ monocyte and cd4+ t cell subsets from this dataset, referred to as the cd <dig>  cd <dig> and cd <dig> datasets, respectively.
microarray
aliquots of 200 ng total rna were amplified and labelled for human gene  <dig>  st 96-array  using the ambion wt expression kit and genechip wt terminal labeling and controls kit , according to the manufacturer’s protocols.
samples were run on a genetitan multi-channel  instrument  as part of a larger dataset acquired over multiple years.
for comparison with ncounter data, we selected only batches with at least  <dig> samples from the desired cohorts covering at least two different diagnoses to ensure batch correction was resistant to outliers and confounding structure.
microarray data for samples used in the ncounter comparison have been deposited in arrayexpress with accession number e-mtab- <dig>
microarray data processing
gene expression microarrays were filtered for sex discordance and global dimness before data processing.
because the robustness of microarray normalization improves with the number of samples included, arrays were normalized in large, cell-type-specific batches, including all available samples from the selected batches with diagnoses tested in this study .
samples were preprocessed with rma using the oligo bioconductor package  <cit>  with pd.hugene.
<dig> .st.v <dig>  <cit>  annotation.
batch correction was performed using the combat function of the sva bioconductor package  <cit>  specifying diagnosis, sex, and age as covariates to avoid removal of biological differences.
quality control was performed with the arrayqualitymetrics bioconductor package  <cit> .
ncounter control gene choice
control genes for the ncounter analysis system were chosen for each cell type on the basis of consistent expression across samples in the large, cell-type-specific microarray datasets described above.
rma-preprocessed datasets were separated by batch, and the variance of each gene calculated across all samples.
gene variances were then averaged across batches.
examining average variance versus mean expression revealed low variance at very high expression levels, suggesting array saturation or preprocessing compression effects .
in order to select control genes that were well-expressed but not completely saturated, we filtered for genes with mean microarray expression values between  <dig> and  <dig> .
these genes were then sorted by average within-batch variance, and control genes were chosen from the 2% with the lowest variance based on functional annotation suggesting stable, high expression in leukocytes .
the use of two control genes per dataset was modelled after reis et al.
<cit> .
ncounter
rna was prepared for and run on the ncounter analysis system , according to the manufacturer’s protocol in a total of  <dig> runs over 8 days.
to test for rna degradation, several sentinel samples from each cell-type-specific dataset were examined by  <dig> bioanalyzer  according to the manufacturer’s protocol.
all sentinel samples were confirmed to be of good quality with rin ≥  <dig> .
one sample was duplicated in a separate ncounter run to evaluate reproducibility and run-specific effects .
rna was loaded at 100 ng per sample with the exception of one sample with low rna yield; this was run at 59 ng and did not result in a low-count quality control flag.
all hybridizations were 17 hours long, and all counts were gathered by scanning on high mode for  <dig> fields of view per sample.
ncounter data processing
no ncounter samples were flagged by nsolveranalysissoftware  for quality control.
ncounter data was normalized for hybridization and counting efficiency in cell-type specific groups.
each sample was multiplicatively normalized to have the same geometric mean of ncounter-provided positive control probe counts.
no normalization factors were outside the nanostring-recommended range of  <dig> - <dig>  thresholds to identify expressed genes from background noise were then calculated as the median of the maxima of the negative control probe measurements for each cell type.
selection of microarray probesets for comparison between platforms
microarray probesets mapping to genes measured by the ncounter analysis system were identified by affymetrix genechip human gene st arrays probeset annotations release  <dig> .
where multiple probesets were available for a given gene, the probeset with the best target region overlap with the ncounter target region was chosen.
where all probesets were equivalent in overlap, one was chosen at random .
correlation between platforms
ncounter data was log-transformed and normalized to either a single, or to the mean of two, log-transformed control gene measurements.
where indicated, microarray expression values were similarly normalized to single or multiple control gene expression values.
the two platforms were compared by pearson correlation of each gene across samples.
log-ratio accuracy and precision analysis
signal detection slopes were calculated by taking the slope of a linear model fit to log-transformed microarray expression values versus log-transformed, control-gene-normalized ncounter measurements.
although spline-fitted curves are often used for similar spike-in experiments to allow for expression-level dependencies, expression of each gene is generally contained within a smaller range than spike-in controls, and therefore we found it appropriate to use linear regression for each individual gene.
for this analysis, each ncounter dataset was normalized to two cell-type-specific control genes, while microarray datasets were not normalized to control genes.
noise in unexpressed microarray probesets was examined by measuring the standard deviation of all possible microarray log-ratios between pairs of samples for each gene in one of two sets: 1) all genes called unexpressed by the ncounter; and 2) all genes called invariant across samples by the ncounter  but strictly unsaturated on the microarray .
RESULTS
microarray expression value level and variance indicate transcript presence and correlation with ncounter measurements
we compared a previously acquired single-color microarray dataset of  <dig> samples from  <dig> batches, containing multiple cell types and diagnoses , with ncounter data from  <dig> of these same rna samples, acquired in  <dig> ncounter analysis system runs over 8 days .
an additional  <dig> rna samples were measured by ncounter for inter-run comparisons.
the ncounter panel was designed to detect transcripts of  <dig> genes, including cell-type-specific control genes, spanning a wide range of expression levels and variances .
two ncounter probesets were excluded: one for poor predicted hybridization specificity, and one based on hybridization failure.
ncounter data was processed as described in methods, using spiked positive and negative control probes to correct for hybridization efficiency and determine count thresholds for unexpressed genes, respectively.
technical replicates of the same sample on different ncounter runs were highly correlated , and variation between samples of different diagnoses was greater than that between samples of the same diagnosis in different ncounter runs , implying very little inter-run variability.
reduced inter-run correlations in cd14+ monocyte  samples were due to one outlier , which was not used in subsequent ncounter versus microarray comparisons.
based on this analysis, we considered ncounter run-effects negligible and processed all samples for each cell type together, disregarding run membership.
as the ncounter analysis system experiment measured a small number of genes, it was not possible to normalize across samples by fitting expression measures to a common distribution  <cit> .
instead, ncounter transcript counts were normalized to selected control genes , as described in methods.
because cell type greatly influences expression patterns  <cit> , and because our microarray datasets consisted of cell-type-specific batches that confound effects from these two variables, each cell type was treated as a separate dataset for processing.
microarray datasets were preprocessed by rma and, where applicable, corrected for batch effects by combat, as described in methods.
the resulting log-transformed, standardized microarray values are referred to as “microarray expression values” throughout this manuscript.
with ncounter thresholds determined from negative control probe counts, we identified expressed and unexpressed genes in each cell type.
unexpressed genes were characterized by both low expression and low variance on the microarray .
it is important to note that many genes identified as expressed by the ncounter analysis system had microarray expression values lower than those identified as unexpressed, confirming on a new platform the observation made by irizarry and colleagues  <cit>  that probe-effects prevent strict, experiment-wide thresholding of expressed genes.
introducing a second variance threshold might improve identification of unexpressed genes, but a larger study examining more genes would be required to develop such a rule.figure  <dig>
median and variance properties of expressed versus unexpressed genes and interplatform correlation.
a) genes are plotted by cell-type-specific rma + combat-  or rma-preprocessed  microarray expression value median and variance.
red indicates genes called unexpressed by ncounter measurement.
b) ncounter datasets were normalized to two cell-type-specific control genes each, log-transformed and compared to rma + combat-  or rma-preprocessed  microarray data without control-gene normalization.
plots show pearson correlation of each expressed gene versus the median or variance of its microarray expression values or control-gene-normalized, log-transformed ncounter measurements: blue = cd <dig>  green = cd <dig>  red = cd <dig> datasets.
c) rma + combat-  or rma-preprocessed  microarray data plotted versus ncounter measurements normalized to two cell-type-specific control genes.
we examined the correlation of each expressed gene across samples between the microarray and ncounter platforms, varying the choice of control genes for ncounter data, the use or absence of control gene normalization for preprocessed microarray data, and the use or absence of microarray batch correction .
choice of ncounter control genes changed correlation results slightly, with multiple control genes generally performing better than single control genes, as previously suggested for other technologies  <cit> .
the poorer performance of cd4+ t cell  samples normalized to pias <dig> suggests that this gene was not as invariant as indicated by its microarray expression values, perhaps due to microarray saturation.
control-gene-normalizing the microarray data generally improved correlation, but the moderate extent of this improvement indicates acceptable robustness of the control genes used.
cd <dig> and cd <dig> rna samples were run on multiple microarray batches, leaving microarray datasets potentially confounded by technical artefacts.
although principal component analyses of the entire datasets before batch correction demonstrated obvious batch effects , batch correction had very little effect on most genes examined in this study .
in fact, correlation between microarray and ncounter measurements remained globally unchanged with batch correction of the cd <dig> and cd <dig> microarray datasets.
direct comparison of batch-corrected and non-batch-corrected data  indicates that this correction dramatically improved correlation of several genes in the cd <dig> dataset while the median gene correlation decreased slightly, possibly due to a reduction in covariance that would be predicted if batch correction reduced microarray variance in comparison to noise levels.
these gene-specific effects highlight the importance of using batch-correction algorithms, such as combat, that can normalize at the gene level even with many covariate groups spread across batches.examination of the inter-platform correlation coefficient for each expressed gene versus either its median expression level or variance  demonstrated that high variance generally corresponded with good correlation.
indeed, pearson correlation relies on the covariance of two variables, inherently requiring variation of both, but it is interesting to note that many low-variance genes also exhibited good inter-platform correlation.
this empirically indicates high precision of microarray measurements, allowing successful identification of true transcript variation over platform-specific noise for many genes, even with low inter-sample variation.
in comparisons of inter-platform correlation with median expression, genes with very low and very high expression generally had poorer correlation.
directly plotting microarray expression values against their corresponding control-gene-normalized ncounter measurements revealed large variability at the low end of expression and flattening of microarray values at the high end .
taken together, these data suggest that low expression measurements on the microarray may have been obscured by background noise while high measurements were likely saturated.
microarray signal detection accuracy depends on expression level and is dataset-specific
one metric used for determining microarray measurement accuracy is the “signal detection slope”  <cit> , or the slope of linear regression relating measured microarray expression values to log-transformed, known input transcript concentrations.
a slope of one indicates that the microarray accurately reflects the input.
to examine microarray accuracy using real experimental data, we calculated the signal detection slope of microarray expression values versus log-transformed, control-gene-normalized ncounter measurements by fitting linear regressions to the paired platform measurements of each gene.
figure 2a shows signal detection slope plotted against correlation coefficient.
because both correlation coefficients and signal detection slopes were determined by comparison of the same measurements, genes with low correlation also showed low signal detection.
however, genes with high correlation did not necessarily have high signal detection, instead exhibiting a much wider range of signal detection slopes.
in order to better understand this spread, we filtered for genes with good inter-platform correlation  and plotted these signal detection slopes versus the median microarray expression value .
signal detection accuracy was globally reduced , with slopes particularly dampened in genes with high  and low  microarray expression values.
even within the same expression level ranges, signal detection varied by dataset, indicating variable effects of independent preprocessing and/or tissue type on accuracy reduction.
batch correction slightly reduced signal detection in general, again possibly due to covariance reduction through decreasing microarray variance .figure  <dig>
signal detection accuracy of microarray gene expression values.
a) signal detection slopes of rma + combat-  or rma-preprocessed  microarray expression values for all expressed genes are plotted against their inter-platform correlation coefficient.
b) signal detection slopes of rma + combat-  or rma-preprocessed  microarray expression values for all genes with correlated ncounter and microarray measures  are plotted against median microarray values.
noise in microarray expression values is dataset-specific
as noted above, inter-platform correlation of low-variance genes demonstrated wide variability within and across datasets , suggesting variable levels of noise in the microarray measurements.
to examine microarray expression value precision in each dataset, we adapted a metric developed by mccall et al.
<cit>  for spike-in data and examined the standard deviation of microarray log-ratios of unexpressed and invariant genes, as described in methods.
noise of unexpressed genes varied between datasets , indicating either a tissue-specific effect or an artefact of the preprocessing of each dataset.
batch correction slightly reduced this variation, suggesting improvements in precision.
comparison with microarray median expression values  demonstrated that noise amplitude of unexpressed genes remained largely independent of microarray expression values.
the cd <dig> dataset had significantly less noise than the cd <dig> and cd <dig> datasets , implying that comparisons of low variance gene expression values might be more reliable in this dataset.
indeed, this was observed in the generally improved inter-platform correlation coefficients of low-variance genes in the cd <dig> subset .
because noise in unexpressed genes only addresses precision at low microarray expression values and because our unexpressed gene sets were of different sizes, we also examined microarray precision in invariant genes over a wider range of expression.
to this end, we used ncounter measurements to select genes with low variance across all samples disregarding diagnosis  and strictly filtered for genes not saturated on the microarray.
examination of this noise metric across datasets  revealed the same trends for batch correction and cell-type-specificity.
comparing precision of invariant gene measurements versus median expression values on the microarray  indicated that genes with very high expression tended to better precision.
although we had filtered strictly to avoid saturated microarray expression values, this precision trend, as well as the binned variance properties depicted in additional file  <dig>  suggests that saturation or compression artefacts of rma preprocessing  <cit>  may begin to reduce inter-sample variation at even lower expression values.figure  <dig>
signal detection precision of microarray gene expression values.
a) boxplots depict the standard deviations of log-ratios of all pairs of samples for unexpressed genes.
stars indicate mann–whitney test p-value <  <dig> .
b) as a for genes invariant by ncounter but unsaturated on the microarray.
discussion
while we are not the first to ask questions about microarray interpretation of gene expression and log-ratios, our approach is unique in its use of a count-based technology to examine diverse biological samples.
this has enabled us to examine the properties of microarray data representative of real experimental questions and to present the new observation of tissue-specific noise.
through examination of inter-platform correlation and expressed versus unexpressed gene properties, we have demonstrated that variance measurement may aid in the determination of whether or not a gene is expressed, particularly at low microarray expression values.
irizarry and colleagues have clearly shown that a low microarray expression value does not necessarily predict an unexpressed gene  <cit> .
the small number of genes tested in the present study would suggest that simultaneous thresholds for microarray gene expression level and variance might perform better than expression level thresholds alone to identify present versus absent transcripts , but ncounter measurement of a larger gene set would be necessary to develop such a method.
indeed present/absent calls, expression thresholds and, most popularly, variance filters  <cit>  are frequently used in downstream microarray analyses to reduce multiple testing and improve power for discovery .
our demonstration that unexpressed genes generally have low microarray expression values and variances provides empirical justification particularly for using the intersection of such filters to remove uninformative probesets.
our comparison of log-fold-changes measured by microarray and ncounter platforms has revealed that signal detection accuracy of the microarray varies dramatically by expression level.
similar to previous studies with constructed datasets  <cit> , our data show global compression most extreme at very high and low gene expression values.
importantly, we noted that signal detection slope compression also varies by dataset, indicating that effect sizes measured by log-fold changes are not necessarily comparable between datasets preprocessed separately.
thus, comparing the magnitudes of expression changes in genes expressed at different levels or genes in different datasets requires knowledge of their individual signal detection slopes.
for precision estimation, we examined noise of genes defined as unexpressed or invariant by ncounter measurements.
the rma algorithm stabilizes variance with respect to expression level  <cit> , and thus noise patterns from probesets detecting unexpressed genes should reflect noise across the whole microarray.
accordingly, we found the same patterns of precision estimates in unexpressed genes as in invariant genes across a wider range of expression levels.
we noted that noise appeared strongly tissue-dependent, with less in the cd <dig> than the cd <dig> or cd <dig> datasets.
binned mean expression versus variance plots  support this pattern of tissue-specific precision, showing a high-variance peak at very low expression levels in the cd <dig> and cd <dig> datasets  that is nearly absent in the cd <dig> dataset.
because the same rna was used for both the ncounter and microarray measurements, dataset-dependent noise differences can only be due to properties intrinsic to the frozen rna samples.
transcriptomes differ by tissue , and thus one explanation for differing noise levels is that levels of cross-hybridizing nucleotides also vary by cell type.
we previously showed that rna transcript profiles of cells from the myeloid lineage  change dramatically if blood is left several hours before processing  <cit> .
although we found no evidence of large-scale loss of rna integrity, it is possible that even during rapid blood processing, rna from myeloid cells suffers slightly more degradation, a factor likely to confound microarray more than the ncounter measurements  <cit> .
another plausible explanation is cell-type-specific contamination with genomic dna, particularly in the cd16+ neutrophil subset, which has comparatively less rna per cell  <cit>  and thus a higher ratio of genomic dna to rna.
genomic dna would likely be problematic in the transcriptional step of microarray sample preparation  <cit>  but not the amplification-free ncounter procedure  <cit> .
regardless of the source of this dataset-specific noise, such effects are important to remember for cross-tissue studies, such as the gene expression barcode  <cit> , where probes reflecting signal in one tissue type may be conflated by noise in another.
our precision results are based on examination of a limited number of genes in three different leukocyte subsets, and future studies of more genes in additional tissue types will begin to shed light on the origin and extent of this dataset-specific noise.
CONCLUSIONS
this analysis of gene expression microarray measurements versus transcript count ratios highlights three aspects of microarray data directly relevant to users of the technology.
first, inter-sample variance may indicate transcript presence in genes with low microarray expression values.
second, signal detection accuracy depends strongly on expression level, even in datasets of diverse biological samples with variable background and small gene expression ranges.
third, precision is dataset-specific, and therefore power to detect subtle biological differences may differ between tissues even when measured on the same microarray platform.
without careful consideration of these biases and confirmatory measurements by a second technology, microarray platform discoveries may be missed or misinterpreted.
availability of supporting data
the data sets supporting the results of this article are available in the arrayexpress repository: https://www.ebi.ac.uk/arrayexpress/experiments/e-mtab-2452/.
electronic supplementary material
additional file 1::
preprocessing samples.
microarray samples used for preprocessing: breakdown of preprocessed microarray batches including biological covariates.
additional file 2::
mean v variance.
binned mean and variance characteristics of microarray datasets: for each gene, a microarray expression value mean and mean within-batch variance was calculated.
genes were then binned by expression value means, and statistics were averaged to achieve an average mean expression value and average mean within-batch-variance for each bin.
these two values are plotted.
additional file 3::
control genes.
microarray properties of ncounter control genes: details of each cell-type-specific control gene used.
additional file 4::
ncounter run comparison.
examination of inter-run technical effects of ncounter data: a) log-transformed raw ncounter counts for technical replicates of the same sample are plotted with pearson correlation indicated.
all genes determined to be globally expressed in cd <dig> samples are included.
b) boxplots depict pearson correlations between log-transformed raw ncounter counts for samples of the same and different diagnoses in the same and in different ncounter runs.
all genes determined to be globally expressed in the designated cell types are included.
in the bottom panel, the outlying cd <dig> sample has been removed.
c) inter-sample pearson correlation coefficients of log-transformed raw ncounter counts between all cd <dig> samples.
red star indicates outlier.
additional file 5::
ncounter probes.
ncounter probe details and mapped affymetrix hugene  <dig>  st array probesets: ncounter probe design schemes, isoform coverage, and microarray probeset mappings are tabulated.
additional file 6::
ncounter samples.
sample composition: details are provided for samples run on the ncounter analysis system.
additional file 7::
correlation comparison.
effects of microarray and ncounter processing on inter-platform correlation: cell-type-specific ncounter datasets were normalized to the indicated control genes and log-transformed.
microarray data were preprocessed by rma and then batch normalized through combat and/or normalized to control genes where indicated.
boxplots show pearson correlation.
additional file 8::
correlation comparison table.
effects of microarray and ncounter processing on inter-platform correlation: table summarizes inter-platform correlation of datasets using different processing and normalization procedures.
additional file 9::
microarray batch effects.
batch effects in microarray datasets: a) samples from full cd <dig> and cd <dig> microarray datasets are plotted by first and second principle components before and after combat batch correction.
color indicates batch membership.
b) pearson correlation of expressed genes across samples in ncounter versus rma-preprocessed microarray datasets was subtracted from the same correlation in ncounter versus rma-preprocessed and combat-corrected microarray datasets.
boxplots depict these differences in cd <dig> and cd <dig> datasets to indicate the effect of batch correction on gene-based platform correlation.
additional file 10::
batch correction and accuracy.
effect of batch correction on signal detection accuracy: a) signal detection slope is plotted versus inter-platform correlation as in figure 2a: blue = rma- and red = rma + combat-preprocessed microarray expression values.
b) signal detection slope of expressed genes across samples in ncounter versus rma-preprocessed microarray datasets was subtracted from the same signal detection slope in ncounter versus rma-preprocessed and combat-corrected microarray datasets.
boxplots depict these differences in cd <dig> and cd <dig> datasets to indicate the effect of batch correction on signal detection accuracy.
additional file 11::
noise v expression value.
comparison of noise versus microarray expression value.
a) for each unexpressed gene, the standard deviation of log-ratios of all pairs of samples from rma + combat-  or rma-preprocessed  microarray data is plotted versus the gene’s median microarray expression value.
b) as  for invariant genes.
additional file 12::
mean expression histograms.
mean expression profiles: histograms depict mean rma + combat-  or rma-preprocessed  microarray expression values from full microarray datasets.
abbreviations
rmarobust multi-array average
qpcrquantitative polymerase chain reaction
aavanti-neutrophil-cytoplasmic-antibody-associated vasculitis
ibdinflammatory bowel disease
cdcrohn’s disease
uculcerative colitis
gpagranulomatosis with polyangiitis
mpamicroscopic polyangiitis.
competing interests
the authors declare that they have no competing interests.
authors’ contributions
acr participated in the design of the study, collected the ncounter measurement data, carried out the comparative analysis and drafted the manuscript.
pal participated in the design of the study, advised the analysis and helped to draft the manuscript.
jep contributed to the analysis methods and execution and helped to draft the manuscript.
db and smf participated in the design of the ncounter assay, and smf helped to edit the manuscript.
jcl recruited patients and processed peripheral blood samples for the ibd cohort.
efm recruited patients and processed samples for the aav cohort.
rms advised and participated in the design of the ncounter assay and helped to draft the manuscript.
kgcs participated in the design of the study, advised the analysis and helped to draft the manuscript.
all authors read and approved the final manuscript.

 over the last decade, gene expression microarrays have become a common tool for examining gene transcript levels in hypothesis-free investigations. microarray data is used for a wide variety of analyses, such as unsupervised clustering, classification, differential expression analysis, and expression quantitative trait loci mapping. these studies aim to differentiate subtle changes in relevant features from other biological and technical variation. sample preparation for gene expression microarray requires one or more transcriptional steps, followed by labelling, hybridization, and intensity measurement. at each step, technical variations and accompanying biases are introduced. microarray data preprocessing and batch correction are important strategies for minimizing such confounding. preprocessing consists of three steps: background correction, normalization, and summarization; and the choice of methods for these steps can dramatically change experiment results. preprocessing algorithms must contend with differing probe hybridization efficiencies that result in greater inter-probe than inter-sample variability, probe intensity variances that change with intensity levels, and inter-sample technical error. in comparative studies with spike-in and dilution datasets, quantile normalization has performed consistently well, and thus robust multi-array average, using a global background correction, quantile normalization, and a linear model fit by median polish for probe summarization, has become a popular method for single-color microarray preprocessing. even after normalization, variation in processing technician, location and time can result in probe-specific batch effects. many algorithms attempt to normalize between batches, some performing location-scale adjustments based on known batch delineations, and others using global matrix factorization under the assumption that technical effects will outweigh biological effects. comparisons of batch effect removal methods have found the location-scale adjustment empirical bayesian method combat to be robust, particularly when the study does not include multiple platforms or tissue-types, and when cross-batch reference samples are unavailable. combat borrows information across genes with similar within-batch expression profiles to correct batch effects while preserving biological covariates, allowing good performance even with small sample sizes. these data processing methods enable comparisons of individual genes between samples, but they leave direct interpretation of the normalized values somewhat ambiguous. this is particularly problematic for genes with low log-transformed expression values, where it can be difficult to differentiate true expression from background noise. after quantile normalization there is no direct calculation of the real expression difference indicated by a log-fold-change in microarray expression values. common validation technologies, such as quantitative polymerase chain reaction and multiplex branched dna assay, also introduce noise through transcript or signal amplification and analog detection: qpcr measures real-time changes in the level of targeted transcripts during amplification through fluorescence changes and extrapolates their relative concentrations, while branched dna assays employ a forked hybridization detection scheme, amplifying reporter fluorescence of hybridized probes for signal detection. in addition, although many biological studies use qpcr for validation of significant findings, they examine a limited number of genes selected for specific expression characteristics and thereby fail to provide a global representation from which microarray data properties might be discerned. previous studies with spike-in and dilution datasets have demonstrated compression of microarray values at high and low levels of expression. however, a number of studies have raised concerns that spike-in and dilution datasets create highly-controlled background noise and are therefore unlikely to reflect the differing levels of cross-hybridizing molecules within real biological samples. in addition, these constructed “truth” samples often contain large target gene variances that may not represent the subtle changes examined in certain experimental settings. several studies have used samples from biological tissue to compare multiple microarray platforms with other measurement technologies, either mixing rna from two tissues at known ratios or harvesting rna in stimulated and unstimulated conditions. however, these datasets have dramatic target gene variances, the studies compare microarray measurements with those from pcr- and branched-dna-based technologies that may introduce bias in amplification or detection steps, and their analyses mainly address differential expression discovery rather than global properties of the microarray. to our knowledge, no study has made a systematic examination of microarray signal detection accuracy and precision with diverse biological samples in reference to an amplification-free, digital rna measurement. here we have used a count-based transcript detection technology to address some of the questions raised above, comparing measurements from the affymetrix human gene st microarray with those from the nanostring technologies ncounter analysis system. in contrast to qpcr and branched dna assays, the ncounter analysis system directly enumerates specific rna molecules by dual probe hybridization, requiring amplification of neither rna nor signal. to avoid hybridization and fluorescence scale biases, ncounter reactions are carried out with a great excess of transcript-specific probes before purification, and measurements are made on a digital instead of analog scale. additionally, because it does not require transcription, the ncounter system protocol never heats samples sufficiently to denature genomic dna, avoiding contamination from genomic dna hybridization to oligonucleotide probes. finally, ncounter data has recently been shown to be highly robust to different normalization methods, providing reassurance that our gold-standard does not suffer computational processing biases. thus, we interpret ratios of test and control gene counts from the ncounter as true measures of the relative expression of these genes in our samples. in this study, we compared ncounter with microarray expression measurements of an experimental dataset composed of leukocyte subset rna from healthy controls and patients with either anti-neutrophil-cytoplasmic-antibody-associated vasculitis or inflammatory bowel disease to exemplify the level of biological variation likely to be encountered in microarray experiments. our analyses provide a better understanding of how preprocessed microarray results reflect rna levels in diverse biological samples, translating microarray expression value differences into molecular changes between samples, and highlighting tissue-specific noise properties. methods rna samples this study was approved by the cambridgeshire research ethics committee and all individuals provided written informed consent. samples were from healthy controls and individuals with active, untreated disease with the following diagnoses: ibd, specifically crohn’s disease and ulcerative colitis; and aav, specifically granulomatosis with polyangiitis and microscopic polyangiitis. whole blood was collected and separated into peripheral blood leukocyte subsets as previously described. briefly, whole blood was passed over a histopaque gradient. red blood cells from the granulocyte-red-blood-cell pellet were lysed in a buffer of 155 mm ammonium chloride, 12 mm sodium bicarbonate, and 100 mm edta, and cd16+ neutrophils were separated by magnetic bead-based positive selection. peripheral blood mononuclear cells were split into two fractions for positive selection of cd14+ monocytes and cd19+ b cells. negative fractions from the cd14+ and cd19+ selections were then used in a second round of positive selection for cd4+ t cells and cd8+ t cells, respectively. rna was extracted with the rneasy mini or allprep dna/rna mini kit, following the manufacturer’s protocols. this study uses the cd16+ neutrophil, cd14+ monocyte and cd4+ t cell subsets from this dataset, referred to as the cd cd and cd datasets, respectively. microarray aliquots of 200 ng total rna were amplified and labelled for human gene st 96-array using the ambion wt expression kit and genechip wt terminal labeling and controls kit, according to the manufacturer’s protocols. samples were run on a genetitan multi-channel instrument as part of a larger dataset acquired over multiple years. for comparison with ncounter data, we selected only batches with at least samples from the desired cohorts covering at least two different diagnoses to ensure batch correction was resistant to outliers and confounding structure. microarray data for samples used in the ncounter comparison have been deposited in arrayexpress with accession number e-mtab- microarray data processing gene expression microarrays were filtered for sex discordance and global dimness before data processing. because the robustness of microarray normalization improves with the number of samples included, arrays were normalized in large, cell-type-specific batches, including all available samples from the selected batches with diagnoses tested in this study. samples were preprocessed with rma using the oligo bioconductor package with pd.hugene..st.v annotation. batch correction was performed using the combat function of the sva bioconductor package specifying diagnosis, sex, and age as covariates to avoid removal of biological differences. quality control was performed with the arrayqualitymetrics bioconductor package. ncounter control gene choice control genes for the ncounter analysis system were chosen for each cell type on the basis of consistent expression across samples in the large, cell-type-specific microarray datasets described above. rma-preprocessed datasets were separated by batch, and the variance of each gene calculated across all samples. gene variances were then averaged across batches. examining average variance versus mean expression revealed low variance at very high expression levels, suggesting array saturation or preprocessing compression effects. in order to select control genes that were well-expressed but not completely saturated, we filtered for genes with mean microarray expression values between and. these genes were then sorted by average within-batch variance, and control genes were chosen from the 2% with the lowest variance based on functional annotation suggesting stable, high expression in leukocytes. the use of two control genes per dataset was modelled after reis et al.. ncounter rna was prepared for and run on the ncounter analysis system, according to the manufacturer’s protocol in a total of runs over 8 days. to test for rna degradation, several sentinel samples from each cell-type-specific dataset were examined by bioanalyzer according to the manufacturer’s protocol. all sentinel samples were confirmed to be of good quality with rin ≥ . one sample was duplicated in a separate ncounter run to evaluate reproducibility and run-specific effects. rna was loaded at 100 ng per sample with the exception of one sample with low rna yield; this was run at 59 ng and did not result in a low-count quality control flag. all hybridizations were 17 hours long, and all counts were gathered by scanning on high mode for fields of view per sample. ncounter data processing no ncounter samples were flagged by nsolveranalysissoftware for quality control. ncounter data was normalized for hybridization and counting efficiency in cell-type specific groups. each sample was multiplicatively normalized to have the same geometric mean of ncounter-provided positive control probe counts. no normalization factors were outside the nanostring-recommended range of - thresholds to identify expressed genes from background noise were then calculated as the median of the maxima of the negative control probe measurements for each cell type. selection of microarray probesets for comparison between platforms microarray probesets mapping to genes measured by the ncounter analysis system were identified by affymetrix genechip human gene st arrays probeset annotations release. where multiple probesets were available for a given gene, the probeset with the best target region overlap with the ncounter target region was chosen. where all probesets were equivalent in overlap, one was chosen at random. correlation between platforms ncounter data was log-transformed and normalized to either a single, or to the mean of two, log-transformed control gene measurements. where indicated, microarray expression values were similarly normalized to single or multiple control gene expression values. the two platforms were compared by pearson correlation of each gene across samples. log-ratio accuracy and precision analysis signal detection slopes were calculated by taking the slope of a linear model fit to log-transformed microarray expression values versus log-transformed, control-gene-normalized ncounter measurements. although spline-fitted curves are often used for similar spike-in experiments to allow for expression-level dependencies, expression of each gene is generally contained within a smaller range than spike-in controls, and therefore we found it appropriate to use linear regression for each individual gene. for this analysis, each ncounter dataset was normalized to two cell-type-specific control genes, while microarray datasets were not normalized to control genes. noise in unexpressed microarray probesets was examined by measuring the standard deviation of all possible microarray log-ratios between pairs of samples for each gene in one of two sets: 1) all genes called unexpressed by the ncounter; and 2) all genes called invariant across samples by the ncounter but strictly unsaturated on the microarray.  microarray expression value level and variance indicate transcript presence and correlation with ncounter measurements we compared a previously acquired single-color microarray dataset of samples from batches, containing multiple cell types and diagnoses, with ncounter data from of these same rna samples, acquired in ncounter analysis system runs over 8 days. an additional rna samples were measured by ncounter for inter-run comparisons. the ncounter panel was designed to detect transcripts of genes, including cell-type-specific control genes, spanning a wide range of expression levels and variances. two ncounter probesets were excluded: one for poor predicted hybridization specificity, and one based on hybridization failure. ncounter data was processed as described in methods, using spiked positive and negative control probes to correct for hybridization efficiency and determine count thresholds for unexpressed genes, respectively. technical replicates of the same sample on different ncounter runs were highly correlated, and variation between samples of different diagnoses was greater than that between samples of the same diagnosis in different ncounter runs, implying very little inter-run variability. reduced inter-run correlations in cd14+ monocyte samples were due to one outlier, which was not used in subsequent ncounter versus microarray comparisons. based on this analysis, we considered ncounter run-effects negligible and processed all samples for each cell type together, disregarding run membership. as the ncounter analysis system experiment measured a small number of genes, it was not possible to normalize across samples by fitting expression measures to a common distribution. instead, ncounter transcript counts were normalized to selected control genes, as described in methods. because cell type greatly influences expression patterns, and because our microarray datasets consisted of cell-type-specific batches that confound effects from these two variables, each cell type was treated as a separate dataset for processing. microarray datasets were preprocessed by rma and, where applicable, corrected for batch effects by combat, as described in methods. the resulting log-transformed, standardized microarray values are referred to as “microarray expression values” throughout this manuscript. with ncounter thresholds determined from negative control probe counts, we identified expressed and unexpressed genes in each cell type. unexpressed genes were characterized by both low expression and low variance on the microarray. it is important to note that many genes identified as expressed by the ncounter analysis system had microarray expression values lower than those identified as unexpressed, confirming on a new platform the observation made by irizarry and colleagues that probe-effects prevent strict, experiment-wide thresholding of expressed genes. introducing a second variance threshold might improve identification of unexpressed genes, but a larger study examining more genes would be required to develop such a rule.figure median and variance properties of expressed versus unexpressed genes and interplatform correlation. a) genes are plotted by cell-type-specific rma + combat- or rma-preprocessed microarray expression value median and variance. red indicates genes called unexpressed by ncounter measurement. b) ncounter datasets were normalized to two cell-type-specific control genes each, log-transformed and compared to rma + combat- or rma-preprocessed microarray data without control-gene normalization. plots show pearson correlation of each expressed gene versus the median or variance of its microarray expression values or control-gene-normalized, log-transformed ncounter measurements: blue = cd green = cd red = cd datasets. c) rma + combat- or rma-preprocessed microarray data plotted versus ncounter measurements normalized to two cell-type-specific control genes. we examined the correlation of each expressed gene across samples between the microarray and ncounter platforms, varying the choice of control genes for ncounter data, the use or absence of control gene normalization for preprocessed microarray data, and the use or absence of microarray batch correction. choice of ncounter control genes changed correlation results slightly, with multiple control genes generally performing better than single control genes, as previously suggested for other technologies. the poorer performance of cd4+ t cell samples normalized to pias suggests that this gene was not as invariant as indicated by its microarray expression values, perhaps due to microarray saturation. control-gene-normalizing the microarray data generally improved correlation, but the moderate extent of this improvement indicates acceptable robustness of the control genes used. cd and cd rna samples were run on multiple microarray batches, leaving microarray datasets potentially confounded by technical artefacts. although principal component analyses of the entire datasets before batch correction demonstrated obvious batch effects, batch correction had very little effect on most genes examined in this study. in fact, correlation between microarray and ncounter measurements remained globally unchanged with batch correction of the cd and cd microarray datasets. direct comparison of batch-corrected and non-batch-corrected data indicates that this correction dramatically improved correlation of several genes in the cd dataset while the median gene correlation decreased slightly, possibly due to a reduction in covariance that would be predicted if batch correction reduced microarray variance in comparison to noise levels. these gene-specific effects highlight the importance of using batch-correction algorithms, such as combat, that can normalize at the gene level even with many covariate groups spread across batches.examination of the inter-platform correlation coefficient for each expressed gene versus either its median expression level or variance demonstrated that high variance generally corresponded with good correlation. indeed, pearson correlation relies on the covariance of two variables, inherently requiring variation of both, but it is interesting to note that many low-variance genes also exhibited good inter-platform correlation. this empirically indicates high precision of microarray measurements, allowing successful identification of true transcript variation over platform-specific noise for many genes, even with low inter-sample variation. in comparisons of inter-platform correlation with median expression, genes with very low and very high expression generally had poorer correlation. directly plotting microarray expression values against their corresponding control-gene-normalized ncounter measurements revealed large variability at the low end of expression and flattening of microarray values at the high end. taken together, these data suggest that low expression measurements on the microarray may have been obscured by background noise while high measurements were likely saturated. microarray signal detection accuracy depends on expression level and is dataset-specific one metric used for determining microarray measurement accuracy is the “signal detection slope”, or the slope of linear regression relating measured microarray expression values to log-transformed, known input transcript concentrations. a slope of one indicates that the microarray accurately reflects the input. to examine microarray accuracy using real experimental data, we calculated the signal detection slope of microarray expression values versus log-transformed, control-gene-normalized ncounter measurements by fitting linear regressions to the paired platform measurements of each gene. figure 2a shows signal detection slope plotted against correlation coefficient. because both correlation coefficients and signal detection slopes were determined by comparison of the same measurements, genes with low correlation also showed low signal detection. however, genes with high correlation did not necessarily have high signal detection, instead exhibiting a much wider range of signal detection slopes. in order to better understand this spread, we filtered for genes with good inter-platform correlation and plotted these signal detection slopes versus the median microarray expression value. signal detection accuracy was globally reduced, with slopes particularly dampened in genes with high and low microarray expression values. even within the same expression level ranges, signal detection varied by dataset, indicating variable effects of independent preprocessing and/or tissue type on accuracy reduction. batch correction slightly reduced signal detection in general, again possibly due to covariance reduction through decreasing microarray variance.figure signal detection accuracy of microarray gene expression values. a) signal detection slopes of rma + combat- or rma-preprocessed microarray expression values for all expressed genes are plotted against their inter-platform correlation coefficient. b) signal detection slopes of rma + combat- or rma-preprocessed microarray expression values for all genes with correlated ncounter and microarray measures are plotted against median microarray values. noise in microarray expression values is dataset-specific as noted above, inter-platform correlation of low-variance genes demonstrated wide variability within and across datasets, suggesting variable levels of noise in the microarray measurements. to examine microarray expression value precision in each dataset, we adapted a metric developed by mccall et al. for spike-in data and examined the standard deviation of microarray log-ratios of unexpressed and invariant genes, as described in methods. noise of unexpressed genes varied between datasets, indicating either a tissue-specific effect or an artefact of the preprocessing of each dataset. batch correction slightly reduced this variation, suggesting improvements in precision. comparison with microarray median expression values demonstrated that noise amplitude of unexpressed genes remained largely independent of microarray expression values. the cd dataset had significantly less noise than the cd and cd datasets, implying that comparisons of low variance gene expression values might be more reliable in this dataset. indeed, this was observed in the generally improved inter-platform correlation coefficients of low-variance genes in the cd subset. because noise in unexpressed genes only addresses precision at low microarray expression values and because our unexpressed gene sets were of different sizes, we also examined microarray precision in invariant genes over a wider range of expression. to this end, we used ncounter measurements to select genes with low variance across all samples disregarding diagnosis and strictly filtered for genes not saturated on the microarray. examination of this noise metric across datasets revealed the same trends for batch correction and cell-type-specificity. comparing precision of invariant gene measurements versus median expression values on the microarray indicated that genes with very high expression tended to better precision. although we had filtered strictly to avoid saturated microarray expression values, this precision trend, as well as the binned variance properties depicted in additional file suggests that saturation or compression artefacts of rma preprocessing may begin to reduce inter-sample variation at even lower expression values.figure signal detection precision of microarray gene expression values. a) boxplots depict the standard deviations of log-ratios of all pairs of samples for unexpressed genes. stars indicate mann–whitney test p-value < . b) as a for genes invariant by ncounter but unsaturated on the microarray. discussion while we are not the first to ask questions about microarray interpretation of gene expression and log-ratios, our approach is unique in its use of a count-based technology to examine diverse biological samples. this has enabled us to examine the properties of microarray data representative of real experimental questions and to present the new observation of tissue-specific noise. through examination of inter-platform correlation and expressed versus unexpressed gene properties, we have demonstrated that variance measurement may aid in the determination of whether or not a gene is expressed, particularly at low microarray expression values. irizarry and colleagues have clearly shown that a low microarray expression value does not necessarily predict an unexpressed gene. the small number of genes tested in the present study would suggest that simultaneous thresholds for microarray gene expression level and variance might perform better than expression level thresholds alone to identify present versus absent transcripts, but ncounter measurement of a larger gene set would be necessary to develop such a method. indeed present/absent calls, expression thresholds and, most popularly, variance filters are frequently used in downstream microarray analyses to reduce multiple testing and improve power for discovery. our demonstration that unexpressed genes generally have low microarray expression values and variances provides empirical justification particularly for using the intersection of such filters to remove uninformative probesets. our comparison of log-fold-changes measured by microarray and ncounter platforms has revealed that signal detection accuracy of the microarray varies dramatically by expression level. similar to previous studies with constructed datasets, our data show global compression most extreme at very high and low gene expression values. importantly, we noted that signal detection slope compression also varies by dataset, indicating that effect sizes measured by log-fold changes are not necessarily comparable between datasets preprocessed separately. thus, comparing the magnitudes of expression changes in genes expressed at different levels or genes in different datasets requires knowledge of their individual signal detection slopes. for precision estimation, we examined noise of genes defined as unexpressed or invariant by ncounter measurements. the rma algorithm stabilizes variance with respect to expression level, and thus noise patterns from probesets detecting unexpressed genes should reflect noise across the whole microarray. accordingly, we found the same patterns of precision estimates in unexpressed genes as in invariant genes across a wider range of expression levels. we noted that noise appeared strongly tissue-dependent, with less in the cd than the cd or cd datasets. binned mean expression versus variance plots support this pattern of tissue-specific precision, showing a high-variance peak at very low expression levels in the cd and cd datasets that is nearly absent in the cd dataset. because the same rna was used for both the ncounter and microarray measurements, dataset-dependent noise differences can only be due to properties intrinsic to the frozen rna samples. transcriptomes differ by tissue, and thus one explanation for differing noise levels is that levels of cross-hybridizing nucleotides also vary by cell type. we previously showed that rna transcript profiles of cells from the myeloid lineage change dramatically if blood is left several hours before processing. although we found no evidence of large-scale loss of rna integrity, it is possible that even during rapid blood processing, rna from myeloid cells suffers slightly more degradation, a factor likely to confound microarray more than the ncounter measurements. another plausible explanation is cell-type-specific contamination with genomic dna, particularly in the cd16+ neutrophil subset, which has comparatively less rna per cell and thus a higher ratio of genomic dna to rna. genomic dna would likely be problematic in the transcriptional step of microarray sample preparation but not the amplification-free ncounter procedure. regardless of the source of this dataset-specific noise, such effects are important to remember for cross-tissue studies, such as the gene expression barcode, where probes reflecting signal in one tissue type may be conflated by noise in another. our precision results are based on examination of a limited number of genes in three different leukocyte subsets, and future studies of more genes in additional tissue types will begin to shed light on the origin and extent of this dataset-specific noise.  this analysis of gene expression microarray measurements versus transcript count ratios highlights three aspects of microarray data directly relevant to users of the technology. first, inter-sample variance may indicate transcript presence in genes with low microarray expression values. second, signal detection accuracy depends strongly on expression level, even in datasets of diverse biological samples with variable background and small gene expression ranges. third, precision is dataset-specific, and therefore power to detect subtle biological differences may differ between tissues even when measured on the same microarray platform. without careful consideration of these biases and confirmatory measurements by a second technology, microarray platform discoveries may be missed or misinterpreted. availability of supporting data the data sets supporting the results of this article are available in the arrayexpress repository: https://www.ebi.ac.uk/arrayexpress/experiments/e-mtab-2452/. electronic supplementary material additional file 1:: preprocessing samples. microarray samples used for preprocessing: breakdown of preprocessed microarray batches including biological covariates. additional file 2:: mean v variance. binned mean and variance characteristics of microarray datasets: for each gene, a microarray expression value mean and mean within-batch variance was calculated. genes were then binned by expression value means, and statistics were averaged to achieve an average mean expression value and average mean within-batch-variance for each bin. these two values are plotted. additional file 3:: control genes. microarray properties of ncounter control genes: details of each cell-type-specific control gene used. additional file 4:: ncounter run comparison. examination of inter-run technical effects of ncounter data: a) log-transformed raw ncounter counts for technical replicates of the same sample are plotted with pearson correlation indicated. all genes determined to be globally expressed in cd samples are included. b) boxplots depict pearson correlations between log-transformed raw ncounter counts for samples of the same and different diagnoses in the same and in different ncounter runs. all genes determined to be globally expressed in the designated cell types are included. in the bottom panel, the outlying cd sample has been removed. c) inter-sample pearson correlation coefficients of log-transformed raw ncounter counts between all cd samples. red star indicates outlier. additional file 5:: ncounter probes. ncounter probe details and mapped affymetrix hugene st array probesets: ncounter probe design schemes, isoform coverage, and microarray probeset mappings are tabulated. additional file 6:: ncounter samples. sample composition: details are provided for samples run on the ncounter analysis system. additional file 7:: correlation comparison. effects of microarray and ncounter processing on inter-platform correlation: cell-type-specific ncounter datasets were normalized to the indicated control genes and log-transformed. microarray data were preprocessed by rma and then batch normalized through combat and/or normalized to control genes where indicated. boxplots show pearson correlation. additional file 8:: correlation comparison table. effects of microarray and ncounter processing on inter-platform correlation: table summarizes inter-platform correlation of datasets using different processing and normalization procedures. additional file 9:: microarray batch effects. batch effects in microarray datasets: a) samples from full cd and cd microarray datasets are plotted by first and second principle components before and after combat batch correction. color indicates batch membership. b) pearson correlation of expressed genes across samples in ncounter versus rma-preprocessed microarray datasets was subtracted from the same correlation in ncounter versus rma-preprocessed and combat-corrected microarray datasets. boxplots depict these differences in cd and cd datasets to indicate the effect of batch correction on gene-based platform correlation. additional file 10:: batch correction and accuracy. effect of batch correction on signal detection accuracy: a) signal detection slope is plotted versus inter-platform correlation as in figure 2a: blue = rma- and red = rma + combat-preprocessed microarray expression values. b) signal detection slope of expressed genes across samples in ncounter versus rma-preprocessed microarray datasets was subtracted from the same signal detection slope in ncounter versus rma-preprocessed and combat-corrected microarray datasets. boxplots depict these differences in cd and cd datasets to indicate the effect of batch correction on signal detection accuracy. additional file 11:: noise v expression value. comparison of noise versus microarray expression value. a) for each unexpressed gene, the standard deviation of log-ratios of all pairs of samples from rma + combat- or rma-preprocessed microarray data is plotted versus the gene’s median microarray expression value. b) as for invariant genes. additional file 12:: mean expression histograms. mean expression profiles: histograms depict mean rma + combat- or rma-preprocessed microarray expression values from full microarray datasets. abbreviations rmarobust multi-array average qpcrquantitative polymerase chain reaction aavanti-neutrophil-cytoplasmic-antibody-associated vasculitis ibdinflammatory bowel disease cdcrohn’s disease uculcerative colitis gpagranulomatosis with polyangiitis mpamicroscopic polyangiitis. competing interests the authors declare that they have no competing interests. authors’ contributions acr participated in the design of the study, collected the ncounter measurement data, carried out the comparative analysis and drafted the manuscript. pal participated in the design of the study, advised the analysis and helped to draft the manuscript. jep contributed to the analysis methods and execution and helped to draft the manuscript. db and smf participated in the design of the ncounter assay, and smf helped to edit the manuscript. jcl recruited patients and processed peripheral blood samples for the ibd cohort. efm recruited patients and processed samples for the aav cohort. rms advised and participated in the design of the ncounter assay and helped to draft the manuscript. kgcs participated in the design of the study, advised the analysis and helped to draft the manuscript. all authors read and approved the final manuscript. 
most importantly, examination of three different cell types revealed that noise levels differed across tissues. although numerous investigations have compared gene expression microarray platforms, preprocessing methods and batch correction algorithms using constructed spike-in or dilution datasets, there remains a paucity of studies examining the properties of microarray data using diverse biological samples. most microarray experiments seek to identify subtle differences between samples with variable background noise, a scenario poorly represented by constructed datasets. using a set of human leukocyte subset rna samples, we compared previously acquired microarray expression values with rna molecule counts determined by the ncounter analysis system in selected genes. confirming previous findings from spike-in and dilution datasets, this “gold-standard” comparison demonstrated signal compression that varied dramatically by expression level and, to a lesser extent, by dataset. microarray measurements generally correlate with relative rna molecule counts within optimal ranges but suffer from expression-dependent accuracy bias and precision that varies across datasets.
 although numerous investigations have compared gene expression microarray platforms, preprocessing methods and batch correction algorithms using constructed spike-in or dilution datasets, there remains a paucity of studies examining the properties of microarray data using diverse biological samples. most microarray experiments seek to identify subtle differences between samples with variable background noise, a scenario poorly represented by constructed datasets. thus, microarray users lack important information regarding the complexities introduced in real-world experimental settings. the recent development of a multiplexed, digital technology for nucleic acid measurement enables counting of individual rna molecules without amplification and, for the first time, permits such a study.  using a set of human leukocyte subset rna samples, we compared previously acquired microarray expression values with rna molecule counts determined by the ncounter analysis system in selected genes. we found that gene measurements across samples correlated well between the two platforms, particularly for high-variance genes, while genes deemed unexpressed by the ncounter generally had both low expression and low variance on the microarray. confirming previous findings from spike-in and dilution datasets, this “gold-standard” comparison demonstrated signal compression that varied dramatically by expression level and, to a lesser extent, by dataset. most importantly, examination of three different cell types revealed that noise levels differed across tissues.  microarray measurements generally correlate with relative rna molecule counts within optimal ranges but suffer from expression-dependent accuracy bias and precision that varies across datasets. we urge microarray users to consider expression-level effects in signal interpretation and to evaluate noise properties in each dataset independently. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords microarraynanostringncountergene expressionissue-copyright-statement© the author 2014 