BACKGROUND
in contemporary high-throughput experiments, very many descriptor values can be measured, leading to the issue of correction for multiple testing to minimize false positives at the cost of a high number of false negatives.
reconciliation entails compromises that are to some extent arbitrary.
a deterministic method is needed for selecting a minimal, distinguished set of descriptors that collectively provide effective, efficient prediction.
researchers can subsequently investigate members of such a subset to determine exactly how they are related  and perhaps why they should be inherently associated with predictions .
meta-analysis is the general body of knowledge that addresses integrating results from multiple experimental programs on one topic; the purpose of this paper is to suggest inclusion of blanket as an additional technique  <cit> .
regarding related papers, we note that hess and iyer found that fisher's combined p method applied to microarray data from spike-in experiments with rt-qpcr validation usually compared favorably to other methods  <cit> .
however, they observed that other probe level testing methods generally selected many of the same genes as differentially expressed.
so the method of finding differentially expressed genes is not the critical issue.
as they further noted, current methods for analyzing microarray data do better at ranking genes rather than maintaining stated false positive rates.
lists of descriptors ranked by informativeness are often encountered in the general pursuit of relationships among diseases, physiological processes, and the action of small molecule therapeutics.
notable examples include the connectivity map by lamb et al.
and the generation of quantitative structure-activity relationships   <cit> .
kazius et al.
considered n compounds, each of which either is or is not toxic   <cit> .
they characterized compounds by substructures, each compound either including or not including a given substructure.
inclusion of any substructure thereby can be considered as a potential toxicity descriptor, and the point of kazius et al.
was analysis of single experiments to determine toxicity.
blanket could be applied to the outcomes of two experiments that use the same set of descriptors.
regarding genes as descriptors , a vast, public repository of data that should support discovery of distinguished descriptor lists is supported by the gene expression omnibus  project.
geo predominantly stores gene expression data generated by microarray technology  <cit> .
another huge data resource is oncomine as developed by rhodes et al.
<cit> .
oncomine includes statistical reports on some  <dig>  cancer gene expression microarrays.
methods
presented first are two synthetic examples.
suppose the number of distilled descriptors n =  <dig> and the ranked list for experiment a is simply labeled  <dig>   <dig>  ...,  <dig>  suppose in the ranked list for experiment b, the first ten are a random permutation of  <dig>   <dig>  ...,  <dig>  and the other  <dig> are a random permutation of  <dig>   <dig> ..., <dig>  we would expect blanket to suggest an optimal subset of the first ten just as is shown in figure  <dig>   note the appearance of the blanket surface: a plateau of rfet values near  <dig> for very low p+q abruptly falls to a floor of values near  <dig> as p+q increases.
by definition of rfet, the extreme pairs with p =  <dig> or n, or q =  <dig> or n values have rfet =  <dig>  a property of all blankets.
so to speak, the square blanket surface is supported at value  <dig> around its edge and dips to positive values ≤ <dig> in its interior.
random blankets  seem to have no such patterns of plateaus and floors and they generally have larger minimum values.
as a second synthetic example, suppose experiment a descriptors again have canonical ordering  <dig> ,..., <dig> while experiment b has the same with local permutations from weighted noise.
figure  <dig> shows that the noise in the ranked lists can be sufficient to preclude shortlists of length <  <dig>  but three survive the <  <dig> criterion.
again there is a plateau of rfet values near  <dig>  falling abruptly to a floor of near  <dig> values.
next the blanket method will be used to evaluate data from a classic microrna  microarray paper by he et al.
<cit> .
the spreadsheet data from the paper are in the ncbi/nlm/geo web site with accession number gse <dig>  entitled "microrna expression in lymphoma lines"  <cit> .
two experiments evaluated mirna expression levels in cell lines oci-ly <dig> and oci-ly <dig> ; these cell lines carry amplification of genomic region of interest 13q31-q <dig> that is thought to be oncogenic.
in each experiment the results from the cell lines were compared to the same measurements of normal b-cells.
he et al.
measured in quadruplicate for cases and for controls  <dig> mature mirna levels for normal b-cells and several cell lines including oci-ly <dig> and oci-ly <dig>  thus experiment a includes the 190-by- <dig> output matrix of normal  b-cell mirna values versus oci-ly <dig>  mirna values, and as experiment b the same from oci-ly <dig> values.
this yields p-values and hence rankings of the two lists of  <dig> mirnas.
we tested the hypothesis that the same mirnas can differentiate control b-cells from both of the two cases oci-ly <dig> and oci-ly <dig> by attempting to find informative subsets of the  <dig> probes.
the blanket for these data is shown as the surface in figure  <dig>  this blanket finds that three combinations of shortlists that achieve the p, q ≤  <dig> threshold for n =  <dig>  namely,  <dig> e- <dig> .
that is, rfet =  <dig> e- <dig> for ; rfet =  <dig> e- <dig> for ; and rfet =  <dig> e- <dig> for .
the top  <dig> of the oci-ly <dig> list are: let-7e, -7g, -7c, -7f, -7d, -7a, and mir-373*.
the top  <dig> of the oci-ly <dig> list are mir-373*, let-7a, -7c, -7f, mir- <dig>  - <dig>  -15a, - <dig>  and let-7g.
it is already obvious from the heatmap in figure  <dig> of the he paper that the let- <dig> family is distinguished by case versus control.
aside from the let- <dig> family, the union of the blanket shortlists contains five other mirnas: hsa-mir-373*, - <dig>  - <dig>  -15a, and - <dig>  there is an interesting alignment among these:
hsa-mir-138   5' agcu-gguguugugaaucaggccg 3'
|||| |||
hsa-mir-423   5' agcucggucugaggccccucagu 3'
this alignment invites investigation because the bases near the 5' terminus  are generally thought by mirna researchers to be most important in terms of targeting and gene regulation  <cit> .
possibly the similarity of mir- <dig> and mir- <dig> in this respect implies the two are actually redundant; redundancy is considered a hallmark of mirna targeting efficacy  <cit> .
redundancy might allow fine tuning when one is upregulated in case and the other downregulated, as is so for these mirnas.
otherwise, shortlisted descriptors might exhibit consistent change associations between the two experiments, as is the case for  <dig> of the other  <dig> mirnas in the blanket union of shortlists for these data.
blanket is next applied to suggest shortlists of genes from experiments with lung adenocarcinoma measurements versus control tissue measurements in microarray studies by stearman et al.
and bhattacharjee et al.
<cit> .
each study contains a statistical contrast of normal lung tissue versus adenocarcinoma tissue.
the gene symbols and associated p-values from each study can be downloaded from oncomine.
the stearman study considers  <dig> genes ; the number for bhattacharjee is  <dig>
ranked by lowest p-values, the top  <dig> genes in each experiment can be selected.
the intersection of the lists can be reranked to a list of  <dig> genes that are possibly informative in both experiments.
blanket yields the surface in figure  <dig>
from the stearman data blanket chooses  <dig> genes: gpc <dig>  sox <dig>  grk <dig>  adh1b, clec3b, mfap <dig>  tek, fh <dig>  aoc <dig>  tbx <dig>  cox7a <dig>  tgfbr <dig>  mylk, vwf.
blanket chooses  <dig> genes from the bhattacharjee data: hyal <dig>  grk <dig>  spock <dig>  eno <dig>  sema5a, cdh <dig>  vwf, cox7a <dig>  thus the intersection is {grk <dig>  cox7a <dig>  vwf} with rfet =  <dig> e- <dig>
interestingly, several additional papers connect some of the shortlisted genes with lung cancer.
regarding gpc <dig>  powell et al.
used microarrays to identify gpc <dig> as one of several genes the expression of which was lower in the healthy lung tissue of smokers than in nonsmokers and was lower in tumor tissue than in healthy tissue  <cit> .
additionally, northern blot analysis demonstrated that gpc <dig> expression was absent in  <dig> of  <dig> lung cancer cell lines.
regarding adh1b, kopantzev et al.
employed cdnas sequencing and rt-qpcr analysis to measure genes differentiated in comparison of human fetal versus adult lungs and in normal lung tissue versus non-small lung cell carcinomas  <cit> .
adh1b was one of  <dig> genes found to have opposite differentiation in the two comparisons.
regarding clec3b, reduced plasma levels have long been associated with cancer and metastasis  <cit> .
regarding tek, millauer et al.
implicated tek among growth factor receptor tyrosine kinases in angiogenesis, and findley et al.
demonstrated that vegf regulates tek signaling  <cit> .
regarding aoc <dig>  singh et al.
found that expression may contribute to the functional heterogeneity of endothelial cells within the lung to create distinct sites for the recruitment of inflammatory cells  <cit> .
regarding hyal <dig>  li et al.
studied genetic aberrations in the genes hyal <dig>  fhit, and other genes in paired tumors and sputum samples from  <dig> patients with stage i non-small cell lung cancer and in sputum samples from  <dig> cancer-free smokers and  <dig> healthy nonsmokers  <cit> .
they found hyal <dig> and fhit were deleted in 84% and 79% tumors and in 45% and 40% paired sputum samples.
regarding eno <dig>  chang et al.
observed that only a limited number of immunogenic tumor-associated antigens have been identified and associated with lung cancer  <cit> .
they reported up-regulation of eno <dig> expression in effusion tumor cells from  <dig> of  <dig> patients compared with human normal lung primary epithelial and non-cancer-associated effusion cells.
regarding mylk, soung et al.
analyzed exons  <dig> and  <dig> encoding the kinase domain for somatic mutations in  <dig> gastric,  <dig> colorectal,  <dig> non-small cell lung, and  <dig> breast cancers  <cit> .
they found one mylk <dig> mutation in lung adenocarcinomas, but not in other cancers.
regarding sema5a, sadanandam et al.
demonstrated an association between the expression of sema5a and plexin b <dig> and the aggressiveness of pancreatic and prostate cancer cells  <cit> .
they deduced that sema5a is among functional tumor-specific cam genes, which may be critical for organ-specific metastasis.
regarding cdh <dig> and intersection gene vwf, smirnov et al.
reported increased numbers of endothelial cells in peripheral blood of cancer patients  <cit> .
they found expression of vwf, dtr, cdh <dig>  tie, and igfbp <dig> genes discriminated between cancer patients and healthy donors with a receiver operating characteristic curve accuracy of  <dig> .
of the other two genes in the intersection, grk <dig> is a g protein-coupled receptor kinase and is highly expressed in lung  <cit> .
lastly, cox7a <dig> is  <dig> kb from and possibly co-expressed with fxyd <dig> , a cancer-associated cell membrane glycoprotein that promotes experimental cancer metastasis  <cit> .
in summary, there are potential lung cancer connections with genes in the stearman-bhattacharjee blanket shortlists.
this illustrates the main output of blanket, namely, suggestions to researchers of small subsets of genes especially worthy of further investigation.
the next analysis pertains to an instance in which blanket does not suggest informative shortlists; this example compares results of stearman with another lung cancer study by beer et al.
<cit> .
preprocessing starting with the  <dig> most differentiated gene lists leads to selection of  <dig> shared genes.
as shown in figure  <dig>  the blanket surface does not display a sharply defined subset of informative descriptors, that is, no plateau that falls precipitously to a floor of rfet values near zero.
lastly, blanket applied to the third possible combination of tests  is not interesting.
it finds a very small subset pair .
such small shortlists are evident by inspection because the top two for beer are adh1b and clec3b while the top five for bhattacharjee are gpc <dig>  sox <dig>  grk <dig>  adh1b, clec3b.
implementation
following is a pipeline  for processing descriptors measured in two experiments called a and b. each experiment routinely yields a matrix of values with descriptors labeling the rows and samples labeling the columns.
an additional column is the informativeness of each descriptor from application of student t-testing or another method; that additional column is used to rank the descriptors by informativeness, yielding the two ranked descriptor lists used as inputs by blanket.
considerable preprocessing might be needed to derive the ranked lists.
this is because the raw data  typically have many thousands of descriptors, from which one distills hundreds that are significantly up or down in case versus control; the researcher might wish to treat up- and downregulated genes separately.
the intersection of the two lists must be found and then a selection made of the topmost descriptors  of the two lists.
some of the top  <dig> in one list might not be in the top  <dig> of the other, so a second intersection is needed to yield a list of genes with different rankings in the two experiments, that is, somewhat fewer, shared, ranked descriptors suitable for blanket.
real data tested in preparation of this paper yielded an intersection n = ~ <dig> to ~ <dig> descriptors.
using the list of shared descriptors and selecting the top p descriptors from a and the top q descriptors from b yields a contingency table.
over the discrete plane of all possible pairs, rfet values can be represented as a blanket-like surface.
after finding shortlist pairs that provide rfet values lower than the values in table  <dig>  the researcher should select shortlists as follows: for each selected shortlist pair , no other shortlist pair  also has p' ≤ p, q' ≤ q, and p'+q' < p+q.
all such values and corresponding descriptors should be noted by the researcher.
all genes that achieve a level of informativeness discovered in a blanket selection might be considered.
that is, the union of all the descriptors in the shortlists might be informative, as well as, of course, the intersection.
if several pairs of shortlists fulfill this condition, then minimizing the rfet values or minimizing p+q might yield especially interesting shortlist pairs.
RESULTS
if two experiments of case versus control have substantially overlapping descriptor sets and a consistent, binary categorization of outcome, then standard statistical analyses can provide two lists, ranked by informativeness, of the shared descriptors.
the ranked lists suggest two questions:
question 1: from the results of the two experiments, is there a minimal subset of descriptors that predicts experimental outcome much better than smaller subsets and about as well as any larger subsets?
question 2: if existence of such a minimal subset is indicated, then what are its members?
the focus here is on one method that answers these questions.
we call our method blanket; this is not an acronym, but merely a term suggestive of a blanket-like surface suspended above a plane of shortlist length pairs.
suppose two experiments such as microarray analyses investigate informativeness of descriptors relative to a property of samples.
here a descriptor  is any tested type of measurement, such as detection of messenger rna of a certain gene in a microarray experiment  or presence or absence of a certain chemical substructure in a compound evaluated for toxicity .
a property of the samples could be case versus control, survival time, or another characteristic or outcome of interest.
blanket treats the set of shared descriptors as two ranked lists.
the informativeness of each descriptor, considered in isolation, can be determined by a by t-test, z-test, or other method; our only requirement is that informativeness analysis for each experiment yields a ranked list.
the two experiments might use the same or different definitions of informativeness.
the basic idea of blanket is consideration of shortlists of descriptors from each experiment, say the top p of n descriptors of experiment a and the top q of the same n descriptors of experiment b. for any such pair of shortlists, four numbers are evident: the number of descriptors appearing in both shortlists, in exactly one shortlist, or in neither shortlist.
the sum of all four is n; the sum of the first two is p; and the sum of the first and third is q. blanket computes right fisher exact test  values used as scores over a discrete plane of all possible pairs of shortlist lengths  with  <dig> ≤ p, q ≤ n).
blanket then chooses one pair or a few pairs with rfet score less than a threshold; the threshold depends upon n and upper bounds of shortlist lengths.
the threshold has been determined by simulations and represents a quality of rfet value achieved by only 5% of random lists.
a further property of a pair  selected by blanket is parsimony, that is, that no other pair  exists with p' ≤ p, q' ≤ q, p'+q' < p+q, and an rfet score that also survives the threshold.
multiple shortlists could be scored by smallness of p+q.
furthermore, we seek to represent the information to the researcher in a visual form such as an excel spreadsheet surface graph that invites assessment based upon a researcher's experience with data of a given type, much in the manner of the commonplace heatmap.
theoretical basis
our approach is to consider the rfet value for all combinations of shortlist lengths  <dig> or  <dig> within ranked descriptor lists of length n =  <dig>   <dig>   <dig>   <dig>  or  <dig>  in the grid of lengths, this can be thought of as the examination of all p-by-q rectangles of rfet values within a given nxn square, subject to p ≤ n and q ≤ n. the rfet attaining the minimum nominal p-value is then compared to the null distribution of such minimum p-values, obtained via permutation, which assumes that the orderings of the two lists of descriptors are random.
the corresponding  <dig>  quantile values are used as rejection thresholds for controlling the overall type i error at  <dig> .
formally, the approach is the single-step westfall-young permutation p-value for potentially correlated tests, which controls the family-wise error and avoids the excessive conservativeness of bonferroni bounds  <cit> .
furthermore, the approach has an exact interpretation as a kind of randomization test of a statistic  in a population of equally likely outcomes  conditioned on some aspect of the data   <cit> .
this is an attractive approach, as it makes very few assumptions about the data and is entirely nonparametric.
discussion
the term blanket reflects the shapes of the surfaces in figures  <dig>   <dig>   <dig>   <dig> and  <dig>  we can reason as follows about the shape.
if the threshold for at least one of the lists is too strict  so that one shortlist is empty or small and there is no intersection, then rfet = 1; likewise, if at least one shortlist is the universe of descriptors, then rfet =  <dig>  thus the boundary of the blanket surface over the full range of all threshold pairs necessarily has fixed value  <dig>  this insures that seeking interior points with relatively low rfet values on the surface makes sense.
to our knowledge, blanket is a novel means for nominating distinguished subsets of descriptors from data from two experiments.
blanket suggests shortlists  of genes from each list, where the shortlists achieve a certain level of informativeness individually.
the subsets then collectively differentiate case from control.
while the preprocessing considers the full ranked lists, blanket does not make global declarations.
that is, blanket ignores very uninformative descriptors but can tolerate descriptors with marginal p-values provided they consistently appear among the best found of ranked lists.
other related scores that might be substituted for the rfet score are pearson's chi-square test and the g-test  <cit> .
once a distinguished set of descriptors has been verified, dependencies among the descriptors might be discovered by applying cronbach's α test  <cit> .
another meta-analysis paper is that of blangiardo and richardson  <cit> .
they also scored 2-by- <dig> contingency tables derived from ranked lists, seeking a "...parsimonious list associated with the strongest evidence of dependence between experiments."
their pioneering work differs from ours three respects.
first is their use of a given number  of bins so that a bin could contain all of a subset of descriptors with close p-values.
second, the hypergeometric distribution is the score of the paired bins as shortlists.
third, and perhaps most importantly, they only scored shortlists of equal length, hence the diagonal of the discrete space of all combinations of bin sizes.
by contrast we consider all "rectangular" combinations of shortlists lengths that lie within a larger "square"  of combinations.
the significance of the restriction of consideration to shortlists of equal length can be illustrated as follows.
suppose that two experiments test  <dig> descriptors for case and control informativeness, providing two ranked lists.
suppose the first experiment is very accurate but the second is not; perhaps the second employs a noisier technology but still might provide a degree of confirmation.
suppose the top four descriptors in the first experiment appear in rank positions  <dig>   <dig>   <dig>   <dig> of the second, and that no other descriptors in the top twenties are shared.
blanket correctly selects the top  <dig> of the first list and the top  <dig> of the second, with rfet = .
<dig> < .
<dig> in table  <dig>  however, requiring shortlists of equal length forces consideration of the top  <dig>   <dig>   <dig>  and  <dig> of both lists.
this results in rfet and hypergeometric distribution p-values all above .
<dig>  that is, restricting consideration to shortlists of equal length would find no informative shortlists and in particular would miss the combination  found by blanket.
regarding other related papers, we note that hess and iyer reported that fisher's combined p method applied to microarray data from spike-in experiments with rt-qpcr validation usually compared favorably to other methods  <cit> .
as they further noted, current methods for analyzing microarray data do better at ranking genes rather than maintaining stated false positive rates.
breitling et al., devised the "rank product" method which in simplest form uses multiplication across n experiments of the reciprocal of rank positions of n descriptors, leading to a kind of global ranking  <cit> .
in some cases, two logically distinct lines of experimentation might lead to two classes, each including many experiments.
the rank product approach might be applied to experiments from one class and then the other, and the two resulting global ranked lists be submitted to blanket.
for example, in the context of a given case versus control study, a global ranked list could be derived from many microarray experiments.
then the same genes could be ranked from keyword studies of research papers associating them with case outcomes, again producing a global ranked list.
finally, the two global ranked lists, from very different lines of investigation, could be analyzed by blanket to discover collectively informative subsets of genes.
an enhancement of blanket in gene expression analysis of microarrays might include consistency of fold change.
that is, the researcher might require that the genes in the intersection of shortlists all have fold change >  <dig> or all have fold change <  <dig> for case versus control.
doing this for randomly generated ranked lists and random fold changes would result in a table like table  <dig> but with increased values.
CONCLUSIONS
the blanket method provides a visual representation of optimal selections of subsets of informative descriptors.
a key observation in our real data is that there can be an abruptly lower  rfet score value, going from a plateau of almost  <dig> to a valley floor of almost  <dig> values as shortlist lengths are slightly incremented.
furthermore, if upper limits on the shortlist lengths are specified as  <dig> or  <dig>  then our simulations provide values for rfet scores that allow rejection of the null hypothesis with 95% certainty.
in such circumstances, blanket can suggest a sharp distinction between slightly too few and slightly too many descriptors, that is, a classifier based upon optimal collective informativeness.
authors' contributions
all three wrote sections of the paper.
cdj conceived an initial version of the blanket algorithm; wow executed early applications and r code and contributed much of the text; dop contributed refinements regarding applications; and faw contributed the random permutation design and simulations, r code, and theoretical foundations and analysis.
supplementary material
additional file 1
r program for blanket.
r program for blanket.
this program yields a value that can be tested in table  <dig> for statistical significance of the discovered shortlists.
click here for file
acknowledgements
this paper reflects several enhancements prompted by critical review of an earlier version; the authors thank the reviewers.
support for our research has included grants from an anonymous donor, stanley medical research foundation grant 08r- <dig> "herpesviruses in schizophrenia risk," nih grant 5p01es014635- <dig> "the system of response to dna damage suppresses environmental melanomagenesis," and nih grant 2r01gm066940-05a <dig> "predictive qsar modeling."
the content is solely the responsibility of the authors, not the funding institutions.
this article was reviewed by the national health and environmental effects research laboratory, us environmental protection agency, and approved for publication.
approval does not signify that the contents necessarily reflect the views and policies of the agency nor does the mention of trade names or commercial products constitute endorsement or recommendation for use.

 in contemporary high-throughput experiments, very many descriptor values can be measured, leading to the issue of correction for multiple testing to minimize false positives at the cost of a high number of false negatives. reconciliation entails compromises that are to some extent arbitrary. a deterministic method is needed for selecting a minimal, distinguished set of descriptors that collectively provide effective, efficient prediction. researchers can subsequently investigate members of such a subset to determine exactly how they are related and perhaps why they should be inherently associated with predictions. meta-analysis is the general body of knowledge that addresses integrating results from multiple experimental programs on one topic; the purpose of this paper is to suggest inclusion of blanket as an additional technique. regarding related papers, we note that hess and iyer found that fisher's combined p method applied to microarray data from spike-in experiments with rt-qpcr validation usually compared favorably to other methods. however, they observed that other probe level testing methods generally selected many of the same genes as differentially expressed. so the method of finding differentially expressed genes is not the critical issue. as they further noted, current methods for analyzing microarray data do better at ranking genes rather than maintaining stated false positive rates. lists of descriptors ranked by informativeness are often encountered in the general pursuit of relationships among diseases, physiological processes, and the action of small molecule therapeutics. notable examples include the connectivity map by lamb et al. and the generation of quantitative structure-activity relationships. kazius et al. considered n compounds, each of which either is or is not toxic. they characterized compounds by substructures, each compound either including or not including a given substructure. inclusion of any substructure thereby can be considered as a potential toxicity descriptor, and the point of kazius et al. was analysis of single experiments to determine toxicity. blanket could be applied to the outcomes of two experiments that use the same set of descriptors. regarding genes as descriptors, a vast, public repository of data that should support discovery of distinguished descriptor lists is supported by the gene expression omnibus project. geo predominantly stores gene expression data generated by microarray technology. another huge data resource is oncomine as developed by rhodes et al.. oncomine includes statistical reports on some cancer gene expression microarrays. methods presented first are two synthetic examples. suppose the number of distilled descriptors n = and the ranked list for experiment a is simply labeled..., suppose in the ranked list for experiment b, the first ten are a random permutation of..., and the other are a random permutation of..., we would expect blanket to suggest an optimal subset of the first ten just as is shown in figure note the appearance of the blanket surface: a plateau of rfet values near for very low p+q abruptly falls to a floor of values near as p+q increases. by definition of rfet, the extreme pairs with p = or n, or q = or n values have rfet = a property of all blankets. so to speak, the square blanket surface is supported at value around its edge and dips to positive values ≤ in its interior. random blankets seem to have no such patterns of plateaus and floors and they generally have larger minimum values. as a second synthetic example, suppose experiment a descriptors again have canonical ordering,..., while experiment b has the same with local permutations from weighted noise. figure shows that the noise in the ranked lists can be sufficient to preclude shortlists of length < but three survive the < criterion. again there is a plateau of rfet values near falling abruptly to a floor of near values. next the blanket method will be used to evaluate data from a classic microrna microarray paper by he et al.. the spreadsheet data from the paper are in the ncbi/nlm/geo web site with accession number gse entitled "microrna expression in lymphoma lines". two experiments evaluated mirna expression levels in cell lines oci-ly and oci-ly; these cell lines carry amplification of genomic region of interest 13q31-q that is thought to be oncogenic. in each experiment the results from the cell lines were compared to the same measurements of normal b-cells. he et al. measured in quadruplicate for cases and for controls mature mirna levels for normal b-cells and several cell lines including oci-ly and oci-ly thus experiment a includes the 190-by- output matrix of normal b-cell mirna values versus oci-ly mirna values, and as experiment b the same from oci-ly values. this yields p-values and hence rankings of the two lists of mirnas. we tested the hypothesis that the same mirnas can differentiate control b-cells from both of the two cases oci-ly and oci-ly by attempting to find informative subsets of the probes. the blanket for these data is shown as the surface in figure this blanket finds that three combinations of shortlists that achieve the p, q ≤ threshold for n = namely, e-. that is, rfet = e- for; rfet = e- for; and rfet = e- for. the top of the oci-ly list are: let-7e, -7g, -7c, -7f, -7d, -7a, and mir-373*. the top of the oci-ly list are mir-373*, let-7a, -7c, -7f, mir- - -15a, - and let-7g. it is already obvious from the heatmap in figure of the he paper that the let- family is distinguished by case versus control. aside from the let- family, the union of the blanket shortlists contains five other mirnas: hsa-mir-373*, - - -15a, and - there is an interesting alignment among these: hsa-mir-138   5' agcu-gguguugugaaucaggccg 3'                  |||| ||| hsa-mir-423   5' agcucggucugaggccccucagu 3' this alignment invites investigation because the bases near the 5' terminus are generally thought by mirna researchers to be most important in terms of targeting and gene regulation. possibly the similarity of mir- and mir- in this respect implies the two are actually redundant; redundancy is considered a hallmark of mirna targeting efficacy. redundancy might allow fine tuning when one is upregulated in case and the other downregulated, as is so for these mirnas. otherwise, shortlisted descriptors might exhibit consistent change associations between the two experiments, as is the case for of the other mirnas in the blanket union of shortlists for these data. blanket is next applied to suggest shortlists of genes from experiments with lung adenocarcinoma measurements versus control tissue measurements in microarray studies by stearman et al. and bhattacharjee et al.. each study contains a statistical contrast of normal lung tissue versus adenocarcinoma tissue. the gene symbols and associated p-values from each study can be downloaded from oncomine. the stearman study considers genes; the number for bhattacharjee is ranked by lowest p-values, the top genes in each experiment can be selected. the intersection of the lists can be reranked to a list of genes that are possibly informative in both experiments. blanket yields the surface in figure from the stearman data blanket chooses genes: gpc sox grk adh1b, clec3b, mfap tek, fh aoc tbx cox7a tgfbr mylk, vwf. blanket chooses genes from the bhattacharjee data: hyal grk spock eno sema5a, cdh vwf, cox7a thus the intersection is {grk cox7a vwf} with rfet = e- interestingly, several additional papers connect some of the shortlisted genes with lung cancer. regarding gpc powell et al. used microarrays to identify gpc as one of several genes the expression of which was lower in the healthy lung tissue of smokers than in nonsmokers and was lower in tumor tissue than in healthy tissue. additionally, northern blot analysis demonstrated that gpc expression was absent in of lung cancer cell lines. regarding adh1b, kopantzev et al. employed cdnas sequencing and rt-qpcr analysis to measure genes differentiated in comparison of human fetal versus adult lungs and in normal lung tissue versus non-small lung cell carcinomas. adh1b was one of genes found to have opposite differentiation in the two comparisons. regarding clec3b, reduced plasma levels have long been associated with cancer and metastasis. regarding tek, millauer et al. implicated tek among growth factor receptor tyrosine kinases in angiogenesis, and findley et al. demonstrated that vegf regulates tek signaling. regarding aoc singh et al. found that expression may contribute to the functional heterogeneity of endothelial cells within the lung to create distinct sites for the recruitment of inflammatory cells. regarding hyal li et al. studied genetic aberrations in the genes hyal fhit, and other genes in paired tumors and sputum samples from patients with stage i non-small cell lung cancer and in sputum samples from cancer-free smokers and healthy nonsmokers. they found hyal and fhit were deleted in 84% and 79% tumors and in 45% and 40% paired sputum samples. regarding eno chang et al. observed that only a limited number of immunogenic tumor-associated antigens have been identified and associated with lung cancer. they reported up-regulation of eno expression in effusion tumor cells from of patients compared with human normal lung primary epithelial and non-cancer-associated effusion cells. regarding mylk, soung et al. analyzed exons and encoding the kinase domain for somatic mutations in gastric, colorectal, non-small cell lung, and breast cancers. they found one mylk mutation in lung adenocarcinomas, but not in other cancers. regarding sema5a, sadanandam et al. demonstrated an association between the expression of sema5a and plexin b and the aggressiveness of pancreatic and prostate cancer cells. they deduced that sema5a is among functional tumor-specific cam genes, which may be critical for organ-specific metastasis. regarding cdh and intersection gene vwf, smirnov et al. reported increased numbers of endothelial cells in peripheral blood of cancer patients. they found expression of vwf, dtr, cdh tie, and igfbp genes discriminated between cancer patients and healthy donors with a receiver operating characteristic curve accuracy of. of the other two genes in the intersection, grk is a g protein-coupled receptor kinase and is highly expressed in lung. lastly, cox7a is kb from and possibly co-expressed with fxyd, a cancer-associated cell membrane glycoprotein that promotes experimental cancer metastasis. in summary, there are potential lung cancer connections with genes in the stearman-bhattacharjee blanket shortlists. this illustrates the main output of blanket, namely, suggestions to researchers of small subsets of genes especially worthy of further investigation. the next analysis pertains to an instance in which blanket does not suggest informative shortlists; this example compares results of stearman with another lung cancer study by beer et al.. preprocessing starting with the most differentiated gene lists leads to selection of shared genes. as shown in figure the blanket surface does not display a sharply defined subset of informative descriptors, that is, no plateau that falls precipitously to a floor of rfet values near zero. lastly, blanket applied to the third possible combination of tests is not interesting. it finds a very small subset pair. such small shortlists are evident by inspection because the top two for beer are adh1b and clec3b while the top five for bhattacharjee are gpc sox grk adh1b, clec3b. implementation following is a pipeline for processing descriptors measured in two experiments called a and b. each experiment routinely yields a matrix of values with descriptors labeling the rows and samples labeling the columns. an additional column is the informativeness of each descriptor from application of student t-testing or another method; that additional column is used to rank the descriptors by informativeness, yielding the two ranked descriptor lists used as inputs by blanket. considerable preprocessing might be needed to derive the ranked lists. this is because the raw data typically have many thousands of descriptors, from which one distills hundreds that are significantly up or down in case versus control; the researcher might wish to treat up- and downregulated genes separately. the intersection of the two lists must be found and then a selection made of the topmost descriptors of the two lists. some of the top in one list might not be in the top of the other, so a second intersection is needed to yield a list of genes with different rankings in the two experiments, that is, somewhat fewer, shared, ranked descriptors suitable for blanket. real data tested in preparation of this paper yielded an intersection n = ~ to ~ descriptors. using the list of shared descriptors and selecting the top p descriptors from a and the top q descriptors from b yields a contingency table. over the discrete plane of all possible pairs, rfet values can be represented as a blanket-like surface. after finding shortlist pairs that provide rfet values lower than the values in table the researcher should select shortlists as follows: for each selected shortlist pair, no other shortlist pair also has p' ≤ p, q' ≤ q, and p'+q' < p+q. all such values and corresponding descriptors should be noted by the researcher. all genes that achieve a level of informativeness discovered in a blanket selection might be considered. that is, the union of all the descriptors in the shortlists might be informative, as well as, of course, the intersection. if several pairs of shortlists fulfill this condition, then minimizing the rfet values or minimizing p+q might yield especially interesting shortlist pairs.  if two experiments of case versus control have substantially overlapping descriptor sets and a consistent, binary categorization of outcome, then standard statistical analyses can provide two lists, ranked by informativeness, of the shared descriptors. the ranked lists suggest two questions: question 1: from the results of the two experiments, is there a minimal subset of descriptors that predicts experimental outcome much better than smaller subsets and about as well as any larger subsets? question 2: if existence of such a minimal subset is indicated, then what are its members? the focus here is on one method that answers these questions. we call our method blanket; this is not an acronym, but merely a term suggestive of a blanket-like surface suspended above a plane of shortlist length pairs. suppose two experiments such as microarray analyses investigate informativeness of descriptors relative to a property of samples. here a descriptor is any tested type of measurement, such as detection of messenger rna of a certain gene in a microarray experiment or presence or absence of a certain chemical substructure in a compound evaluated for toxicity. a property of the samples could be case versus control, survival time, or another characteristic or outcome of interest. blanket treats the set of shared descriptors as two ranked lists. the informativeness of each descriptor, considered in isolation, can be determined by a by t-test, z-test, or other method; our only requirement is that informativeness analysis for each experiment yields a ranked list. the two experiments might use the same or different definitions of informativeness. the basic idea of blanket is consideration of shortlists of descriptors from each experiment, say the top p of n descriptors of experiment a and the top q of the same n descriptors of experiment b. for any such pair of shortlists, four numbers are evident: the number of descriptors appearing in both shortlists, in exactly one shortlist, or in neither shortlist. the sum of all four is n; the sum of the first two is p; and the sum of the first and third is q. blanket computes right fisher exact test values used as scores over a discrete plane of all possible pairs of shortlist lengths with ≤ p, q ≤ n). blanket then chooses one pair or a few pairs with rfet score less than a threshold; the threshold depends upon n and upper bounds of shortlist lengths. the threshold has been determined by simulations and represents a quality of rfet value achieved by only 5% of random lists. a further property of a pair selected by blanket is parsimony, that is, that no other pair exists with p' ≤ p, q' ≤ q, p'+q' < p+q, and an rfet score that also survives the threshold. multiple shortlists could be scored by smallness of p+q. furthermore, we seek to represent the information to the researcher in a visual form such as an excel spreadsheet surface graph that invites assessment based upon a researcher's experience with data of a given type, much in the manner of the commonplace heatmap. theoretical basis our approach is to consider the rfet value for all combinations of shortlist lengths or within ranked descriptor lists of length n = or in the grid of lengths, this can be thought of as the examination of all p-by-q rectangles of rfet values within a given nxn square, subject to p ≤ n and q ≤ n. the rfet attaining the minimum nominal p-value is then compared to the null distribution of such minimum p-values, obtained via permutation, which assumes that the orderings of the two lists of descriptors are random. the corresponding quantile values are used as rejection thresholds for controlling the overall type i error at. formally, the approach is the single-step westfall-young permutation p-value for potentially correlated tests, which controls the family-wise error and avoids the excessive conservativeness of bonferroni bounds. furthermore, the approach has an exact interpretation as a kind of randomization test of a statistic in a population of equally likely outcomes conditioned on some aspect of the data. this is an attractive approach, as it makes very few assumptions about the data and is entirely nonparametric. discussion the term blanket reflects the shapes of the surfaces in figures and we can reason as follows about the shape. if the threshold for at least one of the lists is too strict so that one shortlist is empty or small and there is no intersection, then rfet = 1; likewise, if at least one shortlist is the universe of descriptors, then rfet = thus the boundary of the blanket surface over the full range of all threshold pairs necessarily has fixed value this insures that seeking interior points with relatively low rfet values on the surface makes sense. to our knowledge, blanket is a novel means for nominating distinguished subsets of descriptors from data from two experiments. blanket suggests shortlists of genes from each list, where the shortlists achieve a certain level of informativeness individually. the subsets then collectively differentiate case from control. while the preprocessing considers the full ranked lists, blanket does not make global declarations. that is, blanket ignores very uninformative descriptors but can tolerate descriptors with marginal p-values provided they consistently appear among the best found of ranked lists. other related scores that might be substituted for the rfet score are pearson's chi-square test and the g-test. once a distinguished set of descriptors has been verified, dependencies among the descriptors might be discovered by applying cronbach's α test. another meta-analysis paper is that of blangiardo and richardson. they also scored 2-by- contingency tables derived from ranked lists, seeking a "...parsimonious list associated with the strongest evidence of dependence between experiments." their pioneering work differs from ours three respects. first is their use of a given number of bins so that a bin could contain all of a subset of descriptors with close p-values. second, the hypergeometric distribution is the score of the paired bins as shortlists. third, and perhaps most importantly, they only scored shortlists of equal length, hence the diagonal of the discrete space of all combinations of bin sizes. by contrast we consider all "rectangular" combinations of shortlists lengths that lie within a larger "square" of combinations. the significance of the restriction of consideration to shortlists of equal length can be illustrated as follows. suppose that two experiments test descriptors for case and control informativeness, providing two ranked lists. suppose the first experiment is very accurate but the second is not; perhaps the second employs a noisier technology but still might provide a degree of confirmation. suppose the top four descriptors in the first experiment appear in rank positions of the second, and that no other descriptors in the top twenties are shared. blanket correctly selects the top of the first list and the top of the second, with rfet =. <. in table however, requiring shortlists of equal length forces consideration of the top and of both lists. this results in rfet and hypergeometric distribution p-values all above. that is, restricting consideration to shortlists of equal length would find no informative shortlists and in particular would miss the combination found by blanket. regarding other related papers, we note that hess and iyer reported that fisher's combined p method applied to microarray data from spike-in experiments with rt-qpcr validation usually compared favorably to other methods. as they further noted, current methods for analyzing microarray data do better at ranking genes rather than maintaining stated false positive rates. breitling et al., devised the "rank product" method which in simplest form uses multiplication across n experiments of the reciprocal of rank positions of n descriptors, leading to a kind of global ranking. in some cases, two logically distinct lines of experimentation might lead to two classes, each including many experiments. the rank product approach might be applied to experiments from one class and then the other, and the two resulting global ranked lists be submitted to blanket. for example, in the context of a given case versus control study, a global ranked list could be derived from many microarray experiments. then the same genes could be ranked from keyword studies of research papers associating them with case outcomes, again producing a global ranked list. finally, the two global ranked lists, from very different lines of investigation, could be analyzed by blanket to discover collectively informative subsets of genes. an enhancement of blanket in gene expression analysis of microarrays might include consistency of fold change. that is, the researcher might require that the genes in the intersection of shortlists all have fold change > or all have fold change < for case versus control. doing this for randomly generated ranked lists and random fold changes would result in a table like table but with increased values.  the blanket method provides a visual representation of optimal selections of subsets of informative descriptors. a key observation in our real data is that there can be an abruptly lower rfet score value, going from a plateau of almost to a valley floor of almost values as shortlist lengths are slightly incremented. furthermore, if upper limits on the shortlist lengths are specified as or then our simulations provide values for rfet scores that allow rejection of the null hypothesis with 95% certainty. in such circumstances, blanket can suggest a sharp distinction between slightly too few and slightly too many descriptors, that is, a classifier based upon optimal collective informativeness. authors' contributions all three wrote sections of the paper. cdj conceived an initial version of the blanket algorithm; wow executed early applications and r code and contributed much of the text; dop contributed refinements regarding applications; and faw contributed the random permutation design and simulations, r code, and theoretical foundations and analysis. supplementary material additional file 1 r program for blanket. r program for blanket. this program yields a value that can be tested in table for statistical significance of the discovered shortlists. click here for file acknowledgements this paper reflects several enhancements prompted by critical review of an earlier version; the authors thank the reviewers. support for our research has included grants from an anonymous donor, stanley medical research foundation grant 08r- "herpesviruses in schizophrenia risk," nih grant 5p01es014635- "the system of response to dna damage suppresses environmental melanomagenesis," and nih grant 2r01gm066940-05a "predictive qsar modeling." the content is solely the responsibility of the authors, not the funding institutions. this article was reviewed by the national health and environmental effects research laboratory, us environmental protection agency, and approved for publication. approval does not signify that the contents necessarily reflect the views and policies of the agency nor does the mention of trade names or commercial products constitute endorsement or recommendation for use.
from the associated contingency table, blanket computes right fisher exact test values used as scores over a plane of possible pairs of shortlist lengths. researchers seek within a universe of descriptors some minimal subset that collectively and efficiently predicts experimental outcomes. from resulting lists of n descriptors ranked by informativeness, blanket determines shortlists of descriptors from each experiment, generally of different lengths p and q. for any pair of shortlists, four numbers are evident: the number of descriptors appearing in both shortlists, in exactly one shortlist, or in neither shortlist. combining information from these studies can increase the reliability and generalizability of results and also yield new insights that guide future research. blanket then chooses a pair or pairs with rfet score less than a threshold; the threshold depends upon n and shortlist length limits and represents a quality of intersection achieved by less than 5% of random lists.
 improvements in high-throughput technology and its increasing use have led to the generation of many highly complex datasets that often address similar biological questions. combining information from these studies can increase the reliability and generalizability of results and also yield new insights that guide future research.  this paper describes a novel algorithm called blanket for symmetric analysis of two experiments that assess informativeness of descriptors. the experiments are required to be related only in that their descriptor sets intersect substantially and their definitions of case and control are consistent. from resulting lists of n descriptors ranked by informativeness, blanket determines shortlists of descriptors from each experiment, generally of different lengths p and q. for any pair of shortlists, four numbers are evident: the number of descriptors appearing in both shortlists, in exactly one shortlist, or in neither shortlist. from the associated contingency table, blanket computes right fisher exact test values used as scores over a plane of possible pairs of shortlist lengths. blanket then chooses a pair or pairs with rfet score less than a threshold; the threshold depends upon n and shortlist length limits and represents a quality of intersection achieved by less than 5% of random lists.  researchers seek within a universe of descriptors some minimal subset that collectively and efficiently predicts experimental outcomes. ideally, any smaller subset should be insufficient for reliable prediction and any larger subset should have little additional accuracy. as a method, blanket is easy to conceptualize and presents only moderate computational complexity. many existing databases could be mined using blanket to suggest optimal sets of predictive descriptors. 