BACKGROUND
meta-analysis has long been used to integrate data and/or results from multiple studies targeting the same questions.
it is commonly used in many areas of statistical applications such as clinical studies and psychology experiments.
in recent years, meta-analysis has been frequently adopted in genomic data analysis, due to the fast development of high-throughput technology and the vast amounts of data available in public databases.
many meta-analysis methods have been developed throughout the years.
roughly speaking, there are two main approaches to meta-analysis methods song and tseng  <cit> .
the first directly combines the p-values from the studies, while the second attempts to model the data or the effect sizes from the combined studies.
the former includes methods such as the fisher’s combined probability test  <cit>  and the stouffer’s z-test  <cit> , as well as weighted variations of these classical tests.
the latter includes a variety of fixed effects and random effects models, such as genemeta  <cit> .
each approach has its advantages and disadvantages.
the p-value combining methods are relatively flexible in that they require minimal information and assumptions from the studies.
in this paper we will mainly focus on methods that directly combine the p-values.
most of the traditional meta-analysis methods  aim at testing the alternative hypothesis that at least one of the studies is non-null.
while this aligns with earlier goals of meta-analyses to gain power in detecting signals by combining multiple studies, it is frequently not the case with genomic data.
in meta-analysis of genomic studies for example, the goal is often to identify genes that are differentially expressed in a consistent pattern across multiple studies.
the extreme of this would be to test for the alternative that the null can be rejected for all the studies.
a solution to this extreme alternative dates back to the maxp method by wilkinson  <cit> .
however, the maxp method is often considered too conservative.
a recent approach by phillips and ghosh  <cit>  improves the power of testing for this disjunction of nulls when the rejection of all p-values associated with a gene is required.
in practical meta-analyses, the goal of rejecting all studies may be considered too extreme.
ideally, we would want to target at detecting consistent signals across studies while avoiding being overly exclusive.
this issue has gained attention in recent years, and a number of authors have tried to address this problem.
benjamini and heller  <cit>  discussed a framework for testing partial conjunction hypotheses, where they test for the alternative that at least u out of n null hypotheses are false against the partial conjunction null that no more than n−u+ <dig> of the null hypotheses are true.
song and tseng  <cit>  proposed the rth ordered p-value  method that aims at testing the alternative hypothesis that there is signal in at least a given percentage of studies.
other methods exist that address this problem from different approaches, such as rankprod by hong et al.
<cit>  that looks for consistently highly ranked genes, and a weighted approach by li and ghosh  <cit>  that weights genes by its expression consistency across studies.
we consider the problem of detecting signals in the majority of studies.
our approach adopts one aspect of the rop method song and tseng  <cit>  in that we also consider ordered p-values.
but instead of using a single rth ordered p-value as the statistic , we combine all or a subset of the ordered p-values while weighting them based on their order .
p-values closer to the median are highly weighted and the smallest/largest p-values are down-weighted.
the idea is that among the collection of p-values, the median p-values are likely to be a better reflection of the behavior of the majority of studies than the smallest or largest p-values.
olkin and saner  <cit>  discussed a trimmed fisher’s procedure that leaves out a number of the smallest and/or largest p-values from the calculation of fisher’s statistic to remove the effect of possible aberrant extremes.
in our consideration, we still keep the smallest/largest p-values because they do carry certain information, but we down-weight them because they may be relatively less relevant when considering the “majority” of studies.
to reflect the up-weighting of the medians and down-weighting of the extremes, we calculated our weights based on the binomial distribution.
to summarize the weighted ordered p-values, we mainly considered fisher’s statistic and stouffer’s z-test.
we also explored some other statistics, such as generalized fisher’s test by lancaster  <cit> .
in general, other summary statistics can be used under this framework as well.
while many weighted variations of fisher’s statistic and stouffer’s statistic have been developed throughout the years, most of them distribute weights according to the sample sizes and/or effect sizes of the studies, or other similar considerations .
li and tseng  <cit>  proposed an interesting adaptively weighted statistic, where the weights are used to maximize the significance of the summary statistic.
however, to the best of our knowledge, none of the weighting schemes are based on the ordered p-values, which is what makes our method unique.
xie et al.
<cit>  discussed a meta-analysis approach using confidence distributions, where they incorporated the use of medians and kernel functions, but their approach is under a completely different framework from ours.
by incorporating more than one order statistic into our summary statistic, our method can be considered an expansion of the rth ordered p-value  method song and tseng  <cit> .
in general, both the rop method and the original summary statistic we use  are special cases under the wop framework - one having all the weight on a single ordered p-value and the other having evenly distributed weights.
however, our wop methods have respective advantages over both the traditional summary statistics and the rop method.
compared to the traditional summary statistics, the wop methods better focus on detecting signal in a majority of studies.
on the other hand, the wop methods appear to be more robust compared to the rop method.
these observations are based upon results from simulation studies as well as data applications.
methods
hypothesis settings for meta-analysis
before performing any meta-analysis, it is always important to figure out the goal of combining multiple studies.
when a single hypothesis test is conducted, it is clear what the null and alternative hypotheses are.
in meta-analysis we usually combine studies designed to test the same set of null and alternative hypotheses.
however, the null and alternative hypotheses of the meta-analysis test are not always obvious and largely depends on the researcher’s goals.
here we consider the example of meta-analysis of differential expression studies of genomic data.
sometimes a gene is of interest as long as it is differentially expressed in at least one study, while other times we hope to target genes that are differentially expressed in all studies.
li and tseng  <cit>  and song and tseng  <cit>  both provided extensive discussions on the different scenarios that lead to different hypothesis settings.
following their notation, let θ
g
k
be the true effect size for gene g  in study k .
as in song and tseng  <cit> , for a given 1≤r≤k, a general hypothesis setting for the meta-analysis of the k studies for a given gene g can be formulated as:
hsr:h0:∑k=1ki=0versusha:∑k=1ki≥r.
when r= <dig>  hs <dig> is the classical setting of testing for non-zero effect size in at least one study against the conjunction of nulls.
hs <dig> is the hypothesis setting that fisher’s method, stouffer’s z-test and many other traditional methods test for.
when r=k, hs
k
tests for the alternative that all the studies have non-zero effect size.
for instance, the maxp method  <cit>  tests for hs
k
.
when 1<r<k,hs
r
provides a compromise between the two aforementioned hypothesis settings, and tests for at least a pre-specified number of non-zero effects.
for a given r, the rop  method song and tseng  <cit>  is used to test for hs
r
.
in this paper, we test for non-zero effect sizes in a majority of studies against the null that the effects sizes are zero in all studies.
thus we are testing against the conjunction of nulls while trying to focus on a certain subset of the non-null space.
in general, the hypothesis for our meta-analysis approach can be considered to fall under the hs
r
setting.
song and tseng  <cit>  suggested a few data-driven methods for selecting r. to prevent any potential issues of post hoc choices of r, we choose to fix r before any analysis is conducted.
while it is hard to specify what the term “majority” exactly means, as a general rule, we choose to test the hypothesis setting hs
m
, where m=⌈k/2⌉, ⌈x⌉ being the smallest integer no less than x. essentially we are targeting the alternative that at least half of the studies have non-zero effect sizes.
under our weighted ordered p-values  framework, we are able to develop methods for other hypothesis settings with r ranging from m+ <dig> to k. but in general we hope to provide a simple to use method without having to put too much effort in selecting a particular r, and therefore we will focus mostly on testing hs
m
.
we will later show through data applications that using our wop methods for testing hs
m
provides more robust results compared to using the rop method with different choices of r.
a framework for weighted ordered p-values  methods
we first describe the general framework for the weighted ordered p-values  methods.
suppose we have p-values p <dig> p <dig> ⋯,p
k
for testing the hypothesis of interest for each of the k studies.
let p,p,⋯,p be the ordered p-values.
now consider a set of weights w <dig> w <dig> ⋯,w
k
associated with the corresponding ordered p-values.
summary statistics of weighted ordered p-values can be expressed in the following general form:
t=∑i=1kwih).
as mentioned by previous authors, many traditional p-value combination methods can be expressed in the general form of t′=∑i=1kwi′h .
for instance, stouffer’s z-test takes h to be the inverse normal function, while fisher’s method has h=− <dig> log.
the difference between t′ and t, albeit subtle in notation, is the essence of the wop framework.
in the wop framework, the weight w
i
is associated with the ith ordered p-value p. in other words, the ranking of a p-value in relation to the p-values from the other studies determines its weighting.
in traditional weighted p-value combining methods, the weight assigned to a p-value is associated with the characteristics of that particular study, be it the sample size, the effect size, or other features.
allowing the weights to depend on the ordering of the p-values opens up a whole new area of considerations when combining multiple p-values.
we can consider giving more weights to the p-values that are closer to the center of the distribution of the p-values since they might hold more credibility, and at the same time down-weight the outlying p-values.
for instance, if the entirety of weights is placed on the rth ordered p-value, the statistic reduces to the rop method.
however, using the wop framework, if we highly weight the rth ordered p-value but still distribute some weight to the other p-values, the method can be viewed as a more robust version of the rop method for testing hs
r
.
in this paper, we shall develop a few specific methods under the wop framework.
we consider weights based on the binomial distribution, which will be described in more detail in the next section.
as for the p-value combining methods, we shall focus on fisher’s method, with h
f
)=− <dig> log), and stouffer’s method, with h
z
)=ϕ−1), ϕ being the standard normal distribution function.
binomial weights and half-binomial weights
in this section, we discuss two possible weighting schemes for the wop framework.
we mainly consider testing the alternative hypothesis that the effect sizes are non-zero in the majority of studies, which is the hypothesis setting hs
m
.
we will also briefly discuss extending the weighting schemes to testing other hypothesis settings hs
r
, for m<r≤k.
inspired by the rop method, which uses the rth ordered p-value for testing the hypothesis setting hs
r
, we consider placing the highest weight on the median p-values for testing hs
m
.
this makes intuitive sense, since if a consensus does exist among the studies, we have reason to believe that the behavior of the majority of studies should be best captured by the p-values that are closer to the center of the distribution.
since we do not insist on non-zero effect sizes for every single study, we consider down-weighting the largest p-values among the studies.
on the other hand, p-value combining methods such as fisher’s method are known to be very sensitive to single extremely small p-values, thus the smallest p-values should also be down-weighted, to avoid a small number of extremely small p-values biasing the results of the majority of studies.
in summary, we would like our weighting scheme w
i
, as a function of i, to reflect a unimodal shape, with the highest weights being w
m
or w
m− <dig> and w
m
, and such that w
i
decreases as i goes to  <dig> or k.
to reflect the above properties of the weights, we constructed the weights based on the binomial distribution.
let f be the probability mass function of the binomial distribution b, for x= <dig> ,⋯,n.
we define the binomial weighting scheme such that
wib=f,i= <dig> ,⋯,k.
in the binomial weighting scheme, all the weights are non-zero, thus every p-value contributes to the combined statistic.
to further reduce the influence of the smallest p-values on the summary statistic, we may argue that only p,p,⋯,p matters in testing the alternative that at least m studies have non-zero effect size.
with these considerations, we define what we call the half-binomial weighting scheme such that
wihb= <dig> i= <dig> ,⋯,m− <dig> wib,i=m,m+ <dig> ⋯,k.
we will discuss more on the effects of these two different weighting schemes through simulation studies in later sections.
so far we have constructed the binomial weighting scheme and half-binomial weighting scheme for testing the hypothesis setting hs
m
.
we can extend the ideas of these two weighting schemes to testing hs
r
, for m<r≤k.
instead of placing the highest weights on the medians, the highest weight can now be assigned to p. when r≠m, we lose the natural symmetry of the weights.
however, we can still base the weights on the binomial distribution.
a few possible weighting schemes are defined below:
wib1= <dig> i= <dig> ⋯,r−mwi−b,i=r−m+ <dig> ⋯,k.
wib2=f− <dig> .5),i= <dig> ⋯,k.
wib3= <dig> i= <dig> ⋯,2f−1;k−2− <dig> .5),i=2+ <dig> ⋯,k.
the weighting scheme w
b <dig> is based on the same binomial distribution as w
b
, except that the values are shifted so that the center of the distribution falls on r instead of m. because of the shift, the first few weights are set to be  <dig>  while the last few values in the probability mass function of the binomial distribution are truncated.
the weighting scheme w
b <dig> increases the parameter n to k+2− <dig> in the binomial distribution to ensure that the first few weights are non-zero.
the weighting scheme w
b <dig> decreases the parameter n to k−2− <dig> in the binomial distribution so that the distribution is not truncated on the right.
see figure  <dig> for an illustration of the idea of these three weighting schemes.
corresponding half-binomial weights for these three binomial weighting schemes: w
h
b <dig>  w
h
b <dig> and w
h
b <dig>  can be easily constructed by setting the weights to be  <dig> for i= <dig> ⋯,r− <dig>
implementation of the wop methods
under the null hypothesis, the p
k
’s are assumed to follow a uniform  distribution.
for stouffer’s z-test, h
z
=ϕ− <dig> follows a standard normal distribution.
therefore the traditional weighted z-test in the form of ∑i=1kwi′hz still follows a normal distribution.
for fisher’s method, h
f
=− <dig> log has a chi-square distribution with two degrees of freedom.
therefore the distribution of the traditional weighted fisher’s test ∑i=1kwi′hf is essentially weighted sums of exponential distributions.
the distribution of weighted sums of exponential variables is not as straightforward, though many authors have researched on both the exact and approximations of this distribution, a summary of which can be found in olkin and saner  <cit> .
when we consider weighted ordered p-values in the form of ∑i=1kwih), however, the problem becomes much more complicated.
even for stouffer’s method, the distribution of the sum of weighted ordered normal variables is not readily available.
as for fisher’s method, olkin and saner  <cit>  studied the distribution of the trimmed fisher’s statistic, which is ∑i=1k−2wilog) for the special case of w
i
= <dig> for i= <dig> ⋯,s <dig> and i=s <dig> ⋯,k.
they transformed the distribution of the sum of ordered chi-squared variables back to a weighted sum of exponential variables without order.
but as discussed in their paper, the exact distribution of weighted sums of exponential variables are generally impractical for practitioners, and we would therefore have to use approximation methods.
considering the complexity of the exact distributions of weighted sums of ordered variables, as well as the fact that the uniformness of the original p-values is not always guaranteed in practice, we recommend two methods for obtaining the p-values for the wop statistics:  permutation analysis, in the case that original data from all the studies are available; and  comparing to the numerical distribution, in the case that only the p-values for each gene and each study are known.
we first explain the steps of obtaining the wop p-values through permutation analysis:
•let tg=∑i=1kwih) denote the wop statistic for gene g, where p
g is the ith ordered p-value of p
g <dig> ⋯,p
g
k
.
•permute group labels in each study b times, and recalculate the p-values for the permuted data pgk, for 1≤k≤k, 1≤g≤g, 1≤b≤b.
•calculate the wop statistics for the permuted p-values tg for 1≤g≤g, 1≤b≤b.
•the p-value for the wop statistic t
g
is then computed as
pgt=∑b=1b∑g′=1gi{tg≥tg′}b·g.
once the p-values for the wop statistics for each gene are obtained, we may apply the benjamini-hochberg  method  <cit>  on the pgt’s  to account for multiple testing across the genes and control the false discovery rate .
if the original data are not available, we can simulate the distribution of the wop statistics numerically, by simulating u random variables, the distribution of p-values under the null distribution.
the wop statistics calculated from the data can then be compared to the numerical distribution to obtain the wop p-values.
we simulated numerical distributions of the wop statistics for testing hs
m
based on the fisher’s and stouffer’s combination methods with binomial and half-binomial weighting schemes respectively, for study numbers ranging from  <dig> to  <dig>  we conducted simulation studies to compare the wop p-values obtained either though comparing with the numerical distribution or by performing permutation analysis.
results show that the two methods provide perfectly correlated p-values and that the number of rejections obtained from the two methods after applying the benjamini-hochberg adjustment are very similar .
therefore both methods are reliable choices for obtaining the wop p-values in practice.
the numerical distribution provides an option for when the original data are not available and is also more time-efficient.
the permutation analysis can be used if the uniformness of the original p-values are questionable but that the original data is available.
considerations of one- or two-sided tests
previously we used two-sided alternatives as an example when setting up the hypothesis.
the hypothesis setup hs
r
can be similarly developed for one-sided alternatives.
in fact, the interpretation of the meta-analysis results is easier for one-sided tests, since we do not need to worry about the concordance of the directions of the effect sizes as we do for two-sided tests.
since the wop methods directly combine the p-values, the direction of the effect sizes are not taken into account for two-sided tests.
thus a significant result from the wop meta-analysis of two-sided tests indicate that there are at least r studies with non-zero effect size, but without any implications about the concordance or discordance of the directions of the effects.
this may not be an issue in the case that the direction of effect is not of great importance.
however, in genomic data analysis, it is often desirable to distinguish between up- and down-regulated genes, and a result stating that the gene is differentially expressed across many studies but with possible opposite directions of expression change may be confusing.
in these cases, it might be problematic to directly apply the wop methods to the two-sided p-values.
on the other hand, since both up- and down-regulated genes may be of interest at the same time, we cannot pre-specify one particular one-sided test for all genes.
for such scenarios we recommend using the test of pearson  <cit>  in combination with the wop methods.
to do so, for each gene we need to conduct two wop meta-analyses on one-sided p-values: one on the left-tailed p-values for all studies, and the other on the right-tailed p-values for all studies.
let pwopl and pwopr be the wop meta-analysis p-values for the left-tailed and right-tailed tests respectively.
we shall then adopt the idea of pearson’s test and define pwopc=min{ <dig> min}, where the superscript “c” stands for “concordant”.
as discussed in owen  <cit> , the equation for obtaining pwopc provides a conservative p-value for pearson’s test.
by adopting the pearson’s test, the results are more interpretable.
a significant result now indicates that the gene is consistently up- or down-regulated in at least r studies.
RESULTS
a simulation study
we conducted a simulation study to compare the performances of the wop methods with the original fisher’s and stouffer’s method, as well as with the rop method song and tseng  <cit> .
we shall also demonstrate the differences between the binomial and half-binomial weighting schemes through the simulation.
we simulate the setting of a meta-analysis of differential expression studies, with  <dig> genes and  <dig> studies.
out of the  <dig> genes,  <dig> genes are assumed to be not differentially expressed in any study, while  <dig> genes are assumed to be differentially expressed in  <dig> ,⋯, <dig> studies respectively.
the sample sizes for the treatment and control groups are randomly generated for each study, varying from  <dig> to  <dig>  gene expression values are randomly generated from normal distributions.
control samples are generated from a n distribution, as well as treatment samples that are not differentially expressed.
treatment samples that are differentially expressed are generated from a n distribution.
two-sample t-tests are used to obtain the p-values p
g
k
for each gene and each study.
our wop methods aim at testing the hypothesis that the gene is differentially expressed in the majority of studies.
in this case, m= <dig>  corresponding to the hypothesis setting hs <dig>  the rop statistic song and tseng  <cit>  for testing hs <dig> is the 4th ordered p-value.
note that the original fisher’s and stouffer’s method are supposed to test for hs <dig>  we used permutation analysis to obtain the wop p-values for binomial and half-binomial weighted fisher’s and stouffer’s statistic.
p-values for the rop method were also computed by permutation analysis as recommended in song and tseng  <cit> .
p-values for the original fisher’s and stouffer’s method are computed directly via their respective distributions.
to obtain a list of significant genes, the benjamini-hochberg procedure is applied to the p-values with the fdr controlled at the  <dig>  level.
results are averaged over  <dig> replications.
to better look at the trade off between rejection rates for categories  <dig> to  <dig> versus categories  <dig> to  <dig>  we plotted the roc curves for the methods, as seen in figure  <dig>  since our goal is to test for hs <dig>  we treat rejections in categories  <dig> to  <dig> to be false positives, while rejections in categories  <dig> to  <dig> are considered true positives.
from the roc curves we can clearly see that in terms of the trade off between true and false positives the binomial weighted wop methods beat the original fisher’s or stouffer’s method, while the half-binomial weighted wop methods beat the rop method.
we also gain better insight into the comparison between the binomial and half-binomial weighting schemes.
the half-binomial weighting scheme is relatively more conservative, with relatively higher true positive rates at very low false positive rates.
on the other hand, the binomial weighting scheme achieves higher power when slightly higher false positive rates are allowed.
in summary, the binomial weighted wop methods show improvement over the original fisher’s or stouffer’s method for testing differential expression in a majority of studies, with lower rejection rates for genes that are differentially expressed in a small number of studies, and just as high power for genes that are differentially expressed in almost all studies.
on the other hand, the half-binomial weighted wop methods are more robust versions of the rop method, having similar properties to the rop method, but even lower rejections rates for categories  <dig> to  <dig> and higher power for categories  <dig> and  <dig>  in practice, the binomial weighting scheme is recommended if the user wishes to have a larger pool of significant genes, and when the control of false positives is relatively less important.
for better false positive control, the half-binomial weighting scheme is recommended.
we note that because of our hypothesis setup, false positives are not the same as type i error.
a type i error would be rejecting a gene that is not differentially expressed in any studies.
on the other hand, a false positive would be rejecting a gene that is differentially expressed in less than r studies.
an application to meta-analysis of a set of stem cell studies
as an application of the proposed methodology, we conduct meta-analysis on a set of microarray data studies from four stem cell papers: chin et al.
<cit> , guenther et al.
<cit> , newman and cooper  <cit>  and chin et al.
<cit> .
we wish to find probesets that are differentially expressed between human induced pluripotent stem  cells and human embryonic stem  cells in the majority of studies.
some of the studies contain other samples such as human fibroblasts, but we only used samples from hips cells and hes cells.
we included studies that had at least two samples for each group , giving us a total of  <dig> studies.
all the studies used the affymetrix human genome u <dig> plus  <dig>  array platform, which contains  <dig> probesets.
we directly used the data preprocessed by the original contributors and did not perform any additional normalization, except for taking the log for data that were not already on the log <dig> scale.
we performed a two sample t-test between the hips cell samples and the hes cell samples to obtain the original p-values for differential expression for each probeset and each study.
the hypothesis setting for the meta-analysis is hs <dig>  i.e.
we aim at testing the alternative that the probeset is differentially expressed in at least  <dig> out of the  <dig> studies.
we applied the proposed wop methods, in particular the binomial and half-binomial weighted fisher’s and stouffer’s statistic to the p-values.
the p-values for the wop statistics are obtained by comparing the statistics to the corresponding numerical distributions.
we also applied the original fisher’s and stouffer’s method, and the rop method .
the p-values for the rop method are obtained via its theoretical distribution.
to adjust for multiple testing, we applied the benjamini-hochberg  procedure afterwards, controlling the false discovery rate at the  <dig>  level.
the number of probesets significant for the meta-analysis using the different methods are summarized in table  <dig>  the original fisher’s and stouffer’s methods found the most number of significant probesets.
it is likely that they picked out many probesets that are differentially expressed in a few, but less than half the studies.
the half-binomial weighted wop methods and the rop method are relatively more conservative and focused.
figure  <dig> shows a venn diagram of the probesets found significant by the binomial and half-binomial weighted fisher’s method and the rop method.
we can see that the probesets detected by both the half-binomial weighted fisher’s method and the rop method are mostly detected by the binomial weighted fisher’s method as well.
however, there are still a number of unique probesets that are only detected by either the half-binomial weighted fisher’s method or the rop method.
to get an idea of the types of probesets detected by only one of the three aforementioned methods, we randomly selected some of these probesets and plotted the ordered original p-values from the  <dig> studies for these probesets.
as seen in figure  <dig>  probesets exclusively detected by the binomial weighted fisher’s method tend to have very small values for the two or three smallest p-values, but relatively larger values starting the 5th ordered p-value.
this shows that the binomial weighted fisher’s method is more prone to influences by the smallest p-values, since it takes into account all the p-values in the statistics.
the rop method, which uses the 5th ordered p-value as the statistic, exclusively identifies probesets that are guaranteed to have relatively small p-values up to the 5th ordered p-value, but tend to have very large values starting the 6th ordered p-value.
this shows the sensitivity of the rop method to the particular value of r chosen.
on the other hand, the half-binomial weighted fisher’s method weights in the 5th through the 9th ordered p-values, and thus is able to identify probesets that have relatively small values through larger ordered p-values.
in other words, probesets that have relatively small p-values for most of the studies can be exclusively identified by the half-binomial weighted method, even if the smallest p-values are not very small.
to look at the pathways associated with the significant lists of probesets, we performed functional annotation clustering analysis using david , which is available at http://david.abcc.ncifcrf.gov/home.jsp.
some of the top functions that show up include metal-binding, nucleoplasm, ubl conjugation, vasculature development and head/face development.
we also looked at the pathways for the probesets that were exclusively detected by one of the methods.
functions such as neuron projection, neuron differentiation and development, which are meaningful in the stem cell study setting, were found to be associated with the probesets exclusively identified by the half-binomial weighted fisher’s method.
these functions did not show up in pathway analyses of the other lists.
further applications and comparisons with the rop method
to compare the performances of the wop methods and the rop method song and tseng  <cit>  in real data application, we applied our wop methods to the three microarray meta-analysis applications in song and tseng  <cit> .
the first application consists of comparisons of two subtypes of brain tumors - anaplastic astrocytoma  and glioblastoma multiforme , from  <dig> studies.
the second application combines  <dig> studies comparing post-mortem brain tissues between mdd patients and control samples.
in the third application,  <dig> diabetes microarray studies consisting of different organisms and tissues were combined.
see song and tseng  <cit>  for more details on the contexts of these three meta-analysis applications.
to ensure that the results are directly comparable, for each meta-analysis we directly used the two-sided p-values for each gene and each individual study calculated in song and tseng  <cit> .
in song and tseng  <cit> , permutation analysis is used for the brain cancer studies and the mdd studies, while theoretical distributions are used to obtain results for the diabetes studies.
we follow song and tseng  <cit>  and also directly use the two-sided p-values for the permuted datasets provided by song and tseng  <cit>  for the brain cancer studies and the mdd studies.
see song and tseng  <cit>  for more details on the preprocessing of the data and the calculation of the original p-values.
we applied our wop methods, namely the binomial and half-binomial weighted fisher’s and stouffer’s statistic, to the three sets of studies.
for the brain cancer studies, we test for hs <dig> out of  <dig> studies.
for the mdd studies we test for hs <dig> out of  <dig> studies.
for the diabetes studies, we test for hs <dig> out of  <dig> studies.
we also applied the corresponding rop methods for testing the same hypotheses, using r=m= <dig>   <dig> and  <dig> respectively for the three meta-analyses.
in addition, we applied the rop methods using the selected r values in song and tseng  <cit>  for comparison.
to be specific, song and tseng  <cit>  used r= <dig> for the brain cancer studies, r= <dig> for the mdd studies, and r= <dig> for the diabetes studies.
permutation analysis is used for the brain cancer studies and the mdd studies for all methods.
for the diabetes studies, we used our numerically simulated distribution to obtain the p-values for the wop statistics.
table  <dig> shows the numbers of significant genes found using the different methods for the three meta-analyses with the fdr controlled at the  <dig>  level.
we can observe that the wop methods using the binomial weighting scheme generally detects more significant genes than the half-binomial weighting scheme.
in most cases, the rop methods  detect less genes than the wop methods, although in general closer in number to the wop methods using the half-binomial weighting scheme.
r
=
m
*for the rop method, “selected r” refers to the choice of r in song and tseng  <cit>  for a particular meta-analysis.
one interesting observation is that for the mdd studies, the rop method based on r=m= <dig> detects less genes than based on the selected r= <dig>  this result is counterintuitive, since one would expect that genes that are differentially expressed in at least  <dig> studies would be a subset of the genes that are differentially expressed in at least  <dig> studies.
the result could be due to the fact that r is selected in song and tseng  <cit>  to optimize the number of significant genes and therefore outperforms a general choice of r=m.
nonetheless, this reflects the fact that the rop method is sensitive to the choice of r. to further investigate this problem, we looked at the overlap of the detected genes by the rop methods using either r=m or selected r, as well as with the detected genes by the half-binomial weighted fisher’s method.
see figure  <dig> for a venn diagram of the genes detected by the aforementioned three methods for the mdd studies.
as shown in figure  <dig>  only  <dig> genes overlap between the rop methods with two different choices of r, which is less than half of the genes detected by either method.
however, all  <dig> genes are detected by the half-binomial weighted fisher’s method.
in addition, the wop method also picked up  <dig> of the genes only detected by rop based on selected r and  <dig> of the genes only detected by rop based on r=m, which accounts for most of the genes detected by either method.
further, we noticed that even for the brain cancer studies and the diabetes studies, where rop based on r=m did detect more genes than rop based on selected r, there are still a large number of genes detected by rop based on selected r that are not detected by rop based on r=m.
on the other hand, most of these genes that are detected by only one of the rop methods are detected by the half-binomial weighted fisher’s method.
for comparison, to see how the results would differ for different choices of r for the wop methods, we applied the wop method to the mdd studies again - this time testing hs <dig>  we used the half-binomial versions of the three weighting schemes for m<r≤k that were discussed earlier: w
h
b <dig>  w
h
b <dig> and w
h
b <dig>  we used fisher’s summary statistic in the wop method.
the numbers of genes found by using the three weighting schemes are  <dig>   <dig> and  <dig> respectively.
the three weighting schemes are fairly consistent with each other, since for each weighting scheme more than 90% of the genes found were also found by the other two weighting schemes.
comparing to the number of genes found by the half-binomial weighted fisher’s method for testing hs <dig>  which is  <dig>  notice that the corresponding wop methods for testing hs <dig> yield smaller numbers, conforming to our expectations.
as discussed earlier, the ideal result would be that the genes detected for hs <dig> be a subset of the genes detected for hs <dig>  in reality, over 70% of the genes detected by wop methods for hs <dig> were also detected for hs <dig> .
recall that for the rop method, only  <dig> % of genes detected for r= <dig> were also detected for r= <dig>  even though the wop methods are not perfect, we can still see the great improvement in robustness compared to the rop method.
in summary, our observations confirm that the results of the rop method are indeed heavily dependent on the choice of r. whereas our wop methods show much higher robustness.
in particular, the wop methods for testing hs
m
has shown superior robustness by being able to cover most of the genes detected by the rop methods using different r. since in practice it is not often clear which particular hs
r
should be tested, we believe the wop methods for testing hs
m
is a better choice when the goal is to detect signal in the majority of studies.
CONCLUSIONS
meta-analysis is a useful tool in integrating data from different sources to test a particular hypothesis.
while this paper mainly discussed the application of meta-analysis on microarray differential expression studies, other areas of genomic studies have increasingly relied on the use of meta-analysis, such as genome-wide association studies .
some seminal studies in this area include scott et al.
<cit>  and willer et al.
<cit> .
meta-analysis is also frequently used in clinical studies, psychological studies and statistical applications in other social sciences.
more and more meta-analyses nowadays aim at detecting consistent findings across a number of studies.
while most of the traditional meta-analysis methods test for significance in at least one of the studies, it is important to develop new meta-analysis methods that focus on testing for significance in the majority of studies.
the weighted ordered p-value  method provides such a framework.
it is unique in its use of weights that are based on the order of the p-values.
the rop method song and tseng  <cit> , which is also based on ordered p-values, can be considered a very special case under the wop framework, where all the weight is placed on one single ordered p-value.
the wop methods do not require pre-specification of r and is less sensitive to the choice of its value.
the half-binomial weighted wop methods have been shown to be more robust and have better receiver operating characteristics compared to the corresponding rop method.
as pointed out by a reviewer, one of our previously published meta-analysis methods also considers the issue of heterogeneity and utilizes weights.
however, these two methods are quite different in concept.
the method in  <cit>  applies weights to the genes, essentially to re-rank the genes by adding in information about heterogeneity.
on the other hand, the wop method applies weights to the multiple p-values for each gene.
although, conceptually, we can first apply the wop method to each gene and then use the method in  <cit>  on top of that to re-rank the genes.
one advantage of the wop framework is its flexibility.
the framework allows for different weighting schemes and summary statistics to be used.
even though this paper mainly focused on two particular weighting schemes based on the binomial distribution and two summary statistics , in general, other summary statistics and weighting schemes can be used.
future research can be done to try to optimize the weighting scheme to suit specific meta-analysis purposes.
competing interests
the authors declare that they have no competing interests.
authors’ contributions
yl developed the method’s framework, carried out the simulation studies and data analysis, and drafted the manuscript.
dg conceived of the weighting schemes, supervised the research process and revised the manuscript.
all authors read and approved the final manuscript.

 meta-analysis has long been used to integrate data and/or results from multiple studies targeting the same questions. it is commonly used in many areas of statistical applications such as clinical studies and psychology experiments. in recent years, meta-analysis has been frequently adopted in genomic data analysis, due to the fast development of high-throughput technology and the vast amounts of data available in public databases. many meta-analysis methods have been developed throughout the years. roughly speaking, there are two main approaches to meta-analysis methods song and tseng. the first directly combines the p-values from the studies, while the second attempts to model the data or the effect sizes from the combined studies. the former includes methods such as the fisher’s combined probability test and the stouffer’s z-test, as well as weighted variations of these classical tests. the latter includes a variety of fixed effects and random effects models, such as genemeta. each approach has its advantages and disadvantages. the p-value combining methods are relatively flexible in that they require minimal information and assumptions from the studies. in this paper we will mainly focus on methods that directly combine the p-values. most of the traditional meta-analysis methods aim at testing the alternative hypothesis that at least one of the studies is non-null. while this aligns with earlier goals of meta-analyses to gain power in detecting signals by combining multiple studies, it is frequently not the case with genomic data. in meta-analysis of genomic studies for example, the goal is often to identify genes that are differentially expressed in a consistent pattern across multiple studies. the extreme of this would be to test for the alternative that the null can be rejected for all the studies. a solution to this extreme alternative dates back to the maxp method by wilkinson. however, the maxp method is often considered too conservative. a recent approach by phillips and ghosh improves the power of testing for this disjunction of nulls when the rejection of all p-values associated with a gene is required. in practical meta-analyses, the goal of rejecting all studies may be considered too extreme. ideally, we would want to target at detecting consistent signals across studies while avoiding being overly exclusive. this issue has gained attention in recent years, and a number of authors have tried to address this problem. benjamini and heller discussed a framework for testing partial conjunction hypotheses, where they test for the alternative that at least u out of n null hypotheses are false against the partial conjunction null that no more than n−u+ of the null hypotheses are true. song and tseng proposed the rth ordered p-value method that aims at testing the alternative hypothesis that there is signal in at least a given percentage of studies. other methods exist that address this problem from different approaches, such as rankprod by hong et al. that looks for consistently highly ranked genes, and a weighted approach by li and ghosh that weights genes by its expression consistency across studies. we consider the problem of detecting signals in the majority of studies. our approach adopts one aspect of the rop method song and tseng in that we also consider ordered p-values. but instead of using a single rth ordered p-value as the statistic, we combine all or a subset of the ordered p-values while weighting them based on their order. p-values closer to the median are highly weighted and the smallest/largest p-values are down-weighted. the idea is that among the collection of p-values, the median p-values are likely to be a better reflection of the behavior of the majority of studies than the smallest or largest p-values. olkin and saner discussed a trimmed fisher’s procedure that leaves out a number of the smallest and/or largest p-values from the calculation of fisher’s statistic to remove the effect of possible aberrant extremes. in our consideration, we still keep the smallest/largest p-values because they do carry certain information, but we down-weight them because they may be relatively less relevant when considering the “majority” of studies. to reflect the up-weighting of the medians and down-weighting of the extremes, we calculated our weights based on the binomial distribution. to summarize the weighted ordered p-values, we mainly considered fisher’s statistic and stouffer’s z-test. we also explored some other statistics, such as generalized fisher’s test by lancaster. in general, other summary statistics can be used under this framework as well. while many weighted variations of fisher’s statistic and stouffer’s statistic have been developed throughout the years, most of them distribute weights according to the sample sizes and/or effect sizes of the studies, or other similar considerations. li and tseng proposed an interesting adaptively weighted statistic, where the weights are used to maximize the significance of the summary statistic. however, to the best of our knowledge, none of the weighting schemes are based on the ordered p-values, which is what makes our method unique. xie et al. discussed a meta-analysis approach using confidence distributions, where they incorporated the use of medians and kernel functions, but their approach is under a completely different framework from ours. by incorporating more than one order statistic into our summary statistic, our method can be considered an expansion of the rth ordered p-value method song and tseng. in general, both the rop method and the original summary statistic we use are special cases under the wop framework - one having all the weight on a single ordered p-value and the other having evenly distributed weights. however, our wop methods have respective advantages over both the traditional summary statistics and the rop method. compared to the traditional summary statistics, the wop methods better focus on detecting signal in a majority of studies. on the other hand, the wop methods appear to be more robust compared to the rop method. these observations are based upon results from simulation studies as well as data applications. methods hypothesis settings for meta-analysis before performing any meta-analysis, it is always important to figure out the goal of combining multiple studies. when a single hypothesis test is conducted, it is clear what the null and alternative hypotheses are. in meta-analysis we usually combine studies designed to test the same set of null and alternative hypotheses. however, the null and alternative hypotheses of the meta-analysis test are not always obvious and largely depends on the researcher’s goals. here we consider the example of meta-analysis of differential expression studies of genomic data. sometimes a gene is of interest as long as it is differentially expressed in at least one study, while other times we hope to target genes that are differentially expressed in all studies. li and tseng and song and tseng both provided extensive discussions on the different scenarios that lead to different hypothesis settings. following their notation, let θ g k be the true effect size for gene g in study k. as in song and tseng, for a given 1≤r≤k, a general hypothesis setting for the meta-analysis of the k studies for a given gene g can be formulated as: hsr:h0:∑k=1ki=0versusha:∑k=1ki≥r. when r= hs is the classical setting of testing for non-zero effect size in at least one study against the conjunction of nulls. hs is the hypothesis setting that fisher’s method, stouffer’s z-test and many other traditional methods test for. when r=k, hs k tests for the alternative that all the studies have non-zero effect size. for instance, the maxp method tests for hs k. when 1<r<k,hs r provides a compromise between the two aforementioned hypothesis settings, and tests for at least a pre-specified number of non-zero effects. for a given r, the rop method song and tseng is used to test for hs r. in this paper, we test for non-zero effect sizes in a majority of studies against the null that the effects sizes are zero in all studies. thus we are testing against the conjunction of nulls while trying to focus on a certain subset of the non-null space. in general, the hypothesis for our meta-analysis approach can be considered to fall under the hs r setting. song and tseng suggested a few data-driven methods for selecting r. to prevent any potential issues of post hoc choices of r, we choose to fix r before any analysis is conducted. while it is hard to specify what the term “majority” exactly means, as a general rule, we choose to test the hypothesis setting hs m, where m=⌈k/2⌉, ⌈x⌉ being the smallest integer no less than x. essentially we are targeting the alternative that at least half of the studies have non-zero effect sizes. under our weighted ordered p-values framework, we are able to develop methods for other hypothesis settings with r ranging from m+ to k. but in general we hope to provide a simple to use method without having to put too much effort in selecting a particular r, and therefore we will focus mostly on testing hs m. we will later show through data applications that using our wop methods for testing hs m provides more robust results compared to using the rop method with different choices of r. a framework for weighted ordered p-values methods we first describe the general framework for the weighted ordered p-values methods. suppose we have p-values p p ⋯,p k for testing the hypothesis of interest for each of the k studies. let p,p,⋯,p be the ordered p-values. now consider a set of weights w w ⋯,w k associated with the corresponding ordered p-values. summary statistics of weighted ordered p-values can be expressed in the following general form: t=∑i=1kwih). as mentioned by previous authors, many traditional p-value combination methods can be expressed in the general form of t′=∑i=1kwi′h. for instance, stouffer’s z-test takes h to be the inverse normal function, while fisher’s method has h=− log. the difference between t′ and t, albeit subtle in notation, is the essence of the wop framework. in the wop framework, the weight w i is associated with the ith ordered p-value p. in other words, the ranking of a p-value in relation to the p-values from the other studies determines its weighting. in traditional weighted p-value combining methods, the weight assigned to a p-value is associated with the characteristics of that particular study, be it the sample size, the effect size, or other features. allowing the weights to depend on the ordering of the p-values opens up a whole new area of considerations when combining multiple p-values. we can consider giving more weights to the p-values that are closer to the center of the distribution of the p-values since they might hold more credibility, and at the same time down-weight the outlying p-values. for instance, if the entirety of weights is placed on the rth ordered p-value, the statistic reduces to the rop method. however, using the wop framework, if we highly weight the rth ordered p-value but still distribute some weight to the other p-values, the method can be viewed as a more robust version of the rop method for testing hs r. in this paper, we shall develop a few specific methods under the wop framework. we consider weights based on the binomial distribution, which will be described in more detail in the next section. as for the p-value combining methods, we shall focus on fisher’s method, with h f )=− log), and stouffer’s method, with h z )=ϕ−1), ϕ being the standard normal distribution function. binomial weights and half-binomial weights in this section, we discuss two possible weighting schemes for the wop framework. we mainly consider testing the alternative hypothesis that the effect sizes are non-zero in the majority of studies, which is the hypothesis setting hs m. we will also briefly discuss extending the weighting schemes to testing other hypothesis settings hs r, for m<r≤k. inspired by the rop method, which uses the rth ordered p-value for testing the hypothesis setting hs r, we consider placing the highest weight on the median p-values for testing hs m. this makes intuitive sense, since if a consensus does exist among the studies, we have reason to believe that the behavior of the majority of studies should be best captured by the p-values that are closer to the center of the distribution. since we do not insist on non-zero effect sizes for every single study, we consider down-weighting the largest p-values among the studies. on the other hand, p-value combining methods such as fisher’s method are known to be very sensitive to single extremely small p-values, thus the smallest p-values should also be down-weighted, to avoid a small number of extremely small p-values biasing the results of the majority of studies. in summary, we would like our weighting scheme w i, as a function of i, to reflect a unimodal shape, with the highest weights being w m or w m− and w m, and such that w i decreases as i goes to or k. to reflect the above properties of the weights, we constructed the weights based on the binomial distribution. let f be the probability mass function of the binomial distribution b, for x=,⋯,n. we define the binomial weighting scheme such that wib=f,i=,⋯,k. in the binomial weighting scheme, all the weights are non-zero, thus every p-value contributes to the combined statistic. to further reduce the influence of the smallest p-values on the summary statistic, we may argue that only p,p,⋯,p matters in testing the alternative that at least m studies have non-zero effect size. with these considerations, we define what we call the half-binomial weighting scheme such that wihb= i=,⋯,m− wib,i=m,m+ ⋯,k. we will discuss more on the effects of these two different weighting schemes through simulation studies in later sections. so far we have constructed the binomial weighting scheme and half-binomial weighting scheme for testing the hypothesis setting hs m. we can extend the ideas of these two weighting schemes to testing hs r, for m<r≤k. instead of placing the highest weights on the medians, the highest weight can now be assigned to p. when r≠m, we lose the natural symmetry of the weights. however, we can still base the weights on the binomial distribution. a few possible weighting schemes are defined below: wib1= i= ⋯,r−mwi−b,i=r−m+ ⋯,k. wib2=f−.5),i= ⋯,k. wib3= i= ⋯,2f−1;k−2−.5),i=2+ ⋯,k. the weighting scheme w b is based on the same binomial distribution as w b, except that the values are shifted so that the center of the distribution falls on r instead of m. because of the shift, the first few weights are set to be while the last few values in the probability mass function of the binomial distribution are truncated. the weighting scheme w b increases the parameter n to k+2− in the binomial distribution to ensure that the first few weights are non-zero. the weighting scheme w b decreases the parameter n to k−2− in the binomial distribution so that the distribution is not truncated on the right. see figure for an illustration of the idea of these three weighting schemes. corresponding half-binomial weights for these three binomial weighting schemes: w h b w h b and w h b can be easily constructed by setting the weights to be for i= ⋯,r− implementation of the wop methods under the null hypothesis, the p k ’s are assumed to follow a uniform distribution. for stouffer’s z-test, h z =ϕ− follows a standard normal distribution. therefore the traditional weighted z-test in the form of ∑i=1kwi′hz still follows a normal distribution. for fisher’s method, h f =− log has a chi-square distribution with two degrees of freedom. therefore the distribution of the traditional weighted fisher’s test ∑i=1kwi′hf is essentially weighted sums of exponential distributions. the distribution of weighted sums of exponential variables is not as straightforward, though many authors have researched on both the exact and approximations of this distribution, a summary of which can be found in olkin and saner. when we consider weighted ordered p-values in the form of ∑i=1kwih), however, the problem becomes much more complicated. even for stouffer’s method, the distribution of the sum of weighted ordered normal variables is not readily available. as for fisher’s method, olkin and saner studied the distribution of the trimmed fisher’s statistic, which is ∑i=1k−2wilog) for the special case of w i = for i= ⋯,s and i=s ⋯,k. they transformed the distribution of the sum of ordered chi-squared variables back to a weighted sum of exponential variables without order. but as discussed in their paper, the exact distribution of weighted sums of exponential variables are generally impractical for practitioners, and we would therefore have to use approximation methods. considering the complexity of the exact distributions of weighted sums of ordered variables, as well as the fact that the uniformness of the original p-values is not always guaranteed in practice, we recommend two methods for obtaining the p-values for the wop statistics: permutation analysis, in the case that original data from all the studies are available; and comparing to the numerical distribution, in the case that only the p-values for each gene and each study are known. we first explain the steps of obtaining the wop p-values through permutation analysis: •let tg=∑i=1kwih) denote the wop statistic for gene g, where p g is the ith ordered p-value of p g ⋯,p g k. •permute group labels in each study b times, and recalculate the p-values for the permuted data pgk, for 1≤k≤k, 1≤g≤g, 1≤b≤b. •calculate the wop statistics for the permuted p-values tg for 1≤g≤g, 1≤b≤b. •the p-value for the wop statistic t g is then computed as pgt=∑b=1b∑g′=1gi{tg≥tg′}b·g. once the p-values for the wop statistics for each gene are obtained, we may apply the benjamini-hochberg method on the pgt’s to account for multiple testing across the genes and control the false discovery rate. if the original data are not available, we can simulate the distribution of the wop statistics numerically, by simulating u random variables, the distribution of p-values under the null distribution. the wop statistics calculated from the data can then be compared to the numerical distribution to obtain the wop p-values. we simulated numerical distributions of the wop statistics for testing hs m based on the fisher’s and stouffer’s combination methods with binomial and half-binomial weighting schemes respectively, for study numbers ranging from to we conducted simulation studies to compare the wop p-values obtained either though comparing with the numerical distribution or by performing permutation analysis. results show that the two methods provide perfectly correlated p-values and that the number of rejections obtained from the two methods after applying the benjamini-hochberg adjustment are very similar. therefore both methods are reliable choices for obtaining the wop p-values in practice. the numerical distribution provides an option for when the original data are not available and is also more time-efficient. the permutation analysis can be used if the uniformness of the original p-values are questionable but that the original data is available. considerations of one- or two-sided tests previously we used two-sided alternatives as an example when setting up the hypothesis. the hypothesis setup hs r can be similarly developed for one-sided alternatives. in fact, the interpretation of the meta-analysis results is easier for one-sided tests, since we do not need to worry about the concordance of the directions of the effect sizes as we do for two-sided tests. since the wop methods directly combine the p-values, the direction of the effect sizes are not taken into account for two-sided tests. thus a significant result from the wop meta-analysis of two-sided tests indicate that there are at least r studies with non-zero effect size, but without any implications about the concordance or discordance of the directions of the effects. this may not be an issue in the case that the direction of effect is not of great importance. however, in genomic data analysis, it is often desirable to distinguish between up- and down-regulated genes, and a result stating that the gene is differentially expressed across many studies but with possible opposite directions of expression change may be confusing. in these cases, it might be problematic to directly apply the wop methods to the two-sided p-values. on the other hand, since both up- and down-regulated genes may be of interest at the same time, we cannot pre-specify one particular one-sided test for all genes. for such scenarios we recommend using the test of pearson in combination with the wop methods. to do so, for each gene we need to conduct two wop meta-analyses on one-sided p-values: one on the left-tailed p-values for all studies, and the other on the right-tailed p-values for all studies. let pwopl and pwopr be the wop meta-analysis p-values for the left-tailed and right-tailed tests respectively. we shall then adopt the idea of pearson’s test and define pwopc=min{ min}, where the superscript “c” stands for “concordant”. as discussed in owen, the equation for obtaining pwopc provides a conservative p-value for pearson’s test. by adopting the pearson’s test, the results are more interpretable. a significant result now indicates that the gene is consistently up- or down-regulated in at least r studies.  a simulation study we conducted a simulation study to compare the performances of the wop methods with the original fisher’s and stouffer’s method, as well as with the rop method song and tseng. we shall also demonstrate the differences between the binomial and half-binomial weighting schemes through the simulation. we simulate the setting of a meta-analysis of differential expression studies, with genes and studies. out of the genes, genes are assumed to be not differentially expressed in any study, while genes are assumed to be differentially expressed in,⋯, studies respectively. the sample sizes for the treatment and control groups are randomly generated for each study, varying from to gene expression values are randomly generated from normal distributions. control samples are generated from a n distribution, as well as treatment samples that are not differentially expressed. treatment samples that are differentially expressed are generated from a n distribution. two-sample t-tests are used to obtain the p-values p g k for each gene and each study. our wop methods aim at testing the hypothesis that the gene is differentially expressed in the majority of studies. in this case, m= corresponding to the hypothesis setting hs the rop statistic song and tseng for testing hs is the 4th ordered p-value. note that the original fisher’s and stouffer’s method are supposed to test for hs we used permutation analysis to obtain the wop p-values for binomial and half-binomial weighted fisher’s and stouffer’s statistic. p-values for the rop method were also computed by permutation analysis as recommended in song and tseng. p-values for the original fisher’s and stouffer’s method are computed directly via their respective distributions. to obtain a list of significant genes, the benjamini-hochberg procedure is applied to the p-values with the fdr controlled at the level. results are averaged over replications. to better look at the trade off between rejection rates for categories to versus categories to we plotted the roc curves for the methods, as seen in figure since our goal is to test for hs we treat rejections in categories to to be false positives, while rejections in categories to are considered true positives. from the roc curves we can clearly see that in terms of the trade off between true and false positives the binomial weighted wop methods beat the original fisher’s or stouffer’s method, while the half-binomial weighted wop methods beat the rop method. we also gain better insight into the comparison between the binomial and half-binomial weighting schemes. the half-binomial weighting scheme is relatively more conservative, with relatively higher true positive rates at very low false positive rates. on the other hand, the binomial weighting scheme achieves higher power when slightly higher false positive rates are allowed. in summary, the binomial weighted wop methods show improvement over the original fisher’s or stouffer’s method for testing differential expression in a majority of studies, with lower rejection rates for genes that are differentially expressed in a small number of studies, and just as high power for genes that are differentially expressed in almost all studies. on the other hand, the half-binomial weighted wop methods are more robust versions of the rop method, having similar properties to the rop method, but even lower rejections rates for categories to and higher power for categories and in practice, the binomial weighting scheme is recommended if the user wishes to have a larger pool of significant genes, and when the control of false positives is relatively less important. for better false positive control, the half-binomial weighting scheme is recommended. we note that because of our hypothesis setup, false positives are not the same as type i error. a type i error would be rejecting a gene that is not differentially expressed in any studies. on the other hand, a false positive would be rejecting a gene that is differentially expressed in less than r studies. an application to meta-analysis of a set of stem cell studies as an application of the proposed methodology, we conduct meta-analysis on a set of microarray data studies from four stem cell papers: chin et al., guenther et al., newman and cooper and chin et al.. we wish to find probesets that are differentially expressed between human induced pluripotent stem cells and human embryonic stem cells in the majority of studies. some of the studies contain other samples such as human fibroblasts, but we only used samples from hips cells and hes cells. we included studies that had at least two samples for each group, giving us a total of studies. all the studies used the affymetrix human genome u plus array platform, which contains probesets. we directly used the data preprocessed by the original contributors and did not perform any additional normalization, except for taking the log for data that were not already on the log scale. we performed a two sample t-test between the hips cell samples and the hes cell samples to obtain the original p-values for differential expression for each probeset and each study. the hypothesis setting for the meta-analysis is hs i.e. we aim at testing the alternative that the probeset is differentially expressed in at least out of the studies. we applied the proposed wop methods, in particular the binomial and half-binomial weighted fisher’s and stouffer’s statistic to the p-values. the p-values for the wop statistics are obtained by comparing the statistics to the corresponding numerical distributions. we also applied the original fisher’s and stouffer’s method, and the rop method. the p-values for the rop method are obtained via its theoretical distribution. to adjust for multiple testing, we applied the benjamini-hochberg procedure afterwards, controlling the false discovery rate at the level. the number of probesets significant for the meta-analysis using the different methods are summarized in table the original fisher’s and stouffer’s methods found the most number of significant probesets. it is likely that they picked out many probesets that are differentially expressed in a few, but less than half the studies. the half-binomial weighted wop methods and the rop method are relatively more conservative and focused. figure shows a venn diagram of the probesets found significant by the binomial and half-binomial weighted fisher’s method and the rop method. we can see that the probesets detected by both the half-binomial weighted fisher’s method and the rop method are mostly detected by the binomial weighted fisher’s method as well. however, there are still a number of unique probesets that are only detected by either the half-binomial weighted fisher’s method or the rop method. to get an idea of the types of probesets detected by only one of the three aforementioned methods, we randomly selected some of these probesets and plotted the ordered original p-values from the studies for these probesets. as seen in figure probesets exclusively detected by the binomial weighted fisher’s method tend to have very small values for the two or three smallest p-values, but relatively larger values starting the 5th ordered p-value. this shows that the binomial weighted fisher’s method is more prone to influences by the smallest p-values, since it takes into account all the p-values in the statistics. the rop method, which uses the 5th ordered p-value as the statistic, exclusively identifies probesets that are guaranteed to have relatively small p-values up to the 5th ordered p-value, but tend to have very large values starting the 6th ordered p-value. this shows the sensitivity of the rop method to the particular value of r chosen. on the other hand, the half-binomial weighted fisher’s method weights in the 5th through the 9th ordered p-values, and thus is able to identify probesets that have relatively small values through larger ordered p-values. in other words, probesets that have relatively small p-values for most of the studies can be exclusively identified by the half-binomial weighted method, even if the smallest p-values are not very small. to look at the pathways associated with the significant lists of probesets, we performed functional annotation clustering analysis using david, which is available at http://david.abcc.ncifcrf.gov/home.jsp. some of the top functions that show up include metal-binding, nucleoplasm, ubl conjugation, vasculature development and head/face development. we also looked at the pathways for the probesets that were exclusively detected by one of the methods. functions such as neuron projection, neuron differentiation and development, which are meaningful in the stem cell study setting, were found to be associated with the probesets exclusively identified by the half-binomial weighted fisher’s method. these functions did not show up in pathway analyses of the other lists. further applications and comparisons with the rop method to compare the performances of the wop methods and the rop method song and tseng in real data application, we applied our wop methods to the three microarray meta-analysis applications in song and tseng. the first application consists of comparisons of two subtypes of brain tumors - anaplastic astrocytoma and glioblastoma multiforme, from studies. the second application combines studies comparing post-mortem brain tissues between mdd patients and control samples. in the third application, diabetes microarray studies consisting of different organisms and tissues were combined. see song and tseng for more details on the contexts of these three meta-analysis applications. to ensure that the results are directly comparable, for each meta-analysis we directly used the two-sided p-values for each gene and each individual study calculated in song and tseng. in song and tseng, permutation analysis is used for the brain cancer studies and the mdd studies, while theoretical distributions are used to obtain results for the diabetes studies. we follow song and tseng and also directly use the two-sided p-values for the permuted datasets provided by song and tseng for the brain cancer studies and the mdd studies. see song and tseng for more details on the preprocessing of the data and the calculation of the original p-values. we applied our wop methods, namely the binomial and half-binomial weighted fisher’s and stouffer’s statistic, to the three sets of studies. for the brain cancer studies, we test for hs out of studies. for the mdd studies we test for hs out of studies. for the diabetes studies, we test for hs out of studies. we also applied the corresponding rop methods for testing the same hypotheses, using r=m= and respectively for the three meta-analyses. in addition, we applied the rop methods using the selected r values in song and tseng for comparison. to be specific, song and tseng used r= for the brain cancer studies, r= for the mdd studies, and r= for the diabetes studies. permutation analysis is used for the brain cancer studies and the mdd studies for all methods. for the diabetes studies, we used our numerically simulated distribution to obtain the p-values for the wop statistics. table shows the numbers of significant genes found using the different methods for the three meta-analyses with the fdr controlled at the level. we can observe that the wop methods using the binomial weighting scheme generally detects more significant genes than the half-binomial weighting scheme. in most cases, the rop methods detect less genes than the wop methods, although in general closer in number to the wop methods using the half-binomial weighting scheme. r = m *for the rop method, “selected r” refers to the choice of r in song and tseng for a particular meta-analysis. one interesting observation is that for the mdd studies, the rop method based on r=m= detects less genes than based on the selected r= this result is counterintuitive, since one would expect that genes that are differentially expressed in at least studies would be a subset of the genes that are differentially expressed in at least studies. the result could be due to the fact that r is selected in song and tseng to optimize the number of significant genes and therefore outperforms a general choice of r=m. nonetheless, this reflects the fact that the rop method is sensitive to the choice of r. to further investigate this problem, we looked at the overlap of the detected genes by the rop methods using either r=m or selected r, as well as with the detected genes by the half-binomial weighted fisher’s method. see figure for a venn diagram of the genes detected by the aforementioned three methods for the mdd studies. as shown in figure only genes overlap between the rop methods with two different choices of r, which is less than half of the genes detected by either method. however, all genes are detected by the half-binomial weighted fisher’s method. in addition, the wop method also picked up of the genes only detected by rop based on selected r and of the genes only detected by rop based on r=m, which accounts for most of the genes detected by either method. further, we noticed that even for the brain cancer studies and the diabetes studies, where rop based on r=m did detect more genes than rop based on selected r, there are still a large number of genes detected by rop based on selected r that are not detected by rop based on r=m. on the other hand, most of these genes that are detected by only one of the rop methods are detected by the half-binomial weighted fisher’s method. for comparison, to see how the results would differ for different choices of r for the wop methods, we applied the wop method to the mdd studies again - this time testing hs we used the half-binomial versions of the three weighting schemes for m<r≤k that were discussed earlier: w h b w h b and w h b we used fisher’s summary statistic in the wop method. the numbers of genes found by using the three weighting schemes are and respectively. the three weighting schemes are fairly consistent with each other, since for each weighting scheme more than 90% of the genes found were also found by the other two weighting schemes. comparing to the number of genes found by the half-binomial weighted fisher’s method for testing hs which is notice that the corresponding wop methods for testing hs yield smaller numbers, conforming to our expectations. as discussed earlier, the ideal result would be that the genes detected for hs be a subset of the genes detected for hs in reality, over 70% of the genes detected by wop methods for hs were also detected for hs. recall that for the rop method, only % of genes detected for r= were also detected for r= even though the wop methods are not perfect, we can still see the great improvement in robustness compared to the rop method. in summary, our observations confirm that the results of the rop method are indeed heavily dependent on the choice of r. whereas our wop methods show much higher robustness. in particular, the wop methods for testing hs m has shown superior robustness by being able to cover most of the genes detected by the rop methods using different r. since in practice it is not often clear which particular hs r should be tested, we believe the wop methods for testing hs m is a better choice when the goal is to detect signal in the majority of studies.  meta-analysis is a useful tool in integrating data from different sources to test a particular hypothesis. while this paper mainly discussed the application of meta-analysis on microarray differential expression studies, other areas of genomic studies have increasingly relied on the use of meta-analysis, such as genome-wide association studies. some seminal studies in this area include scott et al. and willer et al.. meta-analysis is also frequently used in clinical studies, psychological studies and statistical applications in other social sciences. more and more meta-analyses nowadays aim at detecting consistent findings across a number of studies. while most of the traditional meta-analysis methods test for significance in at least one of the studies, it is important to develop new meta-analysis methods that focus on testing for significance in the majority of studies. the weighted ordered p-value method provides such a framework. it is unique in its use of weights that are based on the order of the p-values. the rop method song and tseng, which is also based on ordered p-values, can be considered a very special case under the wop framework, where all the weight is placed on one single ordered p-value. the wop methods do not require pre-specification of r and is less sensitive to the choice of its value. the half-binomial weighted wop methods have been shown to be more robust and have better receiver operating characteristics compared to the corresponding rop method. as pointed out by a reviewer, one of our previously published meta-analysis methods also considers the issue of heterogeneity and utilizes weights. however, these two methods are quite different in concept. the method in applies weights to the genes, essentially to re-rank the genes by adding in information about heterogeneity. on the other hand, the wop method applies weights to the multiple p-values for each gene. although, conceptually, we can first apply the wop method to each gene and then use the method in on top of that to re-rank the genes. one advantage of the wop framework is its flexibility. the framework allows for different weighting schemes and summary statistics to be used. even though this paper mainly focused on two particular weighting schemes based on the binomial distribution and two summary statistics, in general, other summary statistics and weighting schemes can be used. future research can be done to try to optimize the weighting scheme to suit specific meta-analysis purposes. competing interests the authors declare that they have no competing interests. authors’ contributions yl developed the method’s framework, carried out the simulation studies and data analysis, and drafted the manuscript. dg conceived of the weighting schemes, supervised the research process and revised the manuscript. all authors read and approved the final manuscript. 
while many classical meta-analysis methods are developed with the former goal in mind, the latter goal has much more practicality for genomic data analysis. in this paper, we propose a class of meta-analysis methods based on summaries of weighted ordered p-values that aim at detecting significance in a majority of studies. meta-analysis has become increasingly popular in recent years, especially in genomic data analysis, due to the fast growth of available data and studies that target the same questions. fisher’s combined probability testmeta-analysisordered p-valuesweighted order statistic in particular, we consider weights based on the binomial distribution, where the median of the p-values are weighted highest and the outlying p-values are down-weighted. some aim at combining information to find signals in at least one of the studies, while others hope to find more consistent signals across the studies.
 meta-analysis has become increasingly popular in recent years, especially in genomic data analysis, due to the fast growth of available data and studies that target the same questions. many methods have been developed, including classical ones such as fisher’s combined probability test and stouffer’s z-test. however, not all meta-analyses have the same goal in mind. some aim at combining information to find signals in at least one of the studies, while others hope to find more consistent signals across the studies. while many classical meta-analysis methods are developed with the former goal in mind, the latter goal has much more practicality for genomic data analysis.  in this paper, we propose a class of meta-analysis methods based on summaries of weighted ordered p-values that aim at detecting significance in a majority of studies. we consider weighted versions of classical procedures such as fisher’s method and stouffer’s method where the weight for each p-value is based on its order among the studies. in particular, we consider weights based on the binomial distribution, where the median of the p-values are weighted highest and the outlying p-values are down-weighted. we investigate the properties of our methods and demonstrate their strengths through simulations studies, comparing to existing procedures. in addition, we illustrate application of the proposed methodology by several meta-analysis of gene expression data.  our proposed weighted ordered p-value methods displayed better performance compared to existing methods for testing the hypothesis that there is signal in the majority of studies. they also appeared to be much more robust in applications compared to the rth ordered p-value method:777–800). with the flexibility of incorporating different p-value combination methods and different weighting schemes, the weighted ordered p-values methods have great potential in detecting consistent signal in meta-analysis with heterogeneity. fisher’s combined probability testmeta-analysisordered p-valuesweighted order statistic 