Interpretation of P-values: 

Challenges for the Replication and 

Comparison of Statistical Results

Ilana Belitskaya-Lévy, Ph.D.

Cooperative Studies Program Coordinating Center

VA Palo Alto Health Care System 

VA Statisticians’ Association

October 29, 2013
Acknowledgments 

 Joint work with: 

 Laura Lazzeroni (Stanford University) and 

 Ying Lu (VA Palo Alto and Stanford University)

 Funding: 

 VA Palo Alto Cooperative Studies Program-Wide 

DNA Bank (CSP#478)

 NIH/NIMH R01 MH086135 and the Consortium on 

the Genetics of Schizophrenia (COGS)
Outline 

 Genome-wide association studies (GWAS)

 Genome-wide significance 

 Failure to replicate, winner’s curse

 Follow-up (“replication”) studies

 P-values

 Sampling variability of a p-value

 How likely is replication?

 How confident are we in the relative meaning of 

two different p-values?

 Implications and conclusions 
Poll Question #1

• What is your primary role in VA? 

– biostatistician or statistical programmer

– student, trainee, or fellow

– clinician

– researcher 

– manager or policy-maker 

– other 
Genome-wide significance

A single modern genomics study may include a million or 

more statistical tests.

To reduce the number of false positive results, very strict 

significance levels are used. 

Thus, significance levels on the order of 10-8 have 

replaced the traditional 0.05 value.

This talk examines the unintended and unappreciated 

consequences of this change.
Concerns regarding GWAS studies 

1. Failure to replicate initial findings

2. Even when replicated, strength of genetic 

association vary from study to study

Why?

Genetic heterogeneity? 

Winner’s curse? 

Publication bias?
Genomic “Replication” Studies

 Genetic studies often include an initial large study 

followed by a more targeted follow-up (“replication”) 

study within the same publication.

 Why? Initial results may be disappointing or ambiguous 

given stringent genome-wide testing criteria.

 Additionally, many journals (e.g. NEJM) require 

replication BEFORE they will publish a genetic 

association due to skepticism regarding genetic 

associations.
How Are Genetic Variants Selected for 

Replication? 

 Custom-designed genotyping chips (smaller & cheaper 

than the original) have a pre-designated amount of 

“real estate.”

 Selection criteria include:

 Small p-values (< 0.001 or <0.0001)

 Large estimated effect sizes (odds ratios)

 Previous reports of association

 Biologically relevant pathways

 Plausible function of the variant

 Or a combination of the above
Designing Follow-up or Replication 

Studies 

 Investigators often use an ad hoc approach to design 

follow-up studies.

Ignoring important factors such as the winner’s 

curse.

 In this talk, we provide explicit numerical results that 

show that even extremely small p-values provide 

surprisingly little information as to the likelihood of 

subsequent replication or the relative importance of 

underlying association.
Questions

1. What is the sampling variability of a p-value?

2. How likely is an independent study to replicate a 

given p-value?

3. How confident can we be that a more significant pvalue reflects a larger underlying effect than a less 

significant p-value?
The Key Assumption

We are using a test statistic, Z, with an asymptotic Normal 

distribution. In the usual notation,

 

Thus, results apply in most common statistical settings.

Genomics terminology is used for convenience: 

GWAS = genome-wide association studies

 SNP = single nucleotide polymorphisms
Question 1: 

What is the sampling variability of a p-value?

P-value is an informative measure of strength of 

statistical results 

P-values are also highly variable random statistics of 

the data, with sampling distributions of their own. 
Question 1: 

What is the sampling variability of a p-value?

 1,000 Simulated studies of the same SNP

 For each simulation study:

 2000 Sample size

 1.74 Odds ratio

 30% Minor allele frequency

 10% Major homozygote disease risk

 -log p (base 10) from a logistic regression analysis
Median p: 10-8 

Min. p: 5.6 10-21

Max. p: 0.015

In real study, we see 

one value of p, but do 

not know distribution or 

median relative to p 

___________________

Vertical line at p = 0.05 

Solid curve = null exponential 

density 

1000 simulated studies of same SNP
Does p-value estimate anything?

Yes. We define π or 

the π-value to be 

the value of p

when the 

estimate (i.e. test 

statistic) exactly 

equals true, 

unknown θ.
Does p-value estimate anything?

Yes. We define π or 

the π-value to be 

the value of p

when the 

estimate (i.e. test 

statistic) exactly 

equals true, 

unknown θ.
 Observe p1 in the initial study

 Calculate Z1 = Z(p1) 

 A 95% confidence interval for Z(π) is:

 The 95% confidence interval for π is:

Confidence interval for π
 95% confidence 

intervals apply to pvalues in any largesample context 

X-axis: -log p1 

Y-axis: -log π 

Confidence intervals 

extend vertically 

from top to bottom 

sides of the green 

funnel. 

Confidence Intervals for π
Question 2: 

How likely is replication?
 95% confidence 

intervals apply to pvalues in any largesample context 

X-axis: -log p1 

Y-axis: -log π 

Confidence intervals 

extend vertically from 

top to bottom sides of 

the green funnel. 

Results for Equal Study Sizes
 95% prediction 

intervals extend 

vertically from 

top to bottom of 

outer red 

funnel.

Y-axis: -log p2

predicted for 

replication.

Results for Equal Study Sizes
Unequal study sizes shift π & the predicted p

Here, the 

replication 

study is 1/4 the 

size of the 

original study.
Unequal study sizes shift π & the predicted p

Here, the 

replication study 

is 4 times the 

size of the 

original study.
Effect of accounting for multiple comparisons. 

1M tests in 

original study

1 test in 

replication study

Bonferroni 

correction for 1M 

tests in original 

study. 

No correction in 

replication study.
Meta-analysis

• Suppose a sample of size n1 has been analyzed. 

• Question: replicate in a sample of size n2 or augment 

the original sample and use a combined analysis?

• 95% prediction interval for combined Z: 

11 12

1 2

combined

nZ nZ

Z

n n

+

=

+
Replication vs Combined Analysis

Equal study 

sample sizes

Blue: prediction 

interval for 

Zcombined

Red: prediction 

interval for 

replication p-value
Question 3: How confident are we in the relative 

meaning of two different p-values?

Suppose we observe p-values for two different SNPs.

We ask “Is the effect of the more significant SNP really 

bigger than the effect of the less significant SNP?”

For example, which of the two SNPs to select for a 

follow-up study? 
SNP comparison

SNP 1: SNP2: 

Observe p-values p1 and p2 such that p1 < p2 

How confident are we that the estimated order is the 

“correct order”? 

We want to construct a test of θ θ 1 2

=

0 1

1

: 0

: 0 a

H

H

θ

θ

=

>

0 2

2

: 0

: 0 a

H

H

θ

θ

=

>
What to compare?

 H0: θ1 = θ2

E.g. 

• the log odds ratios 

• the π-values, π1 and π2 
Test Statistic

11 2

1 2

1

() ()

1

Z p rZ p T

r

− =

+

2 2

1

1 1

/

/

n

r

n

σ

σ

=

Ti is a N(0,1) test statistic of H0: θ1 = θ2

If T is positive => stronger effect of SNP1

If T is negative => stronger effect of SNP2
Evidence Ratio

Let q1 be p-value for a one-sided test with Ha: θ2 > θ1 

Let q2 be p-value for a one-sided test with Ha: θ1 > θ2 

Evidence Ratio

 q1/q2 in favor of SNP1 compared to SNP2 

Evidence ratio of 95:5 means that we can conclude θ1 > θ2 in 

a one-sided , 5% significance test of H0: θ1 = θ2. 

 
Evidence ratio in favor of θ1 vs θ2 

r = 1

Without accounting for multiple comparisons
Implications and conclusions

P-values vary greatly across studies, even when 

underlying effects, population and study design 

are identical.

Studies can have good power to reject the null 

hypothesis of no association, while providing 

little information with respect to the 

reproducibility or relative strength of the true 

association.

The more significant the p-value, the greater 

the variability.
Implications and conclusions

• Replication p-values can differ from an initially 

significant GWAS finding, especially after 

multiple testing correction.

• P-values provide little resolution to distinguish 

the true, relative importance of different SNPs.
Our results

 Support proposals of others to combine multiple lines of 

biological evidence in deciding which results to investigate 

further, rather than depending on association p-values 

alone. 

 Can be used to design successful replication studies.

 The prediction intervals depend on the relative size of 

the replication and initial study. 

 The prediction intervals require no assumptions about 

the unknown effect sizes or the initial sample size. 

 Apply in most statistical settings, with or without multiple 

testing issues.
We gratefully acknowledge support from:

 

The VA Palo Alto Cooperative Studies Program DNA Bank

NIH/NIMH R01 MH086135 and the

Consortium for the Genetics of Schizophrenia (COGS)
Additional slides
P-value variability 

In a real study, we see only one p-value and don’t know 

where the rest of the distribution lies relative to it.

p (unlogged) is most variable under the null hypothesis. 

log p is more variable under the alternative hypothesis.

Using log p instead of p affects our intuitive 

interpretation.
• Consider a one-sided lower-tail hypothesis test for the 

population parameter θ: 

 

• Assume without loss of generality that

• Given p-value, we can construct Z = Z(p) 

• Given Z, we can find p = p(Z) 

• Z(p) & p(Z) are functions mapping one to the other 

– P(Z) is the cumulative distribution function of the 

standard Normal distribution and Z(p) is its inverse.

0 0

0

:

: a

H

H

θ θ

θ θ

≥

<

θ0= 0

Confidence interval for π
Prediction interval for replication 

p-value

• Zi is 

• Consider which is Normal with 

expected value 0 and variance 

• Confidence interval for R is 

• Confidence interval for Z2 is

( / ,1) N ni

θ σ

2 2 11 R Z n nZ = − /

2 1 1 / + n n

2 1 ±− + Z nn (1 / 2) 1 / β
Coverage rate of the prediction interval

For each of the 1,000 p-values in Figure 1, we 

computed a separate 95% prediction interval without 

a multiple testing adjustment and calculated how 

many of the other 999 p-values it covered. The 

estimated coverage rate for our prediction interval 

procedure was 94.9%, consistent with theoretical 

results. The percentage of replication p-values 

covered varied across the distribution of initial pvalues, ranging from 17.3% for p = 0.015 to 99.6% for 

p = 10-8. It should be noted that these conditional 

coverage percentages would differ if the unknown 

effect size or sample size differed, even if the same 

initial p-value was observed. 
Two-sided p-values

If the p-values are two-sided, dividing the observed 

value of p1 yields the more significant of the two 

corresponding one-sided p-values. 

The prediction interval for a one-sided p-value is then 

calculated and converted back to two-sided p-values 

using the equation 

 ptwo-sided = 2 min(pone-sided, 1-pone-sided) 
What to compare?

1. We can compare E1 = θ1 to E2 = θ2

 E.g. to compare the log odds ratios (must know relative 

ratio of standard errors, and ) 

2. Or we can compare E1 = θ1/σ1 to E2 = θ2/σ2 

E.g. to compare R2s from a linear regression model (need 

only n1/n2) 

3. Or we can compare the π-values, π1 and π2 

1 1 σ / n 2 2 σ / n
Test statistics

1. To compare θ1 to θ2, 

 the test statistic is

 

2. To compare θ1/σ1 to 

 θ2/σ2, the test 

 statistic is

3. To compare π1 and π2, 

 the test statistic is 

 

11 2

1 2

1

() ()

1

Z p rZ p T

r

− =

+

2 2

1

1 1

/

/

n

r

n

σ

σ

=

1 2

3

() ()

2

Zp Zp T − =

12 2

2 2

2

() ()

1

Z p rZ p T

r

− =

+

1

2

2

n

r

n

=

3r =1
Evidence Ratio

• Ti is a N(0,1) test statistic of H0: E1=E2, where

 E1= θ1 or θ1/σ1 or 1-π1 

 E2= θ2 or θ2/σ2 or 1-π2

• If T is positive => stronger effect of SNP1

If T is negative => stronger effect of SNP2

• P-value q1 for a one-sided test with Ha: E2>E1 is p(T) 

P-value q2 for a one-sided test with Ha: E1>E2 is 1-q1 

• Evidence Ratio: q1/q2 in favor of SNP1 compared to 

SNP2
Evidence Ratio

Evidence Ratio

 q1/q2 in favor of SNP1 compared to SNP2

Evidence ratio of 95:5 means that we can conclude that 

E1>E2 in a one-sided , 5% significance test of H0: 

E1=E2. 

 
Evidence ratio in favor of E1 vs. E2 

2 2

1

1 1

/

/

n

r

n

σ

σ

=

1

2

2

n

r

n

=

3r =1