**Transcript of Cyberseminar**

**VA Statisticians' Association**

**Interpretation of P-values: Challenges for the Replication and Comparison of Statistical Results**

**Presenter: Ilana Belitskaya-Levy, PhD**

**October 29, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact Ilana Belitskaya-Lévy, Ph.D. at Ilana.Belitskaya-Levy@va.gov*

Moderator:...going to have Dr. Zhou introduce you. And he is the director and the research career scientist in the bio statistics unit at the HSR&D Center of Excellence and the President of the VA Statisticians Association. He is also a professor at the Department of Bio Statistics and an adjunct professor in the Department of Psychiatry and Behavior Sciences at the University of Washington. So Dr. Zhou is going to introduce this series and also our speaker for today. 

Dr. Zhou:Oh good. Thank you, Molly. This is the series for the VA Statisticians Association. Which is an organization that tries to promote the sound use of statistics and introduce the sound statistical practices and  methods in VA studies. Including HSR&D and also clinical R&D. Our members are including those HSR&D researchers and statistician and also the clinical R&D, particularly co-op study programs statisticians such as today's speakers. 

This is a series put together by the VASA VA Statisticians and we will have more coming up the next few months and the next year. We welcome suggestions from the audience about what type of statistic projects they would like to hear some of. I am very happy we have Ilana Levy give her interpretation of the P value challenges for the replications and comparison of statistical results. Dr. Levy has worked in a co-op study program coordinator center of the VA Palo Alto. And she has done many research in the GWAS study. We are very happy she was able to make time to talk to us. Dr. Levy? 

Dr. Belitskaya-Levy:Thank you. Thank you so much for the introduction and for inviting me to present. I will be talking the interpretation of P value and the challenges for the replication and comparison of statistical results. 

This is joint work with my colleague Laura Lazzeroni at Stanford University and Ying Lu at the Palo Alto VA and Stanford University. 

The outline of my talk; I will briefly introduce genome-wide association studies. Concepts of genome wide significance and talk about why genome wide association studies sometimes fail to replicate. And talk about replication and follow-up studies. We will then talk about P-values that are used as a measure of significance of statistical results. And we will talk about primary [inaud.] of p-value. We will discuss how likely it is to replicate a significant finding and how confident can we be in the relative meaning of two different p-values. And implications and conclusions. 

Before I present my talk I would like to know a little bit about who my audience is. Would you please vote as a biostatistician or statistical programmer - student, trainee or fellow - clinician - researcher - manager or policy-maker I would love to get an idea. We will hold off a few seconds. 

Moderator:Thank you very much. So for our attendees please click the circle next to the answer that best describes your primary position. It looks like the answers have stopped streaming in. Ilana, if you can see those feel free to talk through them real quick. 

Dr. Belitskaya-Levy:Okay. So about half of our audience is biostatistician or statistical programmer. About 6% are students, trainees or fellows. 6% are clinicians. About 25% are researchers. There are about 5% who are managers or policy makers. And others are 1%. Something like that. Thank you very much. 

 The concept that I will present today really are inspired by my work - our work in genome-wide association studies. They also apply in any statistical sense. Any research questions where a p-value is obtained all the concept I talked about can be apply there. 

So genome-wide association studies is a genetic study where we look at millions or more genetic variants and see if they are associated with a disease or research. What it means is that we are conducting a million or more statistical tests at once. And even if none of the genetic variants are associated with the disease you would still use the usual 5% significance level we might get a lot of false positives. So if you conduct a million statistical tests you will get 5% of them, 50,000 tests false positives. To review the number of false positives very strict significance levels are used in genome wide association studies. Significance levels on the order of 10-8 have replaced the traditional 5% significance value. Only a p-value the smallest 10 -8 you can be positive that they represent the significant finding. This talk will examine the unintended and unappreciated consequences of this change. 

Genome wide association studies have been around since 2005. And of course there are lots of challenges with them, right? We are testing a million statistical hypothesis at once. And in the past few years lots of the times there are findings on genome wide association studies that fail to replicate. And that can create concern. Even when findings have been replicated the trends of genetic association vary from initial studies to the replication of follow up studies. Complications include for example genetic heterogeneity. For example, initial studies, genome wide association studies, is conducted in a particular population of patients and the findings do not translate to the next study because it is a completely different population and genetics have something to do with it. 

It can also be due to winner's curse. And that is a phenomenon we often encounter in statistics and real life where whoever is the first person to report the significant finding suffers from the winner's curse. Because to replicate... the most significant findings are usually not replicated. They look a lot like significant in the follow-up. That is because a significant finding is a combination of luck, random luck and true signals. It could be due to either true association or luck or a combination of that. Therefore when we try to replicate the more significant findings we are more likely to elect significant association. 

Failure to replicate could also be due to publication bias because only positive findings are released to the publication. And therefore, negative findings - you will never see negative findings. 

Genetic studies often include an initial large study, genome wide association study, followed by a more targeted follow-up, replication study often within the same publication. This is because initial results may be disappointing or ambiguous given their stringent genome wide testing criteria. The significance level starts with a 10-8 not too many variants might be positive significant. 

Also, many journals such s the New England Journal of Medicine require replication before they will publish a finding. That is due to the skepticism over genetic associations. 

How are genetic variants selected for replication? Usually it is done using custom-designed genotyping chips that have a predesignated amount of real estate. Usually a 1,000 or 5,000 variants are usually genotyped on a chip. And therefore we have to select which of the findings we would like to try to replicate. And the criteria - the selection criteria includes small p-values. So the more significant findings from the original studies to try to replicate. Other criteria include large estimated effect sizes if you have a large odd ratio you might try to replicate that finding. Previous reports of association or biologically relevant pathways. Or variants with a clear function that is relevant. And usually a combination of the above criteria are used to select variants for replication. And then we can see this method requires ad hoc. The follow up studies are designed in an ad-hoc fashion ignoring important facts such as winner's curse because the smaller the more significant p-values are the ones who are most likely to suffer from winner's curse and the hardest to replicate. 

And in this talk we will provide explicit numerical results that show that even extremely small, the most significant p-values provide very little information as to how likely we are to replicate the results. And very little information about the relative important of underlying association. 

We will address the following questions about p-values that are used to select findings for replication. What is the sampling variability of a p-value? Second question. How likely are you to replicate your initial finding in an independent study of the same variant? And how confident can we be that a more significant p-value reflects a larger true underlying effect than a less significant p-value? 

All of the findings, all of these contents discussed here rely on a Z statistic which is the most difficult statistical test. And therefore they apply to any statistical setting. It can be used [inaud.] under any other study. They are quite general. Statements made here apply to general. And apply to any statistical test. And the following abbreviations, GWAS, genome wide association studies and SNP is one variant that is usually studied in genome wide association study. We study millions of tests in one study. 

What is the sampling variability of a p-value? Many of the biostatisticians and researchers will probably deal with p-values a lot and we know that it is an informative measure of strength of statistical results. But also p-values are a function of data, of the data you have at hand. They are in fact highly variable, random statistics of data that have sampling distribution of their own. 

I have done a small simulation studies to show the variability of a p-value. Here we have 1,000 studies, simulated studies of the same variant. In each study the sample size is 2,000 and the odds ratio for this variant is 1.74. And let's look at for each study it will give me a p-value. Let's look at the distribution of p-values on a smaller scale. P-values are very small due to the high odds ratio. Most of my p-values are going to be significant. Let's look at the distribution of 1,000 p-values, 1,000 simulation studies of the same variant. 

Here is the distribution of the 1,000 p-value. You can see all of them are quite significant. In fact, the smaller p-value here is 10-21 it is very significant. And the largest p-value is about .015. With a median p-value being 10-8. What it means is if you conduct in real life you conduct only one study and you will see only one p-value here. And you don't know the distribution or median relative to the p-value that you actually have. 

Our next question is does p-value estimate anything? We define - p-value estimates the true unknown p-value of the variant that you are studying. We define pi or the pi value to be the value of p when the estimate the test statistics is exactly equal to true unknown parameter. If this is the true data then p-value is the true p-value, the pi value. Of course in real life we don't know the pi value and what we see is an estimate of it which is the p-value which corresponds to the observed test statistic. 

The next thing is we derive the confidence interval for the true p-value. So we observe a p-value in the initial study. We can calculate the first finding in this statistic. It is corresponding the p-value in z statistic. Then a 95% confidence interval for the z statistic is written out here and from that you can get a 95% confidence interval for the true p-value. And let's look at the picture here. So on the X axis you have the observed p-value in your study on the log scale. And on the Y axis you have the true unknown p-value. So the confidence interval are vertical lines, green lines within the green funnel. With the median here. So what you see is that if the p-value that you observe is 10-6 right here our 95% confidence in the p-value is between 10-3 to 10-11. That is how variable p-values are. 

If you repeat the same experiment you might get a p-value between 10-3 to 10-11 while we all we observe is one p-value of 10-6. 

We then looked at the next session how likely is it to replicate a finding. What we did is we constructed prediction intervals for the p-value in the replication given the p-value in the initial study. So if n1 and n2 are the original and replication sample sizes here the prediction interval for z2 in the replication study. This can be converted to make a prediction interval for the p-value in the replication study. 

Let's look at the figure. So these are confidence intervals and now we add prediction intervals. The red intervals are 95% prediction intervals for the p-value in the replication studies. The p-value in the initial study. You can see how it has more variability. So if I observe a very significant p-value of 10-9 in the replication study you might observe anywhere from 10-3 to a very, very small p-value. 

In the previous figure the original and replication studies had the same size, the same sample size. Let's look at what happens if the replication study is a quarter of the size of the original study. So if the replication is quarter of the size of the original then the prediction intervals for the p-value replication study are even less significant. Right? So for a very significant original p-value you observe less significant p-value in the replication study. Now, if your replication study is four times larger than the original study this is good news. Because prediction intervals cover more significant p-values. 

In the previous figure we did not account for multiple comparisons. The fact that in our study we are testing a million or more hypothesis tests. If you account for a million comparisons, a million tests in the original study using a Bonferroni correction than you will get prediction intervals for the replication p-value in yellow. The yellow intervals are accounting for the fact that in the original study a million tests are being conducted. As you can see if you observe a p-value of 10 to the negative 6 in the original study while testing in the [inaud.] variants. In the replication you won't have replicated at all the intervals. You will get a p-value of 1 in the replication study. You see the more likelihood of replication when you are testing that many statistical tests at once. 

So why did we look at it? Obviously, we want to understand the meaning of getting a very significant p-value in a study with so many tests. But also this numerical result can help to design replication studies. We know we can calculate how large a replication study has to be in order to replicate the finding. 

Now next we are going to look at metaanalysis, which is often used in statistics where instead of a replication study we combine the original and the co-op study we combine different studies conducted. 

So the question is is it better to replicate in a sample size of n2 or is it better to augment original study and use a combined meta analysis. We derive a 95% prediction interval for the combined Z statistic. We see here prediction intervals for the replication study in red and in the combined study. We can see that of course combining the study gives you more significant p-value. 

The second question is how confident can we be in the relative meaning of two different p-values? So suppose we observe p-values with two different variants - two different SNPs. And one p-value is more significant than the other. And we ask, "is the effect of the  more significant SNP really bigger than the effect of the less significant SNP?" For example, which of the two SNPs to select for a replication or follow-up study? 

What we observe in the SNP line where we are conducting hypothesis tests. Whether the SNP has an effect different from 0. And we are conducting a second test for SNP2 where it in fact is greater than 0. We observe p-values, p1 and p2 for this SNP. So assume that one of the p-values, p1 is more significant than p2. How confident are we that the estimated order is the correct order? That indeed p1 has a larger effect having a smaller p-value? What we want here is to construct a test of data1 = data2. Data1 is the true effect of the first effect and data2 is the true effect of the second. We would like to construct a test of the difference between these two. 

And what kind of effect can we compare? It can be any kind of effect. For example, the log ratios of the two SNPs. Which log ratio is bigger? Or we can compare the true p-value of the p1 and p2. So we can really construct the test depending on what we would like to compare. 

We derived the test statistic for the test of the difference in the true effect. And T1 is the test statistic which is the standard normal under the hypothesis of Data1 = to Data2 Of no difference in the true effect. As we can see if T is positive it means that it is a stronger effect of the first SNP. So T1 is positive it means that the Z statistic is higher for the first SNP and therefore it gives evidence of the effect of T1. If T is negative it gives evidence for the stronger effect of SNP2. 

Let q1 be p-value for the one-sided test of the alternative hypothesis the effect of SNP2 is higher than the effect of SNP1. And q2 be the p-value for the test that the alternative that effects of SNP1 is higher than the effects of SNP2. We call the evidence ratio of q1/q2 in favor of SNP1 compared to SNP2. If evidence ratio is 95% it means that we can conclude that the effect of SNP1 and the effect of SNP2 in a one sided 5% significance test of the null hypothesis. 

Let's look at the numeric results of this test. You see evidence ratios in favor of Data1 versus Data2. Here in the first column you have the more significant p-value. Let's say the more significant of a p-value is 10-10. If you like your evidence ratio to be as high as 95% we have to go all the way to here. So the other p-value has to be as big as 10-6 for the two p-values to be significantly different from each other with the evidence ratio of 95.5. You can see that the difference between the two different p-values has to be quite large in order for us to be confident that one SNP indeed has a larger effect than the other. It corresponds to the more significant p-value. 

The implications here are that p-values vary greatly across studies even when underlying effects, population and study design are identical. And studies can have good power to reject the null hypothesis of no association while providing little information with respect to the reproducibility or relative strength of the true association. The more significant the p-value, the greater the variability. 

So while these concepts apply to even a single hypothesis we have millions of tests in genome wide association studies we are looking at very significant, very, very small, extremely small p-values and they are 10-8 and that is where the variability gets even higher. That is where it makes it challenging. 

Replication p-values can differ from an initially significant finding especially after multiple testing correction. We also learned that p-values provide little resolution to distinguish the true, relative importance of different SNPs. 

How can our results be useful? Well, they support their proposals of others to combine multiple lines of biological evidence in deciding which results to investigate further. Relying on p-values is not good enough. And other factors need to be taken into account to design replications studies for them to be successful. Our results can be used to design more successful replication studies. We show that the prediction intervals for p-values in replication studies depend on the relative size of the replication and initial study. Therefore, we can determine the sample size needed for replication. The prediction intervals require no assumptions about the unknown effect sizes or the initial sample size. They can be many different values and applied in most statistical studies whether or not multiple testing issues are present. 

We would like to acknowledge support from the Palo Alto Cooperative Studies DNA Bank and Laura Lazzeroni and Ying Lu And I would be happy to take questions. Thank you for listening. 

Moderator:  Thank you very much. For our attendees that joined us after the top of the hour you can submit a question or comment by using the box located in the upper right hand corner of the screen that is labeled Q&A. Just simply type your question or comment into the lower box and press send. And we will get to them in the order that they are received. 

The first question we have coming in is regarding access to these materials. In a reminder email you received a few hours ago there is a link to these handouts and you can feel free to download those or pass them on. Also, this call, this session is being recorded and you can access it in our archive catalogs you will get a follow-up email with a link to that that you can share. 

We don't have any other pending questions at this time, but we will give people a few more minutes to take their time writing in. And Alana, you did have additional slides, no? 

Dr. Belitskaya-Levy:  Only their questions. 

Moderator:  Somebody wrote in and they want some more clarification on what is the winner's curse? 

Dr. Belitskaya-Levy:  Okay. So winner's curse. Whenever we have a statistical result - statistical finding it can be due to let's say a very significant p-value, okay? It can be due to the fact that indeed there is a true signal or it can be pure luck. In real life it is usually a combination of those two. It is true effect but yes there is some luck too. Therefore, when somebody starts to report a new finding that is very significant it may be due to luck possibly and not just the true underlying effects. Therefore, whoever is trying to replicate the findings might be less lucky and would get a less significant result. That is called the winner's curse. Especially when you are conducting a genome wide association study with new means of test let's say you have a genetic variant that is most significant with a very significant p-value of 10 to the 8th. With a million tests even if this variant is very significant there is definitely a lot of luck in this being the more significant out of so many - out of a million other tests that we have conducted. And therefore on the replication study it is very likely that the most significant p-value is a lot less significant. Maybe it will be a 1,000 instead of one in a replication study. That doesn't mean that the finding is not significant. It just means there is a lot of - when you have a million tests being number one is definitely due to some combination of luck and to [inaud.]. You are very likely one p-value from a GWAS study and you are trying to replicate it will not look as significant on any other subsequent GWAS - any other test or study. That is simply winner's curse. It has nothing much to do with how true the signal is there. 

Moderator:  Thank you for that reply. That was the only pending question at this time. This is a lot of material for people to digest. Do you have any concluding comments you would like to make? 

Dr. Belitskaya-Levy:  We haven't published the results yet, but hopefully soon the first paper will come out. It might be easier to digest this material when you see it on paper. So I will be happy to answer any questions when it finally comes out. 

Moderator:  Well, thank you so much for sharing your expertise with the field. We really do appreciate it and I want to thank your attendees also for joining us. And please, hold on the line for just a second. When I end the meeting there will be a brief survey that pops up on the screen. And we do appreciate your feedback as your topic choices help guide the sessions that we schedule. Once again, I want to very much thank Dr. Belitskaya-Levy for joining us and please stay tuned for future VA Statistician Association sessions that will be coming up in the future. Thank you so much, Ilana. Thank you to our viewers for joining us. 

Dr. Belitskaya-Levy:  Thank you. 

Moderator:  Great. Have a wonderful day, everyone. 

Page 1 of 7