**Transcript of Cyberseminar**

**Timely Topic of Interests Series**

**Power and Sample Size**

**Presenter:  Martin L. Lee, PhD**

**February 28, 2013**

**Transcript of Cyberseminar**

**Timely Topic of Interests Series**

**Power and Sample Size**

**Presenter:  Martin L. Lee, PhD**

**February 28, 2013**

Moderator:I’d like to introduce our speaker. We have Dr. Martin Lee. He is a statistician at Sepulveda Veteran’s Affairs Medical Center, chief scientific officer at Prolactin Bioscience Incorporated, director of regulatory and clinical programs at MedClone, Inc., and president of International Quantitative Consults Lecture at the University of California, Los Angeles School of Public Health. At this time I’d like to turn it over to Dr. Lee.

Dr. Lee:Thank you very much Molly, and welcome everyone. I appreciate the opportunity to talk to you this morning. What I want to do today is talk to you about a subject near and dear, I think, to everybody’s hearts that are probably on the phone, and that is the issue of what statisticians call power in sample size – meaning essentially how big does your study need to be? As a long-term statistician - I’ve been doing this for 30+ years – it’s usually the first question that somebody comes up to me and asks. It’s like an obvious question, but the answer is not obvious at all because really the appropriate response is it depends. 

What we’re going to learn this morning is what does it depend on. Specifically we’re going to talk about the situation as it pertains to two very, very simple sets of circumstances – that is when you're doing a study that’s comparing two groups, either you're dealing with quantitative data and you're talking about the comparison of two means, or you're dealing with categorical data and you're comparing the hypothesis of two portions. Now, admittedly these are all very simplistic situations and admittedly there are many other circumstances that you're all interested in. But, I think to get a basic understanding of what the concepts are here, at least in these circumstances, will allow you to appreciate what you need to know and what you need to be thinking about when you talk about something a little bit more complex. 

Essentially, the issues that we’re going to be talking about today are – as I said before – what you need to know, the basic parameters, the power in sample size. I’m going to show formula, but obviously you have the handouts. I’m not really going to go into the formula. I just want to show you exactly what the calculations entail. Needless to say, there is software these days, and there’s actually several good programs out there. I’m going to point to one in particular called PASS that I prefer and use – a little disclaimer; I have no financial interest in the company so it’s fair enough for me to be able to talk about it.

To begin, what do you need to know? This is the broad set of topics that one has to ask. The first and fundamental thing is do you have a randomized study? This of course is what we’d expect. Most studies that we think about as good studies tend to be randomized. But, I know a lot of studies we do are naturalistic studies, cohort studies, in which the two groups are generated simply because of who you are and what you do – smokers vs. nonsmokers. That will engender different kinds of thinking in terms of how you design the study. Obviously controlling for extraneous is much more important in a nonrandomized study than it is in a randomized one.

The of course you have to take into account what type of randomization you do. Now, most people think there’s only one type of randomization, right? The answer of course is no. I mean, what most people think of as randomization is what statisticians refer to as simple randomization. Essentially each individual that participates, you get consent, you flip a coin – not literally, I think that would be kind of embarrassing to do that in front of an individual. But, we do that electronically. And, we decide which group they go in. However, there are much more complex randomization schemes, things called cluster randomization, things called stratified randomization, things called biased coin randomization. So, those are going to affect things. Clearly, you have to be thinking in terms of how you're going to assign first and foremost the patients to your study groups. 

Then, we talk about matching. Are you going to be pairing patients up? Now, that’s a very kind of specific type of design that we encounter a lot of times in what we call case control studies, where we actually have the individual with what we're interested in looking for and now we’re going to go out and get some control patients or control individuals who may not have the factor we’re interested in but certainly have all the other aspects. For example,  a study in which we’re looking at environmental factors as a potential cause for a rare cancer. We may think it’s simple because they drank from the wrong water source. So, we want to prove that by taking a group of control patients who have the same risk factors and they’re matched up according to those risk factors. Then, of course they don’t drink from that well and hopefully if you can show a different between the incidences of that cancer then you might have a reason for why that’s occurring.

The number of groups is important. Most of our studies tend to have two groups, but certainly that doesn’t preclude studies involving multiple groups. That of course is going to change the way you do your design. Of course, that’s a different design and of course a different sample size calculation. 

Now, very importantly is what type of measurement you are taking. Are you taking something that’s measurable – what we usually refer to as quantitative, a laboratory measurement or number of days in the hospital or something to that effect where you actually have a number that’s a real number, a measurable quantity – verses something that’s categorical – did your disease relapse, did you survive, etc., etc. So, clearly those are two different types of situations and lead to different kinds of sample size calculations. 

Or, you might be dealing with something which we call time to event or sensor data. This of course occurs when you're dealing with something like survival where the study has a finite length. You observe patients through the length of the study. They may or may not, unfortunately, survive. If they do and your endpoint is how long they survive, at the end of the study you don't know the answer to that because they obviously are going on. That’s called – as many of you know – censored data or right censored data to be specific. That kind of data or that kind of study leads to a different kind of calculation, a bit more complex calculation.

Are you doing repeated measurements on the patient? Clearly, that’s something we do a lot of. If you're doing a longitudinal study and you're following patients for a significant period of time and looking at whether it’s something as simple as a laboratory measurement or something a little bit more involved in terms of things like satisfaction with their care and so on and so forth. 

Are you including covariate adjustments? Of course a lot of our studies do that and again, that makes the situation more complex. Finally, a very fundamental issue that I’m sure you all probably remember from your basic statistics, are you doing a one or two-tailed test? What we mean by that is are you looking for an alternative that is the research hypothesis, which is that your experimental group is doing better, doing worse rather than...that’s one sided...rather than are they different than the control group or the standard care group. 

We like to recommend very strongly that you always consider two-tailed situations because they tend to be more conservative, particularly in terms of sample size. In fact, many of you know from trying to publish things, journal editors like you to be conservative, like you to use what we call two-tailed tests. 

There are a couple of more involved issues that we need to consider. Is the purpose of the study... I’ve been alluding to the fact that I’m assuming you're doing hypothesis testing. Are you trying to show some intervention, some experimental group is doing better or worse or whatever? That’s of course what we know is science as hypothesis testing. But you may be in a different kind of situation. We’ll actually spend a little bit of time at the end of this lecture talking about what we call estimation. 

This is the sort of thing where it’s not so much that you're investigating an effect of some intervention, you're merely trying to find out what’s going on in the population. What is the frequency? What is the proportion of some response or – for instance – pollsters do this all the time as we well know. Election time, they go out and find out what the electorate feels about a particular candidate who is going to project to win the election. So, we’re estimating the proportion of people who are going to vote for that candidate. Estimation is part of our tool box. Not that we do this much, because mostly we’re interested in a scientific advancement which usually means hypothesis assessment. 

Now, there’s and interesting phenomena - the second point on this slide – superiority verses equivalence. Now, most of us think about hypothesis testing for science as being advancing things. Can we come up with something that’s better than we have today? Is there a treatment better? Sometimes however, we’re interested in just showing this is the same as what we have. Now, why the heck would we want to do that? There’s a couple of settings. 

In the world of pharmaceuticals there’s a very fundamental situation where that comes about. That’s if you have a generic drug. Generic drugs of course are not supposed to be, by definition, any better than the name brand. They’re just supposed to be the same. Of course by definition, if they’re the same, then you could use them interchangeably and hopefully they’re going to be cheaper because of the economics of manufacturing generics verses the original research. So, sometimes we’re doing equivalent studies. 

Now, there are other situations that lend themselves to this. We may want to show a treatment is equivalent to another treatment simply because that second treatment – the new treatment – is safer or is cheaper. And, not because of it being a generic, it may be a totally different drug, but because it hopefully functions in the same way we would like to be able to implement the use of that drug and essentially the issue of safety or cost are secondary. We need to show...from an efficacy point of voice, that they are the same. So, equivalency studies or the one-sided version of that, lack of inferiority – it’s no worse than what we already have. We don’t care if it’s better. We just want to show it’s no worse, for the same reasons that I’ve just described.

Now, the other fundamental thing, and this really goes to the heart of how you calculate sample size. That is, what are you trying to show? What’s the point of your study? This is what we like to call delving. In a superiority study, which is the traditional type of research study that most of you do, we try to show that this new treatment, this new intervention, this approach to care of a patient is better than what we have. Now, that’s a nice thing to say – it’s better. But, you've got to tell me as the...and you've got to realize in doing these calculations, what does better mean? 

For example, if we have a drug that has a current standard of care...the survival rate with the current standard of care is 50%. Now, you're going to introduce a new therapy. You think this is going to make patients survive better in terms of percentage wise. What is the point of that drug? If it’s to make survival better by 1%, I don't think most people would really – I hate to say this – care. Part of the reason for that is to show that small delta, that small difference. You're going to take a phenomenally large study. Most people really won’t...it won’t matter. 

Really what we define delta to be is the minimum clinically important difference. In other words, the way I like to define this is, what is the benefit this drug has to have at a minimum to change clinical practice? In my example, from 50% survival to...is it 60%? Is that going to make a difference? Is it 65%? Is it 55%? You as the researcher and at the same time as the statistician have to make that decision jointly. And recognize, not surprisingly as we’ll see, the smaller delta is, the larger your study is going to be. It’s just the simple fact of life.

We have to make kind of an economic decision because sample size means cost and at that same time make a decision that makes sense. In other words, we don’t simply want to go into a study and say okay, I can only do a very small study so I’m going to do delta in that example I’ve been showing you at 30%. Okay, I’m going to improve survival from 50 to 80%. Guess what ladies and gentlemen. That isn’t going to happen. You can count on one hand how many medical breakthroughs have made that much difference. So, delta has got to be realistic and at the same time be economically reasonable. 

Let’s talk about the other part of the picture. That is the idea of sample size really is, as we just said, to demonstrate an effect that we’re interested in. But, and here’s the nasty and deep dark truth about. No matter what you do, no matter how you design this study...

[Loud Beeping]

[Irrelevant small talk.] 

Dr. Lee:[Audio dropped for 8 seconds] with 100% certainty. In other words, if you decide treatment A experimental treatment works better than treatment B, statistically you bet a nice p value that is small, etc., etc. Are we 100% certain of that? Of course not. By the same token, on the other side of the coin, if we don’t find an effect, does that mean we’re 100% certain that this drug doesn’t really do anything, the intervention doesn’t do anything? Of course that’s not true. The very simple reason, and this is something that you all probably recognize but it’s work reiterating, statistics is all about uncertainty. When we take a sample, and that’s what we do in research, we never observe the population – otherwise we wouldn’t be sitting here talking about this. When you take a sample, you're dealing with an incomplete information problem. If the population is a million or the population is a thousand, I don't care what the population size is. We’re going to take a sample, which is a fraction of that. That means, whatever is not observed is obviously unknown, and being unknown, we cannot be sure of the result. 

That’s what makes statistics so interesting. It’s the only branch of mathematics where you do the calculations, you come up with an answer and you're still not sure that you're right. It’s not like algebra. It’s like...if you have a simple algebra problem, x+2=4. We can all agree what x is. There’s no uncertainty of that. We don’t have that in statistics. Statistics, as I like to say sometimes, means never having to say you're certain.

Let’s see where the uncertainty or the error arises here. Now, you basically have a couple different circumstances here. If you're looking at this box here, and if you look at the top you essentially have two different possibilities which I like to call the truth – the reality. Which, of course is not observable but still affects the case we’re dealing with. Now, on the right side, the right column says difference is absent. Statistical difference is absent. Ho is true. Or, I think the better way of thinking about that practically is whatever you're studying really has no real effect. It doesn’t have any real benefit. That’s the truth. Again, we don’t know it, but there is that true. On the other side, the other column, Ho not true. In other words, what you're dealing with is wonders if you've discovered something that really has a benefit, a clinical benefit or whatever it is. That’s the good news.

The bad news is on the left-hand side of this box. You are doing this analysis of course based on your sample. On your sample, you're going to make one of two choices statistically. You're either going to reject a null hypothesis and as we know fundamentally if we’re dealing with the usual classical approach to analysis, that’s going to be when your p value is less than 5%. Or, you’re going to accept a null hypothesis or the better terminology is do not reject. That’s going to happen when your p value is greater than 5%. Now you can see what’s going to happen here. 

If you look in the lower right-hand corner, no error. That means that that’s a situation where you have not rejected the null hypothesis and correctly so because there is no real effect going on. There was no real effect. On the other hand, you have above that a really, I think, the most egregious error you can make, which is what we call a type one error, aka alpha - the alpha error or the type one error. That happens when you reject the null hypothesis when you really shouldn’t have. Why do I call that the most egregious error? I call it that because what you're saying to the medical or scientific community is this works. I found evidence to suggest it works. I’m writing a paper. It may even appear in some real prestigious journal. This works, right? When in fact, it doesn’t – so you're telling people the wrong information and they may change clinical practice as a result. That’s going to affect the patient. 

Therefore, we sometimes refer to this error as consumer risk. It stems from a very analagous situation in the real world of quality control testing where they test, let’s say an automobile to tell you that it works and therefore they sell it to you when it really doesn’t. You're the consumer. You suffer. Same idea. 

Now, on the other side of the table you have the opposite kind of situation. In the upper left-hand corner you reject a null hypothesis when in fact you should. I like to sometimes put a real big...we have a big happy face when I draw this box because this is the holy grail of research. You have invented something that works and you find that to be the case by your research. This is the most important place to be. 

You'll notice in that box I have the designation 1 minus β. That’s, if you will, the statistical symbol for that, but you know that as power. That’s power. I love that term, the term power. People ask me, why do statisticians call this power? I have a theory. Statisticians have a stereotype associated with them. They’re kind of wimpy, kind of nerdy, geeky kind of people. No. We deal in power, so therefore we're countering that. 

So, power is what we’re after. We want to try to get a probability of being in that box as large as we can. At the same time, we want to minimize the errors. Now, we talked about alpha error. That’s the consumer risk. But, what about the other error, that’s type II or beta error. That’s the error we make in missing this great new therapy or intervention - that we want to avoid too. That, if you think about it, is an error you'd make by telling people I’m sorry you invented this thing, but we don’t have any evidence to support it. Who loses there? Well, I guess if you want to use the same analogy to what I was using a minute ago, we’ll call that producer risk. 

Now, you take the two: consumer risk – alpha error, producer risk – beta error. Which is more important if you're going to rank these? Clearly, in most people’s minds, it’s alpha error. The difference between the two is how we design our research studies because we have to...in designing sample sizes, we’re going to see in a minute, we’re going to have to set levels for these things. We have to decide how large an alpha error, how large a beta error, or conversely how much power do we want? The answer is we’re going to deal with alpha first and then beta is going to come along for the ride and turns out to be a little bit more leeway in terms of how much we’re willing to accept. 

So, basically this next slide tells you exactly what I just said. Now, alpha error as you can see here, and I think most of you already know, we usually set at a conservative value of 5%. Now, that’s an interesting number because alpha error is the error we’re making in telling people something works when it really doesn’t. So people say you don’t want to make that error. You don’t want to tell physicians to use this on patients.  Shouldn’t we set this as low as possible? You could argue, well since you control this error, you get to set this error when you design your experiment. Why didn’t you set it to zero for God’s sake? We’ll never make this. But then you run into the interesting dilemma that if you set it to zero, automatically – if you stop and think about it for a minute – you'd never find anything. You'd never be able to reject your null hypothesis because your p value has to be less than that number in order to reject the null hypothesis. Alpha and your significance level are one and the same.

So, what that tells us, it’s a very interesting finding. The mere act of doing research means you have knowingly agreed to incur at least some of this error. You are knowingly going to make a...have certain situations where you're finding something that really isn’t real. But, that’s part of doing research. If you expect that then you're okay. I’m not perfect. I’m going to make mistakes. But, I’m going to try to control them. So, what we’ve decided as a convention you know, after many, many years of doing research, is that 5% is a reasonable level. That’s where we’re at. 

Now, as far as beta error is concerned, the producer risk, it will depend – as we’ll see in a minute – it’s going to depend to a certain extent on what you're delta is and your sample size. You get to maneuver beta error based on your choice of delta and your sample size, or conversely you fix beta and that dictates sample size. Now typically, we have a rule of thumb that says if we’re willing to incur no more than a 5% alpha error that beta error can be as much as four fold, four times that. So typically, most studies will have a beta error no worse than 20% but still a lot larger. The reason very simply is if you try to control beta errors at the same level of alpha error the sample size requirements are going to be so ridiculously large that you just can’t do these studies. So, it’s really a tradeoff and that’s really what research is all about.

Now, here you can actually see what I’m talking about. Here’s a very, very simple situation comparing basically essentially two means. What we have here is – without going into the essentially too complicated detail here – is that when you have the true result, the true mean, of your study, what’s true is actually just to the right of what you hypothesized. In other words, it’s the very small delta. In the top graph is a very, very small delta - the difference between the hypothesis mean and the true mean. And, it turns out that under those circumstances you have a very, very large beta error. It’s basically the area to the left of the line of the blue area. 

Moderator:Lee, I apologize for interrupting. The slides aren’t advancing. Can you click on those again?

Dr. Lee:Are you there? Does that work?

Moderator:Yes, thank you. 

Dr. Lee:Okay. No problem. Now, in the second graph you see there’s a big difference between what the null hypothesis is – which is on the left – and the truth on the right and you can see the probability of making a correct decision – which is the blue area, which is your power essentially – is very, very large. Clearly, the bigger the difference you're looking for, the smaller beta is going to be. 

Now, another aspect of beta error is sample size, even if you're looking for a fairly small difference. Again, now see on the left-hand side of this graph you can see that there’s a very, very small...there’s a difference, which isn’t numerically very large. And, here’s a specific kind of sample size, whatever it happens to be, and you can see that the probability of making the correct decision is relatively small. Now, what happens when you increase your sample size? Now, why all the sudden did your distributions of your data come up so narrow? Well, these distributions, since we’re dealing with means here, are a function of your standard error. Your standard error of your mean is a function of sample size. You can see on the left-hand side I’m assuming a sample size of 25. On the right-hand side a sample size of 100. So the variability of your population, of the population of the mean, is in fact a lot, lot smaller. Guess what happened? You've now separated the two distributions by removing them. That’s because you've reduced variability.

It’s very easy to understand that when you think about what I’m saying. The larger the sample size the easier it is to find things because you're getting a larger proportion of your population to observe. So, that’s the principle of sample size. It reduces beta error, because alpha is staying fixed. We’re keeping alpha at .05. That’s not a function here. The function here is beta error or, on the positive side, power. Power again is the blue area in both of these graphs. 

What do you need to know if we’re doing a simple test of mean...we’re dealing with quantitative data here, a simple comparison of means here. Obviously you've got to choose alpha and beta. You've got to choose your delta, and delta is a clinical decision. It’s basically what exactly are you trying to demonstrate, clinically. And we talked about that. 

Moderator:I apologize for interrupting again. Can you please pick up your handset? We’re having trouble with the audio coming through.

Dr. Lee:...Uh... handset...we don’t really have a handset here. Can you hear me better now? I’m talking right into it. No? 

Moderator:Okay thank you. 

Dr. Lee:Is that better? Yes?

Moderator:Yeah.

Dr. Lee:Okay. Here’s the thing again, go back to what we need. We have alpha and beta error. We have delta – that’s something we choose clinically. And now, we need an estimate of the variability. We need an estimate of the variability of our data and that means that you need to know a priori what kind of patient to patient or subject to subject uncertainty is there? Variation, standard deviation, you all know that. We need an estimate of that. Now, a lot of times people say okay where do I get that from? Well, the obvious answer is you go to the literature. You basically find...suppose you're doing a study of cholesterol. We’re looking at a drug to reduce cholesterol – a new anti-cholesterol drug. There’s a lot of studies gone on with cholesterol. We know what kind of variability there might be in the population of people with high cholesterol, what that kind of standard deviation is. 

Now, a lot of times, however, and I know this is going to be true for a lot of you, as researchers you're dealing with something new. You might have a questionnaire. You might have a new scale that you've invented and you have no idea what standard deviation is going to look like. You don’t know what variability is at all. So, there’s an interesting little rule of thumb, which is like... I sometimes tell my students if you get nothing out of this lecture other than this, this will carry you for a long time. It’s called the rule of four. I don't know if any of you have heard of it, but let me tell you what it is. 

It basically says that sure, you may not know the standard deviation, but I bet you you can guess at least or guestimate the range of your data. Suppose you have a new scale. It goes from one up to 50. I’m just making numbers up. What’s the range? It’s 49. If you take that range and divide it by four, that’s about 12 and a quarter. That’s an estimate or not an unreasonable estimate of your standard deviation. Why does that work? Well, if you assume your population, your data follow a bell shaped curve – a normal distribution – roughly as we know it’s 95%, but it’s almost the range of your data fall within plus or minus two standard deviations of the mean. Plus or minus two standard deviations is four standard deviations. If you know the range, divide by four, wow...you get the standard deviation. Isn’t that cool? So, this is a quick and dirty way to get at standard deviation when you have no idea what to expect. And, we often use it. It’s a way out. Believe me, I’ve used it more times than I’d like to count. 

Now, the one thing to recognize – as we’ll see in a second – that the real parameter of interest when you look at all this and when you look at the different numbers here is we’re talking about delta, we’re talking about standard deviation, but it’s really the ratio of those things. You've probably have heard the term of fact size, well in this situation that’s the fact size – the delta and standard deviation ratio. Sometimes engineers call that the signal, the noise ratio. Why is that the key parameter? Well, think about it. If you're looking for a very small difference, a relatively small delta, but you have a lot of noise in your data - your standard deviation is big. Clearly, you're going to have to have a lot of sample, a lot of signal, in order to detect that. 

On the other hand, if delta is small but very, very small standard deviation the sample size is going to be a lot smaller because you're signal is going to come through much easier. So, clearly the delta, the signal ratio is the key parameter. 

Now, I’m going to throw formulas at you. Okay, wonderful. Let’s not worry about the formulas. You can read all of this in your spare time. But I want to point out that everything is coming into play here that we talked about. If you look in the yellow box in the upper right-hand side of the slide you can see where alpha comes in. It comes in as a z value – in other words a standard normal deviate. Beta value, your beta coming in there, your sigma is coming in there. Your delta is coming in there. And, the only thing we haven’t talked about is r. R is just simply the ratio of what you want your experimental and your control group to be. We almost always want r to be one. But, this just allows for the fact that maybe sometimes you have a study in which you want two controls for every test. There may be situations for that. For the most part, as a rule of thumb, most studies you maximize your power, you maximize your...in other words, you've got your minimum sample size for your desired power by keeping the ratio as one. But again, if you want it whatever, that way, then of course you can include it in your calculation.

Now, you can see in the bottom there, there’s a simplification here of that formula. This is kind of neat. You can impress your friends at parties if they ask you how many people to I need if you simplify the problem to equal sample sizes in the two groups – so a ratio of one – 80% power of 5% significance. Then, it turns out that this whole mess here reduces to 21 over your effect size squared. So, if your delta to sigma ratio is like .5 then that gives you a sample size of 84. Wow. I just did that in my head. Isn’t that amazing. You can do that. That’s a very simple, simplification. 

Now, what’s interesting, I just want to point out again that delta to sigma ratio issue is sometimes we have situations in research where, I got this new scale. I don't know what delta should be. I think maybe it’s a point or two, but I’m not sure. I’m not sure what sigma is. Well yeah, we can use the rule of four, but you can say to yourself, maybe I can just figure out what delta to sigma ratio I want. Because, the delta to sigma ratio is the effect size. Let’s say I want a .3 standard deviation effect. Okay, so then delta to sigma is .3. Now, how do I know what number to pick? Well, there’s a gentlemen who wrote a book 20-30 years ago by the name of Cohen. 

Cohen came up with a set of rules for what he called sizes of effect. He defined effect sizes as small, medium, and large. How did he come up with this, because he was a psychometrician and he looked at a looked at a zillion psychology studies and basically saw what constitutes fairly frequently a small effect, what constitutes a medium effect, and what constitutes a large effect in terms of the clinical outcome of these different studies. He said, essentially – and I’m just quoting roughly so you can get an idea – that an effect size of about .2, in other words .2 standard deviation is considered a small effect. A medium effect is about .5 and a large effect is about .8. Now, that’s also a good guide for us because if you come up with an idea that I want to show a delta of 10 and the standard deviation is 2, then that means you've got an effect size of 5. Wow, you're really looking for something that really doesn’t occur in nature. Obviously you're going to get a tiny sample size, but you're being unrealistic. You kind of use Cohen’s rule of thumb to basically understand what you're really asking for in terms of your study. 

Now, here’s the obviously relationship between sample size and effect size. And, not surprisingly as the effect size gets smaller and smaller sample size increases exponentially and vice versa. You'll notice, interestingly enough, that sample size kind of levels off when you get to an effect size of about .5. What did we say .5 was? A medium effect size. So, that’s kind of...that might be a target for us, as the minimum sample size we want. I’m not saying that we’re going to achieve that with our study, but clearly we don’t want to go below that because it’s unrealistic and it doesn’t make sense to do studies smaller than that.

Here’s a simple example of an oral contraceptive study. This is actually the results of the study where you're looking and blood pressure after oral contraceptive use in a user verses a nonuser. It was a very small study, eight people in the user group. This was a naturalistic study, not surprisingly. Twenty-one in the nonuser group. You can see what the blood pressure average was. The p value here was very large, .46. We can do a calculation here working backwards from our sample size and show that the power to detect the delta of about 5 was only 13%. We have no power here. This was an underpowered study. By the way, one of the concepts that we should all be aware of is that the reason why most studies fail is because they don’t have a large enough sample size. All studies will succeed if you have a large enough sample size. But, that’s not to say that you should always do huge studies. But, we should always make sure that our studies are big enough to detect something that’s reasonable. Otherwise, you're going to find yourself in situations like this.

Basically, if you use the same delta and roughly the same standard deviation you wanted 80%, guess what. You needed 154 subjects because the delta here...the effect size here is .3. It wasn’t a very large effect size. So, clearly you need to increase the sample size by about five and a half fold. 

Now, I just wanted to show you real quick here, this was the screen from the program I was referring to earlier called PASS. It just gives you a rough idea how easy it is to use these programs to do the calculations we just talked about. You can see you enter the means of the two groups and you answer the standard deviation, and of course you're allowed to assume the two standard deviations are the same, and then at the bottom here you can see you can have an alternative that’s two sided or one sided. You can actually adjust for the fact that maybe your data aren’t normal. So, you’re going to do a nonparametric task and it will give you an output like this. I’ll give you powers for different possibilities and the different possibilities, as you can see in the middle there under the columns labeled mean one and mean two, you can see at the top mean one is ten and mean two is eleven. That’s a delta one. The standard deviation is assumed to be 1.5. So, as you're going down this you can see the effect size is increasing. And, it gives you the result for both 90% and 80% power, and not surprisingly the sample size gets smaller as the delta sigma ratio increases. And obviously the sample size is always smaller when the power is smaller. So, a very neat kind of program to do this and it takes you literally three seconds to use it.

Now, if you want to do a test for equivalence – I’ll just briefly mention this because most of our studies don’t deal with this. But, you basically do that opposite of what you normally do. Your null hypothesis is essentially that the difference of the means is nonzero. It’s basically that they’re different. You're alternative is that they’re the same. But, the same or different have to have a definition. This is the delta that’s different from the delta that we just talked about. 

The delta we just talked about is the delta for superiority. The delta for equivalence is not necessarily the same. In other words, we might say – to give you an example, remember our survival example. We might want to show this new drug is 10% better than the old drug. But, we might...if in another circumstance comparing two drugs to replace...you know, basically to show equivalence. We may not say drug two is equivalent to drug one if it had 10% worse survival. We may think that’s a little too broad. We may want to change delta to 5%. Typically that’s the case. So, your delta is going to reflect the different clinical setting because now you're saying that this is the largest difference that would make the clinically indifferent to the two drugs. I would use either drug as long as they're within this amount difference in terms of my outcome. 

You have a slightly different calculation and you'll notice that the true difference comes into play. How do we know what the true difference is? We have to sort of guess and most of the time we put zero in there. We’re assuming that yeah, they’re equivalent, they really are equivalent. It makes it a lot simpler. 

Now, what about proportions? Well, here’s the sample size formula for proportions. The things are basically the same except as we well know the real issue with proportions is – or the nice issue with proportions – we don’t have to worry about standard deviations. We basically have to worry about two things. You have two proportions, p1 and p2, and you're basically testing whether p1=p2. Some people use pie instead of p. Now, the two things you need to know...you might say I don't know what both of them are. But, what you do know, or should know, when you the study is you know what baseline is. You know what the proportion of the outcome is in your control group. For example, if you're testing survival you should pretty much know currently what’s the proportion of patients who survive a certain period of time. The other thing that you need to know then is how much better. That’s what d is up here or what we’ve been calling delta. That’s all you really need to put in here.

Here’s the simple example of where you're comparing the rates of adverse events for two drugs then you want 1%. So, this is a real tightly designed study, maybe because safety is a big deal. One percent significance level, 90% power, a difference of 10% and the sample size here for each group is 675. So ladies and gentlemen, don’t get carried away with this in terms of how well you like to design certain studies. Thirteen hundred and fifty patients may be well beyond our budget.

You can see here, this kind of gives you an idea of how sample size varies as a function of proportions. p1 of course is your baseline proportion. Notice that when you're dealing with numbers right around 40 or 50% it’s where your sample size difference...your sample size kind of maximizes. Obviously as you're getting away from that the sample size is less. Why is that? That’s because - for example – you're looking...let’s look at the first column, a 10% difference. That’s what the columns are. I’m trying to show a 10% difference from baseline. If your baseline is 50% right, 10% difference is a 20% relative effect. But, if your baseline is 10%, your 10% difference is 100% relative effect. So it’s not surprising that your sample size is smaller. 

Again, your effect size here is the difference between baseline and what you're trying to show. And clearly, you can have what looks like the same delta, but it’s not because it’s relative to baseline. So, you can see...and obviously the bigger the difference the smaller the sample size. That’s a truism that’s pretty obvious. Here again is the screen from PASS that shows you how easy this is to do where you just enter. You can enter...you can do sensitivity analysis. You can enter multiple values for your treatment group. Your control group you should know. You shouldn’t have to vary. You should know what your baseline and proportions are. Here’s just some output from PASS that shows something about remission rates and, again, just proves the same point that as your p1 increased...as your sample size increases...your difference increases, your sample size increases. 

And then of course the same thing is true for equivalence. You have to decide what equivalence means in terms of the difference of proportions. Then, you can also factor in, if you want, multiple controls per case, which we do. So that’s the value c. Again, a formula for that – not going to sort of waste your time, but one thing to keep in mind is that as you increase the controls per case you get very...after a while you get very little benefit. So like for example when we’re doing epidemiology studies and we want multiple controls you get to a point where after...usually the rule of thumb is about three or four cases of controls per case. After that you really don’t get a whole lot of benefit. 

Sample size for estimating odds ratio: again a formula. I’m not going to go through that since we’re kind of running out of time here – just an example of that. Occasionally we have an interesting study where we want a sample size for a rare event where the goal is to estimate the probability of a rare event. For example, suppose were giving patients a drug and we want to make sure that none of them come down with some serious adverse event like heart failure for instance. We want to be able to...the example I’m giving here is HIV after infusion of some sort of blood transfusion. Hopefully we get zero. But even if we get zero, what’s the upper bound for that probability? It’s not zero as we know because clearly it’s a sample. So, we have a very interesting thing here. We learned about the rule of fours. Now we learn about the rule of threes. You see, you're learning all kinds of new stuff today. 

The rule of three says that the upper, 95% upper bound for that probability is less than or equal to 3/n. That’s an approximation based on the Poisson distribution. It doesn’t matter how we derived it, but it’s a neat little thing. So, if we do 100 cases, none of them develop HIV, we know that the upper bound is 3%. So obviously we can define the n here to say I want the upper bound - for example – to be 1%. Clearly the n is 300 – very, very simple rule under those circumstances. 

Of course I mentioned here the software I prefer and actually PASS is now up to, I believe, 2010 version. It’s not a very expensive program. It’s a few hundred dollars. There’s another well-known program called N Query. I haven't used it, but I hear a lot of good things about it. Of course you have internet calculators. You can find anything on the internet of course, but as you probably know, don’t trust everything on the internet. Be careful if you use a free site as far as some of these calculations are concerned. There’s even, I’ve even downloaded them, there’s even apps for your IPhone or IPad that will do some of these calculations, which is kind of interesting.

Here’s more of the...now, closing up here. Just the sample size for estimating proportions...for estimation means and proportions. A very, very simple sample size where you basically need two things. Since power and significance level don’t come in, we’re estimating, you need to know your standard deviation of your data. We already talked about that. The only other thing you need to know is d. That is what we now know from our newspapers as margin of error. You need to know what the plus or minus you want on your confidence level. Then, you calculate the sample size, same of course with proportions. So, estimation is all about margin of error. Do we want to know what candidate x is going to get, plus or minus four points, that means a margin of error of 4%. It means your confidence interval is plus or minus 4% on whatever the estimate is. 

So, ladies of gentlemen that is right on time, in spite of our little mishap in the beginning. I hope that kind of gives you a window into how we do these sorts of things and I thank you for your time and attention. I guess it’s questions at this point. I’m not sure...how do we deal with this? 

Molly:Thank you very much. I will be moderating the questions over the call and we do have several, so if you have a few minutes to stay after the top of the hour we can get right to it. 

Dr. Lee:Okay. 

Molly:You can go ahead and leave that last slide up just so we have something to look at. 

Dr. Lee:Yes, of course. 

Molly:Great. Thank you. And the first question: is delta also called effect side?

Dr. Lee:No. It is the delta to sigma ratio that is the effect size. In other words; the difference you are looking for divided by, again, we are talking about quantitative data here. It is the difference you are looking for divided by your standard deviation. That is the key parameter. Because delta could be any number, but it is not going to mean anything if your standard deviation is much larger than that. So you really have to factor in that uncertainty or variability that you have in your data. 

Molly:Great. Thank you very much. I am now going to turn it over to my colleague to moderate the rest of the questions. 

Dr. Lee:Okay. 

Molly:Heidi, are you available? Okay. I’m really echoing right now, too. 

Dr. Lee:Not on my side. 

Molly:Okay. Well if you can hear me I’m going to run with it.

Dr. Lee:I can hear you fine. 

Molly:Okay. The next question I have; can you explain how to do sample calculations for for insuring moderation of effect? 

Dr. Lee:Moderation of effect, I’m not sure I exactly understand what the question is. Okay. 

Molly:I’m not sure. 

Dr. Lee:Okay. I’m not sure quite what that means. I mean there - it has various connotations. I apologize, I’m not quite able to answer that question. 

Molly:Okay. The next question I have here; can you address the issue of data [inaud] that are not normally - for example, hospitalizations for asthma where there will be many people with zero. 

Dr. Lee:Absolutely. That is going to be the simplest way to think about that is one could use - we could do a couple of different things one could use - what we would probably use in the simplest case are non parameteric tests. Now if you remember I showed you in the screen for the path program that you can adjust your sample size to allow for the use of in that case it was a Mann-Whitney test. If it was a PAIR test it would be a Wilcoxon signed-rank test. You can adjust for that. However, that adjustment requires that you make an assumption about what the data distribution does look like. You may assume that it might be logged normal. You may assume it might be exponential. You have to have some assumption. 

Now the alternative, which is a little bit more awkward, but can be done, is to assume that perhaps you can do some sort of data transformation to make it norm. Obviously, with a lot of 0s you can’t use a log, it may be a little tricky, but let’s suppose it is data that didn’t have zeroes but it had a long tail to it. Let’s say like hospital stay. You could do a log transformation, so what you would need to plug into these formulas is the mean and standard deviation in log terms and get the sample size determined that way. That way you would be able to incorporate a basic parametric D test whatever using that kind of data, but it would be on a different scale. 

Molly:Great. Thank you very much. It looks like we have finally taken care of the audio issue. We appreciate your patience everyone. Can you explain how to do a sample size calculation for insuring, detecting moderate effect- moderation effect? 

Dr. Lee:Someone already asked me that and I’m not sure what that question means. I apologize. 

Molly:No problem. We will go to the next one. Can you provide more examples of D and S to help me visualize the effect of size. 

Dr. Lee:Sure. Again, let’s go back to our cholesterol example; suppose I’m interested in developing a new cholesterol lowering drug for people who obviously have high cholesterol. I would like this drug to be able to lower cholesterol by 10 points in six months, 10 points on the cholesterol scale. My D, my delta, the clinically important, minimal clinically important difference is 10. Now, my standard deviation is the variation in cholesterol levels in the patient group, alright? Let’s suppose from a previous experience I know that when I measure cholesterol on a particular point in time on a group of patients I find that the standard deviation is 50 points. Let’s suppose in a high cholesterol population it is an average of 220 plus or minus 50 those are my data. My D is 10, my standard deviation is 50 so under those circumstances I am trying to detect an effect size of 20% or .2 standard deviations. 

Molly:Thank you for that reply. How do you determine power in studies with hierarchal samples? Are there any rules of thumb? 

Dr. Lee:Well, you mean sample size when you have clustering, it is a little bit more complicated because you have to incorporate a new parameter, the intracluster correlation co-efficient. Now, obviously, we didn’t talk about this here because we try - this was basically a very fundamental discussion of power and sample size. But if you look at the formula I gave you earlier for sample size comparing two means what you are going to do is you are going to calculate this or not calculate but you are going to guesstimate. This is always tricky in these studies. I am assuming you have one layer of clustering here. In other words there is a single hierarchical model. So like say patients clustered within physician for instance. You are going to have to multiply, in other words increase your sample size essentially by a multiplier that the square root of the intracluster correlation co-efficient. With proportions it is actually just the ICC. Now, where do you get the ICC from? That is always the $64,000 question people ask me and the answer is you get it from the literature. There are lots of papers out there now that have cataloged ICC. If you know what kind of clustering effects you have in terms of the correlation within a cluster, you can just simple inflate your sample size by a simple function of that number. You use the same calculations, but you just simply use the ICC. 

Now the good news is if you are using a program like PASS, there actually is a module in there for hierarchical design and all it asks you for is tell me what the ICC is, tell me essentially either what your total sample size is or the number of clusters usually you want the total sample size to tell me how many clusters you have and how many people you think you might have per cluster or vice versa. Tell me how many people you have per cluster of a total sample size and it will tell you how many clusters you need. Either way. The point is you can do this calculation within the PASS program, but you simply need to know something about the ICC. 

Molly:Thank you. Can you talk a little about the sample size website in the University of Iowa? 

Dr. Lee:What was that again? I’m sorry. 

Molly:Can you talk a little bit about Russ Lenth’s sample size website on the University of Iowa? 

Dr. Lee:I’ve heard of it. I would - you know, I can’t confirm or deny the validity of any website out there simply because it is all about the internet. Certainly, I would feel fairly comfortable even if it is an academic website a little bit better than I would with just somebody putting something on the internet that you didn’t know. But I have no real knowledge of that in terms of using it. Unfortunately, I can’t comment just beyond that. 

Molly:Thank you. What is your recommended software for calculating sample size? Some people have asked about SaaS, SPS, etc. 

Dr. Lee:Those programs don’t do these calculations. Those programs as far as I am aware there are not any sample size modules in any of the basic statistical packages. There are stand alone software packages that I can discuss in here. As I said, I have been using the PASS program for many, many years. I am very, very happy with it and as best I know it is a valid and accurate program. The other one that I mentioned is N Inquiry which was written by Janet Elashorff  who used to be at UCLA. Given that I am a Bruin too I guess I should give kudos to that one as well. 

Molly:Thank you. Many people are wondering if PASS is available to VA employees or if they must purchase it separately on their own? 

Dr. Lee:I don’t believe it is available, I don’t think it is in VINCI is it? I don’t think so. It is a program you can get as I said it is a really cheap program. It is less, I think it is something like $3, $400 tops and it is a one time license. It is not expensive. I don’t think the VA provides it. I don’t think it is available on VINCI. 

Molly:Thank you. Can you include the website address for the sample size calculator that you were using? 

Dr. Lee:Yes. It is NCSS.com. Number Crunchers Statistical System dot com. They have a software package for doing statistical analysis as well, but it is the same company. It is actually run by a guy who was at the University of Utah. You can find all the information you want at NCSS.com. 

Molly:Thank you. Also is there a book that you would recommend? 

Dr. Lee:Well, certainly the classic book is Cohen’s book. I think it is called I want to say it is called Statistical Power Analysis for the Behavioral Scientist and it is a little outdated only because what it contains is - it is a big fat book that contains tables on all of these things and who needs tables anymore when you have these calculators? I think you will - if you pick up any good basic statistics test book the one I use to teach out of at UCLA is Rosner’s book, Bernard Rosner introduction to bio-statistics. In those basic textbooks they are going to talk about certainly the basics of power and sample size in there. 

Molly:Thank you. Can you speak about the implications of calculating sample size and power for cluster RCTs? 

Dr. Lee:Yes. This is something we addressed in a previous question. The fundamental answer is the easiest situation is when you have a single level of hierarchy like patients within physicians. And then the extra parameter you need to incorporate into the calculation is the intracluster correlation co-efficient, which again most people don’t have a clue what that might be so they - you look in the literature cataloging these values for different kinds of outcomes. I think that is really the only the main change that you have to do to what we have been talking about. 

Molly:Thank you. You mentioned in Cohen’s estimate of .2 for small effect size, .5 for medium and .8 for large would this change for more complex analytical methods? 

Dr. Lee:That is what the beauty is of Cohen’s designation. Those are unitless effect sizes that basically reflect in his experience what constitutes a study that is looking for something that is really a small difference versus an average difference versus a large difference. So it really doesn’t matter the context. He is trying to - and we have done this. I know many grants I have been involved with we basically said we are going to detect a .5 standard deviation effect size because based on Cohen’s designations this is an average effect that we are looking for so it is not out of the ordinary. It is a way to categorise what you are looking for. It doesn’t have - it is not related necessarily to a study or to a particular type of measurement it is simply a way of thinking about what you are looking for in a study. 

Molly:Thank you. Stop me if I have already asked this one; can you address the issue of data that is continuous but not normally distributed? For example, hospitalization for asthma where there will be many people with zeroes. 

Dr. Lee:I answered that a few minutes ago. 

Molly:Okay. Can you restate the rule of four? 

Dr. Lee:Yes. The rule of four says to get a crude estimate of standard deviation you guess what the range of your data is. That is the smallest value you might see, the larger value you might see minus the smallest value and you divide it by four that gives you a crude estimate of standard deviation. So it is the distance from the minimum to the maximum divided by four. 

Molly:Thank you. The sample size calculation has mentioned on effect detection I want to know moderation effect detection for instance, interaction. 

Dr. Lee:Okay. I wasn’t quite clear - so interaction terms. Okay, the idea is still the same. Okay? An interaction is just another is a sort of mean and obviously your hypothesis there is at a zero. It is really not that different in the sense that you are trying to design a study to determine a difference from zero. Your D your delta is really how large of an interaction you want. Of course, that is not always the easiest thing to figure out. Typically, we think about interaction mostly as multiplicative. You think about your effect sizes for your main effects, your deltas for your main effects and you kind of multiply them together and that would define your interaction. That is an overly simplistic way of thinking about it. Really, you can put that in the same context of what we have been talking about today, I think. 

Molly:Thank you. Is the estimation versus hypothesis testing also called post hoc analysis? 

Dr. Lee:No. No. Estimation is a different kind of inferential problem where your goal from the outset - this is usually in a survey - is to determine the frequency or the proportion or some other parameter of your population. Post hoc analysis is where you are going to look at your data in multiple ways after you do your initial hypothesis. For instance, if you have argument’s sake a multi-group comparison where you might have several study groups and one control group you might do an initial analysis which is basically an analysis of variance to compare all of the groups simultaneously and then post hoc you might want to say I want to compare each group to control to see which ones are better than the standard of care what control is. That is where we do you know, things where we might do confidence intervals, typically, but it is still a hypothesis test. If you are looking to generate sample size under those conditions where you know you are going to do post hoc analysis the easiest way to think about that is the realization that your significance level is going to be adjusted for the multiple testing. Let’s suppose you are going to do four - let’s say five it is a good round number, five additional tests. You have six groups. You are going to compare five groups. Each one to the control. You are going to do five tests which are usually the easiest way to consider that is to adjust your significance level using the Bonsignori inequality and change your significance level from .05 to .01. So then you just do your sample size on the basis of .01 significance level and just follow the same paradox. 

Molly:Thank you. Someone is wondering how did you get 154 subjects? 

Dr. Lee:Oh in that OC example? That is because I used the formula - the important thing is that the difference they are looking for which was about 5.4 points on a blood pressure scale and the standard deviation I believe was about 16. You are looking for an effect size which was less than .3. It is a relatively small effect size and that just drives your sample size up significantly. The study did only 29 and as you saw it didn’t generate significant effect if the power was only 13%. So clearly, the sample size had to be increased dramatically to get I believe an 80% power is what we were looking for. 

Molly:Thank you. The next question; type one error. Are you saying the hypothesis is true and you accept the hypothesis? When is it false? 

Dr. Lee:Okay. Type one error occurs when in truth which of course, we do not know, in truth the null hypothesis is correct. That is a technical thing. What we think about or say practically is we are looking at an intervention that has no benefit, has no effect. A type one error occurs when you do a study with an intervention that has no effect yet you still find statistical significance. The probability of that happening is indeed your type one error. Your significance level which is 5%. The probability that that is going to happen a priority with an intervention that doesn’t work is 5% of the time. However, and this is the key thing to remember, we have no idea that the intervention doesn’t work. Clearly, if we did, we wouldn’t be doing the study. So all we can do is say if that were the case this is the chance of making that error and what we are trying to do in designing our study is to control that and obviously to control the other type of error. This would either be dealing with an intervention that works in which case beta error is in play or the intervention that doesn’t work which means alpha is in play. We have to control both errors because we don’t know where we are and we never will. We just simply get data and hope for the best. Unfortunately, that is what statistics and research is all about. You aren’t going to get any better than that, folks. 

Molly:Thank you. We are getting down to the last few. 

Dr. Lee:Okay, good. 

Molly:Is it okay to use the power calculation to see if you had enough power after the data analysis is complete? Alternatively, what do you think about a power analysis for a proposal to analyze existing data? 

Dr. Lee:Power ex post facto is an interesting exercise. It doesn’t really mean anything because obviously if you don’t get significance then you might be wanting to know well if the difference you saw really was the true difference, what sample size would you have needed. That is all it is. The exercise to plan for the next study. Other than that I am not sure why else you would do it. Obviously, if you get significance power is irrelevant because power doesn’t come into play. If you find significance there is no beta here by definition so you don’t have to worry about it. 

Molly:How does comparison of means of more than two groups affect the sample size requirement? 

Dr. Lee:It is a different calculation. What we showed here today was the calculation for two groups. When you have a multi group study you are doing a different kind of test it is called an analysis of variance. The effect size is a little different. Now what you are looking for is the multi group differences and how you put that into the calculation. It is a slightly different calculation because you are using a different statistic. You are using an app statistic. What you are going to do is look at a variation of your means. In other words you are going to put an estimate on how the means differ from one another among your several groups and divide that by your intragroup variation the same S squared we talked about here. That is what is going to go into your formula. It is a different formula. Like I said to begin my talk today I couldn’t cover everything, I just wanted to give you a basic introduction. So keep in mind you are doing that you are going to be doing a little bit different calculation. 

Molly:Thank you. I am new to the world of research. Do I have to know what type of data I need before selecting a type of study? 

Dr. Lee:That is an excellent question. I think it is actually not a bad question because a lot of people say I am going to measure something. I am going to study an intervention. What do I measure. Really it sounds like a silly question but it is not. People are not sure. I have studies where we sat around for six months trying to figure out what is the best way to get at whether this intervention works. The rule of thumb that I often tell people is if you can measure something it is better than categorizing something. Pound for pound your sample size requirements are going to be smaller in a study where the end point is quantitative versus when it is qualitative. I often tell people if you are doing a study where your end point is survival don’t measure look at survival yes or no. Look at survival time and that will give you a better way to assess whether the intervention is effective. 

Molly:Thank you. A couple more people are asking about SaaS in order to calculate the power and sample size mass. One person did write in saying SaaS did have the power and sample size module. 

Dr. Lee:Oh, it does? I’ve never seen it but I take your word for it. I don’t use SaaS I would be talking out of turn. I would recommend very strongly since SaaS is great obviously for analytical uses. I would recommend strongly you use a stand alone sample size program to do these kinds of calculations. It is going to be much more expansive. It is going to cover a lot more of the questions people ask about cases that I discussed today. These programs are going to cover all of them as opposed to SaaS which I am going to guess - don’t anybody shoot a gun at me here - I am going to guess that it is probably just for the most simplest cases that we discussed here. Not the more complex ones. 

Molly:Thank you. Someone wrote in saying that PASS is $795 for government and academia. 

Dr. Lee:Oh, okay. I’m sorry they must have just raised the price. I tell you, it is still a deal. Still a deal. It is a great program. I have used it so many times I can’t even begin to tell you. It is worth the investment. Particularly, for the people that are asking questions about moderation effects, multi group studies, clustering, survival studies, all of these things. You really don’t want to try to figure these things out by hand. Not only that you are going to be able to run sensitivity analysis. What is analysis with these programs that you can’t do otherwise. If you are doing any kind of research at all, disclaimer again, I don’t own any stock in this company. I am not pushing it for any reason other than I really love the program. 

Molly:Thank you. We have had people write in what do you think of G Power as an alternative power calculator? 

Dr. Lee:I do not know that program. I apologize. 

Molly:No problem. Final question: what is the difference between the effect size and coefficient value from a regression? 

Dr. Lee:Okay. If you are looking at effect size in regression you are basically, in other words you are trying to find the significance of a regression co-efficient. Let’s say you want to show interval to variant model regression coefficient for a particular effect is significant because you want to show that effect is significant. The effect size is essentially the regression coefficient - essentially as a function - not as a function, but basically divided by the error of the model, the error of the standard deviation of the model for that particular coefficient. It is like the beta to sigma ratio is essentially the effect size. You have to remember that with quantitative data the effect is always going to be mediated by how much noise there is in the model. In this case the model is a lot more complex and the noise, the standard deviation always comes into play in power calculations of quantitative data. 

Molly:Thank you. We have one final comment, G Power is free and not reliable. It is good for estimates, but I would not use it for my real numbers. I am a statistician and I also use PASS. It is great and I also don’t have any stock in the company. 

Dr. Lee. Thank you for that. I appreciate the vote of confidence. 

Molly:Great. I do want to thank the 130 plus attendees that stuck with us through the end. Of course, to Dr. Lee for sticking with us for all the questions. That was great. As we mentioned, this has been recorded and you will receive a follow-up email with a link to the recording. Thank you, Dr. Lee and thank you all of our attendees. 

Dr. Lee:Thank you. 

**Page 1 of 13**

Page **21** of **21**