**Hierarchical Research Designs: Design Strategies and Statistical Approaches**

>> And we are just about at the top of the hour so I would like to provide an introduction for our tow speakers today.  The main presenter will be Dr. Martin Lee.  He is a senior statistician with the Center of Excellence for the Study of Healthcare Provider Behavior at the Sepulveda VA and has been involved in this project for the past seventeen years.  He is also an adjunct professor with the UCLA department of biostatistics and a professor of internal medicine at Charles R. Drew University of Science and Medicine.  And presenting with him today as a discussant is Dr. Becky Yano who is trained in healthcare epidemiology, biostatistics and health policy at UCLA and Rand Health.  She has 25 years experience in health services research and program evaluation.  She is the co-director and research career scientist at the VA Greater Los Angeles HSR&D Center for Excellence for the Study of Healthcare Provider Behavior and a professor of health services at UCLA School of Public Health and serves as the P.I. for the Women’s Health Consortium.  So I’d like to take both of you for taking the time to present for us today, and I’d like to turn it over at this time.

>> Thank you so much Molly.  I really am grateful for Dr. Lee’s participation today.  He is our senior statistician for the VA Women’s Health Research Consortium and is available for technical consultation. He's been instrumental to the conduct of clinical trials here and across the country and has been truly a leading expert in implementation research study methods and statistical design and analysis. So without further ado, it's my privilege to introduce Dr. Lee. 

>> Thank you very much. It's my pleasure to be able to offer this seminar to everyone this morning. Obviously, the topic here is a fairly -- I wouldn't say arcane, but fairly involved statistical and design issue. That is actually fairly common within the context of what our mission is at the VA, and particularly some of the design and statistical issues that come up with women's health problems. Now, I am a statistician as Dr. Yano just mentioned, but I'm going to try to keep this particular discussion today to a relatively simplistic level.  So some of the things -- I know some of the people on the call here today or on the seminar probably know a little bit about this area, and know that there are some really complex analytical issues involved, but I'm going to stay far away from that in order to make this at a level that everyone can appreciate. 

So first of all, let's think about what we're doing here at the VA with respect to the type of issues that generate what we refer to as a hierarchical model, which in essence means that we're looking at data that has several -- that has a richness to it that also makes it a little bit more complex to deal with. In the sense of normally when we do research, particularly in let's say clinical trial research, we're thinking about an intervention that impacts the patient directly. We gather data, and we analyze the data on the patient.  And that's very traditional, and that's the sort of thing that all of us learned initially when we start talking about statistics and statistical design. But obviously, in situations, particularly if you can see on this first slide, the issue of process of care. In other words, how do we look at the influences on how healthcare is delivered? And then you can start to realize that there's not a simplistic data structure any more, because obviously the concept of the -- as you can see here, the health outcome, which needless to say is going to be measured at the patient level -- that's the end of the line, so to speak. But clearly what we're going to be evaluating here are all kinds of things that go on at a level well beyond the patient and at levels that go even well beyond the actual physician delivering health care. And we're talking about things like the actual, for instance, the organization, the environment, which could mean the entire facility, it could mean even at a level of the VISN and how VISNs vary and what influence that has.  So what happens here is that what we're going to be talking about within the context of this type of design are our influences at various levels in various levels of the healthcare delivery system starting with the patient to the physician, to the clinic, to the institution, the hospital, and maybe to even a higher level than that.  And I think that's what makes this particular design useful, because it does in fact, account for all the influences and the variability associated with all those factors. But of course at the same time, makes it a little bit more complicated. 

Now, the type of interventions that we're talking about here, just to give you an idea here on this next slide, and I think many of you are already familiar with these different types of issues or different types of interventions, involve for instance, implementing new clinical guidelines or clinical pathways to healthcare delivery. There are obviously things like collaborative care models, where groups of individuals or groups of individuals get together to come up with better ways of course to deliver optimal healthcare to the patient. There are of course situations where we are interested in reorganizing the way healthcare is delivered, and that of course can involve many different levels in that process. And then of course managed care practice adoptions and so on.  Again, I think the point is, and I hate to hammer at it and make it sound hard, but in some sense it is, that these are complex research designs and the sampling involved with them. And so needless to say, the methodology that we need to consider is not as necessarily as straightforward as we're used to. 

Now again, just to re-emphasize what we were just saying, as far as the interventions are concerned, and again why we have a hierarchical situation, is because of the way these interventions are implemented. And they're not necessarily right directly at the patient level. They're going to be involving both individual physicians, physician groups, the larger environment, and particularly I think the third point on this slide is particularly important, that they involve across various practices, so you've got that to deal with as well. And of course it goes all the way up the leadership structure because typically when you change the way healthcare is going to be delivered at whatever level, it's going to involve management as well. 

Now, what we want to do today are essentially three things. We want to go over the key research issues, as far as the design is concerned, so we're going to talk about the randomization issues and how those are implemented. We're going to talk about the techniques for how we sample patients. We're going to talk a little bit about power and sample size, because everybody wants to know about that. First question in everybody's mind when you design a study. And finally we'll talk a little bit about the analytical methods and we’ll keep that very simple and short because that’s where it gets complicated, and that's where I don't want to lose everyone at the end of this particular discussion. And then I'll mention a couple of issues with respect to software programs. Needless to say, there are a number of programs out there that are specially suited for this particular type of situation. 

Now, let's ask the question, you know I keep using the term: it's complex, it's difficult.  And why is that? Why is there a need to do this? I mean, why is it something that we just simply can't just avoid altogether and just simply look at the way we normally look at a randomized trial, for instance? As I mentioned earlier, by just simply taking patients, randomizing them, and putting them into whatever groups, whether it's the new intervention group versus standard of care, whatever you want to call it? Well again, the first point, on this next slide, is the point that I keep emphasizing. We're not implementing things at the patient level, and that's really, I think, fundamental to the understanding of what a hierarchical design is. There are -- the work is being done at a level beyond the patient, and that's quite the contrary to what we’re normally thinking about when we’re doing health intervention trials with the patients themselves, whether it's the drug trial, whether it's a procedural trial, whatever it is. We're not making changes directly with the patient. And so you've got, that immediately introduces this hierarchical nature of your study and the hierarchical nature of the data because the outcome is going to of course it still be measured at the patient level, and we'll see some examples in a second.  But of course the effect is really -- I should say the intervention is really being incorporated into a level at a point beyond the patients themselves.  And that also reminds us that we are going to then randomize within a study like this, presuming we're doing a randomized study, at levels other than what we're measuring at. In other words, we're not going to randomize patients. We're going to randomize, for instance, physicians or we’re going to randomize institutions or even VISNs, for that matter. And the question people often ask me is, well, why do you do that? And I think one of the fundamental ideas that one has to keep in mind for that particular question is this concept of a contamination effect. What do I mean by that? Well basically, when you randomize patients, of course, then you have -- you define your two groups. And let's suppose we were going to do a process of care study where we just simply randomize patients across our institutions. Now, you can imagine the following type of scenario for instance, where two patients are sitting in a clinic, a waiting room for a clinic, waiting to see their physician, and one of them is in let’s say the intervention group and one of them is not, and they start talking to one another and discussing what's happening in terms of their care.  Of course one of them their care has been changed and the other one it has not.  And they start basically communicating about what's happening there and as a result, you can see that the two groups no longer seem to stay separate. Now, this obviously is a lot different than what you would -- what would happen for instance in a drug trial, where this kind of thing doesn't even make any sense, let alone worry about. So basically, you have to deal with that. Second of all, trying to implement multiple interventions in one facility is obviously going to be very difficult. If we're changing the way care is being delivered by some physicians and not others, you can see chaos can result because someone needs to think, am I supposed to be doing things this way or that way? I mean, where am I today in terms of these changes? And finally, it turns out that in many of our studies, which in fact don't involve necessarily randomization, there is a preferred technique. A particular facility wants to introduce this new process of care delivery system, and we can't simply say, “no, you can't do that.  You’re going to have to flip a coin to decide whether you can do that”.  So sometimes you have facilities simply saying, OK this is what we’re going to do and  other facilities don't.  And that also introduces a very important and interesting problem. Because as a result of let’s say randomizing or otherwise placing the two different interventions -- emphasizing also this doesn't have to be just two groups in these studies, but let's say we have two.  Then you're going to have just a very small number per group.  In other words, if you have let's say, 20 facilities, 10 of them are going to receive the new intervention and 10 are not.  That's a very small sample size, in spite of the fact we may have tens of thousands of patients involved.  We're going to see in a few minutes that sample size is not what you think it is. It's not necessarily exactly the number of patients that are involved in this trial. It's actually a reduced function of that, and that's simply because of the way that we randomizing here. 

Now, let's take a look in the next slide a look at some examples, some studies, a couple of which are VA trials. The first one is this Rapid Early Action for Coronary Treatment, or REACT trial. This was an interesting trial that was trying to reduce time that people would seek medical attention after a heart attack. And the intervention was a mass media campaign. In other words, basically getting on TV, newspapers, and other media formats to tell people how to -- what the warning signs were, and how to react and therefore get quicker intervention. You can imagine this would be a really hard trial to do if we tried to somehow randomize it other than the obvious unit, which is city by city. Another study involving nutritional education and making people aware -- to make people more aware of reduced fat, salt diet was -- the idea was to label menus in a restaurant with the content. And of course, I think many of you now know in some cities they are already doing this thing, including labeling menus with calories as a result of this kind of research. Now again, the obvious unit of randomization would be the restaurant, nothing short of that would make any sense. And then we have the QUITS study, which was a VA study involving smoking cessation among veterans, and intervention here was to implement the EBQI guidelines for inducing people to stop smoking. And here, what we ended up doing, because this was the only thing that again made logistical sense, was to randomize medical centers. And as I recall in this particular trial, there were about 20 medical centers that were involved so again, this was a very large trial. Approximately more than 10,000 veterans involved. But again, only 20 units were randomized. 

Now, let's talk a little bit about group randomization. It is a concept that's sometimes referred to as cluster or hierarchical random -- or similar, I should say, to cluster or hierarchical random sampling, which is a procedure that's talked about in the statistical text books. The idea here is again, you're going to randomize where the group or the place or the cluster where the unit that you're interested in, which is typically a patient, hangs out or where they're located. So as a result of that, you're not going to be able to view the data as simple random sampling.  And that is a very, very important point, because a lot of people will have historically thought about this as a good idea, they thought about the contamination effect, and they thought about well, let's randomize on the basis of physician or group or something like that, but then we can just go ahead and analyze the data as we normally do. Now, when you do, let’s say a typically something simple like a T test or a Ki squared test, you’re making a very fundamental assumption when you do that calculation.  And that is, your data are all independent units. In other words, all the individuals in a group and between groups are thought of as independent observations of whatever it is you're measuring. Now, when I say independent, it means there's no correlation between the units.  And that's not true. When you randomize patients according to a group, you kind of expect, if you will, a clustering of results. Now, we're going to talk in a few minutes and I'll show you what the implications of that are statistically, but I think you can also begin to realize that when you do that, there's going to be some question about whether or not the standard kind of statistical procedures that you use are going to be valid. And it's really interesting. We did a survey about 10 or 12 years ago before this whole concept of hierarchical design was really thought very hard -- long and hard about in the research community, particularly in the health outcomes community, and we looked at, I think it was about 70 or 80 papers that were out there – this was in the mid to late 90’s – and we found that and we found that about 80 to 90% of those papers did not account for the way the randomization or the way the design was in the way they analyzed the data, which as we'll see,  can have some great consequences on the conclusions.  Now, the issue of randomizing people this way of course immediately we know that he sampling unit is not the same as the analytical unit, because the sampling unit is the institution or the clinic or the physician and the analytical unit always, at the end of the day, is still the patient. For interest in our QUITS study we’re interested in whether a given patient quit smoking or stopped for a period of time.  Even though we're not sampling patients per se, we're sampling or randomizing  the institution.  Again, that has indications for not only analysis but as we'll see in a minute for how we design our study with respect to sample size and the power calculations. And we also have situations where we may do a normal study, in the sense that we do have a typical type of collection of data from patients, randomizing maybe even at the patient level, but there may be ex post facto a realization that there could be clustering of data that may not result from the design. In other words, we think we randomize patients but maybe patients that the ones that see a particular physician versus another physician for instance --- maybe they're not independent of one another. Maybe one physician is a lot better at implementing things than another physician and so there could be clustering of the data in spite of the fact that we didn't actually design the study for that. And one of the questions people often ask is do we test for that? Do we test for clustering, which is a conditional analysis?  And I'll have more to say about that shortly. In fact people often say, even if I design the study with clustering, should I still implement it in my analysis because maybe we thought we needed to -- we designed the study this way.  Maybe it is an issue.  But what if it isn't? And of course this has implications for the power of our analysis. 

Now, one of the interesting things that comes up when you think about designing studies this way and randomizing studies this way is the usual issues that we think about in standard studies, which are matching and stratification. In other words, you have covariates. We always have covariates.  In other words, a covariate of course is any factor with our randomized unit that could correlate with outcome. In other words, for instance, in a VA study involving different institutions, are there going to be differences between institutions that are in an urban location versus a rural location? Or is there a difference between a large center and a smaller center?  These are covariates, because we worry that these particular factors could have an impact on the -- on the outcome. And so as a result, typically in a usual randomized study, we use -- typically stratification, so we will break up the population up into these various sub groupings, let's say large versus small, and then we’ll randomize on that basis. But here we have an interesting problem. We only have a few groups to randomize.  Usually, as I said, for example in the QUITS study, 10 to 20. So, can we do that? Well, we can, but we have to be extremely limited in the variables that we select. It's usually maybe one. And that's an issue. Can we pair match?  That's another strategy that people use for eliminating the effect of covariates, or at least trying to eliminate the effects of covariates. Again this is going to be difficult with a few groups because what you're saying is, you’re going to match up one by one, ok here's a small center, here's a large center. That becomes a pair and we're going to randomize within that pair. Excuse me, two small centers, two large centers, and we will randomize within that pair. Now, we know so there's a gain in power this way, because what we -- as you can see from the equation here, this is a formula that's very familiar from standard textbooks that shows you that the variance of the difference between the results within the pair is going to result in a --- that number will be smaller than the variance within a given group which is what you'd be looking at if you are looking at an unpaired design.  And that reduction is a function of the correlation between the matching factor and the outcome.  Now, there has been some work done in the literature about well, how large of values does this need to be before we go to the trouble of thinking about pair matching under these circumstances? And there's a couple of papers that you can see here that suggest that this correlation needs to be at least 0.3 or the numbers of groups has to be at least 10 to this. I think this first point about what the correlation is a little problematic, because we usually don't know. But I think the second point is a valid one. How many groups do we actually have to randomize? And I think pair matching is really only logistically sensible if you've got at least about 10 groups. Otherwise, you're just kind of flailing away to try to get a small number of groups to match up. The other issue about stratification that you may or may not know is whenever you stratify within a clinical trial or research study, you need to be able to argue that there is no stratum by outcome interaction -- by treatment outcome interaction. And what I mean by that is that the effect of treatment is consistent across the strata. That's something that we always test for when we do a stratified randomized study in a typical clinical trial, because obviously is there is an interaction -- in other words at the effect of treatment varies from stratum to stratum, you really can't pull the data and you have to analyze each of the strata separately. If you only have a handful of centers that you're doing a stratification on, this is going to be virtually impossible to detect, so the question is do you then go ahead and pull regardless because your tests for this interaction are going to be non-significant? I think the recommendation that I always give people is it's probably a good idea. Regardless of all this stuff. If there is the possibility to do this particular type of strategy in terms of how you randomize, then it's a good idea.  Because anything that you can use to reduce the inherent variation in what you are measuring is -- there usually can't be anything too bad about that. 

As you know, there are alternatives to this kind of strategy, and you can do this ex post facto. You can do post hoc stratification, or probably what more people do than anything else is you do regression adjustment for covariates using an analysis of covariates or a more sophisticated type of model where you simply say, let's define the stratification or important covariates pre hoc and then incorporate that into our statistical model post hoc.  

Now, another thing that of course is a standard issue that we worry about with regards to any study, but of course comes up in these types of studies as well, is the issue of repeated measures. Now, we always like to do multiple, particularly baseline measurements, on our units of analysis, because it improves the precision of the data. It also helps to eliminate the regression to the mean effect. If you're doing a simple pre-post design, that's the most simplistic version of this.  But if you do a more than two observations then you have a repeated measures issue to worry about, and that's not a major issue, but of course it just, in a sense, complicates the analysis a little bit. 

Now let's talk about sample size. I'm going to talk today here in a very simple way about how to talk about sample size. There are much more sophisticated ways to do this, but I think what I'm about to tell you is probably the easiest way to implement for most people, and it uses something called the inflation factor based on the design effect of clustering.  And I've referred here to a book, I don't know if it's still in print, but it's an excellent book by Donner and Klar, these are two researchers from Canada who have done an awful lot of work on the statistic issues surrounding hierarchical modeling and design, and so this is definitely a resource if you can get your hands on this look.  It's very basic and direct.  And what we're also going to assume here in the sample size consideration is that we have a fixed number of clusters or groups. In other words, you know in advance you will have this number of physicians that you’re going to randomize, or this many institutions or this many clinics or whatever it is, so this is fixed. And then what we're trying to figure out is how many patients per clusters or group or we going to collect? 

Now, here's what I was talking about as far as this inflation factor is concerned.  This was originally put out by a fairly famous sampling statistician by the name of Kish, and so this is sometimes referred to as the Kish design factor.  And you can see that it basically means that – what it refers to, or the reason it is called an inflation factor, is what it means is-- we’re going to inflate the sample size by an amount equal to this quantity right here.  Now, this quantity basically incorporates two things. One is the average number of individuals per cluster. Now, actually to be really technical, and you want to do this exact, then you could in fact incorporate instead of the average, it would be actual number per cluster, which could vary.  But to simplify this calculation, we’re going to assume just an average value and that it doesn’t change that much from group to group.  And then the real hard part, the stuff that really throws people for a loop is this other factor called roe here, which is called the inter-cluster correlation coefficient, which is really a measure of how much -- there's variation between clusters as a function of total variation. Because what we’re basically saying here is if this is close to zero, then it says essentially that the variation within clusters is essentially relatively or quite small compared to the variation between clusters, which is -- which means that essentially, this is not an issue. If this is quite large, then it says that the difference between individuals within a cluster is actually -- excuse me. I said it backwards, I’m sorry. If this is quite small, then the variation within clusters is quite large. And as a result, there is no obvious difference between individuals within a cluster versus individuals between clusters, which means essentially that there is independence between your observations. On the other hand, if this is quite large, it means this is quite small, and it says that the people within a cluster are very much alike and clearly, then you would have a cluster issue.  Now, we'll talk about how we determine this number in a minute but the important point is this is how much we're going to have to inflate out from our traditional sample size formula the required sample size. Now, notice a couple of interesting things here. One is, if you have a correlation of zero, needless to say you're dealing with a traditional type of problem because there is no inflation factor. Or if you have one person per cluster, what does that mean? It’s the same as the traditional study where we randomize individuals, not groups of individuals. And notice the significance as well of a small cluster correlation but a large M. This is going to have a dramatic impact on this inflation factor.  So what that tells us is that the ideal way, if you're going to do cluster randomization, is to have a lot of clusters and a small number of people in each cluster.  That usually isn't an option, but this formula at least tells us that that's going to help us. If we can do it that way. 

And as I said, this average number is something that we probably know. We can simply say to institutions, for randomizing physicians for instance, what's the average number of patients that you deal with as far as this particular outcome or this particular issue is concerned? For instance with smokers, what's the average number of smoking patients that you have? The inter-cluster correlation is difficult, and what we tend to use is research that's already been published. And let me show you what I mean. 

There is a lot of information out in the literature, and in fact there's some newer papers that actually have catalogued these kinds of values for the ICC, but I'm giving you some results here from various papers just to give you an idea of the size of this number.  Now most people think of a correlation, and we’re talking about a correlation here that is of some importance. When people hear an important correlation, that must mean it's .5, .6, .7, the way we used to think of a standard Pearson correlation. In point of fact, the ICC is typically quite small. As you can see here, for some papers, and you can see various outcomes here, SF36 scorers, satisfaction scores. You can see the ICCs are in fact pretty small. On the order of .01 to .05 which actually is gratifying because it means our inflation factor is not going to get ridiculously large. 

Here are some more results from the north of England.  Aberdeen, where a lot of work had been done in  collecting a lot of this data. And as I say, there are some papers out there that pretty much catalogued these things, and you can pretty much easily find these things. 

Now, what does that mean once we get this information? Now here's an illustration of a very simple problem. Comparing two means. Now, the formula that I'm showing, this part of the formula that you see here that I'm outlining with my pointer, that's the standard formula. You've got your alpha and beta  errors incorporated into this. You've got your within group variation here, and then of course the dominator is the affect that you're looking for, the difference of the means. Now, that's what you would of course use if there wasn't any clustering and of course this is also what you'd use if you're not doing any matching. But now the inflation factor gets involved here by multiplying up this value by this amount.  And so obviously, you can determine from this the number of groups that you need or vice versa the number of individuals per group by determining the sample size and then solving backwards for one or two of these things. And then of course if you match, you're going to inflate the sample size by additional amount, -- not inflate, but reduce the sample size by an amount equal to how correlated you think the covariate is with the outcome. And we usually conservatively set this equal to zero. So in point of fact when we have matching in our statistical design, we kind of ignore it and use a conservative calculation. 

Now, let's talk about the point I was making a minute ago, which is what's the efficiency of increasing the number of clusters versus increasing the number of patients per cluster? Now again, you can see here -- if we look at the variance of the difference of two means, the standard error is this and how this is a function of the various components of clustering, the average cluster size and the correlation coefficient, because obviously what we're tying to do in sample size calculations is reduce this to a minimum, or try to make it as least as small as we can logistically deal with. Now, if you increase K, the number of clusters, you can reduce the variance to zero.  So the more clusters you have, the better off you are. But if you increase the number of patients per cluster, the variance is limited according to this formula by this amount. In other words, if you have a choice and sometimes you do, sometimes you don’t -- you're better off increasing the number of clusters versus the cluster size, because you get more bang for that buck.  

Now, what happens if you have, as we usually do, a fixed number of clusters? And we’ll call that K max. In other words you have a certain number of centers that are going to participate.  You can't say, I'm going to do this calculation, and I need 50 centers, and someone turns around and says well you only have access to 10. So sometimes we have a fixed value for this. How do we then determine your sample size in order to accommodate that? Well, you again need to incorporate into the calculation the inter-cluster correlation, the calculation – well of course you have the K max, and then the denominator here is the sample size if you have no clusters. So basically what you're going to do is calculate this number, the number of people per cluster, as a function of the number of clusters that you have access to and then the sample size that you would have if you had no clustering. And of course notice there's a restriction on the number of clusters that you need to have in the study, because it has to be at least as large as your ICC and your sample size without clustering. And again, going back to what I was saying a few minutes ago, putting a number in here, what we typically do is a sensitivity analysis.  We’ll say what happens if we put in .01, or .02, but usually we knew never assume this is more than about .05 as you saw from those examples. About 90% of the time, you don't have that anymore than that amount of clustering. 

Now, what about analytical issues?  Let’s talk briefly about how we analyze the data.  There is a very very crude way to do this, and some people I've seen in the literature do it this way. You say, okay, here's your intervention, your two groups, intervention one, intervention two, and here's the data basically for all the people in the various clinics.  Clinic 11 up to clinic N and observations 1 up to whatever and so on and so forth.  One thing you can do is very simply average the patients within a given cluster. In other words, just use the average value and then compare the m different averages between the two groups. This particular approach is great in the sense that for most people, I can go back and use what I'm already used to using, like a T test or whatever. The problem with that is you wasted all this information. You've ignored all the information you've collected on all those subjects because now you've averaged it away.  Moreover, you can't do any kind of adjustment for subject level covariates.  You can do an adjustment for cluster level covariates but not subject level covariates. So you've thrown away a huge number of degrees of freedom and as a result, you're really wasting information. So clearly, we don't want to do this. 

And then of course issues concerning whether you have two time points, which oftentimes we do within the context of the analysis. What do we do? How do we have analyze the data? And then this is a real generic issue here, because this going to come up whether you have clustering or not. One way to do it is to adjust for the pre level data with respect to the post level data. This is essentially kind of ANOVA with an adjustment, so it I guess you could consider this an ANCOVA. You could also throw in other covariates in an ANCOVA , or better yet you could use a repeated measures even if you have two time points.  But the question is, what's the best approach? Because as other approaches which involve  repeated measures, even if you have two time points. I'm not sure in the literature there is a universal agreement on the best way to handle this. I think it's probably the easiest thing for most people is to do something like a simple ANCOVA under these circumstances.  

Now, what about -- let's go back to the, again ignoring any other issues, but what's the simple way to look at the analysis of let's say just the two groups in terms of their means and incorporating the clustering into all of this? And the answer is -- and this is really, really trivial because this involves completely unadjusted data. In other words we're not going to take into account covariates or matching or that sort of thing. All we're basically doing here is you notice this part of the T test that we’ve got here, and this should be X bars by the way, that's the usual T test. And now what we’re doing is we're inflating the standard error by a factor equal to, as you can see, our old familiar now design effect. The clustering effect. And you can estimate the inter-cluster correlation by looking at, by that definition I talked about earlier which is the variation between clusters, divided by the variation between clusters, plus the variation within clusters, and you can get all three -- the two values that you really need by looking at an ANOVA output.  So in other words, what you’re going to do is a one way ANOVA on your outcome as a function of cluster. And so you can estimate this. And basically, what we often do is, just to see if this matters, because sometimes people say “do we really need to do this? I don't think there's any clustering of the data”. So what we do conservatively is to do the T test with and without the clustering and see if it really makes a difference. And what we obviously will want to do is be conservative because what of easily -- what clustering does as you can see here is if it makes your T statistics smaller, and smaller means of course it's less likely to be significant. So this tends to be inevitably conservative, and so it's very likely that what you'll see under most circumstances is that either way that you do this, with or without the cluster correction, you're probably going to see the same result. Either it’s significant or it’s not significant. But occasionally you'll see a difference between the two. And if you do, the clustering really ought to be incorporated because this is taking the approach that is clearly more conservative. 

Now, let me show you an example from Kish, the man I mentioned earlier who developed this whole concept, and he was looking at a simple problem of the difference in average income between people who own homes and people who rent homes. And he basically used a cluster random sample, basically by looking at neighborhoods of homeowners and neighborhoods of renters. And there was a total of four hundred people per group, and the average cluster size was 40. So there was about 400 individuals in each of these 40 neighborhoods. The result here is interesting. The intracluster correlation is remarkable, very high.  And it makes sense. It says basically that the income levels of people who own or people who rent are very, very of easily correlated with one another. And can therefore cluster together. So the design effect here is about 2.97. Now notice here that the average income of the homeowner was $40,000 and the renter was $35,000. And with the adjusted T test according to this cluster randomization, cluster factor here, you get a T value of 1.49, P value of .14. If you had not inflated this, I should say, in other words if you had not incorporated this effect, look what happens. You get a really significant difference between these two numbers.  So that in point of fact by taking into account clustering, what this does is it inflates your variation properly, because an uninflected T test assumes random sampling, which is not the case. These people are too much alike to be considered a random sample. These people are too much alike. And as a result, the clustering really eliminates by a dramatic amount the effect of what might have been a difference here one way but is not a difference when you look at it properly.  

Now a lot of people will look at what I just did and say, nobody does that. That's just not the way we do things in practice. What we really do in practice is we do a more sophisticated model, and you’re right.  Because we're not interested in doing T tests for the most part. We’re interested in doing more sophisticated multivariate modeling. We want to take into account the fact that we may have repeated measures -- take into account the fact that we have clustering, and covariates and so on and so forth. So what we're going to do is something that is sometimes -- used to be referred to as a mixed model analysis of variance, is now often referred to as the more general version is called general estimating equations, which is a way of taking into account all of this particular -- or GEE, takes into account all of this information because now you have a repeat factor, because you have multiple observations, you have a clustering issue. And of course it's going to be nested with your intervention group. And so on and so forth. And it allows for the relationship between the outcome and the cluster -- the cluster characteristics to vary from cluster to cluster and of course we can use as I said at covariates and repeated measures. You're not going to see on the next slide, if you're wondering, a statistical demonstration or illustration of this model, because it's just not worth driving you crazy with that sort of thing. 

Another way that people have used -- and this is implemented in the program STATA -- is something called a Huber correction to the standard error, where in fact we’re going to account for all the different things, the clustering, the repeated measure or whatever, by correcting the standard error using something called the bootstrap or jackknife approach. This is an approach that doesn't assume anything about the particular type of model that you have in terms of the clustering, but allows for this correction to take place. And it's really pretty easy in STATA to implement this because there's a macro you can call, I believe it's called cluster. And you can use this within any of your regression techniques -- standard regression, logistic regression where you have binary data, Cox regression where you have survival data, and various other types of multivariate regression models, and this is really, really easy because then you're going to get regression coefficients that are accounted for in terms of their standard errors, accounting for again, this particular design. 

And then there are other approaches, and I just mention a few of them here, and of course some people out there maybe are thinking about what about Bayesian? And there are Bayesian type models that we do use, but I think the issue with a lot of these stuff unless the you're a fairly decent expert or a statistician that knows the sort of thing, these become difficult to implement, different difficult to interpret. I think the easiest for most people to do is to think about things in terms of regression models. 

Now, there is software out there that'll do this sort of thing. We mentioned STATA. There is a very well known program that some of you may be familiar with called SUDAAN. It's essentially a program that's for complex survey data, but clearly is valuable for the kinds of situations that we have here. There are some other programs, something called WESREG. I'm not sure if this is still around.  BMDP which is an old statistical package which has sort of made a comeback.  And then SAS, of course.  I mention something here about using PROC MIXED as opposed to PROC GLM.  This is something I've been told by a SAS programmer.  Since I'm not a SAS programmer I can’t swear by this so you need to set check with your nearest SAS programmer that what I'm saying here is correct. And then there's some very special programs. HLM Is probably the best known ones out there, and these are programs that are specifically designed to look at hierarchical models. And of course you can see here that some of these can deal with a lot more hierarchies. In other words, a lot more levels of your design. But I think most of us end up with situations where there is probably no more than about three, perhaps four levels. And when I'm saying that, you might have VISN, hospital, clinic, and then physician, at the worst. 

Let me just give you a couple of quick examples of how we've implemented this in VA studies. This is a study that Eve Kerr, who is now at the University of Michigan, and I worked on back about 14 years ago in which we were looking at why people are wanting to leave a managed care plan. We surveyed 120 physician groups throughout California for a particular health plan, and obtained as you can see a almost a little more than 17,000 patients.  Now, if you really -- I didn't emphasize this point earlier -- you'd think you'd have 17,196 observations, and therefore 17,195 degrees of freedom in your study, but you don't, because the clustering, obviously you have divided these patients up or selected them from 120 groups approximately, so that in point of fact, what the Kish design effect does is effectively reduce your sample size by an amount equivalent to that 17,000, divided by the effect.  So that's what you're effective sample size is.  Moreover, when you go to do a study -- excuse me, a regression model that accounts for physician group covariates, you really only have 119 degrees of freedom for that, and so that's really at the end of the day what you essentially have to work with. So if you're trying to build a really sophisticated model, you're really not dealing with a huge sample size. You're dealing with a relatively small sample size because of your desire to put covariates associated with a physician group in the model, and that has to always be kept in mind. We were looking at satisfaction scales. We looked at desire, whether or not they wanted to change their health plan, and we use this Huber regression approach based on the average number of enrollees per group. 

Here's another study that was done in Southern California, or VISN 22 group. This was looking at the rate of adherence in treatment programs in veterans with HIV, and I think the reason we mention this particular example is because we found, based on our estimate of the design effect that we had to inflate our sample size by about 50% to account for a certain amount of clustering that we wanted.  And that's the unfortunate and sort of deflating -- and I use that in the emotional sense of the word – deflating effect, deflating result when you have to go do your sample size calculations. We end up having to use the sample sizes that are a lot larger than we really would like. 

I just want to mention quickly a new design, that's called the stepped wedge design that we’re starting to look at here particularly in my institution that is a form of clustering that comes up when you have interventions that are implemented at different time points across institutions. In other words, you have a set of institutions that are going to implement let's say a new health outcome, or excuse me, health delivery system. And you're going to do that in a way -- implemented over time, I'm trying to say. And so you have repeated measures, you have baseline information, and then you're going to get data over time. So what happens is -- let me show you the next slide here.  That kind of gives you a better idea.

So you have your four institutions, and they may be at the initial time point. You're basically had one particular approach here in the stepped wedge for instance that you see up here, you can see that at time one none of the institutions have implemented this particular process of care. Then at time two, one of them has.  At time three, another one has and so on. You can see where it's called a stepped wedge.  As opposed to other types of designs where for instance a parallel group design were you randomize groups at time one or another one in which you use a crossover design, where the groups at one point are using the intervention and another point they're not.  So this is a new kind of design -- it's been in the literature for about 10 years but it's a new approach to thinking about ways to introduce a new process of care, let’s say, approach.  But in point of fact, does this over time. But of course, clustering is still part of the mix. 

And so let me just step over a couple of these things. 

Just to give you a quick example of this, and this has appeared in the literature a few years ago about how you notify your partner of patients with an STD. The standard of care was to have the public health authorities notify the partner. The intervention was that patients were given drugs and drug vouchers to give to their partners. In other words the partner was now told by the patient themselves with the idea that maybe they should consider getting treatment. This was implemented in one county in Washington and then randomly rolled out multiple counties over over time. The idea is to do it in a randomized fashion but in point of fact it doesn't necessarily always work out that way.  Sometimes institutions actually implement the intervention when they can, not when they're told to.  But the idea is this is kind of a neat way to do this, in such a way that we can introduce this in a random or even nonrandom fashion. 

This is the model.  I'll spare you the annoying details here. 

And just point out that this kind of analysis, this type of data is still going to use the same type of approaches. It's just that we've got interesting -- the way that this data is a -- for this model is used is that we get to roll out the data over time.  So that it gets a little more complicated but we can still use the same kind of methods or to give the GEE methods because we have repeated measures. We have pre-and several post observations for a given institution. 

And I think that basically covers what I wanted to talk about today, so I'm going to turn this over back to Becky Yano.  Thank you very much. 

>> Thank you, Martin.  In the interest of time, to give you some minutes for questions, I just wanted to make sure that folks understood that we had this talk arranged for the women's health research enterprise but it's obviously applicable to all of health services research, and especially implementation science because this was in response to a national needs assessment for what folks were hoping to hear more about. We're pleased to have Dr. Lee provide his expertise in this area. 

We do have a couple of applications that are going on. There's a gender sensitivity implementation study that Dr. Don Vogt at the National Center for PTSD at VA Boston and Ellen Yee at VA Albuquerque are leading to use evidence-based quality improvement to adapt, deploy and evaluate implementation effectiveness.  In this situation our work groups, are clusters, the workgroup so you see there. 

And so we're thinking about patients and their experiences in care, clustered within providers. Providers being organized with the clinics, and clinics within facilities and then within the VA networks. 

We also had a secondary analysis project that’s been completed on the impact of practice structure on the quality of care for women veterans which has a similar kind of multilevel embeddedness that we need to consider in terms of adjusting for patient clustering across these levels. 

So those are the only two examples I wanted to provide. And then I'm going to stop there and open it up for questions. 

>> Great.  Thank you. Thanks to both of you. And we do have two questions that have been written in.  And while we get to those, I also want to solicit a little feedback from our attendees. Please do take a moment to fill out these questions on your screen, as it helps us to meet your cyber seminar needs and know what topics to address. The first question is: “could you elaborate on the difference between cluster randomization and group randomization? You said that they were similar, but I'm not clear what the difference is.” 

>> Yeah. Actually, they are two ways of stating the same thing. I guess what I was actually trying to indicate was that there is a technique that we have within the sampling literature called cluster sampling.  And the concept there is analogous to what we're talking about here in the sense that when you are looking at a cluster random sample, for instance, what you're doing is you're randomly selecting groups of people in cluster randomization. What we're doing is we’re randomizing groups of people so that I may have slightly misled. What I was trying to say was that that's where the analogy was. But group and cluster randomization are basically the same thing. 

>> Great. Thank you for that answer. And the next question is: “is it reasonable to estimate the ICC using estimators of variance for within clusters by taking point estimates of the effect from various published studies and assuming the variance of the effect size between studies provides the estimate of what you will see between clusters?” 

>> That's an interesting proposition.  I've never actually -- hadn't thought that one through but of course if you're taking the data from, like, results from multiple studies, you're going to also have an additional source of variation, which is the between-study variation, I would presume, that would tend to inflate that estimate of the between group randomized -- group variation -- by even a larger amount. So I'm not sure how -- I think you'd have to figure out a way to parcel out the between-study variation portion of that, but it's an interesting proposition. It's something that I hadn't thought about before.

>> Thank you for that answer. We do have a couple more questions that were written in. If you have another moment, we can get through them real quick. 

>> Sure.  Absolutely.

>> OK.  “Is analysis different for stepped wedge designs which are randomized versus nonrandomized?” 

>> Absolutely, and that's going to be true in any randomized versus nonrandomized design. Obviously, in a randomized design, you're able to do two things. You're able to control for when the intervention is implemented and second of all, you're able to at least therefore balance the potential effects of when you implement intervention. In a nonrandomized design, it's catch-as-catch-can. What people do, when they do it, and that's going to be a function of the issue of the institution’s ability to do things, so there's going to be, you know, issues of the facility characteristics, so that in a nonrandomized design, you're going to have to throw into the pot so to speak, into your model, you're going to have to look at the effects and covariates, whatever you want to call them, of the institution and try to control for differences that randomization would otherwise have taken care of. It's the same thing when you have a randomized verses a nonrandomized trial. Many of you may be familiar with concept of propensity score where people self select which groups there in. It's the same concept here. Institutions are selecting when they want to participate, and you have to account for that somehow in the way when you get to model the effective intervention. 

>> Excellent. Thank you. The final question is: “how about SPFS for software?” 

>> My impression is that SPFS is not as advanced as either SAS or STATA for doing this sort of thing. I've been told by some people that they’ve been able to implement some of this in SPFS but that the recommendation is to use either SAS or STATA. They have particularly focused in on these types of models and the ability to analyze them in a complete fashion. I just don't think SPFS is necessarily as good a program in this particular regard.  Don't get me wrong. I'm not trashing SPFS. I'm saying for this particular type of application. 

>> Thank you for that.  All right. We have reached the end of today's session, so if you or Becky would like to give any concluding comments, feel free. 

>> Well, I just wanted to first of all thank everybody for their time and attention today. It's been a pleasure to be able to give this seminar for everyone. And I am certainly if anybody has any additional questions if you think about them as you go forward, I'm accessible certainly on the VA system, martin.lee@va.gov . And certainly I encourage people to really give some careful thought when they're designing their studies under these kind of conditions to really think about what the implications are of those designs and particularly in terms of things like sample size.  Importantly, and obviously, the analytical model after the fact.  But I hope this is -- has at least helped some people to pay greater attention to this particular, I think, fundamental and important problem that we have in VA studies.

>> Great. Well, once again I would like to thank you both very much for today's presentation. That does formally conclude today’s HSR&D cyber seminar.