HSR&D QUERI Cyberseminar- 16 -Department of Veterans Affairs

February 9, 2012

**Alternatives to the Randomized Controlled Trial in Implementation Science**

**HSR&D QUERI Cyberseminar**

**February 9,  2012**

# Alexander S. Young, MD, MSHS

Q:  Good morning or good afternoon to everyone.  I’d like to add my welcome to you to the second session in our 2012 QUERI Implementation Research Cyberseminar Series.  

The 2012 session served two purposes in the VA QUERI implementation research program. First they’re meant to address key questions in the field of implementation science to present state of the art ideas, methods and findings.  They also represent follow-up presentations to the September 2011 implementation science training program we conducted for midlevel and advanced implementation researchers. 

The topics we’ve selected for the 2012 cyberseminar series were covered in that training program, but only in an introductory manner.  Today’s topic is observational study designs as alternatives to RCTs or studying and evaluating implementation strategies and programs.  

The presenter, Dr. Alex Young is based at the VA Greater Los Angeles Healthcare System in UCLA.  Alex directs health services research for the VISN 22 MIRECC as the research leader in our local HSR&D Center of Excellence, as well as in the mental health QUERI center.  And Alex has led a very rich and diverse portfolio of implementation and pre-implementation studies, mostly in schizophrenia using a range of designs and methods including experimental and observational methods.  Alex, thanks again for going and present today and the floor is now yours. 

Q1:  All right, Alex.  Do you have your presentation up in slideshow mode?  

Alexander S. Young, MD, MSHS:  No.  

Q1:  Just go ahead and pull up the presentation into slideshow mode and you might want to also shut down your Outlook so that your messages don’t pop up on everyone’s screen. 

Alexander Young:  Okay.  

Q1:  And then just let me know when you’re ready to go and I’ll turn the screen sharing over to you. 

Alexander Young:  Okay.  I’m ready.  

Q1:  Great.  Just go ahead and press accept. 

Alexander Young:  All right, very good, okay.  All right, well thank you, Brian, and thanks for having me on to speak today.  I’m happy to be with you.  

It’s as Brian said.  I’m going to be talking about alternatives to the randomized control trial and implementation science.  The randomized control trial is of course the predominant or leading model of research investigation when we’re studying the effectiveness or efficacy of new treatment to say research designed really has revolutionized our scientific approach to understanding of treatments and their effectiveness, and which things work and which things do not work over the past 50 years. It’s been increasingly the dominant model and has made a tremendous difference to the point where it’s really hard for many of us to imagine what life would have been like seventy-five, a hundred years ago before the randomized controlled trial was seen as the gold standard. 

And but the issue for many of us in implementation science is that the randomized controlled trial is often difficult to apply. There are barriers to its use and so it has not been able to be universally applied in implementation science in the same way it has been in clinical investigation. 

And so the talk today is really focused on alternatives to the randomized controlled trial design.  So what I’m going to do is start by talking a bit about the randomized controlled trial, why we do it, what its strengths and weaknesses are and what barriers have been encountered as we’ve tried to apply it to implementation science. 

I’m also then going to spend some time on observational methods, which I really mean here alternatives to the randomized controlled trial, other trial designs.  And what we know about which observational methods are better, which ones can or cannot consistently produce accurate results and then offer you some things to consider as you’re thinking about developing research, and research protocols and looking for designs in your studies, and then also considering whether you should believe results from studies that you see from others. 

And in this regard I’m going to present two examples.  I’m going to present two examples of large, and I think reasonably convincing, trials that are not randomized controlled trials, one of which is a cohort study of the effects of a major insurance policy change in the federal health benefit program.  And the second one is an instrumental variables analysis of the effectiveness of depression treatment in a very large quality improvement study of depression here. 

So before we start I’d like to have a poll question and, Molly, I don’t know if you can take it away for the questions?

Q1:  Yes, I can.  So what I’ve done is I’ve put up a poll question for everyone to answer.  There is a circle next to the multiple choices, so please click on the one that best describes your primary role in the VA. And we’ll give everybody plenty of time to answer and then I’ll share the results with everyone. 

So far two thirds of our attendees have responded.  All right, we’ve got about eighty-five percent response rate so I’m going to leave it up for just another second or two.  And then we should be all set.  

The answers just keep pouring in.  All right, we’ve reached almost ninety percent response rate.  I’m going to go ahead and close this poll out and I’m going to share results with everyone.  

Brian, I’m going to take back the screen for just a second so that you can also see the results.  And if you’d like to talk through those feel free. 

Q:  Okay.  Thanks.  So it sounds like we had about half of the folks on the call are researchers.  And then we have an even distribution of other folks in the VA.  So that’s good.  And we just have one more question if we can of the audience to get a sense of who’s with us today.  If we can I’m going to advance to the next question.  

Q1:  All right.  I’ve actually already just put it up.  So which best describes your research experience, have not done research, have collaborated on research, have conducted research myself, have applied for research funding, or have led funding research through a grant?

So feel free again to click the circle that best corresponds to your response.  And again we’ve already had about two thirds of people responding, so we will give everybody a few more seconds.  

All right, it looks like the responses have stopped coming in so I’m going to go ahead and close the poll now.  And I will share the results with everybody. 

Q:  Well, good.  So it looks we also have a fair distribution of people, most of whom have some research experience and sounds like a fairly even distribution of folks who have collaborated on research projects while others conducted research themselves or conducted under research. Relatively few have only applied for research funding, so that’s maybe that’s a good sign.  Maybe people are successful in getting research funding.  

So thanks, Molly.  Shall we proceed to the next? 

Q1:  Yeah.  Go ahead and take back the screen and we’ll be all set.  

Q:  Okay.  

Alexander Young:  So I guess the first question to think about is really to stop a moment and think about why we use randomized controlled trials.  It’s these are things— randomized controlled trials are I think at least many of us sort of take them for granted at this point as to that this is the model to use when conducting a clinical investigation or when to look at the assessments of something that’s a statistical standard, but I think it is worth reconsidering at least for a minute why we use these, what they accomplish for us and what the in a sense what the method advantages of them are. 

And the main point to using randomized controlled trials is that we have a common problem when we’re testing a new treatment or a new intervention that there’s an association between a confounding or a third factor that’s a confounding or an additional factor that’s associated both with the exposure and the outcome.  So for instance if you look at the effect of alcohol among cancer you find an association between drinking alcohol and lung cancer. 

However, that’s not that association is actually accounted for by a third factor of cigarette smoking, which is associated with both alcohol and lung cancer.  And so if you didn’t examine that, if we don’t consider that then you see a spurious association because of this confounding factor.  

Now there’s—and what the randomized controlled trial does is it equally distributes these confounding factors between the intervention and the control group.  And you can imagine there might be a number of different contrast that confounding factors within any given treatment.  

So maybe you would have three, or five or some number of factors that could affect both exposure to the intervention and also the outcomes.  And in the randomized controlled trial you want these factors distributed equally between your intervention and your control group so that the treatment effect is what’s being measured, the effect of the treatment itself and not of the confounding factor. 

One thing that the randomized controlled trial requires is a large enough sample size to distribute these factors between the two groups.  The RCT is in its pure form is used mostly when we don’t know what the confounding factors are to begin with.  

So if there is, if you know what the confounding factors are, for instance cigarette smoking, you can actually use variants of the RCT that assure that those are equally present in both the intervention and the control group, but often we are not sure exactly what those are and so we use randomization. 

Now the caveat here is that the sample size has to be large enough so that randomization will equally distribute the factors of interest amongst the two groups.  And it’s always hard to know exactly how many these are, but how many, how large a sample size you need, but if you have a number of different factors, maybe the [range] of three to five factors, something like that, and you want them distributed equally in your intervention and control group, you can imagine pretty easily if you had ten or twenty people in each group that by chance alone it’s pretty easy to wind up with groups that are unbalanced. 

And this impact even occurs in larger trials even with sample sizes at fifty or a hundred.  It’s not uncommon to see important factors that randomization has not completely assorted between the two groups.  And this is one of the causes of clinical trial failures that the intervention group for instance winds up being sicker in some way or has otherwise a poor outcome.  And it can make it so that the effect of the intervention is not measurable or the trial fails in the end. 

But there’s another factor that is common in randomized controlled trials and that’s that they are often blinded, meaning the term double-blinded randomized controlled trial.  And this is because knowledge of exposure can bias the valuation of the outcome.  

So if people know that the exposed group is exposed in the treatment of interest that there’s a tendency to evaluate their progress as greater for psychological reasons.  And this is why evaluation is often blinded along with a control trial. 

I’m not going to talk about this, but this is also something that is critical for implementation science.  In implementation science we can rarely fully blind the evaluators if we’re doing as to which was the implementation site, but it is often worth making some effort to separate the evaluators from the implementation, people doing the implementation so that they’re not biased by their hopes or expectations regarding the effectiveness of the intervention itself. 

Now what about applying this RCT model to implementation research?  I think the issue here is that implementation research usually occurs at the organizational level.  So we’ll take a clinic, or a practice, or a medical center, or even a state or an insurance plan and they will receive implementation. 

Then when we have—we want to have a comparison site so that there’s the controlled intervention so that temporal factors and other factors are controlled and so we know we’re looking at the effectiveness of intervention or the implementation.  However, what that means usually is that we have a reasonably small number of implementation sites.

So if you’re doing implementation for instance at a medical center level it’s often not practical to have a hundred, or 200 or 1,000 medical centers in a trial, whereas in a clinical trial you might have that many patients.  It’s hard to get that many organizations for a variety of reasons in implementation trials.  And most of the reason is related to feasibility and cost that it’s just simply impossible to conduct implementation research at that scale. 

Now for the other issue that we get in implementation science is that to conduct a pure RCT sites have to be willing to both participate in three different things.  They have to willing to participate in implementation, in research and then also be willing to randomly assign as to whether or not they would get the implementation or not, or whether it would remain in a control or a comparison group.  

Now there are many sites in my experience or locations that are not interested in any one or in one or more of these.  So they don’t want to participate in the implementation or they conduct, don’t have the capacity to conduct the research.  Or they also don’t have the ability to accept random assignments. 

And what this means is that if you restrict your sample to the sites that are willing to do all three of these you wind up with a set of sites who may not be generalizable that these may be different in systematic ways than other sites.  And the other issue, as I have said that there are when you have multiple differences between sites you would like to have enough sites to randomly balance these between the two groups.  And this is often not possible given the number of sites that we have in implementation studies. 

So if you’re studying a new intervention for instance at three or four medical centers and you have three or four medical centers as the control group then that’s a relatively small group.  And it would surprising in fact if all the differences between those groups that were important could be balanced at all with a sample size of that small, and certainly not by randomization alone.

So that’s where the challenge comes from and leads us to thinking about observational designs.  Now there are many observational different types of observational designs to choose from.  And in a sense observational designs are anything that’s not a randomized control trial.  

I’m going to go through some of them here and some of these are some general categories that people tend to think about in terms of implementation research.  The simplest design is just to observe and study associations between variables or correlations.  We’re going to—and this is the most problematic one because it’s very difficult to know all the factors that are important and the probability of having some confounding factor that’s accounting for the results is quite high.

And I’d say that most of the studies that we see that have spurious conclusions or the conclusions are not accurate in the end fall into this category of studies of associations or correlations.  This would be I think a weak design or a very dangerous design to draw conclusions from. 

The second design is to do regression analysis to take these confounders and control for them.  So for instance in the example of smoking, and alcohol and lung cancer you could identify smokers in your study, see who is smoking, who is not, how much they’re smoking, and their smoking history and then put that as a variable into your analysis and have a regression model control for it. 

This is an effective strategy if you can identify all the confounding variables, if you know what they are, in a sense if they’re measured, if they’re both known by you and they’re also measured effectively.  And there’s a sophisticated way of putting this together called propensity scoring or propensity weighting, which is basically a statistical technique for using measured confounding factors and distributing the propensity to have those amongst intervention and to intervention and control groups. 

I’m not going to discuss this at length, but it’s again this is a perfectly viable strategy for observational research that can produce quite accurate results, but it does require that the confounders are all measured.  Unfortunately often this is not the case.  

Either we don’t know what the confounders are, or we’re not sure that we do or we cannot measure them.  So there are things like illness severity that can be very hard to measure from the data that we have available.  

And there are some instances where illness severity can be measured from available data if their data sources are robust enough and include some clinical information, but there are unfortunately many instances when they cannot.  I’m going to present an example of instrumental variables, which is an analytic technique that can be effectively used with and can in a sense control for or manage unmeasured confounders or confounders that are non-measured or are not accurately measured. 

And then there’s also the final category, which are cohort designs and case control design. So this is where you have, can identify a group of people that received an intervention, and a group that did not and follow them over time.  That’s the cohort design and then compare them. 

Case-control design is a related sort of design where you identify cases, individuals who have an illness and then have an identifier matched individual that has not.  And this is the design that’s particularly useful when illnesses are relatively rare and you can’t assemble a cohort in another way.  

There are various ways of having comparison groups and I’m going to present that briefly.  People in the comparison group can be lacking, which is of course the worst scenario. The comparison group can be historical, meaning you can draw from the past and groups in the past how they did without the intervention.  This turns out to be a weak design that often produces misleading results as you’ll see later.  

And there’s also the before and after design where you can study a particular group in terms of how well they did before the intervention and after they received the intervention.  This is also a weak.... [inaudible.]

So to look at this question Concato and colleagues have written a number of articles in this area.  They have this one article, one particular article that in the New England Journal Medicine in 2000 where they compared results from randomized control trials with results from observational studies that were examining the same question.  

And so they arrived at these by looking for meta analyses in five clinical treatment areas, wound up reviewing a total of ninety-nine articles and compared them.  Now they limited the observational studies to certain types of design.  They found that early on for instance the studies that used historical controls tended to be quiet biased.  

And that was actually one of the reasons initially why randomized control trials were adopted as they were compared to studies that have used historical control designs which are often the easiest to do, but are also can be quite misleading in their results.  And then they also excluded clinical trials where there was a non randomized assignment to intervention.  

And overall, as you will see on the next slide, they found a remarkable similarity between randomized control trials and these higher quality observational studies, meaning largely cohort designs studying cohorts with and without the particular treatment of interest and their outcomes.  And you can see results here from five different clinical areas.  

The black circles are the randomized control trials and the white circles are the observational studies.  And you can see that the results are fairly well distributed.  There are some randomized control trials that are outliers, which and the—it’s interestingly the observational trials tend to be more cohesive and similar in their results to each other.  

And the randomized control trials, while the overall results are quite similar in terms of the relative risk or the odds ratio for the results, they are also presumably because of non-representative patient samples or samples of individuals, some that wound up being outliers, but the message from this was that there is reason to believe that if you have a reasonably well designed cohort study that you can produce results that are accurate and in some instances with randomized control trials.  

And there are as we will discuss later there are things to be concerned about cohort trials. Particularly we have to be alert to the issue of confounders and whether there is anything in the trial design that is creating some systematic confounding, but as the Concato study showed it does not necessarily need to be the case.  And in fact many high quality observational studies produce quite good results.   

So I am going to present two examples of studies that could not be done in a randomized control trial design, but were designed relatively recently over the past decade using observational methods, cohort or implemental variables methods.  The first of these is a study that was published by in a number of articles, but the lead one in by Goldman and colleagues in the New England Journal of Medicine in 2006.  

This is a study of mental health parity in the federal employee health benefit program.  All federal employees have one health care insurance benefit and that is the FEHB program.  What this program is, is it offers individuals many health plan choices.  There are various out-of-pocket costs and varied insurance designs ranging from fee for service plan, HMO plans, many different designs.  

This is the same health plan that employees at the VA have access to, congressmen, senators.  They all have the same health plan.  And this is the and largest employer-sponsored health insurance plan in the nation.  It has more than eight million beneficiaries and $29 billion annually in healthcare benefits.  

It’s administered by the federal government.  And what we are looking at here is mental health and implementation of mental health parity within that benefit package.  

And the story of mental health parity begins really in the 1970s.  Originally mental health and physical health insurance have been very similar.  They are both fee for service and they are both available with equally insurance coverage.  

Beginning in 1975 the mental health side and the substance abuse side of the benefit began to weaken gradually by the implementation of co-pays, the doctor bills and limits, various insurance arrangements that basically wound up that meaning when individuals developed mental health problems that they had substantially more out-of-pocket expenses for their treatments than folks with other health conditions.  

And this continued incrementally into the 1980s and 1990s where the point where the insurance for mental health and substance abuse in the private sector insurance market and also in the federal government wound up being much more limited than for other physical health conditions.  And so and as a result the total claims accounted for decreased from about eight percent to about two percent of all claims.  

So there was interest in restoring parity so that as a way of insuring that folks who develop mental health problems have insurance coverage that is meaningful and that is useful for them in terms of them getting them the treatment that they need, helping them afford it and insuring that they don’t go bankrupt as a result.   And there was reason to believe this could be done.  

There have been some other smaller trials of this in a number of states and smaller locations.  And so there’s interest in restoring it on a broad basis, but it wasn’t clear at what the cost of this would be.  There was really a concern that this might break the bank that in a sense this couldn’t be done at a feasible cost.  

And there are various concerns of mental health like there are concerns that people might overuse mental health care if it was cheap that sort of thing, but they are also countervailing issues in mental health coverage.  So for instance there are—managed care is the predominant insurance arrangement in mental health and is designed to insure that treatment is medically necessary.  

So in 1999 President Clinton directed the Office of Personnel Management to implement parity within the FEHB program, parity for both mental health and substance abuse benefits, and parity meaning equal co-pays to doctor bills and limits.  And then it was to begin on 2001.  

One of the remarkable things here was that the implementation of parity was actually accompanied by funding for research to evaluate this implementation, which as we know as researchers is often not the case.  Policy and major policy interventions are often conducted with no funding or no attention to studying of their effect.  

And if that occurs it is funded elsewhere, or some more limited scale or are often not at all.  The research questions that the Clinton and OPM had with regard to implementation of mental health parity were whether the plans complied.  So did they actually implement the parity as what is mandated?  How did it affect their benefit design?  Did it change access to mental health care in terms of particularly in terms of improving access people that needed it?  And did if affect the cost of care as a result?

And so this is an example of a study where it’s really not possible to do this implementation with a randomized control trail.  The federal health parity, the federal health benefits program is a very large insurance program.  Plans and private insurance companies compete for this business.  

They design plans to market to people in under this insurance arrangement.  And they are not in a position to randomly assign people to receiving one particular benefit or another.  Nor is the federal government when they are implementing an improvement in their benefit package particularly interested in taking and keeping some subset of people in what may be an inferior insurance arrangement for three to five years.  

So it is just not possible to have a randomized control trial here.  The design that was chosen to evaluate this was a prospective cohort design with a matched control.  So this is one of the stronger designs within the Concato model of looking at these observational designs.  

There are many different FEHB plans.  Nine of these were selected to study in terms of study the affect of parity within them.  These were selected to be diverse in terms of geographical location large enough to evaluate and that the plan was interested and willing to make their data available for evaluation.  

Now the strategy for evaluating this was to choose nine comparison plans that were matched to these FEHB plans.  So these were plans that were matched based on location and type of plan.  

So for instance if the FEHB, if the federal insurance plan was a large fee for service plan in Maryland and a similar large fee for service plan in Maryland was selected or chosen for a control site, a plan that was not participating in the FEHB program, but was in the private market.  And then these comparison plans were studied over time, over the same time period that the FEHB plans were studied.  

And the analysis was difference-in-difference analysis, meaning that we looked at the—and I was part of this team, small part of the team.  We looked at the difference in the FEHB plans over time, the difference there.  We looked at the difference in the other in the comparison plan and compared those two differences.   

And so that is the difference-in-difference approach.  And I will show you some of the results.  

In terms of the first two research questions.  Did the plans comply with the parity policy? The answer was from the researchers was yes, but all plans fully implemented the parity within their benefit design.  

Did it affect benefit design and management?  And the answer is that again yes.  Most plans enhanced their benefit design and entered into managed carve out arrangement.  

These are these behavioral health managed care organization that ensure that mental health treatment is medically necessary.  So it is not like the fee for service model where they pay every claim that someone submits, but there is actually there is a requirement that there be a treatment plan and that the treatment is not and to be necessary and not just psychotherapy for personal enhancement.  

Now in terms of the two further questions the first one was did the policy affect access to mental health and substance abuse care?  And you can see the results here.  What these are, this is difference is the column on the second and third column as the ones to focus on first.  

The plans around the left side of the screen, FFS refers to fee for service.  So you can see that there were seven fee for service plans, two HMO plans and that there were some national plans, some state specific, region specific and that is what these indicators mean here, national, northeast, west, south and so forth.  And what you see is estimates from the difference-in-difference analysis of the change in mental health and substance abuse treatment use in the parity plans compared to the similar non-parity plans, a change over time.  

And what you can see is that the difference in use was quite small.  In fact the NS refers to not significant and you can see that six, in six of these instances there was no significant change between the parity, no significant affect of parity on use of mental health or substance abuse services.  

There were some with changes such as you can see the negative 0.96 or 0.78 percent changes.  These are small changes.  These would be seen as small differences though they were significant.  

And so the conclusion here is that the changes that were seen, and if we hadn’t had the comparison group there would have appeared to be changes because they were secular changes over time and use, meaning just in society use of these services was changing, but because we had the comparison group we were able to show that in fact compared to a non fee for service, I mean non-parity plans that there was no substantial difference that there was no substantial effect of this policy on use of services.  And you can see this is the same chart with the columns on the right are dollar differences.  And these are small differences when compared to the overall cost of an insurance benefit, and or small differences overall, many of them not significant.  And so again you can see that they’re—although there were temporal changes, which I’m not showing here, when you have the comparison group and the cohort design you see that in fact the parity did not produce any substantial change in costs.

There was one effect to the parity that was seen consistently and was one of the intentions.  And that was in essence that the parity mandate resulted in this being better insurance.  That is the out-of-pocket spending, meaning how much individuals and their families had to spend out of their own money if they became ill, declined substantially for most of the plans as a result to the parity intervention.  

And so and in fact this was one of main effects that was seen of parity is that it means that folks who became severely ill and had seriously ill it might require hospitalization even or higher levels of treatment that they didn’t have to pay for those out-of-pocket and that so this was better insurance protected folks and without substantial increase in costs to the program.  So that’s and example of a cohort design.  

I am going to briefly present an example of instrumental variables analysis.  This is a different type of analytic observational strategy, but also one that is good to consider.  This is another way of dealing with this issue of confounders or selection bias. 

And so the example here we are going to look at is the treatment of depression.  So you can see that again this is the same problem that we discussed earlier, which is that if you—if x is the treatment of depression and y is depression outcomes then you can have an unmeasured variable such as u that’s associated with both having depression and also the outcome of treatment.  

And in depression for example this is the severity.  One of these factors is the severity of the illness that people have.  So if people who are persistently or severely ill tend to have—they are more likely receive treatment because they are more severely ill.  

They’re also more likely to not respond to treatment.  And so as a result if you just look at the correlation between treatment of depression and treatment outcomes in depression you would find that people with more treatment have worse outcomes.  

And this is of course because the treatments, they need the treatment more and that they are receiving more treatment because they are more severely ill, but this is a confounder that is virtually impossible to measure.  And so if you looked, didn’t pay attention to this you would draw the conclusion that treatments that we know work from randomized controlled trials make depression worse, which is of course not the case.  

So the question is how to study the effectiveness of depression with an observational design.  Now there is really there are reasons why it is very hard or impossible now to conduct a large clinical trial for depression that produces generalizable or meaningful results.

And the main reason for this is that depression treatment is very well established and widely available.  It is inexpensive and I think particularly in medications.  Antidepressant medications are clearly known to be effective.  They are very well accepted.  They are used very commonly by primary care providers for instance.  And it’s hard to imagine when someone would become depressed who would not receive access or be offered depression treatment.  

The other issue from a clinical trial perspective is that it is hard to deny, conduct, have a control group that is not receiving treatment for depression, when particularly if they are seriously ill because we know we have treatments that work.  And it’s hard ethically to deny treatment to people who are seriously ill and would benefit from it.  

So the question that results from this is when you conduct a randomized control trial now today in depression treatment who enrolls?  And the answer is the people that enroll are people that are treatment refractory, meaning that they have failed the usual depression treatments, or they are mildly ill so maybe they would want treatment, but it’s not particularly necessary, or people who are doing it for the money because they need the money, and/or they can’t afford the regular treatments or there are people who are paid to participate in these trials, and so folks can sometimes exaggerate their symptoms or focus on their symptoms in a way of participating in the trial.  

The results from this is that the response rates in clinical trials for depression have been getting smaller and smaller over the past two decades to the point now they actually are quite small and far below where we think the effectiveness of these medications are.  The implementation science problem in depression care is that we are very interested in improving treatment for depression and implementing treatments for depression and the usual care.  

The challenge is to study from a study of the effectiveness of depression is that in these instances patients and providers choose whether or not to use the treatment.  It’s not a randomized control trial design.  You increase the availability of treatments to folks with the quality and proven intervention, but in the end treatment decisions are made by people, patients and providers, and again you would expect people who are more severely ill to be more widely to receive the treatments. 

And this is where the instrumental variables analysis gets in.  And this a relatively sophisticated technique so I’m not going to get into too much detail, but suffice it to say that this requires identification of an instrumental variable, which is z in the picture here.

So z is a variable that affects receipt of the treatment, but does not directly affect treatment outcomes.  So again x is provision of treatment for depression.  Y is the depression outcome and u is the unmeasured variable, the for instance severity of illness.  

So when you’re developing and the hardest thing about instrumental variables analysis is identifying an appropriate instrument.  The essence of the analytic technique here is that by having a variable z that affects receipt of treatment but doesn’t affect outcomes you in essence have a way of creating randomization statistically within a population. 

It’s not a particularly efficient way of doing this.  There is something much more efficient just to randomize people through one or the other.  As a result of this inefficiency is that it often requires quite large sample sizes, meaning hundreds or thousands of people.  

The other issue is it’s necessary to identify an instrument.  And these are not necessarily easy to come by.  Instruments that have been more commonly accepted are used at for instance geographic proximity to treatment, so rural versus urban locations.  So folks who are for instance further away from treatment providers are less likely to receive a treatment, but there’s no reason to believe that would affect their outcome except as mediated by receipt of that treatment. 

Or in this study that I’m presenting the instrument was assignment to a quality improvement program which is something that increases the change of people getting treatment for depression, but otherwise has no direct effect on their treatment outcomes.  And this was a large study.  This is the Partners in Care study led by Ken Wells and others and published in multiple studies and multiple articles.  

There were 27,000 patients screened.  In the end 938 patients completed the follow up.  And the design here was that clinics were randomized to a quality improvement program or usual care.  

The quality improvement program increased the use of antidepressant medication and psychotherapy in these intervention clinics.  And then there were comparison clinics that were also studied.  And patients were assessed with regard to their use of treatments and clinical improvement over time. 

Now there are the overall results of the study.  You can see these are the results at baseline and six months.  The green bar is with the QI programs and the yellow bar is the usual care programs.  

You can see at baseline they’re reasonably similar, a small difference in the use of treatment at intervention and control sites.  So the percent at the bottom is the percent of folks at these sites, patients at these sites who were receiving effective treatment, either psychotherapy or medication, and/or medication for their depression.  And then you can see at six months that the QI programs produced a substantial increase in depression treatment, whereas the usual care sites actually the depression treatment rates decreased modestly.  

So and the analytic technique there was a primary study here of the effectiveness, the quality improvement intervention on increasing depression treatment, but I’m not going to talk about that, suffice it to say that it succeeded in improving depression treatment and outcomes, but the question also that was of interest was what is the effectiveness of depression treatment under usual treatment arrangements, so arrangements where again you don’t have this highly selective group of people who are actually willing to participate in a trial, but you have whole clinics where you have large numbers of patients coming in with depression, many of whom who have never been treated before.  And what is the rate of their treatment?

This instrumental variables analysis technique was used.  The instrument was assignment to the quality improvement program.  And these you can see the results.  This was published by Michael Schoenbaum and colleagues.

And the key numbers here are the row with percent with probably disorder.  And you can see this is the extent to which people wound up having a disorder at the end of the trial on the basis of whether they received appropriate treatment or not.  And this is again using the instrumental variables approach so this is not confounded by their severity of illness.

And you can see that people who were treated for depression they’re at the end their rates of appropriate, their rates of having a disorder were 23.6 percent.  Those who weren’t were 70 percent.  So this is a large effectiveness of treatment for depression, meaning reducing depression rates from 70 percent to 23 percent.  

And it’s interesting that this is actually quite similar to the numbers that were seen early on in the development of antidepressant medication when they were first being introduced and when we didn’t have this clinical trial problem of who will enroll in these trials.  This is actually the main effect that was seen.  And so this is really an interesting way of getting around the fact that we can no longer study the effectiveness of depression treatments in randomized control trials.   

Let me summarize about some of these alternative designs, alternative randomized control trials.  There are certainly weak designs, correlational to simple correlation designs, designs with no control group, designs that look at before and after comparisons in populations or historical controls.  These are all weak designs that have a very high probability of being confounded.  There are stronger designs such as cohort design, case-control, though when using these stronger designs it is still important to be alert for potential confounding factors and particularly important ones such as policy shocks over time and/or differences in patient SES or illness severity amongst the groups.  

So and when you think about which of these designs to use, you have to ask yourself  if RCT is the gold standard of implementation research, and the challenge here is that there are systematic reviews of implementation trials such as the Cochrane reviews that restrict themselves to randomized control trials.  So if you don’t have a randomized control trial you may not wind up in one.  Your study may not wind up in one of these reviews. 

On the other hand you have to ask yourself if you’re implementing, studying for example an implementation or an intervention at four medical centers, are you going to use a randomized control trial or a non-RCT design?  And I guess I would postulate that if you have just four to randomize intervention or control group that randomization makes no difference.  

And in fact you’re maybe better off matching sites so that they are similar to each other and then do a cohort study.  They will be more likely I think to distribute these confounders between the clinics than a simple randomization of four selected sites, which is almost certain to not meaningfully distribute unmeasured factors between intervention and control groups. 

However, the consequences of this approach is that is you do it randomly there is particularly among clinical researchers there is an implicit acceptance of randomized control trials where they may be skepticism of other designs.  And it may be harder to convince people that this is in fact an appropriate design and that can have some strength of the results. 

The other question to think about is if you’re conducting effectiveness research is are randomized control trials the gold standard here?  And this is the again you have researchers and particularly clinical researchers who are steeped in the RCT design.  Treatment guideline committees often restrict themselves to RCT studies.  So when they’re putting together a national treatment guideline they may not look at studies that are not randomized control trials.  The problem here is this problem that I showed you on the depression case for example is that people who enroll in these RCTs increasingly can be quite, patients can be quite atypical and care is often quite different than that received under usual care arrangements. 

The result of this has been, and many of us implementation researchers have seen this is that treatments that are found in guidelines may be impossible to actually implement because they can actually only be conducted in the context of something that looks like an RCT.  They may also not work in routine practice, again because the patient population is quite different from that seen in the trials.  And then we also have the problem that many national guidelines don’t contain treatments that either are effective or that will wind up at some point in the future and shown to be effective in RCT even though they were—even those these treatments may have been found to be quite effective in cohort designs that’s not considered as acceptable evidence. 

So I’m going to wrap up and I think we have a few minutes for questions.  I think the overall conclusion I would suggest is that of course RCTs can be convincing designs to use.  And if you have an opportunity to use an RCT where you think the results are going to be generalizable and there’s a large enough sample size then it’s a perfectly reasonable approach to use in implementation science. 

The challenge is that we often have sample sizes that are not large enough to assure that the intervention control groups are in fact balanced using randomization and that we need to turn elsewhere for methods and to need to—we also still is it’s critically important to pay attention to which observational methods you’re choosing, and if you’re looking at methods like cohort or case-control designs, which are weak designs? which are stronger cohort and case-control designs?  What are the confounders that may be there?  How are those managed?  

And then if you’re using a correlational design is the propensity weighting or regression sufficient?  Or is an instrumental variables approach necessary?  And that just because there is this is an observational study it’s still important to pay attention to inclusion criteria and the ways in which outcomes are measured.  

And also I’d also just put in a plug for qualitative methods that particularly when the sample sizes are modest, meaning five, ten, twenty it’s critical to understand, critically important to understand variation in sites and locations in terms of their organizational structures and arrangements.  And that’s where qualitative methods can be invaluable.  And that’s whether or not you have randomization.

So for that we have I think about six, seven minutes for questions.  And I think we’d be—I think you’re, Molly, can people enter the questions into their system?  Is that right?  

Q1:  Yeah.  For anybody joining us after the top of the hour that didn’t get the instructions, over on that Go to Webinar panel on the right-hand side there’s a section for questions.  Just click the plus sign next to it and you can put your question directly in.  And we will pass it along to the presenter.  

Q:  Great, Alex.  This is Brian.  Thank you again for the nice overview presentation, have four questions for you.  Let me begin with what I think may be a quick one.  

The question is whether you are aware of any guidance or any examples of the use of instrumental variables in organization level implementation studies. 

Alexander Young:  I’m not aware of that.  The challenge for organizational implementation studies would be assembling a large enough sample to use an instrumental variables approach.  I think it’s a fascinating idea, would require the use of data sets that have a large number of organizations in them.  There may be some examples out there, but I would also—I think this something that people could certainly consider and think about engaging in.  

Q:  Okay.  Next question and that is how do observational studies solve the sample size problem?  And I think the basis here is the idea that as you indicated many times the barriers in implementation studies, I think implementation research are the small numbers of sites we have available, especially at the organizational level.  And so I guess the question is, what are the advantages of an observational approach over an RCT?

Alexander Young:  Right.  And so the advantages of an observational approach are that you can leverage policy experiments, or ongoing or other changes that are occurring and study their effect.  So it’s there are...  the VA for instance is on a regular basis engages in policy both at the network at the medical center of the network and the national level.  

And there are large ends or large organizational sample sizes that are available for these.  And using observational methods allows you to study the effect of these on implementation and outcomes in a large sample size. 

Q:  Okay, great.  Next question, what about multiple baseline designs or permutation designs?  How do those fit within the scheme that you presented?

Alexander Young:  Yeah.  And there are some very interesting approaches.  There are other designs that I didn’t cover, but for instance it’s possible to have sort of roll out designs where some sites receive things earlier or later.  

So you may have if you have for instance five different sites you may start with one site and have the other one serve as the comparison, and then have the other sites sequentially implement the intervention one after another.  And this is a—I think those are perfectly reasonable design strategies and good ones to think about. 

They don’t necessarily solve the sample size problem, but if sites are reluctant to participate in a study that may not provide them with anything, meaning where they may be randomized to a control or a control group that does not receive any assistance or technical assistance, your implementation help, then that sort of design helps to alleviate those concerns and may improve the chances of a more diverse pool of sites and locations participating.  So I think those are definitely types of designs to consider.

Q:  Great.  Thank you.  And we’ll try to get in one or two others, but I should say that lots of questions are coming in.  We will try to respond by e-mail after the session because we’re not likely to get to many of these.

Next question, in the Partners in Care example how is what you described different from an RCT randomizing the level of the clinic?

Alexander Young:  That’s right.  It is I guess you could say it is an RCT randomized the level of training.  It’s not an RCT randomizing the treatment.  So and that’s an important distinction. 

So this is a study where sites were randomly assigned intervention or control based upon their location, based upon being a practice or a large clinical practice.  However, patients were not randomly assigned to treatment.  They would receive treatments at either intervention or control sites.  They were just more likely to receive the treatments at the intervention sites. 

So this is sort of a typical quality improvement or implementation research design where the control groups and the randomization occurs at the organizational level, but the treatment that you’re interested in studying is delivered in both intervention and control sites. 

Q:  And I think we may have time for one more.  This is a related question again regarding the Partners in Care study.  It was originally presented as an RCT of the QI approach, not as an instrumental variable study.  Could you discuss why this study could be conceived of either way and whether there’s a right way to consider this study?

Alexander Young:  Yes.  And yeah from a practical perspective the answer to that was that it’s interesting in talking with the investigators.  They actually conceived of this to some extent as an instrumental variables and effectiveness study initially. 

However, it’s just it wasn’t—it was more likely to be for a variety of reasons was more likely to be funded and accepted as a quality improvement or implementation trial.  So implementation and effectiveness research often go together or can occur at the same time, but there’s often it may be for a variety of reasons it may be easier to convince people who are your funders and critics of studies to accept a design that’s focused on implementation or quality improvement as opposed to one that’s focused on effectiveness, even though at the end you can study both. 

Q:  Okay.  I think again given the time limits we should probably stop here.  Just a reminder to the enhancing implementation science training program trainees that you’re invited to a one-hour open discussion session a week from today.  If anyone else is interested and is not participating go ahead and send me an e-mail.  

To the extent that we have capacity we do welcome you to participate in that session as well.  And I’ve said we will try to provide answers to some of the remaining questions by e-mail, Alex, your time permitting.  

Alexander Young:  Oh sure.  Yeah, no I’m happy to do that.  

Q:  Let me thank you again though for presenting and, Molly, any final remarks at your end?

Q1:  Yeah, just a couple.  Thank you to you both.  And I would like to plug our next QUERI session which will be on March 8.  And you can find that in the cyberseminar catalogue to register.  

And also, as Brian mentioned, I will try and disseminate answers to the remaining questions to all of you by e-mail and also you will receive a direct link to this recording tomorrow in an e-mail that you can share with colleagues.  So thank you to everyone who joined us today and this does formally conclude our presentation.  Thanks, Alex.  

Q:  Yeah.  Thank you. 

**[End of Recording]**