**Transcript of Cyberseminar**

**HERC Cost Effectiveness Analysis Course**

**Evidence Synthesis to Derive Model Transition Probabilities**

**Presenter: Risha Gidwani, PhD**

**June 11, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact**herc@va.gov**.*

*Risha Gidwani:*  So I want to chat today about evidence synthesis for the purposes of driving model inputs for a cost effectiveness or other type of model. And today we’re going to focus mostly on meta-analysis. 

This is a schematic of a decision model and it’s just a hypothetical situation where we have one of two drugs to treat infected patients and we need to understand each of their probabilities of success and failure. This is really similar to a schematic that I presented in the last weeks lecture and just like last week when you have this model you need inputs for your transition probabilities and you need to figure out where to get those inputs. 

This is an example of table one that you’ll often see in a publication of a cost effective analysis and table one will tell you those -- the point estimates that you’re going to include in your decision model as well as an estimate of variation around that point estimate that you can use for sensitivity analyses. And you can see here that the data source could be either a single study or it could be a meta-analysis the latter of which we’re going to focus on today.

So as we spoke about last week there’s a multitude of ways to drive model input to get your transition probabilities. You can either go to the literature and get a data input from a single study and transform it into something that’s useful for your analysis or if there are multiple studies that exist in the literature you can synthesize multiple input, or multiple estimates in the literature in order to drive a singular model input for your decision model. And today we’re going to focus on meta-analysis and briefly touch on mixed treatment comparisons and meta regression.

So you essentially are doing meta-analysis because multiple studies have evaluated your question of interest and now you want to create a single pooled estimate from these multiple studies. The idea behind meta-analysis is that the pooled estimate that is based on multiple studies is going to be of higher quality than the estimate that’s provided by any individual study and that’s because single studies may be too small and so they may not be well powered enough. And multiple studies are going to allow you to determine whether findings are reliable. By pooling they can also be more precise because you’re pooling a multitude of studies together and therefore you’re increasing the sample size reducing the effect that random error and producing a more precise measure of the fact. 

Additionally meta-analyses are going to allow you to explore variation between studies and assess factors that might modify treatment response. So you may be able to investigate reasons for different results across studies such as through some sort of sub-group analysis. 

This is an example of PubMed search I did just looking at the effects of vitamin C on the common cold. And you can see here that just by putting in these search terms of ascorbic acid, which is vitamin C and the common cold I found 66 randomized -- I’m sorry 66 clinical trials. So we see this a lot as there’s more and more research being published, there’s going to be more and more individual studies that are evaluating your research question of interest and that could potentially be used as input for your decision model. So the question then becomes when you’re trying to derive input for your cost effectiveness or other type of decision model which input you should select. And the answer is you should select all the inputs or all the studies that are relevant to your research question, review all of them and you may be able to synthesize these into a single pooled estimate by doing a meta-analysis. 

Before we talk about actually conducting a meta-analysis I want to give you -- just show you on this slide what the results of the meta-analysis look like so that you can sort of keep this in the back of your head as we go through the rest of the presentation. 

So this is an example of a meta-analysis published in JAMA in 2012 looking at the relationship between taking Omega-3 supplements and the risk of major cardiovascular disease events. And these actually evaluated three different types of outcomes. One of them was a mixed prevention outcome, one of them was a secondary prevention outcome and then they also looked at implantable cardiac defibrillators. And so what you can see here is that we have the raw data that’s noted in the columns indicated in the blue, underneath the blue line and each individual study is contributing raw data.

From each individual study we also get a summary statistic which here is the relative risk and then the sort of main visual output of meta-analysis is going to be a forest plot which you see here. And each study has an individual weight and so what happens is the data from each study is combined with the study weight and that is spotted out in a forest plot and what you’ll see here is that there’s going to be an overall pool of estimate from this multitude of studies, which you can see with the diamond bar. And you’ll note here that there are multiple diamond bars one for mixed prevention, one for secondary prevention, another for ICD’s and then an overall diamond bar that looks at all three different outcomes. So this is ultimately what we’re going to try to get at when we do a meta-analysis.

And we do that through a multitude of steps. So the first thing that we do in a meta-analysis is we calculate a summary statistic for each individual study. So this is an example of calculating comparative summary statistic where I have two treatments within one study, one arm looked at treatment A, another arm looked at treatment B and it looks like this might be some sort of probability of events and so I have the relative effect from each study of 30 minus 20, in this case it’s 10. And so this 10 is the summary specific that I’m going to be using as the input in my meta-analysis. 

Now when you do a meta-analysis and you extract the study specific estimate it could be particular to dichotomous data such as being an odd, an odds ratio, a relative risk, a probability, a difference in probability or it could be from continuous data; so it could be a mean or a difference in means. And it’s not just comparative data that we can extract for a meta-analysis; it can also be non-comparative data. 

So let’s say I was interested in understanding the probability of death from sepsis. Hen what I would do is I would just be getting the probability of death from sepsis from my study and extracting that study level estimate and this would be non-comparative data. Once I’ve extracted study estimate from each individual study I need to weigh that study specific estimate. So the summary statistics for studies are almost always weighted. And you can weight these estimates in a number of different ways. What’s often used in meta-analysis is the inverse variance method. But you don’t have to; you can certainly use any other type of method as well. 

The inverse variance method essentially assumes that -- or essentially insures that smaller variance studies, which are often times the larger studies, get more weight in the final pooled meta-analysis estimate. And while you don’t have to weight individual studies, most people do. Folks may have heard of the Cochrane collaboration, which is a not for profit organization that conducts high quality systematic review and meta-analyses. They put out a guide book for how to conduct high quality meta-analyses and they have a number of different methods that they recommend for weighting and we’ll go through these in greater detail in the rest of the presentation. But something to keep in mind is that all of the methods the Cochrane collaboration recommends for use do involve weighting individual studies.

Sometimes people will use quality weights in weighting studies that they okay well studies that maybe will have double blinding in their randomized control files are going to be assigned higher weight than studies that are open label trials. The Cochrane collaboration actually specifically recommends against the use of quality weights and that’s for a number of reasons. First is it’s impossible to know the true risk of bias in a study. And so they feel that better views, investigator assessment of the quality of studies and use that to exclude studies rather than weight studies and assigning them a numerical value of study quality.

So for example if you are interested only in double blind randomized control trials you may exclude any open label trials as opposed to down weighting an open label trials. And the other reason that Cochrane doesn’t like using explicit quality weights when weighting a study specific estimate is that it’s hard to know how to construct an appropriate quality score. So the quality of the study is going to be a function of a number of different variables including things like how the randomization conducted, what type of blinding was there, what was the follow-up time of interest, how homogenous was the intervention if it was something like disease management program or surgical intervention. You then have to decide how to combine all of the different variables to create a quality score. Right now there’s no currently acceptable way to do that and that’s another reason why Cochrane recommends against using the specific quality weight. 

So once you’ve extracted study specific estimate and you’ve assigned a weight to that study specific estimate which is often the inverse variance weight you then combine those to create a single pooled estimate. So the individual weighted estimates are average in order to create a pooled point estimate. Therefore meta-analysis is a computation of a weighted mean estimate and that weighted mean estimate could be of a number of different statistics. It could be a weighted mean of means if each of the individual studies in your meta-analysis are looking to have reported mean data. It could be a weighted mean of probabilities, a weighted mean of odd ratios, a weighted mean of relative risks, really any sort of statistic that’s been reported across your studies you can combine in order to create a pool of weighted mean estimate. 

So once we have the point estimate or the pooled estimate for our meta-analysis we need to calculate a variance around that pooled estimate. So meta-analysis is not just a computation of a weighted mean point estimate it’s also going to give you an estimate of variation around this mean estimate. So just like in a single study if you have mean data reported there’s going to be a variance around it. The same thing happened in the meta-analysis where you have a pooled estimate across multiple studies, that pooled estimate is also going to have some variation around it and so we need to be able to calculate that as well. 

Those are pretty much the four big steps that are involved in the meta-analysis and we’re going to go into more detail about how you actually construct the meta-analysis from start to finish but before we do that I also want to briefly touch on what meta-analysis does not do because I think this will help clarify in our minds really the process that we’re going through. So what meta-analysis does not do it does not combine individual data from each study to create an overall estimate and then calculate summary statistics. 

So what do I mean by that? Let’s say that we had 2 by 2 tables from each study. We are not going to combine 2 by 2 tables from each study to construct an overall 2 by 2 table and then calculate the summary statistics from that overall table. So here’s an example where I might have two studies and you can see a 2 by 2 table from each one of these studies. And you can see here that there are in the first study 15 people who are exposed to an intervention who also have the disease. And in the second study there were 20 people exposed to an intervention who also have this disease. So what meta-analysis is not doing is it’s not combining this 15 and 30 to create a 45 and a bigger overall cell and then calculating the relative risks from this combined 2 by 2 table. This is not what it is doing. 

What meta-analysis is doing is it’s creating a pooled estimate by calculating a summary statistic right here from each individual study; so study A has it’s own summary statistic, study B has its own summary statistic; study C has its own summary statistic. You’ll notice here that in each of these studies I’ve translated the summary statistic, the relative risk to a log relative risk and that’s because when you work with relative risk the meta-analysis you want to work on the log scale which we’ll briefly touch on later. But it’s actually the point is that you’re getting a summary statistic from each study and then you are combining those summary statistics from each individual study to create an overall summary statistic. So this is how we would do it for relative risk is you had continuous data such as the mean, you would do the same thing. You would take the mean estimate from each individual study and then you would pool across each one of those three individual estimates across your three studies in order to derive your summary for your meta-analysis. 

So let’s speak now about the actual steps that are involved in a meta-analysis. So you can see here that’ there’s a number of different steps. Each well conducted meta-analysis is going to start off with a systemic literature review then do a title an abstract review of the literature that’s been selected through step one, extract data from relevant studies, separate out observational studies and randomized control trials, convert all outcomes to the same scale, evaluate the heterogeneity of the selected studies and then actually conduct the meta-analysis. So you can see here that the actual conduct of the meta-analysis only comes after a multitude of steps before it but all of those steps are very important before you can get actually derived your pool estimate.

You can also see here that there’s only three steps here that are quantitative. The steps that are before that do not involve any sort of quantitative analysis even though your final outcome is going to be a quantitative estimate. I should also mention that before you even go through any one of these steps you should first make sure that nobody else is published a meta-analysis that is on your same question of interest. And so to do that you can go to two big sources, you can go to PubMed and search for meta-analyses of your question of interest. You can also go to the Cochrane collaboration which is www.cochrane.org, which is C-O-C-H-RA-N-E.org I’m sorry I don’t have that up here for you. 

Okay step one doing a systematic literature search. When you’re doing a systematic literature search you really are trying to get at -- trying to exhaust your search of all of the different types of literature that you could possibly be evaluating so that you make sure you’re getting any possible relevant study. So the first thing you do when you do a systematic literature search is you have to determine your inclusion and exclusion criteria a priority. So this may be things like saying I want all studies conducted after 1985 that are randomized control trials that were published in English. Or you could be even more restricted and you could like specify things like the type of patients they studied; so maybe you’re interested in looking at women of childbearing age which you define as ages 15 to 45 for example. Whatever your inclusion/exclusion criteria are you should definitely be determining them before you start any literature search. You’re then going to take your inclusion/exclusion criteria and do a database search using them and whatever search strings you use, whether those are mesh search headings that you use in PubMed or other types of search strings you want to save those search strings because you’re going to have to -- you’re often times going to have to report them in a manuscript and so you’re going to have to keep track of whether you use ands or ors or anything that’s conditional. So for example searching for women of childbearing age and an English language publication is going to be different than if you search for women of childbearing age or an English language publication and so you’ll want to keep track of that of course.

Once you have pulled up some articles from your database search and you have articles that you keep which we’ll talk about in the next step you want to also make sure that you search the reference sections of those articles because I’ve found multiple times that if you use a search string like a MeSH search string or any other search string in PubMed or another database it’s not always going to get you all of the articles that are relevant to your research question but going through the literature or the reference sighted at the articles that you find to be relevant for your analysis will often be a really good way to pick up stray articles here and there. 

Another good point is that if you are searching for data that has been evaluated and randomized control trial and you think it may have been evaluated in randomized control trial you should go to www.clinicaltrials.gov to search for randomized control trials. I’m sorry that’s showing up in yellow but this is a pretty good website. It was launched in September of 2008 and it doesn’t have data on all clinical trials because not all clinical trials are required by law to be registered here but the FDA does require all phase 2 and phase 3 drug biologic or device clinical trials to be registered here so you’re going to find many, many clinical trials here. The benefit of searching this site is that the studies that are registered here also have to report results. So let’s say there’s a new device or a new drug that’s coming out and it doesn’t actually look favorable after the randomized control trial has been conducted; so it may not be in the best interest of the drug or the device manufacturer to publish an article in a peer review journal saying the drug or device and manufacturer is not safe or effective. And so if it’s not published in a peer review journal you’re not necessarily going to know that this drug or device is not safe or effective compared to you know whatever most likely it’s been compared to placebo. However you can go to clinicaltrials.gov and they have to report those results on this website so you can get results from this website and include them in your meta-analysis.

You can also search the gray literature and the gray literature is that which is not peer review. So this could be policy memos, it could be documents from organization plays to congressional budget office, it could be documents from specialty societies, it could even be abstract from conferences. You definitely want to include these in your systematic literature search but you may for example have exclusion criteria that say that you need something to be you know a randomized study and that may or may not be the case in the gray literature. And so you might exclude this gray literature later on for whatever reason but you at least want to capture it in the first place and so that’s really the jist of what we’re trying to do is we’re trying to cast a wide net to get everything we possibly can and then in later steps we’re going to whittle down and remove studies from our big pool that we’ve created. 

The first way that we’re going to whittle down studies is to through a title and abstract review. So the first thing you want to do is read through every single title and discard those that are irrelevant and so a lot of times you’re just going to be able to exclude studies right off the bat because you realize that they’re not really studying the question of your interest. Once you’ve done that you’re going to read through all your remaining abstracts and discard the ones that are irrelevant after you’ve read through those abstracts and you don’t need to spend a ton of time on these; a good rule of thumb is spending 60 seconds per abstract.

Once you’ve whittled down the studies through title and abstract review that are irrelevant you’re going to do a full text review of the remaining studies meaning that you’re going to read each one in detail and then discard those that are irrelevant after your full text review. It’s very important here to keep track of why you discard the studies from which you did a full text review. And there’s different ways that you could do that; you could come up with your own sort of rubric for why you’re going to discard certain studies. There are also some templates out there for example the Cochrane collaboration has a risk of bias tool and you can go to their website and you can search for that risk of bias tool and that will separate essentially studies with the high, medium and low risk and you may say “This is a high risk of bias study and that’s why I’m discarding it”. 

Once you’ve done all of these things you’re going to create a PRISMA diagram which you can see here on the next slide and the PRISMA diagram is going to be something that many high quality journals are going to require you to show in -- when you’re writing up the results of your meta-analysis. And it may not always be a PRISMA diagram, it may be some other diagram that’s akin to this but this is one high quality way to do this and you can see the references right here for interest in reading more about this PRISMA diagram. The idea here is that you’re telling your readers how many records or articles you identify through searching and databases and to other sources which could be the gray literature and clinicaltrials.gov. The number of records you have after duplicates are removed, which you may find then the number of records that you screened through the title and abstract review and the number that you excluded through that title an abstract review. 

The number of full text articles you assess for eligibility and here you can see in this red box that we also need to tell our readers not just a number of full text articles excluded but also the reasons for why we’ve excluded them and so that’s important to keep track of that as you do your review of these studies because trying to go back at a later date and remember why you excluded a study is extremely difficult; so save yourself the headache and keep track of it as you’re doing it. 

Then you’re going to tell your readers after you’ve assessed your full text article why you’ve excluded certain full text articles, the number of studies that were included in your qualitative synthesis and then the number that were included in your quantitative synthesis and we’ll talk more about the qualitative versus the quantitative synthesis here. 

So once you’ve done your title and abstract review and kept track of the number of studies you’re excluding for based on titles and abstract and the number and reasons for why you’re excluding studies after doing the full text review your next step is to actually extract data from the studies that you’ve decided to keep at this point in your -- for your potential meta-analysis. So there’s a -- you can either look online for existing templates for data extraction and PRISMA has one on their website or you can create your own template. And I’ve noted here some of the variables that I often use when creating my own template. I’m not going to go into all of these in the interest of time but you can use this as a reference in the future. I will say that I often times like to create my own template that’s specific to my own research question, but this is really a matter of personal preference.

When you create a data extraction template you can use Microsoft Access, Excel, there’s the software called Redcap is soon available in the VA that you can also use to capture all of the data from the studies in data extraction templates. One thing I do want to note is that if you have multiple follow-up time here you can see the last four rows I’m talking about the outcome of interest in my treatment arms. If you have multiple follow-up times that have been recorded in your studies you want to be able to capture those outcomes and so you would note in your follow up time one what the value of the outcome was for follow up time one what the value of the variation of the outcome was, for follow up time two the value of the outcome, for follow up time two the value of the variation around the outcome and so on and so forth. So it’s just important to make sure that you’re not just recording one follow up time if it’s been -- if multiple follow up times have been reported in the study.

Data extraction can be an area in which you -- you can create a lot of work for yourself if you are not really thinking thoroughly about how to do this well. And so there’s a number of good research practices I would recommend for data extraction. So for example all categorical variables should be reported in the exact same way. So often times what you’re going to have is a multitude of research assistants that are helping you do your data extraction. So if it’s not just one person reading through all the different studies you can have people that record variables in different ways that can cause a large hassle down the road. For example one person might know a randomized control trial and RCT while another person may actually spell out those words. And when you’re actually trying to filter your study to see how many randomized control trials there are you need to make sure that all the variables have been spelled in the exact same way so that you can do your filtering properly.

What you should do is set up your data extraction template before you -- after reading through a couple articles the principle investigator, the person leading this aspect of the research is set up the data extraction template and then you should test your template with a small number of studies and revise as needed. So what I would recommend is that if you are PI that’s doing a meta-analysis that you as the PI create the template. And let’s say you have two research assistants that are doing your data extraction you should have -- you, yourself as well as each one of your research assistants should review the exact same articles. I would take three to five articles, have each one of you review that and fill in the data extraction template for those articles and then convene and compare results and you can use it as a learning experience to make sure that you’re all on the same page about how to extract data and then what you’ll often times find is that you’re missing a column in your data extraction template or you need to specify what you mean and don’t mean by a variable in your data extraction template and you can modify that template accordingly.

So otherwise you can end up with a number of different hassles. So for example what I’ve often seen is someone sub sample size as one of the variables of their data extraction template. And so sample size could mean -- the total sample size for the study or it could mean the sample size for each individual arm. What you would want to do is o an arm based sample size that you then want to have one column for arm A and one column for arm B, even if you do that you may come across the three arm trial that has arms A, B and C and so you need to have another column in your data extraction template to account for any trials that have more than two arms. 

Other examples could be things like you’re really interested in understanding the mean and standard deviation of outcomes reported in studies and you come across articles that reports mean, median and inter quartile range. In this case you have a mean but you have no estimate of variation around that mean; you have the median and estimate of variation around that median and say you want to make some decisions about which one you’re going to prioritize. In this case I would actually record all three of these things and slide that in your data extraction template knowing that you’re going to have to come back to that later and deal with that through statistical difficulty.

Once you’ve done your data extraction you want to separate out observational studies and randomized control trials and I just want to stop for a minute and have us all think about that and I have a couple questions for you. And the first question is why should we separate out the randomized control trials and observational studies and the second question is why should we conduct meta-analyses if we have observational studies? So Heidi maybe we can switch to the white board and folks can type of their answers to these two different questions.

*Moderator 1:*  So for the audience to use the white board. At the top of the screen there are some tools, there is a capital T. If you click on that T and go down to the screen, just click on the screen and you will be able to type. You don’t need to use the pencil to draw, you can click on the T and you’ll be able to type your text in. We will not be able to read it until you have clicked off of the text; so that is what will get it printed onto the screen.

*Risha Gidwani:*  All right so folks feel free to throw up your answers there. Any thoughts you have about why we should separate out randomized control trials and observational studies. All right, great. Okay and it looks like we have a lot of different answers coming up in terms of why separate out RCT’s and observational studies. We have RCT’s more rigorous, causation versus association, studies representing different levels of evidence, different validity and generalized ability, sample selection bias and [Indiscernible] bias, RCT’s being higher quality research, better able to determine these types of treatments, RCT’s better controlled by randomization, RCT’s not having bias while observational estimates are usually adjusted, observational studies having immeasurable confounding.

So yeah these are all fantastic responses. So when we are doing a meta-analysis we are extracting a relative effect so if we have from each study and so with a randomized control trial we know that if randomization was conducted appropriately that -- that at baseline the two groups are going to be balanced according to covariant that may influence the outcome. And then observational study we don’t know that. We could have groups that are very imbalanced at baseline and that imbalance at baseline is going to affect any sort of effects that we extract -- relative effects that we extract from the study. And so because there sort of -- because the randomized control trial will allow us to determine causality whereas the observational study will not as has been noted here, the causation versus the association studies we want to separate them out and not combine causation and association studies in single pooled estimate of meta-analysis. 

So then the question becomes why even conduct a meta-analysis without an observational study? Maybe we should just exclude observational studies entirely and focus our meta-analyses only on randomized controlled trials. Well as folks are pointing out here randomized control trials are limited in their generalized ability. They often times have very strict inclusion and exclusion criteria and so they may not -- the results from these were RCT’s may not apply to the clinical population of interest. That doesn’t mean of course that we don’t want evidence to guide our treatment of clinical populations that may have comorbidities or may be thicker than those folks included in the randomized control trial.

The other component of why to do a meta-analysis on the observational study is because sometimes you cannot conduct an RCT. It might be unethical to do so, you can’t for example randomized people to smoking versus not smoking but you still may want to know some pool of meta-analysis estimate of the effect of smoking on the incidents of lung cancer for example. 

Other things are things like mortality where you just need an understanding of a mortality rate and that’s just going to be observationally collected data of let’s say all caused mortality. So all right, great, Heidi can we switch back to the Power Point? Okay great, thank you.

So as we talked about before we separate out observational studies and RCT’s because observational studies are going to have systematic differences between groups that randomized control trials will not and that’s very important because we’re extracting relative effects from each study. We conduct observation -- meta-analysis on observational studies because the RCT’s may not be generalized the population that we’re evaluating for our cost effectiveness analysis question. 

So once we’ve separated out our observational studies and our randomized control trials our next step is converting all outcomes to the same scale. So if you have a number of different studies that are reporting outcomes to different scales you’re going to have to make sure that you transform all those individual study estimates into outcomes reported on a singular type of scale. So when you have -- when you’re doing meta-analysis with the ultimate goal of using inputs in a cost effectiveness or decision model you’re going to need binary outcomes and so you’re going to have to convert every single outcome of interest into something that’s binary and this may require the involvement of the PC statistician because you don’t just to convert your study as point estimates but you need to also understand variation around those drive point estimates and that can be a bit of a tricky process depending on what you’re starting -- what type of statistics you’re starting with.

Often times what you’re going to see recorded in the literature are things like odd ratio, relative risks, mean data, risk differences. When you are doing a meta-analysis with study specific estimates that are odds ratios and relative risks you want to work in the log scale, which allows you to maintain symmetry in the analysis. If you have to continuous data you can work in raw means if the data are all reported on the same scale, like a data are all reported as hemoglobin A1C that’s a singular scale so you can work with raw mean data. If data are not all reported in the same scale you should work in the standardized means and when you have risk differences you can work in the raw or the absolute scale.

Once you’ve converted all outcomes to same scale you need to evaluate the heterogeneity of your selected studies. And this is an extremely important step. If you find that you have very few studies or you have multiple studies that are of low quality or the studies you have are heterogeneous you cannot continue to the meta-analysis and you must end at your qualitative work. So you could do a qualitative synthesis of the literature but you would not be able to continue on to do a quantitative estimate of all of your literature. 

There’s two big ways that you can evaluate heterogeneity for your selective studies. There’s an informal way and a formal way. The informal way is looking at your data extraction template and going over the information that you’ve extracted to see if there’s differences in the study population and the length of follow up, the way that outcomes are measured or differences in the intervention. 

So for example if you’re evaluating a disease management program that could be implemented in a number of different ways across the studies that you’re evaluating and so if there’s a lot of heterogeneity in the way that disease management programs have been implemented you may not actually be able to do a quantitative synthetics, that’s because the programs themselves differ wildly in what they contain. So for this reason that there can be heterogeneity in the way that intervention is carried out, at times meta-analysis can be better suited to things like pharmaceuticals which are going to be consistent across settings unlike program. 

Once you’ve done your informal assessment of heterogeneity you can move on to your formal assessment of heterogeneity and when I talk about heterogeneity I’m talking about differences in effect size. One thing to keep in mind is that there’s always going to be some difference in effect size from different studies and some difference doesn’t mean that things are necessarily heterogeneous. We might consider a group of studies to be homogenous for there to be homogeneity if the differences in effect size from different studies is pretty minor we would consider it often times to be due to random variation like sampling error. We would consider a group of studies to be heterogeneous if the difference in effect sizes exceeds that which can be affected from sampling error alone and so that heterogeneity could occur when effect sizes are in different directions that’s a pretty good indication. But it could also occur when the magnitude of effect size and cross study differs.

There are statistical tests that you can use for formal assessments of heterogeneity. One of these is the Cochrane’s Q statistical test and that tests null hypothesis that the true treatment effects are the same in all of the studies against the alternative hypothesis that at least one study effect differs from the remaining study. But this test is -- has its own issues. There is low power to detect heterogeneity when you have 10 or fewer studies, which in a meta-analysis can happen pretty easily. You can have heterogeneity but fail to reject a null hypothesis. And so for this reason it’s often times recommended that you use a more generous alpha level when you are testing your null hypothesis. Often times the P value of less than .10 is recommended to be used as the significant level versus the conventional P less than .05. 

There’s also the opposite problems that if you have studies with large sample sizes you can reject the non-hypothesis even when the effect sizes don’t differ very much. And so you can have problems with failing to reject a null hypothesis inappropriately and then rejecting the null hypothesis inappropriately. So the Q statistic has its own problems, you shouldn’t put a ton of stock into the Q statistic. I bring it up here because you often times seem this reported in the literature as a way of accepting heterogeneity and I want you all to be aware of some of the limitations of this statistic.

And I should also mention that a lot of diagnostic tests have low tolerance, not just the Q test and so you kind of need to use your best judgment here. If you think you have heterogeneity then move forward as you do and handle that. 

Another statistical test that’s often times used for heterogeneity is the I2 statistic, which is actually based off of the Q statistic. And I2 tells you that percentage of total variation across your pull studies is due to heterogeneity rather than due to chance and it sort of reflects the extent of overlap and the confidence in our goals of your individual studies. A rough guide to interpreting I2 statistic is that if you have an I2 statistic of less than 25%, low heterogeneity. If you have 26 to 50% -- sorry that should say 26 instead of 25. So 26 to 50% moderate heterogeneity, something above 51% high heterogeneity. If you’re getting above 75% that is really something where you shouldn’t be trying to quantitatively pool these estimates. Sometimes people just look at the point estimate of the I2 statistic; I would encourage you to also look at the confidence intervals around the I2 statistic to really understand how much heterogeneity you might have as interpreted by this statistic, again keeping in mind that it’s based off of the Q statistic and the Q statistic itself is problematic. So really you should just use all of these as very general rules of thumb; if you think you have heterogeneity and use your best investigator judgment and then you know one might argue that that should actually trump the results of the physical test that you are running.

One thing that you can do to assess heterogeneity in a way that might be very useful is do a forest plot. So in the beginning slides you saw a forest plot as the outcome from the meta-analysis but the forest plot can also be useful sort of in an intermediate step once you have all of your summary specifics from the individual studies that you’re planning to include in your meta-analysis. It can tell you how heterogeneous studies are; it can also have an added benefit of helping you easily visualize data entry errors. This is a meta-analysis looking at the use of diuretics or in pregnancy conducted by Collins, et al and published in the British Medical Journal in 1985. 

And you can see that -- that we have relative risks so of course 1.0 is a null effect here, anything less is a reduction incident, anything more than 1.0 is an increase in incidents. And what you can see here is that there is heterogeneity in studies and one of the ways that you and see this is you can see that the confidence intervals of different studies don’t overlap. And so that to us indicates that there’s some heterogeneity that’s occurring here. You can see that this and this don’t overlap and that the Cuadros and Tervila don’t overlap, Cuadros and Campbell don’t overlap. And so this is the -- a sort of a tip off that there’s some heterogeneity. If I was in the situation and I saw this type of heterogeneity I would also actually want to go back to the data I extracted from the Cuadros study and make sure that this wasn’t just a data entry error. 

So when you are looking at your forest plot, you’re going to see -- you can potentially see some different things. So if you have consistent effect sizes across the different studies that are shown in your forest plot and that’s great and you can focus on your pooled estimate. If you see that there are variations in effect sizes you can still report the pooled estimate but you’ll probably want to note to your readers that the true estimate could be higher or lower. And if you see substantial variations in effect sizes then you’re going to want to focus on variation rather than pooled effect; so there might be -- this might be for example need to do a sub group analysis. If you have -- so let’s see -- so in summary if you have -- in order to assess whether you have heterogeneity you should be doing an informal assessment by examining your data extraction table. You should also be doing a formal assessment -- I would prioritize forest plots but I would also run Q tests and I2 statistics as well. I just wouldn’t put all of my eggs into the basket, but I would do both the informal and the formal assessment. And if you have heterogeneity that you find through these mechanisms unfortunately there’s no clear guidelines for how much heterogeneity is going to sink the ship, meaning that you can’t continue on.

I will say the excluding studies is frowned upon. You really have to have a very good reason to do so. If you do exclude studies you should test excluding these studies and including them in some analyses and see how that affects your pooled meta-analysis estimate. One of the things that -- some of the things I should say that can be useful here is you can do sub group analyses where you analyze groups of studies. Ideally you would want a group -- you would want to determine those groups a priority. You could use random effects meta-analysis or you could conduct a meta regression. 

So to recap these are the things that we’ve done so far. We’ve conducted systematic literature search, we’ve completed a title of abstract review, extracted data from the studies that we think are relative to include, separated out our randomized control trials from our observational studies, converted all of our outcomes to the same scale and we’ve evaluated heterogeneity of our studies and either found that there is no heterogeneity or that we’re going to -- we’ve decided that any heterogeneity that we’ve found is going to be handled either through subgroup analyses, random effects meta-analyses or meta regression. 

So now we get into the actual conduct of the meta-analysis. You can see here that there’s a number of different steps and we’ve spent a long time on them and that sort of reflected on what’s going to happen in reality. Most of the time that you spend on your meta-analysis project will be on doing the pre work before you actually get to do any sort of quantitative combining of your study estimates. When you conduct a meta-analysis you have two big decisions to make. The first is whether you’re going to do six or random effects meta-analysis and the second is going to be how you decide to pool your studies. 

So I’m going to go over fixed and random effects analysis quickly but in the interest of time I’m not going to spend a ton of time here but I hope that these slides will serve as references to you in the future. In a fixed effect meta-analysis we are essentially saying that the variance amongst studies is due to sampling error and we think that there is some underlying true effect. The idea is that if each study had an infinite sample size the sampling error would be zero. But of course in practice we don’t have incident sample size and that’s why we have variation across studies is due to the sampling error. But there’s some true effect going on underneath.

A random effects analysis has a totally different view point. We think that variance amongst the studies is due to both the sampling error and also because the true effect could vary from study to study and that could be because they have different participants, there are different ways intervention was administered, there are a number of different things. And the random effects meta-analysis we think that the studies are similar enough to combine but we don’t think that they’re identical. We don’t think that the true effect is exactly the same in all the studies. Instead we assume that there’s a distribution of effects and we generally assume that this distribution is a normal distribution. So in a random effects meta-analysis there’s both sampling error going on and a true variation effect sizes and we assume that the studies that we have represent a random sample of the studies in the universe.

The difference in effect -- random effect meta-analysis in terms of the output you get is that the confidence intervals are going to be narrower in your fixed effects meta-analysis but in your random effects meta-analysis. And that’s because the fixed effect meta-analysis is only looking at one type of variation that’s within study variation, whereas the random effects meta-analysis is including both within study and between study variation in its estimate and therefore we get the wider constant intervals. And of course the inference of these is different. The fixed effect meta-analysis is telling us X is the true effect, the random effects meta-analysis is telling us X or the pool to mean estimate is the mean of the effects rather than the true effect.

An important consideration here or an important thing to keep in mind is that small studies are given less weight in a random effects meta-analysis -- I’m sorry in a fixed effect meta-analysis and given more weight in a random effects meta-analysis. So the fixed effect meta-analysis assumes that the only reason we have variation in studies is due to sampling error. So it says okay because of that we can largely ignore the smaller studies because we have better information about the same effect size from the larger studies analysis. Conversely the random effects meta-analysis assumes that we’re estimating the mean of the distribution of effects and we therefore can’t discount a small study because it might have information about the effects that no other study has estimated. We also don’t want our -- in the random effects meta-analysis the larger studies to have too much of an effect on the pooled estimate. 

So random effects distribution are often times more suitable because there’s almost always going to be differences amongst studies that you see in your meta-analysis. But you should keep in mind that random effects meta-analysis even though they’re often times more suitable are not always more conservative. If smaller studies are systematically different from larger studies maybe the smaller studies have a bigger population and bigger populations demonstrate greater improvement. So smaller studies are systematically different than larger studies, and increase simulated the smaller studies in a random effects meta-analysis is going to over estimate the treatment effect. Just keep that in mind.

Once you’ve figured out whether you’re doing fixed effects meta-analysis or random effects meta-analysis you have to decide how you’re going to pool your studies. In the interest of time I’m not going to go into all of these different ways of doing things. I will just point out that there are different pooling options depending on whether you have binary data or continuous data and I’m just going to point out inverse-variance and DerSimonian and Laird. So inverse-variance is using the inverse of the variance of the study in order to find the study weight and so larger studies are going to be assigned more weight using the mechanism. An inverse-variance is what it’s called when you are using a fixed effect meta-analysis. If you’re doing a random effects meta-analysis it’s referred to as a DerSimonian and Laird method; you’re still doing an inverse-variance, it’s just with a random effects so it’s called DerSimonian and Laird after the two statisticians that created this. 

And then there’s also these other methods as well like the Knapp-Hartung, the profile likelihood and the Bayesian approach and then I want you guys to keep this in mind as well. DerSimonian and Laird is often times used in the literature as one of the most frequently used ways of combining individual studies or pooling studies that you’ll see in a meta-analysis but it’s actually often times will -- recently called into great question. Again I won’t go into this in the interest of time but there are a couple references here you can search statistics in medicine -- I’m sorry I don’t have the full reference here. But if you search these terms along with meta-analysis you’ll get some articles that come up talking about why there’s some problems with DerSimonian and Laird and inverse-variance. 

Very quickly Shuster had a series of papers in specific to medicine in 2010 where he essentially was saying that the problem with these methods is that you can’t apply them to binary data because the DerSimonian and Laird methods assume that the point estimate in the variance are independent. But when we have a binomial distribution our variance calculation actually uses the point estimate, which is P to calculate the variance. And so the variance was not independent of point estimate. And then Cornell in 2014 had some other points he raised with DerSimonian and Laird about estimating the between study variance and not assuming that we’ve done that exactly.

I will point out that DerSimonian and Laird inverse-variance methods are the default weighting method that’s used in the RevMan software which is often times required by Cochrane collaboration meta-analysis. 

So to move on to some other points. Publication bias, you guys have all probably heard about publication bias and that’s this idea that studies with significant results are more likely to be published and we’ve seen that. If that’s the case -- it’s also called the file drawer syndrome which is why we have this nice picture here. So studies with significant results are more likely to be published than studies with null results. Then the meta-analysis is going to over estimate the effect. Its larger studies are more likely to be published which is often the case and the results of larger studies are systematically different than those from smaller studies than if you’re going to random effects meta-analysis you’re going to have problems like we talked about before.

Effects in publication bias is an important part of the meta-analysis and you can do that through a funnel plot. And you essentially want to look for asymmetry in a funnel plot which is problematic. S this is an example of some funnel plots that have been published in the BMJ by Sterne et al in 2011 and you can see on the left a pretty symmetric funnel plot here And on the right you can see a pretty asymmetric funnel plot and so that asymmetry in the funnel plot is going to indicate to you that there may be some level of publication bias.

But you do have to be careful here. It’s possible for funnel plots to be asymmetric due to something other than publication bias. If the quality of the studies varies with their size and let’s say larger studies are higher quality than smaller studies and they look like they’re publication bias but the asymmetry wouldn’t be actually due to the publication bias, it would be due to the relationship of the quality of the study with their size. Also nothing is perfect here and just because the funnel plots are symmetric doesn’t mean that there is not publication bias. 

The moral of the story is if you see an asymmetric funnel plot you want to explore that, make sure that it’s not due to the quality of the studies varying with their size. If there isn’t -- if that is not the reason there’s asymmetry then you may be thinking there’s publication bias. Unfortunately just because you have the symmetric funnel plot doesn’t mean all is well and good; you may still have publication bias.

So I’m going to skip over this in the interest of time. Just a little bit more information about funnel plots here if you do have publication bias there’s a number of different things you can use to deal with that publication bias. I’ve listed some of these here like cumulative meta-analyses or by precision and then different methods that have been created by different statisticians. If you’re interested in learning more about these there is a chapter on publication bias and its Sutton et all that is fantastic and I highly recommend that you use it.

So this is our result of our meta-analysis to bring this all full circle and see what we’re actually looking at. This is a meta-analysis that was recently published in the annals of internal medicine 2014 looking at diagnostic accuracies point of care test for testing albuminuria and you can see here that they have five individual studies in their meta-analysis so this is pretty common to have this number of studies and that they have a combined estimate noted here. Here their outcome of interest is they were looking at the sensitivity and specificity of the diagnostic test and here they’ve noted the sensitivity. And what you would do here is once you’ve done your meta-analysis you would use this point -- this full point estimate as your input in your base case cost effective analysis and you would take this 95% consonance interval that’s reported alongside the pooled estimate and use that in your sensitivity analyses that you conduct on your cost effectiveness analysis.

So there’s a number of different software programs that you can use to conduct meta-analyses. I will just say that you should be careful with plug and chug software because often times you’re going to have defaults that you’re actually going to want to change and so just sort of like anything else you really want to be understanding of the underlying statistical mechanism that are being used in the software programs that you’re implementing.

So there’s a few advanced topics that I’m not going to go into in detail but I do just want to touch on them briefly so that you’re aware of these and that’s meta regression, mixed treatment comparisons and individual patient level meta-analysis. So meta regression is doing a regression that adjusts for differences of the study level. So the regression that we’re all very familiar with we’re adjusting for differences often times in a patient level, here in a meta regression if we think that there’s differences across our studies we can adjust to those differences but we’re going to be doing so at the level of the study rather than the level of the individual. 

There is -- it isn’t recommended when the number of studies is small; so we know often times that in regression there can be a rule of thumb that you need at least 10 events per co-variant in your model. In meta regression there’s no established rule but it is something to keep in mind. Another topic is mixed treatment comparison and this is a statistical method for estimating the relative treatment effective interest when it hasn’t actually been evaluated in any individual study but there might be a network of evidence you can use to evaluate this.

So here’s an example of what you might have is as one study that looks at treatment A versus placebo and from that you get an estimate of STATA placebo. You may have another treatment -- another study that looks at placebo versus treatment B and from that you get an estimate of data B placebo. What you’re actually really interested in is the estimate -- the relative difference between treatment A and treatment B which is data AB and you can evaluate that using this network of evidence and then equation that you see below. And also get an estimate of the variance surround that point estimate. Again this is an advanced topic but I just want to mention what it is that uses meta-analysis and builds upon it. 

And then there’s individual patient level meta-analysis, regular meta-analysis uses the summary statistic from each study. So if you have eight studies, you can have eight data inputs. Individual patient data meta-analysis uses the individual patient data from each study. If you have eight studies with 50 patients each you’re going to have 400 data inputs and so you’re going to have richer data and you can also have subgroup analysis.

So in summary meta-analysis is giving a single pooled estimate and variance from waiting and combining individual estimates from multiple studies. It does require a systematic literature review and when you conduct the meta-analysis you need to be cognizant of study heterogeneity, whether you want to use fixed versus random effects, how you’re going to pool individual studies and assessing and handling publication bias. For people that are interested in this further, there’s a few different references here that I would strongly recommend if you want to conduct meta-analysis. I’ve also noted in the back of these slides some reference equations of how we’re actually deriving estimates, pooled estimates from meta-analyses and so you can go through these at your leisure and hope they will serve as some point of liquidation for the folks that are more mathematically oriented.

So I think we’ve -- we’re almost up against the top of the hour. I’m sorry this has been too much information but Todd if there’s any questions I’m happy to take them now.

*Moderator 2:*  Sure and there’s a lot of information, but thank you so much for going through that. We have a couple questions. Can you describe and I think I responded to this correctly, but can you describe what determines the width of the funnel plot?

*Risha Gidwani:*  So the funnel plot is -- I’ll go back to this. Okay so the funnel plot is going to give you the scale on the X axis and so if you have studies that have wide variation in their estimates then you’re going to have a wide funnel plot. If most of your studies have really similar estimates then maybe they cluster around 1.0, maybe they go from .90 to 1.1 and you’re going to have a narrow funnel plot. 

*Moderator 2:*  And I responded -- it’s just a scatter plot of this --

*Risha Gidwani:*  Mm-hmm, yes that’s correct. With the summary estimate on X axis and then the standard error on the Y axis.

*Moderator 2:*  Great thanks. Do we use equal weight for each study irrespective of a sample size or is there a way to construct the weight?

*Risha Gidwani:*  Sure so actually most people often times use the inverse variance method and that inverse variance method is actually going to sort of take into account the study sample size. So a larger study is going to have a smaller variance and therefore the inverse of that variance is going to mean that that larger study gets more weight.

*Moderator 2:*  And then you have a question back on the funnel plots; so you have the symmetric and the asymmetric funnel plots here. If you look at the asymmetric ones there’s one outside it, why did the program choose to draw the line not at the farthest width and so is it just purely user -- sort of discretion here and can you make asymmetric things look symmetric?

*Risha Gidwani:*  You won’t make asymmetric things look symmetric; you know I don’t know what -- what software programs these authors used to actually derive their funnel plot. So I think what this person might be referring to is sort of this diagonal less line and how there’s some dots outside of it and I’m not really sure why that’s the case. It might depend on the software program but what we’re -- when we’re interested in a symmetry we’re interested in the symmetry around the mean estimate, which here is shown with a vertical dotted line. And so that’s what that’s sort of the symmetry that we’re talking about.

*Moderator 2:*  Okay. I think we’re actually over the hour. If there are more questions -- that’s all the questions right now but if there are more questions that come in -- and one of the comments Risha is this was a great talk, a lot of material. And so if people have comments we can get -- revise this talk. One person said perhaps we can break this into two different talks, one on sort of the basics and one on the advance --

*Risha Gidwani:*  That’s a good point, actually if folks would be interested what we could do is do one on sort of steps one through six of what you do before you pool things quantitatively and then do an entirely separate seminar on the quantitative pooling. Often times this is you know work -- this is a technique that’s shown throughout an entire course at graduate level institution. So I apologize if it is too much information; I think that’s a good point of feedback.

*Moderator 2:*  Yeah and there’s no perfect here; so -- but we do value your input and Heidi I don’t know if you want to -- when they click out they’ll get comments and if they feel like it’s the right level we’ll stay with it. If they think it’s too much and they really want to break it we definitely use that feedback; so thanks again Risha that was fantastic talk.

*Risha Gidwani:*  Great, thank you Todd. And folks if you’re still on board if you do think that it is too much detail please let me know if you’d prefer that we break this up into two talks or you want it as one talk but more high level information.

*Moderator 1:*  All of that can go into the feedback form that’s up on the screen right now. There is no submit button so once you have put your information in there it is captured on the back end so don’t worry about that. Risha thank you so much for taking the time to prepare and present today. Todd thank you for helping out with the Q & A portion of today’s session.

*Moderator 2:*  My pleasure.

*Moderator 1:*  And for the audience we want to thank everyone for joining us for today’s session. Our next session in the series is two weeks from today; we will be sending registration information out for that next Wednesday; so keep your eye on your email for that. But I want to thank everyone for joining us at today’s HSR&D cyber seminar and we hope to see you at a future session. Thank you.

*Risha Gidwani:*  Thanks everyone, bye bye.

*Moderator 2:*  Thanks Risha.

Page **19** of **19**