**Transcript of Cyberseminar**

**Evidence-based Synthesis Program (ESP)**

**Using Comparative Effectiveness Review Findings in Practice: A new model**

**Presenter: Mark Helfand, MD**

**October 24, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact the ESP{ Coordinating Center at**ESP.CoordinatingCenter@va.gov**.*

Moderator:And it looks like we are just at the top of the hour here. I would like to do a quick introduction of Dr. Mark Helfand. He is a Staff Physician at the Portland VA Medical Center and Professor of Medicine and Medical Informatics and Clinical Epidemiology at Oregon Health and Science University here. He holds a Bachelor of Science and Bachelor of English Literature degrees from Stanford University, a Medical degree from the University of Illinois, and completed post-graduate training in Internal Medicine, and a Master of Science at Health Services Research at Stanford Medical School. Dr. Helfand’s research focuses on the use of systematic review to inform clinical and public policies and patient narratives to improve clinical study design. His current projects include the Coordinating Center for the VA’s Evidence-based Synthesis Program, a randomized trial of an intervention to detect and treat Orthostatic Hypertension in frail, elderly end patients, and development of a collection of patient narratives related to two topics, reintegration into society after traumatic brain injury, and diabetes. He directs the Scientific Resource Center for AHRQ’s Effective Health Care Program, and with that, I would like to turn things over to Dr. Helfand.     

Mark, do I have you on the call, we cannot hear you right now. Dr. Helfand?

Dr. Helfand:Yes, hello.

Moderator:Okay, we are ready for you.

Dr. Helfand: Okay, great. Hello, I guess as you know, I am Mark Helfand, I direct the Evidence-based Synthesis Program Coordinating Center for the VA. Before that I have been involved in what has been called evidence-based medicine since before it was called that. The learning objective for today are a little different from most in evidence-based medicine or in Systematic Reviews. They are to recognize the clinician’s role and ensuring that the evidence base is complete. And to become familiar with a recent example bone morphogenetic protein 2 which was used in final surgery for many years, as well as some other landmarks in the area of reporting bias. 

While the title of the talk is about Comparative Effectiveness Reviews, the story starts with the story of evidence-based medicine. And evidence-based medicine   one could say starts with a responsibility of clinicians. As clinicians, we want to base treatment decisions on all of the relevant clinical research data. We want to ask ourselves, when we are making a recommendation, we want to ask ourselves, what is the kind and strength of evidence I am relying on to make a recommendation to a patient. In evidence-based medicine as you know it, emerged in the 1990’s as a way of making medical decisions more focused on outcomes that are important to patients, and I am using the best available evidence to make medical decisions. 

The original principal of evidence-based medicine was decisions about individual patient care should be based on the conscience explicit and judicious use of current best evidence. And a strong motivation behind this movement was that many medical practices have been adopted despite a lack of evidence that they were better than alternatives or even safe for patients. And in many instances, better evidence eventually showed these treatments were ineffective or harmful. So the need to find and consider all the relevant evidence led to a central role in evidence-based medicine for systematic review. 

Systematic reviews are a cornerstone of what, in this talk I am calling evidence-based medicine   version one. The one that we are accustomed to. In the comparative effectiveness era we speak of systematic reviews that focus on comparing alternative treatments as comparative effectiveness reviews. But the basic features of comparative effeteness reviews are the same as those of other systematic reviews. Now someone could say they place added emphasis on meaningful comparison of patient import outcomes for different treatments, but the fundamentals are the same. Specifically, systematic reviews of the literature – are systematic to remove bias in finding and reviewing the literature. If you had to reduce the whole point of systematic reviews to one thing, that would be it. The reason for the systematic part is to remove bias in finding and reviewing sympathizing literature. 

And other key points about systematic reviews is that the emphasis the best evidence, partly based on its validity, and that is where all the attention to study quality comes in. And partly based on their relevance or how well they reflect patient concerns, people who are facing the actual outcome of alternatives. 

And finally, the third characteristic which we may overlook sometimes, is that systematic reviews should be synthetic. They should synthesize the information, not just list all the studies or characteristics. So today we are going to focus on the first characteristic, the one about removing bias, and from the beginning of evidence-based medicine, I think leaders recognize that to get all the evidence, one cannot just rely on the evidence that is easily found and most talked about. One has to go further than that. And the reason for that, or one reason for that, is that experts may underplay controversy or select only supportive evidence. 

Now for historical reasons, I guess I would say, I am going to offer you a couple examples of that, examples that I use perhaps ten years ago to illustrate what we mean by expert may underplay controversy or select only supportive evidence. So these examples come from studies of antipsychotic medication. So since later we are going to be talking about spine surgery, this may be a bit of a detour, but let me give you some examples. So this is the conclusion reproduced from an abstract of a study that compared head to head, comparative efficacy study of two anti-psychotics. And the conclusion by the investigator was that during six weeks treatment, ziprasidone and olanzapine demonstrated comparable antipsychotic efficacy. Differences favoring ziprasidone were observed in metabolic parameters. Now this is the bottom line, right. The conclusion of the abstract. Abstracts are not the whole paper, but they are certainly what people see first.

The next slide shows the findings that they are talking about regarding metabolic parameters. And this is a table from that article of the adverse events in the two groups, ziprasidone and olanzapine. And the blue arrow, if you can see that, is what they are talking about, that there was a 3.7% rate incidence of metabolic and nutritional adverse events for ziprasidone versus 10.5% for olanzapine. So what the abstract has done is select out that finding to highlight it. 

The other adverse events marked by a funny looking blobby star, are all ones where the difference favored olanzapine. Those are adverse events for body as a whole, which was 38% for ziprasidone and 29% for olanzapine, for digestive complications, for nervous system complications, for respiratory, for skin and appendage, and for urogenital. So the abstract and everybody is familiar with this, sort of pulls out a favorable finding and makes it kind of the highlight of the abstract. Comparative effective reviews are more, in evidence-based medicine, are more interested in the balance of benefits and harms for all patient important outcomes. 

Here is another example, this is a longer-term study comparing two antipsychotics. And looking at the graph, one would say they look about the same, up to 52 weeks, that would be the conclusion. The actual conclusion of the study that the investigators made was in a double blind study versus Risperidone, Geodon sustained control of positive symptoms at one year. Now this is sort of subtle thing, right. That is true but they sort of drop the idea of the comparison, perhaps because it did not really add anything to the impression you get of Geodon. So remembering that the literature is sort of rife with this kind of selective attention to what people want to highlight. Just keep that in mind because we are going to come back to that issue.

Now, the other thing that makes it hard to find all the literature is that some evidence is just hard to find. And everybody has heard a lot recently about the problem of publication bias. The – some of you may know, that for many years have used FDA reports under the Freedom of Information Act, things posted on the FDA site to find out about studies or evidence that is not easily found in the purview of literature. This is the front page of that site. But even with that resource, some data can be very hard to identify, so this is the list of the antipsychotics that were included in a review that I did in the early 2000’s. And on that site, only two of the six available atypical antipsychotics had information there. The other four – the reports of the FDA that would have told us what all the studies of those drugs found were not posted yet, and sometimes these were not posted nine years after the drug was approved. Even when you find a report for some of these things, there is redacted information. 

This is an excerpt from one of those FDA reports and it says the medical review team requested additional analysis which included data obtained from, and whatever that was, that was censored. This reviewer performed, this reviewer meaning the FDA employee, the statistician, performed the ITT last observation carried forward analysis after excluding patients and ruled by blank. So even with the FDA resource, we were not really able to get all the information. We do not know if the redacted information here would be useful, but not knowing is something inconsistent with getting all the evidence.   

Now back to the issue of evidence-based medicine . To implement this idea of evidence-based medicine  and relying on the literature and getting all of the relevant issue, the medical schools and other educational entities implemented training and critical appraisal. And you probably all remember the McMaster JAMA series because most people have had, at some point, journal clubs teaching them how to look at journal article in this way. And so this is from the JAMA series user guide to the medical literature for looking at a study of therapy. It was actually published in 1993. Are the results of the study valid, was the assignment of patients to treatment randomized, and were all patients entered properly accounted for. And there are some more questions which I deleted to fit on the slide, but there are about six or seven that we all learned. And all of that helps us understand if the study is valid.

Then we ask what were the results, and then whether the results will help me in caring for my patient. And what is familiar about all of this I think is that we associate very strongly this kind of critical appraisal of the study design, and whether the study was randomized or not, as maybe the core of evidence-based medicine   of trying to be evidence based in our practice. In this way, evidence-based medicine   influenced the culture of medicine. And attending physicians, health officers, other health providers in practice are likely to refer to the kind and quality of evidence underlying their decisions. They are much more likely to do that than before there was evidence-based medicine and before it became entrained in clinical training and thinking. And these changes represent a marked improvement in clinical thinking, and form the basis of most efforts to improve decision-making. But there is a potential downside, and I do not want people to think that I am again evidence-based medicine, but I do want to highlight the downside and its relevance in this RHBMP-2 study that they completed and that I think is important to share.

And the downside is, that evidence-based medicine tends to encourage us to rely on the published literature because it is peer reviewed. And consequently, traditional systematic reviews which are an extension of evidence-based medicine and are a key tool in evidence-based medicine, often rely on published articles that do not adequately represent all the relevant information about treatment alternatives. At its most fundamental level, evidence-based medicine is based on the assumption that the evidence base, particularly clinical trials published in peer review journals, can be trusted. And also focuses attention on features of study design and analysis in a study, but all of that examination of those characteristics, depends on that trust. 

So what I want to talk about today, is recent developments in medical decision-making and comparative effectiveness research, and clinical epidemiology that makes the teaching materials for evidence-based medicine. And particularly this assumption about trust a bit out of date. There are two aspects to this that I think are important to consider. Both in reading the literature and in practicing medicine. The first, in teaching the fundamental principle of evidence based medicine that we should rely on the best available peer reviewed literature, in a way we minimize a fundamental point underlying comparative effectiveness. And that is that our research enterprise, the sum of all the funders and so on that conduct research, do not always ask the right question, do not always make the right comparison, can give an incomplete or bias picture of the balance of benefits and harms. Sometimes over generalized despite heterogeneity among patients, and can be undermined by the investigators biases or by the practices of journals, how they decide what to accept and what they want the authors to do with that. And so essentially by saying we are going to rely on the published literature, we are kind of forgetting that the published literature does not always serve society in the best ways, or asking and answering the questions that make the most difference to patients and clinicians. And some of you may think of like PCORI the new organization at least in part in trying to correct that. 

Now you would think of the VA research enterprise is one that also tends to correct that problem, that choose questions based on how much we need the information in order to inform good decision. But this is a fundamental criticism of research in general and is somewhat in conflict with the idea of relying heavily on what is published to make decisions. The second point has to do with the corruption of the evidence base. That is the situation where the literature, the published articles do not accurately represent what the actual research found, or even worse, the use of the peer review journal as a vehicle for deliberately misleading or presenting flawed information. And while this, we do not know how widespread this is, this is the subject of what I am going to talk about in the example that we are going to use. 

Now this illustration I probably should say a word about this. This is the illustration that accompanied the illustration on the screen, accompanied an article in science from I think, well October 4, I guess, that you may have heard about. This is where science is essentially, they did a sting operation, they sent a journal article to over 160 open access journals, the journal article, the submission had deliberate and obvious flaws in the area it was in, chemistry and sort of a clinical study derivative of mooching. They found that the study was accepted in the majority of journal despite the flaws, and the picture represents sort of looking under the peer review box. How much really should we be trusting peer review in the first place. That is not the subject of anything that I am going to say today, but it is an interesting article and it does at least peripherally get at the idea of how much we should trust with research just because it is published in the literature.

What I do want to talk about is something called the YODA Project, the Yale University Open Data Access project. The YODA project, which is led by Harlan Krumholz of Yale, this is an excerpt from his website. I will read the introduction, because usually when something is on the screen, people should read it. 

“*A new approach to evaluation and transparency*.” It says each day patients and their physician's make treatment decisions with access to only a fraction of the relevant clinical research data. Many clinical studies, including randomized trials are never published. The Yale University Open Data Access project has developed a model to facilitate access to patient level clinical research data, patient level clinical research data, to promote wider availability and independent analysis by external investigators. 

The YODA project model provides a means for rigorous and objective evaluation of clinical trial data to ensure that patients and physicians possess all necessary information about a drug or device when making treatment decisions. The process includes both coordinating independent examinations of all relevant product data by two separate qualified research groups and making all patient level clinical research data available for analysis by other external investigators. The model is designed to provide industry with confidence that the analysis will be scientifically rigorous, objective and fair. 

So the first actual project that YODA undertook, had to do with spinal surgery. As you know, spinal fusion, vertebral fusion is the most common surgery for chronic low back pain with lumbar disk degenerative condition. And its purpose is to restrict spinal motion and remove the presumed cause of pain. Spinal fusion usually use graph material from the patients iliac crest for most fusions, so the patient has to undergo essentially two operations at the same time. One to remove bone from the iliac crest, and the other to fuse the vertebrae and to use that bone from the iliac crest to promote the formation of new bone that will fuse the vertebrae. 

In 2002, the food and drug administration approved Recombinant Human Bone Morphogenetic Protein-2, rhBMP-2 for short which is a genetically engineered protein, a biologic with bone growth stimulating properties. So that instead of using iliac pressed bone to stimulate bone growth, it could be a substitute for that bone graph in conjunction with a device implant for lumbar, particularly lumbar surgery. And so while there is a lot from the orthopedic surgery viewpoint, there were a lot of different procedures for spinal fusion and this was intended for one particular procedure among all of those. Essentially, that is the background for the project. 

Now what happened that prompted this to be evaluated further, Medtronic had the infused bone graft which I just described to be used with this cage, this LT cage which holds the rhBMP-2 in place so it does not go all over the place. Remember as an aside, it stimulates bone growth so you do not want it spilling out everywhere. And approved it for a procedure called anterior lumbar interbody fusion. But over the years it was widely used for a number of off label indications as well. And often an indication would be a different surgical approach or fusion surgery in other parts of the spine such as the cervical spine. 

Now the journal articles that reported the premarketing studies of this stuff reported few or no adverse events in those journal publications. But other studies came along and reported more or increased adverse events. And so at that point, Yale University talked with Medtronics and developed a contract which would, as I just read from their website, have two independent review groups analyze all the original data from the trials, all the original data, patient by patient, not just a systematic review of published or summarized data. And we, my group here in Oregon, was one of those two review groups. The other was in England. The two groups had minimal contact with each other, they were not allowed to collaborate. The editorial processes at the annals of internal medicine which eventually published these two papers in June, kept the review process separate, so they used separate editorial teams, so not even at the journal did they know how similar the two reviews were. All of that was part of the idea to try to see if there was a convergent or divergent when independent groups looked at the data. 

So we had the data on the total of 17 trials, most of which were randomized trials compared to the usual treatment of iliac crest bone graft with spinal fusion versus rhBMP-2 with spinal fusion. We got a giant amount of documents for each trial, we got a protocol CT imaging protocols, statistical consideration, spinal report, antibody report, voluminous stuff. And then we got the individual patient data for each patient in every trial, including raw data and some things that would be called derived data, like if they used the measure to say what the overall success was, that might be a composite formula for using fusion as well as quality of life and functional results.

We also got 1,000 or so Medwatch online reporting forms and some cancer reports, anything anytime that cancer happened there was also a separate report on that from the manufacturer. And this just gives you – we also conducted a systematic review of the published literature that gives you a sense of how much literature there was about rhBMP-2. And the bold part is what we focused on, the lumbar spine. But we also did what would be other spine. We did not look at tibia or space where this material is also used for promoting fusion.

Let me give you some idea of the results of that analysis, rhBMP-2 and iliac for effective, rhBMP-2 and iliac pressed bone graph had similar outcome. So for the intermediate outcome of fusion as well as for other outcomes such as what you might call success, that is fusion along with some improvement in pain function, quality of life or other outcomes, there was really no difference in those outcomes. But the journal articles that were published based on those studies, made it seem that rhBMP-2 had a better fusion rate for functional outcome. And some of the ways they did that, the primary outcome in most of the studies was the composite overall success. And it really did not come out better. So most of the journal articles did not mention overall success or say that it was the primary outcome, they just focused on fusion.

Some of the articles stressed favorable numbers. For instance just 72% versus 68%, something like that. Even when they were not statistically significant, and kept repeating those comparisons without really emphasizing that there was no statistical significance to them. In some cases there were multiple publications and journals in different specialties, and with each new publication, there were sort of stronger claims for the superiority of rhBMP-2, even though the underlying data that the FDA has looked at would not have justified those kinds of comments. 

In one case, and when I say in one case, what I mean is for one of kinds of surgeries that was evaluated in these studies, one particular site of a multi-site study had very favorable results for rhBMP-2. So they publish those results separately, even though for the overall study and the multiple sites, there was no real difference between the two groups. But that one site study was published in a journal article, and subsequent journal articles always cited that paper even though it was not representative of the overall findings in the study. And then there was what I would call a nutty “pooled analysis.” Where pretty much they arbitrarily took certain studies and threw them together into a statistical analysis with the goal of making something look statistically significant, when really it was not from the randomized trials that were the ostensible subject, or the best evidence about this. Without going into all the statistical details, it appeared to be a very misleading analysis. 

So that was on the effectiveness side, on the safety or harm side, generally they failed to mention stopping one of the trials for safety concerns, the journal articles, when they did report retrograde ejaculation, they reported the number for the whole study. For instance, 11 cases rather than how many occurred in each group. When you added everything up there were more cancers in the rhBMP-2 groups and many adverse events. In many of the studies, adverse events were described in sort of terms that made it sound like there were not any. Terms like there were no unanticipated adverse events attributable to rhBMP-2. 

And so the recording of the adverse events in the trials, in the journal articles, was quite discrepant with what we found when we analyzed the original data. And this table, which may be hard to read, indicates some of those. So for instance, in the first publication you see there, Boden 2000, which was only 14 patients. The journal article reporting that reported six adverse events in the rhBMP-2 group and two in the control group that actually the total number of adverse events was twenty in the rhBMP-2 group and seven in the control group. And if you look at the next line, Burkus 2002, the journal article mentioned a total of nineteen adverse events, six in the rhBMP-2 groups and thirteen in the control group, but the actual data showed there were three hundred fifteen in the rhBMP-2 and two hundred seventy four adverse events in the control group. And the next two studies did not report any adverse events in the journal articles. And the last one actually reports things, the 2011 study actually reported them pretty accurately in the journal article. But most of the journal articles, and I have only shown you five of them here out of I think more like seventeen or fifteen or so, really gave the impression that there were not any adverse events or that they were very, very, few in number. Not just for the rh-BMP-2 group but for both groups. 

So the conclusions from the papers was that the journal articles overstated the benefits and minimized the harms. That there were clues in the early studies that the FDA material picked up on or that the data themselves would have told us, that were not followed up by more careful studies. If you and I were directing the research program, we might have said look, it looks like there is something there with retrograde ejaculation. Could you do a definitive trial of this, or definitive study of this so we know what the rate is, or how safe it is before we start using it. And so it leads to the conclusion, we cannot say this with certainty from what we did, but it did seem that many surgeons believe the promotional material and the peer reviewed literature, even though it contradicted the FDA’s original findings and what the data showed. What a reanalysis of the data showed. 

The editorial accompanying this, these two articles, ours and the one from YOUR written by Harlan Krumholz and others, said that currently, even the most conscientious physician's, those committed to knowing the latest literature, cannot fully understand the true risks and benefits of many treatments. Patients therefore are hampered in their ability to make truly informed decisions. In addition, missing data undermine evidence-based medicine, as recommendations based on the published literature, whether in systematic reviews, guidelines, book chapters or online resources, are not based on the totality of the evidence. To improve the care of patients, clinical trial data protocols and results need to be made more widely available and shared for public benefit.                  

So before I go onto the next section of the talk, which is the bit of background or history on this concept, I go back to the evidence based method which is picking up and reading the articles or searching carefully to find all the relevant articles. And conducting a systematic review, systematic reviews would have missed the main findings that were in the data for this intervention. And this intervention was used very lively for over ten years before this exercise was done. Systematic reviews that missed them since the systematic review is supposed to be using all the evidence, if we cannot trust that the systematic review has access to all the evidence, we really do not know what the ratings in the systematic review of the quality of the data, or the effectiveness and safety of an intervention really are. And this raises a question, if you had a choice of journal publications or of an independent analysis of the original data conducted by independent research teams, which would you rely on to decide what to do in practice. And if the answer to that question is that you would prefer to use independent analysis of original data to what is published in the journal articles, then obviously that undermines one of the basic assumption of evidence-based medicine and of the way we have conducted systematic reviews. 

So this is not coming out of the blue entirely, there are some antecedents. In 2004, you may know that there was legislation that eventually led to clinicaltrials.gov for exactly this reason, to find out what studies had been done that we might not have heard about. And that was about the time that there were concerns about what the literature, what the published literature said versus what other information was available on Vioxx, and antidepressants and suicidal thoughts in children. Those were kind of the current issues that were debated at the time. 

One of the Vioxx story that I think is familiar to everybody, it was voluntarily pulled from the market in 2004 and an individual patient data med analysis, the kind that I just showed you, confirmed what it looked like. It was the case from the biggest trial, the bigger trial. That there was an increased risk of cardiovascular thromboembolic adverse events, or any investigator reported death from any clause. That those were increased with Vioxx compared to the comparator, usually Naprosyn or another NSAID.   

About that time, this is from September 2004, it is an article, I guess I cut it off, but it was on CNN at the time. Reports about a meeting, a congressional hearing I should say, about this issue. And one of the interesting things, not the part that is circled there in blue, but right under it, it says Dr. John Hayes, Product Team Leader at Ely Lilly and Company, said a single report about a drug can number more than 400,000 pages. Flooding a website with 120,000 clinical trials may dilute the usefulness of the information said the Vice President of Glaxo Smith Kline. This concern that there is so much information, if you were going to ask systematic reviewers or clinicians to take a count of the 400,000 pages instead of the 20 or 50 or so that appear in journals, people would be overwhelmed and they would be lost. And that is probably true. And I can tell you that the number of systematic reviews we can do, if we have to do them that way, would be very few. But the other side is, if you get lost in it because it is giving you more information, then it sort of says what is in the journal article may not be giving you all the information. And in fact, the journal article, unless it is done by independent researchers who have looked at all 400,000 pages, may be incomplete even if the researchers are sincere in trying to report what they have found. So it comes down to well if there is 400,000 pages or 120,000 trials, somebody needs to look at that. But it is not really clear from this who that is. The result, as I said of this, is sort of go in the direction of clinicaltrials.gov. 

In 2008, Eric Turner published a paper that you may have seen about selective publication of the antidepressant trials and its influence on a current efficacy. And that paper contrasted what was published in journal articles to what was available from FDA reports. And essentially what it found was that 37% of the trials had not been published, 91% of the published trials had positive findings, whereas when you looked at all the trials 51% positive trials. And some journal articles described studies as positive when the underlying data, according to the FDA, was not positive in favor of the antidepressant. So it was another landmark in sort of looking at how we do systematic reviews. 

And the last example I wanted to talk about here is the Neurontin settlement in 2010. In 2002, Neurontin sales were $2.3 billion, 94% of which was off label. And in 2004, there was a $430 million Neurontin settle for false advertising, and that is well known. And that was done I believe by the Attorney Generals of State. But what I am talking about here is that in 2010, Keiser, the Keiser Health System, got a settlement on the basis that Keiser physician's and formulary committees were misled by the evidence published in peer review journals. Now this was remarkable because it was not about false advertising, it was about the journal articles, what it said in peer review journal articles, not advertising that was inconsistent with the proven benefits or the evidence about the drugs. 

The really remarkable thing about this settlement, is that in the course of her expert report, and being an expert witness in this case, Kay Dickersin at Johns Hopkins said that her expert report, usually court cases will keep this material from the public, that is the proceedings of a court case. But she said that they should be published, that that is part of the idea that transparency was important here, and so the entire set of her – the entire report that she wrote, is available to the public. And it is an eye opening report about the reporting bias in that case. And so, let me just give you a quote from her. This is where she said basically in her report, “I recommend that the documents reviewed by me, (including sealed documents) and other expert witnesses in the case, be made publically available for the education of the public, students, clinicians, payers, and other decision makers. As well as scholarly work that can be used to guide future understanding of potential changes in how drugs are marketed and used.” And that, in fact happens. There is a little website at the bottom of the slide where you can find that report in full. There was a “New England Journal of Medical” NAJM article about this, from this expert report. But I would say the original report, the one at that URL is far broader and more in depth as well. It is really well worth looking up. 

What it kind of points out, I am going to try to summarize what the bottom line of it was, is that traditionally we think of publication bias is trying to publish something that you cannot because negative findings, journals are not interested. Or trying to publish it but it takes longer because it is not that interesting to journal editors or multiple publication bias and then there are some others. And these are kind of well-known long-standing categories of publication bias. But the kind of reporting bias that is now getting more attention is deliberate preventing or delaying publication or public disclosure of study. Selective outcome reporting, picking out outcomes that are favorable and just reporting those in a study like we saw in rhBMP-2, selective analysis where you might do a bunch of different statistical operations and then kick the one that turned out the best. Selective cooling bias, which is to just sort of cherry-pick the studies you are going to pool and then call that a men analysis. And some other that all of which that she found some form of in that report. 

So that particular report provides unique material for education clinicians and others because it concerns a sensibly scientific communication, not commercial communication. The kind of communication we are taught to rely on by evidence-based medicine. Now I am not against evidence-based medicine, I am super for it. And I am looking at all this as far as what is the implications for how we can improve doing systematic reviews. And of course we make every effort to find detectable publication and recording bias. Even when FDA records are available though, additional information from internal correspondence by investigators, can change the interpretation of public evidence. And this means there is an additional level of uncertainty not uncounted for in systematic reviews, unless we have the original data or some independent analysis of the original data for our systematic review. 

And so where I want another quote on that subject, I think may go back to the Vioxx case, but Steve Nissen said, “The reality is that a deliberate fraud is extremely difficult to unearth. If scientists and companies agree to report results in a way that was not initially intended, unless you have access to original documents, it is extremely difficult to actually figure out what happened and how it happened. 

How many other examples like this are out there that we simply do not know about, that is what is frightening. So going back to evidence-based medicine 1 and picking up a journal article and asking are the results valid was the assignment of patients to treat randomized for all patients who entered the trial properly accounted for and additional questions about blinding or masking, and other quality things and what were the results and is it relevant to my practice. 

We still, and we always need to do that. But as clinicians, what we are really after from that early slide, we are really after finding out what the balance of benefits and arms are from all the relevant evidence. And so we need to incorporate into evidence based medicine, questions like what additional research do I need to see to weigh the benefits and risks of this drug versus alternative. And that might be research that has been done, but we have not seen it, or more likely research that has not been done yet. Because the questions that were asked in the research were not the patient centered questions that we might want if we were really after kind of an independent look at what the benefits and harms are. 

How do I advocate for patient centered comparative studies that answer important questions, and how do take a stand that such evidence must be produced and made public before I will adopt an unproven practice on the basis of the value proposition. And all of this get at, both of those issues of comparative effeteness, one is let us try to be advocates to get research organizations to do the research to answer questions that we want to know, that we need to know the answer to, and that patients do. And also let us have more transparency of the original data so that we have more confidence in the studies that have been done. 

I will just close by saying that I think this is true for Kay Dickersin, it was true for me, that actually doing one of these often changes our view on systematic reviews and to some extent on the breadth of evidence-based medicine. And we have a lot of people who have sort of a before/after experience once they do an individual patient date men analysis or once they look at the entire body of evidence and correspondence about it that led to the journal articles. This, by no means, I am not saying in any way, that this is confined to manufacturers. There are examples of similar problems for interventions that have nothing to do with manufacturers. What I am saying, is that we can increase our confidence if we become advocates, if we say, we are the prescribers, or we are the decision makers, here is what we want to know. And if somebody has it, demanding that we get to see it so we can make better decisions.

Thank you.

Moderator:Thank you Dr. Helfand and we actually do not have any pending questions for this time. For the audience, if you do have any questions you can type those into the Q and A box, located in the lower right hand corner of your screen. We do have a few minutes left here if you do have any question we can get those answered on the call here today. I do not see anything coming in quite yet here. And just a reminder for the audience, we did record todays call and we will be making this available on the catalog archives. We will be sending that link out to everyone as soon as we have that posted. And we do have a question in here. Dr. Helfand, what do you recommend that the VA do to support these goals? Any practical recommendations for research programs the VA could implement?

Dr. Helfand:Well first off, two things. I think the VA, this kind of thinking is not new to the VA and the VA sort of has made use over the years early on of asking for more information or certainly in its formularly processes of asking for more information than might be available from journal articles. So on the one hand – that is one issue of transparency, the VA is in a position as an institution to promote that kind of transparency. 

As far as the research program, you know when we look at sort of classic or exemplars of comparative effeteness studies that were done to really get the answers about viable alternative treatments. There are many from NIH, there are some from Industry, there are some from other sources, but the VA has a tremendous track record of selecting studies, particularly in its cooperative studies programs that really take the view of we do not know what is best from the literature that has been published, we have to do our own study to really do this. And all the way back to the studies of bypass surgery versus medical therapy for angina and other forms of coronary disease back in the 70’s and 80’s. The VA has had mechanisms for that. In a sense, those mechanisms are that any clinician or anybody in the VA can nominate such a – it is not that simple but can nominate these things. And that the source of research questions should be what is important to patients and what arises from the insights and practice of clinicians. 

So I do not have specific recommendations for R&D service except of course, I would say to expand that model and do more of it. But I do think the VA is in a good position, has been in, continues to be in a good position to kind of lead the way on these ideas.

Moderator:  Great, thank you, and that is the only question that we have received. I am thinking that if anyone had one they would have typed it in during that type. So Dr. Helfand, do you have any final remarks you wanted to make before we close things out this morning?

Dr. Helfand:No, well then I guess if I say no and the say something that really is not very good. So I mean, I guess my final remark would be this. You know, we in the systematic review area really are quite trusting, and really are not on sort of a hunt for fraud or things like that. What we do, generally, is identify and synthesize literature. What we are less comfortable with is detecting reporting bias. Detecting means something like what a detective does, and that is what we usually do, what we usually do take what we – try hard to find what we can and then synthesize it. And so this role of being more of a detective looking for signs of reporting bias, trying to figure out what the chance of it is, has become a big obsession and a big frustration for those of us who are trying to do a good job of synthesizing evidence because it is reading tea leaves. It is making presumptions about well it could be, there could be publication bias, they could have published this figure after doing it a different way. And their first choice of method and it did not work out, they could have sort of done selective analysis. And all these could ofs and might ofs and so on we are very uncomfortable with. 

And so I think that the Yale project sort of illustrates that it could be to everybody’s advantage. To have all the data out there, if you have got, after all, if you really have got something better and the data shows that, it deserve to succeed in the market, it deserves to be a more popular wide spread treatment than its alternative. And so this is way, in a sense, of improving the market. And so I think it is important to look at it like we are not comfortable being in the position of sort of hunting for stuff and never knowing if we found it or not. Clinicians should not be comfortable in the position of not knowing if the sources that they rely on are reliable. So I think we all kind of have an interest in figuring out how to do this in a way that maintains our objectivity and our fairness. 

Moderator:  Great, thank you. Dr. Helfand we did have one last question come in if you want to handle that before we finalize.

Dr. Helfand:  Sure, yes. 

Moderator:  Great, the question we have, how do you negotiate the tension between ensuring safety and effectiveness, and at the same time promoting innovation. 

Dr. Helfand:  That is a very big question, and of course it is a fundamental issue with the FDA. And nothing I have said in a way directly affects it in this sense. That if the FDA  continues to do exactly what it has been doing, in its approval process, making things available on the market, that is approving them for use in the United States for certain indications, that is almost a separate issue from whether we as clinicians, adopt that practice. Whether we start to prescribe or use that device in surgeries or otherwise, the separate issue is, an innovation in that case, is available to us. But it is our job to decide for whom if anyone is the better choice than the alternatives. 

I did touch on the antipsychotics before and the atypical antipsychotics came on the market and were very widely adopted because of their advantage with respect to movement disorders. And, it benefitted a lot of people, but it took many, many years for people to recognize fully, sort of the downside or the harms, the balance of benefits and harms for these things. And as you know, the downside, including metabolic problems and diabetes and so on, obesity and diabetes. And so in a way this just says, if you are going to have a system that allows things on the market because we want to promote innovation, that is fine. But let us, perhaps be more demanding on our side, on the demand side. And we will say, you know, what, if you have really got something better, prove it by looking at the benefits and risks that have come up in that process. Not just by emphasizing one or deemphasizing the other. And I think that a research program, whether this research is premarketing, meaning it holds up approval or post marketing means it does not hold up approval but adoption by patients and physicians may depend more on having good comparative evidence. From my viewpoint, one does not have to be in the way of the other. It does not have to be a premarketing requirement for it to be worth advocating for. 

Moderator:  Great, thank you. And that is all of the questions that we have received in here today. Dr. Helfand, I really want to thank you for putting together and presenting this session today. I know it was a little tough, we had to reschedule but it worked out very well and it was an excellent session, thank you very much. 

Dr. Helfand:  Okay.

Moderator:For the audience as you are leaving today, I will be putting up a feedback survey. If you could take a few moments to fill that we really do appreciate and read through all of your feedback. Thank you everyone for joining us for today’s spotlight on evidence-based synthesis programs cyber seminar, and we hope to see you at a future session, thank you.    

PAGE  

Page 1 of 15