Evidence-based Synthesis Program 

(ESP)

Using Comparative Effectiveness Review 

Findings in Practice: A new model

Mark Helfand, Director

ESP Coordinating Center

October 2013
Learning Objectives

• Recognize the clinician’s role in ensuring 

that the evidence base is complete.

• Become familiar with rhBMP-2 (bone 

morphogenetic protein 2) and other 

landmarks in reporting bias.

2 
“EBM 1”

As clinicians, we want to base treatment 

decisions on all of the relevant clinical 

research data. 

“What is the kind and strength of the evidence 

I am relying on to make a recommendation to 

a patient?”

3 
“EBM 1”

• Systematic reviews are a cornerstone of 

“EBM 1”.

4 
Systematic literature reviews

• are systematic to remove bias in finding and 

reviewing the literature.

• Emphasize the best evidence

 Valid 

 Reflects patients’ concerns

• Synthesize, don’t just list 
Systematic literature reviews

• are systematic to remove bias in 

finding and reviewing the literature. 

Experts may underplay controversy or select only 

supportive evidence

 
Simpson et al, 2004
Simpson et al, 2004

In a double-blind study vs risperidone... 

GEODON sustained control of positive 

symptoms at 1 year 
Systematic literature reviews

• are systematic to remove bias in 

finding and reviewing the literature.

Experts may underplay controversy or select only 

supportive evidence

Some evidence may be hard to find. 

 

Clozapine not posted

risperidone (1993) not posted

olanzapine (1996) not posted

quetiapine (1997) not posted

ziprasidone (2001) posted 

aripiprazole (2002) posted 

Included Drugs
Trial 114
“EBM 1”

To implement EBM 1, medical schools and 

other entities implemented training in 

“critical appraisal.”

15
JAMA series “User’s Guide to the 

Medical Literature”

• Are the Results of the Study Valid?

 Was the assignment of patients to 

treatment randomized?

 Were all patients who entered the trial 

properly accounted for and attributed at 

its conclusion?

 .... 

• What Were the Results?

• Will the Results Help Me in Caring for My 

Patients? 16

Dec 1, 1993 JAMA
“EBM 1”

EBM tends to encourage us to rely on the 

published literature because it is peer 

reviewed.

Consequently, traditional systematic reviews, 

a key tool in evidence-based medicine, often 

rely on published articles that do not 

adequately represent all of the relevant 

information about treatment alternatives. 

17
18

4 OCTOBER 60 2013 VOL 342 SCIENCE 

www.sciencemag.org

How We Got Here 

• Medtronic’s INFUSE Bone Graft (Recombinant Human 

Bone Morphogenetic Protein-2) + LT Cage: approved for 

anterior lumbar interbody fusion 

• Widely used off-label.

• Few or no adverse events were reported in 

journal publications of Medtronic sponsored trials. 

• Other studies report more or increased adverse events.

Medtronic contracted Yale University which subcontracted 

2 independent review groups to analyze the evidence.

We were one of two review groups

Annals of internal medicine, June 18, 2013
Documents and datasets from Yale

• A total of 17 trials, n = 2090, 4 single-arm 

• Documents for each trial (> 5 feet)

o trial protocol, CT imaging protocol, statistical 

considerations 

o Final report(s), antibody report

 The final reports included individual patient 

case history for adverse event 

• Individual patient data (IPD) for each 

trial

o Raw data

o Derived data
Documents and datasets from Yale

• Documents for 1016 Medwatch online 

reporting (MDR) forms.

o Obtained from two different systems: Global 

complaint handling (GCH) and Enterprise 

Product Comment Reporting (EPCR). 

o Include only complaints that resulted in the 

submission of Documents for 1016 

Medwatch online reporting (MDR) forms.

• Cancer reports
Distribution of 174 Studies

Location RCT Cohort Intervent

ion 

Series

Case Series/ Case

Report

Lumbar 

Spine

11 21 32 15 

Tibia 4 2 8 0 

Face 7 2 18 5 

Other

Spine

1 7 16 5 

All Others 2 2 12 4 

Total 25 34 86 29
Effectiveness 

• rhBMP-2 and iliac crest bone graft had similar 

outcomes.

• Journal articles made it seem that rhBMP-2 had 

better fusion rates or functional outcomes

o “Overall success” (primary outcome) often not 

reported

o Stressed favorable numbers even when they weren’t 

statistically significant

o Multiple publications

o Publication and citation of a paper about one 

particular site not representative of overall findings

o Nutty “pooled analysis” 24
Harms 

• Failed to mention stopping one trial for safety 

concerns

• Reported one adverse event, retrograde 

ejaculation, for the whole study instead of by 

treatment group

• There were more cancers in the rhBMP-2 groups

• Adverse events described as “no unanticipated 

adverse events attributable to rhBMP-2”

25
Adverse events

26

Comparison of reported adverse events in the published trials versus adverse events in the IPD at 24 

months

Author

Trial

Study number

Patients, n Number of AE reported by published study Number of AE based on 

IPD*

rhBMP2 Control rhBMP-2 Control rhBMP-2 Control

ALIF

Boden, 2000

INFUSE®/LTCAGE® Pilot 

(Study 1)

11 3 6 (1 ileus and delay in 

gait training, 1 wound 

dehiscence, 1 low back 

pain and 3 trauma)

2 (1 ileus and delay in gait 

training, 1 urinary 

retention)

20 7

Burkus, 2002

INFUSE®/LTCAGE® Open 

Pivotal

(Study 2)

143 136 6 (6 intraoperative 

vascular)†

13 (5 intraoperative 

vascular, 8 graft side 

related)

315 274

Burkus, 2002

INFUSE®/ Bone 

Dowel Pilot

(Study 4)

24 22 0 0 40 24

Burkus, 2005‡

INFUSE®/ Bone 

Dowel Pivotal

(Study 5)

55 30 0 0 95 76

Gornet, 2011

MAVERICKTM Disc 

Pivotal§

(Study 10)

172 405 407, complete reporting 

of AE in a supplemental 

table

982, complete reporting of 

AE in a supplemental 

table

449 1139
Conclusion on rhBMP-2 

• Journal articles overstated benefits and 

minimized harms.

• Clues in the early studies were not followed 

up.

• Many surgeons believed promotional 

material and the peer reviewed literature, 

even though it contradicted the FDA’s 

findings. 

27
Antecedents 

• 2004 ClinicalTrials.gov (Vioxx, antidepressants 

and suicidal thoughts in children) 

28
Rofecoxib (Vioxx) 

• Voluntarily pulled from the market in 2004

• IPD analysis of completed placebo-controlled RCTs for 

risk of cardiovascular thromboembolic adverse events or 

any investigator-reported death from any cause:

Year RR (95% CI) p-value

2000 2.18 (0.93, 5.81) 0.07 

2001 1.35 (1.00, 1.96) 0.05 

2002 1.39 (1.07, 1.80) 0.02 

2004 1.43 (1.16, 1.76) <0.001 
Septembe 

Posted: 10 
Antecedents

• 2004 ClinicalTrials.gov (Vioxx, antidepressants 

and suicidal thoughts in children) 

• 2008 Selective Publication of Antidepressant Trials 

and Its Influence on Apparent Efficacy

• 2010 Neurontin settlement

31
Dickersin expert report p18 available at 

http://dida.library.ucsf.edu/ 

“I recommend that the documents reviewed 

by me (including sealed documents) and other 

expert witnesses in the case be made publicly 

available for the education of the public, 

students, clinicians, payers and other 

decision-makers, as well as scholarly work 

that can be used to guide future 

understanding of and potential change in how 

drugs are marketed and used.”

Neurontin Settlement (2004)
Traditional Publication Bias

• Trying to publish but can’t

• Trying to publish but it takes longer (Time lag bias)

• Multiple publication bias

• Location bias

• Language bias

• Citation bias
Newer types of Reporting bias

• Deliberately preventing or delaying publication or public 

disclosure

• Selective outcome reporting

• Selective analysis

• Selective pooling bias

• Ghost and guest authorship

• Reframing and spin

• “Publication strategy”

• Subverting the peer review process (Sperell)
Implications for Systematic 

Reviews

• Make every effort to find detectable publication and 

reporting bias

• Even when FDA records are available, additional 

information from internal correspondence (available 

from court documents) changes the interpretation 

of published evidence.

• This means there is an additional level of 

uncertainty not accounted for in systematic reviews.
Impact on systematic reviewers

"The reality is that a deliberate fraud is extremely 

difficult to unearth. If scientists and companies agree 

to report results in a way that wasn't initially 

intended, unless you have access to original 

documents, it is extremely difficult to actually figure 

out what happened and how it happened," said Dr. 

Steven Nissen, chairman of cardiovascular medicine 

at the Cleveland Clinic Foundation in Ohio. "How 

many other examples like this are there out there 

that we simply don't know about? That's what's 

frightening." 
From EBM 1

From

• Are the Results of the Study Valid?

 Was the assignment of patients to treatment 

randomized?

 Were all patients who entered the trial properly 

accounted for and attributed at its conclusion?

 ....

• What Were the Results?

37
...to EBM 2

1. “What additional research do I need to see to 

weigh the benefits and risks of this drug versus 

alternatives?” 

2. “How do I advocate for patient-centered

comparative studies that answer (rather than 

evade) important questions?” 

3. “How do I take a stand that such evidence must 

be produced before I will adopt an unproven 

practice on the basis of its “value proposition”?

38