Transparency and accuracy inreporting health research

Doug Altman
The EQUATOR Network
Centre for Statistics in Medicine, Oxford, UK

Transparency and value
 Research only has value if– Study methods have validity– Research findings are published in a usable form

2

Research article
 A published research article is often the only
permanent record of a research study– Some readers might be satisfied with scanning an article, or abrief summary– Others will study it in detail for possible inclusion in asystematic review or to influence a clinical practice guideline Only an adequately reported research study can be
fully appraised and used appropriately– to assess reliability and relevance3

Research article
 Readers need a clear understanding of exactly
what was done and what was found– Clinicians, Researchers, Systematic reviewers, Policy makers, …

 The goals should be transparency and accuracy– Should allow replication (in principle)– Can be included in systematic review and meta-analysis– Should not mislead

4

What do we mean by poor
reporting?
 Key information is missing, incomplete orambiguous– methods and findings

 Misrepresentation of the study Misleading interpretation

Of particular concern
 Non-publication of whole studies
 Selective reporting of methods or findings

5

Taxonomy of poor reporting Non-reporting
Failure to publish a report of a completed study(even if was presented at a conference) Selective reporting
Biased reporting of data within a published report Incomplete reporting
Key information is missing Misleading presentation
e.g. claiming study is an RCT when it isn’t;
post hoc change of focus (spin) Inconsistencies between sources
e.g. publication conflicts with protocol

All are very common

6Incomplete reporting of research isvery common
 Hundreds of published reviews show that key
elements of trial methods and findings arecommonly missing from journal reports
 We often cannot tell exactly how the research wasdone
 These problems are generic– not specific to randomised trials– not specific to studies of medicines– not specific to commercially sponsored research7Incomplete reporting of research isvery common“In 37% of papers patient numbers were inadequately
reported;
20% of papers introduced new statistical methods in the‘results’ section not previously reported in the ‘methods’
section, and
23% of papers reported no measurement of error with themain outcome measure.”[Parsons et al, J Bone Joint Surg Br 2011]8

5/228 trials (2%) met all 7 CONSORT criteria reviewed.
52% specified a primary outcome
43% reported attrition (loss to follow up)
36% reported information about blinding
28% described randomization scheme
27% described allocation concealment
22% described an adequate power calculation

J Am Coll Surg 2013

9Ecological studies
The quality of modern cross-sectional ecologic studies:a bibliometric review [Dufault & Klar, Am J Epidemiol 2011]
N=125
 “Most investigators who adjusted their outcomes for age orsex did so improperly (64%)
 Statistical validity was a potential issue for 20% of regression
models Many authors omitted important information when discussing
the ecologic nature of their study (31%), the choice of study
design (58%), and the susceptibility of their research to theecological fallacy (49%).”10

“Spin”
 Review of breast cancer trials“… spin was used frequently to influence, positively, theinterpretation of negative trials, by emphasizing theapparent benefit of a secondary end point. We found bias
in reporting efficacy and toxicity in 32.9% and 67.1% oftrials, respectively, with spin and bias used to suggest
efficacy in 59% of the trials that had no significant
difference in their primary endpoint.”[Vera-Badillo et al, Ann Oncol 2013]11

Inconsistency between sources
Comparison of content of RCT reports in surgicaljournals and trial registry entries (n=51)
Primary
No discrepancy
Complete omission
New introduction
Change in definitionDowngrading fromprimary to secondary
Upgrading fromsecondary to primary

Secondary

55%
8%
8%
10%

33%
31%
39%
6%

22%
14%

[Rosenthal & Dwan, Ann Surg 2013]12

Consequences of inadequate
reporting Assessing the reliability of published articles isseriously impeded by inadequate reporting– Clinicians cannot judge whether to use a treatment– Data cannot be included in a systematic review

 Serious consequences for clinical practice,research, policy making, and ultimately for patients13

Poor reporting is a serious problem forsystematic reviews and clinical guidelines“Risk of bias assessment was hampered by poor reporting of trialmethods.”[Meuffels et al. Computer assisted surgery for knee ligament reconstruction, CDSR 2011]“Poor reporting of interventions impeded replication”[Gordon and Findlay. Educational interventions to improve handover in health care: a systematicreview. Med Educ 2011]“15 trials met the inclusion criteria for this review but only 4 could
be included as data were impossible to use in the other 11.”
[Nolte et al. Amphetamines for schizophrenia. CDSR 2004]“Poor reporting of data meant that individual effect size could not
be calculated for any of these studies.”
Bleakley et al. Some conservative strategies are effective when added to controlled mobilisation withexternal support after acute ankle sprain: a systematic review. Aust J Physiother 2008.

14

We need research we can rely on“Assessment of reliability of published articles is anecessary condition for the scientific process”[Ziman. Reliable Knowledge, 1978]“… clinical research involving human participants canonly be justified ethically when such experiments
are done to produce generalizable knowledge.”[Korn & Ehringhaus. PLoS Clin Trials 2006]
 Authors (and journals) have an obligation toensure that research is reported adequately

15

Reporting research is not new concern,
but it is a relatively neglected one“… incompleteness of evidence is not merely afailure to satisfy a few highly critical readers. It not
infrequently makes the data that are presented oflittle or no value.”[Mainland. The treatment of clinical and laboratory data, 1938]

16

[Altman and Moher, BMJ 2013]17

18