PEER REVIEW

Why skepticism is essential
Donna Laframboise

The Global Warming Policy Foundation
GWPF Report 20

GWPF REPORTS
Views expressed in the publications of
the Global Warming Policy Foundation
are those of the authors, not those of
the GWPF, its Academic Advisory Council members or its directors
THE GLOBAL WARMING POLICY FOUNDATION
Director
Benny Peiser
BOARD OF TRUSTEES
Lord Lawson (Chairman)
Peter Lilley MP
Lord Donoughue
Charles Moore
Lord Fellowes
Baroness Nicholson
Rt Revd Dr Peter Forster, Bishop of Chester Graham Stringer MP
Sir Martin Jacomb
Lord Turnbull
ACADEMIC ADVISORY COUNCIL
Professor Christopher Essex (Chairman)
Sir Samuel Brittan
Sir Ian Byatt
Dr John Constable
Professor Vincent Courtillot
Professor Freeman Dyson
Christian Gerondeau
Dr Indur Goklany
Professor William Happer
Professor David Henderson
Professor Terence Kealey
Professor Deepak Lal
Professor Richard Lindzen
CREDITS
Cover image© Joshwww.cartoonsbyjosh.com

Professor Ross McKitrick
Professor Robert Mendelsohn
Professor Ian Plimer
Professor Paul Reiter
Dr Matt Ridley
Sir Alan Rudge
Professor Nir Shaviv
Professor Philip Stott
Professor Henrik Svensmark
Professor Richard Tol
Professor Anastasios Tsonis
Professor Fritz Vahrenholt
Dr David Whitehouse

PEER
REVIEWWhy skepticism is essential
Donna LaframboiseISBN 978-0-9931189-3-7© Copyright 2016 The Global Warming Policy Foundation

Contents
Forewordvii

About the authorx

Summaryxi

1

The reproducibility crisis
Context and history
Bias and conflicting interests
Peer review’s Achilles heel
No defence against fraud
Flawed analyses
Wobbly standards
NASA’s arsenic debacle
Hiding behind peer review
Errors easy to find, hard to fix
No quality assurance
Conclusion to Part 1

1
2
4
5
6
7
8
9
11
12
13
14

2

Climate implications

14

Notes

17

Foreword
By Christopher Essex and Matt Ridley

The Intergovernmental Panel on Climate Change has repeatedly and falsely claimed
that it depends entirely on peer-reviewed papers. Donna Laframboise used volunteers to check this claim and found that a significant part of the references in the
fourth assessment report of the IPCC were to ‘grey literature’ – that is, press releases,‘reports’ from pressure groups and the like, which are not remotely the normal peer
reviewed scientific literature.
Yet even if all the citations used by the IPCC were peer-reviewed, this would notmean they were infallible. Peer review is not, never was, and never can be a general protection against prejudice, error, or misconception about scientific matters.
That it seems otherwise to some people is a misapprehension on their part, reflecting
widespread myths about the reality of human investigations into the natural world.
It is startling for non-scientists who actually visit the sausage factory of science forthe first time. There, peer review proves to be an often biased, prejudicial, and perfunctory process contrary in every respect to popular expectations about science. But
scientists know that no increased regulation or standards can ever improve things,
because there are no higher authorities to appeal to in a domain of human endeavour where no one knows or ever knew the answers – hence the name ‘peer review’and not ‘expert correction’.
Donna Laframboise observes that, ‘There is a reason publishing insiders are among
peer review’s most derisive critics. They know it’s mostly just a game. Everyone is pretending that all is well despite a mountain of evidence to the contrary.’ Most scientists
grudgingly tolerate peer review because they cannot think of anything better. Experienced ones do not expect much from it, even if they must play along to succeed
given modern customs (until about the mid-20th century it hardly existed).
Most scientists cringe when they hear other scientists claim that because theirwork is peer-reviewed, they do not have to respond to criticisms, even those from
qualified colleagues, whether peer-reviewed or not. Some surely do make such claims:‘. . . many academics insist that the research they present to the world has been fully
vetted. Indeed, they often behave as though it meets a standard unrivalled elsewhere’, observes Laframboise.
Furthermore, those same scientists retreat to the truth about the state of humanknowledge of Nature when facts go against their claims. She points out that: ‘On theother hand, they take no responsibility when information they’ve produced turns out
to be mistaken. In such cases everyone is then reminded that scholarly publishing isvii

really just an exchange of ideas.’ Few competent scientists regard current scientificthinking as anything more than provisional. It is always fully open to challenge.
Peer-review is also abused as a form of gatekeeping to defend orthodox ideas
from challenge, as Laframboise says: ‘Alternative schools of thought are more likely
to encounter scorn than a fair hearing, and the secretive nature of peer review provides ample cover for intolerance and tribalism. . . It places unconventional thinkers
at the mercy of their more conventional colleagues. Indeed, this approach seems
designed to extinguish – rather than nurture – the bold, original thinking that leads
to scientific breakthroughs.’ Many unorthodox ideas prove to be wrong, but they are
the lifeblood of scientific advance. They challenge our orthodoxies, either sharpening
them or overthrowing them. Thus the notion of challenging the orthodox is acceptedin science by necessity, even if grudgingly.
Gatekeeping against the unorthodox is not remotely a new problem. Oracular
mediocrities down the centuries have doggedly resisted human advances in knowledge from Galileo to Semmelweis to Einstein, and thousands of other cases that only
the most learned science historians will ever know. Spectacular scandals come andgo, but science is in the end a long game of the generations, not something playedout in news cycles. So why then has the public debate about the perfunctory, crony,gate-keeping aspects of peer review grown in volume in the media now? Partly it isbecause science has become a ‘bigger’ and more centralized endeavor, with massive
budgets invested in conventional wisdom, and more politicians involved in pushingcertain conclusions. How else can one comprehend the term ‘orchestrated’ used by
one founder of the IPCC to describe how scientific opinion was designed to be treated
for the use of policymakers?
It is clear that people who have never studied the history of science, or have neverbeen on the unfashionable side of a scientific debate are in for a shock upon encountering this messy and sordid reality for the first time. Not least in for a shock is themedia, which has been busy identifying heartbreaking science scandals in medicine,social science, neuroscience, and economics. But curiously they offer none from thesubject of climate, despite it being one of the most policy driven and lavishly fundedbranches of science today.
Is this because there are few examples of bad practice, irreproducibility, retraction, pal review and gatekeeping in climatology? Far from it. The Climategate emailsof 2009 revealed gatekeeping at its most blatant. Who can forget Phil Jones writing
to Michael Mann on 8 July 2004 ‘can’t see either of these papers being in the next
IPCC report. Kevin and I will keep them out somehow – even if we have to redefine
what the peer-review literature is!’ Or Steve McIntyre and Ross McKitrick struggling to
publish (leading to a US Congressional hearing, no less) their comprehensive demolition of the statistical errors and data-selection issues in the infamous ‘hockey stick’
paper? Or Richard Tol’s exposure of the practices employed in the Cook et al ‘97%viii

consensus’ paper? Again and again, peer-reviewed climate papers have fallen apartunder post-publication scrutiny from the likes of McIntyre, Willis Eschenbach, Donna
Laframboise, Judith Curry and Nic Lewis. And these do not even touch on the challenge of independently reproducing climate model output without the machineryand resources necessary to do so, as Laframboise rightly observes in the following
paper.
Indeed, the field of climate science could supply a rich harvest of examples of thiscrisis of scientific credibility all on its own. Yet it is the scandal that dare not speak its
name. The discussions of the crisis in peer review in Nature, Science, the Economist
and elsewhere studiously ignore any examples from climate science. Why is this?
It is an article of faith among certain scientists and science journalists that because
climate scepticism is also a position supported by those on the right of politics, so
nobody in science must give fodder to the sceptics.
This is nothing less than the modern manifestation of gatekeeping continuing its
ancient legacy, driven by sheer ignorance and self-delusion, to keep the forces that
actually advance science away from the door. Scientific research stretches humanfaculties to their limits, and it is at such limits where human frailties become mostprominent.
Humans are fallible. That is one of the greatest lessons from the history of science.
The message to be taken from these heartbreaking scientific scandals and absurdities
is not one of chagrin and a temptation to adopt cynicism. The true authors of suchscandals are the laymen, academics, journalists, and policymakers who do not givea fair hearing to the many highly trained scientists motivated by alternative views
who would put such dubious claims to the test. A pervasive uneducated appeal toscience as a monolithic incomprehensible authority, assessed only in terms of moral
purity rather than factual accuracy, has made such a fair hearing nearly impossible,and done great harm to science and us all.
Christopher Essex
Matt Ridley
September 2016ix

About the author
Donna Laframboise is a Canadian investigative journalist. She is the author of a 2011
exposé of the Intergovernmental Panel on Climate Change, The Delinquent Teenager
Who Was Mistaken for the World’s Top Climate Expert. Her 2013 book, Into the Dustbin:
Rajendra Pachauri, the Climate Report & the Nobel Peace Prize, examines the leadershipof that organization. She is a former columnist and editorial board member of the
National Post, and a former vice president of the Canadian Civil Liberties Association.
She has never belonged to any political party, and has voted across the political spectrum. She currently blogs at BigPicNews.com.

xSummary
Prior to the 2009 Copenhagen climate summit, Peter Doherty, winner of the Nobel
Prize in medicine, defended the Intergovernmental Panel on Climate Change (IPCC)
from its critics. The IPCC involves hundreds of scientists and ‘draws its evidence exclusively from peer-reviewed, published scientific literature’, he wrote. 1 Around thesame time, the IPCC chairman was asked if an Indian environment ministry reportmight alter the IPCC’s pessimistic view of Himalayan glaciers. The ‘IPCC studies only
peer-review science’, Rajendra Pachauri replied dismissively. Until the report’s data
appears in ‘a decent credible publication’, he said, ‘we can just throw it into the dustbin’.2
Peer-reviewed research is reliable, so the reasoning goes. Non-peer-reviewed research is not. The IPCC makes exclusive use of the former, therefore its conclusions
can be trusted. This argument has long been used to deflect criticism and to repel
contrary climate perspectives.
But behind it lies a dubious assumption: that academic publications are a soundfoundation on which to base real-world decisions. In fact, science is currently in thegrip of a ’reproducibility crisis’ so severe that the editor of a prominent journal has
declared that ‘much of the scientific literature, perhaps half, may simply be untrue’. 3
Media coverage declaring that ’science is broken’ has become commonplace. 4
Part 1 of this report demonstrates that a journal’s decision to publish a paper provides no assurance that its conclusions are sound. Large swathes of peer-reviewed
work contain errors. Fraudulent research makes it past gatekeepers at even the mostprestigious journals. And while science is supposed to be self-correcting, the process
by which this occurs is haphazard and byzantine.
A policy cannot be considered evidence-based if the evidence on which it depends was never independently verified. Peer review does not perform that function.
News from the worlds of astrobiology, ecology, economics, chemistry, computer science, management studies, medicine, neuroscience, psychology, and physics all tell
the same tale: ’peer-reviewed’ does not equal ’policy-ready.’
Part 2 of this report invites us to re-examine what we think we know about theclimate. While good scientists have always understood that peer review doesn’t certify accuracy, IPCC officials – supported by politicians, activists, and journalists – thinkglobal climate decision-making should rest on this shaky foundation.
If half of all peer-reviewed research ‘may simply be untrue’, half of all climate research may also be untrue. The policy implications of this idea are immense.

xi

1 The reproducibility crisis
The US National Institutes of Health (NIH) is the world’s largest funder of medical research. In early 2014 it issued a statement that began by citing a lengthy Economist
news story titled ‘Trouble at the lab’. According to the news story, tens of billionsof dollars are spent on medical research each year, yet an unidentified NIH official‘reckons, despairingly, that researchers would find it hard to reproduce at least threequarters’ of published medical findings. 5
By calling attention to the news story without disputing the above statistic, Francis
Collins, the director of the NIH, all but shouted from the rooftops that 75% of medical research is unreliable – a shocking state of affairs. In his words: ‘the checks andbalances that once ensured scientific fidelity’ have collapsed. 6
UK officials have begun to acknowledge the existence of a crisis. 7 A US National
Academy of Sciences workshop has addressed ways of combatting it 8 and that country’s National Science Foundation has restated a fundamental principle: ‘If a scientificfinding cannot be independently verified, then it cannot be regarded as an empirical fact’.9 The elite scientific journal, Nature, has announced new measures aimed at‘reducing our irreproducibility’. 10 The editor-in-chief of the equally prestigious Lancet
declares that ‘science has taken a turn toward darkness’ and that ‘much of the scientific literature, perhaps half, may simply be untrue’. 11
Computer scientists, alarmed at how little of the research in their own field isreproducible, refer to a credibility crisis. 12 The field of psychology is similarly grappling with high rates of irreproducibility. 13 An entire 2012 issue of Perspectives in
Psychological Science explored this problem, with one paper referring to ‘the mythof self-correction in science’ and another titled ‘Why science is not necessarily selfcorrecting’.14
In 2007, Daniel Hamermesh remarked that economists ‘treat replication the way
teenagers treat chastity – as an ideal to be professed but not to be practised’. 15 Twoyears later, a report concluded that ‘economics journals do not ensure replicability,
even when their own policies require it’. Rather, ‘the evidence strongly suggests thatmost results published in economics journals are not independently’ verifiable. 16 Little improvement was observed by the authors of a 2015 paper prepared for the Boardof Governors of the US Federal Reserve. Since ‘less than half of the papers in our sample’ can be reproduced even with the assistance of the original authors, they wrote,‘we assert that economics research is usually not replicable’. 17
The idea that research published in a scholarly journal is as likely to be wrong as
it is to be right is difficult to absorb. But this is old news to venture capital investors.
Bruce Booth of Atlas Ventures holds a PhD in molecular immunology from Oxford University. The ‘unspoken rule’, he wrote in early 2011, ‘is that at least 50% of the studies
published even in top tier academic journals. . . can’t be repeated with the same conclusions by an industrial lab’.18
1That same year, employees of Germany’s Bayer Healthcare reported that attempts
to reproduce the findings of 67 studies involving promising drugs had resulted in a
75% failure rate. The investigators were surprised that research published in prestigious journals was no more reliable than research published in journals lower down
the hierarchy.19 An equally alarming report appeared in Nature in 2012. Amgen, an
American pharmaceutical company, had attempted to verify the findings of 53 landmark papers connected to cancer research. It was unable to do so in 47 cases (89%). 20
A world in which nine out of ten cancer studies aren’t worth the paper they’reprinted on is a world that needs to look more closely at the way research currently
gets vetted. The academic publishing industry makes use of a mechanism known as
pre-publication peer review. This, we are told, is the watchdog that prevents poorquality work from making its way into the scholarly record.
When a committee of the UK House of Commons examined peer review in 2011,
Edmund Lamb, editor of the Annals of Clinical Biochemistry, declared that ‘the peer
review process protects the scientific community and the public from erroneous science’. The UK Academy of Social Sciences similarly asserted that peer review ‘provides a guarantee for the public of the validity and scientific warrant of knowledge
produced with public funds and placed in the public domain’. 21
The mechanics of the process are straightforward. Because academic journals often have room for less than 10% of the research papers submitted to them, many
thousands are summarily rejected. 22 Those that survive this first hurdle are typicallysent to two or three individuals external to the journal who are thought to possess
relevant scholarly expertise. These individuals, known as referees or reviewers, canrecommend that a paper be published as is, rejected outright, or amended in orderto make it publication-worthy. The decision rests with journal officials, and the identities of referees are usually kept secret from the authors whose research they evaluate,
as well as from the public.

Context and history
It is difficult to exaggerate the central role peer review plays in modern academic life.
Since at least the 1970s, educational administrators and funding bodies have placed
inordinate emphasis on how many papers a researcher publishes, particularly in top
journals.23 Individual career advancement, university department funding, and academic prestige are all closely linked to this single metric. 24 The House of Commonscommittee was advised that a publishing ‘arms race’ is currently underway throughout the academy,25 and that professional ‘careers are built and destroyed on thesenumbers’.26
Peer review is, therefore, the mechanism by which otherwise invisible research
gets transformed into academic gold. Sandra Goldbeck-Wood has observed that it‘confers legitimacy’ on academics themselves as well as on their work, and is ‘sur2

rounded by an almost religious mystique". 27 A New York Times headline has referred to‘the sacred rite of peer review’.28 Richard Smith, a former editor-in-chief of the British
Medical Journal (BMJ) and an outspoken critic, acknowledges that peer-reviewed research is considered ‘in some sense blessed’. 29
Given all of the above, it’s disconcerting to discover that nothing more than ametaphorical sniff test actually takes place. A 2008 survey found that referees typically spend a total of five hours reading a paper, preparing written feedback, andexchanging e-mail with journal personnel. 30 Some reviews are a single paragraph inlength.31 Others appear to be little more than a posterior-covering formality. 32 Thereis no expectation on anyone’s part that referees are conducting an audit. They typically don’t examine raw data, computer codes, or even verify that cited sources saywhat the paper claims they do. If one receives what one pays for, it’s worth notingthat scholarly publishing’s vaunted vetting process relies almost entirely on free, volunteer labour.
The Royal Society, the world’s oldest scientific academy, told the Commons committee that peer review has ‘stood the test of time’ and that all of its own publishing decisions have been made this way since 1660. 33 But peer review didn’t become
widespread until centuries later. As Melinda Baldwin, the author of a book about Nature, has observed, ‘many of the most influential texts in the history of science were
never put through the peer review process, including Isaac Newton’s 1687 Principia
Mathematica, Albert Einstein’s 1905 paper on relativity, and James Watson and Francis
Crick’s 1953 Nature paper on the structure of DNA’. In fact, she says, ‘Nature publishedsome papers without peer review up until 1973’. 34
The fragmentary historical record indicates that scholarly journals began making
use of external referees at different times, for different reasons, and in a variety ofconfigurations. The turning point appears to have been World War II, after which thenumber of people involved in academic life ballooned. As the supply of manuscripts
rose and research became more specialized, most journals began using outside experts to help them evaluate submissions. 35 That said, as late as 1965 a survey of American humanities journals identified 16 publications in which editors continued to rely
solely on their own judgment, and another 50 in which the editor normally sought
input from only one additional person. 36 Institutionalized external peer review involving multiple referees is, therefore, a recent phenomenon.
Because of this disorderly history, peer review remains highly non-standardized.
Michael Callaham compares the current situation to a bygone era in which no licence
was required to practice medicine. ‘Anyone can produce a journal and use any standards they see fit’, he says.37 After organizing four international peer review conferences between 1989 and 2001, Drummond Rennie concludes that ‘the term ’‘peer
review” still seems to mean a great many things to different journal editors, and practices still seem to vary widely between journals’ (italics added). 38
3

The Commons committee was repeatedly told that peer review is a quality assurance mechanism.39 But industry-wide definitions, standards, and best practices arewholly absent. No certification or monitoring by independent third parties occurs,
as is common in other quality assurance contexts. Rather, the academy appears tobelieve that every ad hoc, idiosyncratic process to which the peer review label has
been affixed produces equal benefits. But these benefits are difficult to document.
An extensive body of research finds scant evidence that peer review accomplishesmuch at all, other than modestly improving the clarity of some manuscripts. 40 On theother hand, a great deal of duly peer-reviewed scholarship has identified numerous
deficiencies.
Smith, the BMJ’s former editor, describes peer review as a roulette wheel, a lottery,
and a black box.41 As early as 1982, researchers demonstrated its random, arbitrary
nature by resubmitting papers to journals that had already published them. Cosmeticchanges (including the use of fictitious author and institution names) were made to
12 papers that had appeared in a dozen highly regarded psychology journals during
the previous 18–32 months. The duplication was noticed in three instances, but theremaining nine papers underwent review by two referees each. The 16 referees (89%)
who recommended rejection didn’t cite lack of originality as a concern. Instead, themanuscripts ‘were rejected primarily for reasons of methodology and statistical treatment’. Only one of the nine papers was deemed worthy of seeing the light of day thesecond time it was examined by reviewers at the same journal. 42

Bias and conflicting interests
Peer review is prone to all manner of bias. After an ecology journal began removing author names prior to forwarding manuscripts to reviewers, the number of published papers whose lead author was female rose. 43 Conversely, a 2013 study foundthat female referees were significantly less likely to recommend publication of maleauthored papers than female-authored papers. 44 When a researcher’s gender, nationality, prominence, or institutional affiliation influences publishing decisions, scientificintegrity suffers.
In the publish-or-perish environment in which they spend their professional lives,modern scientists are hugely incentivised to pursue flashy, superficially tested theories rather than triple-checked, non-glamorous ones. Referees are routinely asked
to pass judgment on a paper’s novelty and originality, and journals display a welldocumented bias against publishing underwhelming (in other words, negative), results.45 Keen to secure publication in high-prestige journals, scientists are practicallyinvited to grasp at straws and to exaggerate borderline findings. Eager to maintaintheir industry rankings and to attract mainstream media attention, scholarly journals
frequently select papers based on criteria other than scientific virtue.
4

A 2010 New Yorker article explores the significant and distorting role that confirmatory bias can play. Titled ‘The truth wears off’, it describes how early, persuasive
scientific results often dissipate over time. Energized by a new idea, researchers startfinding proof of it everywhere. As the years pass, however, the strength of the evidence declines. This phenomenon applies to topics as diverse as the mating ritualsof barn swallows and the effectiveness of antipsychotic drugs. Being mere mortals,scientists aren’t impervious to ‘fleeting fads’. When a community of researchers succumbs to groupthink (the New Yorker calls it ‘a collective illusion nurtured by strong
a-priori beliefs’) referees – who are drawn from that same community – provide thepublic with little protection against erroneous scholarship. 46
Outsiders naively assume that scientists are open-minded intellectual explorers,
but back in 1977 Michael Mahoney demonstrated that identical manuscripts suffer‘very different fates’ depending on whether their conclusions confirm or challengea referee’s own theoretical perspective. 47 Facts are supposed to prevail over dogma
but scientists, alas, are only human. Alternative schools of thought are more likely to
encounter scorn than a fair hearing, and the secretive nature of peer review provides
ample cover for intolerance and tribalism.

Peer review’s Achilles heel
Three decades ago Herbert Ayres,48 then president of the Institute of Management
Sciences, identified peer review’s Achilles heel: it places unconventional thinkers at
the mercy of their more conventional colleagues. Indeed, this approach seems designed to extinguish – rather than nurture – the bold, original thinking that leads toscientific breakthroughs. Most scientists, and therefore most referees, says Ayres, ‘arenot groundbreakers’. When asked to review research, they represent ‘the establishment’. He eloquently invites us to consider matters from the perspective of a hypothetical ‘Professor Zee’, who has tenure, is a recognized authority in his field, and isjuggling numerous professional responsibilities:
In the 31 years since he was a freshman, he has invested 9 solid man-years ofwork mastering his specialty. If its structure is basically disrupted, he will need up
to a solid man-year to rework it. Where is he going to get a man-year? Nowhere,
unless he turns his life inside out. As a referee of a paper that threatens to disrupt his life, he is in a conflict-of-interest position, pure and simple. Unless we’re
convinced that he, we, and all our friends who referee have integrity in the upper fifth percentile of those who have so far qualified for sainthood, it is beyond
naive to believe that censorship does not occur. It need not be entirely conscious. . . Zee is not a bad buy. Zee is a good guy. . . He never said he was a saint; itis we who are asking him to be.49

Many Nobel laureates received a thumbs-down verdict from journals when submitting the very work that would later win them accolades. Juan Miguel Campanario
5reports50 that both Nature and Science rejected Kary Mullis’ description of a new approach to analyzing DNA. Eight years later, it won him a Nobel prize in chemistry. In
another instance, a referee dismissed as ‘not interesting enough’ work that led directly
to a physics Nobel.51 A 1955 letter from the editor of The Journal of Clinical Investigation advised that its peer reviewers ‘have been particularly emphatic in rejecting’discoveries that would earn Rosalind Yalow the 1977 Nobel prize in medicine. 52

No defence against fraud
The same peer review process that fails to recognize Nobel-quality work when it sees
it offers the public no protection against fraud. This is true even at top-tier journals
where, according to the Royal Society, peer review is the most exacting. 53 Smith has
observed that having one’s paper published ‘in a major journal like Nature ’ is equivalent to winning ‘the jackpot’.54 In early 2003, Nature formally retracted seven papers
authored by Jan Hendrik Schön.55 The condensed matter physicist said he’d discovered a new form of superconductor, amongst other things, but had actually faked his
data. The similarly high-status Science retracted eight of Schön’s papers.56 Between
them, three other journals withdrew an additional twelve. In the Schön case alone,
therefore, fraudulent research was published on 27 occasions by five separate scientific journals, including Nature and Science.57
In 2009, the editor-in-chief of the highly-ranked Anesthesia & Analgesia journal released a list of 21 papers that a third-party investigation had determined were basedon fabricated data. Scott Reuben, a prominent professor of anesthesiology, had foiled
the peer review process of that journal ten times over a twelve-year period. Elevenmore of his fraudulent papers had been published by eight additional journals. 58
In 2011, Science retracted highly-publicized research by the social psychologist
Diederik Stapel that claimed to have found a link between litter and racism but was
actually fabricated whole cloth. 59 A lengthy New York Times Magazine profile, entitled ‘The mind of a con man’, describes Stapel as ‘an academic star in the Netherlands
and abroad’.60 A 2012 report written by investigators at three Dutch universities concluded that Stapel had published a total of 55 fraudulent papers. 61
His work has appeared ‘in nearly all the respected international journals in his
field’, yet the report said it was ‘extremely rare’ for his findings ‘to be subjected toserious doubt, even. . . where the fraud was blatant’. 62 These investigators consider it‘almost inconceivable’ that referees at these journals failed to notice ‘all the impossibilities, peculiarities and sloppiness’ they themselves unearthed. ‘Time and again’,
they say, the ‘journals accepted that Mr Stapel’s hypotheses had been confirmed ina single experiment, with extremely large effect sizes’. 63 Overall, Stapel’s fraudulent
research got past the peer reviewers of almost two dozen journals.
The Dutch investigators assert that the Faculty of Psychology at the University of
Amsterdam failed to enforce scientific standards:
6

It was easy for researchers to go their own way. Nobody looked over their shoulder. . . there was no culture in which scientific integrity was given a high priority
through explicit attention, detailed monitoring, an implicit exemplary functionof senior staff and a collective responsibility within research groups, where themembers correct each other and keep each other alert. 64

Instead, say these investigators, the academic culture in which Stapel rose to prominence was one ‘of careless, selective and uncritical handling of research and data. The
observed flaws were not minor. . . but violations of fundamental rules of proper scientific research’.65
These investigators also say that journal referees sometimes ‘encouraged irregular practices’ by urging the omission of untidy details in Stapel’s manuscripts in theinterests ‘of telling an interesting, elegant, concise and compelling story’. 66 The authors who reported that 89% of cancer studies couldn’t be reproduced noticed thesame thing. Data originally included in manuscripts is ‘removed during the peer review and publication process’ they say, because editors and referees ‘often look for ascientific finding that is simple, clear and complete’. 67
That peer review may, in fact, be undermining reproducibility is further supported
by a report published in 2003. Drawing on 30 years’ experience as a journal editor,
Arthur Bedeian invited nearly 200 lead authors who’d recently had papers publishedin premier management journals to complete a survey about the peer review process those papers had undergone. A majority (74%) felt ‘the revisions they were required to make in their manuscripts were beneficial enough to justify the additional
labor and delay in publication’. Nevertheless, 24% reported that, during the processof making their work acceptable to the referees and editors involved, ‘they had actually made changes they felt were wrong’. 68 Academic desperation to be published
may be introducing error into a quarter of the research in top-tier journals.

Flawed analyses
Another deficiency that peer review provides no shield against is the improper useof statistics. In 2011, an examination of neuroscience research papers published in‘five top-ranking journals’ found that half had applied a statistical procedure incorrectly.69 Why, wondered one news account, are journals ‘not properly vetting such
papers before publishing them?’70
In 2014, third-party academics alerted the Obesity Facts journal to significant flaws
in a paper it had published. While offering statistical guidance to other researchers,
the authors had misstated key concepts. A statistician was consulted and he confirmed these shortcomings, but this flawed statistical advice remains part of the official scientific record. The journal subsequently published an editorial that declares‘it is each author’s responsibility to make sure that statistical procedures are correctly
used’ in the work they submit since ‘it cannot be expected that all of our reviewers,
7

in addition to their expertise in various aspects of obesity research, are designatedexperts for advanced statistical procedures’. 71 In other words, caveat emptor. Just
because a paper passed peer review doesn’t mean the referees were intellectuallyequipped to evaluate it.
The Economist news story mentioned at the beginning of this report sums up thecurrent statistical quagmire:‘Some scientists use inappropriate techniques because those are the ones they
feel comfortable with; others latch on to new ones without understanding theirsubtleties. Some just rely on the methods built into their software, even if they
don’t understand them.72

In 2014, Science announced measures to provide deeper scrutiny of statistical claimsin the research it publishes.73 John Ioannidis, the author of a seminal 2005 paper
asserting that most published research findings are false, 74 called this announcement‘long overdue’. In his opinion, statistical review has become more important than
traditional peer review for a ‘majority of scientific papers’. 75
In February 2016, the American Statistical Association issued a formal statement
clarifying ‘the proper use and interpretation’ of a statistical measure ‘commonly misused’ in published scientific research. 76 The degree to which efforts such as thesewill improve the quality of the peer review that is applied to the 1.5 million scholarly
articles published each year77 remains unclear.78

Wobbly standards
As mentioned earlier, educational administrators and funding agencies assess academic performance according to certain metrics. Despite the extravagant shortcomings associated with the peer-review process, they continue to equate publication ina peer-reviewed venue with scientific quality. The folly of this position is highlighted
by a headline that appeared in Nature in 2014. ‘Publishers withdraw more than 120
gibberish papers’, it announced.79 In contrast to other fields, computer scientists publish their work primarily in conference proceedings rather than journals. 80 These arenormally fully refereed publications but here, too, peer review is a poor watchdog. Between 2008 and 2013, papers containing computer-generated nonsense made theirway into 30 conference proceedings issued by reputable scholarly publishers. Whencontacted, one publisher was adamant that the papers had, indeed, been peer reviewed.
There is a reason publishing insiders are among peer review’s most derisive critics.
They know it’s mostly just a game. Everyone is pretending that all is well despite amountain of evidence to the contrary. Smith tells a story about Robbie Fox, ‘the great
20th century editor of The Lancet ’, who joked that he ‘had a system of throwing a pileof papers down the stairs and publishing those that reached the bottom’. Adds Smith,8

When I was editor of the BMJ I was challenged by two of the cleverest researchers
in Britain to publish an issue of the journal comprised only of papers that had
failed peer review and see if anybody noticed. I wrote back ’How do you know I
haven’t already done it?’81

Smith’s book, The Trouble With Medical Journals, devotes an entire chapter to the subject of editorial misconduct. 82 A BMJ blog post he wrote in 2008 begins with thissentence: ‘In what has been called the age of accountability, editors have continued
to be as unaccountable as kings’. The upshot of the melodrama he then relates is that
two academics who attempted to challenge a 2003 paper published in the Journalof Health Economics, felt that the manner in which their own paper was handled violated ‘almost every ethical standard established for editors’. Nevertheless, they wereadvised that no mechanism existed within academic publishing to complain aboutsuch behaviour. In the wake of this incident, writes Smith, the owner of the journal –scholarly publishing giant Elsevier – ‘has now signed up all of its journals to be members of the Committee on Publication Ethics (COPE)’. 83 A spokesperson from COPE
advised the Commons committee that it provides ‘advice to the editors, which they
are free to ignore. We don’t have any particular powers, except that we also provide acode of conduct and we ask all our members to adhere to that code’. Authors, editors,
and members of the public can now file a complaint with COPE should a misbehaving
editor work for an affiliated journal. 84
In 2013, COPE issued a five-page ethical conduct guideline for peer reviewers.
Tucked into the final page of that document is a bullet point declaring that, should
an editor decide to ‘provide a review’ of a manuscript they themselves are handling,
this fact must be revealed. According to COPE coordinator Irene Hames, editors struggling to find referees sometimes write up fake reviews under the cover of anonymity:‘I have come across editors who have almost boasted about it and said: ’I never havea worry about finding reviewers because I just do it myself’. 85
Many scientific insiders are fully aware that the watchdog is feeble. For years,
these critics have reminded anyone who’d listen that peer review doesn’t certify accuracy and therefore can’t possibly signal that a research paper is reliable in a publicpolicy context. But there’s yet another problem. Some academics have seized holdof the leash and have turned the watchdog against those who pay the vet bills. In
that context, a mechanism that helps publishers decide what to print has metamorphosed into an aggressive, snarling beast patrolling the perimeter between the Ivory
Tower and the rest of us.

NASA’s arsenic debacle
On a Monday in late November 2010, NASA issued a press release about a news conference scheduled for Thursday that would ‘impact the search for evidence of extraterrestrial life’.86 On the day of the news conference, the public was told that bi9

ology textbooks would need to be rewritten due to research that had just been published on Science ’s website. NASA-funded scientists had discovered a microbe withthe unique ability to build its DNA out of arsenic rather than phosphorus, altering our‘fundamental knowledge about what comprises all known life on Earth’. 87
The paper’s lead author, geobiochemist Felisa Wolfe-Simon, became an instantcelebrity. Three months to the day later, she spoke at a TED event. 88 A month after
that, Time magazine designated her one of the 100 most influential people in theworld.89 But on the Saturday following the news conference, Rosie Redfield, a microbiology professor at the University of British Columbia, posted a detailed and scathingcritique of this research on her own blog. If the data in the Science paper had beenpresented to her by a PhD student, she said, she’d have sent him or her ‘back to thebench to do more cleanup and controls’. 90
Journalists began eliciting second opinions about what had been a celebratory,
front page story. Carl Zimmer, who writes a science column for the New York Times,
contacted a dozen experts who almost unanimously agreed that the NASA scientists
had ‘failed to make their case’. One said the paper shouldn’t have been published.
Another said it was ‘pretty trivial to do a much better job’. 91 Wolfe-Simon was invited
to respond. But although she’d had a great deal to say at the press conference, shesuddenly didn’t think the public square was where these matters should be discussed.
Her e-mail response to Zimmer urged him to. . . honor the way scientific work must be conducted. Any discourse will haveto be peer-reviewed in the same manner as our paper was. . . The items you arepresenting do not represent the proper way to engage in a scientific discourse
and we will not respond in this manner.

One of Wolfe-Simon’s co-authors similarly advised Zimmer that researchers who ‘debate the questions or comments of others’ in the media ‘have crossed a sacred boundary’.92 The publicly-funded NASA also demurred. A Canadian news outlet reported
that,. . . when asked about public criticisms of the paper in the blogosphere, [NASAspokesperson Dwayne Brown] noted that the article was peer-reviewed and published in one of the most prestigious scientific journals. He added that WolfeSimon will not be responding to individual criticisms, as the agency doesn’t feel
it is appropriate to debate the science using the media and bloggers. Instead, it
believes that should be done in scientific publications. 93

Zimmer was stunned. In his words, ‘I’ve been doing this kind of thing for a longtime, and I have never encountered a response like this one from the hundreds ofscientists I’ve interviewed’.94 David Dobbs, a science writer at Wired, was caustic. Employing adjectives such as ’anti-empirical’ and ’pre-Enlightenment,’ he insisted thatmicrobiology professor Redfield ‘is a peer, and her blog is peer review’. NASA was dismissing criticism not on its merits, he said, but because Redfield hadn’t delivered it10‘standing on the proper altar’ in the ‘Church of the Peer-reviewed Journal’. 95 For hispart, Jonathan Eisen, an evolutionary biologist at the University of California-Davis,
described the situation as absurd. These researchers, he said, ‘carried out science bypress release and press conference. . . they are now hypocritical if they say that the
only response should be in the scientific literature’. 96

Hiding behind peer review
Zimmer’s remarks notwithstanding, this behaviour isn’t unique. The summer prior toNASA’s press conference, in a column published in the Independent, British journalist
Nigel Hawkes addressed what he called a ‘sinister development’. Academics who’ve
authored ‘provocative or implausible claims in peer-reviewed journals have started
arguing that they won’t listen to criticism unless it has undergone the same layingon of hands’, he wrote.97
Hawkes points to professors Kate Pickett and Richard Wilkinson, the authors ofthe 2010 book, The Spirit Level: Why Equality is Better for Everyone. In fairness, they did
issue a 22-page response to their detractors. But at the top of that document thisstatement appears in bold, italicized type:
NOTE: Almost all of the research we present and synthesise in The Spirit Level had
previously been peer-reviewed, and is fully referenced therein. In order to distinguish between well founded criticism and unsubstantiated claims made forpolitical purposes, all future debate should take place in peer-reviewed publications.

Feedback from the academic community is ‘well founded’. Feedback from the restof humanity is unsubstantiated and politically motivated. This is an odd argument for
champions of equality to be making. A few paragraphs later, Pickett and Wilkinson
declare that their overall analysis has been subjected to peer review and that theirresearch has been funded by four separate agencies, ‘all of whom subject research
proposals to rigorous reviews’. On page 18, part of their response to a question aboutlife expectancy includes this assertion: ‘if you have good evidence [to support an alternate perspective] it should be presented in a peer-reviewed journal’. 98
The Amazon UK website currently describes The Spirit Level as ‘the most influential
and talked-about book on society in the last decade’. 99 Journalist Hawkes says thatwhile the authors’ thesis may be correct, their views about public debate are not.
Surely, he says, when you write a book for a lay audience ‘about a hugely politicalsubject such as inequality, you’ve surrendered any right to hide behind the flak-jacket
of peer review’. Attempting to do so ‘is a betrayal of all the principles of academic life,of open dialogue, of freedom of expression and. . . of equality as well. It’s a disgrace’.
Hawkes points to another incident, involving World Health Organization research
about Caesarean births published in The Lancet. As a journalist with a special interestin the misuse of statistics, Hawkes had challenged the study’s analysis in a January112010 column. ‘Did no one check the arithmetic in the tables, which are full of errors?’
he’d asked. ‘The Lancet is a distinguished journal – were its referees asleep?’ 100
Hawkes reports that after a copy of his column was forwarded to the journal by achildbirth advocacy group, an editor snootily replied: ‘We are a scientific journal, andas such prefer to see the scientific debate continued by reference to other academic
articles that have been peer-reviewed’. 101 The only knowledge we scholars recognize
is the knowledge we ourselves generate. Challenges to this knowledge can only takeplace on territory we control.

Errors easy to find, hard to fix
Power dynamics aside, the above (admittedly truncated) response displays no zeal
for ensuring an accurate scientific record. Although the Commons committee was repeatedly told that peer-reviewed journals perform this important service, once again
we find a gap between rhetoric and reality. 102 A 2016 Nature article provocatively titled ‘A tragedy of errors: mistakes in peer-reviewed papers are easy to find but hard
to fix’ provides a disturbing demonstration. 103 Over a period of 18 months, a teamof four American academics attempted to correct the scholarly record after noticing dozens of substantial errors in published medical papers. Some of these papers‘described mathematically or physiologically impossible results’ such as an ‘averageheight change of about 7 centimetres in adults over 8 weeks’. While one study theytargeted was retracted in response, that case isn’t representative. The majority of theirexperiences suggest that the academic publishing system was never designed to be
self-correcting.
The American academics observe that journals rarely indicate who should be contacted about errors, and don’t deal with these matters in a systematic fashion. Editors‘seemed unprepared or ill-equipped to investigate, take action or even respond’. Discussions involving editors, authors, and ‘unidentified journal representatives’ would
often drag on for months ‘without any public statement added to the original article’ to indicate that concern had been expressed about its soundness. Even when
gaffes were eventually established or admitted, the process was Kafkaesque. ‘Somejournals that acknowledged mistakes required a substantial fee to publish our letters’, they report, citing a US$1,716 charge in one instance and a US$2,100 charge inanother. While trying to do the right thing, they say, ‘we were asked to spend our research dollars on correcting other people’s errors’. One publisher, they assert, ‘states
it will charge the author who initiates withdrawal of a published paper US$10,000’. 104
An industry that prizes accuracy does not look like this.
Now let us return to NASA’s arsenic-based life paper published online in December 2010.105 In May 2011, Science published eight short Technical Comments by third
parties expressing doubt about the paper. 106 Then, in July 2012, the story took a sharp
turn. Science published two studies by two teams of researchers (one of which in12

cluded microbiology professor Redfield). 107 It also issued an editorial statement to
the media that declared unequivocally: ‘Contrary to an original report, the new research clearly shows that the bacterium. . . cannot substitute arsenic for phosphorus
to survive’. This means that the NASA paper ‘does not break the long-held rules of life,
contrary to how Wolfe-Simon had interpreted her group’s data’. 108
Science has not, however, formally retracted the NASA study – a step common
enough that an entire blog, RetractionWatch.com, is devoted to tracking such announcements. Nearly four years after this study was refuted, the web page on which
it is published makes no mention of this fact. Readers aren’t warned that they’re looking at discredited research.109 Ambiguous corrections to the scientific record are oflimited value. The public isn’t clairvoyant. Nor are journalists who weren’t watchingclosely when these events unfolded.
When Science published the eight technical comments in 2011, it told the media:
These post-publication responses are an essential part of the process by whichscience moves forward, correcting itself when necessary. We hope that the study
and the subsequent exchange being published today will stimulate further experiments – whether they support or overturn this conclusion. In either case, theoverall result will advance our knowledge about conditions that support life. 110

This is called playing the peer review ‘Get out of jail, free’ card. On the one hand, many
academics insist that the research they present to the world has been fully vetted.
Indeed, they often behave as though it meets a standard unrivalled elsewhere. On
the other hand, they take no responsibility when information they’ve produced turns
out to be mistaken. In such cases everyone is then reminded that scholarly publishingis really just an exchange of ideas. The unwashed masses may be foolish enough to
confuse peer-reviewed studies with the gospel truth, but sophisticated academics
know that all knowledge is just provisional.

No quality assurance
When the House of Commons committee examined peer review in early 2011, one
witness after another insisted it was a ‘robust’ and ‘rigorous’ quality-control mechanism.111 But it is not. The remarks of Martin Hill, a systems engineer who submitted
written evidence, are highly relevant. Peer review, he says, can’t provide bona fidequality assurance ‘because it occurs far too late’ in the research process. ‘Quality has
to be assured from the beginning of the work, not in retrospect’, he says. ‘You cannot check that equipment has been operated correctly after the event’. Unless aninspection was conducted at the time, there is no way of knowing ‘if equipment wasproperly clean’. Nor are referees in a position to know if some data was discarded andother data fabricated.112
Examination of a manuscript by a few unpaid volunteers is not a serious quality
control mechanism.
13

Conclusion to Part 1
When Nobel laureates, scientific lobby groups, academic publishers, or climate officials imply that peer-reviewed research is self-evidently sound and accurate, they arepromoting a myth. Rigorous skepticism is a sensible response to myths – the same
skepticism that animates genuine scientific inquiry. Back in 1974, the late physicist
Richard Feynman delivered a commencement address in which he discussed SouthSea islanders who had constructed an airstrip, a hut for an air traffic controller, andequipment resembling headphones and antennas. After taking these steps, the islanders were disappointed that planes laden with goods didn’t automatically begin
landing on the island as they had during wartime. The islanders had, after all, followed
the recipe conscientiously.113
Feynman coined the term ‘cargo cult science’ because it’s possible to go throughwhat appear to be all the right motions and yet not end up with a valid result. Peerreview is an example of cargo cult thinking. Despite the overwhelming evidence tothe contrary, the academic establishment continues to believe that if a series of prepublication procedures are followed, sound science emerges at the other end. In thewords of former BMJ editor Richard Smith, ‘How odd’ that a tool so closely linked toscience should, in fact, ‘be rooted in belief’. 114
In reality, peer review is an often perfunctory process. It takes place behind closed
doors, with no enforcement of even minimum standards. The fact that a single scholarly journal – among an estimated 25,000 – has performed a peer review ritual tells us
little.115 The paper’s data and computer codes have not been thoroughly examined.
Its arithmetic hasn’t necessarily been checked. Its statistical analysis may or may nothave received informed scrutiny.
As the US National Science Foundation has recently reminded us, a scientific finding ‘cannot be regarded as an empirical fact’ unless it has been ‘independently verified’.116 Peer review does not perform that function.
Policymakers, journalists, and members of the public need to abandon the ideathat peer-reviewed research is a sound foundation on which to base public policy. The
reproducibility crisis currently gripping the scientific world indicates that, despite thewidespread use of peer review, much of published academic research ‘may simply be
untrue’.

2 Climate implications
In 2008, Rajendra Pachauri, the chairman of the Intergovernmental Panel on Climate
Change (IPCC), told a committee of the North Carolina legislature that the global
warming that had occurred since the middle of the 20th century was ‘most probably’ the fault of humanity. A recent IPCC report had reached this conclusion, he said,
14and his audience could be assured the report was authoritative because:
we carry out an assessment of climate change based on peer-reviewed literature,so everything that we look at and take into account in our assessments has tocarry [the] credibility of peer-reviewed publications, we don’t settle for anything
less than that.117

Pachauri’s chain of logic went like this: scientific literature is reliable because it
has been peer-reviewed. IPCC conclusions are reliable because they rest on that firm
foundation.118 But peer review is a castle built on sand. Pachauri’s linking of theterms ’credibility’ and ’peer-reviewed’ highlights the disturbing fact that unverified
academic research frequently becomes the basis of real-world policy decisions. 119
This report has shown that academics across different fields are prone to ’collective illusions’ that may lead them to exaggerate the strength of certain evidence. It
has explained that scientists are typically neither saints nor groundbreakers. Rather,
they are professionally committed to the theoretical perspective on which their own
careers have been built, and are frequently hostile to alternative viewpoints. We havealso seen that one quarter of academics may be so desperate to be published in prestigious venues they will permit opinionated referees and editors to introduce errors
into their work.
There is no reason to believe that the politically charged arena of climate science is exempt from these problems, or that it doesn’t share the alarming rates of
irreproducibility observed in medicine, economics, and psychology. Indeed, nontransparency is an acute problem in climate science due to the use of climate modelling via supercomputers that cost tens of millions of dollars and employ millions oflines of code.120 Outsiders – whether they be other scientists, peer reviewers, or journalists – have no access to the specialized software and hardware involved, and it isdifficult to imagine how such access might be arranged, never mind the person-years
that would be required to fully explore the subtle computational and mathematicalissues arising from a careful audit of such a model.
Reproducibility is the backbone of sound science. If it is infeasible to independently evaluate the numerous assumptions embedded within climate model software, and if third parties lack comparable computing power, a great deal of climate
science would appear to be inherently non-reproducible.
The world is currently spending billions on measures intended to combat humanity’s allegedly significant role in climate change. The IPCC tells us this is prudent andnecessary. But IPCC reports are based on thousands of research papers whose conclusions have never been independently verified. If half of published, peer-reviewed
papers ‘may simply be untrue’, half of the papers cited by the IPCC may also be untrue.
We need to re-examine what we think we know about the climate. In response to
the reproducibility crisis in medicine, the beginnings of a system of accountability are
now being established. Researchers can confidentially submit their completed work15

to an independent, fee-for-service lab prior to publication. Findings successfully reproduced in this manner earn an ’independently validated’ designation. 121 The Lauraand John Arnold Foundation is currently funding replication of 50 important cancer
studies. The Prostate Cancer Foundation has arranged for select research to be assessed. And the Antibody Validation Project is attempting to ‘validate thousands ofcommercial antibodies’ so that healthcare dollars and expertise aren’t wasted by aseries of researchers stumbling down the same blind alleys. 122
Currently, climate research is not subjected to meaningful due diligence prior to
the IPCC presenting it as sound in its reports. Wealthy green charitable foundations
and government funding agencies have the power to change that by pursuing measures similar to those now occurring in medicine.
Until key climate findings meet a higher standard than mere peer review, we cannot claim that our climate policies are evidence-based.

16

Notes
1. Doherty, Peter. Sceptical thinking: Copenhagen and beyond, The Monthly, Nov. 2009;http://mnth.ly/q9apgqv.
2. No proof of Himalayan ice melting due to climate change. Times of India, 10 Nov. 2009;http://www.webcitation.org/5wZuwHUZE.
3. Horton, Richard. What is medicine’s 5 sigma? The Lancet, 11 Apr. 2015;
http://www.webcitation.org/6g9lRzki3.
4. Headines include the following: ‘How science goes wrong’
http://www.economist.com/news/leaders/21588069-scientific-research-has-changedworld-now-it-needs-change-itself-how-science-goes-wrong, ‘Science has lost its way’
http://articles.latimes.com/print/2013/oct/27/business/la-fi-hiltzik-20131027, ‘Broken
science’ http://reason.com/archives/2016/01/19/broken-science, ‘Lies, damned lies,
and medical science’ http://www.theatlantic.com/magazine/archive/2010/11/liesdamned-lies-and-medical-science/308269/, ‘Science’s big scandal’
http://www.slate.com/articles/health_and_science/science/2015/04/fake_peer_
review_scientific_journals_publish_fraudulent_plagiarized_or_nonsense.single.html,‘Academic publishing is broken’
http://www.the-scientist.com/?articles.view/articleNo/31858/title/Opinion--AcademicPublishing-Is-Broken/, ‘Scientific regress’
http://www.firstthings.com/article/2016/05/scientific-regress.
5. Trouble at the lab. The Economist, 19 Oct. 2013; http://www.webcitation.org/6g9lGFe8R.
6. Collins, Francis and Lawrence Tabak. NIH plans to enhance reproducibility, Nature 27 Jan.
2014; http://www.webcitation.org/6g9kqqJIW.
7. Horton, Richard. What is medicine’s 5 sigma? The Lancet, 11 Apr. 2015;
http://www.webcitation.org/6g9lRzki3.
8. Statistical challenges in assessing and fostering the reproducibility of scientific results: aworkshop. 26–27 Feb. 2015; http://www.webcitation.org/6g9mH9R4n.
9. Social, behavioral, and economic perspectives on robust and reliable science. Report of
the Subcommittee on Replicability in Science Advisory Committee to the National
Science Foundation Directorate for Social, Behavioral, and Economic Sciences. May
2015; http://www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf.
10. Reducing our irreproducibility. Nature, 24 Apr. 2013;
http://www.webcitation.org/6g9mPEEGA.
11. Horton, Richard. What is medicine’s 5 sigma? The Lancet, 11 Apr. 2015;
http://www.webcitation.org/6g9lRzki3.
12. Donoho, David et al. Reproducible research in computational harmonic analysis.
Computing in Science & Engineering, Jan–Feb. 2009;http://doi.ieeecomputersociety.org/10.1109/MCSE.2009.15. See also Stodden, Victoria
et al. Setting the default to reproducible in computational science research. SIAM News,
3 Jun. 2013; https://web.archive.org/web/20130802003527/http:/www.siam.org/news/news.php?id=2078.

1713. American Psychological Association. A reproducibility crisis? Monitor on Psychology, Oct.
2015; http://www.apa.org/monitor/2015/10/share-reproducibility.aspx. Open Science
Collaboration. Estimating the reproducibility of psychological science. Science, 28 Aug.
2015; http://science.sciencemag.org/content/349/6251/aac4716. Yong, Ed. Replication
studies: bad copy. Nature, 17 May 2012;
http://www.nature.com/news/replication-studies-bad-copy-1.10634. Jussim, Lee. Are
most published social psychology findings false? Heterodox Academy, 23 Feb. 2016;
http://heterodoxacademy.org/2016/02/23/are-most-published-social-psychologyfindings-false/.
14. Perspectives in Psychological Science, Nov. 2012; http://pps.sagepub.com/content/7/6.
15. Hamermesh, Daniel. Replication in economics. Institute for the Study of Labor, Apr.
2007; http://ftp.iza.org/dp2760.pdf.
16. McCullough, Bruce and Ross McKitrick, Check the numbers: the case for due diligence inpolicy Formation. Fraser Insitute: Studies in Risk & Regulation, Feb. 2009, p. 11;https://www.fraserinstitute.org/studies/check-numbers-case-for-due-diligence-inpolicy-formation.
17. Chang, Andrew and Philip Li. Is economics research replicable? Sixty published papers
from thirteen journals say ’usually not’. Finance and Economics Discussion Series
2015-083. Washington: Board of Governors of the Federal Reserve System, 2015, p. 1;
http://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf. See also:
Is research in finance and economics reproducible? The Mathematical Investor, 10 Oct.
2015; http://www.financial-math.org/blog/2015/10/is-research-in-finance-andeconomics-reproducible/.
18. Booth, Bruce. Academic bias & biotech failures. Life Sci VC, 28 Mar. 2011;https://lifescivc.com/2011/03/academic-bias-biotech-failures/.
19. Prinz, Florian. Believe it or not: how much can we rely on published data on potential
drug targets? Nature Reviews Drug Discovery, Sept. 2011;
http://www.nature.com/nrd/journal/v10/n9/full/nrd3439-c1.html.
20. Begley, Glenn and Lee Ellis. Drug development: raise standards for preclinical canner
research. Nature, 29 Mar. 2012;
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html.
21. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 2, 2011, pp. 16 and 61; http://tinyurl.com/hl34gjt.
22. Ibid. p. 8. Macdonald, Stuart. Emperor’s new clothes: the reinvention of peer review as
myth. Journal of Management Inquiry, 6 Nov. 2014, p. 11;
http://jmi.sagepub.com/content/early/2014/11/06/1056492614554773.abstract.
23. Mahoney, Michael. Publication prejudices: an experimental study of confirmatory bias
in the peer review system. Cognitive Therapy and Research, Jun. 1977, bottom of p. 162;
http://link.springer.com/article/10.1007%2FBF01173636.
24. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 2, 2011, pp. 62, 138,162,168,169,173; http://tinyurl.com/hl34gjt.
25. Ibid, p. 170.
26. Ibid, p. 174.

18

27. Goldbeck-Wood, Sandra. Evidence on peer review – scientific quality control orsmokescreen? BMJ, 2 Jan. 1999; http://www.bmj.com/content/318/7175/44.
28. Cohen, Patricia. For scholars, web changes sacred rite of peer review. New York Times, 23
Aug. 2010; http://www.webcitation.org/6fi25OUx1.
29. Smith, Richard. Peer review: a flawed process at the heart of science and journals.
Journal of the Royal Society of Medicine, Apr. 2006;
http://jrs.sagepub.com/content/99/4/178.full.
30. Mark Ware Consulting Ltd. Peer review in scholarly journals: perspective of the scholarly
community – an international study, 2008, p. 10;
http://publishingresearchconsortium.com/index.php/prc-documents/prc-researchprojects/36-peer-review-full-prc-report-final/file.
31. In Chapter 6 of Plastic Fantastic: How the Biggest Fraud in Physics Shook the ScientificWorld, Eugenie Samuel Reich reports that a 2001 review prepared for Science thatassessed a Jan Hendrik Schön paper ‘was only a paragraph long’. Science accepted
another Schön paper, she says, on the basis of a single review - ‘a clear violation of
Science’s published guidelines [which promise]. . . that research has been reviewed by atleast two experts.’
32. In ‘Early careers spent grinding teeth, not cutting them’, Times Higher Education, 25 Feb.
2005, Chris Bunting quotes a prominent academic: ‘This friend had been sent my article
to referee. He thought that, in all honesty, he could not referee it because we were so
close and he telephoned the editor to say that he would have to find someone else. The
editor responded: “Don’t be silly, this is Charles Oppenheim. We both know we aregoing to publish it anyway. This is really just a formal exercise, so could you just go
through the motions?”’ https://www.timeshighereducation.com/features/earlycareers-spent-grinding-teeth-not-cutting-them/194364.article
33. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 1, 2011, p. 205; http://tinyurl.com/hkd8h2w.
34. Baldwin, Melinda. Did Isaac Newton need peer review? Zocalo Public Square, 29 Apr.
2014; http://www.zocalopublicsquare.org/2014/04/29/did-isaac-newton-need-peerreview/ideas/nexus/.
35. Burnham, JC. The evolution of editorial peer review. JAMA, March 1990; https://www.
researchgate.net/publication/20748579_The_Evolution_of_Editorial_Peer_Review.
Kronick, David. Peer review in 18th-century scientific journalism. JAMA, 9 Mar. 1990;
http://jama.jamanetwork.com/article.aspx?articleid=380935.
36. Lavelle, John. Facts of journal publishing, IV. PMLA, Nov. 1966;
http://www.jstor.org/stable/1261117.
37. Berger, Eric. Peer review: a castle built on sand or the bedrock of scientific publishing?
Annals of Emergency Medicine, Feb. 2006. See quote at
http://www.annemergmed.com/article/S0196-0644%2805%2902102-5/abstract.
38. Rennie, Drummond. Editorial peer review: its development and rationale. In Peer Review
in Health Sciences, BMJ Books, 2003; http://tinyurl.com/z9kk2hn.
39. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 1, 2011, pp. 153, 168, 201, 209, 237–239; http://tinyurl.com/hkd8h2w.

19

House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 2, 2011, pp. 48, 62, 102, 106, 115, 124, 133, 155, 162, 167;
http://tinyurl.com/hl34gjt.
40. In a written submission to the House of Commons Science & Technology Committee,
Richard Horton, editor of The Lancet, observes: ‘our zeal for and confidence in the peer
review process seem inversely proportional to our knowledge about what it actually
does for science. Those who make big claims for peer review need to face up to this
disturbing absence of evidence’. Peer review in scientific publications, vol. 2, 2011, p. 10;
http://tinyurl.com/hl34gjt.
41. Smith, Richard. Peer review: a flawed process at the heart of science and journals.
Journal of the Royal Society of Medicine, Apr. 2006;
http://jrs.sagepub.com/content/99/4/178.full.
42. Peters, Douglas and Stephen Ceci. Peer-review practices of psychological journals: thefate of published articles submitted again. Behavioral and Brain Sciences, Jun. 1982;
http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=
6577844&fileId=S0140525X00011183.
43. Budden, Amber et al. Double-blind review favours increased representation of female
authors. Trends in Ecology and Evolution, Jan. 2008; http://tinyurl.com/hth2xpj.
44. Lloyd, Margaret. Gender factors in reviewer recommendations for manuscript
publication. Journal of Applied Behavior Analysis, Winter, 1990;
http://onlinelibrary.wiley.com/doi/10.1901/jaba.1990.23-539/abstract.
45. Yong, Ed. Replication studies: bad copy. Nature, 17 May 2012;
http://www.nature.com/news/replication-studies-bad-copy-1.10634. Matthews, David.
Amgen launches new platform to help fix scientific ’reproducibility crisis’. Times Higher
Education, 4 Feb. 2016; https://www.timeshighereducation.com/news/amgenlaunches-new-platform-help-fix-scientific-reproducibility-crisis.
46. Lehrer, Jonah. The truth wears off. The New Yorker, 13 Dec. 2010;
http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off.
47. Mahoney, Michael. Publication prejudices: an experimental study of confirmatory bias
in the peer review system. Cognitive Therapy and Research, Jun. 1977;
http://link.springer.com/article/10.1007%2FBF01173636.
48. Herbert F. Ayres, TIMS president 1982-83. INFORMS.org website;
https://www.informs.org/About-INFORMS/History-and-Traditions/Miser-HarrisPresidential-Portrait-Gallery/Herbert-F.-Ayres.
49. Armstrong, Scott. Is review by peers as fair as it appears? Interfaces, Oct. 1982, p. 7 (p. 9of the PDF); http://repository.upenn.edu/marketing_papers/114/.
50. Companario, Juan Miguel. Rejecting and resisting Nobel class discoveries: accounts byNobel Laureates. Scientometrics, 16 Apr. 2009;http://link.springer.com/article/10.1007%2Fs11192-008-2141-5.
51. Heinrich Rohrer, Nobel Prize in Physics 1986. NobelPrize.org website;
http://www.nobelprize.org/nobel_prizes/physics/laureates/1986/.
52. The letter is reproduced on page 449 of the paper at
http://www.nobelprize.org/nobel_prizes/medicine/laureates/1977/yalow-lecture.pdf

20

(page 3 of the PDF).
53. The Royal Society: ‘The reputation of the best journals is intimately involved with therigour of their peer review systems. . . ’ House of Commons Science & Technology
Committee. Peer review in scientific publications, vol. 1, 2011, p. 206;
http://tinyurl.com/hkd8h2w.
54. Smith, Richard. Peer review: a flawed process at the heart of science and journals.
Journal of the Royal Society of Medicine, Apr. 2006;
http://jrs.sagepub.com/content/99/4/178.full.
55. Retractions’ realities. Nature, 6 Mar. 2003;
http://www.nature.com/nature/journal/v422/n6927/full/422001a.html.
56. Retraction. Science. 1 Nov. 2002;
http://science.sciencemag.org/content/298/5595/961.2. See also note 31, regarding
the book Plastic Fantastic.
57. Schön scandal. Exquisitely detailed Wikipedia entry:
http://www.webcitation.org/6gAhvMtQu.
58. Journal rankings in the anesthesiology and pain medicine category;
http://www.scimagojr.com/journalrank.php?category=2703. A complete list of
retracted Reuben papers can be found at http://tinyurl.com/zpba5tb. Retraction notice:
Anesthesia & Analgesia, Apr. 2009; http://journals.lww.com/anesthesiaanalgesia/Fulltext/2009/04000/Retraction_Notice.54.aspx.
59. Retraction. Science. 2 Dec. 2011;https://nofrakkingconsensus.files.wordpress.com/2016/03/2011sciencemagazineretraction-diederik-stapel-case-litter-racism-fraud.pdf. Stapel, Diederik and Siegwart
Lindenberg. Coping with chaos: how disordered contexts promote stereotyping anddiscrimination. Science, 8 Apr. 2011.
60. Bhattacharjee, Yudhijit. The mind of a con man. New York Times Magazine, 26 Apr. 2013;
http://www.nytimes.com/2013/04/28/magazine/diederik-stapels-audaciousacademic-fraud.html.
61. Levelt Committee, Noort Committee, Drenth Committee. Flawed science: the fraudulent
research practices of social psychologist Diederik Stapel. Final Report, 28 Nov. 2012,
pp. 25–31; http://tinyurl.com/hcgzw2c.
62. Ibid, p. 48.
63. Ibid, pp. 53–54.
64. Ibid, p. 39.
65. Ibid, p. 47. The word ’violations’ is used 11 times in this 104-page report.
66. Ibid, p. 53.
67. Begley, Glenn and Lee Ellis. Drug development: raise standards for preclinical canner
research. Nature, 29 Mar. 2012;
http://www.nature.com/nature/journal/v483/n7391/full/483531a.html.
68. Bedeian, Arthur. The manuscript review process: the proper roles of authors, referees,and editors. Journal of Management Inquiry, 1 Dec. 2003;
http://jmi.sagepub.com/content/12/4/331.abstract.

21

69. Nieuwenhuis, Sander et al. Erroneous analyses of interactions in neuroscience: aproblem of significance. Nature Neuroscience, 26 Aug. 2011;
http://jmi.sagepub.com/content/12/4/331.abstract.
70. Yirka, Bob. Study finds statistical error in large numbers of neuroscience papers.
Phys.org, 13 Sept. 2011;
http://phys.org/news/2011-09-statistical-error-large-neuroscience-papers.html.
71. Allison, David et al. A tragedy of errors: mistakes in peer-reviewed papers are easy tofind but hard to fix. Nature, 3 Feb. 2016; http://www.nature.com/polopoly_fs/1.19264!
/menu/main/topColumns/topLeftColumn/pdf/530027a.pdf. Hauner, Hans. Editorial:
Quality management in scientific publishing – the importance to critically scrutinize
scientific work. Obesity Facts, 31 Mar. 2015;
https://www.karger.com/Article/FullText/381481.
72. Trouble at the lab. The Economist, 19 Oct. 2013; http://www.webcitation.org/6g9lGFe8R.
73. McNutt, Marcia. Raising the bar. Science, 4 Jul. 2014;
http://science.sciencemag.org/content/345/6192/9.full.
74. Ioannidis, John. Why most published research findings are false. PLoS Medicine, 30 Aug.
2005; http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124.
75. Van Noorden, Richard. Science joins push to screen statistics in papers. Nature, 3 Jul.
2014; http://www.nature.com/news/science-joins-push-to-screen-statistics-in-papers1.15509#/.
76. Wasserstein, Ronald and Nicole Lazar. The ASA’s statement on p-values: context,
process, and purpose. American Statistical Association website, 7 Mar. 2016;
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108.
77. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 2, 2011, p. 109; http://tinyurl.com/hl34gjt. 1.5 million statistic is
supplied in written evidence from the Publishers Association.
78. See also Nuzzo, Regina. Scientific method: statistical errors. Nature, 12 Feb. 2014;
http://www.nature.com/news/scientific-method-statistical-errors-1.14700.
79. Van Noorden, Richard. Publishers withdraw more than 120 gibberish papers. Nature, 25
Feb. 2014; http://www.nature.com/news/publishers-withdraw-more-than-120gibberish-papers-1.14763.
80. Hermenegildo, Manuel. Conferences vs. journals in CS, what to do? Position paper for
Dagstuhl meeting 12452: Publication Culture in Computing Research, 7 Nov. 2012;
http://tinyurl.com/h5vcwyb.
81. Smith, Richard. Peer review: a flawed process at the heart of science and journals.
Journal of the Royal Society of Medicine, April 2006;
http://jrs.sagepub.com/content/99/4/178.full.
82. Smith, Richard. The Trouble with Medical Journals. Royal Society of Medicine Press, 2006,
Chapter 12.
83. Smith, Richard. A ripping yarn of editorial misconduct. BMJ blogs, 21 Oct. 2008;
http://blogs.bmj.com/bmj/2008/10/21/richard-smith-a-ripping-yarn-of-editorialmisconduct/.

2284. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 1, 2011, p. 118; http://tinyurl.com/hkd8h2w.
85. Jump, Paul. Journal editors’ anonymous reviews criticised by COPE. Times Higher
Education, 4 Apr. 2013; https://www.timeshighereducation.com/news/journal-editorsanonymous-reviews-criticised-by-cope/2002937.article. COPE guidelines may be
downloaded at
http://publicationethics.org/files/Ethical_guidelines_for_peer_reviewers_0.pdf.
86. NASA Media Advisory. NASA sets news conference on astrobiology discovery;
http://www.nasa.gov/home/hqnews/2010/nov/HQ_M10-167_Astrobiology.html.
87. NASA-funded research discovers life built with toxic chemical. NASA website, 2 Dec.
2010;
http://www.nasa.gov/topics/universe/features/astrobiology_toxic_chemical.html.
88. TED2011: The Rediscovery of Wonder program schedule;
https://conferences.ted.com/TED2011/program/schedule.php. On Mar 2, Wolfe-Simon(who is described there as a geobiochemist), was part of a six-member panel discussionof ’Deep Mystery’.
89. Kluger, Jeffrey. Felisa Wolfe-Simon: The 2011 Time 100. 21 Apr. 2011; http://content.time.
com/time/specials/packages/article/0,28804,2066367_2066369_2066250,00.html.
90. Redfield, Rosie. Arsenic-associated bacteria (NASA’s claims). RRResearch, 4 Dec. 2010;
http://rrresearch.fieldofscience.com/2010/12/arsenic-associated-bacteria-nasas.html.
91. Zimmer, Carl. ’This paper should not have been published’. Slate, 7 Dec. 2010;
http://www.slate.com/articles/health_and_science/science/2010/12/this_paper_
should_not_have_been_published.html.
92. Zimmer, Carl. ArsenicLife goes longform, and history gets squished. Discover Magazine,
30 Sep. 2011; http://blogs.discovermagazine.com/loom/2011/09/30/arseniclife-goeslongform-and-history-gets-squished/.
93. NASA’s arsenic microbe science slammed. CBCnews, 6 Dec. 2010; http://www.cbc.ca/news/technology/nasa-s-arsenic-microbe-science-slammed-1.909147.
94. Zimmer, Carl. ArsenicLife goes longform, and history gets squished. Discover Magazine,
30 Sep. 2011; http://blogs.discovermagazine.com/loom/2011/09/30/arseniclife-goeslongform-and-history-gets-squished/.
95. Dobbs, David. The wrong stuff: NASA dismisses arsenic critique because critical priest
not standing on altar. Wired, 7 Dec. 2010; http://www.wired.com/2010/12/the-wrongstuff-nasa-dismisses-arsenic-critique-because-critical-priest-not-standing-on-altar/.
96. Eisen is quoted in Zimmer, Carl. ’This paper should not have been published’. Slate, 7
Dec. 2010; http://www.slate.com/articles/health_and_science/science/2010/12/this_paper_should_not_have_been_published.html.
97. Hawkes, Nigel. Peer-reviewed journals aren’t worth the paper they’re written on.
Independent, 20 Aug. 2010;
http://www.independent.co.uk/voices/commentators/nigel-hawkes-peer-reviewedjournals-arent-worth-the-paper-theyre-written-on-2058067.html.
98. Professors Richard Wilkinson and Kate Pickett, authors of The Spirit Level, reply to critics,
Equality Trust website, 28 Jul. 2010; http://tinyurl.com/jjpkro7.

23

99. The Spirit Level: Why equality is better for everyone. Amazon UK website, accessed 21 Mar.
2016; http://www.amazon.co.uk/Spirit-Level-Equality-Better-Everyone/dp/0241954290
100. Hawkes, Nigel. A bad case of bias against Caesareans: behind the numbers.
Independent, 29 Jan. 2010;
http://www.independent.co.uk/life-style/health-and-families/health-news/nigelhawkes-a-bad-case-of-bias-against-caesareans-1883667.html.
101. Hawkes, Nigel. Peer-reviewed journals aren’t worth the paper they’re written on.
Independent, 20 Aug. 2010;
http://www.independent.co.uk/voices/commentators/nigel-hawkes-peer-reviewedjournals-arent-worth-the-paper-theyre-written-on-2058067.html.
102. In the words of publishing giant Elsevier: ‘Publishers have robust procedures to takeaction and to correct the scientific record when errors or fraud are detected’. House ofCommons Science & Technology Committee. Peer review in scientific publications, vol.
1, 2011, p. 220. See also pp. 201, 203, 205, 206, 219; http://tinyurl.com/hkd8h2w.
103. Allison, David et al. A tragedy of errors: mistakes in peer-reviewed papers are easy tofind but hard to fix. Nature, 3 Feb. 2016; http://www.nature.com/polopoly_fs/1.19264!
/menu/main/topColumns/topLeftColumn/pdf/530027a.pdf.
104. Ibid.
105. Wolfe-Simon, Felisa et al. A bacterium that can grow by using arsenic instead ofphosphorus. Science, 3 Jun. 2011 (date here refers to publication in the hardcopy
journal); http://science.sciencemag.org/content/332/6034/1163.full.
106. Editor’s note and technical comments. Science, 3 Jun. 2011 (date refers to thehardcopy; text of this letter says online publication occurred on 27 May);
http://science.sciencemag.org/content/332/6034/1149.1.
107. Reaves, Marshall Louis et al. Absence of detectable arsenate in DNA fromarsenate-grown DGAJ-1 cells. Science, 27 Jul. 2012;
http://science.sciencemag.org/content/337/6093/470. Erb, Tobias. GRAJ-1 is anarsenate-resistant, phosphate-dependent organism. Science, 27 Jul. 2012;
http://science.sciencemag.org/content/337/6093/467.
108. Editorial statement from the journal Science regarding new fingings on the GFAJ-1
bacterium. 8 Jul. 2012;https://nofrakkingconsensus.files.wordpress.com/2016/03/editorial-statement-fromthe-journal-science-regarding-new-findings-on-the-gfaj-1-bacterium-8-july-2012.pdf.
109. Archived snapshot of Science website. 18 Mar. 2016;
http://www.webcitation.org/6g6kZrqqK.
110. These remarks from Science are quoted in Moskowitz, Clara, Debate reignited overclaim of arsenic-based life. Live Science, 27 May 2011;
http://www.webcitation.org/6g7kDNO1N and in Vergano, Dan, Glowing reviews on’arseniclife’ spurred NASA’s embrace. USA Today, 12 Feb. 2013;
http://www.usatoday.com/story/tech/columnist/vergano/2013/02/01/arseniclife-peerreviews-nasa/1883327/.
111. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 1, 2011, pp. 153, 168, 201, 209, 237–239; http://tinyurl.com/hkd8h2w.

24

House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 2, 2011, pp. 48, 62, 102, 106, 115, 124, 133, 155, 162, 167;
http://tinyurl.com/hl34gjt.
112. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 2, 2011, p. 144; http://tinyurl.com/hl34gjt.
113. Feynman, Richard. Cargo cult science: Commencement address at California Instituteof Technology, 1974; ttp://calteches.library.caltech.edu/51/2/CargoCult.htm.
114. Smith, Richard. Peer review: a flawed process at the heart of science and journals.
Journal of the Royal Society of Medicine, Apr. 2006;
http://jrs.sagepub.com/content/99/4/178.full.
115. House of Commons Science & Technology Committee. Peer review in scientificpublications, vol. 1, 2011, p. 219; http://tinyurl.com/hkd8h2w. Written evidence from
Elsevier says there are ‘c. 25,000 peer-reviewed journals’.
116. Social, behavioral, and economic perspectives on robust and reliable science. Report of
the Subcommittee on Replicability in Science Advisory Committee to the National
Science Foundation Directorate for Social, Behavioral, and Economic Sciences. May
2015; http://www.nsf.gov/sbe/AC_Materials/SBE_Robust_and_Reliable_Research_Report.pdf.
117. Global Climate Change Commission. Minutes. 11 Feb. 2008, see pp. 2 and 4;
http://www.webcitation.org/6g9EqBf9R.
118. Astonishingly, the claim that the IPCC relies solely on peer-reviewed sources is actually
untrue. Roughly one-third of the sources cited by the IPCC’s 2001 and 2007 assessment
reports are to non-peer-reviewed material. See Chapter 2, p. 16 of the 2010
Interacademy Council report on the IPCC(http://bit.ly/29duuvj), the 2010 Citizen Audit
Report (http://www.noconsensus.org/ipcc-audit/IPCC-report-card.php), and my 2011blog post, Speeding Tickets and the IPCC
(https://nofrakkingconsensus.com/2011/04/19/speeding-tickets-and-the-ipcc/).
119. Other examples, most unrelated to climate, are discussed in McCullough, Bruce andRoss McKitrick, Check the numbers: the case for due diligence in policy formation.
Fraser Insitute: Studies in Risk & Regulation, Feb. 2009;https://www.fraserinstitute.org/studies/check-numbers-case-for-due-diligence-inpolicy-formation.
120. Michaels, Patrick and David Wojick. Climate modeling dominates climate science. Cato
Institute, 13 May 2016;
http://www.cato.org/blog/climate-modeling-dominates-climate-science. Hulme, Mike.
How climate models gain and exercise authority. In: Hastrup, Kirsten, The Social Life of
Climate Change Models, 2013, p. 30. Peckham, Matt. The world’s most powerful climatechange supercomputer powers up. Time, 17 Oct. 2012;
http://techland.time.com/2012/10/17/the-worlds-most-powerful-climate-changesupercomputer-powers-up/.
121. Reproducibility Initiative;
http://validation.scienceexchange.com/#/reproducibility-initiative,
ScienceExchange.com.

25

122. See Scienceexchange.com web pages on: Reproducibility Project: Cancer Biology(http://validation.scienceexchange.com/#/cancer-biology), Movember Foundation –PCF Scientific Reproducibility Initiative(http://validation.scienceexchange.com/#/prostate-cancer), and the Antibody
Validation Project (http://validation.scienceexchange.com/#/antibody-validation).

GWPF REPORTS
1Montford
The Climategate Inquiries
2
RidleyThe Shale Gas Shock3
HughesThe Myth of Green Jobs
4
McKitrick
What Is Wrong With the IPCC?
5
BookerThe BBC and Climate Change
6
Montford
Nullius in Verba: The Royal Society and Climate Change
7
Goklany
Global Warming Policies Might Be Bad for Your Health8
HughesWhy Is Wind Power So Expensive?
9
LilleyWhat Is Wrong With Stern?
10 Whitehouse
The Global Warming Standstill
11 Khandekar
The Global Warming-Extreme Weather Link
12 Lewis and Crok
Oversensitive
13 Lewis and Crok
A Sensitive Matter
14 Montford and Shade Climate Control: Brainwashing in Schools
15 De Lange and Carter Sea-level Change: Living with Uncertainty
16 Montford
Unintended Consequences of Climate Change Policy
17 Lewin
Hubert Lamb and the Transformation of Climate Science
18 Goklany
Carbon Dioxide: The Good News
19 Adams
The Truth About China
20 LaframboisePeer Review: Why Scepticism is Essential
21 Constable
Energy Intensive Users: Climate Policy Casualties
22 Lilley
The Climate Change Act: The Cost to the Country

The Global Warming Policy Foundation is an all-party and non-party thinktank and a registered educational charity which, while openminded on
the contested science of global warming, is deeply concerned about thecosts and other implications of many of the policies currently being advocated.
Our main focus is to analyse global warming policies and their economic
and other implications. Our aim is to provide the most robust and reliableeconomic analysis and advice. Above all we seek to inform the media,
politicians and the public, in a newsworthy way, on the subject in generaland on the misinformation to which they are all too frequently being subjected at the present time.
The key to the success of the GWPF is the trust and credibility that we haveearned in the eyes of a growing number of policy makers, journalists andthe interested public. The GWPF is funded overwhelmingly by voluntary
donations from a number of private individuals and charitable trusts. In
order to make clear its complete independence, it does not accept gifts
from either energy companies or anyone with a significant interest in anenergy company.
Views expressed in the publications of the Global Warming Policy Foundation are those of the authors, not those of the GWPF, its trustees, its
Academic Advisory Council members or its directors.

Published by the Global Warming Policy Foundation
For further information about GWPF or a print copy of this report,
please contact:
The Global Warming Policy Foundation
55 Tufton Street, London, SW1P 3QL
T 0207 3406038 M 07553 361717
www.thegwpf.org
Registered in England, No 6962749
Registered with the Charity Commission, No 1131448